On optimal designs for non-regular models
Yi Lin Ryan Martin Min Yang


Impact of the malicious input data modification on the efficiency of 
quantum spatial search
    [
    
============================================================================================





Classically, Fisher information is the relevant object in defining optimal experimental designs.  However, for models that lack certain regularity, the Fisher information does not exist and, hence, there is no notion of design optimality available in the literature.  This article seeks to fill the gap by proposing a so-called Hellinger information, which generalizes Fisher information in the sense that the two measures agree in regular problems, but the former also exists for certain types of non-regular problems.  We derive a Hellinger information inequality, showing that Hellinger information defines a lower bound on the local minimax risk of estimators.  This provides a connection between features of the underlying model—in particular, the design—and the performance of estimators, motivating the use of this new Hellinger information 

for non-regular optimal design problems.  Hellinger optimal designs are derived for several non-regular regression problems, with numerical results empirically demonstrating the efficiency of these designs compared to alternatives.   



Keywords and phrases: E-optimality; experimental design; Fisher information; Hellinger distance; information inequality. 







§ INTRODUCTION

  

 Optimal experimental design is a classical problem with substantial recent developments.  For example, <cit.>, <cit.>, <cit.>, and <cit.> studied optimal designs for dose-response models; <cit.> and <cit.> investigated optimal designs for correlated observations; 
<cit.> and <cit.>  studied robustness issues in optimal designs; <cit.>, <cit.>, <cit.>, <cit.>, and <cit.> studied optimal discrimination designs; <cit.> studied optimal design for additive partially nonlinear models; <cit.>, <cit.>, <cit.>, and <cit.> investigated algorithms for deriving optimal designs; and <cit.>, <cit.>, <cit.>, <cit.>, and <cit.> built a new theoretical framework for studying optimal designs.  The focus of these developments has been exclusively on regular models that enjoy certain normal features asymptotically, 
such as generalized linear models.  However, certain non-regular models may be appropriate in practical applications <cit.>.  In particular, <cit.> describes a class of non-regular linear regression models, 

    y = x^⊤θ + ,

where the error  is non-negative, which implies a non-regular model for y, given x, since its distribution has θ-dependent support. Such models are useful if the goal is to study extremes; for example, x^⊤θ might represent the lower bound on remission time when a patient is subjected to treatment settings described by the vector x. To date, there is no literature on optimal designs for cases like this, and the goal of this paper is to fill this gap by developing an approach to optimal design in non-regular problems.  

Towards formulating a design problem in a non-regular model, the first obstacle is that the Fisher information matrix—the fundamental object in the classical optimal design context—does not exist.  To overcome this, we draw inspiration from recent work on the development of non-informative priors in the Bayesian context, thereby backtracking the path taken by <cit.> and <cit.> from information in an experiment to non-informative priors. In particular, <cit.> proposes an alternative to Fisher information and generalizes the non-informative prior construction of Jeffreys.  An important feature of the Fisher information is how it describes the local behavior of the Hellinger distance (see Section <ref>), leading to its connection to estimator quality via the information inequality.  Unfortunately, the role that Shemyakin's information 

plays in the local approximation of Hellinger distance for multi-parameter models remains unclear; 

see Remark <ref>. 

Since a connection to the quality of estimators is essential to our efforts to define a meaningful notion of optimal designs, we take an alternative approach where the focus is on a local approximation of Hellinger distance.





We start by looking at the local behavior of the squared Hellinger distance between models P_θ and P_ϑ, for ϑ≈θ.  In the regular cases, there is a local quadratic approximation to the squared distance and the Fisher information matrix appears in the approximating quadratic form.  In non-regular problems, by definition, the squared Hellinger distance is not locally quadratic, so there is no reason to expect that an “information matrix” can be extracted from this approximation.  In fact, not being differentiable in quadratic mean implies that the Hellinger distance is continuous at θ, but not differentiable, so important features of the local approximation will generally depend on both the magnitude and the direction of the departure of ϑ from θ.  From the local Hellinger distance approximation for a given direction, we define a direction-dependent Hellinger information, which is additive like Fisher information for independent data, and establish a corresponding information inequality that suitably lower-bounds the risk function of an arbitrary estimator along that direction.  The direction-dependence is removed via profiling, and the result is a locally minimax lower bound on the risk of arbitrary estimators, which is inversely related to our direction-free Hellinger information. Therefore, just like in the familiar Cramér–Rao inequality for regular models, larger Hellinger information means a smaller lower bound and, consequently, better estimation in terms of risk.  

The established connection between our Hellinger information for non-regular models and the quality of estimators provides a natural path to approach the optimal design problem.  In particular, our Hellinger information depends on the design, so we define the optimal design as one that maximizes the Hellinger information.  The intuition, just like in the regular case, is that maximizing the information minimizes the lower bound on the risk, thereby leading to improved estimation.  If the model happens to be regular, then our proposed optimal design corresponds to the classical E-optimal design that maximizes the minimum eigenvalue of the Fisher information matrix, so the new approach at least has intuitive appeal.  After formally defining the notion of optimal design in this context, we develop some novel theoretical results, in particular a complete class theorem for symmetric designs in the context of non-regular polynomial regression.  This theorem, along with some special cases presented in Propositions <ref>–<ref>, suggests the potential for a line of developments parallel to that for regular models.  

The remainder of the paper is organized as follows.  Section <ref> sets our notation and briefly reviews the Fisher information and its properties under regularity conditions.  We relax those regularity conditions in Section <ref> and develop a notion of Hellinger information for certain non-regular models.  The main result of the paper, Theorem <ref>, establishes a connection between this Hellinger information and the quality of estimators, thus paving the way for a framework of optimal designs for non-regular models in Section <ref>.  Some specific non-regular regression models are considered in Section <ref>, and we derive some analytical optimality results and some numerical demonstrations of the improved efficiency of the optimal designs over other designs.  Some concluding remarks are given in Section <ref> and proofs of the two main theorems are presented in Appendix <ref>; the remaining details are given in the Supplementary Material <cit.>.






§ REVIEW OF INFORMATION IN REGULAR MODELS



The proposed model assumes that the -valued observations Y_1,…,Y_n are independent, and the marginal distribution of Y_i is P_i,θ, where θ is a fixed and unknown parameter in Θ⊆^d.  For example, P_i,θ might be a distribution that depends on both the parameter θ and a fixed covariate vector x_i.  We will further assume that, for each i=1,…,n, P_i,θ has a density p_i,θ with respect to a fixed dominating σ-finite measure μ.  When the index i is not important, and there is no risk of confusion, we will drop the index and write simply p_θ for the density function with respect to μ.  

It is common to assume that the model is regular in the sense that θ↦ p_i,θ(y) is smooth for each y, and that θ-derivatives of expectations can be evaluated by interchanging differentiation and integration.  For example, under conditions (6.6) in <cit.>, one can define the d × d Fisher information matrix I_i(θ), whose (k,ℓ) entry is given by 



    E_θ{∂/∂θ_klog p_i,θ(Y_i) ·∂/∂θ_ℓlog p_i,θ(Y_i) },    k,ℓ = 1,…,d.

The Fisher information matrix can be defined in broader generality for families of distributions with a differentiability in quadratic mean property <cit.>.  That is, assume that there exists a function ℓ̇_θ, typically the gradient of log p_θ, taking values in ^d, such that 

    ∫( p_θ+^1/2 - p_θ^1/2 - 12 ^⊤ℓ̇_θ p_θ^1/2)^2  dμ = o(^2),   → 0,

where · denotes the ℓ_2-norm.  Then the Fisher information matrix exists and is given by the formula I(θ) = ∫ℓ̇_θℓ̇_θ^⊤  p_θ dμ.  If we let H(P_θ,P_ϑ) denote the Hellinger distance and define h as 

    h(θ; ϑ) ≡ H^2(P_θ, P_ϑ) := ∫ (p_θ^1/2 - p_ϑ^1/2)^2  dμ = 2 - 2 ∫ (p_θ p_ϑ)^1/2 dμ,

then the above condition amounts to h being locally quadratic:

    h(θ; θ+) = 14  ^⊤ I(θ)  + o(^2).


Therefore, a model is regular if the squared Hellinger distance is locally approximately quadratic, with the Fisher information matrix characterizing that quadratic approximation.  This is the description of Fisher information that we will attempt to extend to the non-regular case below.  

Recall, also, that Fisher information is additive under independence.  That is, if Y_1,…,Y_n are independent, with Y_i ∼ p_i,θ, regular as above for each i, then the Fisher information in the sample of size n satisfies 

    _n(θ) = ∑_i=1^n I_i(θ),

where I_i(θ) is the Fisher information matrix in (<ref>) based on p_i,θ alone.  This property has a nice interpretation: larger samples have more information.  

Under differentiability in quadratic mean, one can prove an information inequality which states that, for any unbiased estimator T=T(Y_1,…,Y_n) of m(θ) = E_θ(T) ∈ with finite second moment, the variance is lower-bounded and satisfies

    V_θ(T) ≥ṁ(θ)^⊤ _n(θ)^-1 ṁ(θ),

where ṁ(θ) is the gradient of m(θ); see <cit.>.  The information inequality above, and its various extensions, establishes a fundamental connection between the quality of an estimator—in this case, the variance of an unbiased estimator—and the Fisher information matrix.  This connection has been essential to the development of optimal design theory and practice since the quality of an estimator can be “optimized” by choosing a design that makes the quadratic form in the lower bound as small as possible, or the Fisher information as large as possible.    



Finally, differentiability in quadratic mean implies local asymptotic normality <cit.> which is almost all one needs to show that maximum likelihood estimators are efficient in the sense that they attain the information inequality lower bound <cit.>.  Therefore, in sufficiently regular problems, there is a general procedure for constructing high-quality estimators, and that the quality of such estimators is controlled by the Fisher information matrix.  The remainder of this paper is concerned with non-regular cases

and, unfortunately, these differ from their regular counterparts in several fundamental ways.  First, the Fisher information is not well-defined in non-regular cases, so we have no general way of measuring the quality of estimators.  Second, one cannot rely on maximum likelihood for constructing good estimators.  For example, Le Cam writes <cit.>


The author is firmly convinced that a recourse to maximum likelihood is justifiable only when one is dealing with families of distributions that are extremely regular. The cases in which maximum likelihood estimates are readily obtainable and have been proved to have good properties are extremely restricted.

Therefore, to achieve our goals, we need a measure of information that is flexible enough to handle non-regular problems and is connected to estimation quality in general, but does not depend on a particular estimator.  The Hellinger information, defined in Section <ref>, will meet these criteria and will provide a basis for defining optimal designs in non-regular problems. 







§ INFORMATION IN NON-REGULAR MODELS





 §.§ Definition and basic properties



To start, we consider the scalar case with d=1.  Suppose that there exists a constant α∈ (0,2] such that, for each θ, the limit J(θ) = lim_→ 0 ||^-α h(θ; θ + ) exists, is finite, and non-zero.  If such an α exists, then it must be unique; but there are cases where existence fails, e.g., when θ is not identifiable, so that h(θ, θ+) ≡ 0 for all sufficiently small .  The case α=2 corresponds to differentiable in quadratic mean and, hence, “regular,” while α∈ (0,2) corresponds to “non-regular.” Differentiability of ϑ↦ H(P_θ, P_ϑ) or lack thereof determines a model's regularity, so the largest value α can take is 2; otherwise, the limit is infinite. From the above limit, there is a local approximation, 

    h(θ;ϑ) = J(θ) |θ-ϑ|^α + o(|θ-ϑ|^α).

This resembles the local Hölder condition considered in <cit.>.  We call α the index of regularity and J(θ) the Hellinger information.  Of course, if α=2, then J(θ) is proportional to I(θ), the Fisher information.  Next are a few quick examples, all with α=1. 


  * If P_θ=(0,θ), θ > 0, then J(θ) = θ^-1.


  * If P_θ=(θ^-1, θ), θ > 1, then J(θ) = (θ^2 + 1) {θ (θ^2 - 1)}^-1. 


  * If P_θ=(θ; θ^2), θ > 1, then J(θ) = (2θ + 1) {θ (θ - 1)}^-1.


A class of non-regular models of particular interest to us here are those in <cit.> based on location shifts of distributions supported on the positive half-line.  Consider a density p_0 on (0,∞) that satisfies

    p_0(y) = β  c   y^β - 1,   as y → 0,

where β≥ 1 and c=c(β) ∈ (0,∞).  For example, the gamma and Weibull families, with shape parameter β and scale σ, have c = {βσ^βΓ(β)}^-1 and c=σ^-β, respectively.  The next result identifies the regularity index α and the Hellinger information J(θ) for this class of location parameter problems, with p_θ(y) = p_0(y-θ). It shows that α need not be an integer and the Hellinger information, like Fisher's, is constant in location models. When β≥ 2, the model is regular—with α=2 and the Fisher information defined as usual—so we focus here on the non-regular case with β∈ [1,2).  



Let p_0 satisfy (<ref>) with β∈ [1,2).  If, for some Δ > 0, 

    ∫_Δ^∞( d/dylog p_0(y) )^2 p_0(y)  dy < ∞,

then α = β and J(θ) ≡ c{1 + β  r(β)}, where c is as in (<ref>) and  

    r(β) = ∫_0^∞{(w+1)^(β-1)/2 - w^(β-1)/2}^2  dw.


 

See Section S2.1 in the Supplementary Material.


<cit.> show that, in this case, h(θ;θ + )= O(||^β) as → 0, but they do not identify J(θ).  

Similar results have appeared elsewhere in the literature on non-regular models; our condition (<ref>) is basically the same as Condition C_5 in <cit.>, which is basically the same as Assumption 9 in <cit.>.  

Turning to the general, non-regular multi-parameter case, where Θ is an open subset of ^d, defining Hellinger information requires some additional effort. 

In particular, non-regularity implies that the familiar local quadratic approximation of h fails, so we should not expect to have an “information matrix” to describe the local behavior in such cases.   In fact, h(θ; ϑ) depends locally on the direction along which ϑ approaches θ, so there is no “direction-free” summary of the local structure and, hence, no “information matrix”; see Remark <ref>.  But this lack of a convenient quadratic approximation need not stop us from defining a suitable Hellinger information.  



Let Θ be an open subset of ^d, for d ≥ 1, and let u denote a generic direction, a d-vector with u=1.  Suppose there exists α∈ (0,2] such that, for all θ∈Θ and all directions u, the following limit exists and is neither 0 nor ∞:

    lim_→ 0 h(θ; θ +  u) /||^α= J(θ; u).

Then, the following local approximation holds: 

    h(θ; θ + u) = J(θ; u)   ||^α + o(||^α),   → 0.

This defines the index of regularity α and the Hellinger information J(θ; u) at θ in the direction of u. 


Since the approximation (<ref>) is in terms of ||, it follows that J(θ; u) = J(θ; -u), so J(θ;u) really only depends on the line defined by u.  If d=1, then there is only one line, i.e., u=± 1, hence, for the scalar case, we can drop the u argument entirely and write J(θ) as described above.  It is also worth pointing out that Definition <ref> assumes that a single index α suffices to describe the regularity of a model with a d-dimensional parameter.  This is appropriate for the kinds of regression models we have in mind here, but can be a limitation in other cases; see Remark <ref> below.  

As a quick example, let P_θ = (θ_1,θ_1 + θ_2), where θ_1 ∈ and θ_2 > 0.  In this form, θ_1 and θ_2 are location and scale parameters, respectively.  If u=(u_1,u_2) is a generic vector on the unit circle, then J(θ; u) = θ_2^-1 g(u),
 
where g(u) has a form which is slightly too complicated to present here; see Section S1 in the Supplementary Material.  This expression agrees with the familiar properties of Fisher information for location–scale models. 





Although we do not define an “information matrix” in the non-regular case (see Remark <ref>), when the model is regular, i.e., when α=2, there are still some connections between our Hellinger information and the familiar Fisher information.  In particular, J(θ; u) is a quadratic form involving the Fisher information I(θ) and the direction u.  This gives an alternative explanation of how the regular models admit a separation of the dependence on θ and on the direction u of departure from θ.  



For a regular model, with α=2, if I(θ) denotes the d × d Fisher information matrix, then J(θ; u) = 14 u^⊤  I(θ)   u.


Another useful and familiar feature of Fisher information that also holds for Hellinger information is the reparametrization formula (Proposition <ref>), which comes in handy for regression problems where the natural parameter is expressed as a function of covariates and another parameter.





 §.§ Hellinger information inequality



We now return to our original setup where Y_1,…,Y_n are independent, but not necessarily identically distributed, with Y_i ∼ P_i,θ, i=1,…,n, and θ is an unknown parameter taking values in an open subset Θ of ^d for some d ≥ 1.    Let P_θ^n denote the joint distribution of Y^n=(Y_1,…,Y_n).  Motivated by the regression problems below, we assume that each P_i,θ has the same index of regularity, α∈ (0,2]. Following our intuition from the regular case, define the Hellinger information at θ, in the direction of u, based on the sample of size n, as 

    _n(θ; u) = ∑_i=1^n J_i(θ; u).

where J_i(θ; u) is the Hellinger information based on P_i,θ as described above.  See Remark <ref> for more on this additivity property. Theorem <ref> below will establish a suitable connection between _n(θ; u) and the quality of an estimator, and this will provide the necessary foundation for defining optimal designs for non-regular models.  

Suppose the goal is to estimate ψ(θ), where ψ: ^d →^q, q ≤ d, is sufficiently smooth.  Let T_n = T(Y^n) be an estimator of ψ(θ), and measure its quality by the risk

    R_ψ(T_n, θ) = E_θ^nT_n - ψ(θ)^2,

the q-vector version of mean square error, where expectation, E_θ^n, is with respect to P_θ^n.  This covers the case where ψ(θ)=θ and q=d, so that interest is in the full parameter θ, and the case where ψ(θ) is a single component of θ and q=1, as well as other intermediate cases.  Next is the aforementioned lower bound on the risk in terms of the total Hellinger information.  




Let Y^n=(Y_1,…,Y_n) consist of independent observations with Y_i ∼ P_i,θ, i=1,…,n.  Let α∈ (0,2] denote the common index of regularity, and _n(θ; u) the corresponding Hellinger information in (<ref>).  Let ψ: Θ→^q be a differentiable function with full-rank q × d derivative matrix D_ψ(θ), and let T_n = T(Y^n) be any estimator of ψ(θ) with risk function defined in (<ref>). If _n,u = {3 _n(θ; u)}^-1/α, and

    lim_n →∞inf_u [ n^-1_n(θ; u)] > 0,

then, for all large n, 






    inf_T_nsup_ϑ∈ B_n(θ) R_ψ(T_n, ϑ) ≳[ inf_u  {D_ψ(θ)   u^-α _n(θ; u) }]^-2/α,

where B_n(θ) ⊂Θ is the region whose boundary is determined by the union of {θ + _n,u u} over all directions u.



See Appendix <ref>.  


Two very brief comments: first, the universal constant hidden in “≳” is known and given in the proof; second, there is nothing special about “3” in the definition of _n,u, any number strictly greater than 2 would suffice.

Some additional comments about the interpretation of Theorem <ref> are in order.  First, the reason for taking supremum over a small “neighborhood" of θ is that a lucky choice of T_n ≡ψ(θ) has excellent performance at θ, but poor performance at a nearby ϑ. The theorem basically says that, if one looks at a locally uniform measure of risk, which prevents  “cheating" towards or luck at a particular θ, then one cannot have smaller risk than that in the lower bound (<ref>).  The classical Cramér–Rao lower bound uses unbiasedness of the estimator to prevent this kind of cheating/luck.

To assess the sharpness of the bound in (<ref>) when regularity conditions do not apply, consider the case where q=1, so that ψ(θ) is a scalar function.  For the rate, if we consider the identically independently distributed case, so that _n(θ; u) = n J_1(θ; u), then it follows that the lower bound is of order n^-2/α, which agrees with the known minimax rate for estimators in non-regular models <cit.>. Therefore, the bound cannot be improved in terms of dependence on the sample size. To assess the quality of the lower bound in terms of its dependence on θ, if the observations come from (0,θ), which has α=1 and J(θ)=θ^-1, the maximum likelihood estimator is the sample maximum, and its mean square error is given by 

    θ^2 n/(n+1)^2(n+2) + (θ n/n+1 - θ)^2.

Asymptotically, this expression is proportional to θ^2 n^-2, which agrees with our lower bound. Therefore, up to universal constants, the bound in Theorem <ref> is sharp.  Whether there exists an estimator that can attain the bound exactly or asymptotically is unclear in general; see Remark <ref>. 

It is worth stating the special case where α=2 as a corollary to Theorem <ref>.  This reveals some connection to the classical Cramér–Rao bound, even though we do not have access to an information matrix, and demonstrates the generality of our result.  



When α=2, if ψ: Θ→^q has q × d derivative matrix D_ψ(θ) of rank q ≤ d, and _n(θ) is the positive definite d × d Fisher information matrix, then the lower bound in (<ref>) is proportional to 

    λ_max{ D_ψ(θ) _n(θ)^-1 D_ψ(θ)^⊤},


where λ_max(A) denotes the maximal eigenvalue of a matrix A.  



See Section S2.2 in the Supplementary Material. 


For comparison to the classical setting, if we take ψ(θ) = θ, then the expression in the above display simplifies to 

    λ_max{_n(θ)^-1} = λ_min^-1{_n(θ)}.

Wanting the information matrix to have a large minimal eigenvalue is a familiar concept in the classical optimal design theory; see Section <ref>.  



This and the previous subsection, along with the remarks in Section <ref>, establish some important properties and insights concerning our proposed Hellinger information.  A difficulty that has not yet been addressed is the dependence of _n(θ; u) on the arbitrary direction u.  However, the lower bound in (<ref>) is free of a direction, so it makes sense to formulate a direction-free Hellinger information based on that.  For a non-regular model as formulated above, with index of regularity α∈ (0,2], we set the direction-free Hellinger information at θ, for interest parameter ψ(θ), as 

    _n^ψ(θ) = inf_u {D_ψ(θ)   u^-α_n(θ; u) }.

In the special case where ψ(θ) = θ, this simplifies to 

    _n(θ) = inf_u _n(θ; u).

Moreover, in the regular case with α=2, it follows from Corollary <ref> and, in particular, (<ref>), that _n(θ) above is (proportional to) the smallest eigenvalue of the Fisher information matrix.  Therefore, definition (<ref>) seems very reasonable; more details are presented in Section <ref>.  






 §.§ Technical remarks





Definition <ref> does not allow α to depend on u, so each component of θ, treated individually, must have the same index of regularity.  To see this, consider an exponential distribution with location and rate parameters θ_1 and θ_2, respectively.  If θ_1 was fixed and only θ_2 was unknown, then it is a regular problem and the above definition would hold with α=2.  Similarly, if θ_2 was fixed and only θ_1 was unknown, then the definition holds with α=1 according to Proposition <ref>.  However, if both θ_1 and θ_2 are unknown, then the model does not satisfy the conditions of Definition <ref>.  Consider two unit vectors u=(1,0) and u'=(0,1).  If α=1, then J(θ; u) is in (0,∞) but J(θ;u') is zero; likewise, if α=2, then J(θ;u') is in (0,∞) but J(θ; u) is infinite.  Therefore, the above definition cannot accommodate situations where the components of θ, treated individually, would have different regularity indices.  But the design applications we have in mind in this paper fit naturally within a setting where all components have the same regularity; the more general case will be considered elsewhere.  




Our definition of Hellinger information coincides with that in <cit.> for one-parameter models, but our perspectives differ when it comes to multi-parameter models.  Shemyakin defines a “Hellinger information matrix” for non-regular problems, which seems to contradict our above claim that no such matrix is available, so some more detailed comments are necessary.  Shemyakin makes no claim that his information matrix is related to the local behavior of h, and we are unable to conclude definitively whether it is or is not.  We do know, however, that ϑ↦ h(θ,ϑ) is “bowl-shaped” (though not smooth) at each θ, so if such a matrix could describe the local behavior, then it ought to be non-negative definite.  However, <cit.> admits that a general non-negative definiteness result has not been established for his information matrix.  Without a non-negative definiteness result for his Hellinger information matrix, lower bounds like those in, e.g., <cit.> may not be informative, and its use in defining optimal designs lacks justification. 






In (<ref>) we defined the Hellinger information in an independent sample of size n as _n(θ; u) = ∑_i=1^n J_i(θ; u), the sum of the individual Hellinger information measures.  This, however, is not a choice made by us, it is a consequence of the proof of Theorem <ref>.  To see this, heuristically, start with the Hellinger distance between joint distributions P_θ^n and P_ϑ^n, assuming independence.  A straightforward calculation reveals 

    H^2(P_θ^n, P_ϑ^n) 
        = 2 - 2 ∏_i=1^n ∫{ p_i,θ(y_i) p_i,ϑ(y_i) }^1/2  dy_i 
        = 2 - 2 exp{∑_i=1^n log[ 1 - 12 H^2(P_i,θ, P_i,ϑ) ] }

Since log(1+x) ≈ x for x ≈ 0, if ϑ is sufficiently close to θ, then the exponent is approximately -1/2∑_i=1^n H^2(P_i,θ, P_i,ϑ) and then, by Taylor's theorem applied to x ↦ e^-x at x ≈ 0, we conclude that 

    H^2(P_θ^n, P_ϑ^n) ≈∑_i=1^n H^2(P_i,θ, P_i,ϑ).

Therefore, a local approximation of the left-hand side is roughly equal to a sum of local approximations on the right-hand side, which leads to (<ref>).  





An important unanswered question in the above theory is if there are any estimators that are efficient in the sense that they attain the lower bound in Theorem <ref> in some generality.  In the simple (0,θ) example above, we showed that the bound is asymptotically attained, up to universal constants, by the sample maximum.  General results about the rate of convergence in non-regular models are consistent with our lower bound, but, to our knowledge, more precise results concerning the asymptotic behavior of estimators in non-regular problems is limited to certain special cases.  Our work here provides some motivation for further investigation of these asymptotic properties.  Not having an estimator that provably attains the lower bound complicates our attempts to demonstrate the efficiency gains of our proposed optimal designs in Section <ref>, but a quality estimator is available in the applications we have in mind; see Section <ref>.  








§ OPTIMAL DESIGNS FOR NON-REGULAR MODELS





 §.§ Definition


The previous section built up a framework of information, based on a local approximation of the squared Hellinger distance, suitable for non-regular problems where Fisher information does not exist.  Our motivation for building such a framework was to address the problem of optimal experimental designs in cases where the underlying statistical model is non-regular.  This section defines what we mean by an optimal design for non-regular models, and provides some additional details about the Hellinger information that are particularly relevant to the design problem.  

We start here with a slightly different setup than in the previous section, but quickly connect it back to the preceding.  Let Y_1,…,Y_n be independent observations, where Y_i has density function q_η_i, for i=1,…,n.  That is, each Y_i has its own parameter η_i, which we will assume is real-valued, as is typical in linear and generalized linear models.  Then the design problem proceeds by expressing the unit-specific parameter η_i as a given function g(x_i,θ) of a common parameter θ∈^d and a vector of unit-specific covariates; here, of course, the covariates are constants that the investigator is able to set in any way he/she pleases, but preferably in a way that is “optimal” in some sense.  By linking each η_i to a common θ, we obtain the setup from previous sections, i.e., Y_i ∼ p_i,θ, independent, for i=1,…,n.  

The next result, stated in the context of n=1, parallels a familiar one in the regular case for Fisher information.  It aids in computing the Hellinger information under a reparametrization like the one described above. 



Let q_η be a density function depending on a scalar parameter η, and suppose that the index of regularity is α∈ (0,2] and the Hellinger information is J̃(η).  Define a new density p_θ, for θ∈Θ⊆^d, as q_g(θ) where g: Θ→ is a smooth function with non-vanishing gradient ġ.  Then p_θ also has index of regularity α, and the corresponding Hellinger information at θ, in the direction of u, is 

    J(θ; u) = |ġ(θ)^⊤ u|^αJ̃(g(θ)).




See Section S2.3 in the Supplementary Material.


From the general theory in Section <ref>, if Y_1,…,Y_n are independent, then under the assumptions in Proposition <ref>, i.e., Y_i ∼ p_i,θ = q_g_i(θ), the Hellinger information at θ, in direction of u, is 

    _n(θ; u) = ∑_i=1^n |ġ_i(θ)^⊤ u|^αJ̃(g_i(θ)).


For the special case where g_i(θ) = g(x_i, θ) for covariates x_i, it is clear that _n(θ; u) depends on x_1,…,x_n. For example, if Y_1,…,Y_n are independent, with Y_i ∼ g(x_i, θ) + (β, 1), where g(x,θ) = θ_0 + ∑_k=1^p θ_k x^k+1, then it follows from Propositions <ref> and <ref> that 

    _n(θ; u) = 1 + β r(β)/βΓ(β)∑_i=1^n|∑_k=0^px_i^ku_k+1|^β.

The Hellinger information's dependence on the covariates (x_1,…,x_n) is what makes our theory of optimal design possible.  




In what follows, we focus exclusively on the case of ψ(θ) = θ, and the direction-free definition of Hellinger distance in (<ref>), though this is only for simplicity.  The same derivations can be carried out with any specific interest parameter ψ(θ) in mind.  

Following the now-standard approximate design theory put forth by <cit.>, let ξ denote a discrete probability measure defined on the design space—the space where the covariates x_i live—with at most m distinct atoms, representing the design itself.  That is, the atoms of ξ represent the specific design points, and the probabilities correspond to the weights (more details below).  Next, with a slight abuse of our previous notation, we write _ξ(θ; u) to indicate that the Hellinger information in the direction u depends on the design ξ through the specific covariate values. For example, given design ξ={(w_i, x_i): i=1,...,m}, then _ξ(θ; u)=∑_i=1^m w_i J_i(θ;u), where J_i(θ;u) is the Hellinger information in the direction u based on one observation taken at location x_i. Following (<ref>), the Hellinger information based on design ξ is defined as 

    _ξ(θ) = inf_u _ξ(θ; u).


Naturally, the optimal design under this setup would be defined as the one that maximizes this measure of information.  



Under the non-regular model setup presented above, the optimal design ξ^⋆ is one which maximizes the Hellinger information, i.e., 


    ξ^⋆ = max_ξ_ξ(θ).



For comparison to the classical design theory, property (<ref>) implies that our optimal design in Definition <ref>, under a regular model, corresponds to an E-optimal design, one that maximizes the minimum eigenvalue of the Fisher information matrix.  For the non-regular case, however, we do not have an information matrix, so it is not clear if other common notions of optimality, such as A- and D-optimality, have any meaning.  For example, non-regularity will cause sampling distributions of estimators to be non-ellipsoidal, so we cannot expect the determinant of some information matrix to correspond to the volume of a confidence ellipsoid. 




Definition <ref> formulates a new class of optimal design problems, deserving further attention.  As discussed briefly in Section <ref>, there is now a substantial literature on theory and computation related to the optimal design problem in regular cases, and we hope that this paper stimulates a parallel line of work with similar developments for non-regular cases.  There are some similarities to the regular case, in particular, the Hellinger information is non-negative and additive like Fisher information.  Also, the map ξ↦_ξ(θ) is concave for fixed θ, i.e., for any two designs ξ and ξ' and any w ∈ [0,1], 

    _w ξ+(1-w)ξ'(θ) ≥ w _ξ(θ)+(1-w)_ξ'(θ),
 
which is important for numerical and/or analytical solution of the optimal design problem.  The following gives some first results along these lines.  






 §.§ A general result for non-regular polynomial models



Motivated by the setup in <cit.>, we consider a non-regular model of the form 

    y_i = g(x_i, θ) + _i,    i=1,…,n,
 
where x_i are scalars, g(x,θ) = θ_0 + ∑_k=1^p θ_k x^k is a degree-p polynomial, θ∈^d, with d=p+1, is an unknown parameter, and _i are independent and identically distributed with density p_0 given in (<ref>) and known shape parameter α∈ [1,2).  As is customary <cit.>, we will insist that the design points be centered at the origin, which puts a constraint on the design itself.  In particular, we will consider the space of designs ξ given by 

    Ξ = {ξ = (w_i, x_i): ∑_i w_i x_i = 0, x_i ∈ [-A,A]},

i.e., designs on [-A,A] that are “balanced” in the sense that the mean x value is 0, where A > 0 is fixed and known.  

The following result shows that, among balanced designs, the subclass of symmetric designs is complete in the sense that the maximum information over symmetric designs is the same as that over the larger class of balanced designs.  This implies that the search for an optimal design can be simplified by restricting it to the smaller class of symmetric designs.  



Let Ξ_sym⊂Ξ denote the set of all balanced designs that are also symmetric in the sense that if x is a design point, then it assigns equal weight to both x and -x.  Then 

    max_ξ∈Ξ_sym_ξ(θ) = max_ξ∈Ξ_ξ(θ).




See Appendix <ref>. 


The next section applies this general result to identify optimal designs in some special cases of the non-regular polynomial regression model above.  The two results, Propositions <ref> and <ref>, suggest that there is a de la Garza phenomenon <cit.> in the non-regular case as well, which would be an interesting theoretical topic to pursue in future work.




§ OPTIMAL DESIGNS FOR SOME NON-REGULAR REGRESSION MODELS



In this section, we apply the general result in Theorem <ref> to identify optimal designs in two important special cases of the polynomial model, namely, linear and quadratic.  Throughout we assume the model stated in (<ref>), namely, that the regression model has non-negative errors with distribution having density of the form (<ref>), with known shape parameter α∈ [1,2).  



 §.§ Linear model


Consider the linear version of (<ref>), where g(x,θ) = θ_0 + θ_1 x.  For linear models we have a strong intuition from the regular case as to what the optimal design might be. It turns out that the same result holds in the non-regular case as well. 


The optimal design ξ^⋆, according to Definition <ref>, for the non-regular linear regression model is the symmetric two-point design with weight 1/2 on x=± A.  



See Section S2.4 in the Supplementary Material. 





 §.§ Quadratic model


Consider a quadratic case where g(x,θ) = θ_0 + θ_1 x + θ_2 x^2.  Here we restrict our attention to the case where the errors _i in the model are exponential, α=1.  



For the quadratic model, with α=1 and the balanced design constraint, the optimal design ξ^⋆, according to Definition <ref>, is one with three distinct points {-A, 0, A} with respective weights {1-π/2, π, 1-π/2} for some π∈ (0,1). 



See Section S2.5 in the Supplementary Material. 


Although the proof of Proposition <ref> holds only for the α=1 case, we expect that the result also holds for α∈ [1,2), and the numerical results in Figure <ref> (b) support this conjecture.  The practical importance is that it simplifies the search over Ξ_sym to a search over the scalar π∈ [0,1]. The weight at point {0} of the optimal design—or the likely optimal design for the case of α∈(1,2]—depends on the value of A and α. Based on Proposition <ref> and the definition of Hellinger information, the optimal weight can be obtained by solving the optimization problem

    π_A(α)=max_π∈[0,1] f(π),

where f(π) = f_α,A(π) is given by 

    f(π) = min_u=1{π|u_1|^α+1-π2( |u_1+A u_2+A^2 u_3|^α+ |u_1-A u_2+A^2 u_3|^α) }.

This search for the optimal weight, π_A(α), along with that over u on the surface of the unit sphere, can be handled numerically. 

Figure <ref> shows α↦π_A(α) for several values of A.  In particular, we see that the (likely) optimal designs put more weight on 0 as either α or A increases. Our optimal designs for non-regular regression models have a similar format to their E-optimal counterparts in the regular case.  That is, a regular E-optimal design for quadratic regression over [-A,A] is given by 

    {(-A,1-w_A2),(0,w_A),(A,1-w_A2)},

and, for A in {1,1.5,2}, the corresponding values of w_A are {0.6, 0.75, 0.81}.  From Figure <ref>, as anticipated by Corollary <ref>, we observe that for α=2, π_A(2) matches the weight w_A of the corresponding regular E-optimal design. This is explained by Corollary 1; when α=2, optimal design under Hellinger information is the E-optimal design.

Henceforth, we call the regular E-optimal design counterpart of a non-regular model “regular-optimal." For the non-regular linear  model, based on Proposition <ref>, the optimal design coincides with the “regular-optimal" design. In the numerical results presented below, we compare optimal designs of non-regular quadratic models to their “regular-optimal” counterparts.






 §.§ Numerical results



Here we show some numerical results to demonstrate the efficiency gain in using the proposed optimal designs over other reasonable designs.  Recall our model is of the form (<ref>) with non-negative errors having density (<ref>), with known shape parameter α∈ [1,2). 

One complication is that currently there are no results that identify an estimator whose risk attains the lower bound in Theorem <ref>.  Consequently, we are currently unable to guarantee that minimizing this lower bound will result in improved estimation for any given estimator.  But we do have a reasonable estimator, described next, and the results below do indicate that the design that minimizes the lower bound in Theorem <ref> does indeed result in improved efficiency for this particular estimation.  

For the class of non-regular polynomial regression problems in consideration here, <cit.> proposed an estimator based on solving a linear programming problem: choosing (θ_0,…,θ_p) such that θ_0 is maximized subject to the condition that y_i ≥∑_k=1^p θ_k x_i^k for each i=1,…,n.  This estimator agrees with the maximum likelihood estimator in the case α=1, has a O(n^-1/α) convergence rate, which matches the one given by the lower bound in (<ref>), and can be readily computed using the quantreg package in R <cit.>.  Moreover, as <cit.> argues, it is generally superior to maximum likelihood in non-regular cases. For these reasons, comparisons of designs based on this estimator ought to be informative. 

Figure <ref> presents simulation results on the quality of estimation for the Hellinger optimal design versus 5-, 10-, and 15-point uniform designs for the non-regular linear models, while Figure <ref> presents simulation results comparing Hellinger optimal design versus 5-point uniform design and the regular-optimal design. The study proceeds as follows.  For each design space [-A,A] and candidate design, the n-vector y is simulated from the corresponding model, with the specified value of α and θ, and then Smith's estimator θ̂ is computed.  Repeat this process 1000 times and compute the Monte Carlo estimate of the risk R(θ̂, θ) as usual.  This risk is the sum of mean square errors for each component of the parameter vector.





Figure <ref> shows that, under different regularity conditions, the optimal design from Proposition <ref> is superior in terms of risk. In particular, it is significantly better in the estimation of the slope, θ_1, whereas no design performs significantly better than the others in the estimation of the intercept. The results presented in Figure <ref>(a) are consistent with Proposition <ref> in the case of α=1.  In each case, the optimal design performs significantly better than both the 5-point uniform design and the regular-optimal design, despite the similarity of the optimal and regular-optimal designs in terms of weight at point 0. Similarly, Figure <ref>(b) supports our intuition that Proposition <ref> can be extended to cases with α>1.








§ CONCLUSION



This paper aims to establish a framework for optimal design in the context of non-regular models where the Fisher information matrix does not exist.  Towards this goal, we defined an alternative measure of information, based on a local approximation of the squared Hellinger distance between models, suitable for non-regular problems.  The proposed Hellinger information has some close connection to the Fisher information when both exist and, more generally, the former has many of the familiar properties of the latter.  In particular, in Theorem <ref> we establish a parallel to the classical Cramér–Rao inequality which connects our proposed Hellinger information measure to the quality of estimators.  This naturally leads to a notion of optimal designs in non-regular problems, i.e., the “optimal design” is one that minimizes the lower bound in Theorem <ref>.  

The proposed optimal design framework introduces a new class of optimization problems to solve, what we have considered here is only the tip of the iceberg.  However, the tools currently available in the optimal design literature for regular problems are expected to be useful here.  For example, in a particular non-regular polynomial regression setting, we establish a theorem to simplify the numerical and/or analytical search for a particular optimal design, and we apply this general result in the linear and quadratic cases.  Developing the theory and computational methods to handle more complex non-regular models, as well as identifying estimators that attain the lower bound (<ref>), are interesting topics for future investigation.  

Aside from creating a new class of design problems to be investigated, the developments here also shed light on how much our current understanding of design problems depends on the regularity of the models being considered.  That is, beyond its value in helping us tackle specific cases in which regularity conditions do not apply, the study of non-regular problems also deepens our understanding of regularity itself and how it affects optimal design. For example, questions about the type of optimality criterion to consider (e.g., A- versus D- versus E-optimal) are apparently only relevant for those regular cases where the Fisher information matrix is exactly or approximately related to the dispersion matrix of an estimator. While this paper provides some important insights about non-regular models and corresponding optimal design problems, there is still much more to be done.  




§ ACKNOWLEDGMENTS


The authors are grateful for the helpful suggestions from the Associate Editor referees, and Professor Arkady Shemyakin.  The authors also thank Mr. Zhiqiang Ye for pointing out a mistake in the statement and proof of Proposition <ref> in a previous version.  







§ PROOFS OF THEOREMS





 §.§ Proof of Theorem <ref>



The proof requires a connection between Hellinger distance and risk of an estimator. This first step is based in part on Section I.6 of <cit.>, although our setup and conclusions are more general in certain ways. We summarize this in the following lemma, proved in the Supplementary Material.  



For data Y ∈, consider a model P_θ, with μ-density p_θ, indexed by a parameter θ∈Θ⊆^d.  Let ψ = ψ(θ) be the interest parameter, where ψ: ^d →^q.  For an estimator T=T(Y) of ψ, the risk function R_ψ(T,θ) for the estimator T satisfies 

    R_ψ(T,θ) + R_ψ(T,ϑ) ≥min{1-h(θ;ϑ)/4h(θ; ϑ), 1/16}ψ(θ) - ψ(ϑ)^2.



For the proof of Theorem <ref>, start with the squared Hellinger distance between joint distributions P_θ^n and P_ϑ^n, given by 

    h^n(θ; ϑ) := H^2(P_θ^n, P_ϑ^n) = 2 [ 1 - ∏_i=1^n { 1 - h_i(θ; ϑ)/2}],

where h_i(θ; ϑ) = H^2(P_i,θ, P_i,ϑ) is the squared Hellinger distance between individual components.  If θ and ϑ are sufficiently close, in the sense that h_i(θ; ϑ) ≤ 1 for each i=1,…,n, then, given the following inequalities, 

    1-x ≤ -log x   and   -log(1-x) ≤ 2x,    x ∈ [0,1/2],

it follows that 

    h^n(θ; ϑ) ≤ -2∑_i=1^n log{1 - h_i(θ; ϑ)/2}≤ 2 ∑_i=1^n h_i(θ; ϑ).

According to our assumption about local expansion of the individual h_i's, if ϑ = θ +   u for a unit vector u, then 

    h^n(θ; θ +   u) ≤ 2 _n(θ; u)  ^α + o(n^α),   → 0.

When we take  equal to _n,u = {3 _n(θ; u)}^-1/α, then we get 

    h^n(θ; θ + _n,u  u) ≤23 + o(1),    n →∞,

where the latter “o(1)” conclusion is justified by the assumption (<ref>) about the rate of information accumulation.  Therefore, for large enough n, with ϑ_n,u = θ + _n,u  u, h^n(θ; ϑ_n,u) ≤3/4, it follows from the above lemma that 

    R_ψ(T_n, θ) + R_ψ(T_n, ϑ_n,u) ≥116ψ(θ) - ψ(ϑ_n,u)^2.

Since ψ is differentiable, there is a Taylor approximation at θ:

    ψ(θ) - ψ(ϑ_n,u) = D_ψ(θ) (θ-ϑ_n,u) + o(θ-ϑ_n,u),

where the latter little-oh means a q-vector whose entries are all of that magnitude.  Plugging in the definition of ϑ_n,u gives 

    ψ(θ) - ψ(θ + _n,u  u) = -_n,u D_ψ(θ)   u + o(_n,u),    n →∞,

and, hence, 

    ψ(θ) - ψ(θ + _n,u  u)^2 = _n,u^2 D_ψ(θ)   u + o(1)^2 ≥12 _n,u^2 D_ψ(θ)   u^2.

Plugging in the definition of _n,u establishes that

    R_ψ(T_n, θ + _n,u u) + R_ψ(T_n, θ) ≳D_ψ(θ)   u^2  _n(θ; u)^-2/α.

Also, the constant that has been absorbed in “≳” is (32)^-1 3^-2/α.  Finally, the claim (<ref>) follows from the above display and the general fact that, for a function f defined on a set A, f(y_1) + f(y_2) is smaller than 2 sup_A f(y). 




 §.§ Proof of Theorem <ref>



Take any fixed design ξ = {(w_m, x_m): m=1,…,M}, and define a function

    L(u; x) = _ξ(θ; u) = ∑_m=1^M w_m | ∑_k=0^p x_m^k u_k+1|^α.

The L function does not depend on θ because it is based on the information in a location parameter problem, but it does depend implicitly on the w component of the design ξ.  From the trivial identity, 

    a   x_m^k = a   (-1)^k   (-x_m)^k,   any a ∈, any m, and any k,

it follows immediately that L(u; x) = L(v; -x), for any unit vector u ∈^p+1, where v_k+1 = (-1)^k u_k+1, k=0,…,p.  Since this new vector v is also a unit vector, we have 

    min_u L(u; x) = min_v L(v; -x).

This implies that the reflected design ξ'—the one that replaces the original x_m in ξ with -x_m, but keeps the same weights—satisfies _ξ(θ) = _ξ'(θ).  Define the mixture design ξ^† = 1/2ξ + 1/2ξ', which is symmetric by construction, and by concavity (<ref>) satisfies

    _ξ^†(θ)     = min_u {12 _ξ(θ; u) + 12 _ξ'(θ; u) }
       ≥12 min_u _ξ(θ; u) + 12 min_u _ξ'(θ; u).

We showed above that the two terms in the lower bound are equal and, consequently, _ξ^†(θ) ≥_ξ(θ).  Therefore, for any design ξ there exists a symmetric design with Hellinger information at least as big; hence, symmetric designs form a complete class.    





1=1
apalike












§ S   SUPPLEMENTARY MATERIAL






 §.§ S1. A multi-parameter example


As an illustrative example, consider the case where θ=(θ_1,θ_2) is two-dimensional and P_θ is (θ_1, θ_1 + θ_2), where θ_1 ∈ and θ_2 > 0.  It is not difficult to show that 

    h(θ, ϑ) = 2{ 1 - (θ_1 + θ_2) ∧ (ϑ_1 + ϑ_2) - θ_1 ∨ϑ_1/(θ_2 ϑ_2)^1/2}.

Writing ϑ = θ +  u for a unit vector u=(u_1,u_2), and by considering all the possible configurations of u, it can be shown that α=1 and 

    J(θ; u) = θ_2^-1 g(u),

where g(u) is a function that depends only on u, not on θ.  This expression is consistent with what we would expect from the familiar Fisher information, since θ_1 and θ_2 are, in this formulation, location and scale parameters, respectively.  The function g(u) is not complicated, just that the expression varies depending on where on the unit circle u is.  For example, if u is in the first or third quadrants, then 

    g(u) = |2u_1 + u_2|.

To derive the corresponding expressions for other values of u, the second and fourth quadrants need to be split in half along the line y=-x.  Figure <ref>(a) shows a plot of ω↦ J(θ; u_ω), where u_ω = (cosω, sinω), as ω varies over (0, 2π), for several θ values.  Notice that the shape of the function does not depend on θ, only the scale, which means the value of u that minimizes J(θ; u) does not depend on θ.  This plot also reveals the symmetry with respect to reflections u → -u through the origin.  

To gain some intuition about the somewhat complicated lower bound established in Theorem 1, suppose that our goal is to estimate the scale parameter θ_2, i.e., ψ(θ) = θ_2.  Then the relevant Hellinger information is 

    J^ψ(θ; u) = J(θ; u)/|u_2| = g(u)/θ_2 |u_2|.

Figure <ref>(b) plots ω↦ J^ψ(θ; u_ω) for three different θ values.  The minimum value of these functions would be the relevant Hellinger information for estimating θ_2, and the horizontal lines drawn there correspond to θ_2^-1.  Given independent data Y_1,…,Y_n, an oracle who knows the value of θ_1 and uses the maximum likelihood estimator of θ_2 when θ_1 is fixed and known, would have mean square error of the order n^-2θ_2^2, which agrees with the theorem's lower bound up to constants.  







 §.§ S2. Remaining proofs




  §.§.§ S2.1. Proof of Proposition 1



Let f(y) be a function such that f(0)=c and the density p_0 in Equation (1) of the main text satisfies 
 
    p_0(y) = β  f(y)   y^β - 1,    y > 0.

Without loss of generality, consider > 0 and small.  For the squared Hellinger distance, we have 

    h(θ, θ + )     = ∫_-∞^∞{ p_θ+^1/2(y) - p_θ^1/2(y) }^2  dy 
        = ∫_θ^θ +  p_θ(y)   dy + ∫_θ + ^∞{ p_θ^1/2(y) - p_θ + ^1/2(y) }^2   dy
        = _θ(θ≤ Y ≤θ + ) + ∫_θ + ^∞{ p_θ^1/2(y) - p_θ + ^1/2(y) }^2   dy.

The first term equals _0(Z ≤) and it is easy to see that this is f(0) ^β + o(^β).  For the second term, we make a change of variable, z=y-θ-, so that 

    ∫_θ + ^∞{ p_θ^1/2(y) - p_θ + ^1/2(y) }^2   dy = ∫_0^∞{p_0^1/2(z + ) - p_0^1/2(z)}^2  dz.

If we split up this latter integral as 

    ∫_0^Δ{p_0^1/2(z + ) - p_0^1/2(z)}^2  dz + ∫_Δ^∞{p_0^1/2(z + ) - p_0^1/2(z)}^2  dz,

for Δ > 0 as in the statement of the proposition, then the right-most integral is O(^2) = o(^β) by Equation (4) in the main text and the dominated convergence theorem.  It remains to investigate the left-most integral, which equals 

    β∫_0^Δ{f(z+)^1/2(z+)^(β-1)/2 - f(z)^1/2 z^(β-1)/2}^2  dz.

We proceed by adding and subtracting f(z+)^1/2 z^(β-1)/2 inside {⋯}, so that the new integrand looks like 

    {f(z+)^1/2(z+)^(β-1)/2 - f(z)^1/2 z^(β-1)/2}^2 = ∑_i=1^3 I_i(z; ),
 
where 

    I_1(z; )     = f(z+) { (z+)^(β-1)/2 - z^(β-1)/2}^2 
    
    I_2(z; )     = { f(z+)^1/2 - f(z)^1/2}^2 z^(β-1)/2
    
    I_3(z; )     = 2 f(z+)^1/2{ f(z+)^1/2 - f(z)^1/2}{ (z+)^(β-1)/2 - z^(β-1)/2}.

The second term, I_2, is the easiest to deal with, so we take this one first.  Because f is smooth and slowly varying near zero, the mean value theorem says that f(z+)^1/2 - f(z)^1/2≲, which implies that 

    ∫_0^Δ I_2(z; )  dz ≲^2 ∫_0^Δ z^(β-1)/2 dz ≲^2 = o(^β),   → 0.

The third term, I_3, is similar.  That is, after applying the mean value theorem to both of the differences in I_3, we have that 

    ∫_0^Δ I_3(z; )  dz ≲^2 ∫_0^Δ z^-(3-β)/2 dz

and, since the integral converges, the upper bound is O(^2) = o(^β) as → 0.  It remains to deal with the I_1 term, namely, 

    β∫_0^Δ f(z+) {(z+)^b - z^b}^2   dz,   where   b=β-12.

Make a change-of-variable, w=z/, so that the above integral becomes 

    β^β∫_0^Δ/ f( w + ) {(w + 1)^b - w^b}^2   dw.

By the mean value theorem, we have that {(w+1)^b - w^b}^2 ≤min{1, b^2w^2(b-1)}, and, since 2(b-1) = β-3 < -1, this upper bound is integrable over w ∈ (0,∞).  Since f is also bounded, it follows from dominated convergence theorem that, as → 0, 

    ∫_0^Δ/ f( w + ) {(w + 1)^b - w^b}^2   dw → f(0) ∫_0^∞{(w + 1)^b - w^b}^2   dw.

The integral on the right-hand side is to be recognized as r(β) in Equation (5) of the main text.  Putting everything together, we have 

    h(θ, θ + ) = f(0){1 + β  r(β)}^β + o(^β),

which implies that the regularity index is α=β and the Hellinger information is 

    J(θ) := lim_→ 0h(θ, θ + )/^α = f(0) {1 + β  r(β)}.








  §.§.§ S2.2. Proof of Corollary 1



For notational simplicity, write =_n(θ) and D=D_ψ(θ).  Then, for the case α=2, it is easy to check that the lower bound in (9) from Theorem 1 is proportional to 

    sup_u: u=1u^⊤ D^⊤ D u/u^⊤ u.

Let = E L E^⊤ be the spectral decomposition of , and let M=DEL^-1/2; note that M is q × d of rank q ≤ d.  Then the above display equals 

    sup_v: v=1 v^⊤ M^⊤ M v = λ_max(M^⊤ M).

The matrix M^⊤ M is non-negative definite, in general, with d-q eigenvalues equal to zero.  But the q positive eigenvalues of M^⊤ M coincide with those of the q × q positive definite matrix M M^⊤.  And since 

    M M^⊤ = D E L^-1/2 L^-1/2 E^⊤ D^⊤ = D ^-1 D^⊤,

it follows that λ_max(M^⊤ M) = λ_max(MM^⊤) = λ_max(D ^-1 D^⊤).  





  §.§.§ S2.3   Proof of Proposition 3



Let h̃(η,η') = H^2(q_η, q_η'), and recall that η = g(θ), a function of θ.  Since g is smooth, a small change in θ corresponds to a small change in η.  In particular, 

    g(θ +  u) - g(θ) = ġ(θ)^⊤ u + o(),   → 0.

Call the above difference δ, so that h(θ; θ +  u) = h̃(η, η + δ) for small .  Then 

    h(θ; θ +  u)/||^α = h̃(η, η + δ)/||^α = h̃(η, η + δ)/|δ|^α  |ġ(θ)^⊤ u|^α + o(1).

Now let → 0 and, therefore, δ→ 0, and recall the definition of J̃(η).  This proves that J(θ; u) = |ġ(θ)^⊤ u|^αJ̃(g(θ)) as was to be shown.  







  §.§.§ S2.4. Proof of Proposition 4



Theorem 2 in the paper says that the optimal design must be symmetric.  So it suffices to show that the two-point symmetric design with points on the boundary, {(0.5,-A),(0.5,A)}, has information at least as big as the maximum among the symmetric designs.  That is, we intend to show that 

    _{(0.5,-A),(0.5,A)}(θ) ≥max_ξ∈Ξ_sym_ξ(θ).


Denote _{(0.5, ± A)}(θ; u) as the Hellinger information of θ in the direction of u based on design {(0.5,-A),(0.5,A)}. For non-regular linear model, g(θ;x)=θ_0+θ_1x, the expressions for _{(0.5, ± A)}(θ; u) and _ξ^†(θ; u) for any symmetric design, ξ^†∈Ξ_sym, are listed below:

    _{(0.5, ± A)}(θ; u)   =0.5(|u_1+u_2A|^α+|u_1-u_2A|^α),
    _ξ^†(θ; u)=    ∑_i=1^m w_i(|u_1+u_2x_i|^α+|u_1-u_2x_i|^α).

Assuming that,

    J_{(0.5, ± A)}(θ; u)≥ J_ξ^†(θ; u),   for all unit vectors u and all ξ^†∈Ξ_sym,

let ũ=min_u:u=1 J_(0.5,± A)(θ; u); then following from (<ref>), for any ξ^†,
 
    u:u_2=1minJ_(0.5,± A)(θ; u)=J_(0.5,± A)(θ; ũ)≥ J_ξ^†(θ; ũ)≥u:u_2=1minJ_ξ^†(θ; ũ).

Consequently, min_u J_(0.5,± A)(θ; u) ≥max_ξ^†min_u  J_ξ^†(θ; u), which implies (<ref>). 

To complete the proof, we only need to establish (<ref>).  Towards this, 


    J_{(0.5, ± A)}   (θ; u)  -J_ξ^†(θ; u)
       = 0.5(|u_1+u_2A|^α+|u_1-u_2A|^α)-∑_i=1^m w_i(|u_1+u_2x_i|^α+|u_1-u_2x_i|^α)
        = ∑_i=1^m w_i(|u_1+u_2A|^α+|u_1-u_2A|^α-|u_1+u_2x_i|^α-|u_1-u_2x_i|^α)

To evaluate the above expression, first see that one can rewrite any unit vector, except u=(1,0),[The case for u=(1,0) can be ignored, since J_ξ(θ; (1,0)) are the same for all ξ∈Ξ.] in the following format: 

    u=±(±B/√(1+B^2),1/√(1+B^2)),    B∈[0,∞).

Notice that no matter what choices of sign combination of u_1, u_2 is given, 

    |u_1+u_2x_i|^α+|u_1-u_2x_i|^α=(1+B^2)^-0.5 α(|B+x_i|^α+|B-x_i|^α),

then, 

    J_{(0.5, ± A)}(θ; u)-J_ξ^†(θ; u)   
    
    =   (1+B^2)^-0.5 α∑_i=1^m w_i(|B+A|^α+|B-A|^α-(|B+x_i|^α+|B-x_i|^α)).

Based on the expression above, to see if J_{(0.5, ± A)}(θ; u)-J_ξ^†(θ; u) is non-negative or not for all u,ξ^† boils down to checking the sign of  ∑_i=1^m w_i(|B+A|^α+|B-A|^α-(|B+x_i|^α+|B-x_i|^α)) for any 0≤ x_1,..,x_m≤ A,B∈[0,∞).

The following shows that |B+A|^α+|B-A|^α-(|B+x_i|^α+|B-x_i|^α) is non-negative for all possible cases (three cases in total) defined by relationships between x_i, A, B in location:  x_i≤ A≤ B,  B≤ x_i≤ A and  x_i≤ B≤ A. 



  * Case 1, 0 ≤ x_i≤ A≤ B, 

    |B+A|^α+|B-A|^α-(|B+x_i|^α+|B-x_i|^α)
    =(B+A)^α+(B-A)^α-(B+x_i)^α-(B-x_i)^α


When α=1,  

    (B+A)+(B-A)-(B+x_i)-(B-x_i)=2B-B-B=0.

When α>1, function f_1(x)=(B+x)^α+(B-x)^α is an increasing function, since its first derivative is always positive when B>x,

    ∂ f_1(x)/∂ x=α[(B+x)^α-1-(B-x)^α-1]>0.


Then, for any x_i, 0≤ x_i≤ A, f_1(A)-f_1(x_i)≥ 0, i.e.

    (B+A)^α+(B-A)^α-(B+x_i)^α-(B-x_i)^α≥0,  for all  i=1,..,m.


  * Case 2: 0 ≤ B≤ x_i≤ A

    |B+A|^α+|B-A|^α-(|B+x_i|^α+|B-x_i|^α)
    =(B+A)^α+(A-B)^α-(B+x_i)^α-(x_i-B)^α

When α=1, 

    (B+A)+(A-B)-(B+x_i)-(x_i-B)=2A-2x_i≥ 0.

When α>1, function f_2(x)=(B+x)^α+(x-B)^α,  0≤ B<x, is an increasing function since it's first derivative is always positive,  

    ∂ f_2(x)/∂ x=α[(B+x)^α-1+(x-B)^α-1]>0 .



Since x_i≤ A, f_2(A)-f_2(x_i)≥ 0,  for all i, 

 
    (B+A)^α+(A-B)^α-(B+x_i)^α-(x_i-B)^α>0, i=1,...,m,
 

  * Case 3 When 0 ≤ x_i≤ B≤ A,

    |B+A|^α+|B-A|^α-(|B+x_i|^α+|B-x_i|^α)
     
    =   (B+A)^α+(A-B)^α-(B+x_i)^α-(B-x_i)^α
     
    =   (B+A)^α-(B+x_i)^α+(A-B)^α-(B-x_i)^α.


When α=1, (B+A)-(B+x_i)+(A-B)-(B-x_i)=2A-2B ≥ 0.

When α>1, if A-B≥ B-x_i≥ 0, then (A-B)^α-(B-x_i)^α≥0, so (<ref>) is non-negative. 

When α>1, if 0≤ A-B<B-x_i, then (A-B)^α-(B-x_i)^α<0. Let 
A-B=d_m, B-x_i=d_i.
Notice that this assumption means 0≤  d_m<d_i. Set 

    B+x_i=W,  then B+A=x_i+d_i+B+d_m=W+d_i+d_m.
  
Consider f_3(x)=(x+y)^α-x^α-y^α, y>0, x≥ 0, f_3(x) is an increasing function, as its first derivative is positive, 

    f_3'(x)=α(x+y)^α-1-α x^α-1>0.

Also notice that f_3(0)=0, so f_3(x) is a non-negative function. 

Therefore, due to w>0, d_i>0, 
    (W+d_i+d_m)^α-(W)^α-(d_i+d_m)^α>0  and (d_i+d_m)^α-d_i^α-d_m^α>0.

Therefore, when 0≤ A-B<B-x_i

    (B+A)^α-(B+x_i)^α+(A-B)^α-(B-x_i)^α
    
    =   (W+d_i+d_m)^α-(W)^α+(d_m)^α-(d_i)^α
    
    >    (d_i+d_m)^α+ (d_m)^α-(d_i)^α
    
    >   d_i^α+d_m^α+ (d_m)^α-(d_i)^α
    ≥   0.



In summary of all three cases, no matter where B is in relation to x_i and A, 
    |B+A|^α+|B-A|^α-(|B+x_i|^α+|B-x_i|^α)≥0 for all i=1,...,m.





 §.§ S2.5. Proof of Proposition 5



Theorem 2 says that optimal design for the quadratic model must be a symmetric design, so here we only need to search among the collection of symmetric designs. 

Given any symmetric design 

    ξ^†={(w_1,-x_1),...(w_m,-x_m),(w_1,x_1),...(w_m,x_m)}

and direction vector u, for the non-regular quadratic regression model, the Hellinger information of ξ^† in the direction of u=(u_1,u_2,u_3) has the expression of  
J_ξ^†(θ; u)= ∑_i=1^m w_i(|u_1+u_2x_i+u_3x^2_i|^α+ |u_1+u_2(-x_i)+u_3x^2_i|^α). For simplicity, denote  f_u(x)=u_1+u_2x+u_3x^2;  then, when α=1, the above becomes 
    J_ξ^†(θ; u)= ∑_1^m w_i( |f_u(x_i)|+|f_u(-x_i)|).



Let's assume that there exists r_i∈[0,1] such that, for all x_i∈[-A,A], the following relation is true:

    2r_i|f_u(0)|+(1-r_i)|f_u(A)|+(1-r_i)|f_u(-A)|>|f_u(x_i)|+|f_u(-x_i)|.

Then, given w_i, after multiplying w_i on both sides of the inequality (<ref>), 
we have 

    2w_ir_i|f(0)|+w_i(1-r_i)|f(A)|+w_i(1-r_i)|f(-A)|
    ≥ w_i(|f(x_i)|+ |f(-x_i)|).







Let w=∑_i=1^m(1-r_i)w_i, based on the fact that ∑_i=1^mw_i=0.5, 1-2w= ∑_i=1^m2w_ir_i. We can denote a three-point symmetric design based on the left hand side of (<ref>) as 

    ξ_w={(w,-A),(1-2w,0),(w,A)}, 0≤ w≤ 0.5.
 
Hellinger information based on design ξ_w in the direction of a given u has the expression 

    J_ξ_w(θ; u)=(1-2w)|f(0)|+w|f(A)|+w|f(-A)|.

Thus, based on (<ref>), for any u, for any symmetric design ξ^†, there is a w such that  

    J_ξ_w(θ; u)≥ J_ξ^†(θ; u).

Now, via Theorem 2 and the exact same argument that established Proposition 4, the conclusion of this proposition holds.

The only step we need in order to complete the proof is to show (<ref>) is true. Notice that |f_u(x)|=|f_-u(x)|, i.e. |u_1+u_2x+u_3x^2|=|-u_1-u_2x-u_3x^2|. Thus, for every given u̅ with u̅_3<0, 
there is a u̇=-u̅ such that |f_u̇(x)|=|f_u̅(x)|, and  f_u̇(x) is convex. Thus, for simplicity, the following only shows (<ref>) is true for f_u(x) with u_3>0, i.e.  only when f_u(x) is convex. There are seven cases based on the locations of x-intercepts of f_u(x), and for each case,  (<ref>) can be shown to be true. Here we only consider cases in which u is such that its u_3≠0, as the case for  u_3=0 is equivalent to the linear regression case. In the rest of the proof, for simplicity,  let f(x)≡ f_u(x). 

By convexity, if f(x_i)>0 over [-B,B] for some B>0 and x_i∈[0,B], and there is a r_i∈(0,1), such that x_i=r_i 0+(1-r_i)B, and

    r_if(0)+(1-r_i)f(B)>f(x_i) and  r_if(0)+(1-r_i)f(-B)>f(-x_i),
 then 

    2r_i|f(0)|+(1-r_i)|f(B)|+(1-r_i)|f(-B)|>|f(x_i)|+|f(-x_i)|.


Given direction vector u and design point location -x_i,x_i, with x_i>0 and the assumption that u_3>0, there are seven cases that describe the possible relationships between -x_i,x_i and the left, right roots of f(x), x_L<x_R.


  * Case 1: x_i<x_L,x_R, 

  * Case 2: x_L,x_R<-x_i

  * Case 3: -x_i≤ x_L,x_R≤ x_i 

  * Case 4: x_L≤ -x_i , x_i≤ x_R 

  * Case 5: -x_i≤ x_L ≤ x_i≤ x_R 

  * Case 6: x_L≤ -x_i ≤ x_R≤ x_i

  * Case 7: There is at most one root for f(x), i.e. f(x)≥0 for all x∈ R


The following goes through these cases and shows that (<ref>) is true for each of them. Notice that cases 1 and 2 are equivalent, and cases 5 and 6 are equivalent. So we shall focus on cases 1, 3, 4, 5, and 7.



  * In case 1 both roots are above x_i; there are two possible ways that this can happen regarding the given value of A: 


  * 1.1) The left root x_L is above A, i.e. A≤ x_L. This implies that f(x_i)>0 over [-A,A], so by the argument of convexity in (<ref>), (<ref>) is true.


  * 1.2) The left root x_L is below A, i.e. x_L< A. Here, f(-x_i),f(x_i),f(-A)>0, which implies that 

    |f(-x_i)|+|f(x_i)|=2u_1+2u_3x^2,  and  f(-A)=u_1-u_2A+u_3A^2.

If A is smaller than right root, A<x_R, then f(A)<0, so

|f(A)|=-u_1-u_2A-u_3A^2>0, and -u_2A>u_1+u_3A^2. Then with (<ref>),  

    |f(A)|+|f(-A)|=-2u_2A>2u_1+2u_3A^2>2u_1+2u_3x_i^2=|f(-x_i)|+|f(x_i)|.


If A is larger than right root, A>x_R, then f(A)>0, so

|f(A)|=u_1+u_2A+u_3A^2>0. Then with (<ref>),

    |f(A)|+|f(-A)|=2u_1+2u_3A^2>2u_1+2u_3x_i^2=|f(-x_i)|+|f(x_i)|.

Then for 1.2) one can find a ratio r_A such that r_A(|f(A)|+f(-A))>f(x_i)+f(-x_i),  letting r_i=1-r_A, then (<ref>) is true, i.e. 

    2r_i|f(0)|+(1-r_i)|f(A)|+(1-r_i)|f(-A)|>|f(x_i)|+|f(-x_i)|.





  * Case 3: -x_i≤ x_L,x_R≤ x_i, is the case of both roots of f(x) are in [-x_i,x_i], so f(x) would be positive and increasing over [x_i,A], while positive and decreasing over [-A,-x_i], i.e. 
    f(A)>f(x_i)>0,   f(-A)>f(-x_i)>0,
 






Let r_i=1-r_A, then, under α=1,  (<ref>) is true, i.e., 

    2r_i|f(0)|+(1-r_i)|f(A)|+(1-r_i)|f(-A)|>|f(x_i)|+|f(-x_i)|.


  * Case 4: x_L≤ -x_i , x_i≤ x_R. In this case,  f(x)≤ 0 over [-x_i,x_i], which means |f(x)|=-f(x)=-u_1-u_2x-u_3x^2 is concave over  [-x_i,x_i]. Thus, |f(0)|>1/2|f(x_i)|+1/2|f(-x_i)|, and consequently,  (<ref>) holds. 


  * Case 5: -x_i≤ x_L ≤ x_i≤ x_R.

First, the assumption of case 5, -x_i≤ x_L ≤ x_i≤ x_R, implies that -u_2/2u_3=x_L+x_R/2>-x_i+x_i/2=0, i.e. u_2<0. 

Also notice that -x_i≤ x_L implies that 0<f(-x_i)<f(-A) and 

    |f(-A)|=u_1-u_2A+u_3A^2,   |f(-x_i)|=u_1-u_2x_i+u_3x_i^2.

Based on the set up of case 5, and the possible relations of A and direction u, the expression of f(x_i) and f(A) depends on the following two sub-cases: 



  * The right boundary A is below right intercept, i.e. A<x_R, i.e.
f(x_i)<0 and f(A)<0, so

    |f(A)|=-f(A)=-u_1-u_2A-u_3A^2, and  |f(x_i)|=-f(x_i)=-u_1-u_2x_i-u_3x_i^2.

Therefore, with the fact that -u_2> 0, A≥ x_i, and (<ref>), we have 

    |f(-A)|+|f(A)|-|f(-x_i)|-|f(x_i)|
    
    =   -2u_2A+2u_2x_i
    
    =   -2u_2(A-x_i)
    ≥    0.






  * The right boundary A is above right intercept, i.e.  x_R<A, 
 which implies that f(x_i)<0<f(A), 

    |f(A)|=u_1+u_2A+u_3A^2, and  |f(x_i)|=-f(x_i)=-u_1-u_2x_i-u_3x_i^2.

Therefore, with -u_2>0, A≥ x_i, and (<ref>)  

    |f(-A)|+|f(A)|-|f(-x_i)|-|f(x_i)|
    
    =   2u_1+2u_3A^2+2u_2x_i
    
    =   2(u_1+u_2A+u_3A^2)-u_2(A-x_i)
    
    =   2|f(A)|-u_2(A-x_i)
    ≥   0.



Combining these two sub-cases, we can conclude that under case 5, 

    |f(-A)|+|f(A)|≥ |f(-x_i)|+|f(x_i)|.

Then one can find a ratio r_A such that r_A(|f(A)|+f(-A))>f(x_i)+f(-x_i),  letting r_i=1-r_A, then (<ref>) is true, as 

    2r_i|f(0)|+(1-r_i)|f(A)|+(1-r_i)|f(-A)|>|f(x_i)|+|f(-x_i)|.


  * Case 7: There is at most one root, which means, f(x)≥0 for all x∈ [-A,A]. Thus, by the argument in (<ref>), implies (<ref>).







In summary of these all seven cases, (<ref>) holds.






  §.§.§ S2.6. Proof of Lemma 1


Define the mean function of the estimator T, i.e., m_ψ(θ) = E_θ(T).  Since integration of a constant function with respect to the (signed) measure with density p_θ - p_ϑ is zero, we have the following identity:

    m_ψ(θ) - m_ψ(ϑ) = ∫[ T(y) - 12{m_ψ(θ) + m_ψ(ϑ)}] [ p_θ(y) - p_ϑ(y) ]  μ(dy).

Write v_θ,ϑ(y)=T(y) - 12{m_ψ(θ) + m_ψ(ϑ)}.  Now bound the norm of the quantity in the above display:

    m_ψ(θ)-m_ψ(ϑ)    = ∫ v_θ,ϑ (p_θ - p_ϑ)  dy 
       ≤∫v_θ,ϑ  |p_θ^1/2 + p_ϑ^1/2|   | p_θ^1/2-p_ϑ^1/2|  dy.

Next, apply the Cauchy–Schwartz inequality, to get 

    m_ψ(θ)-m_ψ(ϑ)^2 ≤∫v_θ,ϑ^2   |p_θ^1/2 + p_ϑ^1/2|^2  dy · h(θ;ϑ).

For two non-negative numbers a and b, we have (a+b)^2 ≤ 2(a^2 + b^2), so the first term in the above upper bound is itself bounded by 

    2 ∫v_θ,ϑ^2 p_θ dy + 2 ∫v_θ,ϑ^2 p_ϑ dy.

If we rewrite v_θ,ϑ as 

    v_θ, ϑ(y) = {T(y) - m_ψ(θ)} + 12{ m_ψ(ϑ) - m_ψ(θ)},

and use the fact that ∫{T - m_ψ(θ)} p_θ dy = 0, then we get 

    ∫v_θ,ϑ^2   p_θ dy ≤ R_ψ(T, θ) + 14m_ψ(θ) - m_ψ(ϑ)^2.

An analogous bound holds for ∫v_θ,ϑ^2   p_ϑ dy, yielding the expression 

    m_ψ(θ)-m_ψ(ϑ)^2 ≤ 2 h(θ; ϑ) {R_ψ(T,θ) + R_ψ(T,ϑ) + 12m_ψ(θ) - m_ψ(ϑ)^2 }.


Rearranging terms gives the bound 

    R_ψ(T,θ) + R_ψ(T,ϑ) ≥1-h(θ;ϑ)/2h(θ; ϑ)m_ψ(θ) - m_ψ(ϑ)^2.

Finally, write b_ψ(θ) = m_ψ(θ) - ψ(θ) for the bias function of T, and consider the following two exhaustive cases based on the magnitude of the bias:


  * Suppose that max{|b_ψ(θ)|, |b_ψ(ϑ)|} < 1/4ψ(θ) - ψ(ϑ).  Then it follows from the triangle inequality that 

    m_ψ(θ) - m_ψ(ϑ) = ψ(θ) - ψ(ϑ) + b_ψ(θ) - b_ψ(ϑ)≥12 ψ(θ) - ψ(ϑ).



  * Next, suppose that, say, b_ψ(θ)≥1/4ψ(θ) - ψ(ϑ).  Then we trivially have R_ψ(T,θ) ≥b_ψ(θ)^2 and, therefore, R_ψ(T, θ) + R_ψ(T, ϑ) ≥116ψ(θ) - ψ(ϑ)^2.  

Putting these two cases together proves the claim.  












