Comparison of Confidence Interval Estimators: an Index Approach
Richard Minkah Tertius de Wet

#1 1

Projected Hartree-Fock as a Polynomial of Particle-Hole Excitations and Its Combination With Variational Coupled Cluster Theory
    Gustavo E. Scuseria
    December 30, 2023
===============================================================================================================================








[0.5ex]1pt
In many statistical problems, several estimators are usually available for interval estimation of a parameter of interest, and hence, the selection of an appropriate estimator is important. The criterion for a good estimator is to have a high coverage probability close to the nominal level and a shorter interval length. However, these two concepts are in opposition to each other: high and low coverages are associated with longer and shorter interval lengths respectively. Some methods, such as bootstrap calibration, modify the nominal level to improve the coverage and thereby allow the selection of intervals based on interval lengths only. Nonetheless, these methods are computationally expensive. In this paper, we propose an index which offers an easy to compute approach of comparing confidence interval estimators based on a compromise between the coverage probability and the confidence interval length. We illustrate that the confidence interval index has range of values within the neighbourhood of the range of the coverage probability, [0,1]. In addition, a good confidence interval estimator has an index value approaching 1; and a bad confidence interval has an index value approaching 0. A simulation study was conducted to assess the finite sample performance of the index. The proposed index is illustrated with a practical example from the literature.


[0.5ex]1pt
MSC: 	62F99 , 62G99

Keywords:  Confidence interval;  Empirical coverage probability ;  Confidence interval length;  Bootstrap calibration



1.45 




§ INTRODUCTION






Most statistical problems involve the estimation of some unknown parameter, θ, of a population from an observed sample using an estimator, θ̂ <cit.>. In order to provide a complete description of the information in the sample about θ, a confidence interval is usually constructed. The key concepts associated with confidence intervals are the coverage probability and interval length. The former is the proportion of times the confidence interval encloses θ under many replications; and the latter refers to the difference between the upper and the lower confidence limits. These two key concepts are related: longer confidence intervals have higher coverage probabilities approaching the nominal level and shorter confidence intervals have lower coverage probabilities. In statistics, one is often faced with a number of confidence intervals for a parameter arising from different estimators or methods of estimation, and a decision has to be made on the “best" method of estimation. Since these two key concepts are in opposition to each other, that is, better coverage probability goes with weaker length and vice versa, 
it is useful to have some practical way of combining these measures. In this paper, such an easy to compute measure is proposed and applied to two well-known problems as well as a practical example from the literature.

Suppose we have a sample, x={x_1,...,x_n}, drawn from an unknown distribution function, F. Let ℓ(α;x)=(ℓ_L(α;x),ℓ_U(α;x)) be the α-level confidence interval for the unknown parameter θ. Also, denote by L(α), the average of the confidence interval length between the upper confidence limit ℓ_U(α;x) and the lower confidence limit, ℓ_L(α;x). Furthermore, let the coverage probability be given by η(α)=P(θ∈ℓ(α;x)). 

The estimation of η(α)  is an important issue for statisticians and the goal is to obtain a confidence interval estimator with estimated coverage probability (usually referred to as empirical coverage probability), η̂(α), equal to the nominal coverage, 1-α <cit.>. However, it is often the case that η̂(α) is not exactly equal to 1-α. A requirement for a good confidence interval estimator is to have a short interval length and a coverage probability equal to or approximately equal to the nominal coverage. As a result, confidence interval estimator selection can be done by a comparison of the intervals' coverage probabilities and lengths. However, this can be subjective especially if several interval estimators are involved and a compromise is sought between coverage probability and interval length.

A handful of methods to overcome the difficulties in comparing confidence interval estimators rely on an adjustment of the interval lengths such that each interval gives a coverage probability close or equal to the nominal level, 1-α. In that case, the comparison of the confidence interval estimators can be done using the confidence interval lengths only. Examples of these methods in the literature include bootstrap calibration <cit.> and prepivoting <cit.>. The basic idea underlying the bootstrap calibration is to obtain β  (β<α) such that the resultant interval's coverage probability equals 1-α i.e. η(β)=1-α.  Also, prepivoting involves the transformation of the lower (and/or upper)  confidence level(s) by using its estimated bootstrap distribution function. This has an important application in reducing the coverage error of bootstrap confidence intervals. In addition, prepivoting can be iterated and this automatically moves the empirical coverage closer to the desired level, 1-α.


However, in practice, these bootstrap-based procedures generally require computationally costly nested bootstraps (i.e. bootstrapping from the bootstrapped data) from the data. For example, in the case of a double bootstrap, the first bootstrap sample needs B_1 resamples from the data and then resampling B_2 times from each of single bootstrap samples. Thus, the computational cost involves B_1× B_2 samples in addition to the confidence interval calculations. Also, in <cit.>, the authors were constrained in terms of the number of estimators for impulse responses in large Vector Autoregressive Models due to the prohibitive computational cost. Even for the limited confidence interval estimators considered, in cases like the bias-corrected and accelerated bootstrap method, the computing time required for the evaluation of the estimators was over one year. 

Furthermore, <cit.> shows that the coverage precision increases with increasing levels of resamples. Thus, the level of resamples can be done until a point where the coverage is approximately equal to 1-α. At this stage, the comparison of interval estimators can be done on the interval lengths only. However, in applications, this is limited by the huge computing power and time needed for such levels of bootstrap. As a result, some work has been done to reduce the computational burden involved in the use of these bootstrap based procedures. Among these, <cit.> and <cit.> proposed a linear and a nonlinear interpolation respectively to reduce the level of bootstrap replications in calibration. Also, <cit.> provides an algorithm for the double bootstrap, illustrated above, to reduce the B_1× B_2 total resamples to an appreciable level. These algorithms have varying degrees of success in implementation. Nevertheless, in practical application of these methods, a determination is needed of the benefits of higher levels of resamples against the computational cost. 


In this paper, we propose an index which offers a straightforward approach of comparing confidence interval estimators without the use of Monte Carlo simulation or analytical derivations. The index is based on a compromise between the coverage probability and the confidence interval length. The need for such an index arose from a recent very large simulation study of comparing different estimators of tail index in extreme value theory. Running the simulation to obtain a variety of confidence intervals based on the different estimators, was already computationally very intensive. Applying a further computationally intensive calibration or double bootstrap, would have been too costly in terms of computing time and resources. Hence, the proposed index was developed as a computationally inexpensive compromise between coverage probability and confidence interval length.

The rest of the paper is organised as follows. In section <ref>, the bootstrap calibration method is presented. The proposed confidence interval index is presented in Section <ref>. In Section 4, we conduct a simulation study to assess the finite sample performance of the proposed index on four popular confidence interval estimators of the mean from a symmetric and skewed distribution. In addition, several confidence interval estimators of the binomial proportion are examined using the index. Section <ref> deals with an application of the index on a study of the performance of several confidence interval estimators of the coefficient of variation from <cit.>.  Finally, we present some concluding remarks in Section <ref>. 



§ BOOTSTRAP CALIBRATION


<cit.> catalogs some procedures for generating confidence intervals with improved coverage probabilities. These include Edgeworth expansion (analytical) and bootstrapping (simulation). The author states, with references, that given that the Edgeworth expansion and bootstrap procedures are valid, both produce results that have the same asymptotic error rates. In particular, the bootstrap procedure implements the Edgeworth correction through simulation in an automatic fashion. In view of this, we consider the bootstrap calibration method of <cit.> only. 

Let  x_1,…,x_n be a random sample of size n from the distribution function F. We consider the estimation of the 100(1-α)% 2-sided normal-theory confidence interval of the mean, θ=θ(F), given by


    [θ_L, θ_U]=[θ̂+z_α/2σ̂/√(n), θ̂-z_α/2σ̂/√(n)].
  
Here, σ̂ and z_α/2=Φ^-1(α/2) are the unbiased estimate of the variance and the quantile of the standard normal distribution, Φ, respectively.

If F is normally distributed and n large, the estimated coverage probability, η̂(α), will be close to 1-α. However, for smaller n and non-normal distributions, η̂(α) may differ substantially from 1-α. The idea of calibration introduced by <cit.> is to replace α with β  (β<α) such that 

    η̂(β)≊1-α.

This invariably implies several or possibly infinite search for β, satisfying (<ref>), and each of this searches is accompanied by bootstrapping samples to obtain η̂(β). Thus, the method seems impractical. 
However,  <cit.> and <cit.> respectively proposed a linear and a smooth nonlinear interpolation, to reduce the level of bootstrap resampling needed for the seemingly infinite search for β to be replaced with just one level. Thus, the calibration is obtained by generating B bootstrap replications.

Let x^*={x_1^*,…,x_n^*} be a bootstrap sample from x={x_1,x_2,…,x_n}.  Also, let θ̂^*=θ̂(x^*), σ̂^*=σ̂(x^*) and  t^*_j=√(n)(θ̂^*_j-θ̂)/σ̂^* the t statistic computed from the jth bootstrap sample. <cit.> defines λ̂_j=1-Φ(|t^*_j|) and β is taken as the α-quantile of (λ̂_1,…,λ̂_B). In addition, the author argues that the calibration method above is equivalent to the bootstrap root method of <cit.>.

The implementation of the calibration method leads to intervals with error rates comparable to bootstrap t, and the accelerated bias-corrected percentile method. However, it is known that these confidence interval estimation methods have limitations with respect to coverage probability and interval lengths <cit.>.  



§ THE INDEX


We introduce an index which offers a straightforward approach and avoids the computational burden in the bootstrap-based methods in comparing confidence interval estimators. In addition, the index abstracts the information provided by the confidence interval length and coverage probability, thereby making it a standalone value for comparative purposes. The idea behind the proposed index was to obtain a value that is simple, easy to interpret, and takes into account confidence interval length and coverage probability. In addition, the index is expected to have a range within the neighborhood of the desired coverage probability and hence, can easily be reported together (e.g. graphically) for comparative purposes. 



Consider R confidence interval estimators and let η={η_1,…,η_R}' and L={L_1,…,L_R}' denote the vectors of realised coverage probabilities and average interval lengths respectively.

The confidence interval index, I, is defined as


    I(L_j,η_j;α)=k_α(1-1/2(1+H(η_j;α)/1+(η_j/1+L_j))),    L≥0,  0≤η_j≤1,  j=1,…,R,

where k_α is a constant depending on the significance level, α. Here, H is a loss function which describes the penalty incurred by the deviation of the empirical coverage probability from 1-α. In this study, we choose H as a simple absolute loss function defined by


    H(η_j;α)=|1-α-η_j|,  0≤η≤ 1, j=1…,R.

Consequently, using (<ref>), the scaling parameter is taken as

    k_α=4-2α/3-2α,

to obtain the range of values of I(L_j,η_j;α) within the neighbourhood of the desired coverage probability. To derive the range of values of the index, I(L_j,η_j;α), we examine the limit at four extreme cases: 


	
  * L_j→ 0, η_j → 0  ⟹ I(L_j,η_j;α) →k_αα/2.
	
  * L_j→∞, η_j → 0  ⟹ I(L_j,η_j;α) →k_αα/2.
	
  * L_j→∞, η_j → 1-α ⟹ I(L_j,η_j;α) →k_α/2.
	
  * L_j→ 0, η_j → 1-α ⟹ I(L_j,η_j;α) → 1.


Thus, I(L_j,η_j;α) has a range [k_αα/2,  1 ].  A bad confidence interval estimator (i.e. an interval with low coverage probability and large interval length) corresponds to cases I and II, with I(L_j,η_j;α) → k_αα/2 . On the other hand, a good confidence interval estimator  (i.e. case IV)  has  I(L_j,η_j;α) → 1. We note that the range of I(L_j,η_j;α) can be transformed to the desirable range of the coverage probability, [0,1], via an affine function f(x)=2x/(2-k_αα)-k_αα/(2-k_αα), for increased interpretability. 

From the aforementioned limits, we conclude that generally a higher value of the index means a better confidence estimator of the parameter θ. That is, such an estimator has coverage probability close to the nominal value and shorter interval lengths. In addition, as the index penalises for deviation from the nominal level and larger interval lengths, estimators with small coverage probabilities and/or large interval lengths generally have smaller confidence interval index. Therefore, in using this index for comparative purposes, the estimator with largest index value will be chosen ahead of the smaller values. In the subsequent sections, we take α=0.05 and, thus, I(L_j,η_j)∈ [0.034, 1.000],  j=1,…,R. 

We note that, other loss functions can be chosen for this purpose, for example, quadratic, a Huber function <cit.>, among others and appropriate values of k_α determined analytically. For example, if we consider the case of the square loss function, H(η_j;α)=(1-α-η_j)^2,  0≤η≤ 1, j=1…,R. The value of k_α can be taken as in (<ref>). However, the limits of I(L_j,η_j;α) corresponding to cases I, II, III and IV are respectively k_αα(2-α)/2,  k_αα(2-α)/2, k_α/2 and 1. Thus, the range of the resulting I(L_j,η_j;α), is [α(2-α)^2/(3-2α),1]. In the case of the two loss functions considered, the rationale behind the choice of k_α, is to obtain a range of values of I(L_j,η_j;α) in the neighbourhood of the range of the coverage probability for ease of interpretation. Lastly, the effect of the choice of loss function is reflected in the range of values of I(L_j,η_j;α). 




§ SIMULATION STUDY


In this section, we study the performance of the confidence interval index, I, through a simulation study. In this regard, we assess the performance of several confidence interval estimators of the mean from a symmetric and skewed (or asymmetric) distributions. In addition, several estimators of the binomial proportion are assessed using I. 



 §.§ Confidence Interval Index for the Mean


We present a simulation study on the estimation of the mean from a symmetric and a skewed distribution in the two subsections that follow. In the former, we considered samples generated from a normal distribution and the latter from a lognormal distribution.

To study the behaviour of the estimators, samples of size, 
n  (n=10, 50, 100,
 200, 500, 1000) were generated from a normal or a lognormal distribution with mean, μ, and variance, σ^2. The parameter of interest is the population mean, μ, which is estimated by the sample mean, x̅. The 95% two-sided confidence interval of  μ  was constructed using four different methods, namely, the normal theory interval, the Johnson t interval ( unlike the normal theory interval, adjust for positive and negative skewness in a data set by shifting the endpoints right and left respectively. The Johnson t interval is given by (x̅+ κ̂_3/6√(n)(1+2t_α^2))± t_αs/√(n), where κ̂_3 is the estimate of the population skewness E(X-μ)^3/σ^3,  t_α is the α quantile of the t distribution with n-1 degrees of freedom and s is the sample standard deviation.) <cit.> and the bootstrap-based intervals-the bootstrap percentile and the  Bias-Corrected and accelerated (BCa) <cit.>. 

The following procedure was used to compute the index and its summary statistics:


	
  A1. Generate N (N=1000) samples each of size n  from N(μ,σ^2).
	
	
  A2. Draw B (B=1000) bootstrap samples from each sample in A1 and use these to compute B bootstrap confidence intervals (i.e. bootstrap percentile and the BCa) of the mean. Compute the average of the B interval lengths, L, and the empirical coverage probability η̂ for both bootstrap interval types separately. 
	
	
  A3. Compute the confidence interval for the mean using the normal theory interval and Johnson t interval using each of the N samples in A1. Calculate the average of the N interval lengths, L, and the empirical coverage probability η̂ for the two interval types.  
	
	
  A4. Repeat A1-A3 a large number of times R (R=5000), to obtain the pairs {(η̂_1,L_1),…,(η̂_R,L_R)} and, hence, the confidence interval index, I^(i,j), i=1,…,4, j=1,…,R.
	
	
  A5. Compute summary statistics for the indexes, I^(i,.) i=1,…,4.  




  §.§.§ Mean of a Symmetric Distribution

Table <ref> shows the summary statistics of the index for the four interval types computed for observations from N(2,1). It can be seen that, as the sample size increases, I tends to 1: the confidence interval estimators improve with increasing sample size. This is expected in line with the weak law of large numbers:  x̅ approaches μ as n→∞. 





In addition, for smaller sample sizes (i.e. n≤50), the Johnson t interval has the largest I values in most cases followed by the normal interval. Generally, these two estimators provide better confidence intervals as they have larger index values for measures of location, smaller variability, large negative skewness and  large peakedness. In the case of large sample sizes (i.e. n>50), there is not much difference between the performance of the normal theory and Johnson t interval estimators of the mean. The bootstrap percentile is the next best confidence interval estimator of the mean followed by the BCa interval estimator based on the summary statistics. Since the sample mean is an unbiased estimator of the population mean, the percentile interval is expected, as shown in the simulation study, to give better intervals in terms of coverage and interval lengths. We remark that the simulation was carried out for larger sample variances and the results show wider interval length leading to smaller values of the index. Due to space consideration, the results are not included but can be obtained from the authors upon request. 




Furthermore, we consider the performance of I in relation to bootstrap calibration of <cit.> and <cit.>. Again, the estimation of the mean of a normal distribution is considered. Here, we considered smaller sample sizes where the empirical coverage probability tends to be smaller than the nominal level, 1-α. In that case, calibration can be used to increase the empirical coverage probability to approximately equal to 1-α. Our aim in this case is to assess the conclusions reached for calibrated intervals in relation to the index. The results of the simulation study for observations from N(2,1) are presented in Table <ref>. 






For smaller sample sizes (n≤20), the Johnson t interval has empirical coverage probabilities close to the nominal level of 0.95. Calibration of such interval leads to overestimation of the coverage probability. Therefore, we failed to calibrate interval estimators with empirical coverage probability close to 0.95.  The performance of the Johnson t confidence interval estimator is expected as it adjusts for the skewness in the data (in particular for small sample sizes where skewness is prevalent). However, this interval consistently has the largest interval length compared with the normal theory, bootstrap percentile and BCa. The index values for the Johnson t interval are the largest, and thus, can be considered as the most appropriate confidence interval estimator of the mean.

As the sample size increases, the normal theory interval  and the Johnson t interval estimators outperform the other intervals in terms of coverage probability. Also, the normal theory interval estimator has interval lengths fairly competitive to the bootstrap-based intervals and outperforms the Johnson t interval. This can be seen from the index of the normal theory interval having the largest values. 

In general, we note that calibration as demonstrated in Table <ref> does not necessarily bring the empirical coverage up to the desired level of 1-α. Thus, calibration, although expensive, is not always attractive. We find that the conclusions of the confidence interval index for the non-calibrated intervals agree mostly with that of the calibrated intervals. Again, if the coverage probability is close to the nominal value, calibration leads to overestimation of the coverage probability. However, the index penalises such intervals for the deviation from the nominal level, and thus, discriminates good intervals from the bad ones. 



  §.§.§ Mean of a Skewed Distribution




In this section, we consider the performance of the confidence interval index for the estimation of the mean of a skewed distribution. Observations were generated from a lognormal distribution with mean 0. Since the skewness of the lognormal distribution depends only on the variance, we took variances of 3, 2 and 0.2 corresponding to skewness of 23.732, 6.185 and 1.516 respectively. The results for these three values are shown in Tables <ref>, <ref> and <ref> respectively. 

Firstly, for a largely skewed distribution, it is evident that the normal theory interval has the smallest average confidence interval indexes but relatively larger standard deviations. The normal intervals are symmetric, and hence, has a challenge when it is used to provide a confidence interval for the mean of a heavily-skewed distribution. On the other hand, the BCa interval records the best performance as it has the largest average confidence interval indexes. This results from the fact that for a heavily-skewed distribution, large bias is expected but the BCa interval corrects for bias and skewness and hence, provides better intervals that enclose the actual parameter being estimated. 




Secondly, in the case of a moderately skewed distribution, the Johnsons-t interval estimator is by far the best estimator of the mean of the lognormal distribution with larger index values. This is followed by the BCa interval estimator especially for smaller sample sizes where skewness is high. However, as the sample size increases, the normal interval estimator improves and surpasses the BCa with larger confidence interval indexes.  

Thirdly,  for the case of low skewness, i.e. σ^2=0.2, the performance relatively follows a similar pattern but with some notable differences. The Johnson t remains the best estimator but the performance of the normal theory interval improves significantly for large sample sizes giving large values of the index. At n=1000, there is not much difference between the two estimators. However, the performance of the BCa interval reduces as its index values are smaller compared with the other estimators. This may be attributed to the low skewness of the distribution, and hence, being close to a symmetric distribution similar to the normal distribution presented in Section <ref>. 

Lastly, the index values increase with decreasing variance of the lognormal distribution (i.e. decreasing skewness) signifying better performances than for the case of larger variances (i.e. increasing skewness). This is to be expected as the confidence interval estimators give intervals that have better coverage and smaller interval lengths when skewness is small. This is in conformity with earlier studies that compared estimators of the mean of a skewed distribution based on interval length and coverage probability <cit.>. Therefore, in general, the confidence interval index works well for selecting confidence interval estimators of the mean of a skewed distribution.  




 §.§ The Confidence Interval Index for the Binomial Proportion



In this section, we consider the estimation of the binomial proportion by sampling from a binomial distribution. This issue usually arises in applied statistics e.g. incidence rates (in medical science), proportion of defective items (in manufacturing), among others. In addition, unlike the confidence interval for the mean, that of the proportion enables us to measure the performance of the index on non-symmetric intervals.  

Assume X is binomially distributed with parameters n and p, written, X∼ Bin(n,p). Here, the estimator, p̂, is the maximum likelihood estimator given by p̂=X/n. This estimator is consistent and, since, the expected value of X is equal to np, it is also unbiased. 

The most basic form of an interval estimate for the proportion, p, is the Wald interval, 

    p̂± Z_α/2√(p̂(1-p̂)/n)

<cit.>. The properties of this interval estimator have been studied extensively in the literature. Its performance is known to be erratic with respect to coverage probability. In addition, recommendations concerning the values of n and p where this interval is appropriate are conflicting <cit.>. Several attempts have been made to obtain better confidence interval estimators of the binomial proportion. For example, <cit.> proposed an amendment of the interval (<ref>) by defining p̂ as (X+2)/(n+2). Some further modifications and other intervals that are not based on the normality assumption are presented in Table <ref>. 



In the present study, we generated samples of size, n, and proportion, p, from a binomial distribution. Each estimator in Table <ref> is used to obtain a confidence interval for p. We repeat the process R  (R=1000) times and obtain the average confidence interval length and coverage probability. We then compute diagnostic checks on these intervals using the index I in (<ref>). 

Tables <ref>-<ref> show the results of the simulation for combinations of n and p. 




















We find that the confidence interval length improves with increasing sample size. In addition, most of the empirical coverage probabilities of the estimators become much closer to the nominal level of 0.95 as the sample size increases. In particular, the Pois estimator for estimating p=0.1, improves drastically from η̂=0.556 to  η̂=0.942 respectively for sample sizes 10 and 100. Together with the corresponding confidence interval lengths, the values of index for the Pois estimator increases from 0.6940 (compared with the best estimator's index of 0.9395) to 0.9715 (joint best with Arc.CC) for sample sizes 10 and 100 respectively. However, for other values of p (more generally p>0.1), the Pois estimator overestimates the coverage probability and has larger confidence intervals relative to the other estimators. Therefore, the Pois estimator has smaller index values compared with the other estimators, and hence, is not appropriate for the estimation of p.  

Furthermore, the Exact estimator overestimates the coverage probability in all cases. In addition, it has large confidence interval lengths and these are shown in its index values. This is consistent with results reported in <cit.>. In most cases, estimators such as Wilson, AgreC, Ag.add4 and midP have relatively good coverage properties and interval lengths and are shown in their I values usually approaching 1.

In general, for the estimation of a proportion, the index is able to distinguish between estimators that are appropriate or not based on their interval lengths and coverage probabilities.  




§ APPLICATION


To illustrate the application of our index, we consider the paper by <cit.>. The authors compared several confidence interval estimators for the coefficient of variation (CV). The coefficient of variation is defined as the variability of a random variable relative to its mean. It is usually expressed as a percentage. The confidence interval estimators of the CV were compared based on their interval lengths and the empirical coverage probabilities. The authors used separate plots for the coverage probabilities and the interval lengths across different sample sizes, CV values and distributions. We take a different approach in this paper by constructing plots showing simultaneously the coverage probabilities and the confidence interval index, I. 




The various confidence intervals considered and their abbreviations are presented in Table <ref>. We compute the confidence interval indexes for the estimators in Table 8 using the values in Table 4 of <cit.>. In addition, we assess the conclusions reached in that paper with that of the computed confidence interval indexes. The confidence interval indexes for each combination of n and CV are shown in Tables <ref>-<ref> in Appendix B. In addition, the plots of the coverage probabilities and the corresponding confidence interval index values are presented in Figure <ref>. We can now make inferences from the graphs and compare these with the conclusions reached in <cit.>.





Firstly, it can be seen that the most visible estimator that performs badly is the S.K estimator. It has mostly low coverage probability and this is reflected in it having smaller values on the index. It must be noted that some of the corresponding interval lengths of the S.K estimator were 2 to 8 times shorter than the other interval lengths. However, having a shorter interval length with low coverage probability is not practically desirable. As the sample size increases, there is a remarkable increase in the performance of the S.K estimator especially for CV=0.5. Therefore, we can conclude that, in this case, the index discriminates the bad estimator from the good ones even though shorter interval lengths were recorded. 

Secondly, <cit.> concludes that “By n = 100, almost all intervals are performing at a similar level (Figure 1). All C.P intervals (C.P, Med C.P, and BS C.P) over exceeded the expected coverage probability of 95% and reached 100% and are clear outliers". From Figure <ref>, it can easily be seen from  the bottom panel (i.e. for n=100) that the index values for these estimators are smaller compared to the other estimators: this indicates that the C.P-based estimators are inappropriate for the estimation of the CV relative to the other estimators.

Thirdly, we can plot the index against sample size and CV values as an alternative to the four cases: CP against sample size; CP against CV; Interval length against n; and interval length against CV. These graphs, not shown here, lead to the same conclusions obtained in <cit.>. 

In general, the index values are consistent with the conclusions from the CP and the interval lengths. Therefore, the index provides a useful, but computationally inexpensive method for measuring the relative performance of the estimators of confidence intervals for CV.  



§ CONCLUSION


In this paper, an index for measuring the performance of confidence interval estimators was proposed. The index is based on the traditional trade-off between confidence interval length and empirical coverage probability. Unlike the confidence interval length which has range, ^+,  the index has range of values within that of the coverage probability. We showed that index values close to 1 indicate a good confidence interval estimator whereas values far removed from 1 indicate a bad confidence interval estimator. Thus, it can easily be superimposed on a plot of coverage probabilities to aid in the selection of estimators with good coverage probabilities and interval lengths. The index can be used alone or to complement the coverage probability for measuring the performance of confidence interval estimators. 

In all the simulations and practical application, we assessed the performance of estimators through the sizes of the values of the index. However, an issue of practical importance is the statistical difference between indexes. In practice, we propose that a hypothesis of equality or otherwise can be performed on any observed differences between indexes. Since the sampling distribution of the index remains an open problem, a non-parametric or the estimation of standard errors based on resampling methods can be used in such a test.  













99
	
	
	
	
	
	
	
	
	Agresti2000
	Agresti, A., Caffo, B.: Simple and Effective Confidence Intervals for
		Proportions and Differences of Proportions Result from Adding Two Successes
		and Two Failures.
	 The American Statistician 54(4), 280–288 (2000)
	
	Agresti1998
	Agresti, A., Coull, B.A.: Approximate is Better than “Exact" for Interval
		Estimation of Binomial Proportions.
	The American Statistician 52(2), 119–126 (1998)
	
	Banik2010
	Banik, S., Kibria, B.M.G.: Comparison of Some Parametric and Nonparametric Type One Sample Confidence Intervals for Estimating the Mean of a Positively Skewed Distribution.
	Communications in Statistics -Simulation and Computation 39,
	361–389 (2010)
	
	Beran1987
	Beran, R. (1987). Prepivoting to Reduce Level Error of Confidence Sets.
	Biometrika 74(3), 457–468 
	
	Brown2001
	Brown, L.D., Cai, T.T., DasGupta, A. (2001). Interval Estimation for a Binomial
		Proportion. 
	Statistical Science 16(2), 101–117 
	
	Brown2002
	Brown, L.D., Cai, T.T., DasGupta, A. (2002). Confidence Intervals for a Binomial
		Proportion and Asymptotic Expansions.
	The Annals of Statistics 30(1), 160–201 
	
	Curto2009
	Curto, J.D., Pinto, J.C. (2009). The Coefficient of Variation Asymptotic Distribution
		in the Case of Non-iid Random Variables.
	Journal of Applied Statistics 36(1), 21–32 
	
	Efron1993
	Efron, B., Tibshirani, R.J. (1993). An Introduction to the Bootstrap.
	Chapman and Hall, London 
	
	Gulhar2012
	Gulhar, M., Golam Kibria, B.M., Albatineh, A.N., Ahmed, N.U. (2012). A Comparison of Some Confidence Intervals for Estimating the population coefficient of variation: A simulation study.
	SORT 36(1), 45–68 
	
	Huber1992
	Huber, P.J. (1992). Robust estimation of a location parameter.
	In: Breakthroughs in Statistics, pp. 492–518. Springer 
	
	Johnson1978
	Johnson, N.J. (1978). Modified t Tests and Confidence Intervals for Asymmetrical
		Populations Modified L Tests and Confidence Intervals for Asymmetrical
		Populations.
	Journal of the American Statistical Association 73(363),
	536–544 
	
	Kilian2000
	Kilian, L., Chang, P.L. (2000). How accurate are confidence intervals for impulse
	responses in large var models?
	Economics Letters 69(3), 299–307 
	
	Lee2003
	Lee, S.M.S., Young, G.A. (2003). Prepivoting by Weighted Bootstrap Iteration.
	Biometrika 90, 393–410 
	
	Leemis1996
	Leemis, L., Trivedi, K. (1996). A Comparison of Approximate Interval Estimators for
		the Bernoulli Parameter.
	The American Statistician 50(1), 1–20 
	
	Loh1987
	Loh, W.Y. (1987). Calibrating Confidence Coefficients.
	Journal of the American Statistical Association 82(397),
	155–162 
	
	Loh1988
	Loh, W.Y. (1988). Discussion: Theoretical Comparison of Bootstrap Confidence
		Intervals.
	The Annals of Statistics 16(3), 972–976 
	
	Loh1991
	Loh, W.Y.: Bootstrap Calibration for Confidence Interval Construction and
		Selection.
	Statistica Sinica 1(2), 477–491 (1991)
	
	Martin1990
	Martin, M. (1990). On the Double Bootstrap.
	Tech. rep., (Report No. 347) Department of Statistics, Stanford
	University, California. 
	
	McKay1932
	McKay, A.T.: Distribution of the Coefficient of Variation and the Extended t	Distribution.
	Journal of the Royal Statistical Society 95 (4), 695–698
	(1932)
	
	Miller1991
	Miller, G.E. (1991). Asymptotic Test Statistics for Coefficients of Variation.
	 Communications in Statistics - Theory and Methods 20(10),
	3351–3363 
	
	Nankervis2005
	Nankervis, J.C. (2005). Computational Algorithms for Double Bootstrap Confidence
		Intervals.
	 Computational Statistics & Data Analysis 49, 461–475
	
	
	Panich2009
	Panichkitkosolkul, W. (2009). Improved Confidence Intervals for a Coefficient of Variation of a Normal Distribution.
	Thailand Statistician 7(2), 193–199 
	
	Pires2008
	Pires, A.M., Amado, C. (2008). Interval Estimators for a Binomial Proportion: Comparison of Twenty Methods.
	REVSTAT 6(2), 165–197 
	
	Sharma1994
	Sharma, K., Krishna, H. (1994). Asymptotic Sampling Distribution of Inverse
		Coefficient-of-Variation and its Applications.
	IEEE Transactions on Reliability 43(4), 630–633 
	
	Vangel2012
	Vangel, M.G. (1996). Confidence Intervals for a Normal Coefficient of Variation.
	The American Statistician 50(1), 21–26 
	
	Wilson1927
	Wilson, E.B. (1927). Probable Inference, the Law of Succession, and Statistical
		Inference.
	Journal of the American Statistical Association 22(158),
	209–212 
	
	Zaane2012
	Zaane, B.V., Vergouwe, Y., Donders, A.R.T., Moons, K.G.M. (2012) Comparison of
		Approaches to Estimate Confidence Intervals of Post-test Probabilities of
		Diagnostic Test Results in a Nested Case-Control Study.
	 BMC Medical Research Methodology 12(166), 1–9 
	










toc













§ APPENDIX A











§ APPENDIX B















