Selective Inference via Marginal Screening for High Dimensional Classification
Yuta Umezu Ichiro Takeuchi





Yuta Umezu Nagoya Institute of Technology,  Aichi, Japan 

           Ichiro Takeuchi Nagoya Institute of Technology,  Aichi, Japan/
              RIKEN Center for Advanced Intelligence Project, Tokyo, Japan/
              Center for Materials Research by Information Integration, National Institute for Materials Science, Ibaraki, Japan 

              ichiro.takeuchi@nitech.ac.jp           






Selective Inference via Marginal Screening for High Dimensional Classification
    Yuta Umezu         Ichiro Takeuchi 

    Received: date / Accepted: date
==============================================================================





Post-selection inference is a statistical technique for determining salient variables after model or variable selection. Recently, selective inference, a kind of post-selection inference framework, has garnered  the attention in the statistics and machine learning communities. By conditioning on a specific variable selection procedure, selective inference can properly control for so-called selective type I error, which is a type I error conditional on a variable selection procedure, without imposing excessive additional computational costs. While selective inference can provide a valid hypothesis testing procedure, the main focus has hitherto been on Gaussian linear regression models. In this paper, we develop a selective inference framework for binary classification problem. We consider a logistic regression model after variable selection based on marginal screening, and derive the high dimensional statistical behavior of the post-selection estimator. This enables us to asymptotically control for selective type I error for the purposes of hypothesis testing after variable selection. We conduct several simulation studies to confirm the statistical power of the test, and compare our proposed method with data splitting and other methods.






§ INTRODUCTION


Discovering statistically significant variables in high dimensional data is an important problem for many applications such as bioinformatics, materials informatics, and econometrics, to name a few.
To achieve this, for example in a regression model, data analysts often attempt to reduce the dimensionality of the model by utilizing a particular model selection or variable selection method.
For example, the Lasso <cit.> and marginal screening <cit.> are frequently used in model selection contexts.
In many applications, data analysts conduct statistical inference based on the selected model as if it is known a priori, but this practice has been referred to as “a quiet scandal in the statistical community” in <cit.>.
If we select a model based on the available data, then we have to pay heed to the effect of model selection when we conduct a statistical inference.
This is because the selected model is no longer deterministic, i.e., random, and statistical inference after model selection is affected by selection bias.
In hypothesis testing of the selected variables, the validity of the inference is compromised when a test statistic is constructed without taking account of the model selection effect.
This means that, as a consequence, we can no longer effectively control type I error or the false positive rate.
This kind of problem falls under the banner of post-selection inference in the statistical community and is recently attracted a lot of attention <cit.>.

Post-selection inference consists of the following two steps: 


  Selection: The analyst chooses a model or subset of variables and constructs hypothesis, based on the data.

  Inference: The analyst tests the hypothesis by using the  selected model.

Broadly speaking, the selection step determines what issue to address, i.e., a hypothesis selected from the data, and the inference step conducts hypothesis testing to enable a conclusion to be drawn about the issue under consideration.
To navigate the issue of selection bias, there are several approaches for conducting the inference step.

Data splitting is the most common procedure for selection bias correction.
In a high dimensional linear regression model, <cit.> and <cit.> succeed in assigning a p-value for each selected variable by splitting the data into two subsets.
Specifically, they first reduce the dimensionality of the model using the first subset, and then make the final selection using the second subset of the data, by assigning a p-value based on a classical least square estimation.
While such a data splitting method is mathematically valid straightforward to implement, it leads to low power for extracting truly significant variables because only sub-samples, whose size is obviously smaller than that of the full sample, can be used in each of the selection and inference steps.

As an alternative, simultaneous inference, which takes account all possible subsets of variables, has been developed for correcting selection bias.
<cit.> showed that the type I error can be successfully controlled even if the full sample is used in both the selection and inference steps by adjusting multiplicity of model selection.
Since the number of all possible subsets of variables increases exponentially, computational costs associated with this method become excessive when the dimension of parameters is greater than 20.

On the other hand, selective inference, which only takes the selected model into account, is another approach for post-selection inference, and provides a new framework for combining selection and hypothesis testing.
Since hypothesis testing is conducted only for the selected model, it makes sense to condition on an event that “a certain model is selected”.
This event is referred to as a selection event, and we conduct hypothesis testing conditional on the event.
Thus, we can avoid having to compare coefficients across two different models.
Recently, <cit.> succeeded in using this method to conduct hypothesis testing through constructing confidence intervals for selected variables by the Lasso in s linear regression modeling context.
When a specific confidence interval is constructed, the corresponding hypothesis testing can be successfully conducted
They also show that the type I error, which is also conditioned on the selection event and is called selective type I error, can be appropriately controlled.
It is noteworthy that by conditioning on the selection event in a certain class, we can construct exact p-values in the meaning of conditional inference based on a truncated normal distribution.

Almost all studies which have followed since the seminal work by <cit.>, however, focus on linear regression models.
Particularly, normality of the noise is crucial to control selective type I error.
To relax this assumption, <cit.> developed an asymptotic theory for selective inference in a generalized linear modeling context.
Although their results can be available for high dimensional and low sample size data, we can only test a global null hypothesis, that is, a hypothesis that all regression hypothesis is zero, just like with covariance test <cit.>.
On the other hand, <cit.> proposed a procedure to test individual hypotheses in a logistic regression model with the Lasso.
By debiasing the Lasso estimator for both the active and inactive variables, they require a joint asymptotic distribution of the debiased Lasso estimator and conduct hypothesis testing for regression coefficients individually.
However, the method is justified only for low dimensional scenarios since they exploit standard fixed dimensional asymptotics.

Our main contribution is that, by utilizing marginal screening as a variable selection method, we can show that the selective type I error rate for logistic regression model is appropriately controlled even in a high dimensional asymptotic scenario.
In addition, our method is applicable not only with respect to testing the global null hypothesis but also hypotheses pertaining to individual regression coefficients.
Specifically, we first utilize marginal screening for the selection step in a similar way to <cit.>.
Then, by considering a logistic regression model for the selected variables, we derive a high dimensional asymptotic property of a maximum likelihood estimator.
Using the asymptotic results, we can conduct selective inference of a high dimensional logistic regression, i.e., valid hypothesis testing for the selected variables from high dimensional data.

The rest of the paper is organized as follows.
Section <ref> briefly describes the notion of selective inference and intruduces several related works.
In Section <ref>, the model setting and assumptions are described.
An asymptotic property of the maximum likelihood estimator of our model is discussed in Section <ref>.
In Section <ref>, we conduct several simulation studies to explore the performance of the proposed method before application to real world empirical data sets in Section <ref>.
Theorem proofs are relegated to Section <ref>.
Finally, Section <ref> offers concluding remarks and suggestions for future research in this domain.



 §.§ Notation

Throughout the paper, row and column vectors of X∈ℝ^n× d are denoted by x_i (i=1,…, n) and x̃_j, (j=1,…, d), respectively.
An n× n identity matrix is denoted by I_n.
The ℓ_2-norm of a vector is denoted by · provided there is no confusion.
For any subset J⊆{1,…, d}, its complement is denoted by J^={1,…,d}\ S.
We also denote v_J=(v_i)_i∈ J∈ℝ^|J| and X_J=(x_J,1,…,x_J,n)^⊤∈ℝ^n× |J| as a sub-vector of v and a sub-matrix of X, respectively.
For a differentiable function f, we denote f' and f” as the first and second derivatives and so on.



§ SELECTIVE INFERENCE AND RELATED WORKS


In this section, we overview fundamental notion of selective inference through a simple linear regression model <cit.>.
We also review related existing works on selective inference.



 §.§ Selective Inference in Linear Regression Model


Let y∈ℝ^n and X∈ℝ^n× d be a response and non-random regressor, respectively, and let us consider a linear regression model 

    y=Xβ^*+ε,

where β^* is the true regression coefficient vector and ε is distributed according to N(0,σ^2 I_n) with known variance σ^2.
Suppose that a subset of variables S is selected in the selection step (e.g., Lasso or marginal screening as in <cit.>) and let us consider hypothesis testing for j∈{1,…, |S|}:

    H_0,j:β_S, j^*=0
        vs.    H_1,j:β_S, j^*≠ 0.

If S is non-random, a maximum likelihood estimator β̂_S=(X_S^⊤ X_S)^-1X_S^⊤y is distributed according to N(β_S^*,σ^2(X_S^⊤ X_S)^-1), as is well-known.
However, we cannot use this sampling distribution when S is selected based on the data, and the selected variable S is also random.

If a subset of variables, i.e., the active set, Ŝ is selected by the Lasso or marginal screening, the event {Ŝ=S} can be written as an affine set with respect to y, that is, in the form of {y;Ay≤b} for some non-random matrix A and vector b <cit.>, in which the event {Ŝ=S} is called a selection event.
<cit.> showed that if y follows a normal distribution and the selection event can be written as an affine set, the following lemma holds:



Suppose y∼ N(μ,Σ).
Let c=Ση(η^⊤Ση)^-1 for any η∈ℝ^n, and let z=(I_n-cη^⊤)y.
Then we have

    {y;Ay≤b}={y; L(z)≤η^⊤y≤ U(z), N(z)≥ 0},

where

    L(z)
    =max_j:(Ac)_j<0b_j-(Az)_j/(Ac)_j,     
    U(z)
    =min_j:(Ac)_j>0b_j-(Az)_j/(Ac)_j

and N(z)=max_j:(Ac)_j=0b_j-(Az)_j.
In addition, (L(z),U(z),N(z)) is independent of η^⊤y.

By using the lemma, we can find that the distribution of the pivotal quantity for η^⊤μ is given by a truncated normal distribution.
Specifically, let F^[L,U]_μ,σ^2 be a cumulative distribution function of a truncated normal distribution TN(μ, σ^2, L, U), that is,

    F^[L,U]_μ,σ^2(x)
    =Φ((x-μ)/σ)-Φ((L-μ)/σ)/Φ((U-μ)/σ)-Φ((L-μ)/σ),

where Φ is a cumulative distribution function of a standard normal distribution.
Then, for any value of z, we have

    [
    F^[L(z),U(z)]_η^⊤μ,η^⊤Ση(η^⊤y)
    | Ay≤b]
     ∼ Unif(0,1),

where L(z) and U(z) are defined in the above lemma.
This pivotal quantity allows us to construct a so-called selective p-value.
Precisely, by choosing η=X_S(X_S^⊤ X_S)^-1e_j, we can construct a right-side selective p-value as

    P_j
    =1-F^[L(z_0), U(z_0)]_0,η^⊤Ση(η^⊤y),

where e_j∈ℝ^|S| is a unit vector whose j-th element is 1 and 0 otherwise, and z_0 is a realization of z.
Note that the value of P_j represents a right-side p-value conditional on the selection event under the null hypothesis H_0,j:β_S,j^*=η^⊤μ=0 in (<ref>).
In addition, for the j-th test in (<ref>), a two-sided selective p-value can be defined as 

    P̃_j=2min{P_j, 1-P_j },

which also follows from standard uniform distribution under the null hypothesis.
Therefore, we reject the j-th null hypothesis at level α when P̃_j≤α, and the probability

    P(H_0,j is falsely rejected|Ŝ=S)
    =P(P̃_j≤α|Ŝ=S)

is referred to as a selective type I error.




 §.§ Related Works

In selective inference, we use the same data in variable selection and statistical inference.
Therefore, the selected model is not deterministic and we can not apply classical hypothesis testing due to selection bias.

To navigate this problem, data splitting has been commonly utilized.
In data splitting, the data are randomly divided into two disjoint sets, and one of them is used for variable selection and the other is used for hypothesis testing.
This is a particularly versatile method and is widely applicable if we can divide the data randomly <cit.>.
Since the data are split randomly, i.e., independent of the data, we can conduct hypothesis testing in the inference step independent of the selection step.
Thus, we do not need to concerned with selection bias. 
It is noteworthy that data splitting can be viewed as a method of selective inference because the inference is conducted only for the selected variables in the selection step.
However, a drawback of data splitting is that only a part of the data are available for each split, precisely because the essence of this approach involves rendering some data available for the selection step and the remainder for the inference step.
Because only a subset of the data can be used in variable selection, the risk of failing to select truly important variables increases.
Similarly, the power of hypothesis testing would decrease since inference proceeds on the basis of a subset of the total data.
In addition, since data splitting is executed at random, it is possible and plausible that the final results and conclusions will vary non-trivially depending on exactly how this split is manifested.

On the other hand, in the traditional statistical community, simultaneous inference has been developed for correcting selection bias <cit.>.
In simultaneous inference, type I error is controlled at level α by considering all possible subsets of variables.
Specifically, let Ŝ⊆{1,…, d} be the set of variables selected by a certain variable selection method and P_j(Ŝ) be a p-value for the j-th selected variable in Ŝ.
Then, in simultaneous inference, the following type I error should be adequately controlled:

    P(P_j(Ŝ)≤α for any Ŝ⊆{1,…,d})≤α.

To examine the relationship between selective inference and simultaneous inference, note that the left-hand side in (<ref>) can be rewritten as

    P(P_j(Ŝ)≤α for any Ŝ⊆{1,…,d}) 
       =∑_S⊆{1,…,d} P(P_j(S)≤α|Ŝ=S) P(Ŝ=S).

The right-hand side in the above equality is simply a weighted sum of selective type I errors over all possible subsets of variables.
Therefore, if we control selected type I errors for all possible subsets of variables, we can also control type I errors in the sense of simultaneous inference.
However, because the number of all possible subsets of variables is 2^d, it becomes overly cumbersome to compute the left-hand side in (<ref>) even for d=20.
In contrast to simultaneous inference, selective inference only considers the selected variables, and thus the computational cost is low compared to simultaneous inference.
 
Following the seminal work of <cit.>, selective inference for variable selection has been intensively studied <cit.>.
All these methods, however, rely on the assumption of normality of the data.



 §.§ Beyond Normality


It is important to relax the assumption of the normality for applying selective inference to more general cases such as generalized linear models.
To the best of our knowledge, there is death of research into selective inference in such a generalized setting.
Here, we discuss the few studies which do exist in this respect.

<cit.> derived an exact post-selection inference for a natural parameter of exponential family, and obtained the uniformly most powerful unbiased test in the framework of selective inference.
However, as suggested in their paper, the difficulty in constructing exact inference in generalized linear models emanates from the discreteness of the response distribution.

Focusing on an asymptotic behavior in a generalized linear model context with the Lasso penalty, <cit.> directly considered the asymptotic property of a pivotal quantity.
Although their work can be applied in high dimensional scenarios, we can only test a global null, that is, H_0:β^*=0, except for the linear regression model case.
This is because that, when we conduct selective inference for individual coefficient, the selection event does not form a simple structure such as an affine set.

On the other hand, <cit.> proposed a procedure to test individual hypotheses fin logistic regression model context based on the Lasso.
Their approach is fundamentally based on solving the Lasso by approximating the log-likelihood up to the second order, and on debiasing the Lasso estimator.
Because the objective function now becomes quadratic as per the linear regression model, the selection event reduces to a relatively simple affine set.
After debiasing the Lasso estimator, they derive an asymptotic joint distribution of active and inactive estimators.
However, since they required d dimensional asymptotics, high dimensional scenarios can not be supported in their theory.

In this paper, we extend selective inference for logistic regression in <cit.> to high dimensional settings in the case where variable selection is conducted by marginal screening. 
We do not consider asymptotics for a d dimensional original parameter space, but for a K dimensional selected parameter space. 
Unfortunately, however, we cannot apply this asymptotic result directly to the polyhedral lemma (Lemma <ref>) in <cit.>. 
To tackle this problem, we consider a score function for constructing a test statistic for our selective inference framework.
We first define a function T_n(β_S^*) based on a score function as a “source” for constructing a test statistic.
To apply the polyhedral lemma to T_n(β_S^*), we need to asymptotically ensure that 
i) the selection event is represented by affine constraints with respect to T_n(β_S^*), and 
ii) the function in the form of η^⊤T_n(β_S^*) is independent of the truncation points. 
Our main technical contribution herein is that, by carefully analyzing problem configuration and by introducing reasonable additional assumptions, we can show that those two requirements for the polyhedral lemma are satisfied asymptotically.

Figure <ref> shows the asymptotic distribution of selective p-values in our setting and in <cit.> based on 1,000 Monte-Carlo simulation.
While the theory in <cit.> does not support high dimensionality, their selective p-value (red solid line) appears to effective in high dimensional scenarios, although it is slightly mode conservative compared to the approach developed in this paper (black solid line).
Our high dimensional framework means that the number of selected variables grows with the sample size in an appropriate order, and a proposed method allows us to test (<ref>) individually even in high dimensional contexts.





§ SETTING AND ASSUMPTIONS


As already noted, our objective herein is to develop a selective inference approach applicable to logistic regression models when the variables are selected by marginal screening.
Let (y_i,x_i) be the i-th pair of the response and regressor.
We assume that the y_i's are independent random variables which take values in {0,1}, and the x_i's are a d dimensional vector of known constants.
Further, let X=(x_1,…,x_n)^⊤∈ℝ^n× d and y=(y_1,…,y_n)^⊤∈{0,1}^n.
Unlike <cit.>, we do not require that the dimension d be fixed, that is, d may increase, as well as the sample size n.



 §.§ Marginal Screening and Selection Event



In this study, we simply select variables based on a score between the regressor and response z=X^⊤y as per a linear regression problem.
Specifically, we select the top K coordinates of absolute values in z, that is, 

    Ŝ={j;|z_j| is among the first  K largest of all}.

To avoid computational issues, we consider the event {(Ŝ, s_Ŝ)=(S, s_S)} as a selection event (see, e.g., <cit.>).
Here, s_S is a vector of sign z_j (j∈ S).
Then, the selection event {(Ŝ, s_Ŝ)=(S, s_S)} can be rewritten as

    |z_j|≥ |z_k|,
        ∀ (j,k)∈ S× S^,

which is equivalent to

    -s_jz_j≤ z_k≤ s_jz_j,
      
    s_j z_j≥ 0,
        ∀(j,k)∈ S× S^.

Therefore, {(Ŝ, s_Ŝ)=(S, s_S)} is reduced to an affine set {z; Az≤0} for an appropriate {2K(d-K)+K}× d dimensional matrix A.

In the following, we assume that a sure screening property holds.
This is desirable property for variable selection <cit.> and the statement is as follows:


  (C0) For the true active set S^*={j;β_j^*≠ 0}, the probability P(Ŝ⊃ S^*) converges to 1 as n goes to infinity.

In the above assumption, we denote β^*∈ℝ^d as a true value of the coefficient vector.
This assumption requires that the set of selected variables contain the set of true active variables with probability tending to 1.
In the linear regression model, (C0) holds under some regularity conditions in high dimensional settings <cit.>.
The sufficient condition concerning about high dimensionality for (C0) is log d=O(n^ξ) for some ξ∈(0,1/2), and thus we allow d to be exponentially large.
Because (C0) is not directly related in selective inference, we do not further discuss it.




 §.§ Selective Test

For a subset of variables Ŝ (=S) selected by marginal screening, we consider K selective tests (<ref>) for each variable β_j^*, j∈ S.
Let us define the loss function of logistic regression with the selected variables as follows:

    ℓ_n(β_S)
    =∑_i=1^n{y_ix_S,i^⊤β_S-ψ(x_S,i^⊤β_S)},

where ψ(x_S,i^⊤β_S)=log(1+exp(x_S,i^⊤β_S)) is a cumulant generating function.
Observe that ℓ_n(β_S) is concave with respect to β_S.
Thus we can define the maximum likelihood estimator of β_S as the optimal solution that attains the maximum of the following optimization problem:

    β̂_S
    =_β_S∈ Bℓ_n(β_S),

where B⊆ℝ^K is a parameter space.

Suppose that S (⊃ S^*) is fixed.
Then, it holds that

    ψ'(x_S,i^⊤β_S^*)=ψ'(x_S^*,i^⊤β_S^*^*),   ψ”(x_S,i^⊤β_S^*)=ψ”(x_S^*,i^⊤β_S^*^*),

and thus, we have

    P(y_i=1)= E[y_i]=ψ'(x_S^*,i^⊤β_S^*^*),    V[y_i]=ψ”(x_S^*,i^⊤β_S^*^*).



We construct test statistics for (<ref>) by deriving an asymptotic distribution of β̂_S.
To develop our asymptotic theory, we further assume the following conditions in addition to (C0) for a fixed S with |S|=K:


  (C1)
max_ix_S,i= O(√(K)).
In addition, for a K× K dimensional matrix

    Ξ_S,n=1/nX_S^⊤ X_S=1/n∑_i=1^nx_S,ix_S,i^⊤∈ℝ^K× K,

the following holds:

    0<C_1<λ_ min(Ξ_S,n)≤λ_ max(Ξ_S,n)<C_2<∞,

where C_1 and C_2 are constants that depend on neither n nor K.

  (C2)
There exists a constant ξ (<∞) such that max_i|x_S,i^⊤β_S^*|<ξ.
In addition, parameter space B is 

    B={β_S∈ℝ^K;max_i|x_S,i^⊤β_S|<ξ̃}

for some constant ξ̃ (∈(ξ,∞)).

  (C3)
K^3/n= o(1).

  (C4)
For any p× q dimensional matrix A, we denote the spectral norm of A by A=sup_v≠0Av/v.
Then the following holds:

    1/√(n)X_S^^⊤ X_S= O(K).


The condition (C1) pertains to the design matrix.
Note that we only consider a high dimensional and small sample size setting for the original data set, and not for selected variables.
This assumption is reasonable for high dimensional and large sample scenarios.
(C2) requires that P(y_i=1) not converge to 0 or 1 for any i=1,…,n.
Observe that the parameter space B is an open and convex set with respect to β_S.
This assumption naturally holds when the space of regressors is compact and β_S does not diverge.
In addition, if the maximum likelihood estimator β̂_S is √(n/K)-consistent, then β̂_S lies in B with probability converging to 1.
The condition (C3) represents the relationship between the sample size and the number of selected variables for high dimensional asymptotics in our model.
As related conditions, <cit.> employs K^5/n→0, and <cit.> employs K^6+δ/n→0 for some δ>0 to derive an asymptotic expansion of a posterior distribution in a Bayesian setting.
Furthermore, <cit.> employs the same condition as in (C3) in the scenario for M-estimation.
Finally, (C4) requires that regressors of selected variables and those of unselected variables be only weakly correlated.
A similar assumption is required in <cit.> for deriving an asymptotic distribution for a bridge estimator.
This type of assumption, e.g., a restricted eigenvalue condition <cit.>, is essential for handling high dimensional behavior of the estimator.




§ PROPOSED METHOD


In this section, we present the proposed method for selective inference for high dimensional logistic regression with marginal screening.
We first consider a subset of features Ŝ = S (⊃ S^*) as a fixed set, and derive an asymptotic distribution of β̂_S under the assumptions (C1) – (C3). 
Then, we introduce the “source” of the test statistic T_n(β_S^*), which is defined by a score function, and apply it to the polyhedral lemma, where we will show that the truncation points are independent of the η^⊤ T_n(β_S^*) with the assumption (C4).

To extend the selective inference framework to logistic regression, we first consider a subset of variables Ŝ=S (⊃ S^*) as a fixed set.
From (<ref>), let us define a score function and observed information matrix by

    s_n(β_S)
       =1/√(n)ℓ'_n(β_S) 
    =1/√(n)∑_i=1^nx_S,i(y_i-ψ'(x_S,i^⊤β_S))
    andΣ_n(β_S)
       =-1/nℓ”_n(β_S)
    =1/n∑_i=1^nψ”(x_S,i^⊤β_S)x_S,ix_S,i^⊤,

respectively.
To simplify the notation, we denote s_n(β_S^*) and Σ_n(β_S^*) by s_n and Σ_n, respectively, for the true value of β_S^*.
Because ψ”(x_S,i^⊤β_S^*) is uniformly bounded on B from (C2), Σ_n is a symmetric and positive definite matrix when (C1) holds.
Then, by the same argument as in <cit.>, if K^2/n→ 0, we have

    β̂_S-β_S^*= O_ p(√(K/n)).

By using Taylor's theorem, we have

    0
    =ℓ'_n(β̂_S)
    ≈√(n)s_n-nΣ_n(β̂_S-β_S^*),

and thus

    √(n)(β̂_S-β_S^*)
    ≈Σ_n^-1s_n.

As per Remark 1, S⊃ S^* implies

    E[s_n]
    =1/√(n)∑_i=1^nx_S,i( E[y_i]-ψ'(x_S,i^⊤β_S^*))
    =0.

In addition, because the y_i's are independent of each other, we observe that

    V[s_n]
    =1/n∑_i=1^n V[y_i]x_S,ix_S,i^⊤
    =Σ_n.

Therefore, by recalling asymptotic normality of the score function, we expect that a distribution of Σ_n^-1s_n can be approximated by a normal distribution with mean 0 and covariance matrix Σ_n^-1.
Indeed, if S is fixed, this approximation is true under the conditions (C1) – (C3):


Suppose that the conditions (C1) – (C3) hold.
Then, for any fixed S (⊃ S^*) and η∈ℝ^K with η<∞, we have

    √(n)σ_n^-1η^⊤(β̂_S-β^*_S)
    =σ_n^-1η^⊤Σ_n^-1s_n+ o_ p(1)  d→ N(0,1),

where σ_n^2=η^⊤Σ_n^-1η and o_ p(1) is a term that converges to 0 in probability uniformly with respect to η and S.


Note that, under the conditions (C1), (C2) and d^3/n→ 0, Theorem <ref> also holds when we do not enforce variable selection (see e.g., <cit.>).
To formulate a selective test, let us consider 

    T_n(β_S^*)
    =Σ_n^-1s_n
    =Σ_n^-1{1/√(n)X_S^⊤(y-ψ'(β_S^*))
    }

as a “source" of a test statistic, where ψ'(β_S^*)=(ψ'(x_S,i^⊤β_S^*))_i=1,…, n.
The term “source" means that we cannot use it as a test statistic directly because T_n(β_S^*) depends on β_S^*.
In the following, for notational simplicity, we denote T_n(β_S^*) and ψ'(β_S^*) by T_n and ψ', respectively.

As noted in Section <ref>, by using an appropriate non-random matrix A∈ℝ^K(2d-2K+1)× d, the marginal screening selection event can be expressed as an affine constraint with respect to z=X^⊤y, that is, {z;Az≤0}.
Then, by appropriately dividing A and X based on the selected S, we can rewrite it as follows:

    Az≤0  ⇔  
    A_SX_S^⊤y+A_S^X_S^^⊤y≤0  ⇔  ÃT_n≤b̃.

The last inequality is an affine constraint with respect to T_n, where

    Ã=A_SΣ_n
        and    b̃=-1/√(n)(A_SX_S^⊤ψ'+A_S^X_S^^⊤y).

Unlike the polyhedral lemma in Section <ref>, b̃ depends on y and so is a random vector.
By using (C4), we can prove that b̃ is asymptotically independent of η^⊤T_n, which implies the polyhedral lemma holds asymptotically.


Suppose that (C1) – (C4) all hold.
Let c=Σ_n^-1η/σ_n^2 for any η∈ℝ^K with η<∞, and w=(I_K-cη^⊤)T_n, where σ_n^2=η^⊤Σ_n^-1η.
Then, for any fixed S (⊃ S^*), the selection event can be expressed as

    {T;ÃT≤b̃}
    ={T;L_n≤η^⊤T≤ U_n,N_n=0},

where

    L_n
    =max_l:(Ãc)_l<0b̃_l-(Ãw)_l/(Ãc)_l,
        
    U_n
    =min_l:(Ãc)_l>0b̃_l-(Ãw)_l/(Ãc)_l,

and N_n=max_l:(Ãc)_l=0b̃_l-(Ãw)_l.
In addition, (L_n,U_n,N_n) is asymptotically independent of η^⊤T_n.


As a result of Theorem <ref>, Theorem <ref> and (C0), we can asymptotically identify a pivotal quantity as a truncated normal distribution, that is, by letting η=e_j∈ℝ^K, 

    [F^[L_n,U_n]_0,σ_n^2(η^⊤T_n) |ÃT_n≤b̃]
     d→ Unif(0,1)

for any w, under H_0,j.
Therefore, we can define an asymptotic selective p-value for selective test (<ref>) under H_0,j as follows:

    P_n,j=2min{F^[L_n,U_n]_0,σ_n^2(η^⊤T_n), 1-F^[L_n,U_n]_0,σ_n^2(η^⊤T_n) },

where L_n and U_n are evaluated at the realization of w=w_0.
Unfortunately, because T_n, Σ_n, L_n and U_n are still dependent on the true value of β_S^*, we construct a test statistic by introducing a maximum likelihood estimator (<ref>), which is a consistent estimator of β_S^*.




 §.§ Computing Truncation Points

In practice, we need to compute truncation points in (<ref>).
When we utilize marginal screening for variable selection, it becomes difficult to compute L_n and U_n because Ã becomes a {2K(d-K)+K}× K dimensional matrix.
For example, even when d=1,000 and K=20, we need to handle a 39,220 dimensional vector.
To reduce the computational burden, we derive a simple form of (<ref>) in this section.

We first derive A_S.
As notedd in Section <ref>, selection event {(Ŝ, s_Ŝ)=(S, s_S)} can be rewritten as

    -s_jz_j≤ z_k≤ s_jz_j, s_j z_j≥ 0,
         ∀ (j,k)∈ S× S^,

where s_j= sgn(z_j) is the sign of the j-th element of z=X^⊤y.
Let S={j_1,…, j_K} and q=2(d-K)+1.
Then, by a simple calculation, we have

    A_S
    =( [ -s_j_11_q         O;                   ⋱          ;         O -s_j_K1_q ])
    =-J⊗1_q,

where J is a K× K dimensional diagonal matrix whose j-th diagonal element is s_j and ⊗ denotes a Kronecker product.
Since Ã=A_SΣ_n and c=Σ_n^-1η/σ_n^2, the denominator in (<ref>) reduces to Ãc=A_Sη//σ_n^2.
For η=e_j, we can further evaluate A_Sη as

    A_Sη=-s_j(0_(j-1)q^⊤,1_q^⊤,0_(K-j)q^⊤)^⊤∈ℝ^Kq.

Further, by the definition of Ã, b̃, and w, we have

    b̃-Ãw
    =b̃-ÃT_n+(η^⊤T_n)Ãc
    =-1/√(n)Az+T_n,jÃc.

Because σ_n^2, the j-th diagonal element of Σ_n^-1, is positive, it is straightforward to observe that

    {l:(Ãc)_l<0}
    =
    {(j-1)q+1,…,jq},    if s_j=1 
    ∅,    otherwise

for j=1,…,K.
Note that, for each j=1,…,K, (Az)_l=(j-1)q+1,…,jq consists of q elements of z_j and z_j± z_k for any k∈ S^.
Therefore, for each j=1,…,K, we have

    max_l=(j-1)q+1,…,jq(Az)_l
    =max_k∈ S^{z_j,z_j± z_k}
    =z_j+max_k∈ S^|z_k|.

As a consequence, we obtain

    L_n
       =max_l:(Ãc)_l<0b̃_l-(Ãw)_l/(Ãc)_l
       =max_l:(Ãη)_l<0-(Az)_l/√(n)/(Ãη)_l/σ_n^2+T_n,j
       =σ_n^2/√(n)max_l=(j-1)q+1,…,jq(Az)_l+T_n,j
       =σ_n^2/√(n)(|z_j|+max_k∈ S^|z_k|)+T_n,j,

if s_j=1, and L_n=-∞, otherwise.
Similarly, we obtain

    U_n
       =min_l:(Ãc)_l>0b̃_l-(Ãw)_l/(Ãc)_l
       =min_l:(Ãη)_l>0-(Az)_l/√(n)/(Ãη)_l/σ_n^2+T_n,j
       =-σ_n^2/√(n)max_l=(j-1)q+1,…,jq(Az)_l+T_n,j
       =σ_n^2/√(n)(|z_j|-max_k∈ S^|z_k|)+T_n,j,

if s_j=-1, and U_n=∞, otherwise.
Because of this simple form, we can calculate truncation points efficiently.
We summarize the algorithm to compute selective p-values of the K selective test in Algorithm <ref>.






 §.§ Controlling Family-wise Error Rate


Since selective test (<ref>) consists of K hypotheses, we may be concerned about multiplicity when K>1.
In this case, instead of selective type I error, we control the family-wise error rate (FWER) in the sense of selective inference and we term it selective FWER.

For the selected variable Ŝ=S, let us denote a family of true null by H={ H_0,j: H_0,j(j∈ S) is true null}.
Then, let us define the selective FWER by

    sFWER
    = P(at least one  H_0,j∈ H is rejected|Ŝ=S)

in the same way as the classic FWER.
Next, we asymptotically control the selective FWER at level α by utilizing Bonferroni correction for K selective tests.
Specifically, we adjust selective p-values (<ref>) as follows.
Let us define α̃=α/K.
Since selective p-value P_n,j is asymptotically distributed according to Unif(0,1), we have that a limit superior of (<ref>) can be bounded as follows:

    lim sup_n→∞ P(⋃_j: H_0,j∈ H{P_n,j≤α̃}|Ŝ=S)
       ≤lim sup_n→∞∑_j: H_0,j∈ H P(P_n,j≤α̃|Ŝ=S) 
       ≤∑_j: H_0,j∈ Hlim sup_n→∞ P(P_n,j≤α̃|Ŝ=S) 
       ≤ | H|α̃≤α.

In the last inequality, we simply use | H|≤ K.
Accordingly, letting p_n,j be a realization of (<ref>), we reject a null hypothesis when {p_n,j≤α̃}.
In the following, we refer to p̃_n,j=min{1,Kp_n,j} as an adjusted selective p-value.
Note that we can utilize not only Bonferroni's method but also other methods for correcting multiplicity such as Scheffé's method, Holm's method, and so on.
We use Bonferroni's method for expository purposes.




§ SIMULATION STUDY


Through simulation studies, we explore the performance of the proposed method in Section <ref>, which we term ASICs (Asymptotic Selective Inference for Classification) here.

We first identify if the ASICs can control selective type I error.
We also check the selective type I error when data splitting (DS) and nominal test (NT) methods are used.
In DS, we first randomly divide the data into two disjoint sets.
Then, after selecting Ŝ=S with |S|=K by using one of these sets, we construct a test statistic T_n(β̂_S) based on the other sets and reject the j-th selective test (<ref>) when |T_n,j/σ_n|≥ z_α/2, where z_α/2 is an upper α/2-percentile of a standard normal distribution.
In NT, we cannot control type I errors since selection bias is ignored:
it selects K variables by marginal screening first, then rejects the j-th selective test (<ref>) when |T_n,j/σ_n|≥ z_α/2, where the entire data set is used for both selection and inference steps.
Finally, we explore whether the ASICs can effectively control selective FWER, and at the same time, confirm its statistical power by comparing it with that of DS.

The simulation settings are as follows.
As d dimensional regressor x_i (i=1,…,n), we used vectors obtained from N(0,Σ), where Σ is a d× d dimensional covariance matrix whose (j,k)-th element is set to ρ^|j-k|.
We set ρ=0 or 0.5 in Case 1 and Case 2, respectively.
Note that each element of x_i is independent in Case 1 but correlated in Case 2.
Then, for each x_i, we generate y_i from Bi(ψ'(x_i^⊤β^*)), where β^* is a d dimensional true coefficient vector and Bi(p) is a Bernoulli distribution with parameter p.
In the following, we conduct simulations using 1,000 Monte-Carlo runs.
We use the glm package in R for parameter estimation.




 §.§ Controlling Selective Type I Error



To check if ASICs can control selective type I error, we consider a selective test (<ref>).
Specifically, we first select K=1 variable by marginal screening and then conduct a selective test at the 5% level.
By setting β^*=0∈ℝ^d, we can confirm selective type I error because the selective null is always true.
Therefore, we assess the following index as an estimator of the selective type I error:
letting β be the selected variable in each simulation, we evaluate an average and standard deviation of

    I{ H_0 is rejected},

where I is an indicator function and H_0:β^*=0 is a selective null.
We construct a selective test at the 5% level in all simulations.
In the same manner as classical type I error, it is desirable when the above index is less than or equal to 0.05, with particularly small values indicating that the selective test is overly conservative.

Table <ref> presents averages and standard deviations of (<ref>) based on 1,000 runs.
It is clear that NT cannot control selective type I error;
it becomes larger as the dimension d increases.
In addition, NT does not improve even if the sample size becomes large, because there exist selection bias in the selection step.
On the other hand, both ASICs and DS adequately control selective type I error, although the latter appears slightly more conservative than the former.
Moreover, unlike NT, these two methods can adequately control selective type I error, even when the covariance structure of x_i and the number of dimensions change.






 §.§ FWER and Power


Here, we explore selective FWER and statistical power with respect to ASICs and DS for K selective tests (<ref>), where we set K=5, 10, 15, and 20.
Note that, as discussed in the above section, NT is disregarded here because it does no adequately control selective type I error.
We adjust multiplicity by utilizing Bonferroni's method as noted in Section <ref>.

The true coefficient vector is set to be β^*=(2×1_5^⊤,0_d-5^⊤)^⊤ and β^*=(2×1_5^⊤,-2×1_5^⊤,0_d-10^⊤)^⊤ in Model 1 and Model 2, respectively.
In the following, we assess the indices as an estimator of selective FWER and power.
Letting Ŝ=S be the subset of selected variables for each simulation, we evaluate an average of

    I{at least one  H_0,j∈ H is rejected}

and

    1/|S^*|∑_j∈ SI{ H_0,j∉ H is rejected},

where, for each j∈ S, H_0,j:β_j^*=0 is the selective null and H is a family of true nulls.
Note that, by using Bonferroni's method, we use α̃=α/K as an adjusted significance level for α=0.05.
Similar to the selective type I error, it is desirable when (<ref>) is less than or equal to α.
In addition, higher values of (<ref>) are desiable in the same manner as per classical power.
We evaluate (<ref>) as the proportion of rejected hypotheses for false nulls to that of true active variables.
We employ this performance index because it is important to identify how many truly active variables are extracted in practice.

Figure <ref> shows the average (<ref>) for each method.
ASICs and DS are both evaluated with respect to four values of K, thus eight lines are plotted in each graph.
Because of the randomness of simulation, some of the ASICs results are larger than 0.05 especially in small sample size and large variable dimension cases.
For both methods, it is clear that selective FWER tends to be controlled at the desired significance level, although DS is more conservative than ASICs.
To accord with our asymptotic theory, the number of selected variables must be K= o(n^1/3), which means that the normal approximation is not ensured in the case of K=15 and 20.
However, we observe that selective FWER is correctly controlled even in these cases, which suggests that assumptions (C3) and (C4) can be relaxed.



Figures <ref> and <ref> show the average of (<ref>) for each method and settings in Model 1 and Model 2, respectively.
In Case 1 of Figure <ref>, ASICs and DS have almost the same power for each K and d.
In addition, ASICs is clearly superior to DS in Case 2. 
This is reasonable since DS uses only the half of the data for inference.
On the other hand, in all cases, the power of ASICs becomes higher as the number of selected variables K decreases.
This can be explained by the condition (C3), that is, we need a much larger sample size when K becomes large for assuring the asymptotic result in Theorem <ref>.
In Figure <ref>, it is clear that the power of ASICs is superior in almost all settings.
However, neither AISCs nor DS appears to perform well when K=5.
In this case, the power of ASICs and DS cannot be improved by 50% or more.
This is because we can only select at most 5 true nonzero variables, while there are 10 true nonzero variables.








§ EMPIRICAL APPLICATIONS


We further explore the performance of the proposed method by applying it to several empirical data sets, all of which are available at LIBSVM[<https://www.csie.ntu.edu.tw/ cjlin/libsvmtools/datasets/>].
In all experiments, we standardize the design matrix X to make the scale of each variable the same.
We report adjusted selective p-values for selected variables.
To explore the selection bias, we also report naive adjusted p-values.
That is, we first compute p-values for selected variables based on NT, then we adjust these p-values by multiplying the number of selected variables.
The results are plotted in Figures <ref> – <ref>.
The result shows that almost all adjusted nominal p-values are smaller than those of selective inference, and the difference between these p-values is interpreted as the effect of selection bias.










§ THEORETICAL ANALYSIS


In this section, we provide proofs of the theoretical results derived herein.
We use the notation p≲ q, which means that, if for any p,q∈ℝ, there exists a constant r>0 such that p≤ rq, and p≳ q is defined similarly.
All proofs are based on fixed S (⊃ S^*);
thus we simply denote β̂_S and X_S by β̂ and X, respectively.
This is because we need to verify several asymptotic condition before selections in the same way as in <cit.>. 




 §.§ Proof of (<ref>)


Let α_n=√(K/n) and define a K dimensional vector u satisfying u=C for a sufficiently large C>0.
The concavity of ℓ_n implies

    P(β̂-β^*≤α_nC)
    ≥ P(sup_u=Cℓ_n(β^*+α_nu)<ℓ_n(β^*)),

and thus we need to show that for any ε>0, there exists a sufficiently large C>0 such that

    P(sup_u=Cℓ_n(β^*+α_nu)<ℓ_n(β^*))≥ 1-ε.

In fact, the  above inequality implies that β̂∈{β+α_nu; u≤ C}, that is, β̂-β^*= O_ p(α_n).

Observe that |ψ'(x_i^⊤β)|,|ψ”(x_i^⊤β)| and |ψ”'(x_i^⊤β)| are bounded uniformly with respect to β∈ B and i.
By using Taylor's theorem, we have

    ℓ_n(β^*+α_nu)-ℓ_n(β^*) 
       =∑_i=1^n[α_ny_ix_i^⊤u-{ψ(x_i^⊤(β^*+α_nu))-ψ(x_i^⊤β^*)}] 
       =α_n∑_i=1^n(y_i-ψ'(x_i^⊤β^*))x_i^⊤u
    -α_n^2/2∑_i=1^nψ”(x_i^⊤β^*)(x_i^⊤u)^2
    -α_n^3/6∑_i=1^nψ”'(θ_i)(x_i^⊤u)^3 
       ≡ I_1+I_2+I_3,

where for i=1,2,…,n, θ_i is in the line segment between x_i^⊤β^* and x_i^⊤(β^*+α_nu).
From (C1) and (C2), we observe that

    E[{∑_i=1^n(y_i-ψ'(x_i^⊤β^*))x_i^⊤u}^2] 
       =∑_i=1^n E[(y_i-ψ'(x_i^⊤β^*))^2(x_i^⊤u)^2] 
       =∑_i=1^nψ”(x_i^⊤β^*)(x_i^⊤u)^2
    ≲ nu^⊤Ξ_nu≲ nu^2,

and thus we have |I_1|= O_ p(α_n√(n)u)= O_ p(√(K)u).
Next, by using (C1) again, I_2 can be bounded as

    I_2
    ≲ -α_n^2∑_i=1^n(x_i^⊤u)^2
    ≲ -Ku^2
    <0.

Finally, for I_3, we have

    |I_3|
       =|α_n^3/6∑_i=1^nψ”'(θ_i)(x_i^⊤u)^3| 
    ≲α_n^3∑_i=1^n|x_i^⊤u|^3
    ≤ nα_n^3u^⊤Ξ_numax_1≤ i≤ n|x_i^⊤u| 
       ≲ nα_n^3√(K)u^3
    = O(K^2/√(n)u^3).

Combining all the above, if K^2/n→0 is satisfied, we observe that for sufficiently large C, I_1 and I_2 are dominated by I_2 (<0).
As a result, we obtain (<ref>).

From (<ref>) and (2), we have

    |x_i^⊤β̂|
    ≤ |x_i^⊤(β̂-β^*)|+|x_i^⊤β^*| 
    = O_ p(K/√(n))+ξ,

and thus, with probability tending to 1, β̂∈ B holds.




 §.§ Proof of Theorem <ref>


First, we prove that √(n)(β̂-β^*) is asymptotically equivalent to Σ_n^-1s_n.
By using Taylor's theorem, we have

    0
    =ℓ_n'(β̂)
    =ℓ_n'(β^*)+ℓ”_n(β^*)(β̂-β^*)
    +1/2∑_i=1^nψ”'(θ̃_i)x_i{x_i^⊤(β̂-β^*)}^2,
 
where for i=1,2,…,n, θ̃_i is in the line segment between x_i^⊤β^* and x_i^⊤β̂.
In addition, (<ref>) can be rewritten as

    √(n)(β̂-β^*)
    =Σ_n^-1s_n+R_n,

where

    R_n
    =-1/2√(n)Σ_n^-1∑_i=1^nψ”'(θ̃_i)x_i{x_i^⊤(β̂-β^*)}^2.

Noting that, from (C1),

    λ_ min(Σ_n)
    ≳λ_ min(Ξ_n)
    >C_1
    >0,

(C1), (C3) and (<ref>) imply

    R_n   ≲1/√(n)max_1≤ i≤ n|x_i^⊤(β̂-β^*)|
    ×∑_i=1^nΣ_n^-1ψ”'(θ̃_i)x_ix_i^⊤(β̂-β^*)
       ≲1/√(n)max_1≤ i≤ nx_iβ̂-β^*× nλ_ max(Ξ_n)β̂-β^*
       = O_ p(K√(K)/√(n))
    = o_ p(1).


Now we can prove the asymptotic normality of σ_n^-1Σ_n^-1s_n.
For any K dimensional vector η with η<∞, define σ_n^2=η^⊤Σ_n^-1η and ω_n such that

    η^⊤Σ_n^-1s_n
    =1/√(n)∑_i=1^nη^⊤Σ_n^-1x_i(y_i-ψ'(x_i^⊤β^*))
    =∑_i=1^nω_ni.

Then, since S⊃ S^*, we observe that

    ∑_i=1^n E[ω_ni]
     =∑_i=1^n1/√(n)η^⊤Σ_n^-1x_i E[y_i-ψ'_i]
     =0,

and

    ∑_i=1^n V[ω_ni]
    =1/n∑_i=1^nη^⊤Σ_n^-1x_i V[y_i]x_i^⊤Σ_n^-1η
    =σ_n^2.

To state the asymptotic normality of σ_n^-1Σ_n^-1s_n, we check the Lindeberg condition for ω_n: for any ε>0,

    1/σ_n^2∑_i=1^n E[ω_ni^2I(|ω_ni|>σ_nε)]
    = o(1).


For any ε>0, we have

    1/σ_n^2∑_i=1^n E[ω_ni^2I(|ω_ni|>σ_nε)] 
       =1/σ_n^2·1/n∑_i=1^n(η^⊤Σ_n^-1x_i)^2 E[(y_i-ψ'_i)^2I(|ω_ni|>σ_nε)] 
       ≤1/σ_n^2max_1≤ i≤ n E[(y_i-ψ'_i)^2I(|ω_ni|>σ_nε)]
    ×1/n∑_i=1^n(η^⊤Σ_n^-1x_i)^2.

By using the Cauchy-Schwarz inequality and (C1), 

    1/n∑_i=1^n(η^⊤Σ_n^-1x_i)^2
    ≤1/n∑_i=1^n(η^⊤Σ_n^-1η)(x_i^⊤Σ_n^-1x_i)
    ≲1/n∑_i=1^nx_i^2
    = O(K).


Noting that each y_i is distributed according to a Bernoulli distribution with parameter ψ', E[(y_i-ψ'_i)^4] is uniformly bounded on B for any i=1,…,n by a simple calculation.
Thus, by using the Cauchy-Schwarz inequality and Chebyshev's inequality, we have

    max_1≤ i≤ n E[(y_i-ψ'_i)^2I(|ω_ni|>σ_nε)]
       ≤max_1≤ i≤ n E[(y_i-ψ_i)^4]^1/2 P(|ω_ni|>σ_nε)^1/2
       ≲1/σ_nmax_1≤ i≤ n E[ω_ni^2]^1/2
       =1/σ_n√(n)max_1≤ i≤ n|η^⊤Σ_n^-1x_i|√(ψ'_i(1-ψ'_i))
       ≲1/√(n)max_1≤ i≤ nx_i
    = O(√(K)/√(n)).

Finally, since

    σ_n^2
    =η^⊤Σ_n^-1η≤λ_ max(Σ_n^-1)η^2
    =η^2/λ_ min(Σ_n)
    = O(1),

we have

    1/σ_n^2∑_i=1^n E[ω_ni^2I(|ω_ni|>σ_nε)]
    = O(√(K)/√(n)· K ).

From (C3), this implies the Lindeberg condition (<ref>).



 §.§ Proof of Theorem <ref>


First, we prove that, for any K dimensional vector η, the selection event can be expressed as an inequality with respect to η^⊤T_n.
Let us define w=(I_K-cη^⊤)T_n, where c=Σ_n^-1η/σ_n^2.
Then, since T_n=(η^⊤T_n)c+w, we have

    ÃT_n≤b̃   ⇔ (η^⊤T_n)Ãc≤b̃-Ãw
       ⇔ (η^⊤T_n)(Ãc)_j≤ (b̃-Ãw)_j, ∀ j 
       ⇔η^⊤T_n≤ (b̃-Ãw)_j/(Ãc)_j,    j:(Ãc)_j>0 
    η^⊤T_n≥ (b̃-Ãw)_j/(Ãc)_j,    j:(Ãc)_j<0 
    
    0=(b̃-Ãw)_j,    j:(Ãc)_j=0

and this implies the former result in Theorem <ref>.

To prove the theorem, we need to verify asymptotic independency between (L_n, U_n, N_n) and η^⊤T_n.
By the definition of w and Theorem <ref>, 

    ( [ η^⊤T_n;      w ])
    =
    ( [      η^⊤; I_K-cη^⊤ ])T_n

is asymptotically distributed according to a Gaussian distribution.
Thus, w and η^⊤T_n are asymptotically independent since

    Cov[w,η^⊤T_n]
    =(I_K-cη^⊤) E[T_nT_n^⊤]η
    =(I_K-cη^⊤)Σ_n^-1η
    =0.

Now we only need to prove asymptotic independency between b̃ and η^⊤T_n.
Letting ψ'=ψ'(β^*), the definition of T_n and Σ_n imply

    X_S^⊤{ (y-ψ')-1/√(n)Ψ X_ST_n}
    =0,

and thus

    y=ψ'+1/√(n)Ψ X_ST_n

Then, we observe that

    b̃   =-1/√(n)A_SX_S^⊤ψ'-1/√(n)A_S^X_S^^⊤y
       =-1/√(n)AX^⊤ψ'-1/√(n)A_S^X_S^^⊤(ψ'+1/√(n)Ψ X_ST_n) 
       =-1/√(n)AX^⊤ψ'-1/nA_S^X_S^^⊤Ψ X_ST_n.

Since b̃ can be expressed as a linear combination of T_n as well as w, the theorem holds when the covariance between b̃ and η^⊤T_n converges to 0 as n goes to infinity.
By noting that Σ_n=X_S^⊤Ψ X_S/n, we have

    Cov[b̃,η^⊤T_n]
       =-1/nA_S^X_S^^⊤Ψ X_S E[T_nT_n^⊤]η
       =-A_S^(X_S^^⊤Ψ X_S)(X_S^⊤Ψ X_S)^-1η.

In addition, letting a=(1,-1)^⊤, it is straightforward that

    A_S^
    =1_K⊗( [ 0 ⋯ 0; a O;   ⋱  ; O a ])
    =1_K⊗J̃

by the definition of the selection event, where J̃=(0_d-K, I_d-K⊗a^⊤)^⊤.
This implies A_S^^⊤ A_S^=2KI_d-K.
Finally, (C1), (C3), and (C4) imply

    Cov[b̃,η^⊤T_n]^2
    =2K(X_S^^⊤Ψ X_S)(X_S^⊤Ψ X_S)^-1η^2 
    ≲ K1/nX_S^^⊤ X_S^2 
    = O(K^3/n),

and this proves the asymptotic independency between b̃ and η^⊤T_n.



§ CONCLUDING REMARKS AND FUTURE RESEARCH


Recently, methods for data driven science such as selective inference and adaptive data analysis have become increasingly important as described by <cit.>.
Although there are several approaches for carrying out post-selection inference, we have developed a selective inference method for high dimensional classification problems, based on the work in <cit.>.
In the same way as that seminal work, the polyhedral lemma (Lemma <ref>) plays an important role in our study.
By considering high dimensional asymptotics concerning sample size and the number of selected variables, we have shown that a similar result to the polyhedral lemma holds even for high dimensional logistic regression problems.
As a result, we could construct a pivotal quantity whose sampling distribution is represented as a truncated normal distribution which converges to a standard uniform distribution.
In addition, through simulation experiments, it has been shown that the performance of our proposed method, in almost all cases, superior to other methods such as data splitting.

As suggested by the results from the simulation experiments, conditions might be relaxed to accommodate more general settings.
In terms of future research in this domain, while we considered the logistic model in this paper, it is important to extend the results to other models, for example, generalized linear models.
Further, higher order interaction models are also crucial in practice.
In this situation, the size of the matrix in the selection event becomes very large, and thus it is cumbersome to compute truncation points in the polyhedral lemma.
<cit.> have shown that selective inference can be constructed in such a model by utilizing a pruning algorithm.
In this respect, it is desirable to extend their result not only to linear regression modeling contexts but also to other models.


econ


