In the high-dimensional regression model a response variable is linearly related to @math covariates, but the sample size @math is smaller than @math . We assume that only a small subset of covariates is active' (i.e., the corresponding coefficients are non-zero), and consider the model-selection problem of identifying the active covariates. A popular approach is to estimate the regression coefficients through the Lasso ( @math -regularized least squares). This is known to correctly identify the active set only if the irrelevant covariates are roughly orthogonal to the relevant ones, as quantified through the so called irrepresentability' condition. In this paper we study the Gauss-Lasso' selector, a simple two-stage method that first solves the Lasso, and then performs ordinary least squares restricted to the Lasso active set. We formulate generalized irrepresentability condition' (GIC), an assumption that is substantially weaker than irrepresentability. We prove that, under GIC, the Gauss-Lasso correctly recovers the active set.
@cite_4: Given @math noisy samples with @math dimensions, where @math , we show that the multi-step thresholding procedure based on the Lasso -- we call it the Thresholded Lasso , can accurately estimate a sparse vector @math in a linear model @math , where @math is a design matrix normalized to have column @math norm @math , and @math . We show that under the restricted eigenvalue (RE) condition (Bickel-Ritov-Tsybakov 09), it is possible to achieve the @math loss within a logarithmic factor of the ideal mean square error one would achieve with an oracle while selecting a sufficiently sparse model -- hence achieving sparse oracle inequalities ; the oracle would supply perfect information about which coordinates are non-zero and which are above the noise level. In some sense, the Thresholded Lasso recovers the choices that would have been made by the @math penalized least squares estimators, in that it selects a sufficiently sparse model without sacrificing the accuracy in estimating @math and in predicting @math . We also show for the Gauss-Dantzig selector (Cand e s-Tao 07), if @math obeys a uniform uncertainty principle and if the true parameter is sufficiently sparse, one will achieve the sparse oracle inequalities as above, while allowing at most @math irrelevant variables in the model in the worst case, where @math is the smallest integer such that for @math , @math . Our simulation results on the Thresholded Lasso match our theoretical analysis excellently.
The same conditions can be used to prove model-selection guarantees. In particular, Zhou @cite_4 studies a multi-step thresholding procedure whose first steps coincide with the Gauss-Lasso. While the main objective of this work is to prove high-dimensional @math consistency with a sparse estimated model, the author also proves partial model selection guarantees. Namely, the method correctly recovers a subset of large coefficients @math , provided @math , for @math . This means that the coefficients that are guaranteed to be detected must be a factor @math larger than what is required by our results.