DropLasso: A robust variant of Lasso for single cell RNA-seq data
Beyrem Khalfaoui Jean-Philippe Vert
















DropLasso: A robust variant of Lasso for single cell RNA-seq data
    Beyrem Khalfaoui ^1,2 and Jean-Philippe Vert ^1,2,3,
 

^ 1  MINES ParisTech, PSL Research University, 
CBIO - Centre for Computational Biology, F-75006 Paris, France

^ 2 Institut Curie, PSL Research University, INSERM, U900, F-75005 Paris, France.
  
^ 3 Ecole Normale Supérieure, Department of Mathematics and Applications, 
CNRS, PSL Research University, F-75005 Paris, France.

<firstname.lastname@mines-paristech.fr>
    
==========================================================================================================================================================================================================================================================================================================================================================================================================================================




Single-cell RNA sequencing (scRNA-seq) is a fast growing approach to measure the genome-wide transcriptome of many individual cells in parallel, but results in noisy data with many dropout events. Existing methods to learn molecular signatures from bulk transcriptomic data may therefore not be adapted to scRNA-seq data, in order to automatically classify individual cells into predefined classes.

We propose a new method called DropLasso to learn a molecular signature from scRNA-seq data. DropLasso extends the dropout regularisation technique, popular in neural network training, to estimate sparse linear models. It is well adapted to data corrupted by dropout noise, such as scRNA-seq data, and we clarify how it relates to elastic net regularisation. We provide promising results on simulated and real scRNA-seq data, suggesting that DropLasso may be better adapted than standard regularisations to infer molecular signatures from scRNA-seq data.


DropLasso is freely available as an R package at <https://github.com/jpvert/droplasso>







§ INTRODUCTION

The fast paced development of massively parallel sequencing technologies and protocols has made it possible to measure gene expression with more precision and less cost in recent years. Single-cell RNA sequencing (scRNA-seq), in particular, is a fast growing approach to measure the genome-wide transcriptome of many individual cells in parallel <cit.>. By giving access to cell-to-cell variability, it represents a major advance compared to standard “bulk” RNA sequencing to investigate complex heterogeneous tissues <cit.> and study dynamic biological processes such as embryo development <cit.> and cancer <cit.>.

The analysis of scRNA-seq data is however challenging and raises a number of specific modelling and computational issues <cit.>. In particular, since a tiny amount of RNA is present in each cell, a large fraction of polyadenylated RNA can be stochastically lost during sample preparation steps including cell lysis, reverse transcription or amplification. As a result, many genes fail to be detected even though they are expressed, a type of errors usually referred to as dropouts. In a standard scRNA-seq experiment it is common to observe more than 80% of genes with no apparent expression in each single cell, an important proportion of which are in fact dropout errors <cit.>. The presence of so many zeros in the raw data can have significant impact on the downstream analysis and biological conclusions, and has given rise to new statistical models for data normalisation and visualisation <cit.> or gene differential analysis <cit.>.

Besides exploratory analysis and gene-per-gene differential analysis, a promising use of scRNA-seq technology is to automatically classify individual cells into pre-specified classes, such as particular cell types in a cancer tissue. This requires to establish cell type specific “molecular signatures” that could be shared and used consistently across laboratories, just like standard molecular signatures are commonly used to classify tumour samples into subtypes from bulk transcriptomic data <cit.>. From a methodological point of view, molecular signatures are based on a supervised analysis, where a model is trained to associate each genome-wide transcriptomic profile to a particular class, using a set of profiles with class annotation to select the genes in the signature and fit the parameters of the models. While the classes themselves may be the result of an unsupervised analysis, just like breast cancer subtypes which were initially defined from a first unsupervised clustering analysis of a set of tumours <cit.>, the development of a signature to classify any new sample into one of the classes is generally based on a method for supervised classification or regression.

Signatures based on a few selected genes, such as the 70-gene signature for breast cancer prognosis of <cit.>, are particularly useful both for interpretability of the signature, and to limit the risk of overfitting the training set. Many techniques exist to train molecular signatures on bulk transcriptomic data  <cit.>, however, they may not be adapted to scRNA-seq data due to the inflation of zeros resulting from dropout events.

Interestingly and independently, the term “dropout” has also gained popularity in the machine learning community in recent years, as a powerful technique to regularise deep neural networks <cit.>. Dropout regularisation works by randomly removing connexions or nodes during parameter optimisation of a neural network. On a simple linear model (a.k.a. single-layer neural network), this is equivalent to randomly creating some dropout noise to the training examples, i.e., to randomly set some features to zeros in the training examples <cit.>. Several explanations have been proposed for the empirical success of dropout regularisation. <cit.> motivated the technique as a way to perform an ensemble average of many neural networks, likely to reduce the generalisation error by reducing the variance of the estimator, similar to other ensemble averaging techniques like bagging <cit.> or random forests <cit.>. Another justification for the relevance of dropout regularisation, particularly in the linear model case, is that it performs an intrinsic data-dependent regularisation of the estimator <cit.> which is particularly interesting in the presence of rare but important features. Yet another justification for dropout regularisation, particularly relevant for us, is that it can be interpreted as a data augmentation technique, a general method that amounts to adding virtual training examples by applying some transformation to the actual training examples, such as rotations of images or corruption by some Gaussian noise; the hypothesis being that the class should not change after transformation. Data augmentation has a long history in machine learning <cit.>, and is a key ingredient of many modern successful applications of machine learning such as image classification <cit.>. As shown by <cit.>, dropout regularisation in the linear model case can be interpreted as a data augmentation technique, where corruption by dropout noise enforces the model to be robust to dropout events in the test data, e.g., to blanking of some pixels on images or to removal of some words in a document. <cit.> show that in some cases, data augmentation with dropout noise allows to train model that should be insensitive to such noise more efficiently than without.

Since scRNA-seq data are inherently corrupted by dropout noise, we therefore propose that dropout regularisation may be a sound approach to make the predictive model robust to this form of noise, and consequently to improve their generalisation performance on scRNA-seq supervised classification. Since plain dropout regularisation does not lead to feature selection and to the identification of a limited number of genes to form a molecular signature, we furthermore propose an extension of dropout regularisation, which we call DropLasso regularisation, obtained by adding a sparsity-inducing ℓ_1 regularisation to the objective function of the dropout regularisation, just like lasso regression adds an ℓ_1 penalty to a mean squared error criterion in order to estimate a sparse model <cit.>. We show that the ℓ_1 penalty can be integrated in the standard stochastic gradient algorithm used to implement dropout regularisation, resulting in a scalable stochastic proximal gradient descent formulation of DropLasso. We also clarify the regularisation property of DropLasso, and show that it is to elastic net regularisation what plain dropout regularisation is to the plain ridge regularisation. Finally, we provide promising results on simulated and real scRNA-seq data, suggesting that specific regularisations like DropLasso may be better adapted than standard regularisations to infer molecular signatures from scRNA-seq data.




§ METHODS




 §.§ Setting and notations

We consider the supervised machine learning setting, where we observe a series of n pairs of the form (x_i,y_i)_i=1,…,n. For each i∈[1,n],  x_i ∈^d represents the gene expression levels for d genes measured in the i-th cell by scRNA-seq, and y_i ∈ or {-1,1} is a label to represent a discrete category or a real number associated to the i-th cell, e.g., a phenotype of interest such as normal vs tumour cell, or an index of progression in the cell cycle. For i∈[1,n] and j∈[1,d], we denote by x_i,j∈ the expression level of gene j in cell i. From this training set of n annotated cells, the goal of supervised learning is to estimate a function to predict the label of any new, unseen cell from its transcriptomic profile. We restrict ourselves to linear models f_w : ^d →, for any w∈^d, of the form

    ∀ u∈^d ,   f_w(u) = ∑_i=1^d w_i u_i .

To estimate a model on the training set, a popular approach is to follow a penalised maximum likelihood or empirical risk minimisation principle and to solve an objective function of the form

    min_w ∈ℝ^d{1/n∑_i=1^n L(w, x_i,y_i)  + λΩ(w) } ,

where L(w, x_i,y_i) is a loss function to assess how well f_w predicts y_i from x_i, Ω is an (optional) penalty to control overfitting in high dimensions, and λ>0 is a regularisation parameter to control the balance between under- and overfitting. Examples of classical loss functions include the square loss:
 
    L_square(w,x_i,y_i) =   (y_i - ∑_j=1^d w_j  x_i,j)  ^2   ,

 and the logistic loss:
 
    L_logistic(w,x_i,y_i) = log(1+ exp(- y_i∑_j=1^d w_j  x_i,j) )  ,

 which are popular losses when y_i is respectively a continuous (y_i∈) or discrete (y_i∈{-1,1}) label. As for the regularisation term Ω(w) in (<ref>), popular choices include the ridge penalty <cit.>:
 
    Ω_ridge(w) =  w_2^2 = ∑_i=1^d w_i^2 ,

and the lasso penalty <cit.>:

    Ω_lasso(w) =  w_1 = ∑_i=1^d | w_i| .

 The properties, advantages and drawbacks of ridge and lasso penalties have been theoretically studied under different assumptions and regimes. The lasso penalty additionally allows  feature selection by producing sparse solutions, i.e., vectors w with many zeros; this is useful to in many bioinformatics applications to select “molecular signatures”, i.e., predictive models based on the expression of a limited number of genes only. It is known however that lasso can be unstable in particular when there are several highly correlated features in the data. It also cannot select more features than the number of observations and its accuracy is often dominated by that of ridge. For these reasons, another popular penalty is elastic net, which encompasses the advantages of both penalties <cit.> : 

    Ω_ridge(w) = α w_2^2 + (1-α) w_1  ,

where α∈[0,1] allows to interpolate between the lasso (α=0) and the ridge (α=1) penalties.



 §.§ DropLasso

For scRNA-seq data subject to dropout noise, we propose a new model to train a sparse linear model robust to the noise by artificially augmenting the training set with new examples corrupted by dropout. Formally, given a vector u ∈^d and a dropout mask δ∈{0,1}^d, we consider the corrupted pattern δ⊙ u ∈^d obtained by entry-wise multiplication (δ⊙ u)_i = δ_i u_i. In order to consider all possible dropout masks, we make δ a random variable with independent entries following a Bernoulli distribution of parameter p∈[0,1], i.e., P(δ_i = 1)=p, and consider the following DropLasso regularisation for any λ>0, p∈[0,1] and loss function L:

    min_w ∈ℝ^d( 1/n∑_i=1^nδ_i ∼ B(p)^d𝔼  L(w,δ_i  ⊙x_i,/p  , y_i)   +  λ w_1)   .

In this equation, the expectation over the dropout mask corresponds to an average of 2^d terms. The division by p in the term x_i/p is here to ensure that, on average, the inner product between w and δ_i  ⊙x_i,/p is independent of p, because:

    δ_i ∼ B(p)^d𝔼∑_j=1^d w_j (δ_i  ⊙x_i,/p)_j    = ∑_j=1^d δ_i,j∼ B(p)𝔼  w_j δ_i,jx_i,j/p
       = ∑_j=1^d w_j x_i,j .

When p=1 and λ>0, the only mask with positive probability is the constant mask with all entries equal to 1, which performs no dropout corruption. In that case, DropLasso (<ref>) therefore boils down to standard lasso. When λ=0 and p<1, on the other hand, DropLasso boils down to the standard dropout regularisation proposed by <cit.> and studied, among others, by <cit.>. In general, DropLasso interpolates between lasso and dropout. For λ>0, it inherits from lasso regularisation the ability to select features associated with ℓ_1 regularisation <cit.>. We therefore propose DropLasso as a good candidate to select molecular signatures (thanks to the sparsity-inducing ℓ_1 regularisation) for data corrupted with dropout noise, in particular scRNA-seq data (thanks to the dropout data augmentation).



 §.§ Algorithm

For any convex loss function L such as the square or logistic losses, DropLasso (<ref>) is a non-smooth convex optimisation problem whose global minimum can be found by generic solvers for convex programs. Due to the dropout corruption, the total number of terms in the sum in (<ref>) is n× 2^d. This is usually prohibitive as soon as d is more than a few, e.g., in practical applications when d is easily of order 10^4 (number of genes). Hence the objective function (<ref>) can simply not be computed exactly for a single candidate model w, and even less optimised by methods like gradient descent.

To solve (<ref>), we instead propose to follow a stochastic gradient approach to exploit the particular structure of the model, in particular the fact that it is fast and easy to generate a sample randomly corrupted by dropout noise. A similar approach is used for standard dropout regularisation when L is differentiable <cit.>, however in our case we additionally need to take care of the non-differentiable ℓ_1 norm; this can be handled by a forward-backward algorithm which, plugged in the stochastic gradient loop, leads to the proximal stochastic gradient algorithm presented in Algorithm <ref>. The fact that Algorithm <ref> is correct, i.e., converges to the solution of (<ref>), follows from general results on stochastic approximations <cit.>.




§ INTRODUCTION AND FRAMEWORK [OLD VERSION]



  
The fast paced development of sequencing methods and technology has made it possible to measure gene expression with more precision and with less cost. This has led to new biological discoveries such as the study of intra-tumoral heterogeneity or the transcriptional landscape of some tissue or cell type. The increase of single cell RNA-sequencing data is promising more refinement of these results and tackling a whole new set of previously inaccessible biological problems involving heterogeneous samples, rare cell types, cell lineage relationships and subpopulations of somatic tissues. However, the high dimensional nature of the data, the type and the amount of technical noise in single cell sequencing methods are issues for analysis, and therefore are particularly interesting for statistical learning research <cit.>.
A lot of the information provided by this can be retrieved in an unsupervised fashion (without providing a label for each observation), using a wide array of statistical analysis and clustering tools, or in a supervised fashion, when a label is available for each observation.The label can be any phenotype of interest or clinical information, categorical or real-valued, e.g: normal vs tumour cell, level of stress and so on     




  
The goal in supervised learning is to build a model relating the values of the data features to the label using labeled examples. 
To formulate the problem, we are given a data matrix X ∈ℝ^n × d where each row x_i,. is an observation of a cell, and each column x_,j  represents a feature that is here a gene or a genome region, (x_i,j is then the level of gene expression j for cell i). Each observation x_i,. corresponds to a label y_i 
Accuracy is then measured on an unseen test dataset and if the model has good generalisation properties, it could be used for predicting labels of new observations or select relevant features for the studied problem.  The selection of the right model is usually done through the maximum likelihood principle or more generally though minimising a loss function averaged (or summed) over the dataset. Here we focus on the family of linear models : 

Given a loss L the cost of an error of prediction for an observation i  will be: L(w,x_i,y_i). The goal is to minimise the empirical risk which is the average of the loss function at the training data points: 
 
    w ∈ℝ^dmin(R_emp(w,x,y)  )  =   w ∈ℝ^dmin(  1/n∑_i=1^n L(w, x_i,y_i)  )
 

Despite the generality of our method, we will particularly focus here on the: 


  * Square loss: L(w,x_i,y_i) =   (y_i - ∑_j=1^d w_j  x_i,j)  ^2, used widely for regression.

  * Logistic loss: L(w,x_i,y_i) = log  (1+ exp(- y_i∑_j=1^d w_j  x_i,j) ) for classification.







  
Building models that are less sensitive to noise in the data is indeed a central challenge in machine learning. A model that is very sensitive to the noise in the training data will likely have poor accuracy on unseen data. It might also select variables that are not meaningful for the problem, which can be misleading for later investigation. There are many techniques tackling this issue of overfitting.Methods such as Ridge <cit.> and Lasso <cit.> are classically used in this case. They both can be expressed in different forms but they are equivalent to adding a penalty term to the empirical risk in the minimisation problem (1). The penalty in this case is only a function of the model and participates in reducing the variance of the model and its sensitivity to sporadic noise in the data. 


  * Ridge penalises the l_2-norm   
    w ∈ℝ^dmin( 1/n∑_i=1^nL(w,x_i,y_i)   +   λ w_2^2)
 

  * Lasso l_1-norm : 
    w ∈ℝ^dmin( 1/n∑_i=1^nL(w,x_i,y_i)   +  λ w_1)


λ is a hyper parameter that is usually estimated by cross-validation. 




  
The properties, advantages and drawbacks of Ridge and Lasso have been theoretically studied under different assumptions and regimes. The Lasso penalty additionally allows for feature selection by producing sparse weights, which became an apparent need in many bioinformatics applications, especially for high dimension data such as RNA-seq data. It is known however that this method can be unstable in particular when there are several highly correlated features in the data. It also cannot select more features than the number of observations and its accuracy can be dominated by that of Ridge <cit.>. For these reasons, a preferred penalty is elastic net which encompasses the advantages of both penalties <cit.> : 

    w ∈ℝ^dmin( 1/n∑_i=1^n L(w,x_i,y_i)   +  λ_1   w_1  +    λ_2   w_2^2)

We can see for instance that for Lasso is a particular case of elastic net when λ_2 =0, and ridge is a particular case when λ_1=0.


Besides elastic net, there have been other methods that try to alleviate Lasso limitations. One can mention the adaptive Lasso <cit.>, the relaxed Lasso <cit.>, VISA   <cit.>, Stability Selection <cit.> and Random Lasso <cit.> among the most popular approaches. Stability selection and Random Lasso are particularly interesting approaches in that they use bootstraps of the data to improve stability and accuracy over the Lasso estimate. Bootstrapping relies on random sampling observations with replacement and thus it is a heuristic that intentionally introduces noise in the data. However, the mentioned methods using the bootstraps are computationally intensive as they train the Lasso many times on several bootstrapped versions of the data. It is even used twice before the averaging step for Random Lasso, and for Stability Selection an arbitrary threshold is used in the averaging step. We have therefore  compare my method mainly with Lasso and elastic net for accuracy and stability. 




  
Another recent technique tackling the overfitting problem in particular for deep neural networks is dropout. Dropout can be incorporated in any gradient descent method consists for a linear model of replacing a part of the input features by zero at each training iteration.Introducing perturbations such as additive gaussian noise in the model or the data during the gradient training of neural networks has been tried and partially studied previously (). Dropout, however, partilcularly improved the performance of deep and shallow neural networks a on several benchmark datasets for image and text classification such as Imagenet, CIFAR, MNIST for images and Reuters, Amazon and Imdb reviews for documents, (<cit.>,<cit.>,<cit.> and references therein) so significantly that it generated a particular interest in the method and its effects. One intuition of dropout as an ensemble learning procedure (or at least an approximation to ensemble averaging) was mentioned in the original paper by was reinforced theoretically by who show that one can approximate the effect of dropout as a normalised geometric averaging in the case of logistic activation functions, and thus reduces the generalisation error in the least squares case for example by reducing the estimate variance, just like averaging estimates after bootstrapping. Another intuition is that dropout enforces the classifier invariance under blanking noise, as for some data even after the removal of a part of features of an observation (e.g by blanking some pixels in an image or removing some words from a document), this new noisy observation still belongs the same class, and thus dropout can also be interpreted as a data augmentation technique with regularisation properties. <cit.> for instance show empirically that dropout works empirically well for two particular settings: Heavy-tailed feature distributions and in the case where some of the features are blanked at test time. 

Using a generative Poisson model and linear classifiers, Wager et al. also provide some theoretical understanding of why dropout regularisation works well for high-dimensional single-layer natural language tasks such as document classification, by studying the rate of convergence of the empirical risk<cit.>.  
  


  
One of the main challenges for normalisation and analysis of single-cell RNA-seq data is the low amount of RNA present in a single cell. A large fraction of polyadenylated RNA is in fact stochastically lost during sample preparation steps including cell lysis, reverse transcription or amplification <cit.> . This results in a very sparse dataset where technical and biological noise are mixed and where read count distributions are heavy-tailed. 
Thus, classification models on Single Cell RNA-seq might also benefit from the regularisation effect of dropout as for images or documents. We follow this intuition to make a simple modification of the LASSO method that requires seeing the dataset only once, that keeps the feature selection property and improves the accuracy and sometimes feature selection stability on simulated data and real single Cell RNA-seq data.








§ METHOD



 §.§ Implementation and formulation


In order to solve problem (1) one can use gradient descent as an iterative optimisation algorithm if the function is differentiable and the gradient can be easily computed at all the training points.One can also use the stochastic version computing only the gradient at one training point or a batch of fixed or varying size, popular variant lately due to the big size of the data the model. Even though for problem (3), the function to be minimised is not differentiable at 0, the problem can be also iteratively minimised using the proximal operator as an extension of the gradient. In the case of l_1-norm penalty or Lasso, the proximal is the soft-thresh holding operater, and the proximal descent algorithm is called ISTA. This algorithm can be.  We use this use this iterative soft-thresholding scheme developed for Lasso, adding a simple step where we perturb the training point (or points if we use a batch) before computing the proximal by setting a part of the feature values to 0, and rescale the point such that on expectation we recover the same original point. (see algorithm ). 


By the stochastic approximation theorem <cit.>, the DropLasso minimises the marginalised risk over the added noise and thus solves the following problem: 

    w ∈ℝ^dminδ_i ∼ B(p)^d𝔼( 1/n∑_i=1^n L(w,δ_𝐢⊙x_i,/p  , y_i)   +  λ w_1)

 where  ⊙ is the component-wise product vector and B(p) is the Bernoulli distribution with probability p.










 §.§ DropLasso and elastic net

As we already mentioned, DropLasso interpolates between lasso (p=1,λ>0) and dropout (p∈[0,1], λ=0). On the other hand, dropout regularisation is known to be related to ridge regularisation <cit.>; in particular, for the square loss, dropout regularisation boils down to ridge regression after proper normalisation of the data, while for more general losses it can be approximated by reweighted version of ridge regression. Here we show that DropLasso largely inherits these properties, and in a sense is to elastic net what dropout is to ridge.
It is interesting to see that DropLasso bridges the Lasso and Dropout penalties as elastic net bridges Ridge and Lasso penalties. In fact, for λ=0  DropLasso is equivalent to the dropout penalty for the case of a linear model and for p=1 it is equivalent to the Lasso penalty. We will see that the similarity with elastic net is stronger especially in the case of least squares. 
It is interesting to notice which properties of Lasso will be kept for DropLasso and which properties will be changed. A very special property of the Lasso estimate is its ability to produce sparse coefficients and thus to perform an embedded feature selection. 

As for Lasso, the DropLasso induces sparsity in the estimate (which depends on the tuning parameters λ and p). 



  
Proof: We know that DropLasso solves problem (4). Setting:
 
    L(w,x,y)= δ∼ B(p)^d𝔼  L(w,δ⊙x/p ,y)
 
Convexity of L induces that of L , and the sparsity property of DropLasso is induced by that of Lasso for convex loss (See <cit.> for a general analysis of Lasso estimate sparsity)  □ 


Let us start with the square loss. In that case we have the following:
For the square loss, DropLasso is equivalent to an elastic net regression if the data are normalised so that all features have the same norm. If data are not normalised, DropLasso is equivalent to an elastic net regression with a weighted ridge penalty.




  
Proof: By developing the error function and marginalising over the Bernoulli variables we get : 


    1/n   ∑_i=1^nδ_i∼ B(p)^d𝔼  L(w,δ_i⊙x_i,/p  , y_i)   +   λ w_1
    
    =   1/n∑_i=1^nδ_i∼ B(p)^d𝔼(y_i - ∑_j=1^d w_jδ_i,jx_i,j/p) ^2   +   λ w_1
    
     =   1/n∑_i=1^n(y_i -∑_j=1^d w_j x_i,j) ^2  + ∑_i=1^n ∑_j=1^d w_j^2  x_i,j^2   Var (δ_i,j/p )  +  λ w_1
    
     =   1/n∑_i=1^n   L(w,x_i,y_i) +  1-p/p∑_j=1^d  x_.,j_2^2 w_j^2  +  λ w_1.
 
 

In the case of the logistic loss, we can also adapt a result of <cit.> which relates dropout to an adaptive version of ridge regression:
:
For the logistic loss, DropLasso can be approximated when the dropout probability p is close to 1 by an adaptive version of elastic net that automatically scales the data but also that encourages more confident predictions. 




  
Proof: Writing the Taylor expansion for the logistic loss up to the second order when the dropout is small (p close to 1), we get:
 
    L(w,δ_𝐢⊙x_i,/p  , y_i)      ≃  L(w,x_i,  , y_i)  
       + ∑_j=1^d∂ L(w,x_i,,y)/∂ x_i,j ( δ_i,j/p - 1 )  x_i,j
       +  1/2∑_j=1^d∂^2 L(w,x_i,,y)/∂^2 x_i,j ( δ_i,j/p - 1  )^2   x_i,j ^2 .

Taking the expectation with respect to δ_i, the first order term cancels out since 

    δ_i ∼ B(p)^d𝔼 [δ_i⊙x_i,/p ] = x_i .

We then get:
 
    1/n∑_i=1^nδ_i ∼ B(p)^d𝔼  L(w,δ_i⊙x_i,/p  , y_i)   +  λ w_1
       ≃∑_i=1^n  L(w,x_i, ,y_i)   + 1-p/p∑_j=1^dα_j w_j ^2 +  λ w_1 ,

where, for the logistic loss,
 
    α_j= 
       ∑_i=1^n  P(Y=1 | X=x_i , w) P(Y=0 | X=x_i , w)   x^2_i,j .

  In words, this shows that the dropout penalty can be approximated by a weighted data-dependent version of ridge regression, where the ridge penalty is controlled both by the size of the features x_i,j^2, but also by the fact that the prediction for each sample is confident or not. 
   

  


§ RESULTS





 §.§ Simulation results


We first investigate the performance of DropLasso on simulated data, and compare it to standard dropout and elastic net regularisation. We design a toy simulation to illustrate in particular how corruption by dropout noise impacts the performances of the different methods. The simulation goes as follow : 


  * We set the dimension to d=100.

  * Each sample is a random vector z∈𝐍^d with entries following a Poisson distribution with parameter 1. We introduce correlations between entries by first sampling a Gaussian copula with covariance Σ_d = 𝐈_d + 1_d^⊤1_d, then transforming each entry in [0,1] into an integer using the Poisson quantile function.

  * The “true” model is a logistic model with sparse weight vector w∈^d satisfying w_i=0.05 for i=1,…,10 and w_i=0 for i=1,…,d.

  * Using w as the true underlying model and z as the true observations, we simulate a label y ∼Bernoulli( 1/(1+exp(-∑_j=1^d w_j z_j )) )  

  * We introduce corruption by dropout events by multiplying entry-wise z with an i.i.d Bernoulli variables δ with probability q.

We simulate n=100 (z,x,y) samples to train different models, evaluate their performance on m=400 independent samples, and repeat the whole process 10 times. Each method estimates a model using the (x,y) pairs in the training set only, i.e., does only see the corrupted samples. Elastic net and DropLasso both have two parameters. In order to make a fair comparison, we fixed the α parameter of elastic net to α=0.5, and the dropout probability in DropLasso to p=0.5. In both cases, we vary the remaining λ parameter over a large grid of 100 values, estimate the classification performance on the test set in terms of area under the receiving operator curve (AUC), and report the best average AUC over the grid. 



Table <ref> shows the classification performance in terms of best average AUC of elastic net, dropout and DropLasso, when we vary the amount of dropout corruption in the data.
We first observe that for all methods, the performance drastically decreases when dropout noise increases, confirming the difficulty induced by dropout events to learn predictive models. 
Second, we note that, whatever the amount of noise, DropLasso outperforms dropout. This illustrates the benefit of incorporating the ℓ_1 lasso penalty in the objective function of dropout when the true model is sparse. Third, and more importantly, we observe that DropLasso outperforms elastic net in all settings, and that the difference in performance increases when the amount of dropout noise increases. This confirms the intuition that DropLasso is more efficient that elastic net in situations where data are corrupted by dropout noise.

Besides classification accuracy, it is also of interest to investigate to what extent the different methods select the correct variables, which are known in our simulations. For each λ value in the grid, we compute the number of true and false positives among the features selected by elastic net and DropLasso, and plot these values averaged over the 10 repeats in Figure <ref>.

Interestingly, we observe that elastic net seems to outperform DropLasso when there is no noise in the observed data (q=1), but seems to lose its ability to recover the correct features as the amount of dropout noise increases quicker than DropLasso, and in the high noise regime (q=0.4) DropLasso eventually outperforms elastic net. Although the differences are limited in this simulation setting, this illustrates again that DropLasso is more robust than elastic net in the presence of dropout noise.



 §.§ Classification on Single Cell RNA-seq



We now turn on to real scRNA-seq data. To evaluate the performance of methods for supervised classification, we collected 7 publicly available scRNA-seq datasets amenable to this setting, as summarised in Table <ref>. The first 6 datasets were processed <cit.>, and we obtained them from the conquer website[http://imlspenticton.uzh.ch:3838/conquer/http://imlspenticton.uzh.ch:3838/conquer/], a collection of consistently processed, analysis-ready and well documented publicly available scRNA-seq data sets. We used the already preprocessed length-scaled transcripts per million mapped reads <cit.>. These datasets were used by <cit.> to assess the performance of methods for gene differential analysis between classes of cells, and we follow the same splits of cells into classes for our experiments of supervised classification. The last dataset was collected from <cit.> and was originally used for the analysis of transcriptional heterogeneity in colorectal tumours. Gene expression levels were quantified as fragments per kilo-base per million reads (FPKM) which also allowed to compare the classifiers after different normalisations. We used the available sample annotations to create a binary classification problem where we want to discriminate tumour from normal epithelial cells, as described in Table <ref>. For all datasets, we filtered out genes that were not expressed in any sample. In order to also study the performance of the different methods for a varying number of variables, we also created for each dataset 3 reduced datasets by first selecting respectively the top 100, 1,000 and 10,000 genes with highest variance among the samples, regardless of their labels. 

On each of the 7× 4=28 resulting dataset, we compare the performance of 4 regularisation methods for logistic regression: lasso, dropout, elastic net and DropLasso. As in the simulation study, we fix the probability of dropout to p=0.5 for dropout and DropLasso regularisation. For elastic net, we fix α=0.5 to balance the ℓ_1 and ℓ_2 norms. Finally, for lasso, elastic net and DropLasso, we run the models with 100 values for λ, regularly spaced after log transform between λ_min=10^-5 and λ_max=10^5, on 20% of the data chosen in such way that labels are balanced, and evaluate the performance of the resulting models on the 80% remaining data. We report in Table <ref> the maximal test AUC, averaged over the 5 repeats, taken over the grid of λ.
 


On each of the 7× 4=28 resulting dataset, we compare the performance of 4 regularisation methods for logistic regression: lasso, dropout, elastic net and DropLasso. As in the simulation study, we fix the probability of dropout to p=0.5 for dropout and DropLasso regularisation. For elastic net, we fix α=0.5 to balance the ℓ_1 and ℓ_2 norms. Finally, for lasso, elastic net and DropLasso, we run the models with 100 values for λ, regularly spaced after log transform between λ_min=10^-5 and λ_max=10^5, on 20% of the data chosen in such way that labels are balanced, and evaluate the performance of the resulting models on the 80% remaining data. We report in Table <ref> the maximal test AUC, averaged over the 5 repeats, taken over the grid of λ. 


The first observation is that the performances reached by all methods on all datasets are generally very high, and can reach an AUC above 0.9 on each of the 7 datasets. This suggests that the labels chosen in these datasets are sufficiently different in terms of transcriptomic profiles that they can be easily recognised most of the time. We still notice some differences in performance between datasets, with GSE60749-GPL13112 being the easiest one while EMTAB2805 is the most challenging, for all methods. <cit.> also noticed a difference in signal-to-noise ratio between these datasets, in the context of gene differential analysis. Second, we observe that the best performance is obtained by DropLasso on 27 out of the 28 datasets. The difference between the 4 regularisers is visualised in the box plots on Figure <ref>, which summarise the AUC values over the 7 experiments with all genes for each method. In 14 of the 28 experiments, DropLasso significantly outperforms elastic net at significance level P<0.05, and in 3 of these cases the significance level is P<0.01.

This confirms that on real scRNA-seq data, DropLasso also brings a consistent benefit over dropout of elastic net regularisation. Finally, regarding the impact of the number of features selected, we observe that in most datasets and for most methods the best performance is reached for 100 or 1,000 genes. Remembering that genes are only selected based on their variance, independently of any class label information, this suggests that all method suffer in the high-dimensional regime and that a simple pre-filtering of genes can help by reducing the dimension of the problem. This also indirectly confirms the importance of regularisation in high dimension, and the relevance of developing adequate regularisers incorporating prior knowledge about the data and their noise.


To conclude this section, we now compare the lists of genes in the molecular signatures estimated by elastic net and DropLasso regularisation. We illustrate this comparison on the first dataset, EMTAB2805, where the goal is to discriminate mouse cells at the G1 from the G2M cell cycle stages. To this end, we retrain the different methods with the tuning parameter λ corresponding to the best accuracy on the 1000-filtered datasets with all the samples, and then we perform a bioinformatic analysis of Gene Ontology annotations using DAVID <cit.> on the subset of genes with non-zero coefficients for each method.

For this dataset and the best tuning parameters, DropLasso selects 186 variables while elastic net only selects 48 variables. The analysis of the selected genes shows enrichment in the functional terms “cell division”, “cell cycle” and “mitosis” for both methods. For DropLasso, 21 genes are related to the functional term “cell division”. Among those, one can find CDC28, CDC23 and CDca8 which were not selected by elastic net, that only selected 10 genes related to cell division. Similarly 30 genes that have previously been annotated with cellular processes consistent with the “cell cycle” term are selected in DropLasso versus 9 in elastic net, and 22 genes related to “mitosis” versus 10 in elastic net. Therefore, DropLasso could potentially allow for the discovery of more functionally related genes in its signature than elastic net.







§ DISCUSSION


ScRNA-seq is changing the way we study cellular heterogeneity and investigate a number of biological process such as differentiation or tumourigenesis. Yet, as the throughput of scRNA-seq technologies increases and allows to process more and more cells simultaneously, it is likely that the amount of information captured in each individual cell will remain limited in the future and that dropout noise will continue to affect scRNA-seq (and other single-cell technologies).

Several techniques have been proposed to handle dropout noise in the context of data normalisation or gene differential expression analysis, and shown to outperform standard techniques widely used for bulk RNA-seq data analysis. In this paper we investigate a new setting which, we believe, will play an important role in the future: supervised classification of cell populations into pre-specified classes, and selection of molecular signatures for that purpose. Molecular signatures for the classification of tissues from bulk RNA-seq data has already had a tremendous impact in cancer research, and as more and more cell types are investigated and discovered with scRNA-seq it is likely that specific molecular signatures will be useful in the future to automatically sort cells into their classes.

DropLasso, the new technique we propose, borrows the recent idea of dropout regularisation from machine learning, and extends it to allow feature selection. While a parallel between dropout regularisation and (data-dependent) ridge regression has already been shown by <cit.> and <cit.>, it is reassuring that we are able to extend this parallel to DropLasso and elastic net regularisation.

More interesting is the fact that, on both simulated and real data, we obtained promising results with DropLasso. They suggest that, again, specific models tailored to the data and noise give an edge over generic models developed under different assumptions.

The intuition behind why dropout (and DropLasso) perform well on scRNA-seq data, however, remains a bit unclear. Our main motivation to use them in this context was to see them as data augmentation techniques, where training data are corrupted according to the noise we assume in the data. While we believe this is fundamentally the reason why we obtained promising results, alternative explanations for the success of dropout have been proposed, and may also play a role in the context of scRNA-seq. They include for example the interpretation of dropout as a regulariser similar to a data-dependent weighted version of ridge regularisation, which works well in the presence of rare but important features <cit.>; it would be interesting to clarify if the regularisation induced by DropLasso on scRNA-seq data exploits some fundamental property of these data, and may be replaced by a more direct approach to model this.

Finally, this first study of dropout and DropLasso regularisation on biological data paves the way to many future direction. For example, it is known that the probability of dropout in scRNA-seq data depends on the gene expression level <cit.>. It would therefore be interesting to study both theoretically and empirically if a dropout regularisation following a similar pattern may be useful. Second,  instead of independently perturbing the different features one may create a correlation between the dropout events in different genes. Creating a correlation may be a way to create new regularisation by generating a structured dropout noise. It may for example be possible to derive a correlation structure for dropout noise from prior knowledge about gene annotations or gene networks in order to enforce a structure in the molecular signature, just like structured ridge and lasso penalties have been used to promote structure molecular signatures with bulk transcriptomes <cit.>.















§ FUNDING


This work has been supported by the European Research Council (grant ERC-SMAC-280032).













natbib



[Bach et al.(2011)Bach, Jenatton, Mairal, and
  Obozinski]Bach2011Optimization
Bach, F., Jenatton, R., Mairal, J., and Obozinski, G. (2011).
Optimization with sparsity-inducing penalties.
Foundations and Trends® in Machine Learning,
  4(1), 1–106.

[Bacher and Kendziorski(2016)Bacher and Kendziorski]Bacher2016Design
Bacher, R. and Kendziorski, C. (2016).
Design and computational analysis of single-cell RNA-sequencing
  experiments.
Genome Biology, 17(63).

[Baldi and Sadowski(2013)Baldi and Sadowski]Baldi2013Understanding
Baldi, P. and Sadowski, P. J. (2013).
Understanding dropout.
In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q.
  Weinberger, editors, Adv. Neural. Inform. Process Syst., pages
  2814–2822. Curran Associates, Inc.

[Breiman(1996)Breiman]Breiman1996Bagging
Breiman, L. (1996).
Bagging predictors.
Mach. Learn., 24(2), 123–140.

[Breiman(2001)Breiman]Breiman2001Random
Breiman, L. (2001).
Random forests.
Mach. Learn., 45(1), 5–32.

[Deng et al.(2014)Deng, Ramsköld, Reinius, and
  Sandberg]Deng2014Single
Deng, Q., Ramsköld, D., Reinius, B., and Sandberg, R. (2014).
Single-cell RNA-seq reveals dynamic, random monoallelic gene
  expression in mammalian cells.
Science, 343(6167), 193–6.

[Haury and Vert(2010)Haury and Vert]Haury2010stability
Haury, A.-C. and Vert, J.-P. (2010).
On the stability and interpretability of prognosis signatures in
  breast cancer.
In Proceedings of the Fourth International Workshop on Machine
  Learning in Systems Biology (MLSB10).
To appear.

[Hoerl and Kennard(1970)Hoerl and Kennard]Hoerl1970Ridge
Hoerl, A. E. and Kennard, R. W. (1970).
Ridge regression : biased estimation for nonorthogonal problems.
Technometrics, 12(1), 55–67.

[Huang et al.(2009)Huang, Sherman, and
  Lempicki]Huang2009Bioinformatics
Huang, D. W., Sherman, B. T., and Lempicki, R. A. (2009).
Bioinformatics enrichment tools: paths toward the comprehensive
  functional analysis of large gene lists.
Nucl. Acids Res., 37, 1–13.

[Jacob et al.(2009)Jacob, Obozinski, and Vert]Jacob2009Group
Jacob, L., Obozinski, G., and Vert, J.-P. (2009).
Group lasso with overlap and graph lasso.
In ICML '09: Proceedings of the 26th Annual International
  Conference on Machine Learning, pages 433–440, New York, NY, USA. ACM.

[Kharchenko et al.(2014)Kharchenko, Silberstein, and
  Scadden]Kharchenko2014Bayesian
Kharchenko, P. V., Silberstein, L., and Scadden, D. T. (2014).
Bayesian approach to single-cell differential expression analysis.
Nat. Methods, 11(7), 740–742.

[Kolodziejczyk et al.(2015)Kolodziejczyk, Kim, Svensson, Marioni,
  and Teichmann]Kolodziejczyk2015technology
Kolodziejczyk, A. A., Kim, J. K., Svensson, V., Marioni, J. C., and Teichmann,
  S. A. (2015).
The technology and biology of single-cell RNA sequencing.
Molecular Cell, 58(4), 610–620.

[Krizhevsky et al.(2012)Krizhevsky, Sutskever, and
  Hinton]Krizhevsky2012ImageNet
Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012).
ImageNet classification with deep convolutional neural networks.
In F. Pereira, C. Burges, L. Bottou, and K. Weinberger, editors, Adv. Neural. Inform. Process Syst., volume 25, pages 1097–1105. Curran
  Associates, Inc.

[Li et al.(2017)Li, Courtois, Sengupta, Tan, Chen, Goh, Kong,
  Chua, Hon, and Tan]Li2017Reference
Li, H., Courtois, E. T., Sengupta, D., Tan, Y., Chen, K. H., Goh, J. J. L.,
  Kong, S. L., Chua, C., Hon, L. K., and Tan, W. S. (2017).
Reference component analysis of single-cell transcriptomes elucidates
  cellular heterogeneity in human colorectal tumors.
Nat. Genet., 49(5), 708.

[Macosko et al.(2015)Macosko, Basu, Satija, Nemesh, Shekhar,
  Goldman, Tirosh, Bialas, Kamitaki, Martersteck, Trombetta, Weitz, Sanes,
  Shalek, Regev, and McCarroll]Macosko2015Highly
Macosko, E. Z., Basu, A., Satija, R., Nemesh, J., Shekhar, K., Goldman, M.,
  Tirosh, I., Bialas, A. R., Kamitaki, N., Martersteck, E. M., Trombetta,
  J. J., Weitz, D. A., Sanes, J. R., Shalek, A. K., Regev, A., and McCarroll,
  S. A. (2015).
Highly Parallel Genome-wide Expression Profiling of Individual Cells
  Using Nanoliter Droplets.
Cell, 161(5), 1202–1214.

[Ozsolak and Milos(2011)Ozsolak and Milos]Ozsolak2011RNA
Ozsolak, F. and Milos, P. M. (2011).
RNA sequencing: advances, challenges and opportunities.
Nat. Rev. Genet., 12, 87–98.

[Patel et al.(2014)Patel, Tirosh, Trombetta, Shalek, Gillespie,
  Wakimoto, Cahill, Nahed, Curry, Martuza, Louis, Rozenblatt-Rosen, Suvà,
  Regev, and Bernstein]Patel2014Single
Patel, A. P., Tirosh, I., Trombetta, J. J., Shalek, A. K., Gillespie, S. M.,
  Wakimoto, H., Cahill, D. P., Nahed, B. V., Curry, W. T., Martuza, R. L.,
  Louis, D. N., Rozenblatt-Rosen, O., Suvà, M. L., Regev, A., and Bernstein,
  B. E. (2014).
Single-cell RNA-seq highlights intratumoral heterogeneity in
  primary glioblastoma.
Science, 344(6190), 1396–1401.

[Perou et al.(2000)Perou, Sørlie, Eisen, van de Rijn, Jeffrey,
  Rees, Pollack, Ross, Johnsen, Akslen, Fluge, Pergamenschikov, Williams, Zhu,
  Lønning, Børresen-Dale, Brown, and Botstein]Perou2000Molecular
Perou, C. M., Sørlie, T., Eisen, M. B., van de Rijn, M., Jeffrey, S. S.,
  Rees, C. A., Pollack, J. R., Ross, D. T., Johnsen, H., Akslen, L. A., Fluge,
  O., Pergamenschikov, A., Williams, C., Zhu, S. X., Lønning, P. E.,
  Børresen-Dale, A. L., Brown, P. O., and Botstein, D. (2000).
Molecular portraits of human breast tumours.
Nature, 406(6797), 747–752.

[Pierson and Yau(2015)Pierson and Yau]Pierson2015Dimensionality
Pierson, E. and Yau, C. (2015).
Dimensionality reduction for zero-inflated single cell gene
  expression analysis.
Genome Biol., 16(241).

[Ramaswamy et al.(2001)Ramaswamy, Tamayo, Rifkin, Mukherjee,
  Yeang, Angelo, Ladd, Reich, Latulippe, Mesirov, Poggio, Gerald, Loda, Lander,
  and Golub]Ramaswamy2001Multiclass
Ramaswamy, S., Tamayo, P., Rifkin, R., Mukherjee, S., Yeang, C., Angelo, M.,
  Ladd, C., Reich, M., Latulippe, E., Mesirov, J., Poggio, T., Gerald, W.,
  Loda, M., Lander, E., and Golub, T. (2001).
Multiclass cancer diagnosis using tumor gene expression signatures.
Proc. Natl. Acad. Sci. USA, 98(26),
  15149–15154.

[Rapaport et al.(2007)Rapaport, Zynoviev, Dutreix, Barillot, and
  Vert]Rapaport2007Classification
Rapaport, F., Zynoviev, A., Dutreix, M., Barillot, E., and Vert, J.-P. (2007).
Classification of microarray data using gene networks.
BMC Bioinformatics, 8, 35.

[Risso et al.(2018)Risso, Perraudeau, Gribkova, Dudoit, and
  Vert]Risso2018ZINB
Risso, D., Perraudeau, F., Gribkova, S., Dudoit, S., and Vert, J.-P. (2018).
ZINB-WaVE: A general and flexible method for signal extraction from
  single-cell RNA-seq data.
Nature Comm., 9(1), 284.

[Robbins and Siegmund(1971)Robbins and
  Siegmund]Robbins1971convergence
Robbins, H. and Siegmund, D. (1971).
A convergence theorem for non negative almost supermartingales and
  some applications.
In Optimizing methods in statistics, pages 233–257.
  Elsevier.

[Schölkopf et al.(1996)Schölkopf, Burges, and
  Vapnik]Scholkopf1996Incorporating
Schölkopf, B., Burges, C., and Vapnik, V. (1996).
Incorporating invariances in support vector learning machines.
In C. von der Malsburg, W. von Seelen, J. C. Vorbrüggen, and
  B. Sendhoff, editors, ICANN 96: Proceedings of the 1996 International
  Conference on Artificial Neural Networks, pages 47–52, London, UK.
  Springer-Verlag.

[Soneson and Robinson(2017)Soneson and Robinson]Soneson2017Bias
Soneson, C. and Robinson, M. D. (2017).
Bias, robustness and scalability in differential expression analysis
  of single-cell RNA-seq data.
Technical Report 143289, bioRxiv.

[Sørlie et al.(2001)Sørlie, Perou, Tibshirani, Aas,
  Geisler, Johnsen, Hastie, Eisen, van de Rijn, Jeffrey, Thorsen, Quist,
  Matese, Brown, Botstein, Eystein Lønning, and
  Børresen-Dale]Soerlie2001Gene
Sørlie, T., Perou, C. M., Tibshirani, R., Aas, T., Geisler, S., Johnsen, H.,
  Hastie, T., Eisen, M. B., van de Rijn, M., Jeffrey, S. S., Thorsen, T.,
  Quist, H., Matese, J. C., Brown, P. O., Botstein, D., Eystein Lønning, P.,
  and Børresen-Dale, A. L. (2001).
Gene expression patterns of breast carcinomas distinguish tumor
  subclasses with clinical implications.
Proc. Natl. Acad. Sci. USA, 98(19), 10869–10874.

[Sørlie et al.(2003)Sørlie, Tibshirani, Parker, Hastie,
  Marron, Nobel, Deng, Johnsen, Pesich, Geisler, Demeter, Perou, Lønning,
  Brown, Børresen-Dale, and Botstein]Sorlie2003Repeated
Sørlie, T., Tibshirani, R., Parker, J., Hastie, T., Marron, J., Nobel, A.,
  Deng, S., Johnsen, H., Pesich, R., Geisler, S., Demeter, J., Perou, C.,
  Lønning, P., Brown, P., Børresen-Dale, A., and Botstein, D. (2003).
Repeated observation of breast tumor subtypes in independent gene
  expression data sets.
Proc. Natl. Acad. Sci. USA, 100(14), 8418–8423.

[Srivastava et al.(2014)Srivastava, Hinton, Krizhevsky,
  Sutskever, and Salakhutdinov]Srivastava2014Dropout
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov,
  R. (2014).
Dropout: A simple way to prevent neural networks from overfitting.
J. Mach. Learn. Res., 15(1), 1929–1958.

[Tasic et al.(2016)Tasic, Menon, Nguyen, Kim, Jarsky, Yao, Levi,
  Gray, Sorensen, Dolbeare, Bertagnolli, Goldy, Shapovalova, Parry, Lee, Smith,
  Bernard, Madisen, Sunkin, Hawrylycz, Koch, and Zeng]Tasic2016Adult
Tasic, B., Menon, V., Nguyen, T. N., Kim, T. K., Jarsky, T., Yao, Z., Levi, B.,
  Gray, L. T., Sorensen, S. A., Dolbeare, T., Bertagnolli, D., Goldy, J.,
  Shapovalova, N., Parry, S., Lee, C., Smith, K., Bernard, A., Madisen, L.,
  Sunkin, S. M., Hawrylycz, M., Koch, C., and Zeng, H. (2016).
Adult mouse cortical cell taxonomy revealed by single cell
  transcriptomics.
Nat. Neurosci., 19(2), 335–346.

[Tibshirani(1996)Tibshirani]Tibshirani1996Regression
Tibshirani, R. (1996).
Regression shrinkage and selection via the lasso.
J. R. Stat. Soc. Ser. B, 58(1), 267–288.

[van de Vijver et al.(2002)van de Vijver, He, van't Veer, Dai,
  Hart, Voskuil, Schreiber, Peterse, Roberts, Marton, Parrish, Atsma,
  Witteveen, Glas, Delahaye, van der Velde, Bartelink, Rodenhuis, Rutgers,
  Friend, and Bernards]Vijver2002gene-expression
van de Vijver, M. J., He, Y. D., van't Veer, L. J., Dai, H., Hart, A. A. M.,
  Voskuil, D. W., Schreiber, G. J., Peterse, J. L., Roberts, C., Marton, M. J.,
  Parrish, M., Atsma, D., Witteveen, A., Glas, A., Delahaye, L., van der Velde,
  T., Bartelink, H., Rodenhuis, S., Rutgers, E. T., Friend, S. H., and
  Bernards, R. (2002).
A gene-expression signature as a predictor of survival in breast
  cancer.
N. Engl. J. Med., 347(25), 1999–2009.

[van der Maaten et al.(2013)van der Maaten, Chen, Tyree, and
  Weinberger]Maaten2013Learning
van der Maaten, L., Chen, M., Tyree, S., and Weinberger, K. Q. (2013).
Learning with marginalized corrupted features.
In Proceedings of the 30th International Conference on Machine
  Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013, number 28 in JMLR
  Proceedings, pages 410–418. JMLR.org.

[Villani et al.(2017)Villani, Satija, Reynolds, Sarkizova,
  Shekhar, Fletcher, Griesbeck, Butler, Zheng, Lazo, et al.]Villani2017Single
Villani, A.-C., Satija, R., Reynolds, G., Sarkizova, S., Shekhar, K., Fletcher,
  J., Griesbeck, M., Butler, A., Zheng, S., Lazo, S., et al. (2017).
Single-cell RNA-seq reveals new types of human blood dendritic
  cells, monocytes, and progenitors.
Science, 356(6335), eaah4573.

[Wager et al.(2013)Wager, Wang, and Liang]Wager2013Dropout
Wager, S., Wang, S., and Liang, P. S. (2013).
Dropout training as adaptive regularization.
In C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and
  K. Weinberger, editors, Adv. Neural. Inform. Process Syst., volume 26,
  pages 351–359. Curran Associates, Inc.

[Wager et al.(2014)Wager, Fithian, Wang, and
  Liang]Wager2014Altitude
Wager, S., Fithian, W., Wang, S., and Liang, P. S. (2014).
Altitude training: Strong bounds for single-layer dropout.
In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q.
  Weinberger, editors, Adv. Neural. Inform. Process Syst., pages
  100–108. Curran Associates, Inc.

[Zeisel et al.(2015)Zeisel, Manchado, Codeluppi, Lonnerberg,
  La Manno, Jureus, Marques, Munguba, He, Betsholtz, Rolny, Castelo-Branco,
  Hjerling-Leffler, and Linnarsson]Zeisel2015Cell
Zeisel, A., Manchado, A. B. M., Codeluppi, S., Lonnerberg, P., La Manno, G.,
  Jureus, A., Marques, S., Munguba, H., He, L., Betsholtz, C., Rolny, C.,
  Castelo-Branco, G., Hjerling-Leffler, J., and Linnarsson, S. (2015).
Cell types in the mouse cortex and hippocampus revealed by
  single-cell RNA-seq.
Science, 347(6226), 1138–42.

[Zou and Hastie(2005)Zou and Hastie]Zou2005regularization
Zou, H. and Hastie, T. (2005).
Regularization and variable selection via the Elastic Net.
J. R. Stat. Soc. Ser. B, 67, 301–320.


