Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We propose a new output layer for deep neural networks that permits the use of logged contextual bandit feedback for training. Such contextual bandit feedback can be available in huge quantities (e.g., logs of search engines, recommender systems) at little cost, opening up a path for training deep networks on orders of magnitude more data. To this effect, we propose a Counterfactual Risk Minimization (CRM) approach for training deep networks using an equivariant empirical risk estimator with variance regularization, BanditNet, and show how the ing objective can be decomposed in a way that allows Stochastic Gradient Descent (SGD) training. We empirically demonstrate the effectiveness of the method by showing how deep networks -- ResNets in particular -- can be trained for object recognition without conventionally labeled images. Log data can be recorded from online systems such as search engines, recommender systems, or online stores at little cost and in huge quantities. For concreteness, consider the interaction logs of an ad-placement system for banner ads. Such logs typically contain a record of the input to the system (e.g., features describing the user, banner ad, and page), the action that was taken by the system (e.g., a specific banner ad that was placed) and the feedback furnished by the user (e.g., clicks on the ad, or monetary payoff). This feedback, however, provides only partial information -"contextual-bandit feedback" -limited to the actions taken by the system. We do not get to see how the user would have responded, if the system had chosen a different action (e.g., other ads or banner types). Thus, the feedback for all other actions the system could have taken is typically not known. This makes learning from log data fundamentally different from traditional supervised learning, where "correct" predictions and a loss function provide feedback for all actions. In this paper, we propose a new output layer for deep neural networks that allows training on logged contextual bandit feedback. By circumventing the need for full-information feedback, our approach opens a new and intriguing pathway for acquiring knowledge at unprecedented scale, giving deep neural networks access to this abundant and ubiquitous type of data. Similarly, it enables the application of deep learning even in domains where manually labeling full-information feedback is not viable. In contrast to online learning with contextual bandit feedback (e.g., BID11 BID0), we perform batch learning from bandit feedback (BLBF) BID1 BID5 and the algorithm does not require the ability to make interactive interventions. At the core of the new output layer for BLBF training of deep neural networks lies a counterfactual training objective that replaces the conventional cross-entropy objective. Our approach -called BanditNet -follows the view of a deep neural network as a stochastic policy. We propose a counterfactual risk minimization (CRM) objective that is based on an equivariant estimator of the true error that only requires propensity-logged contextual bandit feedback. This makes our training objective fundamentally different from the conventional cross-entropy objective for supervised classification, which requires full-information feedback. Equivariance in our context means that the learning is invariant to additive translations of the loss, and it is more formally defined in Section 3.2. To enable large-scale training, we show how this training objective can be decomposed to allow stochastic gradient descent (SGD) optimization. In addition to the theoretical derivation of BanditNet, we present an empirical evaluation that verifies the applicability of the theoretical argument. It demonstrates how a deep neural network architec-ture can be trained in the BLBF setting. In particular, we derive a BanditNet version of ResNet for visual object classification. Despite using potentially much cheaper data, we find that Bandit-ResNet can achieve the same classification performance given sufficient amounts of contextual bandit feedback as ResNet trained with cross-entropy on conventionally (full-information) annotated images. To easily enable experimentation on other applications, we share an implementation of BanditNet. 1 2 RELATED WORK Several recent works have studied weak supervision approaches for deep learning. Weak supervision has been used to pre-train good image features and for information retrieval BID3. Closely related works have studied label corruption on CIFAR-10 recently BID12. However, all these approaches use weak supervision/corruption to construct noisy proxies for labels, and proceed with traditional supervised training (using crossentropy or mean-squared-error loss) with these proxies. In contrast, we work in the BLBF setting, which is an orthogonal data-source, and modify the loss functions optimized by deep nets to directly implement risk minimization. Virtually all previous methods that can learn from logged bandit feedback employ some form of risk minimization principle BID9 over a model class. Most of the methods BID1 BID2 BID5 employ an inverse propensity scoring (IPS) estimator as empirical risk and use stochastic gradient descent (SGD) to optimize the estimate over large datasets. Recently, the self-normalized estimator BID8 ) has been shown to be a more suitable estimator for BLBF BID7. The self-normalized estimator, however, is not amenable to stochastic optimization and scales poorly with dataset size. In our work, we demonstrate how we can efficiently optimize a reformulation of the self-normalized estimator using SGD.Previous BLBF methods focus on simple model classes: log-linear and exponential models (a) or tree-based reductions BID1 ). In contrast, we demonstrate how current deep learning models can be trained effectively via batch learning from bandit feedback (BLBF), and compare these with existing approaches on a benchmark dataset .Our work, together with independent concurrent work BID4, demonstrates success with off-policy variants of the REINFORCE BID11 algorithm. In particular, our algorithm employs a Lagrangian reformulation of the self-normalized estimator, and the objective and gradients of this reformulation are similar in spirit to the updates of the REINFORCE algorithm. This connection sheds new light on the role of the baseline hyper-parameters in REINFORCE: rather than simply reduce the variance of policy gradients, our work proposes a constructive algorithm for selecting the baseline in the off-policy setting and it suggests that the baseline is instrumental in creating an equivariant counterfactual learning objective. To formalize the problem of batch learning from bandit feedback for deep neural networks, consider the contextual bandit setting where a policy π takes as input x ∈ X and outputs an action y ∈ Y. In response, we observe the loss (or payoff) δ(x, y) of the selected action y, where δ(x, y) is an arbitrary (unknown) function that maps actions and contexts to a bounded real number. For example, in display advertising, the context x could be a representation of the user and page, y denotes the displayed ad, and δ(x, y) could be the monetary payoff from placing the ad (zero if no click, or dollar amount if clicked). The contexts are drawn i.i.d. from a fixed but unknown distribution Pr(X).In this paper, a (deep) neural network is viewed as implementing a stochastic policy π. We can think of such a network policy as a conditional distribution π w (Y | x) over actions y ∈ Y, where w are the parameters of the network. The network makes a prediction by sampling an action y ∼ π w (Y | x), where deterministic π w (Y | x) are a special case. As we will show as part of the empirical evaluation, many existing network architectures are compatible with this stochastic-policy view. For example, any network f w (x, y) with a softmax output layer DISPLAYFORM0 can be re-purposed as a conditional distribution from which one can sample actions, instead of interpreting it as a conditional likelihood like in full-information supervised learning. The goal of learning is to find a policy π w that minimizes the risk (analogously: maximizes the payoff) defined as DISPLAYFORM1 Any data collected from an interactive system depends on the policy π 0 that was running on the system at the time, determining which actions y and losses δ(x, y) are observed. We call π 0 the logging policy, and for simplicity assume that it is stationary. The logged data D are n tuples of observed context x i ∼ Pr(X), action y i ∼ π 0 (Y | x i) taken by the logging policy, the probability of this action p i ≡ π 0 (y i | x i), which we call the propensity, and the received loss δ i ≡ δ(x i, y i): DISPLAYFORM2 We will now discuss how we can use this logged contextual bandit feedback to train a neural network policy π w (Y | x) that has low risk R(π w). While conditional maximum likelihood is a standard approach for training deep neural networks, it requires that the loss δ(x i, y) is known for all y ∈ Y. However, we only know δ(x i, y i) for the particular y i chosen by the logging policy π 0. We therefore take a different approach following (; BID6), where we directly minimize an empirical risk that can be estimated from the logged bandit data D. This approach is called counterfactual risk minimization (CRM) BID6, since for any policy π w it addresses the counterfactual question of how well that policy would have performed, if it had been used instead of π 0.While minimizing an empirical risk as an estimate of the true risk R(π w) is a common principle in machine learning BID9, getting a reliable estimate based on the training data D produced by π 0 is not straightforward. The logged bandit data D is not only incomplete (i.e., we lack knowledge of δ(x i, y) for many y ∈ Y that π w would have chosen differently from π 0 ), but it is also biased (i.e., the actions preferred by π 0 are over-represented). This is why existing work on training deep neural networks either requires full knowledge of the loss function, or requires the ability to interactively draw new samples y i ∼ π w (Y | x i) for any new policy π w. In our setting we can do neither -we have a fixed dataset D that is limited to samples from π 0.To nevertheless get a useful estimate of the empirical risk, we explicitly address both the bias and the variance of the risk estimate. To correct for sampling bias and handle missing data, we approach the risk estimation problem using importance sampling and thus remove the distribution mismatch between π 0 and π w (; ; BID6 : DISPLAYFORM0 The latter expectation can be estimated on a sample D of n bandit-feedback examples using the following IPS estimator (; ; BID6 : DISPLAYFORM1 This IPS estimator is unbiased and has bounded variance, if the logging policy has full support in the sense that ∀x, y : π 0 (y | x) ≥ > 0. While at first glance it may seem natural to directly train the parameters w of a network to optimize this IPS estimate as an empirical risk, there are at least three obstacles to overcome. First, we will argue in the following section that the naive IPS estimator's lack of equivariance makes it sub-optimal for use as an empirical risk for high-capacity models. Second, we have to find an efficient algorithm for minimizing the empirical risk, especially making it accessible to stochastic gradient descent (SGD) optimization. And, finally, we are faced with an unusual type of bias-variance trade-off since "distance" from the exploration policy impacts the variance of the empirical risk estimate for different w. While Eq. FORMULA4 provides an unbiased empirical risk estimate, it exhibits the -possibly severe -problem of "propensity overfitting" when directly optimized within a learning algorithm BID7. It is a problem of overfitting to the choices y i of the logging policy, and it occurs on top of the normal overfitting to the δ i. Propensity overfitting is linked to the lack of equivariance of the IPS estimator: while the minimizer of true risk R(π w) does not change when translating the loss by a constant (i.e., ∀x, y : δ(x, y) + c) by linearity of expectation, DISPLAYFORM0 the minimizer of the IPS-estimated empirical riskR IPS (π w) can change dramatically for finite training samples, and c + min DISPLAYFORM1 Intuitively, when c shifts losses to be positive numbers, policies π w that put as little probability mass as possible on the observed actions have low risk estimates. If c shifts the losses to the negative range, the exact opposite is the case. For either choice of c, the choice of the policy eventually selected by the learning algorithm can be dominated by where π 0 happens to sample data, not by which actions have low loss. The following self-normalized IPS estimator (SNIPS) addresses the propensity overfitting problem BID7 and is equivariant: DISPLAYFORM2 In addition to being equivariant, this estimate can also have substantially lower variance than Eq. FORMULA4, since it exploits the knowledge that the denominator DISPLAYFORM3 always has expectation 1: DISPLAYFORM4 The SNIPS estimator uses this knowledge as a multiplicative control variate BID7. While the SNIPS estimator has some bias, this bias asymptotically vanishes at a rate of O(1 n) . Using the SNIPS estimator as our empirical risk implies that we need to solve the following optimization problem for training: DISPLAYFORM5 Thus, we now turn to designing efficient optimization methods for this training objective. Unfortunately, the training objective in Eq. FORMULA0 does not permit stochastic gradient descent (SGD) optimization in the given form (see Appendix C), which presents an obstacle to efficient and effective training of the network. To remedy this problem, we will now develop a reformulation that retains both the desirable properties of the SNIPS estimator, as well as the ability to reuse established SGD training algorithms. Instead of optimizing a ratio as in Eq., we will reformulate the problem into a series of constrained optimization problems. Letŵ be a solution of Eq., and at that solution let S * be the value of the control variate for πŵ as defined in Eq.. For simplicity, assume that the minimizerŵ is unique. If we knew S *, we could equivalently solve the following constrained optimization problem: DISPLAYFORM0 Of course, we do not actually know S *. However, we can do a grid search in {S 1, . . ., S k} for S * and solve the above optimization problem for each value, giving us a set of solutions {ŵ 1, . . .,ŵ k}. Note that S is just a one-dimensional quantity, and that the sensible range we need to search for S * concentrates around 1 as n increases (see Appendix B). To find the overall (approximate)ŵ that optimizes the SNIPS estimate, we then simply take the minimum: DISPLAYFORM1 This still leaves the question of how to solve each equality constrained risk minimization problem using SGD. Fortunately, we can perform an equivalent search for S * without constrained optimization. To this effect, consider the Lagrangian of the constrained optimization problem in Eq. FORMULA0 with S j in the constraint instead of S *: DISPLAYFORM2 The variable λ is an unconstrained Lagrange multiplier. To find the minimum of Eq. for a particular S j, we need to minimize L(w, λ) w.r.t. w and maximize w.r.t. λ. DISPLAYFORM3 However, we are not actually interested in the constrained solution of Eq. for any specific S j. We are merely interested in exploring a certain range S ∈ [S 1, S k] in our search for S *. So, we can reverse the roles of λ and S, where we keep λ fixed and determine the corresponding S in hindsight. In particular, for each {λ 1, . . ., λ k} we solvê DISPLAYFORM4 Note that the solutionŵ j does not depend on S j, so we can compute S j after we have found the minimumŵ j. In particular, we can determine the S j that corresponds to the given λ j using the necessary optimality conditions, DISPLAYFORM5 by solving the second equality of Eq.. In this way, the sequence of λ j produces solutionsŵ j corresponding to a sequence of {S 1, . . ., S k}.To identify the sensible range of S to explore, we can make use of the fact that Eq. concentrates around its expectation of 1 for each π w as n increases. Theorem 2 in Appendix B provides a characterization of how large the range needs to be. Furthermore, we can steer the exploration of S via λ, since the ing S changes monotonically with λ: (λ a < λ b) and (ŵ a =ŵ b are not equivalent optima in Eq.) ⇒ (S a < S b). A more formal statement and proof are given as Theorem 1 in Appendix A. In the simplest form one could therefore perform a grid search on λ, but more sophisticated search methods are possible too. After this reformulation, the key computational problem is finding the solution of Eq. for each λ j. Note that in this unconstrained optimization problem, the Lagrange multiplier effectively translates the loss values in the conventional IPS estimate: We denote this λ-translated IPS estimate withR λ IPS (π w). Note that each such optimization problem is now in the form required for SGD, where we merely weight the derivative of the stochastic policy network π w (y | x) by a factor (δ i − λ j)/π 0 (y i | x i). This opens the door for re-purposing existing fast methods for training deep neural networks, and we demonstrate experimentally that SGD with momentum is able to optimize our objective scalably. DISPLAYFORM6 Similar loss translations have previously been used in on-policy reinforcement learning BID11, where they are motivated as minimizing the variance of the gradient estimate BID10 ). However, the situation is different in the off-policy setting we consider. First, we cannot sample new roll-outs from the current policy under consideration, which means we cannot use the standard variance-optimal estimator used in REINFORCE. Second, we tried using the (estimated) expected loss of the learned policy as the baseline as is commonly done in REINFORCE, but will see in the experiment section that this value for λ is far from optimal. Finally, it is unclear whether gradient variance, as opposed to variance of the ERM objective, is really the key issue in batch learning from bandit feedback. In this sense, our approach provides a rigorous justification and a constructive way of picking the value of λ in the off-policy settingnamely the value for which the corresponding S j minimizes Eq.. In addition, one can further add variance regularization BID6 to improve the robustness of the risk estimate in Eq. (see Appendix D for details). The empirical evaluation is designed to address three key questions. First, it verifies that deep models can indeed be trained effectively using our approach. Second, we will compare how the same deep neural network architecture performs under different types of data and training objectives -in particular, conventional cross-entropy training using full-information data. In order to be able to do this comparison, we focus on synthetic contextual bandit feedback data for training BanditNet that is sampled from the full-information labels. Third, we explore the effectiveness and fidelity of the approximate SNIPS objective. For the following BanditNet experiments, we adapted the ResNet20 architecture by replacing the conventional cross-entropy objective with our counterfactual risk minimization objective. We evaluate the performance of this Bandit-ResNet on the CIFAR-10 dataset, where we can compare training on full-information data with training on bandit feedback, and where there is a full-information test set for estimating prediction error. To simulate logged bandit feedback, we perform the standard supervised to bandit conversion BID1 ). We use a hand-coded logging policy that achieves about 49% error rate on the training data, which is substantially worse than what we hope to achieve after learning. This emulates a real world scenario where one would bootstrap an operational system with a mediocre policy (e.g., derived from a small hand-labeled dataset) and then deploys it to log bandit feedback. This logged bandit feedback data is then used to train the Bandit-ResNet. We evaluate the trained model using error rate on the held out (full-information) test set. We compare this model against the skyline of training a conventional ResNet using the full-information feedback from the 50,000 training examples. Both the conventional full-information ResNet as well as the Bandit-ResNet use the same network architecture, the same hyperparameters, the same data augmentation scheme, and the same optimization method that were set in the CNTK implementation of ResNet20. Since CIFAR10 does not come with a validation set for tuning the variance-regularization constant γ, we do not use variance regularization for Bandit-ResNet. The Lagrange multiplier λ ∈ {0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95, 1.0, 1.05} is selected on the training set via Eq.. The only parameter we adjusted for Bandit-ResNet is lowering the learning rate to 0.1 and slowing down the learning rate schedule. The latter was done to avoid confounding the Bandit-ResNet with potential effects from early stopping, and we report test performance after 1000 training epochs, which is well beyond the point of convergence in all runs. Learning curve. FIG0 shows the prediction error of the Bandit-ResNet as more and more bandit feedback is provided for training. First, even though the logging policy that generated the bandit feedback has an error rate of 49%, the prediction error of the policy learned by the Bandit-ResNet is substantially better. It is between 13% and 8.2%, depending on the amount of training data. Second, the horizontal line is the performance of a conventional ResNet trained on the full-information training set. It serves as a skyline of how good Bandit-ResNet could possibly get given that it is sampling bandit feedback from the same full-information training set. The learning curve in FIG0 shows that Bandit-ResNet converges to the skyline performance given enough bandit feedback training data, providing strong evidence that our training objective and method can effectively extract the available information provided in the bandit feedback. Effect of the choice of Lagrange multiplier. The left-hand plot in FIG1 shows the test error of solutionsŵ j depending on the value of the Lagrange multiplier λ j used during training. It shows that λ in the range 0.8 to 1.0 in good prediction performance, but that performance degrades outside this area. The SNIPS estimates in the right-hand plot of FIG1 roughly reflects this optimal range, given empirical support for both the SNIPS estimator and the use of Eq..We also explored two other methods for selecting λ. First, we used the straightforward IPS estimator as the objective (i.e., λ = 0), which leads to prediction performance worse than that of the logging policy (not shown). Second, we tried using the (estimated) expected loss of the learned policy as the baseline as is commonly done in REINFORCE. As FIG0 shows, it is between 0.130 and 0.083 for the best policies we found. FIG1 (left) shows that these baseline values are well outside of the optimum range. Also shown in the right-hand plot of FIG1 is the value of the control variate in the denominator of the SNIPS estimate. As expected, it increases from below 1 to above 1 as λ is increased. Note that large deviations of the control variate from 1 are a sign of propensity overfitting BID7. In particular, for all solutionsŵ j the estimated standard error of the control variate S j was less than 0.013, meaning that the normal 95% confidence interval for each S j is contained in [0.974, 1.026]. If we see aŵ j with control variate S j outside this range, we should be suspicious of propensity overfitting to the choices of the logging policy and discard this solution. We proposed a new output layer for deep neural networks that enables the use of logged contextual bandit feedback for training. This type of feedback is abundant and ubiquitous in the form of interaction logs from autonomous systems, opening up the possibility of training deep neural networks on unprecedented amounts of data. In principle, this new output layer can replace the conventional cross-entropy layer for any network architecture. We provide a rigorous derivation of the training objective, linking it to an equivariant counterfactual risk estimator that enables counterfactual risk minimization. Most importantly, we show how the ing training objective can be decomposed and reformulated to make it feasible for SGD training. We find that the BanditNet approach applied to the ResNet architecture achieves predictive accuracy comparable to conventional full-information training for visual object recognition. The paper opens up several directions for future work. First, it enables many new applications where contextual bandit feedback is readily available. Second, in settings where it is infeasible to log propensity-scored data, it would be interesting to combine BanditNet with propensity estimation techniques. Third, there may be improvements to BanditNet, like smarter search techniques for S, more efficient counterfactual estimators beyond SNIPS, and the ability to handle continuous outputs. DISPLAYFORM0 If the optimaŵ a andŵ b are not equivalent in the sense thatR DISPLAYFORM1 where g(w) corresponds to the value of the control variate S. Sinceŵ a andŵ b are not equivalent optima, we know that DISPLAYFORM2 Adding the two inequalities and solving implies that DISPLAYFORM3 B APPENDIX: CHARACTERIZING THE RANGE OF S TO EXPLORE.Theorem 2. Let p ≤ π 0 (y | x) be a lower bound on the propensity for the logging policy, then constraining the solution of Eq. to the w with control variate S ∈ [1 −, 1 +] for a training set of size n will not exclude the minimizer of the true risk w * = arg min w∈W R(π w) in the policy space W with probability at least DISPLAYFORM4 Proof. For the optimal w *, let DISPLAYFORM5 be the control variate in the denominator of the SNIPS estimator. S is a random variable that is a sum of bounded random variables between 0 and DISPLAYFORM6 We can bound the probability that the control variate S of the optimum w * lies outside of [1−, 1+] via Hoeffding's inequality: DISPLAYFORM7 The same argument applies to any individual policy π w, not just w *. Note, however, that it can still be highly likely that at least one policy π w with w ∈ W shows a large deviation in the control variate for high-capacity W, which can lead to propensity overfitting when using the naive IPS estimator. Suppose we have a dataset of n BLBF samples D = {(x 1, y 1, δ 1, p 1)... (x n, y n, δ n, p n)} where each instance is an i.i.d. sample from the data generating distribution. In the sequel we will be considering two datasets of n + 1 samples, D = D ∪ {(x, y, δ, p)} and D = D ∪ {(x, y, δ, p)} where (x, y, δ, p) = (x, y, δ, p) and (x, y, δ, p), (x, y, δ, p) / ∈ D.For notational convenience, let DISPLAYFORM8 π0(yi|xi), andġ i:= ∇ w g i. First consider the vanilla IPS risk estimate of Eq.. DISPLAYFORM9 To maximize this estimate using stochastic optimization, we must construct an unbiased gradient estimate. That is, we randomly select one sample from D and compute a gradient α((x i, y i, δ i, p i)) and we require that DISPLAYFORM10 Here the expectation is over our random choice of 1 out of n samples. Observe that α((x i, y i, δ i, p i)) =ḟ i suffices (and indeed, this corresponds to vanilla SGD): DISPLAYFORM11 Other choices of α(·) can also produce unbiased gradient estimates, and this leads to the study of stochastic variance-reduced gradient optimization. Now let us attempt to construct an unbiased gradient estimate for Eq.: DISPLAYFORM12 Suppose such a gradient estimate exists, β((x i, y i, δ i, p i)). Then, DISPLAYFORM13 This identity is true for any sample of BLBF instances -in particular, for D and D: DISPLAYFORM14 1 n + 1 β((x i, y i, δ i, p i)) + β((x, y, δ, p)) n + 1, DISPLAYFORM15 1 n + 1 β((x i, y i, δ i, p i)) + β((x, y, δ, p)) n + 1.Subtracting these two equations, DISPLAYFORM16 = β((x, y, δ, p)) − β((x, y, δ, p)) n + 1.The LHS clearly depends on {(x i, y i, δ i, p i)} n i=1 in general, while the RHS does not! This contradiction indicates that no construction of β that only looks at a sub-sample of the data can yield an unbiased gradient estimate ofR SNIPS (π w). Unlike in conventional supervised learning, a counterfactual empirical risk estimator likeR IPS (π w) can have vastly different variances Var(R IPS (π w)) for different π w in the hypothesis space (and R SNIPS (π w) as well) BID6. Intuitively, the "closer" the particular π w is to the exploration policy π 0, the larger the effective sample size will be and the smaller the variance of the empirical risk estimate. For the optimization problems we solve in Eq. FORMULA0, this means that we should trust the λ-translated risk estimateR λj IPS (π w) more for some w than for others, as we useR λj IPS (π w) only as a proxy for finding the policy that minimizes its expected value (i.e., the true loss). To this effect, generalization error bounds that account for this variance difference BID6 ) motivate a new type of overfitting control. This leads to the following training objective BID6, which can be thought of as a more reliable version of Eq. FORMULA0: DISPLAYFORM0 Here, Var(R λj IPS (π w)) is the estimated variance ofR λj IPS (π w) on the training data, and γ is a regularization constant to be selected via cross validation. The intuition behind this objective is that we optimize the upper confidence interval, which depends on the variance of the risk estimate for each π w. While this objective again does not permit SGD optimization in its given form, it has been shown that a Taylor-majorization can be used to successively upper bound the objective in Eq. FORMULA2, and that typically a small number of iterations suffices to converge to a local optimum BID6. Each such Taylor-majorization is again of a form DISPLAYFORM1 for easily computable constants A and B BID6, which allows for SGD optimization.

The paper proposes a new output layer for deep networks that permits the use of logged contextual bandit feedback for training.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

The paper designs two algorithms for the stochastic AUC maximization problem with state-of-the-art complexities when using deep neural network as predictive model, which are also verified by empirical studies.


Question: Represent this text

Question: Protein classification is responsible for the biological sequence, we came up with an idea whichdeals with the classification of proteomics using deep learning algorithm. This algorithm focusesmainly to classify sequences of protein-vector which is used for the representation of proteomics. Selection of the type protein representation is challenging based on which output in terms ofaccuracy is depended on, The protein representation used here is n-gram i.e. 3-gram and Kerasembedding used for biological sequences like protein. In this paper we are working on the Proteinclassification to show the strength and representation of biological sequence of the proteins Human body comprises of the many cells the key for the formation of all these are the DNA(Deoxyribonucleic acid) which is a chain of nucleotides that are thread like chain in nature responsible for carrying genetic instruction for development, functioning of organisms in the body and RNA(Ribonucleic acid) which is a polymeric molecule which is very much essential for the three biological roles namely CDRE(Coding, Decoding,Regulation and Expression) of each and every gene present in the body, which are present in every living being. Like human beings use various language for communication, biological organisms use these type of the codes by using DNA and RNA for the communication. Selection of the type of feature extraction is a challenging task because it helps for the studying of the types of the genes to the machine Email address: naveensivakumarr@gmail.com (Naveenkumar K S) using the machine learning algorithm, Even a most highly sophisticated algorithm would go wrong if the feature extraction is not done in a proper form. The features from the existing data can be obtained by manually or by using unsupervised (data without labels) fashion BID0, BID1. This work focuses on protein family classification wit the publically avilable data Swiss-Prot BID2. In this work application of keras embedding and n-gram technique is used to map the protein sequences into numeric vectors and followed by traditional machine learning and deep neural network (DNN) for classification. The rest of the paper are organized as follows. Section 2 discusses the related works in a detailed manner. Section 3 presents the works. Section 4 presents the description of the data set. Section 5 gives a overview about the proposed architecture which is used in building this work. Section 6 and 7 presents and , future work directions and discussions. In this paper we mainly focus on the related works done previously on protein classification using deep learning. Using Natural language processing which mainly focus on biological sequence, the features are been extracted using n-grams,Bio-vec and Prot-v to identify the protein structure and classified into different structure for which deep neural network has been implemented to find ndimensional vectors BID3 BID4. These are considered as main features,using some of CBOW and skip-gram methods, word2vec, DM and DBOW for doc vec consider another method in which we can classify the protein biological embedding sequences BID0.In which supervised learning has been used as classifier to classify the predictive performance of the data, from the data that has been classified various features were been extracted and compared,after which the accuracy based classification is done BID5. Using the extracted features and considering the values, prediction is done by using Random forestry,Naviers bayes classifier and Decision tree methods. The outputs of these are taken under consideration, which is helpful in prediction solving of protein classifications and time efficient BID6 BID7. In proteins the transformation of sequence of cells requires an analysis to obtain source function of proteins to obtain this sequence gaussian process regression is generalize to find number of variables BID8. By training the sequential data of various proteins regarding the set function analysis accuracy and can be obtained BID3. The proteins are classified based on the sequence with a distribution of the ten classes based on the algorithm which has been developed recently which is an advanced level of the deep learning that is by the means of the ELM (Extreme machine learning). When this algorithm is tested by comparing it with the old that is with the existing algorithm which is none other than the BNN (Backpropagation neural network) method it had a significantly greater response when compared with the BNN, the ELM requires very less time and four times magnitude less training when compared with BNN with a greater accuracy. The ELM (Extreme machine learning) which is used in this architecture of network do not have any of the parameters for controlling automatically only manual operation is needed which is easy to be tuned to the machine BID9.The classification are the implementation part of the data into the algorithm some of the types of the classifiers that are being implemented are GE (Genetic algorithm), Fuzzy ARTMAP which are being used in the deep learning for classification that has to improve the accuracy. This deals with the GE, Fuzzy ARTMAP and RCM (Rough set classifier model) BID10.The classification of the protein is a tough task for the persons to identify their family because it is tedious time consuming process now a days the machines are being used for the classification based on the machine learning techniques such as the SVM(Support vector machine),RF(Random forest).K-NN(K-nearest neighbor) and Fuzzy K-NN are being used BID11. Proteomics is the study of proteins which is concerned with the protein expression analysis of a cell or an organism. Proteins are molecules that contains one or more long chains of amino acids which performs variety of functions such as catalytic metabolic reaction, DNA replication, Responding to stimuli and transporting molecules inside the cell. The amino acids in the proteins are formed of sequences which are stated by their genes of the nucleotide which in protein folding into a three dimensional specific function. Proteome can be expressed as the inverse of the entire protein which can be shown by an organism, tissue or by a cell. They vary with re-spect to time and their requirements based on the stresses of the cells BID3. Proteomics which are combination of various branches has got contributed from genetic information of HGP (Human Genome Project) which is a blooming topic under research of proteins. Proteomics shows that it is responsible for the large scale experiment analysis of proteins and is specifically used on protein purification and mass spectrometry BID0. Genomics a branch of the molecular biology which is a study related to the function, evolution, structure, editing and mapping of genomes. Genome consists of the haploid (single set of unpaired chromosomes) set of chromosomes in each cell of multicellular organism or in gametes of microorganisms they are also referred to a complete set of genetic material or genes present in a cell or organism. An organisms complete set of DNA including all its genes is Genome BID5. On the other hand Genetics means study of genes which are individual and their functions in inheritance, whereas genomics is the study of characterization which are collective and genes of qualification, that recalls the production of the proteins in a method that is direct that is with the AOE (assistance of enzymes) and MM(messenger molecules). Humans used to communicate with each other for the purpose of the communication in the same way the computers communicate with its users by the means of the algorithm that is by performing the task which was assigned by the coder to the machine. The programmer uses the certain algorithm to ask the computer rather the machine to perform certain task. As years passed by the computers are taught linguistics in the form of the data put in an algorithm with the help of the probability, as a a new field is emerged that is computational linguistics along with probability. Many algorithms are being created for the feature extraction from the data, out of which n-gram is one of the feature extraction which is being followed majorly by all. N-Gram is a n items of the given sample which has a sequence of the text or speech those can be of the syllables, words, corpus (which is used for the text data), or text. Based on the type of the words,letters or a text that are used they are classified as "unigram" means taking one word or a letter or a text then as the "bigram" taking two at a time and so on such as "three-gram", "four-gram", "five-gram" etc. They are extensively used in the NLP(Natural Language Processing) and Speech Recognition which is a statistical and used for the extraction of the phonemes(distinct sound unit related to a particular language that are separated)and sequence of phonemes. Word embedding deals with the vector representation of the dense vectors. This method is much more accurate than that of the previous methods that was used which is the bag of words. Keras is the library function which allows the use of the embedding for the word semantically and syntactically. This is an API data as an input which can be prepared by the Tokenizer which has to be integer encoded. They are initialized with the random weights that are responsible for the embedding in the training set. In recent days, Keras embedding has performed well in compared to n-gram as text representation method in most of the applications BID12, BID13, BID14, BID15, BID16, BID17. Naives Bayes The naives Bayes is a classifier in the machine learning which uses the probabilistic methods alongside with the help of the bayes theorem, which is easily scalable with linear number of parameters such as the features or the predictors that help the machine to learn about the problem, is very efficient method of classifiers which has a very greater level of accuracy that are easy to learn, which mainly focus on the dependent attributes and they can easily deal with the parametric attributes which are continuous. The process of maximum training takes place by the method of the closed form evaluation expressions that takes in a linear time for the completion than that of the other method that is the iterative process of approximation. This algorithm has a technique that has the capability of assigning the labels that are given to the class with the problem instance which are represented in the form of the vectors of the values that are featured and the values of the class are taken from the set that is finite. There are two types of the method in the machine learning that is the supervised and the unsupervised machine learning this algorithm goes well with the supervised machine learning and has high efficiency on the machine learning. This has a greater advantages for the users who are using this algorithm for the various purposes is that this need a very little amount the training data set when it is being compared with the other types of the algorithms and gives a greater accuracy. A method which can be said as an algorithm that only depend on the statements which are conditional and control, uses the tool which takes the decision that has the tools which supports or agree with the graph which are like the tree with the models that have decision which can have a possible outcomes that also include the outcomes of the event chances along with the utility and the costs of the resources are the decision trees. This is a machine learning tool that is so popular which are used for the research operations because this type of the algorithm deals with the various types of the analysis mainly decision analysis that helps in identifying a problem or a situation which are similar in the case of the reaching a goal. The decision trees are in the structure of the flowchart which has nodes which are internal and they are referred by the term which is coined to be as "Test", that has a representation of the of the of the test in which it has a leaf node and each of the leaf node represents the class to which the label belongs. In the analysis done by the decision tree which can be also referred as the DA (decision analysis) they are form a close resemblance to that of the influence diagram which are used for the various other methods such as the VAD (visual, analytic and decision support method) by which the values of the alternatives which are competing can be calculated. Data is a point in space which has an infinite dimension, this century is the one which deals with data than with people large amount of data are found and they are increasing day by day so as the machine learning algorithms are used for the studying and dealing with data, one of the algorithm which deals with the data better are the random forest tree method which are used for solving the huge data sets. Algorithms of the recent trend are focused on the accuracy of the data even though they are focused on the accuracy they fail to classify the data with greater accuracy. So this algorithm that is random forest tree is used for the classification of the large amount of the data with the regression technique. Machine learning techniques are the recent trend in this era so this is used along with the random forest tree technique as an ensemble this leads to a cutting edge technique in the field of the data mining. Ensemble is the process of the data mining that consists of many classifiers who are individual for the classification of the data and to create a new data of instances. Random forest technique is one of the most famous and well know technique for ensemble classification because it has a lot of features present in it. Random forest trees are formed by the combination of predictors of the trees such as each of them depend upon the value of sampled random vector which are independent of each other with the similar type of the distribution to all the types of the trees that are present in that forest. The errors that occurs in this type of the trees are much reduced as the number of the tress that are formed are increased and become large. They are generalized so that the occurrence of the error is reduced even if the error occurs they depend on the strength of the trees that are individual, that belong to that forest and the relation between them. By using this method the features are splitted so that error rates are decreased and completely favorable with reduced noise. This method is not new to machine learning by the means of the name this may find different but the expansion for the ADABOOST is Adaptive Boosting, used by the weak learners for the coding kidding, they are also called as the discrete adaboost because it follows the classification method in the process when compared to regression. This type of the algorithm is used for boosting the performance of the machine learning as the name suggest. The performance of this type of the algorithm is for boosting the performance on the decision trees on the classification which are binary none other than the binary classification problem. The algorithm which is most suited with this type are the decision trees so they are used along with the adaboost algorithm, since these trees are short they contain only one decision tree for the purpose of the classification so they are called as the decision stumps. The weight is weighed at each instance of the dataset at the training level, the initial level of the weight is set to the weight that is given by the formula given below DISPLAYFORM0 where x i is the training instance at the i the level and n is the training instance. AdaBoost classifiers are a meta-estimator which begins by fitting classifiers on the datasets that are original, they are additional to that of the classifier which are original. The weights are incorrectly distributed at any instances that are adjusted to classifiers which are subsequent and focuses on difficult tasks much more. Support Vector Machine(SVM) is the classifier in the machine learning which stands for the Support Vector Machine (SVM) which means formally as the discriminative classifier which are separated by the hyperplane. There are three methods of the learning in the machine learning they are the supervised and unsupervised and reinforcement learning the svm deals with the supervised machine learning which means that it consist of the dependent variable that can be predicted using the predictors which is nothing but the independent variables. Here the support vector machine that is the SVM follows the supervised learning (which deals with the datasets that are labelled) along with the other algorithms that help in the analysis of the data that are used for the regression and the classification analysis. They are successful in the high dimensional spaces and also were the number of the dimensions are greater than the number of the samples. It uses the decision functions which are called as the support vectors for the purpose of the training the points in the subset so that the memory would be efficient. The k-nearest neighbors(KNN) stands for the K nearest neighbor algorithm which is a non-parametric methods which are used in the classification of the data and in the process of regression. In the both the methods which are stated above the input factors consists of the k closest examples for the training in the feature space, but the output depends upon the k-NN whether they are used for the process of the classification or for the regression. In this method the training data are the vectors that are in the multidimensional space which are in the feature with the each class with the label. During the training of the algorithm it consists of only the storing of the vectors which belong to the feature called as the feature vectors and the class to which the labels belong of that of the training data. When the k-NN is used in the classification the output which is got after the application of the algorithm is calculated as the class which is having the highest frequency from the base of the K-most similar instances, each of the instance are responsible for the votes of the class which are taken as the prediction. Deep learning is a sub model of machine learning which has the capability to obtain optimal feature representation from raw input samples BID18. Deep neural networks (DNN) is a method of the machine learning tools, which allows one to learn about the complex functions which are non-linear from a given input which reduces the error cost. Recurrent Neural Networks (RNN) is a method of the machine learning tools, which uses the sequential data since each neuron has the internal memory in them to store the information about the input which had occurred already BID18. RNN generates vanishing error gradient issues when try to learn the long-term temporal dependencies. Long short-term memory commonly referred as LSTM (or blocks) are the main units in the building of layers of a RNN (Recurrent Neural Network). This has the capability to handle vanishing and error gradient issue. LSTM has a memory which has the gates they are the input gate, output gate and the forget gate and a self-recurrent connections with that of the fixed weights. The cell which is present is the main responsible for the memory. Set of proteins which are evolutionarily related, similarly involving in structures or functions belongs to protein family.The protein family consists of the higher classifier named G-protein coupled receptors, based on which they are divided given in the picture below. The protein family is represented as the'Pfam' which are classified as the "Family" and "Domain" BID3.' Pfam' which belongs to the protein classification which are made by the biologist for the computational purpose includes a view of description of the family, multiple alignments, protein domain architectures, examine species distribution. Classification is the task next to feature extraction which determines the value for the data set that has been got. In this paper the classifier used for classifying protein family is SVM (Support Vector Machine) based on the sequences that is on primary structures this classifier classifies the given data. SVM (Support vector machine) are the supervised learning methods which are used by machine learning for analysis such as data and regression based classification BID5. The main objective of this process is to create a sequence in biology which is a distributed representation. The data is divided into two they are training data and the test data, the system is trained by the means of the training data and with the help of the test data it is verified. For this many process such as the ten cross ten is used to check the train data with the test, many algorithm are being used for this process, based on the accuracy which is being got and to improve the accuracy the algorithm is varied BID3. A large set of the training data is required for the machine to learn and based on the information present in the data, here it is the protein sequence machine learns as we learn and the output is when a sequence is being given to the machine rather the algorithm it tells to which sequence it belongs to easily within no time. The n-gram algorithm uses the information from the protein that are used for the lapping of the window from three to six residues. Using a two cross fold cross validation are more accurate for different window size. The protein space analysis is used for many process that are used in training space for the analysis of the spread over of the physical and chemical biological properties that are used in the n-gram more specifically the three gram is used in the hundred dimensional space to the two dimensional space by the means of the SNE (Stochastic Neighbor Embedding). Some properties like the volume and mass were being taken into consideration for the classification of the data, the protein space characteristics have been studied and from Lipschitz constant BID3 DISPLAYFORM0 Were f, is scale of the properties on which the n gram has been used, D, is the metric distance. Df, is the score differences of the absolute value. Dw, is the distance that is Euclidian between the 2 three-grams w1 and w2 We gathered family information of about 40433 protein sequences in Swiss-Prot from Protein family database(Pfam), which consists of 200 distinct families. Swiss-Prot is a curated database of primary protein sequences which is manually annotated and reviewed. There is no redundancy of protein sequences in the database and is evaluated based on obtained through experiments. The data contain 84753 protein sequences BID2. An overview of proposed architecture is shown in Fig 1. This contains a protein sequences. These sequences are passed into protein representation layers. This layer transforms protein into numeric vectors i.,e. dense vector representation. There are two types of representation are used. They are n=gram and Keras embedding. In n-gram we used 3 gram with feature hashing of length 1000. These vectors are passed into different deep learning layers such as dnn, rnn, lstm and CNN for optimal feature extraction. DNN model contains 5 hidden layer each hidden layer contain 128 hidden units. RNN and LSTM layer contains 128 units and memory blocks respectively. CNN contain 128 filters with filter length 4. CNN layer followed pooling layer. We used maxpooling which is of length 3. These feature are passed into fully connected layers for classification. The fully connected layer contains connection to every other neurons. This used softmax as an activation function. To reduce the loss, categorical cross entropy is used. The categorical cross entropy is defined as follows. DISPLAYFORM0 where is true probability distribution, is predicted probability distribution. We have used as an optimizer to minimize the loss of binarycross entropy and categorical-cross entropy. All experiments of deep learning algorithms are run on GPU enabled machines. All deep learning algorithms are implemented using TensorFlow BID19 with Keras BID20 framework. TAB0. This paper has proposed deep learning method for protein classification. To transform protein to numeric vector, the n-gram and Keras embedding representation is used. Deep learning method with Keras embedding has performed well in comparison to the n-gram with deep neural network. The main reason is due to the Keras embedding has the capability to preserver the sequential information among the protein sequences. Thus, the deep learning algorithms able to capture the optimal information from the syntactic and semantic information of Keras embedding vectors. The proposed methodology can be employed for other domain in biology tasks such as genomics, DNA classification. This is one of the significant directions towards future work.

Protein Family Classification using Deep Learning

Represent this text to find a much much longer article about it with say hundreds-thousands of words

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.