Question: Represent the text to find a very short summary (1-2 sentences max).

Question: Positive-unlabeled (PU) learning addresses the problem of learning a binary classifier from positive (P) and unlabeled (U) data. It is often applied to situations where negative (N) data are difficult to be fully labeled. However, collecting a non-representative N set that contains only a small portion of all possible N data can be much easier in many practical situations. This paper studies a novel classification framework which incorporates such biased N (bN) data in PU learning. The fact that the training N data are biased also makes our work very different from those of standard semi-supervised learning. We provide an empirical risk minimization-based method to address this PUbN classification problem. Our approach can be regarded as a variant of traditional example-reweighting algorithms, with the weight of each example computed through a preliminary step that draws inspiration from PU learning. We also derive an estimation error bound for the proposed method. Experimental demonstrate the effectiveness of our algorithm in not only PUbN learning scenarios but also ordinary PU leaning scenarios on several benchmark datasets. In conventional binary classification, examples are labeled as either positive (P) or negative (N), and we train a classifier on these labeled examples. On the contrary, positive-unlabeled (PU) learning addresses the problem of learning a classifier from P and unlabeled (U) data, without need of explicitly identifying N data BID6 BID42 ).PU learning finds its usefulness in many real-world problems. For example, in one-class remote sensing classification, we seek to extract a specific land-cover class from an image. While it is easy to label examples of this specific land-cover class of interest, examples not belonging to this class are too diverse to be exhaustively annotated. The same problem arises in text classification, as it is difficult or even impossible to compile a set of N samples that provides a comprehensive characterization of everything that is not in the P class BID24 BID8. Besides, PU learning has also been applied to other domains such as outlier detection BID13 BID36 ), medical diagnosis BID45, or time series classification BID28.By carefully examining the above examples, we find out that the most difficult step is often to collect a fully representative N set, whereas only labeling a small portion of all possible N data is relatively easy. Therefore, in this paper, we propose to study the problem of learning from P, U and biased N (bN) data, which we name PUbN learning hereinafter. We suppose that in addition to P and U data, we also gather a set of bN samples, governed by a distribution distinct from the true N distribution. As described previously, this can be viewed as an extension of PU learning, but such bias may also occur naturally in some real-world scenarios. For instance, let us presume that we would like to judge whether a subject is affected by a particular disease based on the of a physical examination. While the data collected from the patients represent rather well the P distribution, healthy subjects that request the examination are in general highly biased with respect to the whole healthy subject population. We are not the first to be interested in learning with bN data. In fact, both BID22 and BID7 attempted to solve similar problems in the context of text classification. BID22 simply discarded negative samples and performed ordinary PU classification. It was also mentioned in the paper that bN data could be harmful. BID7 adapted another strategy. The authors considered even gathering unbiased U data is difficult and learned the classifier from only P and bN data. However, their method is specific to text classification because it relies on the use of effective similarity measures to evaluate similarity between documents. Therefore, our work differs from these two in that the classifier is trained simultaneously on P, U and bN data, without resorting to domain-specific knowledge. The presence of U data allows us to address the problem from a statistical viewpoint, and thus the proposed method can be applied to any PUbN learning problem in principle. In this paper, we develop an empirical risk minimization-based algorithm that combines both PU learning and importance weighting to solve the PUbN classification problem, We first estimate the probability that an example is sampled into the P or the bN set. Based on this estimate, we regard bN and U data as N examples with instance-dependent weights. In particular, we assign larger weights to U examples that we believe to appear less often in the P and bN sets. P data are treated as P examples with unity weight but also as N examples with usually small or zero weight whose actual value depends on the same estimate. The contributions of the paper are three-fold:1. We formulate the PUbN learning problem as an extension of PU learning and propose an empirical risk minimization-based method to address the problem. We also theoretically establish an estimation error bound for the proposed method. 2. We experimentally demonstrate that the classification performance can be effectively improved thanks to the use of bN data during training. In other words, PUbN learning yields better performance than PU learning. 3. Our method can be easily adapted to ordinary PU learning. Experimentally we show that the ing algorithm allows us to obtain new state-of-the-art on several PU learning tasks. Relation with Semi-supervised Learning With P, N and U data available for training, our problem setup may seem similar to that of semi-supervised learning BID2 BID29. Nonetheless, in our case, N data are biased and often represent only a small portion of the whole N distribution. Therefore, most of the existing methods designed for the latter cannot be directly applied to the PUbN classification problem. Furthermore, our focus is on deducing a risk estimator using the three sets of data, whereas in semi-supervised learning the main concern is often how U data can be utilized for regularization BID10 BID1 BID20 BID25. The two should be compatible and we believe adding such regularization to our algorithm can be beneficial in many cases. Relation with Dataset Shift PUbN learning can also be viewed as a special case of dataset shift 1 BID31 ) if we consider that P and bN data are drawn from the training distribution while U data are drawn from the test distribution. Covariate shift BID38 BID39 ) is another special case of dataset shift that has been studied intensively. In the covariate shift problem setting, training and test distributions have the same class conditional distribution and only differ in the marginal distribution of the independent variable. One popular approach to tackle this problem is to reweight each training example according to the ratio of the test density to the training density BID15. Nevertheless, simply training a classifier on a reweighted version of the labeled set is not sufficient in our case since there may be examples with zero probability to be labeled. It is also important to notice that the problem of PUbN learning is intrinsically different from that of covariate shift and neither of the two is a special case of the other. In this section, we briefly review the formulations of PN, PU and PNU classification and introduce the problem of learning from P, U and bN data. Let x ∈ R d and y ∈ {+1, −1} be random variables following an unknown probability distribution with density p(x, y). Let g: R d → R be an arbitrary decision function for binary classification and ℓ: R → R + be a loss function of margin yg(x) that usually takes a small value for a large margin. The goal of binary classification is to find g that minimizes the classification risk: DISPLAYFORM0 where E (x,y)∼p (x,y) [·] denotes the expectation over the joint distribution p(x, y). When we care about classification accuracy, ℓ is the zero-one loss ℓ 01 (z) = (1 − sign(z))/2. However, for ease of optimization, ℓ 01 is often substituted with a surrogate loss such as the sigmoid loss ℓ sig (z) = 1/(1 + exp(z)) or the logistic loss ℓ log (z) = ln(1 + exp(−z)) during learning. In standard supervised learning scenarios (PN classification), we are given P and N data that are sampled independently from p(x | y = +1) and p(x | y = −1) as X P = {x [ℓ(−g(x) )] partial risks and π = p(y = 1) the P prior. We have the equality R(g) = πR DISPLAYFORM1 DISPLAYFORM2 The classification risk can then be empirically approximated from data bŷ DISPLAYFORM3. By minimizingR PN (g) we obtain the ordinary empirical risk minimizerĝ PN. In PU classification, instead of N data X N we have only access to X U = {x DISPLAYFORM0 a set of U samples drawn from the marginal density p(x). Several effective algorithms have been designed to address this problem. BID23 proposed the S-EM approach that first identifies reliable N data in the U set and then runs the Expectation-Maximization (EM) algorithm to build the final classifier. The biased support vector machine (Biased SVM) introduced in BID24 regards U samples as N samples with smaller weights. BID27 solved the PU problem by aggregating classifiers trained to discriminate P data from a small random subsample of U data. More recently, attention has been paid on the unbiased risk estimator proposed in du BID4 and du BID3. The key idea is to use the following equality: DISPLAYFORM1. As a , we can approximate the classification risk bŷ DISPLAYFORM2 DISPLAYFORM3 We then minimizê R PU (g) to obtain another empirical risk minimizerĝ PU. Note that as the loss is always positive, the classification risk thatR PU (g) approximates is also positive. However, BID19 pointed out that when the model of g is too flexible, that is, when the function class G is too large,R PU (ĝ PU) indeed goes negative and the model seriously overfits the training data. To alleviate overfitting, the authors observed that R DISPLAYFORM4 and proposed the non-negative risk estimator for PU learning: DISPLAYFORM5 In terms of implementation, stochastic optimization was used and when r =R − U (g) − πR − P (g) becomes negative for a mini-batch, they performed a step of gradient ascent along ∇r to make the mini-batch less overfitted. In semi-supervised learning (PNU classification), P, N and U data are all available. An abundance of works have been dedicated to solving this problem. Here we in particular introduce the PNU risk estimator proposed in BID35. By directly leveraging U data for risk estimation, it is the most comparable to our method. The PNU risk is simply defined as a linear combination of PN and PU/NU risks. Let us just consider the case where PN and PU risks are combined, then for some γ ∈, the PNU risk estimator is expressed aŝ DISPLAYFORM0 We can again consider the non-negative correction by forcing the term DISPLAYFORM1 to be non-negative. In the rest of the paper, we refer to the ing algorithm as non-negative PNU (nnPNU) learning (see Appendix D.4 for an alternative definition of nnPNU and the corresponding ). In this paper, we study the problem of PUbN learning. It differs from usual semi-supervised learning in the fact that labeled N data are not fully representative of the underlying N distribution p(x | y = −1). To take this point into account, we introduce a latent random variable s and consider the joint distribution p (x, y, s) DISPLAYFORM0. Both π and ρ are assumed known throughout the paper. In practice they often need to be estimated from data BID17 BID32. In place of ordinary N data we collect a set of bN samples DISPLAYFORM1 The goal remains the same: we would like to minimize the classification risk. In this section, we propose a risk estimator for PUbN classification and establish an estimation error bound for the proposed method. Finally we show how our method can be applied to PU learning as a special case when no bN data are available. DISPLAYFORM0 The first two terms on the right-hand side of the equation can be approximated directly from data by writingR DISPLAYFORM1 We therefore focus on the third termR DISPLAYFORM2 Our approach is mainly based on the following theorem. We relegate all proofs to the appendix. DISPLAYFORM3 In the theorem,R − s=−1 (g) is decomposed into three terms, and when the expectation is substituted with the average over training samples, these three terms are approximated respectively using data from X U, X P and X bN. The choice of h and η is thus very crucial because it determines what each of the three terms tries to capture in practice. Ideally, we would like h to be an approximation of σ. Then, for x such that h(x) is close to 1, σ(x) is close to 1, so the last two terms on the righthand side of the equation can be reasonably evaluated using X P and X bN (i.e., samples drawn from p(x | s = +1)). On the contrary, if h(x) is small, σ(x) is small and such samples can be hardly found in X P or X bN. Consequently the first term appeared in the decomposition is approximated with the help of X U. Finally, in the empirical risk minimization paradigm, η becomes a hyperparameter that controls how important U data is against P and bN data when we evaluateR − s=−1 (g). The larger η is, the more attention we would pay to U data. One may be curious about why we do not simply approximate the whole risk using only U samples, that is, set η to 1. There are two main reasons. On one hand, if we have a very small U set, which means n U ≪ n P and n U ≪ n bN, approximating a part of the risk with labeled samples should help us reduce the estimation error. This may seem unrealistic but sometimes unbiased U samples can also be difficult to collect BID16. On the other hand, more importantly, we have empirically observed that when the model of g is highly flexible, even a sample regarded as N with small weight gets classified as N in the latter stage of training and performance of the ing classifier can thus be severely degraded. Introducing η alleviates this problem by avoiding treating all U data as N samples. As σ is not available in reality, we propose to replace σ by its estimateσ in. We further substitute h with the same estimate and obtain the following expression: DISPLAYFORM4 ].We notice thatR s=−1,η,σ depends both on η andσ. It can be directly approximated from data bŷ DISPLAYFORM5.We are now able to derive the empirical version of Equation FORMULA14 aŝ DISPLAYFORM6 Estimating σ If we regard s as a class label, the problem of estimating σ is then equivalent to training a probabilistic classifier separating the classes with s = +1 and s = −1. Observing that DISPLAYFORM7, it is straightforward to apply nnPU learning with availability of X P, X bN and X U to DISPLAYFORM8 In other words, here we regard X P and X bN as P and X U as U, and attempt to solve a PU learning problem by applying nnPU. Since we are interested in the classposterior probabilities, we minimize the risk with respect to the logistic loss and apply the sigmoid function to the output of the model to getσ(x). However, the above risk estimator accepts any reasonableσ and we are not limited to using nnPU for computingσ. For example, the least-squares fitting approach proposed in BID18 for direct density ratio estimation can also be adapted to solving the problem. Here we establish an estimation error bound for the proposed method. Let G be the function class from which we find a function. The Rademacher complexity of G for the samples of size n drawn from q(x) is defined as DISPLAYFORM0 where X = {x 1, . . ., x n} and θ = {θ 1, . . ., θ n} with each x i drawn from q(x) and θ i as a Rademacher variable BID26. In the following we will assume that R n,q (G) vanishes asymptotically as n → ∞. This holds for most of the common choices of G if proper regularization is considered BID0 BID9. Assume additionally the exis- DISPLAYFORM1 We also assume that ℓ is Lipschitz continuous on the interval DISPLAYFORM2 Theorem 2. Let g * = arg min g∈G R(g) be the true risk minimizer andĝ PUbN,η,σ = arg min g∈GRPUbN,η,σ (g) be the PUbN empirical risk minimizer. We suppose thatσ is a fixed function independent of data used to computeR PUbN,η,σ DISPLAYFORM3. Then for any δ > 0, with probability at least 1 − δ, DISPLAYFORM4 Theorem 2 shows that as DISPLAYFORM5 where O p denotes the order in probability. As for ϵ, knowing thatσ is also estimated from data in practice 3, apparently its value depends on both the estimation algorithm and the number of samples that are involved in the estimation process. For example, in our approach we applied nnPU with the logistic loss to obtainσ, so the excess risk can be written as E x∼p(x) KL(σ(x)∥σ(x)), where by abuse of notation KL(p∥q) = p ln(p/q)+(1−p) ln((1−p)/(1−q)) denotes the KL divergence between two Bernouilli distributions with parameters respectively p and q. It is known that BID44. The excess risk itself can be decomposed into the sum of the estimation error and the approximation error. BID19 showed that under mild assumptions the estimation error part converges to zero when the sample size increases to infinity in nnPU learning. It is however impossible to get rid of the approximation error part which is fixed 2 For instance, this holds for linear-in-parameter model class DISPLAYFORM6 DISPLAYFORM7 where Cw and C ϕ are positive constants BID26.3 These data, according to theorem 2, must be different from those used to evaluateR PUbN,η,σ (g). This condition is however violated in most of our experiments. See Appendix D.3 for more discussion.once we fix the function class G. To circumvent this problem, we can either resort to kernel-based methods with universal kernels BID44 or simply enlarge the function class when we get more samples. In PU learning scenarios, we only have P and U data and bN data are not available. Nevertheless, if we let y play the role of s and ignore all the terms related to bN data, our algorithm is naturally applicable to PU learning. Let us name the ing algorithm PUbN\N, then DISPLAYFORM0 whereσ is an estimate of p(y = +1 | x) and DISPLAYFORM1 ].PUbN\N can be viewed as a variant of the traditional two-step approach in PU learning which first identifies possible N data in U data and then perform ordinary PN classification to distinguish P data from the identified N data. However, being based on state-of-the-art nnPU learning, our method is more promising than other similar algorithms. Moreover, by explicitly considering the posterior p(y = +1 | x), we attempt to correct the bias induced by the fact of only taking into account confident negative samples. The benefit of using an unbiased risk estimator is that the ing algorithm is always statistically consistent, i.e., the estimation error converges in probability to zero as the number of samples grows to infinity. In this section, we experimentally investigate the proposed method and compare its performance against several baseline methods. We focus on training neural networks with stochastic optimization. For simplicity, in an experiment, σ and g always use the same model and are trained for the same number of epochs. All models are learned using AMSGrad BID33 as the optimizer and the logistic loss as the surrogate loss unless otherwise specified. To determine the value of η, we introduce another hyperparameter τ and choose η such that #{x ∈ X U |σ(x) ≤ η} = τ (1 − π − ρ)n U. In all the experiments, an additional validation set, equally composed of P, U and bN data, is sampled for both hyperparameter tuning and choosing the model parameters with the lowest validation loss among those obtained after every epoch. Regarding the computation of the validation loss, we use the PU risk estimator with the sigmoid loss for g and an empirical approximation of DISPLAYFORM0 We assess the performance of the proposed method on three benchmark datasets: MNIST, CIFAR-10 and 20 Newsgroups. Experimental details are given in Appendix C. In particular, since all the three datasets are originally designed for multiclass classification, we group different categories together to form a binary classification problem. Baselines. When X bN is given, two baseline methods are considered. The first one is nnPNU adapted from. In the second method, named as PU→PN, we train two binary classifiers: one is learned with nnPU while we regard s as the class label, and the other is learned from X P and X bN to separate P samples from bN samples. A sample is classified in the P class only if it is so classified by the two classifiers. When X bN is not available, nnPU is compared with the proposed PUbN\N.Sampling bN Data To sample X bN, we suppose that the bias of N data is caused by a latent prior probability change BID40 BID14 in the N class. Let z ∈ Z:= DISPLAYFORM0 In the experiments, the latent categories are the original class labels of the datasets. Concrete definitions of X bN with experimental are summarized in TAB0.Results. Overall, our proposed method consistently achieves the best or comparable performance in all the scenarios, including those of standard PU learning. Additionally, using bN data can effectively help improving classification performance. However, the choice of algorithm is essential. Both nnPNU and the naive PU→PN are able to leverage bN data to enhance classification accuracy in only relatively few tasks. In the contrast, the proposed PUbN successfully reduce the misclassification error most of the time. Clearly, the performance gain that we can benefit from the availability of bN data is case-dependent. On CIFAR-10, the greatest improvement is achieved when we regard mammals (i.e. cat, deer, dog and horse) as P class and drawn samples from latent categories bird and frog as labeled negative data. This is not surprising because birds and frogs are more similar to mammals than vehicles, which makes the classification harder specifically for samples from these two latent categories. By explicitly labeling these samples as N data, we allow the classifier to make better predictions for these difficult samples. Through experiments we have demonstrated that the presence of bN data effectively helps learning a better classifier. Here we would like to provide some intuition for the reason behind this. Let us consider the MNIST learning task where X bN is uniformly sampled from the latent categories 1, 3 and 5. We project the representations learned by the classifier (i.e., the activation values of the last layer of the neural network) into a 2D plane using PCA for both nnPU and PUbN algorithms. The are shown in FIG0. Since for both nnPU and PUbN classifiers, the first two principal components account around 90% of variance, we believe that this figure depicts fairly well the learned representations. Thanks to the use of bN data, in the high-level feature space 1, 3, 5 and P data are further pushed away when we employ the proposed PUbN learning algorithm, and we are always able to separate 7, 9 from P to some extent. This explains the better performance which is achieved by PUbN learning and the benefit of incorporating bN data into the learning process. This paper studied the PUbN classification problem, where a binary classifier is trained on P, U and bN data. The proposed method is a two-step approach inspired from both PU learning and importance weighting. The key idea is to attribute appropriate weights to each example to evaluate the classification risk using the three sets of data. We theoretically established an estimation error bound for the proposed risk estimator and experimentally showed that our approach successfully leveraged bN data to improve the classification performance on several real-world datasets. A variant of our algorithm was able to achieve state-of-the-art in PU learning. DISPLAYFORM0 We obtain Equation after replacing p(x, s = +1) by πp(x | y = +1)+ρp(x | y = −1, s = +1). Forσ and η given, let us define DISPLAYFORM0 The following lemma establishes the uniform deviation bound fromR PUbN,η,σ to R PUbN,η,σ. DISPLAYFORM1 ] be a fixed function independent of data used to computeR PUbN,η,σ and η ∈. For any δ > 0, with probability at least 1 − δ, DISPLAYFORM2 Proof. For ease of notation, let DISPLAYFORM3 ].From the sub-additivity of the supremum operator, we have DISPLAYFORM4 As a consequence, to conclude the proof, it suffices to prove that with probability at least 1 − δ/3, the following bounds hold separately: DISPLAYFORM5 DISPLAYFORM6 DISPLAYFORM7 Below we prove. FORMULA43 and FORMULA0 are proven similarly. Let ϕ x: R → R + be the function defined by DISPLAYFORM8 Following the proof of Theorem 3.1 in BID26, it is then straightforward to show that with probability at least 1 − δ/3, it holds that DISPLAYFORM9 where θ = {θ 1, . . ., θ n P} and each θ i is a Rademacher variable. Also notice that for all x, ϕ x is a (L l /η)-Lipschitz function on the interval [−C g, C g]. By using a modified version of Talagrad's concentration lemma (specifically, Lemma 26.9 in), we can show that, when the set X P is fixed, we have DISPLAYFORM10 After taking expectation over X P ∼ p np P, we obtain the Equation FORMULA42.However, what we really want to minimize is the true risk R(g). Therefore, we also need to bound the difference between R PUbN,η,σ (g) and R(g), or equivalently, the difference between DISPLAYFORM11 Proof. One one hand, we havē DISPLAYFORM12 On the other hand, we can expressR DISPLAYFORM13 The last equality follows from the fact p(DISPLAYFORM14 From the second to the third line we use the Cauchy-Schwarz inequality. |A 1 − A 2 | ≤ C l √ ζϵ can be proven similarly, which concludes the proof. Combining lemma 1 and lemma 2, we know that with probability at least 1 − δ, the following holds: DISPLAYFORM15 Finally, with probability at least 1 − δ, DISPLAYFORM16 The first inequality uses the definition ofĝ PUbN,η,σ . In terms of validation we want to choose the model forσ such that DISPLAYFORM0 The last term does not depend onσ and can be ignored if we want to identifyσ achieving the smallest J(σ). We denote by J(σ) the sum of the first two terms. The middle term can be further expanded using DISPLAYFORM1 The validation loss of an estimationσ is then defined aŝ DISPLAYFORM2 It is also possible to minimize this value directly to acquireσ. In our experiments we decide to learn σ by nnPU for a better comparison between different methods. In the experiments we work on multiclass classification datasets. Therefore it is necessary to define the P and N classes ourselves. MNIST is processed in such a way that pair numbers 0, 2, 4, 6, 8 form the P class and impair numbers 1, 3, 5, 7, 9 form the N class. Accordingly, π = 0.49. For CIFAR-10, we consider two definitions of the P class. The first one corresponds to a quite natural task that aims to distinguish vehicles from animals. Airplane, automobile, ship and truck are therefore defined to be the P class while the N class is formed by bird, cat, deer, dog, frog and horse. For the sake of diversity, we also study another task in which we attempt to distinguish the mammals from the non-mammals. The P class is then formed by cat, deer, dog, and horse while the N class consists of the other six classes. We have π = 0.4 in the two cases. As for 20 Newsgroups, alt., comp., misc. and rec. make up the P class whereas sci., soc. and talk. make up the N class. This gives π = 0.56. For the three datasets, we use the standard test examples as a held-out test set. The test set size is thus of 10000 for MNIST and CIFAR-10, and 7528 for 20 Newsgroups. Regarding the training set, we sample 500, 500 and 6000 P, bN and U training examples for MNIST and 20 Newsgroups, and 1000, 1000 and 10000 P, bN and U training examples for CIFAR-10. The validation set is always five times smaller than the training set. The original 20 Newsgroups dataset contains raw text data and needs to be preprocessed into text feature vectors for classification. In our experiments we borrow the pre-trained ELMo word embedding BID30 from https://allennlp.org/elmo. The used 5.5B model was, according to the website, trained on a dataset of 5.5B tokens consisting of Wikipedia (1.9B) and all of the monolingual news crawl data from WMT 2008 WMT -2012. For each word, we concatenate the features from the three layers of the ELMo model, and for each document, as suggested in BID34, we concatenate the average, minimum, and maximum computed along the word dimension. This in a 9216-dimensional feature vector for a single document. MNIST For MNIST, we use a standard ConvNet with ReLU. This model contains two 5x5 convolutional layers and one fully-connected layer, with each convolutional layer followed by a 2x2 max pooling. The channel sizes are 5-10-40. The model is trained for 100 epochs with a weight decay of 10 −4. Each minibatch is made up of 10 P, 10 bN (if available) and 120 U samples. The learning rate α ∈ {10 −2, 10 −3} and τ ∈ {0. 5, 0.7, 0.9}, γ ∈ {0.1, 0.3, 0.5, 0.7, 0 .9} are selected with validation data. CIFAR-10 For CIFAR-10, we train PreAct ResNet-18 BID11 for 200 epochs and the learning rate is divided by 10 after 80 epochs and 120 epochs. This is a common practice and similar adjustment can be found in BID11. The weight decay is set to 10 −4. The minibatch size is 1/100 of the number of training samples, and the initial learning rate is chosen from {10 −2, 10 −3}. We also have τ ∈ {0. 5, 0.7, 0.9} and γ ∈ {0.1, 0.3, 0.5, 0.7, 0.9}. 20 Newsgroups For 20 Newsgroups, with the extracted features, we simply train a multilayer perceptron with two hidden layers of 300 neurons for 50 epochs. We use basically the same hyperparameters as for MNIST except that the learning rate α is selected from {5 · 10 Our method, specifically designed for PUbN learning, naturally outperforms other baseline methods in this problem. Nonetheless, Table 1 equally shows that the proposed method when applied to PU learning, achieves significantly better performance than the state-of-the-art nnPU algorithm. Here we numerically investigate the reason behind this phenomenon. Besides nnPU and PUbN\N, we compare with unbiased PU (uPU) learning. Both uPU and nnPU are learned with the sigmoid loss, learning rate 10 −3 for MNIST, initial learning rate 10 −4 for CIFAR-10, and learning rate 10 −4 for 20 Newsgroups. This is because uPU learning is unstable with the logistic loss. The other parts of the experiments remain unchanged. On the test sets we compute the false positive rates, false negative rates and misclassification errors for the three methods and plot them in FIG3. We first notice that PUbN\N still outperforms nnPU trained with the sigmoid loss. In fact, the final performance of the nnPU classifier does not change much when we replace the logistic loss with the sigmoid loss. In BID19, the authors observed that uPU overfits training data with the risk going to negative. In other words, a large portion of U samples are classified to the N class. This is confirmed in our experiments by an increase of false negative rate and decrease of false positive rate. nnPU remedies the problem by introducing the non-negative risk estimator. While the non-negative correction successfully prevents false negative rate from going up, it also causes more N samples to be classified as P compared to uPU. However, since the gain in terms of false negative rate is enormous, at the end nnPU achieves a lower misclassification error. By further identifying possible N samples after nnPU learning, we expect that our algorithm can yield lower false positive rate than nnPU without misclassifying too many P samples as N as in the case of uPU. FIG3 suggests that this is effectively the case. In particular, we observe that on MNIST, our method achieves the same false positive rate than uPU whereas its false negative rate is comparable to nnPU. In the proposed algorithm we introduce η to control howR s=−1 (g) is approximated from data and assume that ρ = p(y = −1, s = +1) is given. Here we conduct experiments to see how our method is affected by these two factors. To assess the influence of η, from TAB0 we pick four learning tasks and we choose τ from {0.5, 0.7, 0.9, 2} while all the other hyperparameters are fixed. Similarly to simulate the case where ρ is misspecified, we replace it by ρ ′ ∈ {0.8ρ, ρ, 1.2ρ} in our learning method and run experiments with all hyperparameters being fixed to a certain value. However, we still use the true ρ to compute η from τ to ensure that we always use the same number of U samples in the second step of the algorithm independent of the choice of ρ ′.The are reported in TAB1 and TAB2. We can see that the performance of the algorithm is sensitive to the choice of τ. With larger value of τ, more U data are treated as N data in PUbN learning, and consequently it often leads to higher false negative rate and lower false positive rate. The trade-off between these two measures is a classic problem in binary classification. In particular, when τ = 2, a lot more U samples are involved in the computation of the PUbN risk, but this does not allow the classifier to achieve a better performance. We also observe that there is a positive correlation between the misclassification rate and the validation loss, which confirms that the optimal value of η can be chosen without need of unbiased N data. TAB2 shows that in general slight misspecification of ρ does not cause obvious degradation of the classification performance. In fact, misspecification of ρ mainly affect the weights of each sample when we computeR PUbN,η,σ (due to the direct presence of ρ in and influence on estimating σ). However, as long as the variation of these weights remain in a reasonable range, the learning algorithm should yield classifiers with similar performances. DISPLAYFORM0 Theorem 2 suggests thatσ should be independent from the data used to computeR PUbN,η,σ. Therefore, here we investigate the performance of our algorithm whenσ and g are optimized using different sets of data. We sample two training sets and two validation sets in such a way that they are all disjoint. The size of a single training set and a single validation set is as indicated in Appendix C.2, except for 20 Newsgroups we reduce the number of examples in a single set by half. We then use different pairs of training and validation sets to learnσ and g. For 20 Newsgroups we also conduct standard experiments whereσ and g are learned on the same data, whereas for MNIST and CIFAR-10 we resort to TAB0.The are presented in TAB3. Estimating σ from separate data does not seem to benefit much the final classification performance, despite the fact that it requires collecting twice more samples. In fact,R − s=−1,η,σ (g) is a good approximation ofR − s=−1,η,σ (g) as long as the functionσ is smooth enough and does not possess abrupt changes between data points. With the use of non-negative correction, validation data and L2 regularization, the ingσ does not overfit training data so this should always be the case. As a consequence, even ifσ and g are learned on the same data, we are still able to achieve small generalization error with sufficient number of samples. In subsection 2.3, we define the nnPNU algorithm by forcing the estimator of the whole N partial risk to be positive. However, notice that the term γ(1 − π)R − N (g) is always positive and the chances are that including it simply makes non-negative correction weaker and is thus harmful to the final classification performance. Therefore, here we consider an alternative definition of nnPNU where we only force the term (1 − γ)(R − U (g) − πR − P (g)) to be positive. We plug the ing algorithm in the experiments of subsection 4.2 and summarize the in TAB4 in which we denote the alternative version of nnPNU by nnPU+PN since it uses the same non-negative correction as nnPU. The table indicates that neither of the two definitions of nnPNU consistently outperforms the other. It also ensures that there is always a clear superiority of our proposed PUbN algorithm compared to nnPNU despite its possible variant that is considered here. 17.14 ± 1.87 15.80 ± 0.95 soc. > talk. > sci.15.93 ± 1.88 15.80 ± 1.91 sci. 14.69 ± 0.46 14.50 ± 1.32 talk.14.38 ± 0.74 14.71 ± 1.01 soc. > talk. > sci.14.41 ± 0.70 13.66 ± 0.72

This paper studied the PUbN classification problem, where we incorporate biased negative (bN) data, i.e., negative data that is not fully representative of the true underlying negative distribution, into positive-unlabeled (PU) learning.

The paper designs two algorithms for the stochastic AUC maximization problem with state-of-the-art complexities when using deep neural network as predictive model, which are also verified by empirical studies.


Question: Represent the text to find a very short summary (1-2 sentences max)!

Question: Reinforcement learning (RL) is frequently used to increase performance in text generation tasks, including machine translation (MT), notably through the use of Minimum Risk Training (MRT) and Generative Adversarial Networks (GAN). However, little is known about what and how these methods learn in the context of MT. We prove that one of the most common RL methods for MT does not optimize the expected reward, as well as show that other methods take an infeasibly long time to converge. In fact, our suggest that RL practices in MT are likely to improve performance only where the pre-trained parameters are already close to yielding the correct translation. Our findings further suggest that observed gains may be due to effects unrelated to the training signal, concretely, changes in the shape of the distribution curve. Reinforcement learning (RL) is an appealing path for advancement in Machine Translation (MT), as it allows training systems to optimize non-differentiable score functions, common in MT evaluation, as well as tackling the "exposure bias" in standard training, namely that the model is not exposed during training to incorrectly generated tokens, and is thus unlikely to recover from generating such tokens at test time. These motivations have led to much interest in RL for text generation in general and MT in particular (see §2). Various policy gradient methods have been used, notably REINFORCE and variants thereof (e.g., ;) and Minimum Risk Training (MRT; e.g., ; . Another popular use of RL is for training GANs . Nevertheless, despite increasing interest and strong , little is known about what accounts for these performance gains, and the training dynamics involved. We present the following contributions. First, our theoretical analysis shows that commonly used approximation methods are theoretically ill-founded, and may converge to parameter values that do not minimize the risk, nor are local minima thereof (§2.2). Second, using both naturalistic experiments and carefully constructed simulations, we show that performance gains observed in the literature likely stem not from making target tokens the most probable, but from unrelated effects, such as increasing the peakiness of the output distribution (i.e., the probability mass of the most probable tokens). We do so by comparing a setting where the reward is informative, vs. one where it is constant. In §4 we discuss this peakiness effect (PKE). Third, we show that promoting the target token to be the mode is likely to take a prohibitively long time. The only case we find, where improvements are likely, is where the target token is among the first 2-3 most probable tokens according to the pretrained model. These findings suggest that REINFORCE (§5) and CMRT (§6) are likely to improve over the pre-trained model only under the best possible conditions, i.e., where the pre-trained model is "nearly" correct. We conclude by discussing other RL practices in MT which should be avoided for practical and theoretical reasons, and briefly discuss alternative RL approaches that will allow RL to tackle a larger class of errors in pre-trained models (§7). An MT system generates tokens y = (y 1, ..., y n) from a vocabulary V one token at a time. The probability of generating y i given preceding tokens y <i is given by P θ (·|x, y <i), where x is the source sentence and θ are the model parameters. For each generated token y i, we denote with r(y i ; y <i, x, y (ref) ) the score, or reward, for generating y i given y <i, x, and the reference sentence y (ref). For brevity, we omit parameters where they are fixed within context. For simplicity, we assume r does not depend on following tokens y >i. We also assume there is exactly one valid target token, as de facto, training is done against a single reference . In practice, either a token-level reward is approximated using MonteCarlo methods (e.g.,), or a sentence-level (sparse) reward is given at the end of the episode (sentence). The latter is equivalent to a uniform token-level reward. r is often the negative log-likelihood, or a standard MT metric, e.g., BLEU . RL's goal is to maximize the expected episode reward (denoted with R); i.e., to find 2.1 REINFORCE For a given source sentence, and past predictions y <i, REINFORCE samples k tokens (k is a hyperparameter) S = y,..., y (k) from P θ and updates θ according to this rule: The right-hand side of equation 2 is an unbiased estimator of the gradient of the objective function, i.e., E [∆θ] ∝ ∇ θ R (θ). Therefore, REINFORCE is performing a form of stochastic gradient ascent on R, and has similar formal guarantees. From here follows that if R is constant with respect to θ, then the expected ∆θ prescribed by REINFORCE is zero. We note that r may be shifted by a constant term (called a "baseline"), without affecting the optimal value for θ. REINFORCE is used in MT, text generation, and image-to-text tasks;;; ) -in isolation, or as a part of training . Lately, an especially prominent use for REINFORCE is adversarial training with discrete data, where another network predicts the reward (GAN). For some recent work on RL for NMT, see (; ; ; ;). The term Minimum Risk Training (MRT) is used ambiguously in MT to refer either to the application of REINFORCE to minimizing the risk (equivalently, to maximizing the expected reward, the negative loss), or more commonly to a somewhat different estimation method, which we term Contrastive MRT (CMRT) and turn now to analyzing. CMRT was proposed by , adapted to NMT by, and often used since (; ; ; ;). The method works as follows: at each iteration, sample k tokens S = {y 1, . . ., y k} from P θ, and update θ according to the gradient of where Q θ,S (y i) = P (y i) α yj ∈S P (y j) α Commonly (but not universally), deduplication is performed, so R sums over a set of unique values . This changes little in our empirical and theoretical analysis. Despite the resemblance in definitions of R (equation 1) and R (indeed, R is sometimes presented as an approximation of R), they differ in two important aspects. First, Q's support is S, so increasing Q(y i) for some y i necessarily comes at the expense of Q(y) for some y ∈ S. In contrast, increasing P (y i), as in REINFORCE, may come at the expense of P (y) for any y ∈ V. Second, α is a smoothness parameter: the closer α is to 0, the closer Q is to be uniform. We show in Appendix A.1 that despite its name, CMRT does not optimize R, nor does it optimize E[R]. That is, it may well converge to values that are not local maxima of R, making it theoretically ill-founded. 1 However, given CMRT popularity, the strong it yielded and the absence of theory for explaining it, we discuss it here. Given a sample S, the gradient of R is given by where Comparing Equations 2 and 3, the differences between REINFORCE and CMRT are reflected again. First, ∇ R has an additional term, proportional to ∇ log Z(S), which yields the contrastive effect. This contrast may improve the rate of convergence since it counters the decrease of probability mass for non-sampled tokens. Second, given S, the relative weighting of the gradients ∇ log P (y i) is proportional to r(y i)Q(y i), or equivalently to r(y i)P (y i) α. CMRT with deduplication sums over distinct values in S (equation 3), while REINFORCE sums over all values. This means that the relative weight of the unique value y i is r(yi)|{yi∈S}| k in REINFORCE. For α = 1 the expected value of these relative weights is the same, and so for α < 1 (as is commonly used), more weight is given to improbable tokens, which could also have a positive effect on the convergence rate. 2 However, if α is too close to 0, ∇ R vanishes, as it is not affected by θ. This tradeoff explains the importance of tuning α reported in the literature. In §6 we present simulations with CMRT, showing very similar trends as presented by REINFORCE. Implementing a stochastic gradient ascent, REINFORCE is guaranteed to converge to a stationary point of R under broad conditions. However, not much is known about its convergence rate under the prevailing conditions in NMT. We begin with a qualitative, motivating analysis of these questions. As work on language generation empirically showed, RNNs quickly learn to output very peaky distributions . This tendency is advantageous for generating fluent sentences with high probability, but may also entail slower convergence rates when using RL to fine-tune the model, because RL methods used in text generation sample from the (pretrained) policy distribution, which means they mostly sample what the pretrained model deems to be likely. Since the pretrained model (or policy) is peaky, exploration of other potentially more rewarding tokens will be limited, hampering convergence. Intuitively, REINFORCE increases the probabilities of successful (positively rewarding) observations, weighing updates by how rewarding they were. When sampling a handful of tokens in each context (source sentence x and generated prefix y <i), and where the number of epochs is not large, it is unlikely that more than a few unique tokens will be sampled from P θ (·|x, y <i). (In practice, k is typically between 1 and 20, and the number of epochs between 1 and 100.) It is thus unlikely that anything but the initially most probable candidates will be observed. Consequently, REINFORCE initially raises their probabilities, even if more rewarding tokens can be found down the list. We thus hypothesize the peakiness of the distribution, i.e., the probability mass allocated to the most probable tokens, will increase, at least in the first phase. We call this the peakiness-effect (PKE), and show it occurs both in simulations (§4.1) and in full-scale NMT experiments (§4.2). With more iterations, the most-rewarding tokens will be eventually sampled, and gradually gain probability mass. This discussion suggests that training will be extremely sample-inefficient. We assess the rate of convergence empirically in §5, finding this to be indeed the case. A histogram of the update size (x-axis) to the total predicted probability of the 10 most probable tokens (left) or the most probable token (right) in the Constant Reward setting. An update is overwhelmingly more probable to increase this probability than to decrease it. We turn to demonstrate that the initially most probable tokens will initially gain probability mass, even if they are not the most rewarding, yielding a PKE. recently observed in the context of language modeling using GANs that performance gains similar to those GAN yield can be achieved by decreasing the temperature for the prediction softmax (i.e., making it peakier). However, they proposed no causes for this effect. Our findings propose an underlying mechanism leading to this trend. We return to this point in §7. Furthermore, given their findings, it is reasonable to assume that our are relevant for RL use in other generation tasks, whose output space too is discrete, high-dimensional and concentrated. We experiment with a 1-layer softmax model, that predicts a single token i ∈ V with probability. θ = {θ j} j∈V are the model's parameters. This model simulates the top of any MT decoder that ends with a softmax layer, as essentially all NMT decoders do. To make experiments realistic, we use similar parameters as those reported in the influential Transformer NMT system . Specifically, the size of V (distinct BPE tokens) is 30,715, and the initial values for θ were sampled from 1,000 sets of logits taken from decoding the standard newstest2013 development set, using a pretrained Transformer model. The model was pretrained on WMT2015 training data . Hyperparameters are reported in Appendix A.3. We define one of the tokens in V to be the target token and denote it with y best. We assign deterministic token reward, this makes learning easier than when relying on approximations and our predictions optimistic. We experiment with two reward functions: 1. Simulated Reward: r(y) = 2 for y = y best, r(y) = 1 if y is one of the 10 initially highest scoring tokens, and r(y) = 0 otherwise. This simulates a condition where the pretrained model is of decent but sub-optimal quality. r here is at the scale of popular rewards used in MT, such as GAN-based rewards or BLEU (which are between 0 and 1). 2. Constant Reward: r is constantly equal to 1, for all tokens. This setting is aimed to confirm that PKE is not a of the signal carried by the reward. Experiments with the first setting were run 100 times, each time for 50K steps, updating θ after each step. With the second setting, it is sufficient to take a single step at a time, as the expected update after each step is zero, and so any PKE seen in a single step is only accentuated in the next. It is, therefore, more telling to run more repetitions rather than more steps per initialization. We, therefore, sample 10,000 pretrained distributions, and perform a single REINFORCE step. As RL training lasts about 30 epochs before stopping, samples about 100K tokens per epoch, and as the network already predicts y best in about two thirds of the contexts, 3 we estimate the number of steps used in practice to be in the order of magnitude of 1M. For visual clarity, we present figures for 50K-100K steps. However, full experiments (with 1M steps) exhibit similar trends: where REINFORCE was not close to converging after 50K steps, the same was true after 1M steps. We evaluate the peakiness of a distribution in terms of the probability of the most probable token (the mode), the total probability of the ten most probable tokens, and the entropy of the distribution (lower entropy indicates more peakiness). Results. The distributions become peakier in terms of all three measures: on average, the mode's probability and the 10 most probable tokens increases, and the entropy decreases. Figure 1a presents the histogram of the update size, the difference in the probability of the 10 most probable tokens in the Constant Reward setting, after a single step. Figure 1b depicts similar statistics for the mode. The average entropy in the pretrained model is 2.9 is reduced to 2.85 after one REINFORCE step. Simulated Reward setting shows similar trends. For example, entropy decreases from 3 to about 0.001 in 100K steps. This extreme decrease suggests it is effectively a deterministic policy. PKE is achieved in a few hundred steps, usually before other effects become prominent (see Figure 2), and is stronger than for Constant Reward. Figure 3: The cumulative distribution of the probability of the most likely token in the NMT experiments. The green distribution corresponds to the pretrained model, and the blue corresponds to the reinforced model. The yaxis is the proportion of conditional probabilities with a mode of value ≤ x (the x-axis). Note that a lower cumulative percentage means a more peaked output distribution. A lower cumulative percentage means a more peaked output distribution. We turn to analyzing a real-world application of REINFORCE to NMT. Important differences between this and the previous simulations are: it is rare in NMT for REINFORCE to sample from the same conditional distribution more than a handful of times, given the number of source sentences x and sentence prefixes y <i (contexts); and in NMT P θ (·|x, y <i) shares parameters between contexts, which means that updating P θ for one context may influence P θ for another. We follow the same pretraining as in §4.1. We then follow in defining the reward function based on the expected BLEU score. Expected BLEU is computed by sampling suffixes for the sentence, and averaging the BLEU score of the sampled sentences against the reference. We use early stopping with a patience of 10 epochs, where each epoch consists of 5,000 sentences sampled from the WMT2015 German-English training data. We use k = 1. We retuned the learning-rate, and positive baseline settings against the development set. Other hyper-parameters were an exact replication of the experiments reported in . Results. Results indicate an increase in the peakiness of the conditional distributions. Our are based on a sample of 1,000 contexts from the pretrained model, and another (independent) sample from the reinforced model. The modes of the conditional distributions tend to increase. Figure 3 presents the distribution of the modes' probability in the reinforced conditional distributions compared with the pretrained model, showing a shift of probability mass towards higher probabilities for the mode, following RL. Another indication of the increased peakiness is the decrease in the average entropy of P θ, which was reduced from 3.45 in the pretrained model to an average of 2.82 following RL. This more modest reduction in entropy (compared to §4.1) might also suggest that the procedure did not converge to the optimal value for θ, as then we would have expected the entropy to substantially drop if not to 0 (overfit), then to the average entropy of valid next tokens (given the source and a prefix of the sentence). We now turn to assessing under what conditions it is likely that REINFORCE will lead to an improvement in the performance of an NMT system. As in the previous section, we use both controlled simulations and NMT experiments. We use the same model and experimental setup described in Section 4.1, this time only exploring the Simulated Reward setting, as a Constant Reward is not expected to converge to any meaningful θ. Results are averaged over 100 conditional distributions sampled from the pretrained model. Caution should be exercised when determining the learning rate (LR). Common LRs used in the NMT literature are of the scale of 10 −4. However, in our simulations, no LR smaller than 0.1 yielded any improvement in R. We thus set the LR to be 0.1. We note that in our simulations, a higher learning rate means faster convergence as our reward is noise-free: it is always highest for the best option. In practice, increasing the learning rate may deteriorate , as it may cause the system to overfit to the sampled instances. Indeed, when increasing the learning rate in our NMT experiments (see below) by an order of magnitude, early stopping caused the RL procedure to stop without any parameter updates. Figure 2 shows the change in P θ over the first 50K REINFORCE steps (probabilities are averaged over 100 repetitions), for a case where y best was initially the second, third and fourth most probable. Although these are the easiest settings, and despite the high learning rate, it fails to make y best the mode of the distribution within 100K steps, unless y best was initially the second most probable. In cases where y best is initially of a lower rank than four, it is hard to see any increase in its probability, even after 1M steps. We trained an NMT system, using the same procedure as in Section 4.2, and report BLEU scores over the news2014 test set. After training with an expected BLEU reward, we indeed see a minor improvement which is consistent between trials and pretrained models. While the pretrain BLEU score is 30.31, the reinforced one is 30.73. Analyzing what words were influenced by the RL procedure, we begin by computing the cumulative probability of the target token y best to be ranked lower than a given rank according to the pretrained model. Results (Figure 4) show that in about half of the cases, y best is not among the top three choices of the pretrained model, and we thus expect it not to gain substantial probability following REINFORCE, according to our simulations. We next turn to compare the ranks the reinforced model assigns to the target tokens, and their ranks according to the pretrained model. Figure 6 presents the difference in the probability that y best is ranked at a given rank following RL and the probability it is ranked there initially. Results indicate that indeed more target tokens are ranked first, and less second, but little consistent shift of probability mass occurs otherwise across the ten first ranks. It is possible that RL has managed to push y best in some cases between very low ranks (<1,000) to medium-low ranks (between 10 and 1,000). However, token probabilities in these ranks are so low that it is unlikely to affect the system outputs in any way. This fits well with the of our simulations that predicted that only the initially top-ranked tokens are likely to change. In an attempt to explain the improved BLEU score following RL with PKE, we repeat the NMT experiment this time using a constant reward of 1. Our present a nearly identical improvement in BLEU, achieving 30.72, and a similar pattern in the change of the target tokens' ranks (see Ap- pendix 8). Therefore, there is room to suspect that even in cases where RL yields an improvement in BLEU, it may partially from reward-independent factors, such as PKE. 6 EXPERIMENTS WITH CONTRASTIVE MRT Figure 6: Difference between the ranks of y best in the reinforced and the pretrained model. Each column x corresponds to the difference in the probability that y best is ranked in rank x in the reinforced model, and the same probability in the pretrained model. In §2.2 we showed that CMRT does not, in fact, maximize R, and so does not enjoy the same theoretical guarantees as REINFORCE and similar policy gradient methods. However, being the RL procedure of choice in much recent work we repeat the simulations described in §4 and §5, assessing CMRT's performance in these conditions. We experiment with α = 0.005 and k = 20, common settings in the literature, and average over 100 trials. Figure 5 shows how the distribution P θ changes over the course of 50K update steps to θ, where y best is taken to be the second and third initially most probable token (Simulated Reward setting). Results are similar in trends to those obtained with REINFORCE: MRT succeeds in pushing y best to be the highest ranked token if it was initially second, but struggles where it was initially ranked third or below. We only observe a small PKE in MRT. This is probably due to the contrastive effect, which means that tokens that were not sampled do not lose probability mass. All graphs we present here allow sampling the same token more than once in each batch (i.e., S is a sample with replacements). Simulations with deduplication show similar . In this paper, we showed that the type of distributions used in NMT entail that promoting the target token to be the mode is likely to take a prohibitively long times for existing RL practices, except under the best conditions (where the pretrained model is "nearly" correct). This leads us to conclude that observed improvements from using RL for NMT are likely due either to fine-tuning the most probable tokens in the pretrained model (an effect which may be more easily achieved using reranking methods, and uses but little of the power of RL methods), or to effects unrelated to the signal carried by the reward, such as PKE. Another contribution of this paper is in showing that CMRT does not optimize the expected reward and is thus theoretically unmotivated. A number of reasons lead us to believe that in our NMT experiments, improvements are not due to the reward function, but to artefacts such as PKE. First, reducing a constant baseline from r, so as to make the expected reward zero, disallows learning. This is surprising, as REINFORCE, generally and in our simulations, converges faster where the reward is centered around zero, and so the fact that this procedure here disallows learning hints that other factors are in play. As PKE can be observed even where the reward is constant (if the expected reward is positive; see §4.1), this suggests PKE may play a role here. Second, we observe more peakiness in the reinforced model and in such cases, we expect improvements in BLEU . Third, we achieve similar with a constant reward in our NMT experiments (§5.2). Fourth, our controlled simulations show that asymptotic convergence is not reached in any but the easiest conditions (§5.1). Our analysis further suggests that gradient clipping, sometimes used in NMT , is expected to hinder convergence further. It should be avoided when using REINFORCE as it violates REINFORCE's assumptions. The per-token sampling as done in our experiments is more exploratory than beam search , reducing PKE. Furthermore, the latter does not sample from the behavior policy, but does not properly account for being off-policy in the parameter updates. Adding the reference to the sample S, which some implementations allow may help reduce the problems of never sampling the target tokens. However, as point out, this practice may lower , as it may destabilize training by leading the model to improve over outputs it cannot generalize over, as they are very different from anything the model assigns a high probability to, at the cost of other outputs. The standard MT scenario poses several uncommon challenges for RL. First, the action space in MT problems is a high-dimensional discrete space (generally in the size of the vocabulary of the target language or the product thereof for sentences). This contrasts with the more common scenario studied by contemporary RL methods, which focuses mostly on much smaller discrete action spaces (e.g., video games (; 2016) ), or continuous action spaces of relatively low dimensions (e.g., simulation of robotic control tasks ). Second, reward for MT is naturally very sparse -almost all possible sentences are "wrong" (hence, not rewarding) in a given context. Finally, it is common in MT to use RL for tuning a pretrained model. Using a pretrained model ameliorates the last problem. But then, these pretrained models are in general quite peaky, and because training is done on-policy -that is, actions are being sampled from the same model being optimized -exploration is inherently limited. Here we argued that, taken together, these challenges in significant weaknesses for current RL practices for NMT, that may ultimately prevent them from being truly useful. At least some of these challenges have been widely studied in the RL literature, with numerous techniques developed to address them, but were not yet adopted in NLP. We turn to discuss some of them. Off-policy methods, in which observations are sampled from a different policy than the one being currently optimized, are prominent in RL , and were also studied in the context of policy gradient methods . In principle, such methods allow learning from a more "exploratory" policy. Moreover, a key motivation for using α in CMRT is smoothing; off-policy sampling allows smoothing while keeping convergence guarantees. In its basic form, exploration in REINFORCE relies on stochasticity in the action-selection (in MT, this is due to sampling). More sophisticated exploration methods have been extensively studied, for example using measures for the exploratory usefulness of states or actions , or relying on parameter-space noise rather than action-space noise . For MT, an additional challenge is that even effective exploration (sampling diverse sets of observations), may not be enough, since the state-action space is too large to be effectively covered, with almost all sentences being not rewarding. Recently, diversity-based and multi-goal methods for RL were proposed to tackle similar challenges (; ;). We believe the adoption of such methods is a promising path forward for the application of RL in NLP. Let θ be a real number in [0, 0.5], and let P θ be a family of distributions over three values a, b, c such that: Let r(a) = 1, r(b) = 0, r(c) = 0.5. The expected reward as a function of θ is: is uniquely maximized by θ * = 0.25. Table 1 details the possible samples of size k = 2, their probabilities, the corresponding R and its gradient. Standard numerical methods show that E[∇ R] over possible samples S is positive for θ ∈ (0, γ) and negative for θ ∈ (γ, 0.5], where γ ≈ 0.295. This means that for any initialization of θ ∈ (0, 0.5], Contrastive MRT will converge to γ if the learning rate is sufficiently small. For θ = 0, R ≡ 0.5, and there will be no gradient updates, so the method will converge to θ = 0. Neither of these values maximizes R(θ). We note that by using some g (θ) the γ could be arbitrarily far from θ *. g could also map to (−inf, inf) more often used in neural networks parameters. We further note that resorting to maximizing E[R] instead, does not maximize R(θ) either. Indeed, plotting E[R] as a function of θ for this example, yields a maximum at θ ≈ 0.32. (1-θ-2θ 2) 2 0.5 0 True casing and tokenization were used , including escaping html symbols and "-" that represents a compound was changed into a separate token of =. Some preprocessing used before us converted the latter to ##AT##-##AT## but standard tokenizers in use process that into 11 different tokens, which over-represents the significance of that character when BLEU is calculated. BPE extracted 30,715 tokens. For the MT experiments we used 6 layers in the encoder and the decoder. The size of the embeddings was 512. Gradient clipping was used with size of 5 for pre-training (see Discussion on why not to use it in training). We did not use attention dropout, but 0.1 residual dropout rate was used. In pretraining and training sentences of more than 50 tokens were discarded. Pretraining and training were considered finished when BLEU did not increase in the development set for 10 consecutive evaluations, and evaluation was done every 1,000 and 5,000 for batches of size 100 and 256 for pretraining and training respectively. Learning rate used for rmsprop was 0.01 in pretraining and for adam with decay was 0.005 for training. 4,000 learning rate warm up steps were used. Pretraining took about 7 days with 4 GPUs, afterwards, training took roughly the same time. Monte Carlo used 20 sentence rolls per word. We present graphs for the constant reward setting in Figures 8 and 7. Trends are similar to the ones obtained for the Simulated Reward setting. Each column x corresponds to the difference in the probability that y best is ranked in rank x in the reinforced model, and the same probability in the pretrained model.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Reinforcment practices for machine translation performance gains might not come from better predictions.

represent the tldr for finding the corresponding scientific article.

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.


Question: represent the text to find a very short summary (1-2 sentences max).

Question: Significant advances have been made in Natural Language Processing (NLP) modelling since the beginning of 2018. The new approaches allow for accurate , even when there is little labelled data, because these NLP models can benefit from training on both task-agnostic and task-specific unlabelled data. However, these advantages come with significant size and computational costs. This workshop paper outlines how our proposed convolutional student architecture, having been trained by a distillation process from a large-scale model, can achieve 300x inference speedup and 39x reduction in parameter count. In some cases, the student model performance surpasses its teacher on the studied tasks. The last year has seen several major advances in NLP modelling, stemming from previous innovations in embeddings BID0 BID2 and attention models BID3 BID5 that allow Language Models (LMs) to be trained on very large corpuses: For instance ELMo BID6, OpenAI Transformer BID7 and recently BERT BID8.In addition, the power of building on LM-enhanced contextualised embeddings, using a fine-tuning approach on task-specific unlabelled data BID9, has shown huge benefits for downstream tasks (such as text classification) -especially in a typical industrial setting where labelled data is scarce. In order to make use of these advances, this work shows how a model distillation process BID10 can be used to train a novel'student' CNN structure from a much larger'teacher' Language Model. The teacher model can be fine-tuned on the specific task at hand, using both unlabelled data, and the (small number of) labelled training examples available. The student network can then be trained using both labelled and unlabelled data, in a process akin to pseudo-labelling BID11.Our show it is possible to achieve similar performance to (and surpass in some cases) large attention-based models with a novel, highly efficient student model with only convolutional layers. In this work, we used the OpenAI Transformer BID7 model as the'teacher' in a model-distillation setting, with a variety of different'student' networks (see FIG0 . The OpenAI Transformer model consists of a Byte-Pair Encoded subword BID13 embedding layer followed by 12-layers of "decoder-only transformer with masked self-attention heads" BID3, pretrained on the standard language modelling objective on a corpus of 7000 books. This LM's final layer outputs were then coupled with classification modules and the entire model was discriminatively fine-tuned with an auxiliary language modelling objective, achieving excellent performance on various NLP tasks. To optimize for speed and memory constraints of industrial deployment, a variety of different models were trained (a) on the classification task directly; and (b) via distillation BID10 of the logit layer output by the pretrained OpenAI classification model. To combat label-scarcity and improve distillation quality, we inferred distillation logits for unlabelled samples in a pseudo-labelling manner BID11, while using transfer learning through pretrained GloVe embeddings BID1. A number of common network structures were tested in the student role, specifically:• a two-layer BiLSTM network BID14 • a wide-but shallow CNN network BID15 • a novel CNN structure, dubbed here'BlendCNN'The BlendCNN architecture was inspired by the ELMo'something from every layer' paradigm, and aims to be capable of leveraging hierarchical representations for text classification BID5 BID16.The BlendCNN model is illustrated in FIG0, and comprises a number of CNN layers (with n_channels=100, kernel_width=5, activation=relu), each of which exposes a global pooling output as a'branch'. These branches are then concatenated together and "blended" through a dense network (width=100), followed by the usual classification logits layer. Each of the models was trained and tested on the 3 standard datasets described in BID17: AG News, DBpedia and Yahoo Answers. The experiment proceeded in two phases, the first being to evaluate the performance of two baseline methods (TFIDF+SVM BID18 and fastText BID19) along with that of the student networks (without the benefit of a LM teacher), and the large LM, with a classification'head' trained on the task. The second phase used the large LM in a'teacher' role, to train the other networks as students via distillation of the LM classifier logits layer (with a Mean Absolute Error loss function). Referring to TAB2, the 3-Layer and 8-Layer variants of the proposed BlendCNN architecture achieve the top scores across all studied datasets. However, the performance of the proposed architecture is lower without the'guidance' of the teacher teacher logits during training, implying the marked improvement is due to distillation. The additional given for BlendCNN quantifies the advantage of adding unlabelled data into the distillation phase of the student model training. Notably from TAB1, the 3-Layer BlendCNN student has 39× fewer parameters and performs inference 300× faster than the OpenAI Transformer which it empirically out-scores. For text classifications, mastery may require both high-level concepts gleaned from language under standing and fine-grained textual features such as key phrases. Similar to the larval-adult form analogy made in BID10, high-capacity models with task-agnostic pre-training may be well-suited for task mastery on small datasets (which are common in industry). On the other hand, convolutional student architectures may be more ideal for practical applications by taking advantage of massively parallel computation and a significantly reduced memory footprint. Our suggest that the proposed BlendCNN architecture can efficiently achieve higher scores on text classification tasks due to the direct leveraging of hierarchical representations, which are learnable (even in a label-sparse setting) from a strong teaching model. Further development of specialized student architectures could similarly surpass teacher performance if appropriately designed to leverage the knowledge gained from a pretrained, task-agnostic teacher model whilst optimizing for task-specific constraints.

We train a small, efficient CNN with the same performance as the OpenAI Transformer on text classification tasks

Represent this text to find a much much longer article about it with say hundreds-thousands of words

A variant of capsule networks that can be used for pairwise learning tasks. Results shows that Siamese Capsule Networks work well in the few shot learning setting.