Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Energy-based models (EBMs), a.k.a. un-normalized models, have had recent successes in continuous spaces. However, they have not been successfully applied to model text sequences. While decreasing the energy at training samples is straightforward, mining (negative) samples where the energy should be increased is difficult. In part, this is because standard gradient-based methods are not readily applicable when the input is high-dimensional and discrete. Here, we side-step this issue by generating negatives using pre-trained auto-regressive language models. The EBM then works in the {\em residual} of the language model; and is trained to discriminate real text from text generated by the auto-regressive models. We investigate the generalization ability of residual EBMs, a pre-requisite for using them in other applications. We extensively analyze generalization for the task of classifying whether an input is machine or human generated, a natural task given the training loss and how we mine negatives. Overall, we observe that EBMs can generalize remarkably well to changes in the architecture of the generators producing negatives. However, EBMs exhibit more sensitivity to the training set used by such generators. Energy-based models (EBMs) have a long history in machine learning (; ;). Their appeal stems from the minimal assumptions they make about the generative process of the data. Unlike directed or auto-regressive models which are defined in terms of a sequence of conditional distributions, EBMs are defined in terms of a single scalar energy function, representing the joint compatibility between all input variables. EBMs are a strict generalization of probability models, as the energy function need not be normalized or even have convergent integral. Training an EBM consists of decreasing the energy function at the observed training data points (a.k.a. positives), while increasing it at other data points (a.k.a. negatives) . Different learning strategies mainly differ in how negatives are mined . Some find negatives by gradient descent, or using Monte Carlo methods like Gibbs sampling and hybrid Monte Carlo , which enable the loss to approximate maximum likelihood training . Other approaches instead use implicit negatives, by enforcing global constraints on the energy function, like sparsity of the internal representation , for instance. GANs can be interpreted as a particular form of EBM where the negatives are generated by a learned model. While there are works exploring the use of EBMs for modeling images (; ;), they have not been successfully applied to text. One reason is that text consists of sequences of discrete variables, which makes the energy function not differentiable with respect to its inputs. Therefore, it is not possible to mine negatives using gradient-based methods. Other approaches to mine negatives are also not immediately applicable or may be too inefficient to work at scale. In this work, we start from the observation that current large auto-regressive locally-normalized language models are already strong, and therefore, it may be beneficial to use them to constrain the search space of negatives. We propose to learn in the residual space of a pre-trained language model (LM), which we accomplish by using such LM to generate negatives for the EBM. Given a dataset of positives and pre-generated negatives, the EBM can be trained using either a binary cross-entropy loss or a ranking loss, to teach the model to assign a lower energy to true human generated text than to the text generated by the pre-trained LM. The question we ask in this work is whether such an EBM can generalize well. Understanding this is important for two reason. First, this generalization is a prerequisite for using residual EBMs for modeling text. Second, in our setting, this generalization question is equivalent to the question of whether it is possible for a learned model (the energy function) to discriminate real text from text generated by an auto-regressive model. Discriminating real vs. machine-generated text is an important task on its own that has recently gained a lot of attention . Our contribution is an extensive study of the generalization ability of such residual EBMs, or in other words, the generalization ability of models trained to detect real text from machine generated text. In particular, we assess how well the energy function is robust to changes in the architecture of the generator and to changes in the data used to train the generator. The overall finding is that the energy function is remarkably robust, and the bigger the model and the longer the generation the better its performance. Moreover, the energy function is robust to changes in the architecture of the LM producing negatives at test time. However, it is sensitive to the training dataset of the test generator. Our work can be interpreted as a particular instance of EBMs where negatives are produced by a pre-trained language model as opposed to the energy function itself. Learning a generator and a discriminator relates also to Generative Adversarial Networks , except that in our case the generator is trained beforehand. Since the discriminator is learned after the generator has been trained, it learns from the residual error of the generator, and therefore, our training procedure is a particular instance of a "cascade" model and "boosting" . Using a separately trained scoring function to evaluate and rank candidate outputs has a long history which dates back to work on parsing and machine translation . In that work however, the goal was to improve a weak generator by employing a linear reranker taking as input relatively few hand-design features. The approach has been recently re-discovered in the context of dialogue modeling by , but here negatives are randomly chosen next utterances from the training dataset. Several recent works have studied whether machine generations can be detected automatically, but they do not study how these findings generalize to settings where generator architectures and corpora are different between training and test time. For example, (GROVER) assume that the generator is known and apply only slight fine-tuning in order to train the energy function. (GLTR) assume knowledge of the generator; these Authors say "We further hypothesize that these methods generalize to black-box scenarios, as long as the fake text follows a similar sampling assumption and is generated by a large language model"; our work answers precisely this question, provides a rigorous experimental protocol and quantitative . Finally, there has been a release of a training dataset of the GPT-2 language model generations for the purpose of training discriminators capable of detecting machine generated text. While we share the same motivation, our work is a much broader investigation on the topic. We assess generalization of several discriminator architectures to not just one but several kinds of generators and corpora used for training (including GPT-2). In this section, we describe how we train the energy based model and how we mine negatives. Our goal is to learn an energy function E(w 1, . . ., w n |c; θ) ∈ R that scores the joint compatibility of an input sequence of tokens (w 1, . . ., w n) given some context c and a set of parameters θ. The context depends on the application, it could be the preceding text, some keywords, a bag of words, a title, etc. In this work for simplicity, c is an affix from which we condition the generation. The goal of training is to assign to golden sequences, i.e. sequences taken from a dataset of human generated text, lower energy than other sequences. We parameterize the energy function as a neural network, using the architectures described in §4.3. At training time, the energy function can be trained using a variety of different losses. In this work, we consider two choices: the binary cross-entropy loss and the ranking loss . As the findings are similar, unless otherwise specified we will refer to the binary cross-entropy loss, and report with the ranking loss in Appendix C. + be a positive sample taken from the training set, and consisting of a sequence of n tokens given some context c. Let (x − 1, ..., x − k) be a set of k negative samples each derived from the same context c as above, all containing at least some machine generated tokens. We train our energy function using the (per-sample) binary cross-entropy loss: wherex − is the most offending negative , i.e. its index is the solution of arg min, and σ is the sigmoid function: The most critical component of training an energy based model is the method used to generate negatives, i.e. inputs where the energy should score high (unlikely inputs). In settings with continuous variables, researchers have suggested MCMC or Langevin dynamics . In this work instead, we use the fact that modern auto-regressive models for text are already quite good, and we use them for negative sampling. We train two auto-regressive language models, a left-to-right one which will be used to produce suffixes assuming the prefix is the context, and a right-to-left one which will be used to generate prefixes assuming the suffix is the context. The negatives are generated by top-k sampling setting k equal to 10. Given a trained language model (for instance, a left-to-right autoregressive model) and given a positive example x + = (w i+1, . . ., w i+n), for a given context: c = (w 1, . . ., w i), a negative can be written as: x − = (ŵ i+1, . . .,ŵ i+n), where w j for j ∈ [1, i + n] are ground truth words, the first i of them belonging to the common context, andŵ j for j ∈ [i + 1, i + n] are words generated by the language model conditioned on c. In the same way, we can sample a negative with a right-to-left model yielding x − = (ŵ 1, . . .,ŵ n), for a given context c = (w n+1, . . ., w n+i). In this section we first describe the datasets and preprocessing used, provide architecture details for both generators and scoring functions, and finally introduce the evaluation settings. We train models on three corpora coming from different domains. We report more detailed statistics about the sizes of these corpora in Appendix Table 8: Books: The Toronto books corpus described in;, which consists of fiction books in 16 different genres, totaling about half a billion words. CCNews: We collect a de-duplicated subset of the English portion of the CommonCrawl news dataset , which totals around 16 Billion words. Wikitext: The wikitext103 dataset from , which consists of 103 million words from English Wikipedia articles. While Wikitext and CCNews are factual, Books is fiction and comprises a wide variety of writing styles. The CCNews corpus has the narrowest domain and it is two orders of magnitude larger than Wikipedia. Overall, these datasets are interesting because they enable us to assess the ability of the energy function to fit and generalize across various axes, from the amount of data available at training time to the richness of style and relatedness among the different data sources. On Wikitext and Books, we extract positive sequences from windows of text that are 160 tokens long with a stride of 40. On the larger CCNews we do the same except that we stride by 160 tokens. This protocol to mine positives is used both at training and test time, although at test time we limit the evaluation to 60,000 randomly chosen positive samples. We use a Byte Pair Encoding in order to represent all the dataset with a common vocabulary. In particular, our vocabulary contains 50k tokens that was constructed from a byte level UTF-8 encoding of the CC-News corpus following. We mainly use a transformer based network to generate negatives. We have a medium, large and huge transformer model based on the architecture used in, yielding three language models in total: TransfSmall, TransfBig and TransfHuge; see details also in Appendix B. The small sized models use 6 blocks each containing a multi-head attention module with 8 heads. The large models use 12 blocks each containing a multi-head attention module with 16 heads. The huge models use 48 blocks each containing a multi-head attention module with 25 heads. Transformer models are also implemented in as "transformer_lm", "transformer_lm_big", and "transformer_lm_gpt2_big". The TransfHuge has 10x the number of parameters than TransfBig and it is trained on CCNews only. For each architecture except for TransfHuge we train two models on each each dataset: left to right and right to left. In addition to the transformer generator, we also consider a 12-layer convolutional architecture (Conv) , and we also use a the third-party trained GPT2 models as described in §5.3. As described in §3.2, we use these language models to generate either a prefix or a suffix. Unless otherwise specified, the context is long either 120 or 140 tokens (with equal probability). Positive and negative examples have 40 or 20 tokens depending on the context size, for an overall length of 160 tokens in all cases. In preliminary experiments, we found that increasing the size of the generations and reducing the size of the context makes the learning task significantly easier. We analyze the effect of the context size in §5.5. We consider three architectures for the energy function: Linear which computes an energy value via a bag of tokens:, where u i is a learnt scalar parameter corresponding to the i-th token in the vocabulary. BiLSTM which computes an energy value through L bidirectional layers using LSTM recurrent units , as in Linear(AvgPool(h L,1, . . ., h L,n)), where h L,i is the hidden state at position i and layer L which is the concatenation of the forward and backward hidden states, AvgPool averages hidden states over positions and Linear is a vector of parameters projecting the hidden state down to a scalar value. We consider two versions, referred to as "BiLSTMsmall" and "BiLSTMbig". Both have 4 layers, but BiLSTMsmall has 512 units in both the embedding layer and the hidden layers, while BiLSTMbig has 758 units in the embedding layer and 2014 units in the hidden states. Transformer which computes an energy value similarly to the BiLSTM's, except that each bi-LSTM layer is replaced by a either a bidirectional Transformer layer (BiTransf), or a Transformer with causal self-attention (UniTransf). For unidirectional models we use the same averaging technique as with BiLSTM models. For bidirectional models the energy is computed via: f (w 1, ..., w n) = u h L,1 + b, where h L,1 is the top layer hidden state at the first position (as common practice also in prior work ). BiTransf uses the BERT-Large CORPUS: GENERATOR ARCHITECTURE: in-domain cross-architecture cross-corpus unseen Table 1: Four evaluation settings considered in this work, described in §4.4. architecture initialized from. It uses 24 self-attention layers with 1024 units and 16-head attention each. UniTransf has instead 12 layers with 1024 units and 16 attention heads per layer and it is initialized from a language modeling task as in. For all models, we use Adam optimizer with warmup. Training is stopped after processing 2.5M samples without any improvement on the validation set. We use data-parallel synchronous multi-GPU training with up to 8 nodes, each with 8 Nvidia V100 GPUs. To improve training speed, we use mixed precision training 1. Following common practice we clip the norm of the gradient vector . More details about hyper-parameter setting can be found in Appendix Table 11, while Table 10 in Appendix reports the number of parameters of each energy function. We evaluate the generalization of a residual EBM in four settings: in-domain, cross-architecture, cross-corpus, and unseen. These settings are determined by the corpora C train used to train the training generator G train with architecture A train and the corpora C test used to train the testing generator G test with architecture A test. Note that G train = G test even if A test = A train as we use different training seeds. In all cases, C train is used for fitting the G train and also for the positives for the EBM. In the in-domain setting, C test is C train (but any affixes used as conditioning during testing are from the test-set of the corpus), and A test = A train. In the cross-architecture setting, again C test is C train, but A test is different from A train. In the cross-corpus setting, A test = A train but C test is different than C train, and G test is trained on the training split of C test, while G train trained on the train split of C train. In the unseen setting, both C test is different than C train and A test is different from A train. In all settings, we report performance in terms of average classification accuracy balancing the positive and negative classes. We now present the main of this work and extensively investigate the generalization ability of the energy functions we have considered. In Table 2 we report the of the in-domain generalization experiment using our large language model, TransfBig. We observe that when the EBMs have similar representational power compared with the generator (UniTransf, see Table 10), they are able to distinguish real from fake completions fairly accurately, reaching an accuracy of more than 90% on the Books dataset (which is easier since it exhibits the larger variety of style and topics), and attaining above 88% on the more challenging CCNews dataset (for which generation is easier and hence discrimination harder). The Wikipedia dataset has lower accuracy because the EBM overfits to this smaller dataset. TransfBig (log-likelihood) 57.1 50.8 50.5 Table 2: "In domain" generalization accuracy of EBMs (each row) on various text corpora. A column corresponds to the corpus used to get positives and to fit the train and test language models, which are TransfBig (§4.2) with different initial seeds. The last row is the accuracy when using as energy the log-probability of the training language model over the whole sequence. Conv TransfSmall Conv 92.9 81.2 TransfSmall 86.5 87.9 Table 3: Cross-architecture generalization accuracy using the Wikitext dataset for both training and testing (Ctrain = Ctest). Each row is a model architecture used for generating the training negatives (Atrain), and each column is a model architecture for generating the testing negatives (Atest). The energy function is UniTransf. Weaker energy models are able to do comparably or better at discriminating real from fake than the training generator used as a discriminator by taking the log probability of the sequence as energy. In Table 3, we assess how well the UniTransf energy function generalizes to different generator architectures at test time, namely Conv and TransfSmall. As a reference on the Wikitext dataset, the test perplexity of Conv and TransfSmall are 35.4 and 33.5, respectively. Therefore, these two generators attain roughly the same perplexity, despite Conv having about 4 times more parameters, see Table 9. Surprisingly, UniTransf has significantly harder time discriminating TransfSmall negatives with an in-domain rate of 87.9%, compared to 92.9% of Conv. Also, UniTransf trained with TransfSmall negatives is more robust to the (weaker) Conv generations, than vice versa, with a mild 1.4% accuracy drop. However, if we average values across rows, we see that UniTransf tested with mixed negatives is just slightly more accurate when training with the harder negatives produced by TransfSmall. In Table 4 we show the of generalizing across corpora using UniTransf as an energy function and TransfBig as generator both at training and test time. We observe that models generalize less well across corpora; for instance, when testing on Wikitext an energy function trained with either Books or CCNews, the accuracy is 59.1% and 65.5%, respectively. However, training on the union of two of the corpora gives a large benefit over training on just one or the other when testing on the third. Finally, training on the union of all the three corpora (last two rows) yields an energy function that is very robust to the testing conditions, with an accuracy which is on par if not better than training on in-domain data, even for the largest CC-News dataset (second column). We also tested the bidirectional transformer energy function BiTransf with 355M parameters (almost twice as UniTransf), and found that on CC-News it improves accuracy by more than 5% when it is trained on the union of all corpora, confirming the finding that bigger models trained on more data can achieve substantially better discrimination. As BiTransf was pre-trained using the whole Wikipedia rather than the training part of Wikitext103, we do not report its accuracy on Wiki test set. Test time negatives are generated by models specified in the rows, with their training set in parenthesis and model size in millions of parameters. Note that both the training corpus and GPT2 generator are "unseen" by the energy function during training. In Table 5 we test the generalization of the energy functions to GPT-2 generators 2 that were trained on a completely different dataset, namely WebText a dataset of 8 million web pages. This is an instance of unseen generalization since C train = C test, and A train = A test. We also consider generations from TransfHuge (last row) whose configuration is similar to the unreleased biggest GPT2 model with 1.4 billion parameters, 7 times bigger than TransfBig, the generator used at training time. Expectedly as the generator gets bigger the discrimination tasks gets harder. When the energy function is confronted with generations from the GPT2 small model, which is smaller than the training generator, the accuracy is close to the in-domain setting, however. For instance the BiTransf accuracy increases by 0.4% on the Book corpus and decreases by 5.4% on the CCNews corpus compared to the fully in-domain of Table 4. That suggests that for a known domain, a big enough energy model trained with a single big generator can efficiently discriminate a block-box generator. Of course, accuracy decreases as the black-box generator is made bigger (GPT-2 medium). Finally, we investigate generalization of the energy function to a new domain, such as samples from the dataset of GPT-2 generations. For each model the dataset has a 250k generated texts with either top-k sampling or random sampling. Also, the dataset provides samples from the WebText corpus that was used to train the generator models, and that we use to discriminate against. To adapt our models to this task we split the text segments into sets of intersecting blocks of 160 tokens. During training we treat all blocks in a set as either positives or negatives. During evaluation we take the mean prediction over all blocks in a segment as a prediction for the whole segment. Table 6: Generalization of the energy function to unconditional generation from various GPT2 models (model size in parantheses, followed by sampling method used). Each row contains the accuracy on the corresponding test set. TF-IDF are taken from. In Table 6 we report of the BiTransf energy function compared to the TF-IDF baseline provided with the dataset. We consider two cases. In the in-domain setting, we finetune the energy function on the train set of each of the datasets, following the same protocol used by the provided TF-IDF baseline. In generalization mode, we finetune only on the generations from the small GPT2 model (both top-k and random sampling), and apply the model to the other datasets. Unsurprisingly, in-domain BiTransf beats TF-IDF baseline getting almost 100% across the board. However in generalization mode, we can outperform the TF-IDF baseline only when the generator is less than three times bigger than what was used at training time. Interestingly, our energy function was trained using a fixed length input with a prefix. These generalization are significantly higher than the in-domain experiment of Table 2 because the unconditional task is significantly easier, a topic further discussed next. First, we investigate the dependency between performance of the energy functions and length of the prefix. We trained BiLSTMSmall and UniTransf models on examples with varying prefix length from the Wikitext corpus, and computed the accuracy for each prefix length independently. Figure 1 shows that as the prefix length increases (and the generation gets shorter), the discrimination task gets harder and the difference between the models more prominent. The unconditional case, i.e. zero prefix length, is the easiest, while prefixes of length 120 and 140 that are the main experimental setup in this work, are the hardest. Finally, in Table 7 we study the impact of the number of negatives and using the most offending negative in the loss of Eq.. Using more negatives and harder negatives improves accuracy. In the previous sections we have seen that the energy function is less robust to negatives generated from a model trained on a different corpus. However, even in that case, a negative is still a sample from an auto-regressive neural network. In Appendix F, we show examples where changing a few entities can cause large jumps in the energy (from negative to positive or vice versa), and so fool the EBM. More generally, we see that the energy function is not robust to truly out-of-domain samples. For example, the energy will score blocks of randomly generated text lower than real text. These behaviors are evidence that the energy functions have learned the regularities of generated text, as opposed to learning the regularities of real text. We surmise that it does so because modeling the latter would be much more difficult than the former. By modeling generated text, the energy function assigns low score to anything that is not generated by its training generator. While not surprising, this might be considered a liability of such energy functions. However, as a model of text, the energy functions should be considered as working on the residuals of the language models used to generate negatives. For the examples in Appendix F, the language model records a large decrease in likelihood after the change in entity; and language models of course give much lower likelihood to random text than gold or generated text. Therefore, the energy function needs not to be accurate on examples that are already very unlikely according to these language models. In Figure 2 we show the average effects of applying various perturbations to sequences from Wikitext103 on an in-domain energy and language model at each location (from 1 to 160) in the sequence. We see that for all perturbations, the energy decreases its value, but the language model increases its negative log likelihood. We also see that the energy function is more sensitive to the ends of the text, which is where the negatives were different from real text at training time. The EBM framework could potentially unlock more expressive models of text, as they are not limited to scoring a single word at a time as current locally normalized auto-regressive models do. Unfortunately, training EBMs is challenging because generating negatives using the energy function itself is still an open research problem, and does not scale well in practice. In this work, we propose a simple solution, which is to leverage generations produced by pre-trained language models as negative samples. As a preliminary yet necessary step in this direction we have investigated the generalization ability of such EBMs. We found that EBMs, when trained on large datasets, achieve good generalization. For instance, they behave nicely when tested with negatives produced by generators that have rather different architectures. The generalization is less good when generators are trained on other corpora, but EBMs re-gain robustness once we train them on even bigger composite datasets. In the future, we can improve EBMs for text by simply making their architectures bigger and increasing the diversity and size of their training datasets. Of course, further scaling up of EBMs will pose formidable engineering challenges. On the application side, a natural application of the current formulation of EBMs is real/fake text discrimination. We believe that this is important application in its own right, and that EBMs can be very powerful, as demonstrated by their superior performance compared to discriminating using the original language model log-likelihood. Table 9: Number of parameters (in millions) for the generator language models. The computational cost is directly related to the number of parameters in other layers than the input embedding layer (second row). a We use models from HuggingFace repository (https://github.com/huggingface/ pytorch-transformers) and report here the sizes of these models as they were used to generate data for table 5. Note that the OpenAI GPT2 repository (https://github.com/openai/gpt-2) defines models sizes as 124M and 355M for small and medium model correspondingly. b As reported in. Table 10: Number of parameters in millions for the scoring functions. The computational cost is directly related to the number of parameters in other layers than the input embedding layer (second row). The (per-sample) ranking loss is: In this case we also refer to the negative energy as the model score. The ranking loss makes the energy values local, as the loss takes as input the difference of energies for a pairs of positive and negative that share the same context. Instead, the binary cross entropy loss of Eq. 1 encourages a more global and absolute scoring as the loss forces all positive examples to have negative energy, and all negative samples to have positive energy, regardless of the context. Therefore, the binary cross entropy loss is perhaps more interpretable as it is not context dependent, but the task is also harder to learn. Empirically, we found similar findings with both losses. When the energy function is trained using the ranking loss of eq. 2, we evaluate the model using precision at 1 (P@1), which is the ratio between the number of times the ground truth sequence scores the lowest over its set of negatives, averaged over the number of sequences in the test set. All models are implemented using the PyTorch framework and are optimized using Adam . To train our biggest models (UniTransf and BiTransf) we used 8 machines each with 8 GPUs in synchronous mode using data parallelism. The ing large batch size speeds up training when combined with float16 reduced precision and cosine scheduling of the learning rate without any restarts , i.e. we decay the learning rate to zero over the course of "max steps" updates and then stop training. Using these methods, we reduced training time by five times compared to a single node training. For all other configurations we used a single node with up to 8 GPUs and inverse square root decay. In this section we show that we can change a few words to make a negative example become a "positive" one as judged by the energy function, and vice versa, by using gradient information. Below here, we show an example of a ground truth sentence from the Wikitext dataset. Here the block has 160 BPE tokens, where the first 120 tokens (black font) are used as context and the remaining 40 are the ground truth completion. Next, we use a language model to generate 10 negatives:... In this example, using the big transformer model, UniTransf, as the energy function, we are able to separate real from fake examples as shown (Figure 4). We want to perturb these negatives to violate the margin. To do so, we make use of the gradient information from the energy function ∇ x E θ (x) and use a first order Taylor expansion to approximate the effect of a token replacement (we abuse our notations and use x to denote embeddings in this analysis). Given the original sample x, we change one word x i to x i to arrive at x. The score of x is approximately: Using this approximation, we can search for those token replacements that increase/decrease the energy the most. We can easily change a negative sample to a positive one by replacing the 5 words highlighted below. In paratheses, we report both score and language model perplexity.

Represent the short summary for finding the corresponding scientific article (long).

A residual EBM for text whose formulation is equivalent to discriminating between human and machine generated text. We study its generalization behavior.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the article for finding a single-sentence TLDR.

Question: Semi-supervised learning, i.e. jointly learning from labeled an unlabeled samples, is an active research topic due to its key role on relaxing human annotation constraints. In the context of image classification, recent advances to learn from unlabeled samples are mainly focused on consistency regularization methods that encourage invariant predictions for different perturbations of unlabeled samples. We, conversely, propose to learn from unlabeled data by generating soft pseudo-labels using the network predictions. We show that a naive pseudo-labeling overfits to incorrect pseudo-labels due to the so-called confirmation bias and demonstrate that mixup augmentation and setting a minimum number of labeled samples per mini-batch are effective regularization techniques for reducing it. The proposed approach achieves state-of-the-art in CIFAR-10/100 and Mini-ImageNet despite being much simpler than other state-of-the-art. These demonstrate that pseudo-labeling can outperform consistency regularization methods, while the opposite was supposed in previous work. Code will be made available. Convolutional neural networks (CNNs) have become the dominant approach in computer vision (; ; ;). To best exploit them, vast amounts of labeled data are required. Obtaining such labels, however, is not trivial, and the research community is exploring alternatives to alleviate this (; ;). Knowledge transfer via deep domain adaptation is a popular alternative that seeks to learn transferable representations from source to target domains by embedding domain adaptation in the learning pipeline. Other approaches focus exclusively on learning useful representations from scratch in a target domain when annotation constraints are relaxed (; ;). Semi-supervised learning (SSL) focuses on scenarios with sparsely labeled data and extensive amounts of unlabeled data; learning with label noise seeks robust learning when labels are obtained automatically and may not represent the image content; and self-supervised learning uses data supervision to learn from unlabeled data in a supervised manner. This paper focuses on SSL for image classification, a recently very active research area . SSL is a transversal task for different domains including images , audio , time series (González et al., 2018), and text . Recent approaches in image classification primarily focus on exploiting the consistency in the predictions for the same sample under different perturbations (consistency regularization) , while other approaches directly generate labels for the unlabeled data to guide the learning process (pseudo-labeling) . These two alternatives differ importantly in the mechanism they use to exploit unlabeled samples. Consistency regularization and pseudo-labeling approaches apply different strategies such as a warm-up phase using labeled data , uncertainty weighting , adversarial attacks , or graph-consistency . These strategies deal with confirmation bias , also known as noise accumulation . This bias stems from using incorrect predictions on unlabeled data for training in subsequent epochs and, thereby increasing confidence in incorrect predictions and producing a model that will tend to resist new changes. This paper explores pseudo-labeling for semi-supervised deep learning from the network predictions and shows that, contrary to previous attempts on pseudo-labeling (; ;), simple modifications to prevent confirmation bias lead to state-of-the-art performance without adding consistency regularization strategies. We adapt the approach proposed by in the context of label noise and apply it exclusively on unlabeled samples. Experiments show that this naive pseudo-labeling is limited by confirmation bias as prediction errors are fit by the network. To deal with this issue, we propose to use mixup augmentation as an effective regularization that helps calibrate deep neural networks and, therefore, alleviates confirmation bias. We find that mixup alone does not guarantee robustness against confirmation bias when reducing the amount of labeled samples or using certain network architectures (see Subsection 4.3), and show that, when properly introduced, dropout regularization and data augmentation mitigates this issue. Our purely pseudo-labeling approach achieves state-of-the-art (see Subsection 4.4) without requiring multiple networks (; ; ;), nor does it require over a thousand epochs of training to achieve peak performance in every dataset , nor needs many (ten) forward passes for each sample . Compared to other pseudo-labeling approaches, the proposed approach is simpler in that it does not require graph construction and diffusion or combination with consistency regularization methods , but still achieves state-of-the-art . This section reviews closely related SSL methods, i.e. those using deep learning with mini-batch optimization over large image collections. Previous work on deep SSL differ in whether they use consistency regularization or pseudo-labeling to learn from the unlabeled set , while they all share the use of a cross-entropy loss (or similar) on labeled data. Consistency regularization Imposes that the same sample under different perturbations must produce the same output. This idea was used in where they apply randomized data augmentation, dropout, and random max-pooling while forcing softmax predictions to be similar. A similar idea is applied in , which also extends the perturbation to different epochs, i.e. the current prediction for a sample has to be similar to an ensemble of predictions of the same sample in the past. Here the different perturbations come from networks at different states, dropout, and data augmentation. In , the temporal ensembling method is interpreted as a teacher-student problem where the network is both a teacher that produces targets for the unlabeled data as a temporal ensemble, and a student that learns the generated targets by imposing the consistency regularization. naturally re-define the problem to deal with confirmation bias by separating the teacher and the student. The teacher is defined as a different network with similar architecture whose parameters are updated as an exponential moving average of the student network weights. This method is extended in , where they apply an uncertainty weight over the unlabeled samples to learn from the unlabeled samples with low uncertainty (i.e. entropy of the predictions for each sample under random perturbations). use virtual adversarial training to carefully introduce perturbations to data samples as adversarial noise and later impose consistency regularization on the predictions. More recently, propose to use a contrastive loss on the predictions as a regularization that forces predictions to be similar (different) when they are from the same (different) class. This method extends the consistency regularization previously considered only in-between the same data samples to in-between different samples. Their method can naturally be combined with or to boost their performance. propose interpolation consistency training, a method inspired by that encourage predictions at interpolated unlabeled samples to be consistent with the interpolated predictions of individual samples. Also, authors in apply consistency regularization by guessing low-entropy labels, generating data-augmented unlabeled examples and mixing labeled and unlabeled examples using mixup. Both and adopt to estimate the targets used in the consistency regularization. Co-training uses two (or more) networks trained simultaneously to agree on their predictions (consistency regularization) and disagree on their errors. Errors are defined as different predictions when exposed to adversarial attacks, thus forcing different networks to learn complementary representations for the same samples. measure the consistency between the current prediction and an additional prediction for the same sample given by an external memory module that keeps track of previous representations. They additionally introduce an uncertainty weighting of the consistency term to reduce the contribution of uncertain predictions. Consistency regularization methods such as (; ;) have all been shown to benefit from stochastic weight averaging method , that averages network parameters at different training epochs to move the SGD solution on borders of flat loss regions to their center and improve generalization. Pseudo-labeling Seeks the generation of labels or pseudo-labels for unlabeled samples to guide the learning process. An early attempt at pseudo-labeling proposed in uses the network predictions as labels. However, they constrain the pseudo-labeling to a fine-tuning stage, i.e. there is a pre-training or warm-up to initialize the network. A recent pseudo-labeling approach proposed in uses the network class prediction as hard labels for the unlabeled samples. They also introduce an uncertainty weight for each sample loss, it being higher for samples that have distant k-nearest neighbors in the feature space. They further include a loss term to encourage intra-class compactness and inter-class separation, and a consistency term between samples with different perturbations. Improved are reported in combination with . Finally, a recently published work implements pseudo-labeling through graph-based label propagation. The method alternates between two steps: training from labeled and pseudo-labeled data and using the representations of the network to build a nearest neighbor graph where label propagation is applied to refine hard pseudo-labels. They further add an uncertainty score for every sample (softmax prediction entropy based) and class (class population based) to deal, respectively, with the unequal confidence in network predictions and class-imbalance. We formulate SSL as learning a model h θ (x) from a set of N training samples D. These samples are split into the unlabeled set C the one-hot encoding label for C classes corresponding to x i and N = N l + N u. In our case, h θ is a CNN and θ represents the model parameters (weights and biases). As we seek to perform pseudo-labeling, we assume that a pseudo-labelỹ is available for the N u unlabeled samples. We can then reformulate SSL as training usingD = {(, beingỹ = y for the N l labeled samples. The CNN parameters θ can be optimized using categorical cross-entropy: where h θ (x) are the softmax probabilities produced by the model and log(·) is applied element-wise. A key decision is how to generate the pseudo-labelsỹ for the N u unlabeled samples. Previous approaches have used hard pseudo-labels (i.e. one-hot vectors) directly using the network output class or the class estimated using label propagation on a nearest neighbor graph . We adopt the former approach, but use soft pseudo-labels, as we have seen this outperforms hard labels, confirming the observations noted in in the context of relabeling when learning with label noise. In particular, we store the softmax predictions h θ (x i) of the network in every mini-batch of an epoch and use them to modify the soft pseudo-labelỹ for the N u unlabeled samples at the end of every epoch. We proceed as described from the second to the last training epoch, while in the first epoch we use the softmax predictions for the unlabeled samples from a model trained in a 10 epochs warm-up phase using the labeled data subset D u. We use the two regularizations applied in to improve convergence. The first regularization deals with the difficulty of converging at early training stages when the network's predictions are mostly incorrect and the CNN tends to predict the same class to minimize the loss. Assignment of all samples to a single class is discouraged by adding: where p c is the prior probability distribution for class c and h c denotes the mean softmax probability of the model for class c across all samples in the dataset. As in , we assume a uniform distribution p c = 1/C for the prior probabilities (R A stands for all classes regularization) and approximate h c using mini-batches. The second regularization is needed to concentrate the probability distribution of each soft pseudo-label on a single class, thus avoiding the local optima in which the network might get stuck due to a weak guidance: where h c θ (x i) denotes the c class value of the softmax output h θ (x i) and again using mini-batches (i.e. N is replaced by the mini-batch size) to approximate this term. This second regularization is the average per-sample entropy (R H stands for entropy regularization), a well-known regularization in SSL . Finally, the total semi-supervised loss is: where λ A and λ H control the contribution of each regularization term. We stress that this pseudolabeling approach adapted from is far from the state-of-the-art for SSL (see Subsection 4.2), and are the mechanisms proposed in Subsection 3.1 which make pseudo-labeling a suitable alternative. Network predictions are, of course, sometimes incorrect. This situation is reinforced when incorrect predictions are used as labels for unlabeled samples, as it is the case in pseudo-labeling. Overfitting to incorrect pseudo-labels predicted by the network is known as confirmation bias. It is natural to think that reducing the confidence of the network on its predictions might alleviate this problem and improve generalization. Recently, mixup data augmentation introduced a strong regularization technique that combines data augmentation with label smoothing, which makes it potentially useful to deal with this bias. Mixup trains on convex combinations of sample pairs (x p and x q) and corresponding labels (y p and y q): where δ ∈ {0, 1} is randomly sampled from a beta distribution Be (α, β), with α = β (e.g. α = 1 uniformly selects δ). This combination regularizes the network to favor linear behavior in-between training samples, reducing oscillations in regions far from them. Additionally, Eq. 6 can be reinterpreted in the loss as q, thus re-defining the loss * used in Eq. 4 as: As shown in , overconfidence in deep neural networks is a consequence of training on hard labels and it is the label smoothing effect from randomly combining y p and y q during mixup training that reduces prediction confidence and improves model calibration. In the semi-supervised context with pseudo-labeling, using soft-labels and mixup reduces overfitting to model predictions, which is especially important for unlabeled samples whose predictions are used as soft-labels. Note that training with mixup generates softmax outputs h θ (x) for mixed inputs x, thus requiring a second forward pass with the original images to compute unmixed predictions. Mixup data augmentation alone might not effectively deal with confirmation bias when few labeled examples are provided. For example, when training with 500 labeled samples in CIFAR-10 and mini-batch size of 100, 1 clean sample per batch is seen, which is especially problematic at early stages of training where little correct guidance is provided. We find that setting a minimum number of labeled samples per mini-batch (as done in other works (; ; ;) ) provides a constant reinforcement with correct labels during training, reducing confirmation bias and helping to produce better pseudo-labels. Subsections 4.2 and 4.3 experimentally show that mixup, a minimum number of samples per mini-batch, and other techniques (dropout and data augmentation) reduce confirmation bias and make pseudo-labeling an effective alternative to consistency regularization. We use four image classification datasets, CIFAR-10/100 , SVHN (and Mini-ImageNet , to validate our approach. Part of the training images are labeled and the remaining are unlabeled. Following , we use a validation set of 5K samples for CIFAR-10/100 for studying hyperparameters in Subsections 4.2 and 4.3. However, as done in , we add the 5K samples back to the training set for comparisons in Subsection 4.4, where we report test (model from the best epoch). CIFAR-10, CIFAR-100, and SVHN These datasets contain 10, 100, and 10 classes respectivelly, with 50K color images for training and 10K for testing in CIFAR-10/100 and 73257 images for training and 26032 for testing in SVHN. The three datasets have resolution 32×32. We perform experiments with a number of labeled images N l = 0.25K, 0.5K, and 1K for SVHN and N l = 0.25K, 0.5K, 1K, and 4K (4K and 10K) for CIFAR-10 (CIFAR-100). We use the well-known "13-CNN" architecture for CIFAR-10/100 and SVHN. We also experiment with a Wide ResNet-28-2 (WR-28) and a PreAct ResNet-18 (PR-18) in Subsection 4.3 to study the generalization to different architectures. We emulate the semi-supervised learning setup Mini-ImageNet (a subset of the well-known ImageNet dataset) used in . Train and test sets of 100 classes and 600 color images per class with resolution 84 × 84 are selected from ImageNet, as in . 500 images per-class are kept for train (test) splits. The train and test sets therefore contain 50k and 10k images. As with CIFAR-100, we experiment with a number of labeled images N l = 4K and 10K. Following , we use a ResNet-18 (RN-18) architecture (a). Hyperparameters We use the typical configuration for CIFAR-10/100 and SVHN , and the same for Mini-ImageNet. Image normalization using dataset mean and standard deviation and subsequent data augmentation by random horizontal flips and 2 pixel translations for CIFAR and SVHN (Mini-ImageNet). Additionally, color jitter is applied as in in Subsections 4.3 and 4.4 for higher robustness against confirmation bias. We train using SGD with momentum of 0.9, weight decay of 10 −4, and batch size of 100. Training always starts with a high learning rate (0.1 in CIFAR and SVHN, and 0.2 in Mini-ImageNet), dividing it by ten twice during training. We train for CIFAR and Mini-ImageNet 400 epochs (reducing learning rate in epochs 250 and 350) and use 10 epoch warm-up with labeled data, while for SVHN we train 150 epochs (reducing learning rate in epochs 50 and 100) and use a longer warm-up of 150 epochs to start the pseudo-labeling with good predictions and leading to reliable convergence. We do not attempt careful tuning of the regularization weights λ A and λ H and just set them to 0.8 and 0.4 as done in (see Appendix A.2 for an ablation study of these parameters). When using dropout, it is introduced between consecutive convolutional layers of ResNet blocks in WR-28, PR-18, and RN-18, while for 13-CNN we introduce it as in. Following 1, we use weight normalization in all networks. This section demonstrates that carefully regularized pseudo-labeling is a suitable alternative for SSL. Figure 1 illustrates our approach on the "two moons" toy data. Figure 1 (left) shows the limitations of a naive pseudo-labeling adapted from , which fails to adapt to the structure in the unlabelled examples and in a linear decision boundary. Figure 1 (middle) shows the effect of mixup, which alleviates confirmation bias to better model the structure and gives a smoother boundary. Figure 1 (right) shows that combining mixup with a minimum number of labeled samples k per mini-batch improves the semi-supervised decision boundary. Naive pseudo-labeling leads to overfitting the network predictions and high training accuracy in CIFAR-10/100. Table 1 (left) reports mixup effect in terms of validation error. Naive pseudo-labeling Figure 1: Pseudo-labeling in the "two moons" data (4 labels/class) for 1000 samples. From left to right: no mixup, mixup, and mixup with a minimum number of labeled samples per mini-batch. We use an NN classifier with one hidden layer with 50 hidden units as in . leads to an error of 11.40/48.54 for CIFAR-10/100 when training with cross-entropy (C) loss for 4000 labels. This error can be greatly reduced when using mixup (M) to 7.16/41.80. However, when further reducing the number of labels to 500 in CIFAR-10, M is insufficient to ensure low-error (32.10). We propose to set a minimum number of samples k per mini-batch to tackle the problem. Table 1 (right) studies this parameter k when combined with mixup, showing that 16 samples per mini-batch works well for both CIFAR-10 and CIFAR-100, dramatically reducing error in all cases (e.g. in CIFAR-10 for 500 labels error is reduced from 32.10 to 13.68). Confirmation bias causes a dramatic increase in the certainty of incorrect predictions during training. To demonstrate this behavior we compute the average cross-entropy of the softmax output with a uniform U across the classes in every epoch t for all incorrectly predicted samples {x mt} as: Figure 2 shows that mixup and minimum k are effective regularizers for reducing r t, i.e. confirmation bias is reduced. We also experimented with using label noise regularizations , but setting a minimum k proved more effective. There are examples in the recent literature where moving from one architecture to another modifies the belief of which methods have a higher potential. show that skip-connections in ResNet architectures play a key role on the quality of learned representations, while most approaches in previous literature were systematically evaluated using AlexNet (A.). showed that different architectures lead different and useful image priors, highlighting the importance of exploring different networks. We, therefore, test our method with two more architectures: a Wide ResNet-28-2 (WR-28) (S.) typically used in SSL (1.5M parameters) and a PreAct ResNet-18 (PR-18) (b) used in the context of label noise ) (11M parameters). Table 2 presents the for the 13-CNN (AlexNet-type) and these network architectures (ResNet-type). Our pseudo-labeling with mixup and k = 16 (M*) works well for 4000 and 500 labels across architectures, except for 500 labels for WR-28 where there is large error increase (29.50). This is due to a stronger confirmation bias in which labeled samples are not properly learned, while incorrect pseudo-labels are fit. Interestingly, PR-18 (11M) is more robust to confirmation bias than WR-28 (1.5M), while the 13-layer network (3M) has fewer parameters than PR-18 and achieves better performance. This suggests that the network architecture plays an important role, being a relevant prior for SSL with few labels. Figure 2: Example of certainty of incorrect predictions r t during training when using 500 (left) and 4000 (right) labeled images in CIFAR-10. Moving from cross-entropy (C) to mixup (M) reduces r t, whereas adding a minimum number of samples per mini-batch (*) also helps in 500 labels, where M* (with slightly lower r t than M) is the only configuration that converges, as shown in Table 1 (left). Table 2: Validation error across architectures is stabilized using dropout p and data augmentation (A). Labeled images 500 4000 500 4000 500 4000 500 4000 We found that dropout and data augmentation is needed for good performance across all architectures. Table 2 shows that dropout p = 0.1, 0.3 helps in achieving better convergence in CIFAR-10, whereas adding color jitter as additional data augmentation (details in Subsection 4.1) further contributes to error reduction. Note that the quality of pseudo-labels is key, so it is essential to disable dropout to prevent corruption when computing these in the second forward pass. We similarly disable data augmentation in the second forward pass, which consistently improves performance. This configuration is used for comparison with the state-of-the-art in Subsection 4.4. We compare our pseudo-labeling approach against related work that makes use of the 13-CNN in CIFAR-10/100: Π model , TE , MT , Π model-SN , MA-DNN , Deep-Co , TSSDL , LP , CCL , fast-SWA and ICT . Table 3 divides methods into those based on consistency regularization and pseudo-labeling. Note that we include pseudolabeling approaches combined with consistency regularization ones (e.g. MT) in the consistency regularization set. The proposed approach clearly outperforms consistency regularization methods, as well as other purely pseudo-labeling approaches and their combination with consistency regularization methods in CIFAR-10/100 and SVHN (250 labels are reported in Table 3 and extended are in the Appendix A.3). These demonstrate the generalization of the proposed approach compared to other methods that fail when decreasing the number of labels. Furthermore, Table 4 (left) demonstrates that the proposed approach successfully scales to higher resolution images, obtaining an over 10 point margin on the best related work in Mini-ImageNet. Note that all supervised baselines are reported using the same data augmentation and dropout as in the proposed pseudo-labeling. Table 4 (right) compares against recent consistency regularization approaches that use mixup. We achieve better performance than ICT , while being competitive with MM for 500 and 4000 labels using WR-28. Regarding PR-18, we converge to reasonable performance for 4000 and 500 labels, whereas for 250 we do not. Finally, the 13-CNN robustly converges even for 250 labels where we obtain 9.37 test error (see Appendix A.1 for some details on different architectures convergence). Therefore, these suggest that it is worth exploring the relationship between number of labels, dataset complexity and architecture type. As shown in Subsection 4.3, dropout and additional data augmentation help with 500 labels/class across architectures, but are insufficient for 250 labels. Better data augmentation or self-supervised pre-training might overcome this challenge. Furthermore, hyperparameters such as the regularization weights λ A = 0.8 and λ H = 0.4 from Eq. 4 and the mixup α require further study. However, it is already interesting that a straightforward modification of pseudo-labeling, designed to tackle confirmation bias, gives a competitive semi-supervised learning approach, without any consistency regularization, and future work should take this into account. This paper presented a semi-supervised learning approach for image classification based on pseudolabeling. We proposed to directly use the network predictions as soft pseudo-labels for unlabeled data together with mixup augmentation, a minimum number of labeled samples per mini-batch, dropout and data augmentation to alleviate confirmation bias. This conceptually simple approach outperforms related work in four datasets, demonstrating that pseudo-labeling is a suitable alternative to the dominant approach in recent literature: consistency-regularization. The proposed approach is, to the best of our knowledge, both simpler and more accurate than most recent approaches. Future work should explore SSL in class-unbalanced and large-scale datasets, synergies of pseudo-labelling and consistency regularization, and careful hyperparameter tuning. Figure 3 presents the cross-entropy loss for labeled samples when training with 13-CNN, WR-28 and PR-18 and using 500 and 250 labels in CIFAR-10. This loss is a good indicator of a robust convergence to reasonable performance as the interquartile range for cases failing (250 labels for WR-28 and PR-18) is much higher. This subsection studies the effect of α, λ A, and λ H hyperparameters of our pseudo-labeling approach. Tables 5 and 6 report the validation error in CIFAR-10 using 500 and 4000 labels for, respectively, α and λ A and λ H. Note that we keep the same configuration used in Subsection 4.2 with k = 16, i.e. no dropout or additional data augmentation is used. Table 5 suggest that α = 4 and α = 8 values might further improve the reported using α = 1. However, we experimented on CIFAR-10 with 500 labels using the final configuration (adding dropout and additional data augmentation) and observed marginal differences (8.54 with α = 4, which is within the error range of the 8.80 ± 0.45 obtained with α = 1), thus suggesting that stronger mixup regularization might not be additive to dropout and extra data augmentation in our case. Table 6 shows that our configuration (λ A = 0.8 and λ H = 0.4) adopted from is very close to the best performance in this experiment where marginal improvements are achieved. In , more careful hyperparameter tuning might slighltly improve reported in the paper, while the configuration selected is already good and generalizes across datasets. A.3 SVHN EXPPERIMENTS Table 7 reports a comparison of different state-of-the-art algorithms in SVHN using the 13-CNN network. We train 150 epochs using labeled samples (i.e. warm-up) to compute initial pseudo-labels. Then, we train 150 epochs reducing the learning rate in epochs 50 and 100. The long warm-up makes the method robust to lower levels of labeled samples in SVHN. We also experimented in CIFAR-10 with longer warm-up (our are reported using 10 epochs) and found that are in the same error range already reported.

Represent the TLDR for finding the corresponding scientific article.

Pseudo-labeling has shown to be a weak alternative for semi-supervised learning. We, conversely, demonstrate that dealing with confirmation bias with several regularizations makes pseudo-labeling a suitable approach.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance.