title: GREEK-BERT: The Greeks visiting Sesame Street
authors: John Koutsikakis Ilias Chalkidis Prodromos Malakasiotis Ion Androutsopoulos
text: 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  [1]
  
  Equal contribution.
  
  
  [jkoutsikakis, ihalk, rulller, ion]@aueb.gr
  
    Department of Informatics, Athens University of Economics and Business
  
  
  
  
  
  
  
  
  
  
  
  
    Transformer-based language models, such as and its variants, have achieved state-of-the-art performance in several downstream natural language processing () tasks on generic benchmark datasets (e.g., , , ). However, these models have mostly been applied to the resource-rich English language. In this paper, we present , a monolingual -based language model for modern Greek. We evaluate its performance in three tasks, i.e., part-of-speech tagging, named entity recognition, and natural language inference, obtaining state-of-the-art performance. Interestingly, in two of the benchmarks outperforms two multilingual Transformer-based models (, ), as well as shallower neural baselines operating on pre-trained word embeddings, by a large margin (5%-10%). Most importantly, we make both and our training code publicly available, along with code illustrating how can be fine-tuned for downstream tasks. We expect these resources to boost research and applications for modern Greek.
  
  
  
  
  
  
  
  
  <ccs2012>
     <concept>
         <concept_id>10010147.10010257.10010293.10010294</concept_id>
         <concept_desc>Computing methodologies Neural networks</concept_desc>
         <concept_significance>500</concept_significance>
         </concept>
     <concept>
         <concept_id>10010147.10010178.10010179.10010186</concept_id>
         <concept_desc>Computing methodologies Language resources</concept_desc>
         <concept_significance>500</concept_significance>
         </concept>
   </ccs2012>
  
  
  [500]Computing methodologies Neural networks
  [500]Computing methodologies Language resources
  
  
  
  
  
  
  
  
  
  
  GREEK-BERT: The Greeks visiting Sesame Street
      Ion Androutsopoulos
      Received 18 June 2020; accepted 24 August 2020
  ==================================================
  
  
  
  
  
  § INTRODUCTION
  
  Natural Language Processing () has entered its ImageNet  <cit.> era as advances in transfer learning have pushed the limits of the field in the last two years <cit.>. Pre-trained language models based on Transformers <cit.>, such as  <cit.> and its variants <cit.>, have achieved state-of-the-art results in several downstream tasks (e.g., text classification, natural language inference) on generic benchmark datasets, such as  <cit.>,  <cit.>, and  <cit.>. However, these models have mostly targeted the English language, for which vast amounts of data are readily available. Recently, multilingual language models (e.g., , , ) have been proposed <cit.> covering multiple languages, including modern Greek. While these models provide surprisingly good performance in zero-shot configurations (e.g., fine-tuning a pre-trained model in one language for a particular downstream task, and using the model in another language for the same task without further training), monolingual models, when available, still outperform them in most downstream tasks, with the exception of machine translation, where multilingualism is crucial. Consequently, Transformer-based language models have recently been adapted for, and applied to other languages <cit.> or even specialized domains (e.g., biomedical <cit.>, finance <cit.>, etc.) with promising results. To our knowledge, there are no publicly available pre-trained Transformer-based language models for modern Greek (hereafter simply Greek), for which much fewer resources are available, compared to English. Our main contributions are:
  
  
  
  
      
    * We introduce , a new monolingual pre-trained Transformer-based language model for Greek, similar to <cit.>, trained on 29 GB of Greek text with a 35k sub-word 
      vocabulary created from scratch.
      
    * We compare against multilingual language models based on Transformers 
      
      (, ) and other strong neural baselines operating on pre-trained word embeddings in three core downstream tasks, i.e., Part-of-Speech () tagging, Named Entity Recognition (), and Natural Language Inference (). achieves state-of-the-art results in all datasets, while outperforming its competitors by a large margin (5-10%) in two of them.
      
    * Most importantly, we make publicly available both the pre-trained 
      and our training code, along with code illustrating how can be fine-tuned for downstream tasks in Greek.[The pre-trained model is available at <https://huggingface.co/nlpaueb/bert-base-greek-uncased-v1>. The project is available at <https://github.com/nlpaueb/greek-bert>] We expect these resources to boost research and applications for Greek, since fine-tuning pre-trained Transformer-based language models for particular downstream tasks is the state-of-the-art. 
  
  
  
  
  § RELATED WORK
  
  
  
  
   §.§ BERT: A Transformer language model
  
  <cit.> introduced , a deep language model based on Transformers <cit.>,  which is pre-trained on pairs of sentences to learn to produce high quality context-aware representations of sub-word tokens and entire sentences (Fig. <ref>). Each sentence is represented as a sequence of s, a variation of s <cit.>, 
  while the special tokens and are used to indicate the start of the sequence and the end of a sentence, respectively. In other words, each pair of sentences has the form: <., , , ,  .>, where  is the first and  the second sentence of the pair. is pre-trained in two auxiliary self-supervised tasks: (a) Masked Language Modelling (), also called Denoising Language Modelling in the literature, where the model tries to predict masked-out (hidden) tokens based on the surrounding context, and (b) Next Sentence Prediction (), where the model uses the representation of to predict whether  immediately follows  or not, in the corpus they were taken from. The original English was pre-trained on two generic corpora, English Wikipedia and Children's Books <cit.> with a vocabulary of 32k 
  sub-words
  extracted from the same corpora. <cit.> originally released four English models. Two of them, BERT-BASE-UNCASED (12 layers of stacked Transformers, each of  768 hidden units, 12 attention heads, 110M parameters) and BERT-LARGE-UNCASED (24 layers, each of 1024 hidden units, 16 attention heads, 340M parameters), convert all text to lower-case. The other two models, BERT-BASE-CASED and BERT-LARGE-CASED, have the 
  exact same architectures as the corresponding previous models, but preserve character casing.
  
  
  
   §.§ Multilingual language models
  
  Most work on transfer learning for languages other than English focuses on multilingual language modeling to cover multiple languages at once. Towards that direction, <cit.>, a multilingual version of , supports 100 languages, including Greek. was pre-trained with the same auxiliary tasks as (, ), on the Wikipedias of the supported languages. Each pre-training sentence pair contains sentences from the same language, but now pairs from all 100 languages are used. To cope with multiple languages, relies on an extended shared vocabulary of 110k 
  sub-words. Only a small portion of the vocabulary (1,208 s or approx. 1%) applies to the Greek language, mainly because of the Greek alphabet. By contrast, languages that use the Roman alphabet (e.g., English, French) share more sub-words <cit.>, which as shown by <cit.> greatly improves the alignment of embedding spaces across these languages. has been mainly evaluated as a baseline for zero-shot cross-lingual training <cit.>.
  
  More recently, <cit.> introduced , a multilingual language model pre-trained on the Wikipedias of 15 languages, including Greek. They reported state-of-the-art results in supervised and unsupervised machine translation and cross-lingual classification. Similarly to , was trained in two auxiliary tasks, and the newly introduced Translation Language Modeling task (). is a supervised extension of , where each training pair contains two sentences,  and , from two different languages,  being the translation of . When a word of  (or ) is masked, the corresponding translation of that word in  (or ) is not masked. In effect, the model learns to align representations across languages. <cit.> introduced , which further improved the results of , without relying on the supervised , by using more training data from Common Crawl, and a larger vocabulary of 250k sub-words, covering 100 languages. Again, a small portion of the vocabulary covers the Greek language (4,862 sub-words, approx. 2%).
  
  
  
   §.§ Monolingual language models
  
  <cit.> released , a monolingual language model for French, based on <cit.>.  reported state-of-the-art results on four downstream tasks for French (tagging, dependency parsing, , ), outperforming and among other neural methods. <cit.> is another monolingual language model, for Finish, based on . It achieved state-of-the-art results, in PoS tagging, dependency parsing, , and text classification, outperforming among other models. Monolingual Transformer-based models have also been released for other languages (e.g., Italian, German, Spanish, etc.), showing strong performance in preliminary experiments. Most of them are still under development with no published work describing them.[An extensive list of monolingual Transformer-based models can be found at <https://huggingface.co/models>, along with preliminary results (when available).] 
  
  
  
   §.§ in Greek
  
  Publicly available resources for Greek continue to be very limited, compared to more widely spoken languages, although there have been several efforts to develop datasets, tools, and infrastructure for Greek .[Consult <https://www.clarin.gr/el/content/nlpel-clarin-knowledge-centre-natural-language-processing-greece> and <http://nlp.cs.aueb.gr/software.html> for examples.] Deep learning resources for Greek are even more limited. Recently, <cit.> presented an evaluation of Greek word embeddings, comparing Greek Word2Vec <cit.> models against the publicly available Greek FastText <cit.> model. Neural models that only pre-train word embeddings, however, have been largely superseded by deep pre-trained Transformer-based language models in the last two years. To the best of our knowledge, no Transformer-based pre-trained language model especially for Greek has been published to date. Thus, we aim to develop, study, and release such an important resource, as well as to provide an extensive evaluation across several tasks, comparing against state-of-the-art models. Part of our study compares against strong neural, in most cases -based, methods relying on pre-trained word embeddings. Such neural models are often not considered in other recent studies, without convincing justification. Instead, monolingual pre-trained Transformer-based language models are solely compared to multilingual pre-trained Transformer-based ones. We suspect that the latter, being usually biased towards more resource-rich languages (e.g., English, French, Spanish, etc.), may be outperformed by shallower neural models that rely only on pre-trained word embeddings. Hence, models of the latter kind may be stronger baselines for GREEK-BERT.
  
  
  
  § GREEK-BERT
  
  
  
  In this work, we present , a new monolingual version of for Greek. We use the architecture of BERT-BASE-UNCASED, because the larger BERT-LARGE-UNCASED architecture is computationally heavy for both pre-training and fine-tuning. We pre-trained on 29 GB of text from the following corpora: (a) the Greek part of Wikipedia;[An up-to-date dump can be found at <https://dumps.wikimedia.org/elwiki/>.] (b) the Greek part of the European Parliament Proceedings Parallel Corpus (Europarl) <cit.>; and (c) the Greek part of OSCAR <cit.>, a clean version of Common Crawl.[<https://commoncrawl.org>] Accents and other diacritics were removed, and all words were lower-cased to provide the widest possible normalization.
  The same corpora were used to extract a vocabulary of 35k s with the SentencePiece library <cit.>. Table <ref> presents the statistics of the pre-training corpora.
  To pre-train in the and tasks, we used the official code provided by Google.[<https://github.com/google-research/bert>] Similarly to <cit.>, we used 1M pre-training steps and the Adam optimizer <cit.> with an initial learning rate of 1e-4 on a single Google Cloud TPU v3-8.[The Google Cloud TPU v3-8 was provided for free by the TensorFlow Research Cloud (TFRC) program, to which we are grateful.] Pre-training took approx. 5 days.
  
  
  
  
  
  § BENCHMARKS
  
  
  
  We compare the performance of against strong baselines on datasets for three core downstream tasks.
  
  
  
   §.§ Part-of-Speech tagging
  
  
  For the first downstream task, tagging, we use the Greek Universal Dependencies Treebank (GUDT) <cit.>,[<https://github.com/UniversalDependencies/UD_Greek-GDT>.] which has been derived from the Greek Dependency Treebank,[<http://gdt.ilsp.gr>] a resource developed and maintained by the Institute for Language and Speech Processing, Research Center `Athena'.[<http://www.ilsp.gr/>] The dataset contains 2,521 sentences split in train (1,622), development (403), and test (456) sets. The sentences have been annotated with tags from a collection of 17 universal tags (UPoS).[Additional information on the curation of the dataset can be found at <https://universaldependencies.org/treebanks/el_gdt/index.html>.] We ignore the syntactic dependencies of the dataset, since we consider only tagging.
  
  
  
   §.§ Named Entity Recognition
  
  
  For the second downstream task, named entity recognition (), we use two currently unpublished datasets, developed by I. Darras and A. Romanou, during student projects at NTUA and AUEB, respectively.[The annotated dataset of I. Darras was part of his project for Google Summer of Code 2018 (<https://github.com/eellak/gsoc2018-spacy>), while A. Romanou annotated documents with named entities for another project (<http://greekner.me/info>). We are grateful to both for sharing their datasets.] 
  As both datasets, are fairly small, containing 1,798 and 2,521 sentences, we merged them and eliminated duplicate sentences. The merged dataset contains 4,189 unique sentences. 
  We use the Person, Organization, and Location annotations only, since the other entity types are not shared across the two datasets. 
  
  
  
   §.§ Natural Language Inference
  
  
  Finally, we experiment with the Cross-lingual Natural Language Inference corpus () <cit.>, which contains 5,000 test and 2,500 development pairs from the MultiNLI corpus <cit.>. Each pair consists of a premise and a hypothesis, and the task is to decide whether the premise entails (E), contradicts (C), or is neutral (N) to the hypothesis.
  The test and development pairs, originally in English, were manually classified (as E, C, N) by crowd-workers, and they were then manually translated by professional translators (using the One Hour Translation platform) to 14 languages, including Greek. The premises and hypotheses were translated separately, to ensure that no context is added to the hypothesis that was not there originally. Hence, each pair is available in 14 languages, always with the same class label.
  MultiNLI also has a training set of 340k pairs. In <cit.>, the training set of 340k pairs was automatically translated from English to the other languages, hence its quality is questionable; we discuss this further below. 
  Although has been mainly used as a cross-lingual test-bed for multilingual models <cit.>, we only consider its Greek part, i.e., we only use Greek pairs.
  
  
  
  § EXPERIMENTAL SETUP
  
  
  
  
  
   §.§ Multilingual models
  
  
  For each task and dataset, we compare against and both the cased and uncased versions of .[The vocabulary of retains both character casing and accents. We use the BASE version of to be comparable with the rest of models. The models are available at <https://huggingface.co/xlm-roberta-base>, <https://huggingface.co/bert-base-multilingual-cased>, and <https://huggingface.co/bert-base-multilingual-uncased>.] Recall that these models cover Greek with just a small portion of their sub-word vocabularies (approx. 2% for and 1% for ), which may cause excessive word fragmentation.
  
  
  
  Table <ref> reports the word fragmentation ratio, measured as the average number of sub-word tokens per word, in the three datasets for all Transformer-based language models considered. The multilingual models tend to fragment the words more than , especially variants, whose fragmentation ratio is approximately twice as large. has a lower fragmentation ratio than , as it has four times more sub-words covering Greek. 
  
  All three  multilingual models often over-fragment Greek words. For example, in `κατηγορουμενος'  becomes
  [`κ', `_ατ', `_η', `_γο', `_ρου', `_μενος'], and `γνωμη' 
  becomes [`γ', `_ν', `_ωμη'], but both words exist in 's vocabulary. We suspect that, despite the ability of sub-words to effectively prevent out-of-vocabulary word instances, such long sequences of meaningless sub-words may be difficult to re-assemble into meaningful units, even when using deep pre-trained models. By contrast, baselines that operate on embeddings of entire words do not suffer from this problem, which is one more reason to compare against them.
  
  
  
   §.§ Baselines operating on word embeddings
  
  For both tagging and , we experiment with an established neural sequence tagging model, dubbed , introduced by <cit.>. This model initially maps each word to the corresponding pre-trained embedding (e_i), as well as to an embedding (c_i) produced from the characters of the word by a Convolutional Neural Network (CNN). 
  The two embeddings of each word are then concatenated  (w_i=[e_i;c_i]). 
  Each text T is viewed as a sequence of embeddings <w_1,…,w_|T|>. A stacked bidirectional LSTM <cit.> turns the latter to a sequence of context-aware embeddings, which is fed to a linear Conditional Random Field (CRF) <cit.> layer to produce the final predictions. 
  
  For , we re-implemented the Decomposable Attention Model (DAM) <cit.>, which consists of an attention, a comparison, and an aggregation component. The attention component measures the importance of each word of the premise with respect to each hypothesis word and vice versa, as the normalized (via softmax) similarity of all possible pairs of words between the hypothesis and the premise. The similarities are calculated as the dot products of the corresponding word embeddings, which are first projected through a shared MLP layer. Finally, each word of the premise (or hypothesis) is represented by an attended embedding (a_i), which is simply the similarity weighted average of the embeddings of the hypothesis (or premise). The comparison component concatenates each a_i with the corresponding initial word embedding (e_i) and projects the new representation through an MLP. In effect, the premise and the hypothesis are each represented by a set of comparison vectors
  (P={v_1,…,v_p}, H={u_1,…,u_h}, respectively). Finally, the aggregation component uses an MLP classifier for the final prediction, which operates on the concatenation of s_p with s_h, where s_p=∑_v_i∈ Pv_i and s_h=∑_u_i∈ Hu_i.
  
  
  
   §.§ Implementation details and hyper-parameter tuning
  
  
  All the baselines that require pre-trained word embeddings use the Greek FastText <cit.> model[<https://fasttext.cc>] to obtain 300-dimensional word embeddings. The code for all experiments on downstream tasks is written in Python with the PyTorch[<https://pytorch.org>] framework using the PyTorch-Wrapper library [<https://github.com/jkoutsikakis/pytorch-wrapper>]. For the -based models, we use the Transformers library from HuggingFace <cit.>.[<https://github.com/huggingface/transformers>] The best architecture for each model is selected with grid search hyper-parameter tuning, minimizing the development loss, using early stopping without a maximum number of training epochs. For models, we tuned the learning rate considering the range {2e-5, 3e-5, 5e-5}, the dropout rate in the range {0, 0.1, 0.2}, and the batch size in {16, 32}. For , we used 2 stacked bidirectional LSTM layers and tuned the number of hidden units per layer in {100, 200, 300}, the learning rate in {1e-2, 1e-3}, the dropout rate in {0, 0.1, 0.2, 0.3}, and the batch size in {16, 32, 64}. Finally, for DAM we used 1 hidden layer with 200 hidden units for each MLP, and tuned the learning rate in {1e-2, 1e-3, 1e-4}, the dropout rate in {0, 0.1, 0.2, 0.3}, and the batch size in {16, 32, 64}. Given the best hyper-parameter values, we train each model 3 times with different random seeds and report the mean scores and unbiased standard deviation (over the 3 repetitions) on the test set of each dataset.
  
  
  
   §.§ Denoising XNLI training data
  
  
  As already discussed, includes 2,500 development pairs, 5,000 test pairs, and 340k training pairs, which have been translated from English to 14 languages, including Greek. The training pairs were machine-translated and, unfortunately, many of the resulting training pairs are of very low quality, which may harm performance. Based on this observation, we wanted to assess the effect of using the full training set, including many noisy pairs, against using a subset of the training set containing only high-quality pairs. We estimate the quality of a machine-translated pair as the perplexity of the concatenated sentences of the pair, computed by using as a language model, masking one of the two concatenated sentences at a time.[See Appendix A for a selection of random noisy samples and the best and worst samples according to .] We retain the 40k training pairs (approx. 10%) with the lowest (best) perplexity scores as the high-quality training subset.
  For comparison, we also train (fine-tune) our models on the full training set. Unlike all other experiments, when using the full training set we do not perform three iterations (with different random seeds), because of the size of the full training set, which makes repeating experiments computationally much more expensive.
  
  
  
  § EXPERIMENTAL RESULTS
  
  
  
  Table <ref> reports the tagging results. All Transformer-based models have comparable performance (97.8-98.2 accuracy), and is marginally (+0.1%) better than and . By contrast, performs clearly worse, but the difference from the other models is small (0.8-1.2%).
  The fact that all models achieve high scores can be explained by the observation that the correct tag of a word in Greek can often be determined by considering mostly the word's suffix, or for short function words (e.g., determiners, prepositions) the word itself, and to a lesser extent the word's context. Thus, even multi-lingual models with a high word fragmentation ratio (, ) are often able to guess the tags of words from their sub-words, even if the sub-words correspond to suffixes, other small parts of words, or frequent short words included in the sub-word vocabulary. Hence, there is often no need to consider the context of each word. The latter is difficult if context information gets scattered across too many very short sub-words.[In all multilingual models, each word is usually split in 2 or more sub-words and the last one resembles a suffix (e.g., `ανησυχιες' becomes [`ανησυχ', `_ιες'], `χαρακτηριζεται becomes [`χαρακτηρ', `_ιζεται']
  ), which often suffices to identify tags.] The method, which uses word embeddings, is also able to exploit information from suffixes, because it produces extra word embeddings from the characters of the words (using a CNN).
  
  
  
  For a more complete comparison we also report results per tag for the two best models, i.e., and (Table <ref>). We again observe that the two models have almost identical performance. Interestingly, both models have difficulties in predicting the tag other (X) and to a lesser extent proper nouns (PROPN) and numerals (NUM). In fact, both models tend to confuse these tags, which is reasonable considering that the inflectional morphology and context of Greek proper nouns is similar to that of common nouns, numerals (when written as words, e.g., `χιλιαδες') often have similar morphology with nouns, and words tagged as `other' (X) are often foreign proper nouns; for example, `καστρο' could either refer to `Fidel Castro' (X) or be the Greek noun for `castle'. Overall, there is still room for improvement in these particular tags.
  
  
  
  Table <ref> presents the results. We observe that outperforms the rest of the methods, by a large margin in most cases; it is 9.3% better than , 3.9% better than the two models on average, and 0.9% better than . The task is clearly more difficult than tagging, as evidenced by the near-perfect performance of all methods in tagging (Table <ref>), compared to the far from perfect performance of all methods in (Table <ref>). Being more difficult, the task leaves more space for better methods to distinguish themselves from weaker methods, and indeed clearly performs better than all the other methods, with being the second best method.
  
  
  
  In Table <ref>, we conduct a per entity type evaluation of the two best models. Both models are more accurate when predicting persons and locations. is better on persons, and marginally better on locations. Concerning organizations, is better, but both models struggle, because organizations often contain person names (`μπισκότα Παπαδοπούλου') or locations (`Αθλέτικο Μπιλμπάο'), shown in italics.
  
  
  
  Finally, Table <ref> shows that is again substantially better than the rest of the methods in , outperforming (+10.1%), the two models (+4.9% on average) and (+1.3%). Interestingly, performance improves for all models when trained on the entire training set, as opposed to training only on the high-quality 10% training subset, contradicting our assumption that noisy data could harm performance. Using a larger training set seems to be better than using a smaller one, even if the larger training set contains more noise. We suspect that noise may, in effect, be acting as a regularizer, improving the generalization ability of the models. A careful error analysis would shed more light on this phenomenon, but we leave this investigation for future work.
  
  
  
  As in the previous datasets, we report results per class for the two best models in Table <ref>. is better in all three classes, while both models have difficulties when predicting neutral pairs, which tend to be confused with pairs containing contradiction. 
  
  
  
  
  
  § CONCLUSIONS AND FUTURE WORK
  
  
  
  We presented , a new monolingual -based language model for modern Greek, which has been pre-trained on large modern Greek corpora and can be fine-tuned (further trained) for particular tasks. The new model achieves state-of-the-art performance in Greek tagging, named entity recognition, and natural language inference, outperforming strong baselines in the latter two, more difficult tasks. The baselines we considered included deep multilingual Transformer-based language models (, ), and shallower established neural methods (, DAM) operating on word embeddings. Most importantly, we release the pre-trained model, code to replicate our experiments, and code illustrating how can be fine-tuned for tasks. We expect that these resources will boost research and applications for Greek, a language for which public resources, especially for deep learning, are still scarce.
  
  In future work, we plan to pre-train another version of using even larger corpora. We plan to use the entire corpus of Greek legislation <cit.>, as published by the National Publication Office,[Available at <http://www.et.gr>.] and the entire Greek corpus of EU legislation, as published in Eur-Lex.[EU legislation is translated in the 24 official EU languages.] Both corpora include well-written Greek text describing policies across many different domains (e.g., economy, health, education, agriculture). Following <cit.>, who showed the importance of cleaning data when pre-training language models, we plan to discard noisy parts from all corpora, e.g., by filtering out documents containing tables or other non-natural text. We also plan to investigate the performance of in more downstream tasks, including dependency parsing, to the extent that more Greek datasets for downstream tasks will become publicly available. It would also be interesting to pre-train -based models for earlier forms of Greek, especially classical Greek, for which large datasets are available.[For example, <http://stephanus.tlg.uci.edu/>, <https://www.perseus.tufts.edu/hopper/>.] This could potentially lead to improved tools for classical studies.
  
  
  This project was supported by the TensorFlow Research Cloud (TFRC) program that provided a Google Cloud TPU v3-8 for free, while we also used free Google Cloud Compute (GCP) research credits. We are grateful to both Google programs.
  
  
  ACM-Reference-Format
  
  
  
  
  
  
  § EXAMPLES FROM THE GREEK PART OF XNLI CORPUS
  
  
  
  
  Table <ref> presents examples from the Greek part of XNLI including: (a) random noisy samples with morphology and syntax errors, (b) a selection from the best samples according to language modeling perplexity, and (c) a selection from the worst samples according to language modeling perplexity.
  
  