{'fraction_non_alphanumeric': 0.02967478191042815, 'fraction_numerical': 0.01168011096105413, 'mean_word_length': 4.777730915225643, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Out-of-the-Box and into the Ditch? Multilingual Evaluation of Generic Text Extraction Tools

This article examines extraction methods designed to retain the main text content of web pages and discusses how the extraction could be oriented and evaluated: can and should it be as generic as possible to ensure opportunistic corpus construction? The evaluation grounds on a comparative benchmark of open-source tools used on pages in five different languages (Chinese, English, Greek, Polish and Russian), it features several metrics to obtain more fine-grained differentiations. Our experiments highlight the diversity of web page layouts across languages or publishing countries. These discrepancies are reflected by diverging performances so that the right tool has to be chosen accordingly.


Web corpus construction
Large "offline" web corpora are now standard throughout disciplines among the research community. Corpus construction notably involves "crawling, downloading, 'cleaning' and de-duplicating the data, then linguistically annotating it and loading it into a corpus query tool." (Kilgarriff, 2007) Although text is ubiquitous on the Web, extracting information from web pages can prove to be difficult. They come in different shapes and sizes mostly because of the wide variety of platforms and content management systems, and not least depending on the context, for instance diverging goals followed during publication. This process involves a significant number of design decisions and turning points in data processing. Depending on the purpose of data collection, a substantial filtering and quality assessment can be crucial. Recently, approaches using the CommonCrawl 1 have flourished as they allow for faster download and processing by skipping (or more precisely outsourcing) the crawling phase (Habernal et al., 2016;Schäfer, 2016). Barring the fact that finding one's "own" way through the Web can be preferable, it is clear that such data should not be used without some filtering. Beside the discovery of relevant websites, a major issue consist in selecting appropriate content after download and processing (Schäfer et al., 2013), which may not be straightforward due to unexpected or machinegenerated flaws and biases. Some large-scale algorithms can be expected to smooth out irregularities. However, uses requiring a low margin of error and close reading approaches imply constant refinements in the constitution and processing of the dataset, for example in the context of an aggregated lexical information platform (Geyken et al., 2017). The potential lack of metadata is worsened by a lack of information regarding the content whose adequacy, focus and quality are the object of a post hoc evaluation (Baroni et al., 2009). A major challenge lies in the ability to extract and 1 https://commoncrawl.org pre-process web data to meet scientific expectations with respect to corpus quality (Barbaresi, 2015). Because of the vastly increasing variety of corpora, text types and use cases, it becomes more and more difficult to assess the usefulness and appropriateness of the gathered web texts for given research objectives. Potential answers can reside in methods such as focused web crawling for corpus construction (Schäfer et al., 2014) and in a degree of focus concerning the selection of sources (Barbaresi, 2016;Barbaresi, 2019). Regardless of the chosen construction method, an essential operation consists in retaining the desired content while discarding the rest, a polyonymous goal referring to peculiar subtasks or to the whole, most notably web scraping, boilerplate removal, web page segmentation, web page cleaning, or content extraction (Lejeune and Zhu, 2018). The variety of contexts and text genres leads to important design decisions during the collection of texts: could and should the tooling be adapted to particular sources that are targeted (which often amounts to the development of web scraping tools e.g. for news outlets) or should the extraction be as generic as possible to provide opportunistic ways of gathering information? Due to a lack of time resources in academia and elsewhere, the tools are considered as fieldtested without a thorough evaluation in vitro. This article hopefully makes a step towards the latter.

State of the art of content extraction
As the use of templates is pervasive on the Web ( Bar-Yossef and Rajagopalan, 2002), common approaches to main content detection include heuristic rules, machine learning on labeled training data, and indirectly template-based approaches (for example by identifying duplicated content) (Rae et al., 2018). Although text-based (Kohlschütter and Nejdl, 2008) and visual segmentation algorithms (Cai et al., 2003) have been published on, content extraction mostly draws on Document Object Model (DOM) examination (Gupta et al., 2003). That means considering a given HTML document as a tree structure whose nodes represent parts of the document to be operated on.
Text, tag and/or link density have proven to be good heuristics in order to select or discard content nodes, with approaches such as the Content Extraction via Tag Ratios (CETR) (Weninger et al., 2010) or the Content Extraction via Text Density (CETD) algorithms (Sun et al., 2011). Statistical selection of informative nodes through a combination of both methods proved more efficient on comparable datasets (Qureshi and Memon, 2012). Indeed, the large majority of DOM-based approaches try to leverage semantic information conveyed by HTML tags, notably paragraphs (p) on which text-to-tag ratios are calculated (Carey and Manic, 2016). An earlier, language-independent approach uses entropy measures applied to feature, links, and content in order to discriminate among parts of a webpage (Kao et al., 2004). Machine learning approaches have also been used, whose interest generally consists in leveraging advances in classification tasks by treating a HTML document as a series of blocks to be classified. Relevant algorithms notably include conditional random fields (CRF) learning header, text or noisy blocks using markup-based, content-based, and document-related features (Spousta et al., 2008), support vector machines (SVMs) trained on linguistic, structural and visual features (Bauer et al., 2007), or more recently deep learning, for example with convolutional neural networks (CNNs) learning combinations of DOM-based features (Vogels et al., 2018). Regarding the evaluation of extraction methods, the Cleaneval dataset and metrics (Baroni et al., 2008) have been used as a reference by numerous studies. Granularity and metrics used can have a real impact on results. Character and word-level metrics can be considered as a sequence, in a bag of words approach, or as a set and then ranked by F-score (Gottron, 2007). Web text extraction is not a solved task, user experience in general turns web content extraction into an active field of research, resulting from higher download and rendering speeds overall as well as from a growing tendency to inject content from a wide variety of sources, notably through the development of "reader modes" and "distillers" 2 for web browsers which strive to reduce the amount of "Web bloat" (Ghasemisharif et al., 2019). Furthermore, many existing algorithms have become somewhat obsolete due to the rapid changes in web technologies over the last 15 years (Weninger et al., 2016). Web page structure is also constantly evolving from the perspective of standards. HTML 5 was first released in 2008 to provide support for multimedia and graphical elements. This standard also streamlined syntax while retaining backward-compatibility. It also provided ways to tag the semantic content of documents with a granularity unseen before, with new page structure elements such as main, section, article, header, footer, aside, or nav. The standard has been gradually integrated into publishing practices and content management systems, while the recommendations still evolve, the current standard being HTML 5.2. 3 In addition, publication systems combining HTML code with embedded JavaScript 2 https://chromium.googlesource.com/chromium/dom-distiller 3 https://www.w3.org/TR/2017/REC-html52-20171214/ are on the rise, which also raises the question of "dry" and rendered page code. Last, there is a disciplinary gap between computer scientists and corpus linguists, both at the time of and following the "web as corpus" paradigm. As well as other research traditions sharing the Web as a research object without communicating much (Brügger and Laursen, 2019), both communities do not seem to be interconnected, although they could benefit from each other's results. We believe content extraction does not get the amount of attention it deserves in the corpus linguistics community. Additionally, precise metadata extraction is paramount in the humanities and remains a collateral issue of this disciplinary gap.

Contributions
Distinguishing between whole page and essential parts can help to alleviate many quality problems related to web texts. While this is particularly useful in the case of deduplication and studies relying on frequency-based information, other tasks related to content extraction also benefit from a cleaner text base. In the concrete case of linguistic and lexicographic research, it allows for content checks on the only portion of the document that really counts. In the following, we describe and evaluate text extraction tools published under open-source licenses and whose installation is straightforward. We perform a comparative benchmark on a multilingual setting consisting of realworld data with a manually annotated gold standard. We discuss the results as well as potentially suitable metrics to obtain more fine-grained differentiation. The insights of this paper are thus threefold in terms of software usability, benchmarking, and metrics.

Evaluation method
The evaluation described here focuses on integration and real-world usability of the tested solutions. As in previous evaluation campaigns we target the main content, which is usually the part displayed centrally, without the left or right bars, the header or the footer, but including potential titles and comments. We gathered tools coming from different research and industrial backgrounds, different countries, and developed during different time frames.

Tested solutions
The current benchmark focuses on the Python programming language which is reportedly the most popular programming language in academia 4 and one of the most popular overall. A few algorithms below are adapted from other languages such as Java and JavaScript, which contributes to giving an exhaustive yet incomplete panorama of available solutions overall. The following tools keep the structure intact but don't focus on main text extraction, they are kept in the benchmark to see how they perform in terms of recall, that is in order to measure how easy it would be to simply gather all the extractable text: • HTML2TEXT 5 performs text extraction • INSCRIPTIS 6 converts HTML to text with a particular emphasis on nested tables.
The following tools focus on main text extraction which is the task at hand: • BOILERPY3 7 is a Python version of the boilerpipe algorithm (Kohlschütter et al., 2010) for boilerplate removal and fulltext extraction; • DRAGNET 8 works as a meta-classifier using different methods weighted by machine learning (Peters and Lecocq, 2013), it requires more dependencies and potentially fine-tuning or re-training to work at its best; • GOOSE3 9 can extract information for embedded content but doesn't preserve markup; • JUSTEXT 10 is designed to preserve mainly text containing full sentences along with some markup, it has been explicitly developed to create linguistic resources (Pomikálek, 2011); • NEWSPAPER 11 is mostly geared towards newspaper texts, provides additional functions but no structured text or comment extraction • NEWS-PLEASE 12 is a news crawler that extracts structured information (Hamborg et al., 2017); • PYTHON-READABILITY 13 is a Python port of the Readability library used in Firefox to display distraction-free webpages, it cleans the page and preserves some markup.
The systems are used out-of-the-box or with minimal finetuning. Some of them come from an academic and others from an engineering or commercial background. Some are not being actively developed while others are still being updated. There is no reason to believe some would be disadvantaged as the pages they are tested on are anterior to their development. We use different pre-tuned configurations (here after mode) for the tools that offer this possibility: BOILERPY3 and JUSTEXT. All the code developed for this evaluations is available online. 14 In the results section we will use the following names for the tools:  • GOOSE for GOOSE3; • JT for JUSTEXT (default configuration), JT en for the English mode and JT langid for the language dependent mode; • NPAPER for NEWSPAPER; • NPLEASE for NEWS-PLEASE; • READ for Python-Readability.

Corpus
For our experiments we take advantage of the multilingual, human-annotated corpus DAnIEL, used previously for segmentation and event detection tasks (Lejeune et al., 2012) and extraction (Lejeune and Zhu, 2018). It comprises 1694 documents in five languages: Chinese, English, Greek, Polish and Russian. Each document is present as in its original HTML version and as a cleaned version with the text and some markup. To the best of our knowledge it is the largest multilingual corpus for evaluating web content extraction tools.
The documents have been collected in 2011 and 2012 to evaluate a text classification tool. The HTML 5 standard was not published as a W3C recommendation before 2014, thus it is to be expected that the documents analyzed here almost exclusively ground on HTML 4 which has been a reference since the end of the 1990s. We wish to compare the results of extrinsic evaluation (e.g. how does the web cleaning tool influence the result of classification) and intrinsic evaluation, e.g. to what extent the extracted content matches the expected outcome. We focus on the latter, not only to find the potentially "best" solution but also to provide more insights on the metrics and results of the evaluation. The dataset is available upon request. Table 1 shows some statistics on the corpus, the HTML original files and the manually curated clean versions. We can see two different families of tools: • Recall oriented tools such as HTML2TEXT, INSCRIP-TIS and BP3 KEEPE: they tend to extract much more data than expected • Precision-oriented tools (all the others) which are really devoted to avoid noise.   Table 3: Proportion of empty or almost empty (< 10% of the expected size) files for each language be sure that the output clearly does not fit the result one can expect from a text extractor. Obviously, the three tools of the recall-oriented family seldom output empty or almost empty files. Most tools seem to be primarily designed for English and not well-adapted to Chinese. We can see the importance of the JUSTEXT language models when compared to the English mode (JT EN). But the default configuration performs well, except in Chinese for which we had to adapt the configuration 15 . Because of differences in both data sample and processing it is important to choose appropriate metrics which can highlight disparities in tool efficiency. The metrics are described and discussed in the following section.

Processing time
We present in Table 4 the processing time for each tool.
There are noticeable differences between them, partly due to the fact that some tools go far beyond a mere text extraction, most notably NEWS-PLEASE. We included this information as it needs to be taken into account for users that 15 We followed the recommendations from the author: https://github.com/miso-belica/jusText/issues/12.  need to process data in real time or to clean big datasets but we won't discuss it thoroughly. We can see that DRAGNET and INSCRIPTIS seem to be the fastest systems, whereas language settings for JUSTEXT affect the results significantly.

Evaluation Metrics
Since the CLEANEVAL campaigns (Baroni et al., 2008), a state-of-the-art evaluation scheme has been set up and accepted by the community. This metric is based on the following assumption: a text is a sequence of tokens with or without HTML tags and a good content extraction solution should preserve this sequence. The proposition consists in matching the longest common subsequence between a gold standard version of the text and the result given by an extractor. While there are still unmatched zones, the algorithm recursively finds the next longest common subsequence in these zones. The insertion of a sequence not present in the Gold Standard is a False Positive. Conversely, a sequence that is missing in the result of the extractor is a False Negative. This proved to be convenient since classical metrics like recall, precision and f-score can be computed. However, this metric has some flaws. First of all, it has a quadratic complexity due to the use of the Ratcliff/Obershelp algorithm (Ratcliff and Metzener, 1988). Even on small datasets it is very slow. Secondly, it does not account properly for recall. For instance, copy-pasting the whole content of the document (e.g. with a very naive html-to-text tool) does not achieve 100% recall. As a consequence, we propose to use three additional metrics. Let GT be the Ground Truth and RES be the result of a given extractor and GT tok and RES tok be the sequence of their tokens. Let T P be the number of True Positives, F P the number of False Positives and F N the number of False Negatives.
In order to favor comparisons, the tokenization is produced by the exact same code as in CLEANEVAL except for Chinese where a segmentation in characters has been performed. 16 The first one, voc eval, simply compares the vocabulary of GT and RE: • Let GT voc be the set of GT tok and RES voc the set of RES tok The second one, occ eval compares the number of occurrences for each token.
• For each token t in GT tok : We also wish to apply other indicators in order to make other types of differences visible among all the tested tools. As such, we opt for two metrics: cosine and euclidean distance. These distances are regularly used for assessing the closeness between two documents (Platt et al., 2010;Buck and Koehn, 2016) , therefore we thought it could yield useful insights in this context. The last one (KL eval) uses the Kullback-Leibler divergence (a measure of relative entropy between two probability distributions): • V OC = GT voc ∪ RES tok (union of the vocabularies of GT and RES) • Let P gt (resp. P res ) be the probability distribution in GT (resp. RES) of each token of V OC • for all x in P gt (resp. P res ): if P gt (x) = 0 (resp.P res (x) = 0) * P gt (x) ← 10 −5 (resp. P res (x) ← 10 −5 ) • D KL (P g P res ) = − x∈X P (x) log

Pg(x) Pres(x)
The Kullback-Leibler divergence is not a distance metric since it is not symmetric but it is a way to measure how probability distributions diverge. In our case, we do not need a symmetric measure since we just want to account for the closeness with the GT probability distribution. The first two metrics allow us to compute recall, precision and f-score whereas KL eval yields a single measure: the smaller the divergence, the greater the similarity of the two documents. 16 See https://github.com/rundimeco/waddle Table 5 lists the results of each tool on the clean-eval evaluation scheme. The precision and recall are means, which is important for the interpretation since documentwise evaluation tends to favor systems that do not yield results much smaller that expected. The f-score is the classical version (with β = 1) computed on the mean precision and mean recall. We could also have chosen to compute a mean of the different f-scores but decided it would be strange to have a geometric mean of harmonic means. The first thing we can see is that BP3 is very efficient. READABILITY offers a slightly worse result but with a higher recall whereas JUSTEXT exhibits a drop in recall in comparison. DRAGNET has the highest precision score but with a recall below 60%. The recall-oriented tool family leads to lower scores but we can see that INSCRIPTIS is better than HTML2TEXT in both recall and precision. It seems to be a good tool for task when it is important to get as much content as possible.  The clean-eval measures for the quality of web page cleaning is widely used but it uses a convoluted algorithm relying on the alignment of sequences of words. Its rationale is quite straightforward: nobody wants to have a discontinuous version of the data or to have words in the wrong order. But it appears that in HTML code, the sequence of text blocks is in the same order as the original text. One can see there is not much difference between this evaluation and occ eval (Table 7). There are some differences in ranking concerning the voc eval metric (Table  6. Therefore, we can say that we can use the occ eval metric which has the advantage of being around ten times faster to compute. Table 8 shows the evaluation with cosine distance, euclidean distance and Kullback-Leibler divergence. Interestingly, this metric seems to be able to highlight systems that show a good balance between silence and noise (like READABILITY and JUSTEXT). Moreover, it does not penalize much systems with large recall scores (like INSCRIP-TIS or HTML2TEXT).   This is not surprising since, even with smoothing, this measure tends to favor close probabilities in the same order of magnitude, in other words P (x) = 1 * 10 −4 is closer to Q(x) = 3 * 10 −4 than R(x) = 1 * 10 −5 .

Results by language
The results on the five languages of the corpus describe major discrepancies between the tools. First of all, Table 9 shows the results obtained on English documents with the clean-eval metric and Table 10 the results for the occ eval metric. Again, we can see that occ eval yields comparable results. Since it is a simpler measure we will focus on this one for the remainder of the article.
One can see that the scores are much higher than the scores showed in Tables 5 and 7, which highlights that English is a very specific case. Our results demonstrate that most tools are primarily designed to process English documents. Furthermore, the tools that perform very well in this subcorpus are not as efficient on the multilingual corpus. So, one cannot rely on results evaluated solely on English to draw conclusions on the efficiency of a tool in real-world multilingual settings. Except the three recall-oriented tools, all yield an   Table 9: Evaluation with the clean-eval metric (documents in English) , sorted by descending f-score occ eval f-score of 80% and higher. NEWSPAPER outperforms the other tools with an f-score above 90%. GOOSE is slightly below and close to NEWSPLEASE but it is much faster (around 35 times according to Table 4). The three tools designed for readability (READABILITY itself but also NEWSPAPER and NEWS-PLEASE) all perform very well. Table 11 introduces the results on the Greek subcorpus. The three best tools perform comparably to the three top tools for English. It is interesting to see that the languagedependent JUSTEXT configuration yields results comparable to the default configuration. NEWSPAPER, GOOSE and obviously JT EN perform poorly on this subcorpus. It is obvious for the latter but it is astonishing that the other two do not perform well.      It seems harder to extract text from Russian pages since no system is able to achieve above 80% f-score (Table 13). Again, JUSTEXT is among the best performers. Contrary to the Polish subcorpus, it is BP3 Larg that is the best BP3 configuration. We can see again that READABILITY performs very well on other languages than English. Finally, the worst results are related to the Chinese subcorpus (Table 14). BP3 outperforms the rest of the field by far. One can see that the choice of a tool is much more important for Chinese than for English since many tools result in f-scores below 20%. We can note that it is the only language for which INSCRIPTIS does not achieve 90% recall.

Is there a winner?
The results we presented yield differentiated insights so that it is difficult to give a definitive and universal answer. First of all, if one targets recall and/or speed INSCRIPTIS is clearly the most efficient solution. In general BP3 and READABILITY are the most stable systems and the only ones that perform reasonably well for Chinese.
If we do not consider Chinese, JUSTEXT in its languageindependent setting seems to be the most efficient solution for multilingual corpora. That being said, this setting is much slower and it is not strictly comparable as it uses additional information but most of all it does not appear to perform better. For texts in English GOOSE and NEWSPA-PER outperform the other systems. For Polish, BP3 ART shows a comparable f-score than JUSTEXT but with a better precision. For Russian BP3 LARG is a good solution if one needs precision but JUSTEXT achieves a satisfying trade-off between precision and recall. According to our study, there appears to be no benefit from more intricate machine-learning approaches, DRAGNET does not stand out and does not perform poorly either. However, the amount of additional training data needed to potentially improve its results is a penalty in terms of usability compared to the other solutions for which parameter tuning could lead to improvements much faster. JUSTEXT is such an example where changing settings can be done easily.

Conclusions and outlook
The article focused on a comparative benchmark of opensource tools used on web documents from 2011 and 2012 written in five different languages, along with a discussion of suitable metrics. Content processing is affected by both diatopic and diachronic factors, whereas vocabulary analysis and distance metrics can yield more fine-grained information which complements the CLEANEVAL evaluation standard. Rule-based approaches appear to be more efficient in the long run, all the more since they are both easier to use and to parametrize. Most tools are developed with particular page styles in mind, mostly from the English-speaking world. Our data shows that linguistic factors are most probably reflected in HTML structures, which deeply affects extraction processes. The experiments above highlight the diversity of layouts and web coding practices depending on language and most probably on the country from which a document is published. These discrepancies are reflected by diverging performances so that the right tool has to be chosen accordingly.
In addition, different eras of web development result in diverging "HTMLects". Our corpus provides a snapshot of a past version of the Web which proves to be challenging for some tools. As such, it is useful to assess how data from Web archives can be processed. These findings prompt for further studies on the evaluation of tool robustness with respect to the ever-changing Web. We have reasons to believe that the success of standardized publishing platforms and the consecutive advent of HTML 5 changes the way text is published on the Web, all of which could pave the way for further examinations.