page_header
arXiv:2303.03915v1 [cs.CL] 7 Mar 2023
#/body
BOTTOMLEFT
#/texts/0
arXiv:2303.03915v1 [cs.CL] 7 Mar 2023
section_header
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
#/body
BOTTOMLEFT
#/texts/1
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
text
Hugo Laurençon 1 ∗ Lucile Saulnier 1 ∗ Thomas Wang 1 ∗ Christopher Akiki 2 ∗ Albert Villanova del Moral 1 ∗ Teven Le Scao 1*
#/body
BOTTOMLEFT
#/texts/2
Hugo Laurençon 1 ∗ Lucile Saulnier 1 ∗ Thomas Wang 1 ∗ Christopher Akiki 2 ∗ Albert Villanova del Moral 1 ∗ Teven Le Scao 1*
text
Leandro von Werra 1 Chenghao Mou 3 Eduardo González Ponferrada 4 Huu Nguyen 5 Jörg Frohberg 32 Mario Šaško 1 Quentin Lhoest 1
#/body
BOTTOMLEFT
#/texts/3
Leandro von Werra 1 Chenghao Mou 3 Eduardo González Ponferrada 4 Huu Nguyen 5 Jörg Frohberg 32 Mario Šaško 1 Quentin Lhoest 1
text
Angelina McMillan-Major 1 , 6 Gérard Dupont 7 Stella Biderman 8 , 9 Anna Rogers 10 Loubna Ben allal 1 Francesco De Toni 11 Giada Pistilli 1 Olivier Nguyen 28 Somaieh Nikpoor 12 Maraim Masoud 13 Pierre Colombo 14 Javier de la Rosa 15 Paulo Villegas 16 Tristan Thrush 1 Shayne Longpre 17 Sebastian Nagel 19 Leon Weber 20 Manuel Romero Muñoz 21 Jian Zhu 22 Daniel van Strien 23 Zaid Alyafeai 24 Khalid Almubarak 25 Vu Minh Chien 26 Itziar Gonzalez-Dios 27 Aitor Soroa 27 Kyle Lo 29 Manan Dey 30 Pedro Ortiz Suarez 31 Aaron Gokaslan 18 Shamik Bose 3 David Ifeoluwa Adelani 33 Long Phan 34 Hieu Tran 34 Ian Yu 35 Suhas Pai 36 Jenny Chim 37
#/body
BOTTOMLEFT
#/texts/4
Angelina McMillan-Major 1 , 6 Gérard Dupont 7 Stella Biderman 8 , 9 Anna Rogers 10 Loubna Ben allal 1 Francesco De Toni 11 Giada Pistilli 1 Olivier Nguyen 28 Somaieh Nikpoor 12 Maraim Masoud 13 Pierre Colombo 14 Javier de la Rosa 15 Paulo Villegas 16 Tristan Thrush 1 Shayne Longpre 17 Sebastian Nagel 19 Leon Weber 20 Manuel Romero Muñoz 21 Jian Zhu 22 Daniel van Strien 23 Zaid Alyafeai 24 Khalid Almubarak 25 Vu Minh Chien 26 Itziar Gonzalez-Dios 27 Aitor Soroa 27 Kyle Lo 29 Manan Dey 30 Pedro Ortiz Suarez 31 Aaron Gokaslan 18 Shamik Bose 3 David Ifeoluwa Adelani 33 Long Phan 34 Hieu Tran 34 Ian Yu 35 Suhas Pai 36 Jenny Chim 37
text
Violette Lepercq 1
#/body
BOTTOMLEFT
#/texts/5
Violette Lepercq 1
text
Suzana Ili'c 1
#/body
BOTTOMLEFT
#/texts/6
Suzana Ili'c 1
text
Margaret Mitchell 1
#/body
BOTTOMLEFT
#/texts/7
Margaret Mitchell 1
text
Sasha Luccioni 1
#/body
BOTTOMLEFT
#/texts/8
Sasha Luccioni 1
text
Yacine Jernite 1
#/body
BOTTOMLEFT
#/texts/9
Yacine Jernite 1
text
$^{1}$Hugging Face $^{2}$Leipzig University $^{3}$Independent Researcher $^{4}$Ferrum Health $^{5}$Ontocord.ai $^{6}$University of Washington $^{7}$Mavenoid $^{8}$EleutherAI $^{9}$Booz Allen Hamilton $^{10}$University of Copenhagen $^{11}$University of Western Australia $^{12}$CAIDP $^{13}$Independent Researcher $^{14}$CentraleSupélec $^{15}$National Library of Norway $^{16}$Telefonica I+D $^{17}$MIT $^{18}$Cornell University $^{19}$Common Crawl $^{20}$Humboldt-Universität zu Berlin and Max Delbrück Center for Molecular Medicine $^{21}$Narrativa $^{22}$University of Michigan, Ann Arbor $^{23}$British Library $^{24}$King Fahd University of Petroleum and Minerals $^{25}$Prince Sattam bin Abdulaziz University (PSAU) $^{26}$DETOMO Inc. $^{27}$HiTZ Center, University of the Basque Country (UPV/EHU) $^{28}$ServiceNow $^{29}$Allen Institute for AI $^{30}$SAP $^{31}$Mannheim University $^{32}$Apergo.ai $^{33}$Saarland University $^{34}$VietAI Research $^{35}$Aggregate Intellect $^{36}$Bedrock AI $^{37}$Queen Mary University of London
#/body
BOTTOMLEFT
#/texts/10
$^{1}$Hugging Face $^{2}$Leipzig University $^{3}$Independent Researcher $^{4}$Ferrum Health $^{5}$Ontocord.ai $^{6}$University of Washington $^{7}$Mavenoid $^{8}$EleutherAI $^{9}$Booz Allen Hamilton $^{10}$University of Copenhagen $^{11}$University of Western Australia $^{12}$CAIDP $^{13}$Independent Researcher $^{14}$CentraleSupélec $^{15}$National Library of Norway $^{16}$Telefonica I+D $^{17}$MIT $^{18}$Cornell University $^{19}$Common Crawl $^{20}$Humboldt-Universität zu Berlin and Max Delbrück Center for Molecular Medicine $^{21}$Narrativa $^{22}$University of Michigan, Ann Arbor $^{23}$British Library $^{24}$King Fahd University of Petroleum and Minerals $^{25}$Prince Sattam bin Abdulaziz University (PSAU) $^{26}$DETOMO Inc. $^{27}$HiTZ Center, University of the Basque Country (UPV/EHU) $^{28}$ServiceNow $^{29}$Allen Institute for AI $^{30}$SAP $^{31}$Mannheim University $^{32}$Apergo.ai $^{33}$Saarland University $^{34}$VietAI Research $^{35}$Aggregate Intellect $^{36}$Bedrock AI $^{37}$Queen Mary University of London
text
∗ Equal contributions
#/body
BOTTOMLEFT
#/texts/11
∗ Equal contributions
section_header
Abstract
#/body
BOTTOMLEFT
#/texts/12
Abstract
text
As language models grow ever larger, the need for large-scale high-quality text datasets has never been more pressing, especially in multilingual settings. The BigScience workshop, a 1-year international and multidisciplinary initiative, was formed with the goal of researching and training large language models as a values-driven undertaking, putting issues of ethics, harm, and governance in the foreground. This paper documents the data creation and curation efforts undertaken by BigScience to assemble the Responsible Open-science Open-collaboration Text Sources ( ROOTS ) corpus, a 1.6TB dataset spanning 59 languages that was used to train the 176-billion-parameter BigScience Large Open-science Open-access Multilingual ( BLOOM )(BigScience Workshop, 2022) language model. We further release a large initial subset of the corpus and analyses thereof, and hope to empower large-scale monolingual and multilingual modeling projects with both the data and the processing tools, as well as stimulate research around this large multilingual corpus.
#/body
BOTTOMLEFT
#/texts/13
As language models grow ever larger, the need for large-scale high-quality text datasets has never been more pressing, especially in multilingual settings. The BigScience workshop, a 1-year international and multidisciplinary initiative, was formed with the goal of researching and training large language models as a values-driven undertaking, putting issues of ethics, harm, and governance in the foreground. This paper documents the data creation and curation efforts undertaken by BigScience to assemble the Responsible Open-science Open-collaboration Text Sources ( ROOTS ) corpus, a 1.6TB dataset spanning 59 languages that was used to train the 176-billion-parameter BigScience Large Open-science Open-access Multilingual ( BLOOM )(BigScience Workshop, 2022) language model. We further release a large initial subset of the corpus and analyses thereof, and hope to empower large-scale monolingual and multilingual modeling projects with both the data and the processing tools, as well as stimulate research around this large multilingual corpus.
page_footer
36th Conference on Neural Information Processing Systems (NeurIPS 2022) Track on Datasets and Benchmarks.
#/body
BOTTOMLEFT
#/texts/14
36th Conference on Neural Information Processing Systems (NeurIPS 2022) Track on Datasets and Benchmarks.
section_header
Contents
#/body
BOTTOMLEFT
#/texts/15
Contents
page_footer
2
#/body
BOTTOMLEFT
#/texts/16
2
caption
Figure 1: Overview of ROOTS. Left: A treemap of natural language representation in number of bytes by language family. The bulk of the graph is overwhelmed by the 1321.89 GB allotted to Eurasia. The orange rectangle corresponds to the 18GB of Indonesian, the sole representative of the Papunesia macroarea, and the green rectangle to the 0.4GB of the Africa linguistic macroarea. Right: A waffle plot of the distribution of programming languages by number of files. One square corresponds approximately to 30,000 files.
#/body
BOTTOMLEFT
#/texts/17
Figure 1: Overview of ROOTS. Left: A treemap of natural language representation in number of bytes by language family. The bulk of the graph is overwhelmed by the 1321.89 GB allotted to Eurasia. The orange rectangle corresponds to the 18GB of Indonesian, the sole representative of the Papunesia macroarea, and the green rectangle to the 0.4GB of the Africa linguistic macroarea. Right: A waffle plot of the distribution of programming languages by number of files. One square corresponds approximately to 30,000 files.
section_header
1 Introduction
#/body
BOTTOMLEFT
#/texts/18
1 Introduction
text
BigScience 1 started in May 2021 as a one-year long open collaborative research initiative that gathered over a thousand participants around the world to study large language models (LLM). One of the founding goals of BigScience was to train an open-access, massively multilingual LLM, comparable in scale to GPT-3 (Brown et al., 2020) yet trained on a better documented and more representative multilingual dataset. The overall BigScience workshop was designed as a collaborative (Caselli et al., 2021; Bondi et al., 2021) and value-driven (Birhane et al., 2021) endeavor. Throughout the process of building this corpus we engaged in simultaneous investigation of ethical (Talat et al., 2022), sociopolitical (McMillan-Major et al., 2022), and data governance issues (Jernite et al., 2022) with the explicit goal of doing good for and by the people whose data we collected.
#/body
BOTTOMLEFT
#/texts/19
BigScience 1 started in May 2021 as a one-year long open collaborative research initiative that gathered over a thousand participants around the world to study large language models (LLM). One of the founding goals of BigScience was to train an open-access, massively multilingual LLM, comparable in scale to GPT-3 (Brown et al., 2020) yet trained on a better documented and more representative multilingual dataset. The overall BigScience workshop was designed as a collaborative (Caselli et al., 2021; Bondi et al., 2021) and value-driven (Birhane et al., 2021) endeavor. Throughout the process of building this corpus we engaged in simultaneous investigation of ethical (Talat et al., 2022), sociopolitical (McMillan-Major et al., 2022), and data governance issues (Jernite et al., 2022) with the explicit goal of doing good for and by the people whose data we collected.
text
Sourcing and building the dataset was organized around four working groups: Data Governance which helped define the project's values and design our approach to data usage and release in an international context, Data Sourcing and Preparation which was tasked with overseeing data collection, curation efforts, and Privacy for privacy risks and sanitizing the dataset, Legal Scholarship which helped define the multi-jurisdiction legal context in which the entire workshop was to operate, and we discuss practical implications throughout the paper where appropriate. An overview of the BigScience Corpus is provided in figure 1.
#/body
BOTTOMLEFT
#/texts/20
Sourcing and building the dataset was organized around four working groups: Data Governance which helped define the project's values and design our approach to data usage and release in an international context, Data Sourcing and Preparation which was tasked with overseeing data collection, curation efforts, and Privacy for privacy risks and sanitizing the dataset, Legal Scholarship which helped define the multi-jurisdiction legal context in which the entire workshop was to operate, and we discuss practical implications throughout the paper where appropriate. An overview of the BigScience Corpus is provided in figure 1.
text
The goal of the current paper is twofold: (1) we present a preliminary gated, subject to committing to the BigScience ethical charter$^{2}$, release of a large subset of ROOTS 3 (2) we release the numerous data tools 4 that were developed along the way and enabled us to curate, source, clean and inspect all 498 constituent datasets that come together to constitute ROOTS. This includes a preliminary results of the analyses that are currently being developed to study the corpus.
#/body
BOTTOMLEFT
#/texts/21
The goal of the current paper is twofold: (1) we present a preliminary gated, subject to committing to the BigScience ethical charter$^{2}$, release of a large subset of ROOTS 3 (2) we release the numerous data tools 4 that were developed along the way and enabled us to curate, source, clean and inspect all 498 constituent datasets that come together to constitute ROOTS. This includes a preliminary results of the analyses that are currently being developed to study the corpus.
section_header
1.1 Outline of the Paper
#/body
BOTTOMLEFT
#/texts/22
1.1 Outline of the Paper
text
The remainder of this paper details our approach to curating a web-scale dataset covering 59 languages, 46 natural languages and 13 programming languages - the language choice was chiefly driven by the communities who participated in the effort given the importance we placed on language expertise. Our final corpus is made up of two main components: 62% of the text comes from a community-selected and documented list of language data sources and its collection process is described in section 2, and
#/body
BOTTOMLEFT
#/texts/23
The remainder of this paper details our approach to curating a web-scale dataset covering 59 languages, 46 natural languages and 13 programming languages - the language choice was chiefly driven by the communities who participated in the effort given the importance we placed on language expertise. Our final corpus is made up of two main components: 62% of the text comes from a community-selected and documented list of language data sources and its collection process is described in section 2, and
footnote
$^{1}$https://bigscience.huggingface.co/
#/body
BOTTOMLEFT
#/texts/24
$^{1}$https://bigscience.huggingface.co/
footnote
$^{2}$https://hf.co/spaces/bigscience/ethical-charter
#/body
BOTTOMLEFT
#/texts/25
$^{2}$https://hf.co/spaces/bigscience/ethical-charter
footnote
$^{3}$https://hf.co/bigscience-data
#/body
BOTTOMLEFT
#/texts/26
$^{3}$https://hf.co/bigscience-data
footnote
$^{4}$https://github.com/bigscience-workshop/data-preparation
#/body
BOTTOMLEFT
#/texts/27
$^{4}$https://github.com/bigscience-workshop/data-preparation
page_footer
3
#/body
BOTTOMLEFT
#/texts/28
3
text
38% consists of text extracted from a pre-processed web crawl, OSCAR (Ortiz Suárez et al. (2020)), filtered with the help of native speakers, which is described in section 3.
#/body
BOTTOMLEFT
#/texts/29
38% consists of text extracted from a pre-processed web crawl, OSCAR (Ortiz Suárez et al. (2020)), filtered with the help of native speakers, which is described in section 3.
section_header
1.2 Related Work
#/body
BOTTOMLEFT
#/texts/30
1.2 Related Work
text
Large Language Models and Large Text Corpora The current dominant paradigm in natural language processing relies heavily on pre-trained models: large language models that can then be fine-tuned on a downstream task (Howard and Ruder, 2018; Devlin et al., 2018) or even used as-is without additional data (Radford et al., 2019; Brown et al., 2020). In this paradigm, performance is directly correlated on both the model size and the dataset size and quality (Kaplan et al., 2020), with recent models trained on up to 1.4 trillion tokens (Hoffmann et al., 2022) and dataset creation pipelines representing a significant part of large language model projects. Most such datasets, however, are not released, hindering further research. Exceptions include the Pile (Gao et al., 2020), a curated corpus of datasets for language modeling that has become widely used for training state-of-the-art English-language models (Lieber et al., 2021; Smith et al., 2022; Black et al., 2022; Zhang et al., 2022), and C4 and mC4 (Raffel et al., 2020; Xue et al., 2020), which have powered the T5 family of models; CC100 (Conneau et al., 2020) which has seen heavy use for multilingual modeling; and OSCAR (Ortiz Suárez et al., 2019), which has enabled monolingual non-English models.
#/body
BOTTOMLEFT
#/texts/31
Large Language Models and Large Text Corpora The current dominant paradigm in natural language processing relies heavily on pre-trained models: large language models that can then be fine-tuned on a downstream task (Howard and Ruder, 2018; Devlin et al., 2018) or even used as-is without additional data (Radford et al., 2019; Brown et al., 2020). In this paradigm, performance is directly correlated on both the model size and the dataset size and quality (Kaplan et al., 2020), with recent models trained on up to 1.4 trillion tokens (Hoffmann et al., 2022) and dataset creation pipelines representing a significant part of large language model projects. Most such datasets, however, are not released, hindering further research. Exceptions include the Pile (Gao et al., 2020), a curated corpus of datasets for language modeling that has become widely used for training state-of-the-art English-language models (Lieber et al., 2021; Smith et al., 2022; Black et al., 2022; Zhang et al., 2022), and C4 and mC4 (Raffel et al., 2020; Xue et al., 2020), which have powered the T5 family of models; CC100 (Conneau et al., 2020) which has seen heavy use for multilingual modeling; and OSCAR (Ortiz Suárez et al., 2019), which has enabled monolingual non-English models.
text
Tooling, Visualization, and Replication Upstream from the finalized training datasets is the issue of processing methods and pipelines: both the operations that the datasets go through and the engineering effort required to apply them at terabyte scales. Existing work tends to fall on a spectrum from no details at all (Brown et al., 2020) to detailed filtering instructions, with (Raffel et al., 2020) or without the dataset release (Rae et al., 2021) to detailed filtering instructions with the accompanying code (Gao et al., 2020; Conneau et al., 2020; Ortiz Suárez et al., 2019). Even when the code is released, it tends to be built and tailored for the project's purpose. Consequently, large projects that do not re-use an existing dataset outright usually build their own pipeline rather than re-use an existing one on new data. However, data tools that were built and packaged in order to be used for other projects exist, such as OSCAR's Ungoliant and Goclassy (Abadji et al., 2021; Ortiz Suárez et al., 2019), which provides a distributed Common Crawl processing pipeline; CCNet (Wenzek et al., 2020), built for quality filtering of multilingual Common Crawl dumps; and OpenWebText (Gokaslan and Cohen, 2019), enabling Reddit dump processing.
#/body
BOTTOMLEFT
#/texts/32
Tooling, Visualization, and Replication Upstream from the finalized training datasets is the issue of processing methods and pipelines: both the operations that the datasets go through and the engineering effort required to apply them at terabyte scales. Existing work tends to fall on a spectrum from no details at all (Brown et al., 2020) to detailed filtering instructions, with (Raffel et al., 2020) or without the dataset release (Rae et al., 2021) to detailed filtering instructions with the accompanying code (Gao et al., 2020; Conneau et al., 2020; Ortiz Suárez et al., 2019). Even when the code is released, it tends to be built and tailored for the project's purpose. Consequently, large projects that do not re-use an existing dataset outright usually build their own pipeline rather than re-use an existing one on new data. However, data tools that were built and packaged in order to be used for other projects exist, such as OSCAR's Ungoliant and Goclassy (Abadji et al., 2021; Ortiz Suárez et al., 2019), which provides a distributed Common Crawl processing pipeline; CCNet (Wenzek et al., 2020), built for quality filtering of multilingual Common Crawl dumps; and OpenWebText (Gokaslan and Cohen, 2019), enabling Reddit dump processing.
text
Documenting Textual Corpora in NLP An inspiration for our work is a recent emphasis on a more in-depth documentation of what is included and what is not in the corpora used for training NLP models . The most notable example of this is the Pile, for which the authors themselves analyze and document a variety of syntactic and semantic properties of the dataset including structural statistics (n-gram counts, language, document sizes), topical distributions across its components, social bias and sentiment co-occurrence, pejorative content, and information about licensing and authorial consent, in addition to releasing a datasheet (Biderman et al., 2022). Other LM pre-training datasets that have been documented and analyzed include C4 (Dodge et al., 2021; Luccioni and Viviano, 2021; Kreutzer et al., 2022), OSCAR (Kreutzer et al., 2022) and BookCorpus (Bandy and Vincent, 2021) . While this kind of documentation is far from standard practice, it is becoming increasingly common given recent calls for better documentation (Rogers, 2021; Bender et al., 2021) as well as empirical studies on data memorization in language models (Carlini et al., 2019, 2022).
#/body
BOTTOMLEFT
#/texts/33
Documenting Textual Corpora in NLP An inspiration for our work is a recent emphasis on a more in-depth documentation of what is included and what is not in the corpora used for training NLP models . The most notable example of this is the Pile, for which the authors themselves analyze and document a variety of syntactic and semantic properties of the dataset including structural statistics (n-gram counts, language, document sizes), topical distributions across its components, social bias and sentiment co-occurrence, pejorative content, and information about licensing and authorial consent, in addition to releasing a datasheet (Biderman et al., 2022). Other LM pre-training datasets that have been documented and analyzed include C4 (Dodge et al., 2021; Luccioni and Viviano, 2021; Kreutzer et al., 2022), OSCAR (Kreutzer et al., 2022) and BookCorpus (Bandy and Vincent, 2021) . While this kind of documentation is far from standard practice, it is becoming increasingly common given recent calls for better documentation (Rogers, 2021; Bender et al., 2021) as well as empirical studies on data memorization in language models (Carlini et al., 2019, 2022).
section_header
2 (Crowd) Sourcing a Language Resource Catalogue
#/body
BOTTOMLEFT
#/texts/34
2 (Crowd) Sourcing a Language Resource Catalogue
text
The first part of our corpus, accounting for 62% of the final dataset size (in bytes), was made up of a collection of monolingual and multilingual language resources that were selected and documented collaboratively through various efforts of the BigScience Data Sourcing working group. The first such effort consisted in creating a tool to support metadata collection through open submissions, called the BigScience Catalogue and running a series of hackathons in collaboration with locally-focused ML and NLP communities such as Masakhane, Machine Learning Tokyo and LatinX in AI where participants could add and document entries for their languages to the catalogue (McMillan-Major et al., 2022). This yielded a set of 252 sources, including at least 21 per considered language category. We focused on metadata collection as a way to support selection of the sources for the final dataset and documentation of the final dataset. In parallel, working group participants gathered additional
#/body
BOTTOMLEFT
#/texts/35
The first part of our corpus, accounting for 62% of the final dataset size (in bytes), was made up of a collection of monolingual and multilingual language resources that were selected and documented collaboratively through various efforts of the BigScience Data Sourcing working group. The first such effort consisted in creating a tool to support metadata collection through open submissions, called the BigScience Catalogue and running a series of hackathons in collaboration with locally-focused ML and NLP communities such as Masakhane, Machine Learning Tokyo and LatinX in AI where participants could add and document entries for their languages to the catalogue (McMillan-Major et al., 2022). This yielded a set of 252 sources, including at least 21 per considered language category. We focused on metadata collection as a way to support selection of the sources for the final dataset and documentation of the final dataset. In parallel, working group participants gathered additional
page_footer
4
#/body
BOTTOMLEFT
#/texts/36
4
text
Arabic language resources in the Masader repository (Alyafeai et al., 2021), and proposed a list of websites of interest to increase the geographical diversity of our English, Spanish, and Chinese language data. Finally, in order to explicitly test large language models' ability to handle computer code along with natural language, we selected code data available on GitHub and StackExchange.
#/body
BOTTOMLEFT
#/texts/37
Arabic language resources in the Masader repository (Alyafeai et al., 2021), and proposed a list of websites of interest to increase the geographical diversity of our English, Spanish, and Chinese language data. Finally, in order to explicitly test large language models' ability to handle computer code along with natural language, we selected code data available on GitHub and StackExchange.
section_header
2.1 Obtaining Data from the Identified Resources
#/body
BOTTOMLEFT
#/texts/38
2.1 Obtaining Data from the Identified Resources
text
Gathering Identified Datasets and Collections. First, we leveraged the BigScience Catalogue and the Masader repository to start obtaining text from identified sources, which included both existing NLP datasets and collections of documents of various compositions. Given the diversity of sources, hosting methods, data custodians, and formats, collecting this text required a collaborative effort. To that end, we established a 2-phase approach: first, collect as many data sources as possible in an easily accessible location; second, map all of them to a common format to ease further processing.
#/body
BOTTOMLEFT
#/texts/39
Gathering Identified Datasets and Collections. First, we leveraged the BigScience Catalogue and the Masader repository to start obtaining text from identified sources, which included both existing NLP datasets and collections of documents of various compositions. Given the diversity of sources, hosting methods, data custodians, and formats, collecting this text required a collaborative effort. To that end, we established a 2-phase approach: first, collect as many data sources as possible in an easily accessible location; second, map all of them to a common format to ease further processing.
text
In the first phase, we organized an open hackathon to start gathering identified sources on the Hugging Face Datasets hub (Lhoest et al., 2021), in a dedicated organization 5 (in order to manage access controls). In the second phase, the collected datasets were furthered processed via (1) Language segmentation , whereby data sources were split using metadata for each covered language in order to obtain monolingual datasets, and the use of (2) Uniform interface whereby a document consisted of two fields: "text" for the actual text content, and "meta" with a JSON representation of metadata for a given document, containing sufficient information to trace documents back to their original sources.
#/body
BOTTOMLEFT
#/texts/40
In the first phase, we organized an open hackathon to start gathering identified sources on the Hugging Face Datasets hub (Lhoest et al., 2021), in a dedicated organization 5 (in order to manage access controls). In the second phase, the collected datasets were furthered processed via (1) Language segmentation , whereby data sources were split using metadata for each covered language in order to obtain monolingual datasets, and the use of (2) Uniform interface whereby a document consisted of two fields: "text" for the actual text content, and "meta" with a JSON representation of metadata for a given document, containing sufficient information to trace documents back to their original sources.
text
Pseudo-Crawled Data. Of the various categories of language resources identified through the data sourcing effort, websites stood out as one that required a particular effort and dedicated pipeline. We decided to design such a pipeline based on "pseudo-crawling": that is, rather than crawling the websites ourselves, we retrieved pages corresponding to the target domain names from 18 snapshots archived by Common Crawl in 2020 and 2021 in Web ARChive (WARC) format (Mohr et al., 2008). These domain names came from two main sources: the homepage field in the metadata of the 252 above-mentioned catalogue entries when available (192 in total), and the 456 websites proposed by participants asynchronously to improve the geographical diversity of our language sources; which yielded a total of 614 unique domain names after deduplication.
#/body
BOTTOMLEFT
#/texts/41
Pseudo-Crawled Data. Of the various categories of language resources identified through the data sourcing effort, websites stood out as one that required a particular effort and dedicated pipeline. We decided to design such a pipeline based on "pseudo-crawling": that is, rather than crawling the websites ourselves, we retrieved pages corresponding to the target domain names from 18 snapshots archived by Common Crawl in 2020 and 2021 in Web ARChive (WARC) format (Mohr et al., 2008). These domain names came from two main sources: the homepage field in the metadata of the 252 above-mentioned catalogue entries when available (192 in total), and the 456 websites proposed by participants asynchronously to improve the geographical diversity of our language sources; which yielded a total of 614 unique domain names after deduplication.
text
We collected URLs contained within those domains using the Common Crawl index. The index provides metadata for every document including the page URL, WARC filename and record offsets, fetch status, content MIME type, etc. We ran a query matching all documents that share the domain name with a seed using Amazon Athena on Common Crawl's columnar index$^{6}$. 48 of the 614 initial seed domain names had no matches in the index and were therefore left out. Once we obtained the document metadata, we fetched the WARC records using HTTP range requests with the start and end byte offsets. Since HTML web pages constitute the largest portion of pages contained in the Common Crawl dumps, we decided to only extract text from HTML pages. Documents in other formats were filtered out, ie XML, PDF, etc. 27 domain names were additionally removed from the list at this stage as we had not retrieved any HTML pages for them.
#/body
BOTTOMLEFT
#/texts/42
We collected URLs contained within those domains using the Common Crawl index. The index provides metadata for every document including the page URL, WARC filename and record offsets, fetch status, content MIME type, etc. We ran a query matching all documents that share the domain name with a seed using Amazon Athena on Common Crawl's columnar index$^{6}$. 48 of the 614 initial seed domain names had no matches in the index and were therefore left out. Once we obtained the document metadata, we fetched the WARC records using HTTP range requests with the start and end byte offsets. Since HTML web pages constitute the largest portion of pages contained in the Common Crawl dumps, we decided to only extract text from HTML pages. Documents in other formats were filtered out, ie XML, PDF, etc. 27 domain names were additionally removed from the list at this stage as we had not retrieved any HTML pages for them.
text
To extract the text from the HTML pages, we first minified the HTML code. Minification is the removal of unnecessary characters from the source code of a website. Inspired by Aghajanyan et al. (2022), we removed from the DOM-HTML all the sub-trees contained in a <script> , <style> , <header> , <iframe> , <footer> and <form> tag as well as all the sub-trees associated with a <body> , <div> , <p> , <section> , <table> , <ul> , <ol> or <dl> tag whose textual content was less than 64 characters long. The text was then extracted from the nodes of this new DOM-HTML. While concatenating the text extracted, we applied a set of rules to reconstruct the structure of the text without its HTML code, inspired by what Common Crawl does to extract its WET files (Appendix B.1). The overall procedure enabled us to obtain text datasets for 539 domain names.
#/body
BOTTOMLEFT
#/texts/43
To extract the text from the HTML pages, we first minified the HTML code. Minification is the removal of unnecessary characters from the source code of a website. Inspired by Aghajanyan et al. (2022), we removed from the DOM-HTML all the sub-trees contained in a <script> , <style> , <header> , <iframe> , <footer> and <form> tag as well as all the sub-trees associated with a <body> , <div> , <p> , <section> , <table> , <ul> , <ol> or <dl> tag whose textual content was less than 64 characters long. The text was then extracted from the nodes of this new DOM-HTML. While concatenating the text extracted, we applied a set of rules to reconstruct the structure of the text without its HTML code, inspired by what Common Crawl does to extract its WET files (Appendix B.1). The overall procedure enabled us to obtain text datasets for 539 domain names.
text
GitHub Code. We collected a code dataset from BigQuery 7 using the same language selection as AlphaCode (Li et al., 2022). The dataset was then deduplicated of exact matches and filtered for source files with between 100 and 200,000 characters, between 15-65% alphabetic characters, a max
#/body
BOTTOMLEFT
#/texts/44
GitHub Code. We collected a code dataset from BigQuery 7 using the same language selection as AlphaCode (Li et al., 2022). The dataset was then deduplicated of exact matches and filtered for source files with between 100 and 200,000 characters, between 15-65% alphabetic characters, a max
footnote
$^{5}$https://hf.co/bigscience-catalogue-data
#/body
BOTTOMLEFT
#/texts/45
$^{5}$https://hf.co/bigscience-catalogue-data
footnote
$^{6}$https://commoncrawl.org/2018/03/index-to-warc-files-and-urls-in-columnar-format/
#/body
BOTTOMLEFT
#/texts/46
$^{6}$https://commoncrawl.org/2018/03/index-to-warc-files-and-urls-in-columnar-format/
footnote
$^{7}$“GitHub on BigQuery: Analyze all the open source code”
#/body
BOTTOMLEFT
#/texts/47
$^{7}$“GitHub on BigQuery: Analyze all the open source code”
page_footer
5
#/body
BOTTOMLEFT
#/texts/48
5
text
line length of 20-1000 characters, and a token length standard deviation of more than 3. Due to a bug in the pre-processing pipeline the dataset was also filtered for GPL licenses only.
#/body
BOTTOMLEFT
#/texts/49
line length of 20-1000 characters, and a token length standard deviation of more than 3. Due to a bug in the pre-processing pipeline the dataset was also filtered for GPL licenses only.
text
Merging and Deduplicating Sources. After gathering and processing language data via the three pipelines outlined above, we took a final step to manually inspect, deduplicate, and make a further selection of the sources. First, we addressed dataset overlap we found by looking through our sources. For example: OpenITI was present in both its raw form as well as a processed version. Consensus was reached to choose the latter version. Non-trivial datasets overlap included s2orc (Lo et al., 2020), Arxiv (Clement et al., 2019) and the PubMed Central subset of the Pile (Gao et al., 2020). We also performed cross-pipeline dataset deduplication, removing the pseudo-crawled Wikipedia and GitHub in favor of their other versions. We also removed datasets that we found had a high incidence of documents that were not fully in natural language (e.g. unexpected instances of SEO, HTML tags etc...), as well as very small datasets in the higher-resourced languages. Finally, pseudo-crawled sources were further processed to remove menus (with a heuristic consisting of removing lines that occurred in more than 1% of pages in a given domain) and pages that had a high incidence of character ngram repetition, low language identification confidence, or low proportion of closed class words (see Section 3). We then removed entire domains whose size was less than 2MB after this step, yielding 147 pseudo-crawl-based datasets, and a total of 517 datasets including all three pipelines.
#/body
BOTTOMLEFT
#/texts/50
Merging and Deduplicating Sources. After gathering and processing language data via the three pipelines outlined above, we took a final step to manually inspect, deduplicate, and make a further selection of the sources. First, we addressed dataset overlap we found by looking through our sources. For example: OpenITI was present in both its raw form as well as a processed version. Consensus was reached to choose the latter version. Non-trivial datasets overlap included s2orc (Lo et al., 2020), Arxiv (Clement et al., 2019) and the PubMed Central subset of the Pile (Gao et al., 2020). We also performed cross-pipeline dataset deduplication, removing the pseudo-crawled Wikipedia and GitHub in favor of their other versions. We also removed datasets that we found had a high incidence of documents that were not fully in natural language (e.g. unexpected instances of SEO, HTML tags etc...), as well as very small datasets in the higher-resourced languages. Finally, pseudo-crawled sources were further processed to remove menus (with a heuristic consisting of removing lines that occurred in more than 1% of pages in a given domain) and pages that had a high incidence of character ngram repetition, low language identification confidence, or low proportion of closed class words (see Section 3). We then removed entire domains whose size was less than 2MB after this step, yielding 147 pseudo-crawl-based datasets, and a total of 517 datasets including all three pipelines.
section_header
2.2 Processing Pipeline for Quality Improvement on Crowdsourced Datasets
#/body
BOTTOMLEFT
#/texts/51
2.2 Processing Pipeline for Quality Improvement on Crowdsourced Datasets
text
Once a text field was obtained, we attempted to improve the quality of that text. In the specific case of text extraction from HTML, we observe that not all text are relevant (menus, advertisements, repeated text on each page etc ...). In order to remove noisy data from our dataset, we applied a processing pipeline for each dataset consisting of a sequence of functions.
#/body
BOTTOMLEFT
#/texts/52
Once a text field was obtained, we attempted to improve the quality of that text. In the specific case of text extraction from HTML, we observe that not all text are relevant (menus, advertisements, repeated text on each page etc ...). In order to remove noisy data from our dataset, we applied a processing pipeline for each dataset consisting of a sequence of functions.
text
Functions were categorised as document-scoped or dataset-scoped functions. Document-scoped functions are operations that modify a document independently of other documents and datasetscoped functions are operations that take into account the whole dataset. Orthogonal to this scope, functions were also separated into cleaning and filtering functions. Cleaning functions aim to remove text considered not part of the main document. Document-scoped cleaning functions can for example target leftover HTML tags. On the other end, dataset-scoped cleaning functions need the whole dataset to calculate a heuristic to determine how to modify each document. For instance, advertisements vary across datasets, making it harder to define a dataset-agnostic classifier for advertisement. Instead, we can index all the lines in a dataset and identify repeated lines on multiple pages as likely advertisements. An example is displayed in Appendix B.2. Filtering functions aim at removing an entire document from the corpus. The reasons for choosing to remove a document completely are diverse: it may be because the document is considered to be of too poor quality, to be too complex to automatically fix or too similar to other examples already present in the corpus. In the latter case, we speak of deduplication. Deduplication of a document is dependent on whether an equivalent document already exists somewhere else in the dataset and is thus necessarily a datasetscope function. The notion of equivalent documents has been explored by Lee et al. (2022). In this case we provide deduplication via metadata (urls, normalised urls) and via text (exact string matching). An exhaustive list of functions is available in B.3.
#/body
BOTTOMLEFT
#/texts/53
Functions were categorised as document-scoped or dataset-scoped functions. Document-scoped functions are operations that modify a document independently of other documents and datasetscoped functions are operations that take into account the whole dataset. Orthogonal to this scope, functions were also separated into cleaning and filtering functions. Cleaning functions aim to remove text considered not part of the main document. Document-scoped cleaning functions can for example target leftover HTML tags. On the other end, dataset-scoped cleaning functions need the whole dataset to calculate a heuristic to determine how to modify each document. For instance, advertisements vary across datasets, making it harder to define a dataset-agnostic classifier for advertisement. Instead, we can index all the lines in a dataset and identify repeated lines on multiple pages as likely advertisements. An example is displayed in Appendix B.2. Filtering functions aim at removing an entire document from the corpus. The reasons for choosing to remove a document completely are diverse: it may be because the document is considered to be of too poor quality, to be too complex to automatically fix or too similar to other examples already present in the corpus. In the latter case, we speak of deduplication. Deduplication of a document is dependent on whether an equivalent document already exists somewhere else in the dataset and is thus necessarily a datasetscope function. The notion of equivalent documents has been explored by Lee et al. (2022). In this case we provide deduplication via metadata (urls, normalised urls) and via text (exact string matching). An exhaustive list of functions is available in B.3.
text
As datasets came from heterogeneous sources with different properties, each needs its own set of processing functions to correspond to our definition of natural language documents. In order to support participants in deciding what functions to apply to which, we built and released a streamlit -based visualization tool (figure 2 helps understand the impact of each function, displaying how a document was altered/removed as well as estimated dataset level metrics (quantity of data removed in bytes or samples)). This rapid feedback loop enabled us to update the pipeline consequently in an iterative process to finetune each processing pipelines across datasets and languages with the input of native speakers. A specific example is shared in Appendix B.2. This resulted in 485 non-empty datasets.
#/body
BOTTOMLEFT
#/texts/54
As datasets came from heterogeneous sources with different properties, each needs its own set of processing functions to correspond to our definition of natural language documents. In order to support participants in deciding what functions to apply to which, we built and released a streamlit -based visualization tool (figure 2 helps understand the impact of each function, displaying how a document was altered/removed as well as estimated dataset level metrics (quantity of data removed in bytes or samples)). This rapid feedback loop enabled us to update the pipeline consequently in an iterative process to finetune each processing pipelines across datasets and languages with the input of native speakers. A specific example is shared in Appendix B.2. This resulted in 485 non-empty datasets.
page_footer
6
#/body
BOTTOMLEFT
#/texts/55
6
caption
Figure 2: Partial screenshot of the visualization tool. Users can look at how each function in the processing pipeline influenced high-level statistics. Influence on specific samples can be monitored via the same tool, see Appendix B.2
#/body
BOTTOMLEFT
#/texts/56
Figure 2: Partial screenshot of the visualization tool. Users can look at how each function in the processing pipeline influenced high-level statistics. Influence on specific samples can be monitored via the same tool, see Appendix B.2
section_header
3 Processing OSCAR
#/body
BOTTOMLEFT
#/texts/57
3 Processing OSCAR
text
We chose to complement the data obtained at the end of the process described in the previous section with additional Common Crawl-based 8 data motivated by two main reasons. First, given the project's overall goal of providing a trained LLM as a research artifact comparable to previously released ones that have relied extensively on this source, we assessed that not including it would constitute too much of a departure and risk invalidating comparisons. Relatedly, recent work has put a strong emphasis on the quantity of data being a strong factor in a trained model's performance on evaluation tasks (Kaplan et al., 2020; Hoffmann et al., 2022), and we were missing about one third of data in order to optimize our compute budget in this direction. With that in mind, we chose OSCAR version 21.09 (Ortiz Suárez et al., 2020), based on the Common Crawl snapshot of February 2021, to make up the remaining 38% of our final dataset.
#/body
BOTTOMLEFT
#/texts/58
We chose to complement the data obtained at the end of the process described in the previous section with additional Common Crawl-based 8 data motivated by two main reasons. First, given the project's overall goal of providing a trained LLM as a research artifact comparable to previously released ones that have relied extensively on this source, we assessed that not including it would constitute too much of a departure and risk invalidating comparisons. Relatedly, recent work has put a strong emphasis on the quantity of data being a strong factor in a trained model's performance on evaluation tasks (Kaplan et al., 2020; Hoffmann et al., 2022), and we were missing about one third of data in order to optimize our compute budget in this direction. With that in mind, we chose OSCAR version 21.09 (Ortiz Suárez et al., 2020), based on the Common Crawl snapshot of February 2021, to make up the remaining 38% of our final dataset.
text
However, crawled data suffers from several known issues. First, we wanted to only select documents written by humans for humans, and exclude machine-generated content e.g. search engine optimization (SEO). Crawled content also over-represents pornographic text across languages (Kreutzer et al., 2022), especially in the form of spam ads. Finally, it contains personal information that may constitute a privacy risk. The present section outlines our approach to mitigating those issues.
#/body
BOTTOMLEFT
#/texts/59
However, crawled data suffers from several known issues. First, we wanted to only select documents written by humans for humans, and exclude machine-generated content e.g. search engine optimization (SEO). Crawled content also over-represents pornographic text across languages (Kreutzer et al., 2022), especially in the form of spam ads. Finally, it contains personal information that may constitute a privacy risk. The present section outlines our approach to mitigating those issues.
section_header
3.1 Data cleaning and filtering
#/body
BOTTOMLEFT
#/texts/60
3.1 Data cleaning and filtering
text
Our first approach to addressing the above consists in defining quality indicators for web content. These can then be used to filter out specific pages by defining cutoff thresholds. Extensive descriptions for reproduction are available in appendix C. We filtered out documents with:
#/body
BOTTOMLEFT
#/texts/61
Our first approach to addressing the above consists in defining quality indicators for web content. These can then be used to filter out specific pages by defining cutoff thresholds. Extensive descriptions for reproduction are available in appendix C. We filtered out documents with:
list_item
-
· Too high character repetition or word repetition as a measure of repetitive content.
#/groups/0
BOTTOMLEFT
#/texts/62
· Too high character repetition or word repetition as a measure of repetitive content.
list_item
-
· Too high ratios of special characters to remove page code or crawling artifacts.
#/groups/0
BOTTOMLEFT
#/texts/63
· Too high ratios of special characters to remove page code or crawling artifacts.
list_item
-
· Insufficient ratios of closed class words to filter out SEO pages.
#/groups/0
BOTTOMLEFT
#/texts/64
· Insufficient ratios of closed class words to filter out SEO pages.
list_item
-
· Too high ratios of flagged words to filter out pornographic spam. We asked contributors to tailor the word list in their language to this criterion (as opposed to generic terms related to sexuality) and to err on the side of high precision.
#/groups/0
BOTTOMLEFT
#/texts/65
· Too high ratios of flagged words to filter out pornographic spam. We asked contributors to tailor the word list in their language to this criterion (as opposed to generic terms related to sexuality) and to err on the side of high precision.
list_item
-
· Too high perplexity values to filter out non-natural language.
#/groups/0
BOTTOMLEFT
#/texts/66
· Too high perplexity values to filter out non-natural language.
list_item
-
· Insufficient number of words , as LLM training requires extensive context sizes.
#/groups/0
BOTTOMLEFT
#/texts/67
· Insufficient number of words , as LLM training requires extensive context sizes.
text
The languages that we eventually considered in OSCAR were the languages for which we were able to obtain hyperparameters and the cutoff values for each of these indicators by native speakers. Specifically, we considered Arabic, Basque, Bengali, Catalan, Chinese, English, French, Hindi, Indonesian, Portuguese, Spanish, Urdu, and Vietnamese. The code used for filtering OSCAR, along with the language-specific parameters and cutoff values, are publicly available. We then asked native speakers of each language to use our visualization tool 9 to establish the thresholds for each filter. The percentage of documents removed after applying all these filters is given in Table 1, and the percentage of documents discarded by each filter independently is given in 3.
#/body
BOTTOMLEFT
#/texts/68
The languages that we eventually considered in OSCAR were the languages for which we were able to obtain hyperparameters and the cutoff values for each of these indicators by native speakers. Specifically, we considered Arabic, Basque, Bengali, Catalan, Chinese, English, French, Hindi, Indonesian, Portuguese, Spanish, Urdu, and Vietnamese. The code used for filtering OSCAR, along with the language-specific parameters and cutoff values, are publicly available. We then asked native speakers of each language to use our visualization tool 9 to establish the thresholds for each filter. The percentage of documents removed after applying all these filters is given in Table 1, and the percentage of documents discarded by each filter independently is given in 3.
footnote
$^{8}$https://commoncrawl.org/
#/body
BOTTOMLEFT
#/texts/69
$^{8}$https://commoncrawl.org/
footnote
$^{9}$https://hf.co/spaces/huggingface/text-data-filtering
#/body
BOTTOMLEFT
#/texts/70
$^{9}$https://hf.co/spaces/huggingface/text-data-filtering
page_footer
7
#/body
BOTTOMLEFT
#/texts/71
7
caption
Table 1: Percentage of documents removed by the filtering per language (ISO 639-1 code).
#/body
BOTTOMLEFT
#/texts/72
Table 1: Percentage of documents removed by the filtering per language (ISO 639-1 code).
caption
Figure 3: Percentage of documents discarded by each filter independently for 5 languages
#/body
BOTTOMLEFT
#/texts/73
Figure 3: Percentage of documents discarded by each filter independently for 5 languages
section_header
3.2 Deduplication
#/body
BOTTOMLEFT
#/texts/74
3.2 Deduplication
text
Data deduplication has become a key tool for language model projects following research showing that it both improves performance on downstream tasks (Lee et al., 2022; Zhang et al., 2021) and decreases memorization of training data (Kandpal et al., 2022). To remove near duplicate documents in OSCAR (which is already exact-deduplicated) we initially used SimHash (Charikar, 2002; Manku et al., 2007), a hashing function that associates to two similar texts hashes with a low Hamming distance, with 6-grams and a Hamming distance threshold of 4. About 0.7% of the documents on average (0.07% ∼ 2.7%) were identified as near duplicates. However, because SimHash is essentially a bag-of-words algorithm, long documents are more likely to end up being similar to each other. In practice, we found false positives among long documents and decided not to discard documents in a same cluster of near-duplicates when they were longer than 6000 characters. Instead, we applied substring deduplication (Lee et al., 2022) based on Suffix Array (Manber and Myers, 1993) as a complementary method that clusters documents sharing a long substring, for documents with more than 6000 characters. We found on average 21.67% (10.61% ∼ 32.30%) of the data (in bytes) being duplicated.
#/body
BOTTOMLEFT
#/texts/75
Data deduplication has become a key tool for language model projects following research showing that it both improves performance on downstream tasks (Lee et al., 2022; Zhang et al., 2021) and decreases memorization of training data (Kandpal et al., 2022). To remove near duplicate documents in OSCAR (which is already exact-deduplicated) we initially used SimHash (Charikar, 2002; Manku et al., 2007), a hashing function that associates to two similar texts hashes with a low Hamming distance, with 6-grams and a Hamming distance threshold of 4. About 0.7% of the documents on average (0.07% ∼ 2.7%) were identified as near duplicates. However, because SimHash is essentially a bag-of-words algorithm, long documents are more likely to end up being similar to each other. In practice, we found false positives among long documents and decided not to discard documents in a same cluster of near-duplicates when they were longer than 6000 characters. Instead, we applied substring deduplication (Lee et al., 2022) based on Suffix Array (Manber and Myers, 1993) as a complementary method that clusters documents sharing a long substring, for documents with more than 6000 characters. We found on average 21.67% (10.61% ∼ 32.30%) of the data (in bytes) being duplicated.
section_header
3.3 Personally identifiable information
#/body
BOTTOMLEFT
#/texts/76
3.3 Personally identifiable information
text
We used a rule-based approach leveraging regular expressions (Appendix C). The elements redacted were instances of KEY (numeric & alphanumeric identifiers such as phone numbers, credit card numbers, hexadecimal hashes and the like, while skipping instances of years and simple numbers), EMAIL (email addresses), USER (a social media handle) and IP_ADDRESS (an IPv4 or IPv6 address).
#/body
BOTTOMLEFT
#/texts/77
We used a rule-based approach leveraging regular expressions (Appendix C). The elements redacted were instances of KEY (numeric & alphanumeric identifiers such as phone numbers, credit card numbers, hexadecimal hashes and the like, while skipping instances of years and simple numbers), EMAIL (email addresses), USER (a social media handle) and IP_ADDRESS (an IPv4 or IPv6 address).
section_header
4 A First look at ROOTS
#/body
BOTTOMLEFT
#/texts/78
4 A First look at ROOTS
text
The efforts described in the previous sections come together in an assemblage of 1.6 Terabytes of multilingual text. Figure 4 puts that number into context by comparing the sizes of corpora typically used to train large language models. Documentation of the individual components of the corpus can be found in an interactive dataset card deck. In this section, we take initial steps towards further understanding of the corpus through statistical analyses of the aggregated data.
#/body
BOTTOMLEFT
#/texts/79
The efforts described in the previous sections come together in an assemblage of 1.6 Terabytes of multilingual text. Figure 4 puts that number into context by comparing the sizes of corpora typically used to train large language models. Documentation of the individual components of the corpus can be found in an interactive dataset card deck. In this section, we take initial steps towards further understanding of the corpus through statistical analyses of the aggregated data.
section_header
4.1 Natural Languages
#/body
BOTTOMLEFT
#/texts/80
4.1 Natural Languages
text
The constitution of the corpus reflects the crowdsourcing efforts that enabled its creation. It comprises of 46 natural languages spanning 3 macroareas and 9 language families: Afro-Asiatic, AustroAsiatic, Austronesian, Basque, Dravidian, Indo-European, Mande, Niger-Congo, Sino-Tibetan. At 30.03%, English constitutes the largest part of the corpus, followed by Simplified Chinese (16.16%), French (12.9%), Spanish (10.85%), Portuguese (4.91%) and Arabic (4.6%). A more detailed breakdown of the corpus can be found in the appendix and in an online interactive exploration tool$^{10}$,
#/body
BOTTOMLEFT
#/texts/81
The constitution of the corpus reflects the crowdsourcing efforts that enabled its creation. It comprises of 46 natural languages spanning 3 macroareas and 9 language families: Afro-Asiatic, AustroAsiatic, Austronesian, Basque, Dravidian, Indo-European, Mande, Niger-Congo, Sino-Tibetan. At 30.03%, English constitutes the largest part of the corpus, followed by Simplified Chinese (16.16%), French (12.9%), Spanish (10.85%), Portuguese (4.91%) and Arabic (4.6%). A more detailed breakdown of the corpus can be found in the appendix and in an online interactive exploration tool$^{10}$,
footnote
$^{10}$https://hf.co/spaces/bigscience-data/corpus-map
#/body
BOTTOMLEFT
#/texts/82
$^{10}$https://hf.co/spaces/bigscience-data/corpus-map
page_footer
8
#/body
BOTTOMLEFT
#/texts/83
8
caption
Figure 4: A raw size comparison to other corpora used to train large language models. The asterisk next to GPT-3 indicates the fact that the value in question is an estimate computed using the reported number of tokens and the average number of tokens per byte of text that the GPT-2 tokenizer produces on the Pile-CC , Books3 , OWT2 , and Wiki-en subsets of the Pile (Gao et al., 2020)
#/body
BOTTOMLEFT
#/texts/84
Figure 4: A raw size comparison to other corpora used to train large language models. The asterisk next to GPT-3 indicates the fact that the value in question is an estimate computed using the reported number of tokens and the average number of tokens per byte of text that the GPT-2 tokenizer produces on the Pile-CC , Books3 , OWT2 , and Wiki-en subsets of the Pile (Gao et al., 2020)
caption
Figure 5: Size in bytes of every document in the corpus per language. The y-axis is in logarithmic scale. Box-and-whisker diagrams illustrate median, the first and third quartiles, whiskers drawn within the 1.5 IQR value and outliers
#/body
BOTTOMLEFT
#/texts/85
Figure 5: Size in bytes of every document in the corpus per language. The y-axis is in logarithmic scale. Box-and-whisker diagrams illustrate median, the first and third quartiles, whiskers drawn within the 1.5 IQR value and outliers
text
a screenshot of which is included in figure 1 to depict the byte-distribution of linguistic genera of the Eurasian macroarea subset of the corpus.
#/body
BOTTOMLEFT
#/texts/86
a screenshot of which is included in figure 1 to depict the byte-distribution of linguistic genera of the Eurasian macroarea subset of the corpus.
text
In order for the trained model to have an opportunity to learn long dependencies, the training corpus needs to contain long sequences of coherent text. At the same time, the previous post-processing steps only reduced the size of the documents. The median size of a document in our corpus is 1,129 bytes. Figure 5 shows the distribution of document sizes by language. A more detailed breakdown of the size of corpus on an online interactive tool.$^{11}$.
#/body
BOTTOMLEFT
#/texts/87
In order for the trained model to have an opportunity to learn long dependencies, the training corpus needs to contain long sequences of coherent text. At the same time, the previous post-processing steps only reduced the size of the documents. The median size of a document in our corpus is 1,129 bytes. Figure 5 shows the distribution of document sizes by language. A more detailed breakdown of the size of corpus on an online interactive tool.$^{11}$.
text
The distributions of the filter values for the different filters introduced in Section 3.1 and languages, for the Catalogue, Pseudo-Crawl and OSCAR (filtered) data are available in an online demo$^{12}$. Examples for English are shown in figure 6. The different distributions reflect the diversity of sourcing and filtering of our main components. A notable example is the flagged word filter, for which the distribution for OSCAR is skewed right compared to the catalogue even after filtering.
#/body
BOTTOMLEFT
#/texts/88
The distributions of the filter values for the different filters introduced in Section 3.1 and languages, for the Catalogue, Pseudo-Crawl and OSCAR (filtered) data are available in an online demo$^{12}$. Examples for English are shown in figure 6. The different distributions reflect the diversity of sourcing and filtering of our main components. A notable example is the flagged word filter, for which the distribution for OSCAR is skewed right compared to the catalogue even after filtering.
section_header
4.2 Programming Languages
#/body
BOTTOMLEFT
#/texts/89
4.2 Programming Languages
text
As depicted in the waffle plot in figure 1, the code subset of the corpus spans 13 programming languages, with Java, PHP, and C++ accounting for more than half of all documents.
#/body
BOTTOMLEFT
#/texts/90
As depicted in the waffle plot in figure 1, the code subset of the corpus spans 13 programming languages, with Java, PHP, and C++ accounting for more than half of all documents.
footnote
$^{11}$https://hf.co/spaces/bigscience-data/document-sizes
#/body
BOTTOMLEFT
#/texts/91
$^{11}$https://hf.co/spaces/bigscience-data/document-sizes
footnote
$^{12}$https://hf.co/spaces/bigscience-catalogue-lm-data/filter_values_distributions
#/body
BOTTOMLEFT
#/texts/92
$^{12}$https://hf.co/spaces/bigscience-catalogue-lm-data/filter_values_distributions
page_footer
9
#/body
BOTTOMLEFT
#/texts/93
9
caption
Figure 6: Some distributions of filter values for English. A filter value is the value that the filter gives to a document. These values are generally used to filter out documents that are too low or too high rated and also inform about the composition of the datasets.
#/body
BOTTOMLEFT
#/texts/94
Figure 6: Some distributions of filter values for English. A filter value is the value that the filter gives to a document. These values are generally used to filter out documents that are too low or too high rated and also inform about the composition of the datasets.
caption
Figure 7: Tokens per byte for each English-language component for tokenizers trained on this corpus (BLOOM), the Pile (GPT-NeoX 20B) and C4 (T5). Lower values mean the component (X axis) is more similar in aggregate to the compared training corpus.
#/body
BOTTOMLEFT
#/texts/95
Figure 7: Tokens per byte for each English-language component for tokenizers trained on this corpus (BLOOM), the Pile (GPT-NeoX 20B) and C4 (T5). Lower values mean the component (X axis) is more similar in aggregate to the compared training corpus.
text
Configuration and test files are abundant in most GitHub repositories but not as interesting for code modeling. To that end, we use a heuristic whose first step examines the first 5 lines of a file for the presence of keywords such as "configuration file" or "test file". Failing that, the second step is to see whether the occurrence of the literals config and test in a given file exceeds 5% of the total number of lines of that file. We find that 5.23% of the data consists of configuration files and 7.88% of test files.
#/body
BOTTOMLEFT
#/texts/96
Configuration and test files are abundant in most GitHub repositories but not as interesting for code modeling. To that end, we use a heuristic whose first step examines the first 5 lines of a file for the presence of keywords such as "configuration file" or "test file". Failing that, the second step is to see whether the occurrence of the literals config and test in a given file exceeds 5% of the total number of lines of that file. We find that 5.23% of the data consists of configuration files and 7.88% of test files.
text
Allamanis (2019) and Lopes et al. (2017) highlight the large fraction of near-duplicates present in code datasets and how they can inflate performance metrics. Exact match deduplication alone can miss a fair amount of near-duplicates. To detect them, we first compute the MinHash of all documents, then create a Locality Sensitive Hashing (LSH) index between files to find the duplicate clusters in linear time. We additionally evaluate the Jaccard similarities within duplicate clusters to remove some false positives. We find 10.9M duplicate files in the clusters and 4.1M unique files: almost 32% of the data consists of near-duplicates. Syntax checkers 13 are used to validate 500K samples of Python and PHP code. We find that only 1% of the Python data and 2% of the PHP files do not pass the syntax check.
#/body
BOTTOMLEFT
#/texts/97
Allamanis (2019) and Lopes et al. (2017) highlight the large fraction of near-duplicates present in code datasets and how they can inflate performance metrics. Exact match deduplication alone can miss a fair amount of near-duplicates. To detect them, we first compute the MinHash of all documents, then create a Locality Sensitive Hashing (LSH) index between files to find the duplicate clusters in linear time. We additionally evaluate the Jaccard similarities within duplicate clusters to remove some false positives. We find 10.9M duplicate files in the clusters and 4.1M unique files: almost 32% of the data consists of near-duplicates. Syntax checkers 13 are used to validate 500K samples of Python and PHP code. We find that only 1% of the Python data and 2% of the PHP files do not pass the syntax check.
section_header
4.3 Tokenizer analysis of the component datasets
#/body
BOTTOMLEFT
#/texts/98
4.3 Tokenizer analysis of the component datasets
text
A tokenizer trained on a dataset can be used as a proxy for its content (Gao et al., 2020). The relevant metric is the number of tokens produced for a byte of natural language. The more different the training corpus from the tokenized corpus, the more tokens will be produced as the tokenizer is forced to divide natural text in more numerous, more general, smaller tokens. This property has allowed us to spot errors associated with outlier values, such as incorrectly classified languages, or crawling error. In the following analysis, we use it in two ways: first, we can use tokenizers trained on different corpora to see how ours differs from them; and second, we can use a tokenizer trained on this corpus to assess which components are outliers. We exclude outliers smaller than 5 documents.
#/body
BOTTOMLEFT
#/texts/99
A tokenizer trained on a dataset can be used as a proxy for its content (Gao et al., 2020). The relevant metric is the number of tokens produced for a byte of natural language. The more different the training corpus from the tokenized corpus, the more tokens will be produced as the tokenizer is forced to divide natural text in more numerous, more general, smaller tokens. This property has allowed us to spot errors associated with outlier values, such as incorrectly classified languages, or crawling error. In the following analysis, we use it in two ways: first, we can use tokenizers trained on different corpora to see how ours differs from them; and second, we can use a tokenizer trained on this corpus to assess which components are outliers. We exclude outliers smaller than 5 documents.
footnote
$^{13}$py_compile for Python and the -l flag for PHP
#/body
BOTTOMLEFT
#/texts/100
$^{13}$py_compile for Python and the -l flag for PHP
page_footer
10
#/body
BOTTOMLEFT
#/texts/101
10
caption
Figure 8: Tokens per byte for each French, Simplified Chinese, and Arabic component for tokenizers trained on this corpus. Lower values mean the component (X axis) is more similar in aggregate to the rest of the corpus.
#/body
BOTTOMLEFT
#/texts/102
Figure 8: Tokens per byte for each French, Simplified Chinese, and Arabic component for tokenizers trained on this corpus. Lower values mean the component (X axis) is more similar in aggregate to the rest of the corpus.
text
Figure 7 shows the tokens-per-byte measurement on English component datasets for the BLOOM tokenizer, trained on this corpus, the GPT-NeoX 20B tokenizer (Black et al., 2022), trained on the Pile, and the T5 tokenizer (Raffel et al., 2020), trained on C4. Those tokenizers may differ in algorithms and/or vocabulary size, but we won't be directly comparing them to each other.
#/body
BOTTOMLEFT
#/texts/103
Figure 7 shows the tokens-per-byte measurement on English component datasets for the BLOOM tokenizer, trained on this corpus, the GPT-NeoX 20B tokenizer (Black et al., 2022), trained on the Pile, and the T5 tokenizer (Raffel et al., 2020), trained on C4. Those tokenizers may differ in algorithms and/or vocabulary size, but we won't be directly comparing them to each other.
text
The figure is ordered by BLOOM tokenizer token-per-byte values, which shows that the ordering is very similar for BLOOM and GPT-NeoX. However, it shows several bumps for T5: component datasets that are out of domain in C4 but not our corpus, for example technical and academic datasets such as s2orc or royal_society_corpus , domains absent from C4's Common Crawl-sourced data. Other such datasets include global_voices , which contains news about non-English-speaking regions including quotes in the original languages and no_code_stackexchange , which contains forums which, although in English, may be dedicated to technical matters, foreign languages, or very specific domains. Both are similar to our corpus but not to the Pile or C4.
#/body
BOTTOMLEFT
#/texts/104
The figure is ordered by BLOOM tokenizer token-per-byte values, which shows that the ordering is very similar for BLOOM and GPT-NeoX. However, it shows several bumps for T5: component datasets that are out of domain in C4 but not our corpus, for example technical and academic datasets such as s2orc or royal_society_corpus , domains absent from C4's Common Crawl-sourced data. Other such datasets include global_voices , which contains news about non-English-speaking regions including quotes in the original languages and no_code_stackexchange , which contains forums which, although in English, may be dedicated to technical matters, foreign languages, or very specific domains. Both are similar to our corpus but not to the Pile or C4.
text
Figure 8 additionally shows BLOOM fertilities for Simplified Chinese, French and Arabic components. Outlier, high-fertility components, e.g. datasets that differ from the rest of our corpus, tend to be the same for all languages. project_gutenberg contains old books with their original formatting (for example, "***********" to denote page ends). wiktionary contains definitions of words in foreign languages. wikiversity contains technical terms and L A T E X. wikivoyage contains tables formatted as text. Forums may contain the user and date information of the message, as well as internet slang or emoji. arabench is spoken Arabic, and habibi is classical Arabic with more diacritics than modern. We deem most of those deviations acceptable to represent the diversity of uses of text, which tokenizer analysis is able to surface from the rest of the dataset.
#/body
BOTTOMLEFT
#/texts/105
Figure 8 additionally shows BLOOM fertilities for Simplified Chinese, French and Arabic components. Outlier, high-fertility components, e.g. datasets that differ from the rest of our corpus, tend to be the same for all languages. project_gutenberg contains old books with their original formatting (for example, "***********" to denote page ends). wiktionary contains definitions of words in foreign languages. wikiversity contains technical terms and L A T E X. wikivoyage contains tables formatted as text. Forums may contain the user and date information of the message, as well as internet slang or emoji. arabench is spoken Arabic, and habibi is classical Arabic with more diacritics than modern. We deem most of those deviations acceptable to represent the diversity of uses of text, which tokenizer analysis is able to surface from the rest of the dataset.
section_header
5 Conclusion
#/body
BOTTOMLEFT
#/texts/106
5 Conclusion
text
We have presented ROOTS, a massive multilingual corpus that was the result of an international collaboration between multidisciplinary researchers studying large language models. The efforts to put the corpus together were value-driven and prompted by a data-first approach to training the BLOOM model. We further release the tooling developed throughout the project, and are currently implementing a release strategy that is informed by both the licensing and governance needs of every data source for the corpus itself. We hope this paves the way toward a more reflected use of the data that makes its way into large language models.
#/body
BOTTOMLEFT
#/texts/107
We have presented ROOTS, a massive multilingual corpus that was the result of an international collaboration between multidisciplinary researchers studying large language models. The efforts to put the corpus together were value-driven and prompted by a data-first approach to training the BLOOM model. We further release the tooling developed throughout the project, and are currently implementing a release strategy that is informed by both the licensing and governance needs of every data source for the corpus itself. We hope this paves the way toward a more reflected use of the data that makes its way into large language models.
section_header
Ethical Considerations and Broader Impacts Statement
#/body
BOTTOMLEFT
#/texts/108
Ethical Considerations and Broader Impacts Statement
text
As discussed in Section 1, the BigScience Research Workshop was conceived as a collaborative and value-driven endeavor from the start. This approach shaped many of the decisions described in this paper, spurring many contextual discussions and consensus-seeking on how to articulate the project's core values, those of the contributors to the data efforts, and considerations of social impact on the people directly and indirectly impacted. Of particular relevance were the data release and governance strategy, the choice to center human selection of data while still using OSCAR web-crawled for a significant section of the corpus, and the tools we developed to manage the risks of the latter (including regarding privacy). Each of these were the occasion of moral exercises and technical contributions that we believe were useful and required, and each will require further research and progress. We provide a more detailed discussion of these aspects of our work in Appendix A.
#/body
BOTTOMLEFT
#/texts/109
As discussed in Section 1, the BigScience Research Workshop was conceived as a collaborative and value-driven endeavor from the start. This approach shaped many of the decisions described in this paper, spurring many contextual discussions and consensus-seeking on how to articulate the project's core values, those of the contributors to the data efforts, and considerations of social impact on the people directly and indirectly impacted. Of particular relevance were the data release and governance strategy, the choice to center human selection of data while still using OSCAR web-crawled for a significant section of the corpus, and the tools we developed to manage the risks of the latter (including regarding privacy). Each of these were the occasion of moral exercises and technical contributions that we believe were useful and required, and each will require further research and progress. We provide a more detailed discussion of these aspects of our work in Appendix A.
page_footer
11
#/body
BOTTOMLEFT
#/texts/110
11
section_header
Acknowledgements
#/body
BOTTOMLEFT
#/texts/111
Acknowledgements
text
BigScience. This work was pursued as part of the BigScience research workshop, an effort to collaboratively build a very large multilingual neural network language model and a very large multilingual text dataset. This effort gathered 1000+ reasearchers from 60 countries and from more than 250 institutions.
#/body
BOTTOMLEFT
#/texts/112
BigScience. This work was pursued as part of the BigScience research workshop, an effort to collaboratively build a very large multilingual neural network language model and a very large multilingual text dataset. This effort gathered 1000+ reasearchers from 60 countries and from more than 250 institutions.
text
Compute. The BigScience Workshop was granted access to the HPC resources of the Institut du développement et des ressources en informatique scientifique (IDRIS) du Centre national de la recherche scientifique (CNRS) under the allocation 2021-A0101012475 made by Grand équipement national de calcul intensif (GENCI). Model training ran on the Jean-Zay cluster of IDRIS, and we thank the IDRIS team for their responsive support throughout the project, in particular Rémi Lacroix.
#/body
BOTTOMLEFT
#/texts/113
Compute. The BigScience Workshop was granted access to the HPC resources of the Institut du développement et des ressources en informatique scientifique (IDRIS) du Centre national de la recherche scientifique (CNRS) under the allocation 2021-A0101012475 made by Grand équipement national de calcul intensif (GENCI). Model training ran on the Jean-Zay cluster of IDRIS, and we thank the IDRIS team for their responsive support throughout the project, in particular Rémi Lacroix.
section_header
References
#/body
BOTTOMLEFT
#/texts/114
References
text
Abadji, J., P. J. Ortiz Suárez, L. Romary, and B. Sagot (2021, July). Ungoliant: An Optimized Pipeline for the Generation of a Very Large-Scale Multilingual Web Corpus. In CMLC 2021 - 9th Workshop on Challenges in the Management of Large Corpora , Limerick / Virtual, Ireland.
#/body
BOTTOMLEFT
#/texts/115
Abadji, J., P. J. Ortiz Suárez, L. Romary, and B. Sagot (2021, July). Ungoliant: An Optimized Pipeline for the Generation of a Very Large-Scale Multilingual Web Corpus. In CMLC 2021 - 9th Workshop on Challenges in the Management of Large Corpora , Limerick / Virtual, Ireland.
text
Abdelali, A., F. Guzman, H. Sajjad, and S. Vogel (2014, may). The amara corpus: Building parallel language resources for the educational domain. In N. C. C. Chair), K. Choukri, T. Declerck, H. Loftsson, B. Maegaard, J. Mariani, A. Moreno, J. Odijk, and S. Piperidis (Eds.), Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14) , Reykjavik, Iceland. European Language Resources Association (ELRA).
#/body
BOTTOMLEFT
#/texts/116
Abdelali, A., F. Guzman, H. Sajjad, and S. Vogel (2014, may). The amara corpus: Building parallel language resources for the educational domain. In N. C. C. Chair), K. Choukri, T. Declerck, H. Loftsson, B. Maegaard, J. Mariani, A. Moreno, J. Odijk, and S. Piperidis (Eds.), Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14) , Reykjavik, Iceland. European Language Resources Association (ELRA).
text
Adelani, D. I., J. Abbott, G. Neubig, D. D'souza, J. Kreutzer, C. Lignos, C. Palen-Michel, H. Buzaaba, S. Rijhwani, S. Ruder, S. Mayhew, I. A. Azime, S. H. Muhammad, C. C. Emezue, J. NakatumbaNabende, P. Ogayo, A. Anuoluwapo, C. Gitau, D. Mbaye, J. Alabi, S. M. Yimam, T. R. Gwadabe, I. Ezeani, R. A. Niyongabo, J. Mukiibi, V. Otiende, I. Orife, D. David, S. Ngom, T. Adewumi, P. Rayson, M. Adeyemi, G. Muriuki, E. Anebi, C. Chukwuneke, N. Odu, E. P. Wairagala, S. Oyerinde, C. Siro, T. S. Bateesa, T. Oloyede, Y. Wambui, V. Akinode, D. Nabagereka, M. Katusiime, A. Awokoya, M. MBOUP, D. Gebreyohannes, H. Tilaye, K. Nwaike, D. Wolde, A. Faye, B. Sibanda, O. Ahia, B. F. P. Dossou, K. Ogueji, T. I. DIOP, A. Diallo, A. Akinfaderin, T. Marengereke, and S. Osei (2021). MasakhaNER: Named entity recognition for African languages. Transactions of the Association for Computational Linguistics 9 , 1116-1131.
#/body
BOTTOMLEFT
#/texts/117
Adelani, D. I., J. Abbott, G. Neubig, D. D'souza, J. Kreutzer, C. Lignos, C. Palen-Michel, H. Buzaaba, S. Rijhwani, S. Ruder, S. Mayhew, I. A. Azime, S. H. Muhammad, C. C. Emezue, J. NakatumbaNabende, P. Ogayo, A. Anuoluwapo, C. Gitau, D. Mbaye, J. Alabi, S. M. Yimam, T. R. Gwadabe, I. Ezeani, R. A. Niyongabo, J. Mukiibi, V. Otiende, I. Orife, D. David, S. Ngom, T. Adewumi, P. Rayson, M. Adeyemi, G. Muriuki, E. Anebi, C. Chukwuneke, N. Odu, E. P. Wairagala, S. Oyerinde, C. Siro, T. S. Bateesa, T. Oloyede, Y. Wambui, V. Akinode, D. Nabagereka, M. Katusiime, A. Awokoya, M. MBOUP, D. Gebreyohannes, H. Tilaye, K. Nwaike, D. Wolde, A. Faye, B. Sibanda, O. Ahia, B. F. P. Dossou, K. Ogueji, T. I. DIOP, A. Diallo, A. Akinfaderin, T. Marengereke, and S. Osei (2021). MasakhaNER: Named entity recognition for African languages. Transactions of the Association for Computational Linguistics 9 , 1116-1131.
text
Aghajanyan, A., D. Okhonko, M. Lewis, M. Joshi, H. Xu, G. Ghosh, and L. Zettlemoyer (2022). HTLM: Hyper-text pre-training and prompting of language models. In International Conference on Learning Representations .
#/body
BOTTOMLEFT
#/texts/118
Aghajanyan, A., D. Okhonko, M. Lewis, M. Joshi, H. Xu, G. Ghosh, and L. Zettlemoyer (2022). HTLM: Hyper-text pre-training and prompting of language models. In International Conference on Learning Representations .
text
Agi'c, Ž. and I. Vuli'c (2019, July). JW300: A wide-coverage parallel corpus for low-resource languages. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , Florence, Italy, pp. 3204-3210. Association for Computational Linguistics.
#/body
BOTTOMLEFT
#/texts/119
Agi'c, Ž. and I. Vuli'c (2019, July). JW300: A wide-coverage parallel corpus for low-resource languages. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , Florence, Italy, pp. 3204-3210. Association for Computational Linguistics.
text
Allamanis, M. (2019). The adverse effects of code duplication in machine learning models of code. In Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software , pp. 143-153.
#/body
BOTTOMLEFT
#/texts/120
Allamanis, M. (2019). The adverse effects of code duplication in machine learning models of code. In Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software , pp. 143-153.
text
Alrabiah, M., A. Alsalman, and E. Atwell (2013, 01). The design and construction of the 50 million words ksucca king saud university corpus of classical arabic.
#/body
BOTTOMLEFT
#/texts/121
Alrabiah, M., A. Alsalman, and E. Atwell (2013, 01). The design and construction of the 50 million words ksucca king saud university corpus of classical arabic.
text
Aly, M. and A. Atiya (2013, August). LABR: A large scale Arabic book reviews dataset. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , Sofia, Bulgaria, pp. 494-498. Association for Computational Linguistics.
#/body
BOTTOMLEFT
#/texts/122
Aly, M. and A. Atiya (2013, August). LABR: A large scale Arabic book reviews dataset. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , Sofia, Bulgaria, pp. 494-498. Association for Computational Linguistics.
text
Alyafeai, Z., M. Masoud, M. Ghaleb, and M. S. AlShaibani (2021). Masader: Metadata sourcing for arabic text and speech data resources. CoRR abs/2110.06744 .
#/body
BOTTOMLEFT
#/texts/123
Alyafeai, Z., M. Masoud, M. Ghaleb, and M. S. AlShaibani (2021). Masader: Metadata sourcing for arabic text and speech data resources. CoRR abs/2110.06744 .
text
Armengol-Estapé, J., C. P. Carrino, C. Rodriguez-Penagos, O. de Gibert Bonet, C. Armentano-Oller, A. Gonzalez-Agirre, M. Melero, and M. Villegas (2021, August). Are multilingual models the best choice for moderately under-resourced languages? A comprehensive assessment for Catalan. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , Online, pp. 4933-4946. Association for Computational Linguistics.
#/body
BOTTOMLEFT
#/texts/124
Armengol-Estapé, J., C. P. Carrino, C. Rodriguez-Penagos, O. de Gibert Bonet, C. Armentano-Oller, A. Gonzalez-Agirre, M. Melero, and M. Villegas (2021, August). Are multilingual models the best choice for moderately under-resourced languages? A comprehensive assessment for Catalan. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , Online, pp. 4933-4946. Association for Computational Linguistics.
page_footer
12
#/body
BOTTOMLEFT
#/texts/125
12
text
Artetxe, M., I. Aldabe, R. Agerri, O. Perez-de Viñaspre, and A. Soroa (2022). Does corpus quality really matter for low-resource languages?
#/body
BOTTOMLEFT
#/texts/126
Artetxe, M., I. Aldabe, R. Agerri, O. Perez-de Viñaspre, and A. Soroa (2022). Does corpus quality really matter for low-resource languages?
text
Ashari, A. (2018). Indonesian news articles published at 2017.
#/body
BOTTOMLEFT
#/texts/127
Ashari, A. (2018). Indonesian news articles published at 2017.
text
Bandy, J. and N. Vincent (2021). Addressing" documentation debt" in machine learning research: A retrospective datasheet for bookcorpus. arXiv preprint arXiv:2105.05241 .
#/body
BOTTOMLEFT
#/texts/128
Bandy, J. and N. Vincent (2021). Addressing" documentation debt" in machine learning research: A retrospective datasheet for bookcorpus. arXiv preprint arXiv:2105.05241 .
text
Belinkov, Y., A. Magidow, A. Barrón-Cedeño, A. Shmidman, and M. Romanov (2019). Studying the history of the arabic language: language technology and a large-scale historical corpus. Language Resources and Evaluation 53 (4), 771-805.
#/body
BOTTOMLEFT
#/texts/129
Belinkov, Y., A. Magidow, A. Barrón-Cedeño, A. Shmidman, and M. Romanov (2019). Studying the history of the arabic language: language technology and a large-scale historical corpus. Language Resources and Evaluation 53 (4), 771-805.
text
Bender, E. M., T. Gebru, A. McMillan-Major, and S. Shmitchell (2021). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency , FAccT '21, New York, NY, USA, pp. 610-623. Association for Computing Machinery.
#/body
BOTTOMLEFT
#/texts/130
Bender, E. M., T. Gebru, A. McMillan-Major, and S. Shmitchell (2021). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency , FAccT '21, New York, NY, USA, pp. 610-623. Association for Computing Machinery.
text
Biderman, S., K. Bicheno, and L. Gao (2022). Datasheet for the pile. arXiv preprint arXiv:2201.07311 .
#/body
BOTTOMLEFT
#/texts/131
Biderman, S., K. Bicheno, and L. Gao (2022). Datasheet for the pile. arXiv preprint arXiv:2201.07311 .
text
BigScience Workshop (2022). Bloom (revision 4ab0472).
#/body
BOTTOMLEFT
#/texts/132
BigScience Workshop (2022). Bloom (revision 4ab0472).
text
Bird, S., E. Klein, and E. Loper (2009). Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit . Beijing: O'Reilly.
#/body
BOTTOMLEFT
#/texts/133
Bird, S., E. Klein, and E. Loper (2009). Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit . Beijing: O'Reilly.
text
Birhane, A., P. Kalluri, D. Card, W. Agnew, R. Dotan, and M. Bao (2021). The values encoded in machine learning research. ArXiv abs/2106.15590 .
#/body
BOTTOMLEFT
#/texts/134
Birhane, A., P. Kalluri, D. Card, W. Agnew, R. Dotan, and M. Bao (2021). The values encoded in machine learning research. ArXiv abs/2106.15590 .
text
Black, S., S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Golding, H. He, C. Leahy, K. McDonell, J. Phang, et al. (2022). Gpt-neox-20b: An open-source autoregressive language model. In Proceedings of BigScience Episode \ # 5-Workshop on Challenges & Perspectives in Creating Large Language Models , pp. 95-136.
#/body
BOTTOMLEFT
#/texts/135
Black, S., S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Golding, H. He, C. Leahy, K. McDonell, J. Phang, et al. (2022). Gpt-neox-20b: An open-source autoregressive language model. In Proceedings of BigScience Episode \ # 5-Workshop on Challenges & Perspectives in Creating Large Language Models , pp. 95-136.
text
Bondi, E., L. Xu, D. Acosta-Navas, and J. A. Killian (2021). Envisioning communities: A participatory approach towards AI for social good. In M. Fourcade, B. Kuipers, S. Lazar, and D. K. Mulligan (Eds.), AIES '21: AAAI/ACM Conference on AI, Ethics, and Society, Virtual Event, USA, May 19-21, 2021 , pp. 425-436. ACM.
#/body
BOTTOMLEFT
#/texts/136
Bondi, E., L. Xu, D. Acosta-Navas, and J. A. Killian (2021). Envisioning communities: A participatory approach towards AI for social good. In M. Fourcade, B. Kuipers, S. Lazar, and D. K. Mulligan (Eds.), AIES '21: AAAI/ACM Conference on AI, Ethics, and Society, Virtual Event, USA, May 19-21, 2021 , pp. 425-436. ACM.
list_item
-
Brown, T. B., B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei (2020). Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual .
#/groups/1
BOTTOMLEFT
#/texts/137
Brown, T. B., B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei (2020). Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual .
text
Budiono, H. Riza, and C. Hakim (2009, August). Resource report: Building parallel text corpora for multi-domain translation system. In Proceedings of the 7th Workshop on Asian Language Resources (ALR7) , Suntec, Singapore, pp. 92-95. Association for Computational Linguistics.
#/body
BOTTOMLEFT
#/texts/138
Budiono, H. Riza, and C. Hakim (2009, August). Resource report: Building parallel text corpora for multi-domain translation system. In Proceedings of the 7th Workshop on Asian Language Resources (ALR7) , Suntec, Singapore, pp. 92-95. Association for Computational Linguistics.
text
Bình, V. Q. (2021). Binhvq news corpus. https://github.com/binhvq/news-corpus.
#/body
BOTTOMLEFT
#/texts/139
Bình, V. Q. (2021). Binhvq news corpus. https://github.com/binhvq/news-corpus.
text
Carlini, N., D. Ippolito, M. Jagielski, K. Lee, F. Tramer, and C. Zhang (2022). Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646 .
#/body
BOTTOMLEFT
#/texts/140
Carlini, N., D. Ippolito, M. Jagielski, K. Lee, F. Tramer, and C. Zhang (2022). Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646 .
text
Carlini, N., C. Liu, Ú. Erlingsson, J. Kos, and D. Song (2019). The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19) , pp. 267-284.
#/body
BOTTOMLEFT
#/texts/141
Carlini, N., C. Liu, Ú. Erlingsson, J. Kos, and D. Song (2019). The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19) , pp. 267-284.
text
Carrino, C. P., C. G. Rodriguez-Penagos, and C. Armentano-Oller (2021, March). Tecla: Text classification catalan dataset.
#/body
BOTTOMLEFT
#/texts/142
Carrino, C. P., C. G. Rodriguez-Penagos, and C. Armentano-Oller (2021, March). Tecla: Text classification catalan dataset.
text
Caselli, T., R. Cibin, C. Conforti, E. Encinas, and M. Teli (2021). Guiding principles for participatory design-inspired natural language processing. Proceedings of the 1st Workshop on NLP for Positive Impact .
#/body
BOTTOMLEFT
#/texts/143
Caselli, T., R. Cibin, C. Conforti, E. Encinas, and M. Teli (2021). Guiding principles for participatory design-inspired natural language processing. Proceedings of the 1st Workshop on NLP for Positive Impact .
page_footer
13
#/body
BOTTOMLEFT
#/texts/144
13
page_footer
14
#/body
BOTTOMLEFT
#/texts/145
14
text
Gokaslan, A. and V. Cohen (2019). Openwebtext corpus. http://Skylion007.github.io/ OpenWebTextCorpus .
#/body
BOTTOMLEFT
#/texts/146
Gokaslan, A. and V. Cohen (2019). Openwebtext corpus. http://Skylion007.github.io/ OpenWebTextCorpus .
text
Goldhahn, D., T. Eckart, and U. Quasthoff (2012, May). Building large monolingual dictionaries at the Leipzig corpora collection: From 100 to 200 languages. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12) , Istanbul, Turkey, pp. 759-765. European Language Resources Association (ELRA).
#/body
BOTTOMLEFT
#/texts/147
Goldhahn, D., T. Eckart, and U. Quasthoff (2012, May). Building large monolingual dictionaries at the Leipzig corpora collection: From 100 to 200 languages. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12) , Istanbul, Turkey, pp. 759-765. European Language Resources Association (ELRA).
text
He, W., K. Liu, J. Liu, Y. Lyu, S. Zhao, X. Xiao, Y. Liu, Y. Wang, H. Wu, Q. She, X. Liu, T. Wu, and H. Wang (2018). Dureader: a chinese machine reading comprehension dataset from real-world applications.
#/body
BOTTOMLEFT
#/texts/148
He, W., K. Liu, J. Liu, Y. Lyu, S. Zhao, X. Xiao, Y. Liu, Y. Wang, H. Wu, Q. She, X. Liu, T. Wu, and H. Wang (2018). Dureader: a chinese machine reading comprehension dataset from real-world applications.
text
Heafield, K. (2011, July). KenLM: Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation , Edinburgh, Scotland, pp. 187-197. Association for Computational Linguistics.
#/body
BOTTOMLEFT
#/texts/149
Heafield, K. (2011, July). KenLM: Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation , Edinburgh, Scotland, pp. 187-197. Association for Computational Linguistics.
text
Ho, V. A., D. H.-C. Nguyen, D. H. Nguyen, L. T.-V. Pham, D.-V. Nguyen, K. V. Nguyen, and N. L.-T. Nguyen (2020). Emotion recognition for vietnamese social media text.
#/body
BOTTOMLEFT
#/texts/150
Ho, V. A., D. H.-C. Nguyen, D. H. Nguyen, L. T.-V. Pham, D.-V. Nguyen, K. V. Nguyen, and N. L.-T. Nguyen (2020). Emotion recognition for vietnamese social media text.
text
Hoffmann, J., S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. v. d. Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre (2022). Training compute-optimal large language models.
#/body
BOTTOMLEFT
#/texts/151
Hoffmann, J., S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. v. d. Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre (2022). Training compute-optimal large language models.
text
Howard, J. and S. Ruder (2018). Fine-tuned language models for text classification. CoRR abs/1801.06146 .
#/body
BOTTOMLEFT
#/texts/152
Howard, J. and S. Ruder (2018). Fine-tuned language models for text classification. CoRR abs/1801.06146 .
text
Jawaid, B., A. Kamran, and O. Bojar (2014). Urdu monolingual corpus.
#/body
BOTTOMLEFT
#/texts/153
Jawaid, B., A. Kamran, and O. Bojar (2014). Urdu monolingual corpus.
text
Jernite, Y., H. Nguyen, S. Biderman, A. Rogers, M. Masoud, V. Danchev, S. Tan, A. S. Luccioni, N. Subramani, G. Dupont, J. Dodge, K. Lo, Z. Talat, D. Radev, A. Gokaslan, S. Nikpoor, P. Henderson, R. Bommasani, and M. Mitchell (2022). Data governance in the age of large-scale data-driven language technology. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency , FAccT '22, New York, NY, USA. Association for Computing Machinery.
#/body
BOTTOMLEFT
#/texts/154
Jernite, Y., H. Nguyen, S. Biderman, A. Rogers, M. Masoud, V. Danchev, S. Tan, A. S. Luccioni, N. Subramani, G. Dupont, J. Dodge, K. Lo, Z. Talat, D. Radev, A. Gokaslan, S. Nikpoor, P. Henderson, R. Bommasani, and M. Mitchell (2022). Data governance in the age of large-scale data-driven language technology. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency , FAccT '22, New York, NY, USA. Association for Computing Machinery.
list_item
-
Joulin, A., E. Grave, P. Bojanowski, and T. Mikolov (2017, April). Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers , pp. 427-431. Association for Computational Linguistics.
#/groups/2
BOTTOMLEFT
#/texts/155
Joulin, A., E. Grave, P. Bojanowski, and T. Mikolov (2017, April). Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers , pp. 427-431. Association for Computational Linguistics.
text
Kandpal, N., E. Wallace, and C. Raffel (2022). Deduplicating training data mitigates privacy risks in language models. arXiv preprint arXiv:2202.06539 .
#/body
BOTTOMLEFT
#/texts/156
Kandpal, N., E. Wallace, and C. Raffel (2022). Deduplicating training data mitigates privacy risks in language models. arXiv preprint arXiv:2202.06539 .
text
Kaplan, J., S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei (2020). Scaling laws for neural language models. CoRR abs/2001.08361 .
#/body
BOTTOMLEFT
#/texts/157
Kaplan, J., S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei (2020). Scaling laws for neural language models. CoRR abs/2001.08361 .
text
Kermes, H., S. Degaetano-Ortlieb, A. Khamis, J. Knappen, and E. Teich (2016, May). The royal society corpus: From uncharted data to corpus. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16) , Portorož, Slovenia, pp. 1928-1931. European Language Resources Association (ELRA).
#/body
BOTTOMLEFT
#/texts/158
Kermes, H., S. Degaetano-Ortlieb, A. Khamis, J. Knappen, and E. Teich (2016, May). The royal society corpus: From uncharted data to corpus. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16) , Portorož, Slovenia, pp. 1928-1931. European Language Resources Association (ELRA).
text
Kowsher, M., M. Uddin, A. Tahabilder, M. Ruhul Amin, M. F. Shahriar, and M. S. I. Sobuj (2021, September). Banglalm: Bangla corpus for language model research. Online. IEEE.
#/body
BOTTOMLEFT
#/texts/159
Kowsher, M., M. Uddin, A. Tahabilder, M. Ruhul Amin, M. F. Shahriar, and M. S. I. Sobuj (2021, September). Banglalm: Bangla corpus for language model research. Online. IEEE.
text
Kreutzer, J., I. Caswell, L. Wang, A. Wahab, D. van Esch, N. Ulzii-Orshikh, A. Tapo, N. Subramani, A. Sokolov, C. Sikasote, M. Setyawan, S. Sarin, S. Samb, B. Sagot, C. Rivera, A. Rios, I. Papadimitriou, S. Osei, P. O. Suarez, I. Orife, K. Ogueji, R. Niyongabo, T. Nguyen, M. Müller, A. Müller, S. Muhammad, N. Muhammad, A. Mnyakeni, J. Mirzakhalov, T. Matangira, C. Leong, N. Lawson, S. Kudugunta, Y. Jernite, M. Jenny, O. Firat, B. Dossou, S. Dlamini, N. de Silva, S. Çabuk Ballı, S. Biderman, A. Battisti, A. Baruwa, A. Bapna, P. Baljekar, I. Azime, A. Awokoya, D. Ataman, O. Ahia, O. Ahia, S. Agrawal, and M. Adeyemi (2022). Quality at a glance: An audit of web-crawled multilingual datasets. Transactions of the Association for Computational Linguistics 10 (0), 50-72.
#/body
BOTTOMLEFT
#/texts/160
Kreutzer, J., I. Caswell, L. Wang, A. Wahab, D. van Esch, N. Ulzii-Orshikh, A. Tapo, N. Subramani, A. Sokolov, C. Sikasote, M. Setyawan, S. Sarin, S. Samb, B. Sagot, C. Rivera, A. Rios, I. Papadimitriou, S. Osei, P. O. Suarez, I. Orife, K. Ogueji, R. Niyongabo, T. Nguyen, M. Müller, A. Müller, S. Muhammad, N. Muhammad, A. Mnyakeni, J. Mirzakhalov, T. Matangira, C. Leong, N. Lawson, S. Kudugunta, Y. Jernite, M. Jenny, O. Firat, B. Dossou, S. Dlamini, N. de Silva, S. Çabuk Ballı, S. Biderman, A. Battisti, A. Baruwa, A. Bapna, P. Baljekar, I. Azime, A. Awokoya, D. Ataman, O. Ahia, O. Ahia, S. Agrawal, and M. Adeyemi (2022). Quality at a glance: An audit of web-crawled multilingual datasets. Transactions of the Association for Computational Linguistics 10 (0), 50-72.
page_footer
15
#/body
BOTTOMLEFT
#/texts/161
15
text
Kudo, T. (2018, July). Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , Melbourne, Australia, pp. 66-75. Association for Computational Linguistics.
#/body
BOTTOMLEFT
#/texts/162
Kudo, T. (2018, July). Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , Melbourne, Australia, pp. 66-75. Association for Computational Linguistics.
text
Kunchukuttan, A. (2020). The IndicNLP Library. https://github.com/anoopkunchukuttan/indic_ nlp_library/blob/master/docs/indicnlp.pdf .
#/body
BOTTOMLEFT
#/texts/163
Kunchukuttan, A. (2020). The IndicNLP Library. https://github.com/anoopkunchukuttan/indic_ nlp_library/blob/master/docs/indicnlp.pdf .
text
Kunchukuttan, A., D. Kakwani, S. Golla, G. N.C., A. Bhattacharyya, M. M. Khapra, and P. Kumar (2020). Ai4bharat-indicnlp corpus: Monolingual corpora and word embeddings for indic languages. arXiv preprint arXiv:2005.00085 .
#/body
BOTTOMLEFT
#/texts/164
Kunchukuttan, A., D. Kakwani, S. Golla, G. N.C., A. Bhattacharyya, M. M. Khapra, and P. Kumar (2020). Ai4bharat-indicnlp corpus: Monolingual corpora and word embeddings for indic languages. arXiv preprint arXiv:2005.00085 .
list_item
-
Kunchukuttan, A., P. Mehta, and P. Bhattacharyya (2018, May). The IIT Bombay English-Hindi parallel corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) , Miyazaki, Japan. European Language Resources Association (ELRA).
#/groups/3
BOTTOMLEFT
#/texts/165
Kunchukuttan, A., P. Mehta, and P. Bhattacharyya (2018, May). The IIT Bombay English-Hindi parallel corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) , Miyazaki, Japan. European Language Resources Association (ELRA).
text
Kurniawan, K. and S. Louvan (2018). Indosum: A new benchmark dataset for indonesian text summarization. In 2018 International Conference on Asian Language Processing (IALP) , pp. 215-220. IEEE.
#/body
BOTTOMLEFT
#/texts/166
Kurniawan, K. and S. Louvan (2018). Indosum: A new benchmark dataset for indonesian text summarization. In 2018 International Conference on Asian Language Processing (IALP) , pp. 215-220. IEEE.
text
Külebi, B. (2021, October). ParlamentParla - Speech corpus of Catalan Parliamentary sessions.
#/body
BOTTOMLEFT
#/texts/167
Külebi, B. (2021, October). ParlamentParla - Speech corpus of Catalan Parliamentary sessions.
text
Leahy, C. and S. Biderman (2021). The hard problem of aligning AI to human values. In The State of AI Ethics Report , Volume 4, pp. 180-183. The Montreal AI Ethics Institute.
#/body
BOTTOMLEFT
#/texts/168
Leahy, C. and S. Biderman (2021). The hard problem of aligning AI to human values. In The State of AI Ethics Report , Volume 4, pp. 180-183. The Montreal AI Ethics Institute.
text
Lee, K., D. Ippolito, A. Nystrom, C. Zhang, D. Eck, C. Callison-Burch, and N. Carlini (2022). Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics.
#/body
BOTTOMLEFT
#/texts/169
Lee, K., D. Ippolito, A. Nystrom, C. Zhang, D. Eck, C. Callison-Burch, and N. Carlini (2022). Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics.
text
Lhoest, Q., A. Villanova del Moral, Y. Jernite, A. Thakur, P. von Platen, S. Patil, J. Chaumond, M. Drame, J. Plu, L. Tunstall, J. Davison, M. Šaško, G. Chhablani, B. Malik, S. Brandeis, T. Le Scao, V. Sanh, C. Xu, N. Patry, A. McMillan-Major, P. Schmid, S. Gugger, C. Delangue, T. Matussière, L. Debut, S. Bekman, P. Cistac, T. Goehringer, V. Mustar, F. Lagunas, A. Rush, and T. Wolf (2021, November). Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , Online and Punta Cana, Dominican Republic, pp. 175-184. Association for Computational Linguistics.
#/body
BOTTOMLEFT
#/texts/170
Lhoest, Q., A. Villanova del Moral, Y. Jernite, A. Thakur, P. von Platen, S. Patil, J. Chaumond, M. Drame, J. Plu, L. Tunstall, J. Davison, M. Šaško, G. Chhablani, B. Malik, S. Brandeis, T. Le Scao, V. Sanh, C. Xu, N. Patry, A. McMillan-Major, P. Schmid, S. Gugger, C. Delangue, T. Matussière, L. Debut, S. Bekman, P. Cistac, T. Goehringer, V. Mustar, F. Lagunas, A. Rush, and T. Wolf (2021, November). Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , Online and Punta Cana, Dominican Republic, pp. 175-184. Association for Computational Linguistics.
list_item
-
Li, Y., D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. D. Lago, T. Hubert, P. Choy, C. d. M. d'Autume, I. Babuschkin, X. Chen, P.-S. Huang, J. Welbl, S. Gowal, A. Cherepanov, J. Molloy, D. J. Mankowitz, E. S. Robson, P. Kohli, N. de Freitas, K. Kavukcuoglu, and O. Vinyals (2022). Competition-level code generation with alphacode.
#/groups/4
BOTTOMLEFT
#/texts/171
Li, Y., D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. D. Lago, T. Hubert, P. Choy, C. d. M. d'Autume, I. Babuschkin, X. Chen, P.-S. Huang, J. Welbl, S. Gowal, A. Cherepanov, J. Molloy, D. J. Mankowitz, E. S. Robson, P. Kohli, N. de Freitas, K. Kavukcuoglu, and O. Vinyals (2022). Competition-level code generation with alphacode.
list_item
-
Lieber, O., O. Sharir, B. Lenz, and Y. Shoham (2021). Jurassic-1: Technical details and evaluation. White Paper. AI21 Labs .
#/groups/4
BOTTOMLEFT
#/texts/172
Lieber, O., O. Sharir, B. Lenz, and Y. Shoham (2021). Jurassic-1: Technical details and evaluation. White Paper. AI21 Labs .
text
Lison, P. and J. Tiedemann (2016). Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles.
#/body
BOTTOMLEFT
#/texts/173
Lison, P. and J. Tiedemann (2016). Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles.
list_item
-
Lo, K., L. L. Wang, M. Neumann, R. Kinney, and D. Weld (2020, July). S2ORC: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , Online, pp. 4969-4983. Association for Computational Linguistics.
#/groups/5
BOTTOMLEFT
#/texts/174
Lo, K., L. L. Wang, M. Neumann, R. Kinney, and D. Weld (2020, July). S2ORC: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , Online, pp. 4969-4983. Association for Computational Linguistics.
text
Lopes, C. V., P. Maj, P. Martins, V. Saini, D. Yang, J. Zitny, H. Sajnani, and J. Vitek (2017). Déjàvu: a map of code duplicates on github. Proceedings of the ACM on Programming Languages 1 (OOPSLA), 1-28.
#/body
BOTTOMLEFT
#/texts/175
Lopes, C. V., P. Maj, P. Martins, V. Saini, D. Yang, J. Zitny, H. Sajnani, and J. Vitek (2017). Déjàvu: a map of code duplicates on github. Proceedings of the ACM on Programming Languages 1 (OOPSLA), 1-28.
text
Luccioni, A. S. and J. D. Viviano (2021). What's in the box? a preliminary analysis of undesirable content in the common crawl corpus. Published in the Proceedings of ACL 2021 .
#/body
BOTTOMLEFT
#/texts/176
Luccioni, A. S. and J. D. Viviano (2021). What's in the box? a preliminary analysis of undesirable content in the common crawl corpus. Published in the Proceedings of ACL 2021 .
list_item
-
Mahendra, R., A. F. Aji, S. Louvan, F. Rahman, and C. Vania (2021). Indonli: A natural language inference dataset for indonesian. arXiv preprint arXiv:2110.14566 .
#/groups/6
BOTTOMLEFT
#/texts/177
Mahendra, R., A. F. Aji, S. Louvan, F. Rahman, and C. Vania (2021). Indonli: A natural language inference dataset for indonesian. arXiv preprint arXiv:2110.14566 .
page_footer
16
#/body
BOTTOMLEFT
#/texts/178
16
text
Manber, U. and G. Myers (1993). Suffix arrays: A new method for on-line string searches. SIAM Journal on Computing 22 (5), 935-948.
#/body
BOTTOMLEFT
#/texts/179
Manber, U. and G. Myers (1993). Suffix arrays: A new method for on-line string searches. SIAM Journal on Computing 22 (5), 935-948.
text
Manku, G. S., A. Jain, and A. Das Sarma (2007). Detecting near-duplicates for web crawling. In Proceedings of the 16th International Conference on World Wide Web , WWW '07, New York, NY, USA, pp. 141-150. Association for Computing Machinery.
#/body
BOTTOMLEFT
#/texts/180
Manku, G. S., A. Jain, and A. Das Sarma (2007). Detecting near-duplicates for web crawling. In Proceedings of the 16th International Conference on World Wide Web , WWW '07, New York, NY, USA, pp. 141-150. Association for Computing Machinery.
text
Mayeesha, T. T., A. M. Sarwar, and R. M. Rahman (2020, November). Deep learning based question answering system in Bengali.
#/body
BOTTOMLEFT
#/texts/181
Mayeesha, T. T., A. M. Sarwar, and R. M. Rahman (2020, November). Deep learning based question answering system in Bengali.
list_item
-
McMillan-Major, A., Z. Alyafeai, S. Biderman, K. Chen, F. De Toni, G. Dupont, H. Elsahar, C. Emezue, A. F. Aji, S. Ili'c, N. Khamis, C. Leong, M. Masoud, A. Soroa, P. O. Suarez, Z. Talat, D. van Strien, and Y. Jernite (2022). Documenting geographically and contextually diverse data sources: The bigscience catalogue of language data and resources.
#/groups/7
BOTTOMLEFT
#/texts/182
McMillan-Major, A., Z. Alyafeai, S. Biderman, K. Chen, F. De Toni, G. Dupont, H. Elsahar, C. Emezue, A. F. Aji, S. Ili'c, N. Khamis, C. Leong, M. Masoud, A. Soroa, P. O. Suarez, Z. Talat, D. van Strien, and Y. Jernite (2022). Documenting geographically and contextually diverse data sources: The bigscience catalogue of language data and resources.
text
Moeljadi, D. (2012). Usage of indonesian possessive verbal predicates: A statistical analysis based on questionnaire and storytelling surveys. In 5th Conference on Austronesian and Papuan Languages and Linguistics (APLL5) , SOAS, University of London.
#/body
BOTTOMLEFT
#/texts/183
Moeljadi, D. (2012). Usage of indonesian possessive verbal predicates: A statistical analysis based on questionnaire and storytelling surveys. In 5th Conference on Austronesian and Papuan Languages and Linguistics (APLL5) , SOAS, University of London.
text
Mohr, G., J. Kunze, and M. Stack (2008). The warc file format 1.0 (iso 28500).
#/body
BOTTOMLEFT
#/texts/184
Mohr, G., J. Kunze, and M. Stack (2008). The warc file format 1.0 (iso 28500).
text
Nekoto, W., V. Marivate, T. Matsila, T. E. Fasubaa, T. Fagbohungbe, S. O. Akinola, S. H. Muhammad, S. K. Kabenamualu, S. Osei, F. Sackey, R. A. Niyongabo, R. Macharm, P. Ogayo, O. Ahia, M. M. Berhe, M. Adeyemi, M. Mokgesi-Selinga, L. Okegbemi, L. Martinus, K. Tajudeen, K. Degila, K. Ogueji, K. Siminyu, J. Kreutzer, J. Webster, J. T. Ali, J. Z. Abbott, I. Orife, I. Ezeani, I. A. Dangana, H. Kamper, H. Elsahar, G. Duru, G. Kioko, E. Murhabazi, E. V. Biljon, D. Whitenack, C. Onyefuluchi, C. C. Emezue, B. F. P. Dossou, B. Sibanda, B. I. Bassey, A. Olabiyi, A. Ramkilowan, A. Öktem, A. Akinfaderin, and A. Bashir (2020). Participatory research for low-resourced machine translation: A case study in african languages. In T. Cohn, Y. He, and Y. Liu (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020 , Volume EMNLP 2020 of Findings of ACL , pp. 2144-2160. Association for Computational Linguistics.
#/body
BOTTOMLEFT
#/texts/185
Nekoto, W., V. Marivate, T. Matsila, T. E. Fasubaa, T. Fagbohungbe, S. O. Akinola, S. H. Muhammad, S. K. Kabenamualu, S. Osei, F. Sackey, R. A. Niyongabo, R. Macharm, P. Ogayo, O. Ahia, M. M. Berhe, M. Adeyemi, M. Mokgesi-Selinga, L. Okegbemi, L. Martinus, K. Tajudeen, K. Degila, K. Ogueji, K. Siminyu, J. Kreutzer, J. Webster, J. T. Ali, J. Z. Abbott, I. Orife, I. Ezeani, I. A. Dangana, H. Kamper, H. Elsahar, G. Duru, G. Kioko, E. Murhabazi, E. V. Biljon, D. Whitenack, C. Onyefuluchi, C. C. Emezue, B. F. P. Dossou, B. Sibanda, B. I. Bassey, A. Olabiyi, A. Ramkilowan, A. Öktem, A. Akinfaderin, and A. Bashir (2020). Participatory research for low-resourced machine translation: A case study in african languages. In T. Cohn, Y. He, and Y. Liu (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020 , Volume EMNLP 2020 of Findings of ACL , pp. 2144-2160. Association for Computational Linguistics.
text
Ngo, C. and T. H. Trinh (2021). Styled augmented translation (sat). https://github.com/vietai/SAT .
#/body
BOTTOMLEFT
#/texts/186
Ngo, C. and T. H. Trinh (2021). Styled augmented translation (sat). https://github.com/vietai/SAT .
text
Nguyen, K. V., V. D. Nguyen, P. X. V. Nguyen, T. T. H. Truong, and N. L.-T. Nguyen (2018). Uit-vsfc: Vietnamese students' feedback corpus for sentiment analysis. In 2018 10th International Conference on Knowledge and Systems Engineering (KSE) , pp. 19-24.
#/body
BOTTOMLEFT
#/texts/187
Nguyen, K. V., V. D. Nguyen, P. X. V. Nguyen, T. T. H. Truong, and N. L.-T. Nguyen (2018). Uit-vsfc: Vietnamese students' feedback corpus for sentiment analysis. In 2018 10th International Conference on Knowledge and Systems Engineering (KSE) , pp. 19-24.
text
Nguyen, T., H. Pham, M. Truong, H. Duc, and P. Tan (2021). Vietnamese poem generator. https://github.com/fsoft-ailab/Poem-Generator.
#/body
BOTTOMLEFT
#/texts/188
Nguyen, T., H. Pham, M. Truong, H. Duc, and P. Tan (2021). Vietnamese poem generator. https://github.com/fsoft-ailab/Poem-Generator.
text
Nomoto, H., K. Okano, D. Moeljadi, and H. Sawada (2018). Tufs asian language parallel corpus (talpco). In Proceedings of the Twenty-Fourth Annual Meeting of the Association for Natural Language Processing , pp. 436-439. Association for Natural Language Processing.
#/body
BOTTOMLEFT
#/texts/189
Nomoto, H., K. Okano, D. Moeljadi, and H. Sawada (2018). Tufs asian language parallel corpus (talpco). In Proceedings of the Twenty-Fourth Annual Meeting of the Association for Natural Language Processing , pp. 436-439. Association for Natural Language Processing.
text
Ortiz Suárez, P. J., L. Romary, and B. Sagot (2020, July). A monolingual approach to contextualized word embeddings for mid-resource languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , Online, pp. 1703-1714. Association for Computational Linguistics.
#/body
BOTTOMLEFT
#/texts/190
Ortiz Suárez, P. J., L. Romary, and B. Sagot (2020, July). A monolingual approach to contextualized word embeddings for mid-resource languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , Online, pp. 1703-1714. Association for Computational Linguistics.
text
Ortiz Suárez, P. J., B. Sagot, and L. Romary (2019). Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures. Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, Mannheim, pp. 9 16. Leibniz-Institut für Deutsche Sprache.
#/body
BOTTOMLEFT
#/texts/191
Ortiz Suárez, P. J., B. Sagot, and L. Romary (2019). Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures. Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, Mannheim, pp. 9 16. Leibniz-Institut für Deutsche Sprache.
text
Parida, S., O. Bojar, and S. R. Dash (2020). Odiencorp: Odia-english and odia-only corpus for machine translation. In Smart Intelligent Computing and Applications , pp. 495-504. Springer.
#/body
BOTTOMLEFT
#/texts/192
Parida, S., O. Bojar, and S. R. Dash (2020). Odiencorp: Odia-english and odia-only corpus for machine translation. In Smart Intelligent Computing and Applications , pp. 495-504. Springer.
text
Pisceldo, F., R. Manurung, and M. Adriani (2009). Probabilistic part-of-speech tagging for bahasa indonesia. In Third International Workshop on Malay and Indonesian Language Engineering (MALINDO) , Suntec, Singapore.
#/body
BOTTOMLEFT
#/texts/193
Pisceldo, F., R. Manurung, and M. Adriani (2009). Probabilistic part-of-speech tagging for bahasa indonesia. In Third International Workshop on Malay and Indonesian Language Engineering (MALINDO) , Suntec, Singapore.
page_footer
17
#/body
BOTTOMLEFT
#/texts/194
17
list_item
-
Qi, P., Y. Zhang, Y. Zhang, J. Bolton, and C. D. Manning (2020). Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations .
#/groups/8
BOTTOMLEFT
#/texts/195
Qi, P., Y. Zhang, Y. Zhang, J. Bolton, and C. D. Manning (2020). Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations .
text
Radford, A., J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever (2019). Language models are unsupervised multitask learners.
#/body
BOTTOMLEFT
#/texts/196
Radford, A., J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever (2019). Language models are unsupervised multitask learners.
text
Rae, J. W., S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, H. F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, L. A. Hendricks, M. Rauh, P. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mellor, I. Higgins, A. Creswell, N. McAleese, A. Wu, E. Elsen, S. M. Jayakumar, E. Buchatskaya, D. Budden, E. Sutherland, K. Simonyan, M. Paganini, L. Sifre, L. Martens, X. L. Li, A. Kuncoro, A. Nematzadeh, E. Gribovskaya, D. Donato, A. Lazaridou, A. Mensch, J. Lespiau, M. Tsimpoukelli, N. Grigorev, D. Fritz, T. Sottiaux, M. Pajarskas, T. Pohlen, Z. Gong, D. Toyama, C. de Masson d'Autume, Y. Li, T. Terzi, V. Mikulik, I. Babuschkin, A. Clark, D. de Las Casas, A. Guy, C. Jones, J. Bradbury, M. Johnson, B. A. Hechtman, L. Weidinger, I. Gabriel, W. S. Isaac, E. Lockhart, S. Osindero, L. Rimell, C. Dyer, O. Vinyals, K. Ayoub, J. Stanway, L. Bennett, D. Hassabis, K. Kavukcuoglu, and G. Irving (2021). Scaling language models: Methods, analysis & insights from training gopher. CoRR abs/2112.11446 .
#/body
BOTTOMLEFT
#/texts/197
Rae, J. W., S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, H. F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, L. A. Hendricks, M. Rauh, P. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mellor, I. Higgins, A. Creswell, N. McAleese, A. Wu, E. Elsen, S. M. Jayakumar, E. Buchatskaya, D. Budden, E. Sutherland, K. Simonyan, M. Paganini, L. Sifre, L. Martens, X. L. Li, A. Kuncoro, A. Nematzadeh, E. Gribovskaya, D. Donato, A. Lazaridou, A. Mensch, J. Lespiau, M. Tsimpoukelli, N. Grigorev, D. Fritz, T. Sottiaux, M. Pajarskas, T. Pohlen, Z. Gong, D. Toyama, C. de Masson d'Autume, Y. Li, T. Terzi, V. Mikulik, I. Babuschkin, A. Clark, D. de Las Casas, A. Guy, C. Jones, J. Bradbury, M. Johnson, B. A. Hechtman, L. Weidinger, I. Gabriel, W. S. Isaac, E. Lockhart, S. Osindero, L. Rimell, C. Dyer, O. Vinyals, K. Ayoub, J. Stanway, L. Bennett, D. Hassabis, K. Kavukcuoglu, and G. Irving (2021). Scaling language models: Methods, analysis & insights from training gopher. CoRR abs/2112.11446 .
text
Raffel, C., N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 21 , 1-67.
#/body
BOTTOMLEFT
#/texts/198
Raffel, C., N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 21 , 1-67.
text
Rahman, M., E. Kumar Dey, et al. (2018). Datasets for aspect-based sentiment analysis in bangla and its baseline evaluation. Data 3 (2), 15.
#/body
BOTTOMLEFT
#/texts/199
Rahman, M., E. Kumar Dey, et al. (2018). Datasets for aspect-based sentiment analysis in bangla and its baseline evaluation. Data 3 (2), 15.
text
Rahutomo, F. and A. Miqdad Muadz Muzad (2018). Indonesian news corpus.
#/body
BOTTOMLEFT
#/texts/200
Rahutomo, F. and A. Miqdad Muadz Muzad (2018). Indonesian news corpus.
text
Rajkomar, A., M. Hardt, M. D. Howell, G. Corrado, and M. H. Chin (2018). Ensuring fairness in machine learning to advance health equity. Annals of Internal Medicine 169 , 866-872.
#/body
BOTTOMLEFT
#/texts/201
Rajkomar, A., M. Hardt, M. D. Howell, G. Corrado, and M. H. Chin (2018). Ensuring fairness in machine learning to advance health equity. Annals of Internal Medicine 169 , 866-872.
text
Ramesh, G., S. Doddapaneni, A. Bheemaraj, M. Jobanputra, R. AK, A. Sharma, S. Sahoo, H. Diddee, M. J, D. Kakwani, N. Kumar, A. Pradeep, S. Nagaraj, K. Deepak, V. Raghavan, A. Kunchukuttan, P. Kumar, and M. S. Khapra (2021). Samanantar: The largest publicly available parallel corpora collection for 11 indic languages.
#/body
BOTTOMLEFT
#/texts/202
Ramesh, G., S. Doddapaneni, A. Bheemaraj, M. Jobanputra, R. AK, A. Sharma, S. Sahoo, H. Diddee, M. J, D. Kakwani, N. Kumar, A. Pradeep, S. Nagaraj, K. Deepak, V. Raghavan, A. Kunchukuttan, P. Kumar, and M. S. Khapra (2021). Samanantar: The largest publicly available parallel corpora collection for 11 indic languages.
text
Rodriguez-Penagos, C. G. and C. Armentano-Oller (2021a, June). Enriched conllu ancora for ml training.
#/body
BOTTOMLEFT
#/texts/203
Rodriguez-Penagos, C. G. and C. Armentano-Oller (2021a, June). Enriched conllu ancora for ml training.
list_item
-
Rodriguez-Penagos, C. G. and C. Armentano-Oller (2021b, February). VilaQuAD: an extractive QA dataset from Catalan newswire.
#/groups/9
BOTTOMLEFT
#/texts/204
Rodriguez-Penagos, C. G. and C. Armentano-Oller (2021b, February). VilaQuAD: an extractive QA dataset from Catalan newswire.
text
Rodriguez-Penagos, C. G. and C. Armentano-Oller (2021c, February). ViquiQuAD: an extractive QA dataset from Catalan Wikipedia.
#/body
BOTTOMLEFT
#/texts/205
Rodriguez-Penagos, C. G. and C. Armentano-Oller (2021c, February). ViquiQuAD: an extractive QA dataset from Catalan Wikipedia.
text
Rogers, A. (2021, August). Changing the world by changing the data. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , Online, pp. 2182-2194. Association for Computational Linguistics.
#/body
BOTTOMLEFT
#/texts/206
Rogers, A. (2021, August). Changing the world by changing the data. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , Online, pp. 2182-2194. Association for Computational Linguistics.
list_item
-
Sajjad, H., A. Abdelali, N. Durrani, and F. Dalvi (2020, December). AraBench: Benchmarking dialectal Arabic-English machine translation. In Proceedings of the 28th International Conference on Computational Linguistics , Barcelona, Spain (Online), pp. 5094-5107. International Committee on Computational Linguistics.
#/groups/10
BOTTOMLEFT
#/texts/207
Sajjad, H., A. Abdelali, N. Durrani, and F. Dalvi (2020, December). AraBench: Benchmarking dialectal Arabic-English machine translation. In Proceedings of the 28th International Conference on Computational Linguistics , Barcelona, Spain (Online), pp. 5094-5107. International Committee on Computational Linguistics.
list_item
-
Sakti, S., E. Kelana, H. Riza, S. Sakai, K. Markov, and S. Nakamura (2008). Development of Indonesian large vocabulary continuous speech recognition system within a-STAR project. In Proceedings of the Workshop on Technologies and Corpora for Asia-Pacific Speech Translation (TCAST) .
#/groups/10
BOTTOMLEFT
#/texts/208
Sakti, S., E. Kelana, H. Riza, S. Sakai, K. Markov, and S. Nakamura (2008). Development of Indonesian large vocabulary continuous speech recognition system within a-STAR project. In Proceedings of the Workshop on Technologies and Corpora for Asia-Pacific Speech Translation (TCAST) .
text
Scheuerman, M. K., A. Hanna, and E. Denton (2021). Do datasets have politics? disciplinary values in computer vision dataset development. Proceedings of the ACM on Human-Computer Interaction 5 , 1-37.
#/body
BOTTOMLEFT
#/texts/209
Scheuerman, M. K., A. Hanna, and E. Denton (2021). Do datasets have politics? disciplinary values in computer vision dataset development. Proceedings of the ACM on Human-Computer Interaction 5 , 1-37.
page_footer
18
#/body
BOTTOMLEFT
#/texts/210
18
page_footer
19
#/body
BOTTOMLEFT
#/texts/211
19
section_header
Appendix
#/body
BOTTOMLEFT
#/texts/212
Appendix
section_header
A Ethical Considerations and Broader Impacts Statement
#/body
BOTTOMLEFT
#/texts/213
A Ethical Considerations and Broader Impacts Statement
text
As discussed in Section 1, the BigScience Research Workshop was conceived as a collaborative and value-driven endeavor from the start. All the ethical efforts were concentrated on implementing the values chosen first on the ethical charter and then on how to articulate those core values into specific ethical sensitive issues, such as data governance. This mechanism also allows ethical thinking to guide governance regarding technical matters. The articulation between the BigScience core values and those chosen by the collaborators contributing to data efforts was central. The importance of this collective exercise is due to the social impact that technologies such as LLMs have on the people impacted, directly and indirectly, positively and negatively. Moral exercises based on consensus, discussion around values, and how to link technical actions to ethical reflections is a strength that we believe is important within ML research. A critical analysis from an ethical perspective is fundamental to making different disciplines coexist in thinking around the social impact of these technologies and well define the object of analysis, as in this case, a multilingual dataset.
#/body
BOTTOMLEFT
#/texts/214
As discussed in Section 1, the BigScience Research Workshop was conceived as a collaborative and value-driven endeavor from the start. All the ethical efforts were concentrated on implementing the values chosen first on the ethical charter and then on how to articulate those core values into specific ethical sensitive issues, such as data governance. This mechanism also allows ethical thinking to guide governance regarding technical matters. The articulation between the BigScience core values and those chosen by the collaborators contributing to data efforts was central. The importance of this collective exercise is due to the social impact that technologies such as LLMs have on the people impacted, directly and indirectly, positively and negatively. Moral exercises based on consensus, discussion around values, and how to link technical actions to ethical reflections is a strength that we believe is important within ML research. A critical analysis from an ethical perspective is fundamental to making different disciplines coexist in thinking around the social impact of these technologies and well define the object of analysis, as in this case, a multilingual dataset.
section_header
BigScience Values
#/body
BOTTOMLEFT
#/texts/215
BigScience Values
text
Motivated by recent work on the values encoded in current approaches to research in NLP and ML more broadly (Leahy and Biderman, 2021; Birhane et al., 2021), which finds that narrow definitions of performance and efficiency were often prioritized over considerations of social impact in research and development. Even more relevant to the corpus creation aspect of our project, Scheuerman et al. (2021) outline how data efforts in computer vision tend to prioritize " efficiency [over] care; universality [over] contextuality; impartiality [over] positionality . . . ". These ML research programs and systems in turn support the development of new technologies that carry these same values when deploying these technologies in production (Winner, 2017). This limits the potential positive societal benefits of the rapid advances of NLP research while increasing risks considerably.
#/body
BOTTOMLEFT
#/texts/216
Motivated by recent work on the values encoded in current approaches to research in NLP and ML more broadly (Leahy and Biderman, 2021; Birhane et al., 2021), which finds that narrow definitions of performance and efficiency were often prioritized over considerations of social impact in research and development. Even more relevant to the corpus creation aspect of our project, Scheuerman et al. (2021) outline how data efforts in computer vision tend to prioritize " efficiency [over] care; universality [over] contextuality; impartiality [over] positionality . . . ". These ML research programs and systems in turn support the development of new technologies that carry these same values when deploying these technologies in production (Winner, 2017). This limits the potential positive societal benefits of the rapid advances of NLP research while increasing risks considerably.
text
Aware of these challenges, participants in BigScience collaboratively drafted an ethical charter 2 formalizing our core values and how they are articulated. It establishes the core values in order to allow its contributors to commit to them, both individually and collectively, and to ground discussions and choices made throughout the project in a common document. These values include notably openness and reproducibility as a scientific endeavor aimed at advancing the state of the art in a way that can be understood, interrogated, and re-used; responsibility of the participants to consider the social and legal context, and the social and environmental consequences of their work; and diversity and inclusivity . These last two are especially relevant to our data efforts, which aim to include text representative of diverse languages, varieties, and uses through a participatory approach to curation.
#/body
BOTTOMLEFT
#/texts/217
Aware of these challenges, participants in BigScience collaboratively drafted an ethical charter 2 formalizing our core values and how they are articulated. It establishes the core values in order to allow its contributors to commit to them, both individually and collectively, and to ground discussions and choices made throughout the project in a common document. These values include notably openness and reproducibility as a scientific endeavor aimed at advancing the state of the art in a way that can be understood, interrogated, and re-used; responsibility of the participants to consider the social and legal context, and the social and environmental consequences of their work; and diversity and inclusivity . These last two are especially relevant to our data efforts, which aim to include text representative of diverse languages, varieties, and uses through a participatory approach to curation.
section_header
Putting Our Values into Practice
#/body
BOTTOMLEFT
#/texts/218
Putting Our Values into Practice
text
Centering Participation in Data Curation Participatory approaches play a vital role in bridging the gaps between model development and deployment and in promoting fairness in ML applications (Rajkomar et al., 2018). They have received increased attention in recent years, with newer work calling to involve participants as full stake-holders of the entire research life-cycle rather to catering their role to post hoc model evaluation (Sloane et al., 2020; Caselli et al., 2021; Bondi et al., 2021), as exemplified by an organization like Maskhane (Nekoto et al., 2020) that brings together African researchers to collaboratively build NLP for African languages.
#/body
BOTTOMLEFT
#/texts/219
Centering Participation in Data Curation Participatory approaches play a vital role in bridging the gaps between model development and deployment and in promoting fairness in ML applications (Rajkomar et al., 2018). They have received increased attention in recent years, with newer work calling to involve participants as full stake-holders of the entire research life-cycle rather to catering their role to post hoc model evaluation (Sloane et al., 2020; Caselli et al., 2021; Bondi et al., 2021), as exemplified by an organization like Maskhane (Nekoto et al., 2020) that brings together African researchers to collaboratively build NLP for African languages.
text
With regard to developing LLMs, BigScience stands in contrast to previous work on models of similar size (Brown et al., 2020; Zhang et al., 2022) - where the majority of the development occurs in-house - by promoting engagement with other communities at every stage of the project from its design to the data curation to the eventual model training and release. Specifically, on the data curation aspect which is the focus of this paper, the involvement of a wide range of participants from various linguistic communities aims to help with the following aspects. First, Kreutzer et al. (2022) have shown in recent work that multilingual text data curation done without involving language-specific expertise leads to resources that are very different from the intentions of their creators, and these limitations carry on to the models trained on these datasets. Second, resources that are developed in collaboration with other communities are more likely to be more directly relevant to them, and thus to avoid reduce replication of model development by making the artifacts and tools we develop useful
#/body
BOTTOMLEFT
#/texts/220
With regard to developing LLMs, BigScience stands in contrast to previous work on models of similar size (Brown et al., 2020; Zhang et al., 2022) - where the majority of the development occurs in-house - by promoting engagement with other communities at every stage of the project from its design to the data curation to the eventual model training and release. Specifically, on the data curation aspect which is the focus of this paper, the involvement of a wide range of participants from various linguistic communities aims to help with the following aspects. First, Kreutzer et al. (2022) have shown in recent work that multilingual text data curation done without involving language-specific expertise leads to resources that are very different from the intentions of their creators, and these limitations carry on to the models trained on these datasets. Second, resources that are developed in collaboration with other communities are more likely to be more directly relevant to them, and thus to avoid reduce replication of model development by making the artifacts and tools we develop useful
page_footer
20
#/body
BOTTOMLEFT
#/texts/221
20
text
to more people and for more languages. Third, intentional curation and proper documentation of web-scale corpora takes a significant amount of human work and expertise, which can be distributed between a large number of participants in community efforts. Finally, community involvement can help foster trust and collective ownership of the artifacts we create.
#/body
BOTTOMLEFT
#/texts/222
to more people and for more languages. Third, intentional curation and proper documentation of web-scale corpora takes a significant amount of human work and expertise, which can be distributed between a large number of participants in community efforts. Finally, community involvement can help foster trust and collective ownership of the artifacts we create.
text
Addressing the Legal Landscape The legal status of webscraped datasets is extremely unclear in many jurisdictions, putting a substantial burden on both data creators and data users who wish to be involved with this process. While the principle of fair use generally protects academic researchers, it is not recognized in all jurisdictions and may not cover research carried out in an industry context. In consultation with our Legal Scholarship and Data Governance working groups, we developed a framework (Jernite et al., 2022) to uphold the rights and responsibilities of the many stakeholders in NLP data generation and collection, and provide assurances to downstream users as to how they are and are not authorized to use the dataset (Contractor et al., 2020).
#/body
BOTTOMLEFT
#/texts/223
Addressing the Legal Landscape The legal status of webscraped datasets is extremely unclear in many jurisdictions, putting a substantial burden on both data creators and data users who wish to be involved with this process. While the principle of fair use generally protects academic researchers, it is not recognized in all jurisdictions and may not cover research carried out in an industry context. In consultation with our Legal Scholarship and Data Governance working groups, we developed a framework (Jernite et al., 2022) to uphold the rights and responsibilities of the many stakeholders in NLP data generation and collection, and provide assurances to downstream users as to how they are and are not authorized to use the dataset (Contractor et al., 2020).
section_header
Limitations of the Approach.
#/body
BOTTOMLEFT
#/texts/224
Limitations of the Approach.
text
While we believe that an approach grounded in community participation and prioritizing language expertise constitutes a promising step toward more responsible data curation and documentation, it still has important limitations. Among those, we primarily identify the use of data from the Common Crawl which represents a point of tension between our drive to present a research artifact that is comparable to previous work and values of consent and privacy (see Section 3). Our preprocessing removes some categories of PII but is still far from exhaustive, and the nature of crawled datasets makes it next to impossible to identify individual contributors and ask for their consent. Similar concerns apply to other existing NLP datasets we identified in the catalogue, including notably the WuDao web-based corpus (Yuan et al., 2021) which makes up a significant part of the Chinese language data. Additionally, while we hope that our intentional approach to selecting diverse data sources (mostly along axes of geographical diversity and domains) will lead to a more representative language dataset overall, our reliance on medium to large sources of digitized content still over-represents privileged voices and language varieties.
#/body
BOTTOMLEFT
#/texts/225
While we believe that an approach grounded in community participation and prioritizing language expertise constitutes a promising step toward more responsible data curation and documentation, it still has important limitations. Among those, we primarily identify the use of data from the Common Crawl which represents a point of tension between our drive to present a research artifact that is comparable to previous work and values of consent and privacy (see Section 3). Our preprocessing removes some categories of PII but is still far from exhaustive, and the nature of crawled datasets makes it next to impossible to identify individual contributors and ask for their consent. Similar concerns apply to other existing NLP datasets we identified in the catalogue, including notably the WuDao web-based corpus (Yuan et al., 2021) which makes up a significant part of the Chinese language data. Additionally, while we hope that our intentional approach to selecting diverse data sources (mostly along axes of geographical diversity and domains) will lead to a more representative language dataset overall, our reliance on medium to large sources of digitized content still over-represents privileged voices and language varieties.
section_header
B Details on tools used to obtain crowdsourced dataset
#/body
BOTTOMLEFT
#/texts/226
B Details on tools used to obtain crowdsourced dataset
section_header
B.1 Pseudocode to recreate the text structure from the HTML code
#/body
BOTTOMLEFT
#/texts/227
B.1 Pseudocode to recreate the text structure from the HTML code
text
The HTML code of a web page provides information about the structure of the text. The final structure of a web page is, however, the one produced by the rendering engine of the web browser and any CSS instructions. The latter two elements, which can vary enormously from one situation to another, always use the tag types for their rendering rules (Figure 9. Therefore, we have used a fairly simple heuristic on tag types to reconstruct the structure of the text extracted from an HTML code. To reconstruct the text, the HTML DOM, which can be represented as a tree (Figure 10), is traversed with an depth-first search algorithm. The text is initially empty and each time a new node with textual content is reached its content is concatenated according to the rules presented in the Algorithm 1. Block-type tags are for us: <address> , <article> , <aside> , <blockquote> , <body> , <br> , <button> , <canvas> , <caption> , <col> , <colgroup> , <dd> , <div> , <dl> , <dt> , <embed> , <fieldset> , <figcaption> , <figure> , <footer> , <form> , <h1> , <h2> , <h3> , <h4> , <h5> , <h6> , <header> , <hgroup> , <hr> , <li> , <map> , <noscript> , <object> , <ol> , <output> , <p> , <pre> , <progress> , <section> , <table> , <tbody> , <textarea> , <tfoot> , <th> , <thead> , <tr> , <ul> , and <video> . Inline-type tags are for us: <address> , <cite> , <details> , <datalist> , <iframe> , <img> , <input> , <label> , <legend> , <optgroup> , <q> , <select> , <summary> , <tbody> , <td> , and <time> .
#/body
BOTTOMLEFT
#/texts/228
The HTML code of a web page provides information about the structure of the text. The final structure of a web page is, however, the one produced by the rendering engine of the web browser and any CSS instructions. The latter two elements, which can vary enormously from one situation to another, always use the tag types for their rendering rules (Figure 9. Therefore, we have used a fairly simple heuristic on tag types to reconstruct the structure of the text extracted from an HTML code. To reconstruct the text, the HTML DOM, which can be represented as a tree (Figure 10), is traversed with an depth-first search algorithm. The text is initially empty and each time a new node with textual content is reached its content is concatenated according to the rules presented in the Algorithm 1. Block-type tags are for us: <address> , <article> , <aside> , <blockquote> , <body> , <br> , <button> , <canvas> , <caption> , <col> , <colgroup> , <dd> , <div> , <dl> , <dt> , <embed> , <fieldset> , <figcaption> , <figure> , <footer> , <form> , <h1> , <h2> , <h3> , <h4> , <h5> , <h6> , <header> , <hgroup> , <hr> , <li> , <map> , <noscript> , <object> , <ol> , <output> , <p> , <pre> , <progress> , <section> , <table> , <tbody> , <textarea> , <tfoot> , <th> , <thead> , <tr> , <ul> , and <video> . Inline-type tags are for us: <address> , <cite> , <details> , <datalist> , <iframe> , <img> , <input> , <label> , <legend> , <optgroup> , <q> , <select> , <summary> , <tbody> , <td> , and <time> .
section_header
B.2 Visualization tool use cases
#/body
BOTTOMLEFT
#/texts/229
B.2 Visualization tool use cases
text
The visualisation tool was for us an iterative tool that we used to define new cleaning and filtering methods by visualising their effect on a subset of documents. This visualisation allowed us understand the impact of functions on the dataset at every stage of the processing pipeline (Figure 11 for advertisement detection for example), prompted us to adapt pipelines as well as introduce new functions for specific cases.
#/body
BOTTOMLEFT
#/texts/230
The visualisation tool was for us an iterative tool that we used to define new cleaning and filtering methods by visualising their effect on a subset of documents. This visualisation allowed us understand the impact of functions on the dataset at every stage of the processing pipeline (Figure 11 for advertisement detection for example), prompted us to adapt pipelines as well as introduce new functions for specific cases.
page_footer
21
#/body
BOTTOMLEFT
#/texts/231
21
text
(a) HTML code
#/body
BOTTOMLEFT
#/texts/232
(a) HTML code
text
(b) Web browser rendering
#/body
BOTTOMLEFT
#/texts/233
(b) Web browser rendering
text
Figure 9: Example showing how a single line of HTML code is rendered by a browser's renderer. In this example, we can see that the tags <p> delimit different blocks which are therefore spaced by line breaks while other tags, such as <cite> , are rendered on the same line of text that precedes and follows them.
#/body
BOTTOMLEFT
#/texts/234
Figure 9: Example showing how a single line of HTML code is rendered by a browser's renderer. In this example, we can see that the tags <p> delimit different blocks which are therefore spaced by line breaks while other tags, such as <cite> , are rendered on the same line of text that precedes and follows them.
caption
Figure 10: Simplified version of HTML DOM model on an example. Left: snippet of HTML code. Right: corresponding DOM. The yellow squares represent nodes with textual content.
#/body
BOTTOMLEFT
#/texts/235
Figure 10: Simplified version of HTML DOM model on an example. Left: snippet of HTML code. Right: corresponding DOM. The yellow squares represent nodes with textual content.
text
As a typical usage of the visualisation tool as a development tool, for documents coming from pseudo-crawls, we wanted to create a method to remove the parts of the documents that looked like a template, based on the principle that these templates would be identifiable by the fact that they would be repeated lines between documents. With the first version of the pipeline we could see from the estimates of the size of the final dataset (Figure 12) that a lot of content was removed. Looking at the examples (Figure 12c), we could confirm that a large part of the article text was removed. The cause of this behaviour was due to the fact that the same article was appearing at several different URLs as the website hierarchy had changed between the different common crawl dumps. For the final pipeline, we therefore added a custom deduplication of the urls as a first operation to target this change of addresses. With the final pipeline developed, less content was removed. By manually inspecting the examples, we could observe that the content removed from the documents was indeed the one initially targeted.
#/body
BOTTOMLEFT
#/texts/236
As a typical usage of the visualisation tool as a development tool, for documents coming from pseudo-crawls, we wanted to create a method to remove the parts of the documents that looked like a template, based on the principle that these templates would be identifiable by the fact that they would be repeated lines between documents. With the first version of the pipeline we could see from the estimates of the size of the final dataset (Figure 12) that a lot of content was removed. Looking at the examples (Figure 12c), we could confirm that a large part of the article text was removed. The cause of this behaviour was due to the fact that the same article was appearing at several different URLs as the website hierarchy had changed between the different common crawl dumps. For the final pipeline, we therefore added a custom deduplication of the urls as a first operation to target this change of addresses. With the final pipeline developed, less content was removed. By manually inspecting the examples, we could observe that the content removed from the documents was indeed the one initially targeted.
section_header
B.3 Exhaustive list of functions used in (Crowd)Sourced dataset
#/body
BOTTOMLEFT
#/texts/237
B.3 Exhaustive list of functions used in (Crowd)Sourced dataset
text
We provide an exhaustive list of functions used in each of the processing pipeline for the crowdsourced dataset$^{14}$:
#/body
BOTTOMLEFT
#/texts/238
We provide an exhaustive list of functions used in each of the processing pipeline for the crowdsourced dataset$^{14}$:
text
replace_newline_with_space Takes in a batch of texts and for each text replaces the newline character "
#/body
BOTTOMLEFT
#/texts/239
replace_newline_with_space Takes in a batch of texts and for each text replaces the newline character "
text
n" with a single space.
#/body
BOTTOMLEFT
#/texts/240
n" with a single space.
text
remove_lines_with_code Takes in a batch of texts and removes lines with the following substrings: "{" , "}" , "[if" , "<script" ,
#/body
BOTTOMLEFT
#/texts/241
remove_lines_with_code Takes in a batch of texts and removes lines with the following substrings: "{" , "}" , "[if" , "<script" ,
text
$^{14}$Code is available at preparation/blob/main/preprocessing/training/clean.py
#/body
BOTTOMLEFT
#/texts/242
$^{14}$Code is available at preparation/blob/main/preprocessing/training/clean.py
page_footer
22
#/body
BOTTOMLEFT
#/texts/243
22
text
https://github.com/bigscience-workshop/data-
#/body
BOTTOMLEFT
#/texts/244
https://github.com/bigscience-workshop/data-
section_header
Algorithm 1 Pseudo-code to concatenate texts retrieved from the HTML DOM
#/body
BOTTOMLEFT
#/texts/245
Algorithm 1 Pseudo-code to concatenate texts retrieved from the HTML DOM
list_item
-
1: text ← empty string
#/groups/11
BOTTOMLEFT
#/texts/246
1: text ← empty string
text
2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12:
#/body
BOTTOMLEFT
#/texts/247
2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12:
text
for new _ text in list of texts retrieved by the DFS traversal do if new _ text is attached to a block-type tag then # Block elements are separated from the rest by a line break if text ends with a breaking line then text ← text + new _ text else if text ends with a space then text ← text without end space text ← text + breaking line + new _ text else text ← text + breaking line + new _ text end if
#/body
BOTTOMLEFT
#/texts/248
for new _ text in list of texts retrieved by the DFS traversal do if new _ text is attached to a block-type tag then # Block elements are separated from the rest by a line break if text ends with a breaking line then text ← text + new _ text else if text ends with a space then text ← text without end space text ← text + breaking line + new _ text else text ← text + breaking line + new _ text end if
list_item
-
13: else if new _ text is attached to a inline-type tag then
#/groups/12
BOTTOMLEFT
#/texts/249
13: else if new _ text is attached to a inline-type tag then
text
14:
#/body
BOTTOMLEFT
#/texts/250
14:
text
15: 16: 17: 18: 19: 20: 21: 22: 23:
#/body
BOTTOMLEFT
#/texts/251
15: 16: 17: 18: 19: 20: 21: 22: 23:
text
# space if text ends with a space or a breaking line then text ← text + new _ text else
#/body
BOTTOMLEFT
#/texts/252
# space if text ends with a space or a breaking line then text ← text + new _ text else
text
Inline elements are separated from the rest by a line break or a
#/body
BOTTOMLEFT
#/texts/253
Inline elements are separated from the rest by a line break or a
text
text ← text + space + new _ text
#/body
BOTTOMLEFT
#/texts/254
text ← text + space + new _ text
text
end if else text ← text + new _ text end if end for
#/body
BOTTOMLEFT
#/texts/255
end if else text ← text + new _ text end if end for
section_header
Old text
#/body
BOTTOMLEFT
#/texts/256
Old text
section_header
New text
#/body
BOTTOMLEFT
#/texts/257
New text
caption
Figure 11: Example of showing sample changes throughout each step of the processing pipeline. In the following example, users can notice that advertisement text were removed from the main article.
#/body
BOTTOMLEFT
#/texts/258
Figure 11: Example of showing sample changes throughout each step of the processing pipeline. In the following example, users can notice that advertisement text were removed from the main article.
page_footer
23
#/body
BOTTOMLEFT
#/texts/259
23
text
The purpose of this application is to sequentially view the changes made to dataset.
#/body
BOTTOMLEFT
#/texts/260
The purpose of this application is to sequentially view the changes made to dataset.
caption
(a) Pipeline v0
#/body
BOTTOMLEFT
#/texts/261
(a) Pipeline v0
text
The purpose of this application is to sequentially view the changes made to dataset.
#/body
BOTTOMLEFT
#/texts/262
The purpose of this application is to sequentially view the changes made to dataset.
caption
(b) Pipeline v2
#/body
BOTTOMLEFT
#/texts/263
(b) Pipeline v2
caption
(c) Sample example difference between pipeline versions
#/body
BOTTOMLEFT
#/texts/264
(c) Sample example difference between pipeline versions
text
Figure 12: High level statistics between two seperate pipelines and a sample example of the difference between two pipelines. First iteration (Figure 12a) generated a 7Mb dataset. After some careful tweaking, and some observed samples, we proposed a new pipeline in order to preserve previously wrongly removed data (Figure 12b) which generated a 134Mb dataset (x18). A example sample is available in Figure 12c
#/body
BOTTOMLEFT
#/texts/265
Figure 12: High level statistics between two seperate pipelines and a sample example of the difference between two pipelines. First iteration (Figure 12a) generated a 7Mb dataset. After some careful tweaking, and some observed samples, we proposed a new pipeline in order to preserve previously wrongly removed data (Figure 12b) which generated a 134Mb dataset (x18). A example sample is available in Figure 12c
page_footer
24
#/body
BOTTOMLEFT
#/texts/266
24
text
remove_html_spans Takes in a batch of texts and removes lines with the following substrings: "<span" , "</span>" , "<div" , "<a" , "</div>" , "</a>" , "br>" ,
#/body
BOTTOMLEFT
#/texts/267
remove_html_spans Takes in a batch of texts and removes lines with the following substrings: "<span" , "</span>" , "<div" , "<a" , "</div>" , "</a>" , "br>" ,
text
remove_html_spans_sanad Takes in a batch of texts and removes lines with the following substrings: "<img" , "]]>" , "<![CDATA" , "//DW" , "var " , "xtImg" , "To view this video please enable JavaScript" ,
#/body
BOTTOMLEFT
#/texts/268
remove_html_spans_sanad Takes in a batch of texts and removes lines with the following substrings: "<img" , "]]>" , "<![CDATA" , "//DW" , "var " , "xtImg" , "To view this video please enable JavaScript" ,
text
remove_wiki_mojibake Takes in a batch of texts and removes lines with the following substrings: "À À"
#/body
BOTTOMLEFT
#/texts/269
remove_wiki_mojibake Takes in a batch of texts and removes lines with the following substrings: "À À"
text
strip_substrings_en_wiktionary Takes in a batch of texts and removes the following substrings:
#/body
BOTTOMLEFT
#/texts/270
strip_substrings_en_wiktionary Takes in a batch of texts and removes the following substrings:
list_item
-
· This entry needs pronunciation information
#/groups/13
BOTTOMLEFT
#/texts/271
· This entry needs pronunciation information
list_item
-
· Please try to find a suitable image on Wikimedia Commons or upload one there yourself!This entry need pronunciation information
#/groups/13
BOTTOMLEFT
#/texts/272
· Please try to find a suitable image on Wikimedia Commons or upload one there yourself!This entry need pronunciation information
list_item
-
· You may continue to edit this entry while the discussion proceeds, but please mention significant edits at the RFD discussion and ensure that the intention of votes already cast is not left unclear
#/groups/13
BOTTOMLEFT
#/texts/273
· You may continue to edit this entry while the discussion proceeds, but please mention significant edits at the RFD discussion and ensure that the intention of votes already cast is not left unclear
list_item
-
· This entry is part of the phrasebook project, which presents criteria for inclusion based on utility, simplicity and commonality
#/groups/13
BOTTOMLEFT
#/texts/274
· This entry is part of the phrasebook project, which presents criteria for inclusion based on utility, simplicity and commonality
list_item
-
· If you are a native speaker with a microphone, please record some and upload them
#/groups/13
BOTTOMLEFT
#/texts/275
· If you are a native speaker with a microphone, please record some and upload them
list_item
-
· If you are familiar with the IPA then please add some!
#/groups/13
BOTTOMLEFT
#/texts/276
· If you are familiar with the IPA then please add some!
list_item
-
· Feel free to edit this entry as normal, but do not remove rfv until the request has been resolved
#/groups/13
BOTTOMLEFT
#/texts/277
· Feel free to edit this entry as normal, but do not remove rfv until the request has been resolved
list_item
-
· This entry needs quotations to illustrate usage
#/groups/13
BOTTOMLEFT
#/texts/278
· This entry needs quotations to illustrate usage
list_item
-
· If you are familiar with the IPA then please add some!This entry needs audio files
#/groups/13
BOTTOMLEFT
#/texts/279
· If you are familiar with the IPA then please add some!This entry needs audio files
list_item
-
· Please see that page for discussion and justifications
#/groups/13
BOTTOMLEFT
#/texts/280
· Please see that page for discussion and justifications
list_item
-
· If you are familiar with the IPA or enPR then please add some!A user has added this entry to requests for verification(+) If it cannot be verified that this term meets our attestation criteria, it will be deleted
#/groups/13
BOTTOMLEFT
#/texts/281
· If you are familiar with the IPA or enPR then please add some!A user has added this entry to requests for verification(+) If it cannot be verified that this term meets our attestation criteria, it will be deleted
list_item
-
· This entry needs a photograph or drawing for illustration
#/groups/13
BOTTOMLEFT
#/texts/282
· This entry needs a photograph or drawing for illustration
list_item
-
· A user has added this entry to requests for deletion(+)
#/groups/13
BOTTOMLEFT
#/texts/283
· A user has added this entry to requests for deletion(+)
list_item
-
· Do not remove the rfd until the debate has finished
#/groups/13
BOTTOMLEFT
#/texts/284
· Do not remove the rfd until the debate has finished
list_item
-
· This entry needs audio files
#/groups/13
BOTTOMLEFT
#/texts/285
· This entry needs audio files
list_item
-
· If you come across any interesting, durably archived quotes then please add them!This entry is part of the phrasebook project, which presents criteria for inclusion based on utility, simplicity and commonality
#/groups/13
BOTTOMLEFT
#/texts/286
· If you come across any interesting, durably archived quotes then please add them!This entry is part of the phrasebook project, which presents criteria for inclusion based on utility, simplicity and commonality
list_item
-
· (For audio required quickly, visit WT:APR)
#/groups/13
BOTTOMLEFT
#/texts/287
· (For audio required quickly, visit WT:APR)
text
remove_references_{lang} Removes lines that do not contain a minimum ratio of stopwords, as defined for each language$^{15}$. Note, currently does not support languages with different segmentation (e.g. Chinese). Designed for academic datasets.
#/body
BOTTOMLEFT
#/texts/288
remove_references_{lang} Removes lines that do not contain a minimum ratio of stopwords, as defined for each language$^{15}$. Note, currently does not support languages with different segmentation (e.g. Chinese). Designed for academic datasets.
text
split_sentences_{lang} Builds a sentence splitter depending on the language passed: For Arabic, Catalan, Basque, Indonesian, and Chinese (both simplified and traditional), we use the Stanza tokenizer (Qi et al., 2020). For English, French, Portuguese, and Spanish, we use the NLTK tokenizer (Bird et al., 2009). For Bengalic, Gujarati, Hindi, Kannada, Malayalam, Marathi, Punjabi, Tamil, and Telugu, we use the Indic NLP library tokenizer (Kunchukuttan, 2020). For Vietnamese, we use the Underthesea tokenizer $^{16}$.
#/body
BOTTOMLEFT
#/texts/289
split_sentences_{lang} Builds a sentence splitter depending on the language passed: For Arabic, Catalan, Basque, Indonesian, and Chinese (both simplified and traditional), we use the Stanza tokenizer (Qi et al., 2020). For English, French, Portuguese, and Spanish, we use the NLTK tokenizer (Bird et al., 2009). For Bengalic, Gujarati, Hindi, Kannada, Malayalam, Marathi, Punjabi, Tamil, and Telugu, we use the Indic NLP library tokenizer (Kunchukuttan, 2020). For Vietnamese, we use the Underthesea tokenizer $^{16}$.
text
filter_remove_empty_docs Removes documents that have a length of 0 when whitespace is removed.
#/body
BOTTOMLEFT
#/texts/290
filter_remove_empty_docs Removes documents that have a length of 0 when whitespace is removed.
text
filter_wiki_user_titles Removes documents where the Wikimedia metadata title starts with "user" ,
#/body
BOTTOMLEFT
#/texts/291
filter_wiki_user_titles Removes documents where the Wikimedia metadata title starts with "user" ,
text
filter_wiki_non_text_type Removes documents where the Wikimedia metadata type is not "text"
#/body
BOTTOMLEFT
#/texts/292
filter_wiki_non_text_type Removes documents where the Wikimedia metadata type is not "text"
footnote
$^{15}$https://github.com/bigscience-workshop/catalogue_data/blob/master/clean_helpers/stopwords.py
#/body
BOTTOMLEFT
#/texts/293
$^{15}$https://github.com/bigscience-workshop/catalogue_data/blob/master/clean_helpers/stopwords.py
footnote
$^{16}$https://github.com/undertheseanlp/underthesea
#/body
BOTTOMLEFT
#/texts/294
$^{16}$https://github.com/undertheseanlp/underthesea
page_footer
25
#/body
BOTTOMLEFT
#/texts/295
25
text
filter_small_docs Discards documents with less than 15 words. Tokenization is done via whitespace tokenizer.
#/body
BOTTOMLEFT
#/texts/296
filter_small_docs Discards documents with less than 15 words. Tokenization is done via whitespace tokenizer.
text
filter_small_docs_bytes_{i} Discards documents with less than either 300 or 1024 bytes of text
#/body
BOTTOMLEFT
#/texts/297
filter_small_docs_bytes_{i} Discards documents with less than either 300 or 1024 bytes of text
text
dedup_template_soft Removes lines that are a minimum of 15 characters long and occur 10 or more times.
#/body
BOTTOMLEFT
#/texts/298
dedup_template_soft Removes lines that are a minimum of 15 characters long and occur 10 or more times.
text
dedup_pseudocrawl_newspapers Removes lines that occur 2 or more times.
#/body
BOTTOMLEFT
#/texts/299
dedup_pseudocrawl_newspapers Removes lines that occur 2 or more times.
text
dedup_document Removes duplicate documents ignoring whitespaces and punctuation so only keeping characters and keeps one occurrence.
#/body
BOTTOMLEFT
#/texts/300
dedup_document Removes duplicate documents ignoring whitespaces and punctuation so only keeping characters and keeps one occurrence.
text
dedup_document_on_url Removes duplicate documents based on matched url while ignoring query parameters and keeps one occurrence.
#/body
BOTTOMLEFT
#/texts/301
dedup_document_on_url Removes duplicate documents based on matched url while ignoring query parameters and keeps one occurrence.
text
dedup_document_on_url_lm_es_pseudocrawl-filtered_341_es_cointelegraph_com Removes duplicate documents based on the normalized urls (e.g., $URL and $URL/amp are treated as the same) without the query parameters and keeps one occurrence.
#/body
BOTTOMLEFT
#/texts/302
dedup_document_on_url_lm_es_pseudocrawl-filtered_341_es_cointelegraph_com Removes duplicate documents based on the normalized urls (e.g., $URL and $URL/amp are treated as the same) without the query parameters and keeps one occurrence.
text
dedup_document_on_url_lm_en_pseudocrawl_filtered_619_www_qut_edu_au Removes duplicate documents based on the url without query parameters except for the "id" and "new-id" query parameters. The "new-id" query parameter is changed into a simple "id" parameter.
#/body
BOTTOMLEFT
#/texts/303
dedup_document_on_url_lm_en_pseudocrawl_filtered_619_www_qut_edu_au Removes duplicate documents based on the url without query parameters except for the "id" and "new-id" query parameters. The "new-id" query parameter is changed into a simple "id" parameter.
text
concatenate_lm_fr_ester Concatenate the text sorted by the id number in the metadata.
#/body
BOTTOMLEFT
#/texts/304
concatenate_lm_fr_ester Concatenate the text sorted by the id number in the metadata.
section_header
C Exhaustive list of human curated filters used on OSCAR
#/body
BOTTOMLEFT
#/texts/305
C Exhaustive list of human curated filters used on OSCAR
text
Before performing the filtering step, we did a cleaning step to modify the documents by standardizing whitespace and removing links, non-printable characters, and long words beyond a character threshold. These steps were designed to remove "non natural" language parts of the document (i.e. texts that are machine generated or not language, such as URLs).
#/body
BOTTOMLEFT
#/texts/306
Before performing the filtering step, we did a cleaning step to modify the documents by standardizing whitespace and removing links, non-printable characters, and long words beyond a character threshold. These steps were designed to remove "non natural" language parts of the document (i.e. texts that are machine generated or not language, such as URLs).
text
Many of these filters require to split a document into words. For Chinese, we used the SentencePiece unigram tokenizer. For Vietnamese, since a word can be composed of two or three sub-words separated by spaces, we augmented the list of space separated tokens by the list of two and three consecutive space separated tokens.
#/body
BOTTOMLEFT
#/texts/307
Many of these filters require to split a document into words. For Chinese, we used the SentencePiece unigram tokenizer. For Vietnamese, since a word can be composed of two or three sub-words separated by spaces, we augmented the list of space separated tokens by the list of two and three consecutive space separated tokens.
text
Filter on number of words We discarded documents with too few words, as they often contain incorrect sentences, or contain no context for a model to learn correctly.
#/body
BOTTOMLEFT
#/texts/308
Filter on number of words We discarded documents with too few words, as they often contain incorrect sentences, or contain no context for a model to learn correctly.
text
Filter on character repetition ratio To remove documents containing many repetitions, for a given n (determined in practice according to the language by native speakers), we counted the occurrence of each character n -gram present in the document. We defined the character repetition ratio as the ratio of the sum of the k largest occurrences by the sum of all occurrences, and we discarded documents with a too high ratio.
#/body
BOTTOMLEFT
#/texts/309
Filter on character repetition ratio To remove documents containing many repetitions, for a given n (determined in practice according to the language by native speakers), we counted the occurrence of each character n -gram present in the document. We defined the character repetition ratio as the ratio of the sum of the k largest occurrences by the sum of all occurrences, and we discarded documents with a too high ratio.
text
If k = 1 , short sentences are much more likely to have a high character repetition ratio, since the most frequent n -gram represents a larger proportion of the sentence. If k is the number of occurrences greater than or equal to 2 , very long documents, but not necessarily including repetitions, tend to have a high character repetition ratio, since these texts inherently have a wide diversity of n -grams. We found that k = ⌊ √ N ⌋ , with N the number of different n -grams found in the document, counterbalances well this effect in practice.
#/body
BOTTOMLEFT
#/texts/310
If k = 1 , short sentences are much more likely to have a high character repetition ratio, since the most frequent n -gram represents a larger proportion of the sentence. If k is the number of occurrences greater than or equal to 2 , very long documents, but not necessarily including repetitions, tend to have a high character repetition ratio, since these texts inherently have a wide diversity of n -grams. We found that k = ⌊ √ N ⌋ , with N the number of different n -grams found in the document, counterbalances well this effect in practice.
text
Example: Take the sentence " ok_ok_good_ok " and n = 3 . Character n -grams, with their frequencies, are given in the following table.
#/body
BOTTOMLEFT
#/texts/311
Example: Take the sentence " ok_ok_good_ok " and n = 3 . Character n -grams, with their frequencies, are given in the following table.
text
Since we have 9 different character n -grams, N = 9 and k = ⌊ √ N ⌋ = 3 .
#/body
BOTTOMLEFT
#/texts/312
Since we have 9 different character n -grams, N = 9 and k = ⌊ √ N ⌋ = 3 .
text
The sum of the k largest occurrences is 2 + 2 + 1 = 5 and the sum of all occurrences is 11 . Thus, the character repetition ratio for this sentence is 5 $_{11}$.
#/body
BOTTOMLEFT
#/texts/313
The sum of the k largest occurrences is 2 + 2 + 1 = 5 and the sum of all occurrences is 11 . Thus, the character repetition ratio for this sentence is 5 $_{11}$.
page_footer
26
#/body
BOTTOMLEFT
#/texts/314
26
text
Filter on word repetition ratio As a complement to the previous filter, we remove documents that have commonly repeated similar long sentences. More specifically, we create a filter for the repetitions by looking this time at the occurrences of the word n -grams, for a chosen n parameter. We define the word repetition ratio as the ratio of the sum of the occurrences greater than or equal to 2 to the sum of all occurrences, and we discard documents with too high of a ratio. Contrary to the filter on the character repetition ratios, we did not find a bias of this method giving systematically higher or lower scores to longer or short documents. This filter is more robust in finding documents with long exact duplicated sentences in them, while the previous one is used to find short to medium sized repetitions.
#/body
BOTTOMLEFT
#/texts/315
Filter on word repetition ratio As a complement to the previous filter, we remove documents that have commonly repeated similar long sentences. More specifically, we create a filter for the repetitions by looking this time at the occurrences of the word n -grams, for a chosen n parameter. We define the word repetition ratio as the ratio of the sum of the occurrences greater than or equal to 2 to the sum of all occurrences, and we discard documents with too high of a ratio. Contrary to the filter on the character repetition ratios, we did not find a bias of this method giving systematically higher or lower scores to longer or short documents. This filter is more robust in finding documents with long exact duplicated sentences in them, while the previous one is used to find short to medium sized repetitions.
text
Filter on special character ratio We established a list of special characters, including emojis, and simply discard documents with a special character ratio above a certain threshold.
#/body
BOTTOMLEFT
#/texts/316
Filter on special character ratio We established a list of special characters, including emojis, and simply discard documents with a special character ratio above a certain threshold.
text
Filter on closed class word ratio We found that having a low closed class word ratio in a document was one of the best indicators of a non-human generated content. We built lists of closed class words for each language by taking pre-existing lists, for example from Universal Dependencies$^{17}$, which were then reviewed by native speakers. We discard documents with a too low closed class word ratio.
#/body
BOTTOMLEFT
#/texts/317
Filter on closed class word ratio We found that having a low closed class word ratio in a document was one of the best indicators of a non-human generated content. We built lists of closed class words for each language by taking pre-existing lists, for example from Universal Dependencies$^{17}$, which were then reviewed by native speakers. We discard documents with a too low closed class word ratio.
text
Filter on flagged word ratio To limit the over-representation of pornographic documents, which are in practice much more likely to have shocking and sexist content, and to contain only buzzwords for SEO, we built lists of flagged words for each language by gathering existing lists, and filtering them by native speakers with precise instructions. We are then able to compute the flagged word ratio of a document and discard it if it is too high. About 1% of the documents for each language are removed by this filter.
#/body
BOTTOMLEFT
#/texts/318
Filter on flagged word ratio To limit the over-representation of pornographic documents, which are in practice much more likely to have shocking and sexist content, and to contain only buzzwords for SEO, we built lists of flagged words for each language by gathering existing lists, and filtering them by native speakers with precise instructions. We are then able to compute the flagged word ratio of a document and discard it if it is too high. About 1% of the documents for each language are removed by this filter.
text
Instructions for building the lists of flagged words: Keep only the words associated with porn and systematically used in a sexual context. Remove words that can be used in medical, scientific, colloquial (without referring systematically to porn), or everyday contexts. Remove all insults. Remove all words referring to race or sexual orientation.
#/body
BOTTOMLEFT
#/texts/319
Instructions for building the lists of flagged words: Keep only the words associated with porn and systematically used in a sexual context. Remove words that can be used in medical, scientific, colloquial (without referring systematically to porn), or everyday contexts. Remove all insults. Remove all words referring to race or sexual orientation.
text
Filter on language identification prediction score We used fastText (Joulin et al., 2017) to perform language identification and getting confidence scores for each document. If a score is below a specific threshold, we discard the document. We chose to eliminate few documents with this filter, because the language identification does not perform as well on low-resource languages.
#/body
BOTTOMLEFT
#/texts/320
Filter on language identification prediction score We used fastText (Joulin et al., 2017) to perform language identification and getting confidence scores for each document. If a score is below a specific threshold, we discard the document. We chose to eliminate few documents with this filter, because the language identification does not perform as well on low-resource languages.
text
Filter on perplexity score Following Wenzek et al. (2020), we trained SentencePiece unigram tokenizers (Kudo, 2018) followed by KenLM 5-gram models after tokenization (Heafield, 2011) on Wikipedia article openings for every language that was extracted from OSCAR. As in De la Rosa et al. (2022), we discarded documents to move the perplexity distribution towards the median, to avoid too high perplexity scores (deemed as not useful for the model), but subsampling was done by perplexity thresholding, not by reshaping the distribution as in De la Rosa et al. (2022). This thresholding was done lightly, by having native speakers manually establish the cutoff values per language$^{18}$, so as not to be too biased by the Wikipedia content and keep the dataset diverse.
#/body
BOTTOMLEFT
#/texts/321
Filter on perplexity score Following Wenzek et al. (2020), we trained SentencePiece unigram tokenizers (Kudo, 2018) followed by KenLM 5-gram models after tokenization (Heafield, 2011) on Wikipedia article openings for every language that was extracted from OSCAR. As in De la Rosa et al. (2022), we discarded documents to move the perplexity distribution towards the median, to avoid too high perplexity scores (deemed as not useful for the model), but subsampling was done by perplexity thresholding, not by reshaping the distribution as in De la Rosa et al. (2022). This thresholding was done lightly, by having native speakers manually establish the cutoff values per language$^{18}$, so as not to be too biased by the Wikipedia content and keep the dataset diverse.
section_header
D PII filtering initiative
#/body
BOTTOMLEFT
#/texts/322
D PII filtering initiative
text
Even if not eventually used in our final pipeline, we have released muliwai 19 a library for text preprocessing, augmentation, anonymization, and synthesis. It relies on transformer models and backtranslation to perform NER and associated augmentation and anonymization over 100+ languages (i.e., we rely on XLMRoberta Fan et al. (2021) and M2M100 Conneau et al. (2020)). We either use a specific model for the chosen language or a model with cross-lingual capabilities. Muliwai tags using the aforementioned transformer then translate the sentence to a target language (e.g., English) and test to see if the translation preserves the NER tagging and discounts or increases the weight of a NER decision accordingly. It then performs NER in the target language and back translates to
#/body
BOTTOMLEFT
#/texts/323
Even if not eventually used in our final pipeline, we have released muliwai 19 a library for text preprocessing, augmentation, anonymization, and synthesis. It relies on transformer models and backtranslation to perform NER and associated augmentation and anonymization over 100+ languages (i.e., we rely on XLMRoberta Fan et al. (2021) and M2M100 Conneau et al. (2020)). We either use a specific model for the chosen language or a model with cross-lingual capabilities. Muliwai tags using the aforementioned transformer then translate the sentence to a target language (e.g., English) and test to see if the translation preserves the NER tagging and discounts or increases the weight of a NER decision accordingly. It then performs NER in the target language and back translates to
text
$^{17}$https://universaldependencies.org/ $^{18}$Native speakers used an ad-hoc visualization tool built for the https://huggingface.co/spaces/huggingface/text-data-filtering $^{19}$Pronounced "mu-lee-why" , Hawaiian for river. https://github.com/ontocord/muliwai/tree/main
#/body
BOTTOMLEFT
#/texts/324
$^{17}$https://universaldependencies.org/ $^{18}$Native speakers used an ad-hoc visualization tool built for the https://huggingface.co/spaces/huggingface/text-data-filtering $^{19}$Pronounced "mu-lee-why" , Hawaiian for river. https://github.com/ontocord/muliwai/tree/main
page_footer
27
#/body
BOTTOMLEFT
#/texts/325
27
text
occasion:
#/body
BOTTOMLEFT
#/texts/326
occasion:
text
the source language. Finally it matches the translated sentence to the original sentence to determine which text spans in the source language sentence should be NER tagged based on the target language NER. We also use spacy and regex as added signals for NER tags.
#/body
BOTTOMLEFT
#/texts/327
the source language. Finally it matches the translated sentence to the original sentence to determine which text spans in the source language sentence should be NER tagged based on the target language NER. We also use spacy and regex as added signals for NER tags.
text
We also include in the library specific regexes for detecting age, email, date, time, personal addresses, phone numbers and government-issued identifiers (such as license plates). Some regex matches use also the surrounding text context to improve precision.
#/body
BOTTOMLEFT
#/texts/328
We also include in the library specific regexes for detecting age, email, date, time, personal addresses, phone numbers and government-issued identifiers (such as license plates). Some regex matches use also the surrounding text context to improve precision.
text
However, the scale of the data, the fact that the impact on the resulting text could not be fully assessed in terms of language modeling and the time constraint due to compute allocation, meant this approach could not be operationalized on ROOTS. Instead we fell back to a simpler approach, see Section 3.3.
#/body
BOTTOMLEFT
#/texts/329
However, the scale of the data, the fact that the impact on the resulting text could not be fully assessed in terms of language modeling and the time constraint due to compute allocation, meant this approach could not be operationalized on ROOTS. Instead we fell back to a simpler approach, see Section 3.3.
section_header
E Data Sources
#/body
BOTTOMLEFT
#/texts/330
E Data Sources
page_footer
28
#/body
BOTTOMLEFT
#/texts/331
28
page_footer
29
#/body
BOTTOMLEFT
#/texts/332
29
page_footer
30
#/body
BOTTOMLEFT
#/texts/333
30
text
Dataset
#/body
BOTTOMLEFT
#/texts/334
Dataset
text
Language
#/body
BOTTOMLEFT
#/texts/335
Language
text
Source
#/body
BOTTOMLEFT
#/texts/336
Source
paragraph
Table 2: List of datasets used in crowdsourced dataset.
#/body
BOTTOMLEFT
#/texts/337
Table 2: List of datasets used in crowdsourced dataset.
page_footer
31
#/body
BOTTOMLEFT
#/texts/338
31
caption
Table 3: Linguistic makeup of the corpus.
#/body
BOTTOMLEFT
#/texts/339
Table 3: Linguistic makeup of the corpus.
page_footer
32
#/body
BOTTOMLEFT
#/texts/340
32
caption
Table 4: Pseudocrawled data per language sorted by number of domains crawled
#/body
BOTTOMLEFT
#/texts/341
Table 4: Pseudocrawled data per language sorted by number of domains crawled
page_footer
33
#/body
BOTTOMLEFT
#/texts/342
33
section_header
F Author contributions
#/body
BOTTOMLEFT
#/texts/343
F Author contributions
text
Author contributions in alphabetical order.
#/body
BOTTOMLEFT
#/texts/344
Author contributions in alphabetical order.
text
Aaron Gokaslan set a pre-commit (for code formatting) in a repository and helped with the writing of the Related Work section of the paper.
#/body
BOTTOMLEFT
#/texts/345
Aaron Gokaslan set a pre-commit (for code formatting) in a repository and helped with the writing of the Related Work section of the paper.
text
Aitor Soroa integrated one dataset into crowdsourced data.
#/body
BOTTOMLEFT
#/texts/346
Aitor Soroa integrated one dataset into crowdsourced data.
text
Albert Villanova del Moral led the gathering of identified sources, implemented loading scripts in the datasets library in a single unified interface, and integrated the most datasets into crowdsourced data.
#/body
BOTTOMLEFT
#/texts/347
Albert Villanova del Moral led the gathering of identified sources, implemented loading scripts in the datasets library in a single unified interface, and integrated the most datasets into crowdsourced data.
text
Angelina McMillan-Major gathered the lists of closed class words for many languages used for the filtering of OSCAR.
#/body
BOTTOMLEFT
#/texts/348
Angelina McMillan-Major gathered the lists of closed class words for many languages used for the filtering of OSCAR.
text
Anna Rogers contributed to the writing of the paper.
#/body
BOTTOMLEFT
#/texts/349
Anna Rogers contributed to the writing of the paper.
text
Chenghao Mou was the main contributor for OSCAR deduplication.
#/body
BOTTOMLEFT
#/texts/350
Chenghao Mou was the main contributor for OSCAR deduplication.
text
Christopher Akiki advised on analysis aspects of the project, integrated over a hundred datasets into crowdsourced data, wrote dataset loading scripts, participated in cleaning and filtering efforts, helped with visualization, and contributed to the writing of the paper.
#/body
BOTTOMLEFT
#/texts/351
Christopher Akiki advised on analysis aspects of the project, integrated over a hundred datasets into crowdsourced data, wrote dataset loading scripts, participated in cleaning and filtering efforts, helped with visualization, and contributed to the writing of the paper.
text
Daniel van Strien integrated one dataset into crowdsourced data.
#/body
BOTTOMLEFT
#/texts/352
Daniel van Strien integrated one dataset into crowdsourced data.
text
David Ifeoluwa Adelani participated in the PII filtering initiative (see Appendix D).
#/body
BOTTOMLEFT
#/texts/353
David Ifeoluwa Adelani participated in the PII filtering initiative (see Appendix D).
text
Eduardo González Ponferrada trained SentencePiece and KenLM models used for the filtering of OSCAR.
#/body
BOTTOMLEFT
#/texts/354
Eduardo González Ponferrada trained SentencePiece and KenLM models used for the filtering of OSCAR.
text
Francesco De Toni led one crowdsourcing hackathon, analyzed the distribution of the sources in the catalogue, participated in the PII filtering initiative (see Appendix D), and contributed to the writing of the paper.
#/body
BOTTOMLEFT
#/texts/355
Francesco De Toni led one crowdsourcing hackathon, analyzed the distribution of the sources in the catalogue, participated in the PII filtering initiative (see Appendix D), and contributed to the writing of the paper.
text
Giada Pistilli helped write the Ethical Considerations section of the paper.
#/body
BOTTOMLEFT
#/texts/356
Giada Pistilli helped write the Ethical Considerations section of the paper.
text
Gérard M Dupont contributed to data tooling and sourcing and advised on analysis aspects of the project.
#/body
BOTTOMLEFT
#/texts/357
Gérard M Dupont contributed to data tooling and sourcing and advised on analysis aspects of the project.
text
Hieu Tran helped set the filtering parameters for Vietnamese and contributed to the list of Vietnamese closed-class words.
#/body
BOTTOMLEFT
#/texts/358
Hieu Tran helped set the filtering parameters for Vietnamese and contributed to the list of Vietnamese closed-class words.
text
Hugo Laurençon developed the filtering library used for the cleaning of OSCAR and the visualization tool to help choose the filtering parameters, and ran OSCAR filtering jobs. He was involved in the cleaning of some crowdsourced and pseudo-crawled datasets and the deduplication of OSCAR. He also contributed to the writing of the paper.
#/body
BOTTOMLEFT
#/texts/359
Hugo Laurençon developed the filtering library used for the cleaning of OSCAR and the visualization tool to help choose the filtering parameters, and ran OSCAR filtering jobs. He was involved in the cleaning of some crowdsourced and pseudo-crawled datasets and the deduplication of OSCAR. He also contributed to the writing of the paper.
text
Huu Nguyen contributed to the data tooling and lead the PII filtering initiative (see Appendix D).
#/body
BOTTOMLEFT
#/texts/360
Huu Nguyen contributed to the data tooling and lead the PII filtering initiative (see Appendix D).
text
Ian Yu participated in the PII filtering initiative (see Appendix D) and helped to choose the filtering parameters for Chinese.
#/body
BOTTOMLEFT
#/texts/361
Ian Yu participated in the PII filtering initiative (see Appendix D) and helped to choose the filtering parameters for Chinese.
text
Itziar Gonzalez-Dios , as a Basque native speaker, helped choose the filtering parameters for this language.
#/body
BOTTOMLEFT
#/texts/362
Itziar Gonzalez-Dios , as a Basque native speaker, helped choose the filtering parameters for this language.
text
Javier De la Rosa contributed with perplexity sampling efforts for OSCAR (not used in final pipeline).
#/body
BOTTOMLEFT
#/texts/363
Javier De la Rosa contributed with perplexity sampling efforts for OSCAR (not used in final pipeline).
text
Jenny Chim participated in the PII filtering initiative (see Appendix D) and helped to choose the filtering parameters and closed-class words for Chinese.
#/body
BOTTOMLEFT
#/texts/364
Jenny Chim participated in the PII filtering initiative (see Appendix D) and helped to choose the filtering parameters and closed-class words for Chinese.
text
Jian Zhu integrated two datasets into crowdsourced data.
#/body
BOTTOMLEFT
#/texts/365
Jian Zhu integrated two datasets into crowdsourced data.
text
Jörg Frohberg integrated multiple datasets into crowdsourced data and reached out to license holders.
#/body
BOTTOMLEFT
#/texts/366
Jörg Frohberg integrated multiple datasets into crowdsourced data and reached out to license holders.
text
Khalid Almubarak integrated some datasets into crowdsourced data.
#/body
BOTTOMLEFT
#/texts/367
Khalid Almubarak integrated some datasets into crowdsourced data.
text
Kyle Lo integrated one dataset into crowdsourced data.
#/body
BOTTOMLEFT
#/texts/368
Kyle Lo integrated one dataset into crowdsourced data.
page_footer
34
#/body
BOTTOMLEFT
#/texts/369
34
text
Leandro von Werra participated in cleaning and filtering efforts, built the code dataset, contributed to its analysis, and participated in the sourcing effort for target datasets.
#/body
BOTTOMLEFT
#/texts/370
Leandro von Werra participated in cleaning and filtering efforts, built the code dataset, contributed to its analysis, and participated in the sourcing effort for target datasets.
text
Leon Weber integrated one dataset into crowdsourced data.
#/body
BOTTOMLEFT
#/texts/371
Leon Weber integrated one dataset into crowdsourced data.
text
Long Phan participated in the PII filtering initiative (see Appendix D).
#/body
BOTTOMLEFT
#/texts/372
Long Phan participated in the PII filtering initiative (see Appendix D).
text
Loubna Ben allal contributed the analysis of the code dataset.
#/body
BOTTOMLEFT
#/texts/373
Loubna Ben allal contributed the analysis of the code dataset.
text
Lucile Saulnier co-led pseudo-crawled data acquisition, contributed to filtering, cleaning, and deduplication for the crowdsourced datasets, built visualization tools to inspect the results of preprocessing, scaled the PII filtering process, performed the document size analysis, and participated in paper writing.
#/body
BOTTOMLEFT
#/texts/374
Lucile Saulnier co-led pseudo-crawled data acquisition, contributed to filtering, cleaning, and deduplication for the crowdsourced datasets, built visualization tools to inspect the results of preprocessing, scaled the PII filtering process, performed the document size analysis, and participated in paper writing.
text
Manan Dey , as a Bengali native speaker, helped to choose the filtering parameters for this language.
#/body
BOTTOMLEFT
#/texts/375
Manan Dey , as a Bengali native speaker, helped to choose the filtering parameters for this language.
text
Manuel Romero Muñoz contributed to KenLM models and to corpus visualization with those models, and participated in the PII filtering initiative (see Appendix D).
#/body
BOTTOMLEFT
#/texts/376
Manuel Romero Muñoz contributed to KenLM models and to corpus visualization with those models, and participated in the PII filtering initiative (see Appendix D).
text
Maraim Masoud contributed to the sourcing of some Arabic datasets, the list of Arabic closed-class words, and the writing of the paper.
#/body
BOTTOMLEFT
#/texts/377
Maraim Masoud contributed to the sourcing of some Arabic datasets, the list of Arabic closed-class words, and the writing of the paper.
text
Margaret Mitchell co-led the final regex-based PII efforts.
#/body
BOTTOMLEFT
#/texts/378
Margaret Mitchell co-led the final regex-based PII efforts.
text
Mario Šaško integrated multiple datasets into crowdsourced data.
#/body
BOTTOMLEFT
#/texts/379
Mario Šaško integrated multiple datasets into crowdsourced data.
text
Olivier Nguyen helped build the first blocks of the OSCAR filtering pipeline.
#/body
BOTTOMLEFT
#/texts/380
Olivier Nguyen helped build the first blocks of the OSCAR filtering pipeline.
text
Paulo Villegas participated in the PII filtering initiative (see Appendix D) and the perplexity sampling efforts for OSCAR.
#/body
BOTTOMLEFT
#/texts/381
Paulo Villegas participated in the PII filtering initiative (see Appendix D) and the perplexity sampling efforts for OSCAR.
text
Pedro Ortiz Suarez contributed to crowdsourced data and provided high-level metrics on OSCAR.
#/body
BOTTOMLEFT
#/texts/382
Pedro Ortiz Suarez contributed to crowdsourced data and provided high-level metrics on OSCAR.
text
Pierre Colombo participated in the PII filtering initiative (see Appendix D).
#/body
BOTTOMLEFT
#/texts/383
Pierre Colombo participated in the PII filtering initiative (see Appendix D).
text
Quentin Lhoest integrated multiple datasets into crowdsourced data.
#/body
BOTTOMLEFT
#/texts/384
Quentin Lhoest integrated multiple datasets into crowdsourced data.
text
Sasha Luccioni co-led the final regex-based PII efforts. participated in filtering and cleaning efforts, and contributed to the writing of the paper.
#/body
BOTTOMLEFT
#/texts/385
Sasha Luccioni co-led the final regex-based PII efforts. participated in filtering and cleaning efforts, and contributed to the writing of the paper.
text
Sebastian Nagel helped to implement the pseudo-crawled data acquisition step.
#/body
BOTTOMLEFT
#/texts/386
Sebastian Nagel helped to implement the pseudo-crawled data acquisition step.
text
Shamik Bose participated in the PII filtering initiative (see Appendix D) and contributed the list of Bengali closed-class words.
#/body
BOTTOMLEFT
#/texts/387
Shamik Bose participated in the PII filtering initiative (see Appendix D) and contributed the list of Bengali closed-class words.
text
Shayne Longpre contributed to the writing of the paper.
#/body
BOTTOMLEFT
#/texts/388
Shayne Longpre contributed to the writing of the paper.
text
Somaieh Nikpoor co-led the development of ethical/legal charter and contributed to the section on ethical considerations.
#/body
BOTTOMLEFT
#/texts/389
Somaieh Nikpoor co-led the development of ethical/legal charter and contributed to the section on ethical considerations.
text
Stella Biderman contributed to the writing of the paper.
#/body
BOTTOMLEFT
#/texts/390
Stella Biderman contributed to the writing of the paper.
text
Suhas Pai participated in the PII filtering initiative (see Appendix D).
#/body
BOTTOMLEFT
#/texts/391
Suhas Pai participated in the PII filtering initiative (see Appendix D).
text
Suzana Ili'c coordinated the organization of BigScience working groups.
#/body
BOTTOMLEFT
#/texts/392
Suzana Ili'c coordinated the organization of BigScience working groups.
text
Teven Le Scao led final quality control checks, contributed to filtering, cleaning, and deduplication for all components of the corpus, contributed to the OSCAR filtering visualization tool, contributed and repaired several datasets for crowdsourced data, performed the tokenizer-based analysis, and participated in the writing of the paper.
#/body
BOTTOMLEFT
#/texts/393
Teven Le Scao led final quality control checks, contributed to filtering, cleaning, and deduplication for all components of the corpus, contributed to the OSCAR filtering visualization tool, contributed and repaired several datasets for crowdsourced data, performed the tokenizer-based analysis, and participated in the writing of the paper.
text
Thomas Wang co-led pseudo-crawled data acquisition, built the distributed cleaning pipelines for pseudo-crawled and crowdsourced datasets, handled job monitoring for crowdsourced dataset filtering, and participated in paper writing.
#/body
BOTTOMLEFT
#/texts/394
Thomas Wang co-led pseudo-crawled data acquisition, built the distributed cleaning pipelines for pseudo-crawled and crowdsourced datasets, handled job monitoring for crowdsourced dataset filtering, and participated in paper writing.
text
Tristan Thrush participated in the final regex-based PII efforts.
#/body
BOTTOMLEFT
#/texts/395
Tristan Thrush participated in the final regex-based PII efforts.
text
Violette Lepercq was the primary project manager for the final dataset cleaning efforts and helped to reach out to the native speakers to tune the filtering parameters.
#/body
BOTTOMLEFT
#/texts/396
Violette Lepercq was the primary project manager for the final dataset cleaning efforts and helped to reach out to the native speakers to tune the filtering parameters.
text
Vu Minh Chien participated in the PII filtering initiative (see Appendix D).
#/body
BOTTOMLEFT
#/texts/397
Vu Minh Chien participated in the PII filtering initiative (see Appendix D).
page_footer
35
#/body
BOTTOMLEFT
#/texts/398
35
text
Yacine Jernite defined the primary goals of the project, advised on data collection and filtering efforts, contributed sourcing tools and early cleaning scripts, and contributed to the writing of the paper.
#/body
BOTTOMLEFT
#/texts/399
Yacine Jernite defined the primary goals of the project, advised on data collection and filtering efforts, contributed sourcing tools and early cleaning scripts, and contributed to the writing of the paper.
text
Zaid Alyafeai helped find a list of Arabic datasets that should be integrated into crowdsourced data.
#/body
BOTTOMLEFT
#/texts/400
Zaid Alyafeai helped find a list of Arabic datasets that should be integrated into crowdsourced data.
page_footer
36
#/body
BOTTOMLEFT
#/texts/401
36
BOTTOMLEFT
1 Introduction
BOTTOMLEFT
1 Introduction
BOTTOMLEFT
3
BOTTOMLEFT
1.1
BOTTOMLEFT
Outline of the Paper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
BOTTOMLEFT
3
BOTTOMLEFT
1.2
BOTTOMLEFT
Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
BOTTOMLEFT
4
BOTTOMLEFT
2
BOTTOMLEFT
(Crowd) Sourcing a Language Resource Catalogue 4
BOTTOMLEFT
(Crowd) Sourcing a Language Resource Catalogue 4
BOTTOMLEFT
2.1
BOTTOMLEFT
Obtaining Data from the Identified Resources . . . . . . . . . . . . . . . . . . . .
BOTTOMLEFT
5
BOTTOMLEFT
2.2
BOTTOMLEFT
Processing Pipeline for Quality Improvement on Crowdsourced Datasets . . . . . .
BOTTOMLEFT
6
BOTTOMLEFT
3 Processing OSCAR 7
BOTTOMLEFT
3 Processing OSCAR 7
BOTTOMLEFT
3 Processing OSCAR 7
BOTTOMLEFT
3.1
BOTTOMLEFT
Data cleaning and filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
BOTTOMLEFT
7
BOTTOMLEFT
3.2
BOTTOMLEFT
Deduplication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
BOTTOMLEFT
8
BOTTOMLEFT
3.3
BOTTOMLEFT
Personally identifiable information . . . . . . . . . . . . . . . . . . . . . . . . . .
BOTTOMLEFT
8
BOTTOMLEFT
4 A First look at ROOTS 8
BOTTOMLEFT
4 A First look at ROOTS 8
BOTTOMLEFT
4 A First look at ROOTS 8
BOTTOMLEFT
4.1
BOTTOMLEFT
Natural Languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
BOTTOMLEFT
8
BOTTOMLEFT
4.2
BOTTOMLEFT
Programming Languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
BOTTOMLEFT
9
BOTTOMLEFT
4.3
BOTTOMLEFT
Tokenizer analysis of the component datasets . . . . . . . . . . . . . . . . . . . .
BOTTOMLEFT
10
BOTTOMLEFT
5 Conclusion
BOTTOMLEFT
5 Conclusion
BOTTOMLEFT
A
BOTTOMLEFT
Ethical Considerations and Broader Impacts Statement
BOTTOMLEFT
20
BOTTOMLEFT
B.1
BOTTOMLEFT
Pseudocode to recreate the text structure from the HTML code . . . . . . . . . . .
BOTTOMLEFT
21
BOTTOMLEFT
B.2
BOTTOMLEFT
Visualization tool use cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
BOTTOMLEFT
21
BOTTOMLEFT
B.3
BOTTOMLEFT
Exhaustive list of functions used in (Crowd)Sourced dataset . . . . . . . . . . . .
BOTTOMLEFT
22
BOTTOMLEFT
26
BOTTOMLEFT
C
BOTTOMLEFT
Exhaustive list of human curated filters used on OSCAR
BOTTOMLEFT
Exhaustive list of human curated filters used on OSCAR
BOTTOMLEFT
D
BOTTOMLEFT
PII filtering initiative
BOTTOMLEFT
27
BOTTOMLEFT
E F Author contributions
BOTTOMLEFT
Data Sources
BOTTOMLEFT
28 34
BOTTOMLEFT
1 Introduction
BOTTOMLEFT
3
BOTTOMLEFT
1.1
BOTTOMLEFT
Outline of the Paper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
BOTTOMLEFT
3
BOTTOMLEFT
1.2
BOTTOMLEFT
Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
BOTTOMLEFT
4
BOTTOMLEFT
2
BOTTOMLEFT
(Crowd) Sourcing a Language Resource Catalogue 4
BOTTOMLEFT
2.1
BOTTOMLEFT
Obtaining Data from the Identified Resources . . . . . . . . . . . . . . . . . . . .
BOTTOMLEFT
5
BOTTOMLEFT
2.2
BOTTOMLEFT
Processing Pipeline for Quality Improvement on Crowdsourced Datasets . . . . . .
BOTTOMLEFT
6
BOTTOMLEFT
3 Processing OSCAR 7
BOTTOMLEFT
3.1
BOTTOMLEFT
Data cleaning and filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
BOTTOMLEFT
7
BOTTOMLEFT
3.2
BOTTOMLEFT
Deduplication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
BOTTOMLEFT
8
BOTTOMLEFT
3.3
BOTTOMLEFT
Personally identifiable information . . . . . . . . . . . . . . . . . . . . . . . . . .
BOTTOMLEFT
8
BOTTOMLEFT
4 A First look at ROOTS 8
BOTTOMLEFT
4.1
BOTTOMLEFT
Natural Languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
BOTTOMLEFT
8
BOTTOMLEFT
4.2
BOTTOMLEFT
Programming Languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
BOTTOMLEFT
9
BOTTOMLEFT
4.3
BOTTOMLEFT
Tokenizer analysis of the component datasets . . . . . . . . . . . . . . . . . . . .
BOTTOMLEFT
10
BOTTOMLEFT
5 Conclusion
BOTTOMLEFT
A
BOTTOMLEFT
Ethical Considerations and Broader Impacts Statement
BOTTOMLEFT
20
BOTTOMLEFT
B.1
BOTTOMLEFT
Pseudocode to recreate the text structure from the HTML code . . . . . . . . . . .
BOTTOMLEFT
21
BOTTOMLEFT
B.2
BOTTOMLEFT
Visualization tool use cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
BOTTOMLEFT
21
BOTTOMLEFT
B.3
BOTTOMLEFT
Exhaustive list of functions used in (Crowd)Sourced dataset . . . . . . . . . . . .
BOTTOMLEFT
22
BOTTOMLEFT
26
BOTTOMLEFT
C
BOTTOMLEFT
Exhaustive list of human curated filters used on OSCAR
BOTTOMLEFT
D
BOTTOMLEFT
PII filtering initiative
BOTTOMLEFT
27
BOTTOMLEFT
E F Author contributions
BOTTOMLEFT
Data Sources
BOTTOMLEFT
28 34
table
#/body
BOTTOMLEFT
#/tables/0
#/texts/72
BOTTOMLEFT
AR
BOTTOMLEFT
EU
BOTTOMLEFT
BN
BOTTOMLEFT
CA
BOTTOMLEFT
ZH
BOTTOMLEFT
EN
BOTTOMLEFT
FR
BOTTOMLEFT
HI
BOTTOMLEFT
ID
BOTTOMLEFT
PT
BOTTOMLEFT
UR
BOTTOMLEFT
VI
BOTTOMLEFT
ES
BOTTOMLEFT
20.3
BOTTOMLEFT
5.2
BOTTOMLEFT
48.8
BOTTOMLEFT
21.1
BOTTOMLEFT
23.1
BOTTOMLEFT
17.2
BOTTOMLEFT
17.0
BOTTOMLEFT
25.7
BOTTOMLEFT
10.4
BOTTOMLEFT
12.6
BOTTOMLEFT
15.8
BOTTOMLEFT
21.3
BOTTOMLEFT
16.9
BOTTOMLEFT
AR
BOTTOMLEFT
EU
BOTTOMLEFT
BN
BOTTOMLEFT
CA
BOTTOMLEFT
ZH
BOTTOMLEFT
EN
BOTTOMLEFT
FR
BOTTOMLEFT
HI
BOTTOMLEFT
ID
BOTTOMLEFT
PT
BOTTOMLEFT
UR
BOTTOMLEFT
VI
BOTTOMLEFT
ES
BOTTOMLEFT
20.3
BOTTOMLEFT
5.2
BOTTOMLEFT
48.8
BOTTOMLEFT
21.1
BOTTOMLEFT
23.1
BOTTOMLEFT
17.2
BOTTOMLEFT
17.0
BOTTOMLEFT
25.7
BOTTOMLEFT
10.4
BOTTOMLEFT
12.6
BOTTOMLEFT
15.8
BOTTOMLEFT
21.3
BOTTOMLEFT
16.9
table
#/body
BOTTOMLEFT
#/tables/1
BOTTOMLEFT
Cettolo, M., C. Girardi, and M. Federico (2012, May 28-30). WIT3: Web inventory of transcribed and translated talks. In Proceedings of the 16th Annual conference of the European Association for Machine Translation , Trento, Italy, pp. 261-268. European Association for Machine Translation.
BOTTOMLEFT
Charikar, M. S. (2002). Similarity estimation techniques from rounding algorithms. In Proceedings of the Thiry-Fourth Annual ACM Symposium on Theory of Computing , STOC '02, New York, NY, USA, pp. 380-388. Association for Computing Machinery.
BOTTOMLEFT
Chen, Y. and A. Eisele (2012, May). MultiUN v2: UN documents with multilingual alignments. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12) , Istanbul, Turkey, pp. 2500-2504. European Language Resources Association (ELRA).
BOTTOMLEFT
Clement, C. B., M. Bierbaum, K. P. O'Keeffe, and A. A. Alemi (2019). On the use of arxiv as a dataset. CoRR abs/1905.00075 .
BOTTOMLEFT
Conneau, A., K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov (2020, July). Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , Online, pp. 8440-8451. Association for Computational Linguistics.
BOTTOMLEFT
Contractor, D., D. McDuff, J. Haines, J. Lee, C. Hines, and B. Hecht (2020). Behavioral use licensing
BOTTOMLEFT
David, D. (2020, December). Swahili: News classification dataset.
BOTTOMLEFT
De la Rosa, J., E. G. Ponferrada, M. Romero, P. Villegas, P. González de Prado Salas, and M. Grandury (2022). BERTIN: Efficient pre-training of a Spanish language model using perplexity sampling.
BOTTOMLEFT
Devlin, J., M. Chang, K. Lee, and K. Toutanova (2018). BERT: pre-training of deep bidirectional transformers for language understanding. CoRR abs/1810.04805 .
BOTTOMLEFT
Dodge, J., M. Sap, A. Marasovi'c, W. Agnew, G. Ilharco, D. Groeneveld, M. Mitchell, and M. Gardner (2021). Documenting large webtext corpora: A case study on the colossal clean crawled corpus. pp. 1286-1305.
BOTTOMLEFT
Einea, O., A. Elnagar, and R. Al Debsi (2019). Sanad: Single-label arabic news articles dataset for automatic text categorization. Data in Brief 25 , 104076.
BOTTOMLEFT
El-Haj, M. (2020, May). Habibi - a multi dialect multi national Arabic song lyrics corpus. In Proceedings of the 12th Language Resources and Evaluation Conference , Marseille, France, pp. 1318-1326. European Language Resources Association.
BOTTOMLEFT
arXiv preprint arXiv:1611.04033 .
BOTTOMLEFT
El-Khair, I. A. (2016). 1.5 billion words arabic corpus.
BOTTOMLEFT
Fan, A., S. Bhosale, H. Schwenk, Z. Ma, A. El-Kishky, S. Goyal, M. Baines, O. Celebi, G. Wenzek, V. Chaudhary, et al. (2021). Beyond english-centric multilingual machine translation. J. Mach.
BOTTOMLEFT
Learn. Res. 22 (107), 1-48.
BOTTOMLEFT
Galliano, S., E. Geoffrois, G. Gravier, J.-F. Bonastre, D. Mostefa, and K. Choukri (2006, May). Corpus description of the ESTER evaluation campaign for the rich transcription of French broadcast news. In Proceedings of the Fifth International Conference on Language Resources and Evaluation
BOTTOMLEFT
Gao, L., S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy (2020). The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 .
BOTTOMLEFT
Cettolo, M., C. Girardi, and M. Federico (2012, May 28-30). WIT3: Web inventory of transcribed and translated talks. In Proceedings of the 16th Annual conference of the European Association for Machine Translation , Trento, Italy, pp. 261-268. European Association for Machine Translation.
BOTTOMLEFT
Charikar, M. S. (2002). Similarity estimation techniques from rounding algorithms. In Proceedings of the Thiry-Fourth Annual ACM Symposium on Theory of Computing , STOC '02, New York, NY, USA, pp. 380-388. Association for Computing Machinery.
BOTTOMLEFT
Chen, Y. and A. Eisele (2012, May). MultiUN v2: UN documents with multilingual alignments. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12) , Istanbul, Turkey, pp. 2500-2504. European Language Resources Association (ELRA).
BOTTOMLEFT
Clement, C. B., M. Bierbaum, K. P. O'Keeffe, and A. A. Alemi (2019). On the use of arxiv as a dataset. CoRR abs/1905.00075 .
BOTTOMLEFT
Conneau, A., K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov (2020, July). Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , Online, pp. 8440-8451. Association for Computational Linguistics.
BOTTOMLEFT
Contractor, D., D. McDuff, J. Haines, J. Lee, C. Hines, and B. Hecht (2020). Behavioral use licensing
BOTTOMLEFT
David, D. (2020, December). Swahili: News classification dataset.
BOTTOMLEFT
De la Rosa, J., E. G. Ponferrada, M. Romero, P. Villegas, P. González de Prado Salas, and M. Grandury (2022). BERTIN: Efficient pre-training of a Spanish language model using perplexity sampling.
BOTTOMLEFT
Devlin, J., M. Chang, K. Lee, and K. Toutanova (2018). BERT: pre-training of deep bidirectional transformers for language understanding. CoRR abs/1810.04805 .
BOTTOMLEFT
Dodge, J., M. Sap, A. Marasovi'c, W. Agnew, G. Ilharco, D. Groeneveld, M. Mitchell, and M. Gardner (2021). Documenting large webtext corpora: A case study on the colossal clean crawled corpus. pp. 1286-1305.
BOTTOMLEFT
Einea, O., A. Elnagar, and R. Al Debsi (2019). Sanad: Single-label arabic news articles dataset for automatic text categorization. Data in Brief 25 , 104076.
BOTTOMLEFT
El-Haj, M. (2020, May). Habibi - a multi dialect multi national Arabic song lyrics corpus. In Proceedings of the 12th Language Resources and Evaluation Conference , Marseille, France, pp. 1318-1326. European Language Resources Association.
BOTTOMLEFT
arXiv preprint arXiv:1611.04033 .
BOTTOMLEFT
El-Khair, I. A. (2016). 1.5 billion words arabic corpus.
BOTTOMLEFT
Fan, A., S. Bhosale, H. Schwenk, Z. Ma, A. El-Kishky, S. Goyal, M. Baines, O. Celebi, G. Wenzek, V. Chaudhary, et al. (2021). Beyond english-centric multilingual machine translation. J. Mach.
BOTTOMLEFT
Learn. Res. 22 (107), 1-48.
BOTTOMLEFT
Galliano, S., E. Geoffrois, G. Gravier, J.-F. Bonastre, D. Mostefa, and K. Choukri (2006, May). Corpus description of the ESTER evaluation campaign for the rich transcription of French broadcast news. In Proceedings of the Fifth International Conference on Language Resources and Evaluation
BOTTOMLEFT
Gao, L., S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy (2020). The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 .
table
#/body
BOTTOMLEFT
#/tables/2
BOTTOMLEFT
Shikali, S. C. and M. Refuoe (2019, November). Language modeling data for swahili.
BOTTOMLEFT
Shikali, S. C. and M. Refuoe (2019, November). Language modeling data for swahili.
BOTTOMLEFT
Siripragada, S., J. Philip, V. P. Namboodiri, and C. V. Jawahar (2020, May). A multilingual parallel corpora collection effort for Indian languages. In Proceedings of the 12th Language Resources and Evaluation Conference , Marseille, France, pp. 3743-3751. European Language Resources Association.
BOTTOMLEFT
machine learning. arXiv preprint arXiv:2007.02423 .
BOTTOMLEFT
Sloane, M., E. Moss, O. Awomolo, and L. Forlano (2020). Participation is not a design fix for
BOTTOMLEFT
Smith, S., M. Patwary, B. Norick, P. LeGresley, S. Rajbhandari, J. Casper, Z. Liu, S. Prabhumoye, G. Zerveas, V. Korthikanti, E. Zhang, R. Child, R. Y. Aminabadi, J. Bernauer, X. Song, M. Shoeybi, Y. He, M. Houston, S. Tiwary, and B. Catanzaro (2022). Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. arXiv .
BOTTOMLEFT
Talat, Z., A. Névéol, S. Biderman, M. Clinciu, M. Dey, S. Longpre, S. Luccioni, M. Masoud, M. Mitchell, D. Radev, S. Sharma, A. Subramonian, J. Tae, S. Tan, D. Tunuguntla, and O. van der Wal (2022). You reap what you sow: On the challenges of bias evaluation under multilingual settings. In Challenges & Perspectives in Creating Large Language Models .
BOTTOMLEFT
Vuong, Q.-H., V.-P. La, T.-H. T. Nguyen, M.-H. Nguyen, T.-T. Le, and M.-T. Ho (2021). An ai- enabled approach in analyzing media data: An example from data on covid-19 news coverage in
BOTTOMLEFT
Wenzek, G., M.-A. Lachaux, A. Conneau, V. Chaudhary, F. Guzmán, A. Joulin, and É. Grave (2020). Ccnet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of The 12th Language Resources and Evaluation Conference , pp. 4003-4012.
BOTTOMLEFT
Wibowo, H. A. (2020). Recibrew. https://github.com/haryoa/ingredbrew.
BOTTOMLEFT
Winner, L. (2017). Do artifacts have politics? In Computer Ethics , pp. 177-192. Routledge.
BOTTOMLEFT
Xue, L., N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, and C. Raffel (2020). mt5: A massively multilingual pre-trained text-to-text transformer. CoRR abs/2010.11934 .
BOTTOMLEFT
Zerrouki, T. and A. Balla (2017). Tashkeela: Novel corpus of arabic vocalized texts, data for
BOTTOMLEFT
auto-diacritization systems. Data in brief 11 , 147.
BOTTOMLEFT
Zhang, B., P. Williams, I. Titov, and R. Sennrich (2020). Improving massively multilingual neural machine translation and zero-shot translation. arXiv preprint arXiv:2004.11867 .
BOTTOMLEFT
Zhang, C., D. Ippolito, K. Lee, M. Jagielski, F. Tramèr, and N. Carlini (2021). Counterfactual memorization in neural language models. arXiv preprint arXiv:2112.12938 .
BOTTOMLEFT
Zhang, S., S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al. (2022). Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 .
BOTTOMLEFT
Ziemski, M., M. Junczys-Dowmunt, and B. Pouliquen (2016, May). The United Nations parallel corpus v1.0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16) , Portorož, Slovenia, pp. 3530-3534. European Language Resources Association (ELRA).
BOTTOMLEFT
Shikali, S. C. and M. Refuoe (2019, November). Language modeling data for swahili.
BOTTOMLEFT
Siripragada, S., J. Philip, V. P. Namboodiri, and C. V. Jawahar (2020, May). A multilingual parallel corpora collection effort for Indian languages. In Proceedings of the 12th Language Resources and Evaluation Conference , Marseille, France, pp. 3743-3751. European Language Resources Association.
BOTTOMLEFT
machine learning. arXiv preprint arXiv:2007.02423 .
BOTTOMLEFT
Sloane, M., E. Moss, O. Awomolo, and L. Forlano (2020). Participation is not a design fix for
BOTTOMLEFT
Smith, S., M. Patwary, B. Norick, P. LeGresley, S. Rajbhandari, J. Casper, Z. Liu, S. Prabhumoye, G. Zerveas, V. Korthikanti, E. Zhang, R. Child, R. Y. Aminabadi, J. Bernauer, X. Song, M. Shoeybi, Y. He, M. Houston, S. Tiwary, and B. Catanzaro (2022). Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. arXiv .
BOTTOMLEFT
Talat, Z., A. Névéol, S. Biderman, M. Clinciu, M. Dey, S. Longpre, S. Luccioni, M. Masoud, M. Mitchell, D. Radev, S. Sharma, A. Subramonian, J. Tae, S. Tan, D. Tunuguntla, and O. van der Wal (2022). You reap what you sow: On the challenges of bias evaluation under multilingual settings. In Challenges & Perspectives in Creating Large Language Models .
BOTTOMLEFT
Vuong, Q.-H., V.-P. La, T.-H. T. Nguyen, M.-H. Nguyen, T.-T. Le, and M.-T. Ho (2021). An ai- enabled approach in analyzing media data: An example from data on covid-19 news coverage in
BOTTOMLEFT
Wenzek, G., M.-A. Lachaux, A. Conneau, V. Chaudhary, F. Guzmán, A. Joulin, and É. Grave (2020). Ccnet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of The 12th Language Resources and Evaluation Conference , pp. 4003-4012.
BOTTOMLEFT
Wibowo, H. A. (2020). Recibrew. https://github.com/haryoa/ingredbrew.
BOTTOMLEFT
Winner, L. (2017). Do artifacts have politics? In Computer Ethics , pp. 177-192. Routledge.
BOTTOMLEFT
Xue, L., N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, and C. Raffel (2020). mt5: A massively multilingual pre-trained text-to-text transformer. CoRR abs/2010.11934 .
BOTTOMLEFT
Zerrouki, T. and A. Balla (2017). Tashkeela: Novel corpus of arabic vocalized texts, data for
BOTTOMLEFT
auto-diacritization systems. Data in brief 11 , 147.
BOTTOMLEFT
Zhang, B., P. Williams, I. Titov, and R. Sennrich (2020). Improving massively multilingual neural machine translation and zero-shot translation. arXiv preprint arXiv:2004.11867 .
BOTTOMLEFT
Zhang, C., D. Ippolito, K. Lee, M. Jagielski, F. Tramèr, and N. Carlini (2021). Counterfactual memorization in neural language models. arXiv preprint arXiv:2112.12938 .
BOTTOMLEFT
Zhang, S., S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al. (2022). Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 .
BOTTOMLEFT
Ziemski, M., M. Junczys-Dowmunt, and B. Pouliquen (2016, May). The United Nations parallel corpus v1.0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16) , Portorož, Slovenia, pp. 3530-3534. European Language Resources Association (ELRA).
table
#/body
BOTTOMLEFT
#/tables/3
#/texts/258
BOTTOMLEFT
Business Economy Technology Google services down for users around the world
BOTTOMLEFT
Business & Economy Technology Google services down for users around the world
BOTTOMLEFT
Frustrated customers countries including Australia, Japan , France and the United States complained online of the outage and tracking website DownDetector reported Google services were down in every continent
BOTTOMLEFT
Frustrated customers countries including Australia, Japan , France and the United States complained online of the outage and tracking website DownDetector reported Google services were down in every continent.
BOTTOMLEFT
Popular Google services including Gmail and Drive were down for many users around the world Thursday , with the technology giant telling affected people they were aware of service disruption.
BOTTOMLEFT
Popular Google services including Gmail and Drive were down for many users around the world Thursday , with the technology giant telling affected people they were aware of service disruption.
BOTTOMLEFT
Anyone else having issues with @gmail in Australia?" one person tweeted.
BOTTOMLEFT
Anyone else having issues with @gmail in Australia?" one tweeted
BOTTOMLEFT
Another Twitter user, in Brooklyn, New York wrote: "Nearly years and this the first time can remember Gmail being completely down
BOTTOMLEFT
person Another Twitter user , in Brooklyn, New York wrote: "Nearly years and this is the first time can remember Gmail being completely down .
BOTTOMLEFT
Google' s @Gmail Twitter feed replied to the posts with: Thanks for reporting. are aware service disruption the moment.
BOTTOMLEFT
Google @Gmail Twitter feed replied to the posts with: Thanks for reporting. are aware service disruption the moment
BOTTOMLEFT
The message contained link Gmail service details page that told users are continuing investigate this issue and check back later
BOTTOMLEFT
The message contained link Gmail service details page that told users we are continuing investigate this issue and check back later_
BOTTOMLEFT
well English, the Gail Twitter feed replied people in French, Japanese, Portuguese and German
BOTTOMLEFT
well English, the Gmail Twitter feed replied people in French, Japanese, Portuguese and German
BOTTOMLEFT
Responding AFP enquiry, Goog le refer the Suite Dashboard for status updates said
BOTTOMLEFT
Responding AFP enquiry, Google said refer the Suite Dashboard for status updates
BOTTOMLEFT
Talk show host Ellen DeGeneres apologizes over allegations
BOTTOMLEFT
Trump VOwS to block any TikTok deal that allows Chinese control
BOTTOMLEFT
Business Economy Technology Google services down for users around the world
BOTTOMLEFT
Business & Economy Technology Google services down for users around the world
BOTTOMLEFT
Frustrated customers countries including Australia, Japan , France and the United States complained online of the outage and tracking website DownDetector reported Google services were down in every continent
BOTTOMLEFT
Frustrated customers countries including Australia, Japan , France and the United States complained online of the outage and tracking website DownDetector reported Google services were down in every continent.
BOTTOMLEFT
Popular Google services including Gmail and Drive were down for many users around the world Thursday , with the technology giant telling affected people they were aware of service disruption.
BOTTOMLEFT
Popular Google services including Gmail and Drive were down for many users around the world Thursday , with the technology giant telling affected people they were aware of service disruption.
BOTTOMLEFT
Anyone else having issues with @gmail in Australia?" one person tweeted.
BOTTOMLEFT
Anyone else having issues with @gmail in Australia?" one tweeted
BOTTOMLEFT
Another Twitter user, in Brooklyn, New York wrote: "Nearly years and this the first time can remember Gmail being completely down
BOTTOMLEFT
person Another Twitter user , in Brooklyn, New York wrote: "Nearly years and this is the first time can remember Gmail being completely down .
BOTTOMLEFT
Google' s @Gmail Twitter feed replied to the posts with: Thanks for reporting. are aware service disruption the moment.
BOTTOMLEFT
Google @Gmail Twitter feed replied to the posts with: Thanks for reporting. are aware service disruption the moment
BOTTOMLEFT
The message contained link Gmail service details page that told users are continuing investigate this issue and check back later
BOTTOMLEFT
The message contained link Gmail service details page that told users we are continuing investigate this issue and check back later_
BOTTOMLEFT
well English, the Gail Twitter feed replied people in French, Japanese, Portuguese and German
BOTTOMLEFT
well English, the Gmail Twitter feed replied people in French, Japanese, Portuguese and German
BOTTOMLEFT
Responding AFP enquiry, Goog le refer the Suite Dashboard for status updates said
BOTTOMLEFT
Responding AFP enquiry, Google said refer the Suite Dashboard for status updates
BOTTOMLEFT
Talk show host Ellen DeGeneres apologizes over allegations
BOTTOMLEFT
Trump VOwS to block any TikTok deal that allows Chinese control
table
#/body
BOTTOMLEFT
#/tables/4
BOTTOMLEFT
ok_
BOTTOMLEFT
_ok
BOTTOMLEFT
k_o
BOTTOMLEFT
k_g
BOTTOMLEFT
_go
BOTTOMLEFT
goo
BOTTOMLEFT
ood
BOTTOMLEFT
od_
BOTTOMLEFT
d_o
BOTTOMLEFT
2
BOTTOMLEFT
2
BOTTOMLEFT
1
BOTTOMLEFT
1
BOTTOMLEFT
1
BOTTOMLEFT
1
BOTTOMLEFT
1
BOTTOMLEFT
1
BOTTOMLEFT
1
BOTTOMLEFT
ok_
BOTTOMLEFT
_ok
BOTTOMLEFT
k_o
BOTTOMLEFT
k_g
BOTTOMLEFT
_go
BOTTOMLEFT
goo
BOTTOMLEFT
ood
BOTTOMLEFT
od_
BOTTOMLEFT
d_o
BOTTOMLEFT
2
BOTTOMLEFT
2
BOTTOMLEFT
1
BOTTOMLEFT
1
BOTTOMLEFT
1
BOTTOMLEFT
1
BOTTOMLEFT
1
BOTTOMLEFT
1
BOTTOMLEFT
1
table
#/body
BOTTOMLEFT
#/tables/5
BOTTOMLEFT
Dataset
BOTTOMLEFT
Language
BOTTOMLEFT
Source
BOTTOMLEFT
AraBench
BOTTOMLEFT
ar
BOTTOMLEFT
Sajjad et al. (2020)
BOTTOMLEFT
1.5 billion words Arabic Corpus
BOTTOMLEFT
ar
BOTTOMLEFT
El-Khair (2016)
BOTTOMLEFT
BanglaLM
BOTTOMLEFT
bn
BOTTOMLEFT
Kowsher et al. (2021)
BOTTOMLEFT
bangla sentiment classifica- tion datasets
BOTTOMLEFT
bn
BOTTOMLEFT
Rahman et al. (2018)
BOTTOMLEFT
Question answering in Ben- gali
BOTTOMLEFT
bn
BOTTOMLEFT
Mayeesha et al. (2020)
BOTTOMLEFT
Binhvq News Corpus
BOTTOMLEFT
vi
BOTTOMLEFT
Bình (2021)
BOTTOMLEFT
Books by Book Dash
BOTTOMLEFT
en, fr, xh, zu
BOTTOMLEFT
https://bookdash.org/books/
BOTTOMLEFT
Bloom Library
BOTTOMLEFT
ak, bm, fon, ki, lg, ln, nso, rw, st, sw, tn, ts, xh, zu
BOTTOMLEFT
bloomlibrary.org
BOTTOMLEFT
BRAD 2.0
BOTTOMLEFT
ar
BOTTOMLEFT
Elnagar et al. (2018)
BOTTOMLEFT
brWaC Corpus
BOTTOMLEFT
pt
BOTTOMLEFT
EusCrawl
BOTTOMLEFT
eu
BOTTOMLEFT
Artetxe et al. (2022)
BOTTOMLEFT
Catalan General Crawling
BOTTOMLEFT
ca
BOTTOMLEFT
Armengol-Estapé et al. (2021)
BOTTOMLEFT
Catalan Government Crawling
BOTTOMLEFT
ca
BOTTOMLEFT
Armengol-Estapé et al. (2021)
BOTTOMLEFT
Catalan Textual Corpus
BOTTOMLEFT
ca
BOTTOMLEFT
Armengol-Estapé et al. (2021)
BOTTOMLEFT
Data on COVID-19 News Coverage in Vietnam
BOTTOMLEFT
vi
BOTTOMLEFT
Vuong et al. (2021)
BOTTOMLEFT
DuReader
BOTTOMLEFT
zhs
BOTTOMLEFT
He et al. (2018)
BOTTOMLEFT
Enriched CONLLU Ancora for ML training
BOTTOMLEFT
ca
BOTTOMLEFT
Rodriguez-Penagos and Armentano-Oller (2021a)
BOTTOMLEFT
ESTER
BOTTOMLEFT
fr
BOTTOMLEFT
Galliano et al. (2006)
BOTTOMLEFT
Github
BOTTOMLEFT
code
BOTTOMLEFT
Github on BigQuery
BOTTOMLEFT
Habibi
BOTTOMLEFT
ar
BOTTOMLEFT
El-Haj (2020)
BOTTOMLEFT
HAL
BOTTOMLEFT
fr
BOTTOMLEFT
hal.archives-ouvertes.fr/
BOTTOMLEFT
IIT Bombay English-Hindi Parallel Corpus
BOTTOMLEFT
hi
BOTTOMLEFT
Kunchukuttan et al. (2018)
BOTTOMLEFT
IndicNLP Corpus
BOTTOMLEFT
bn, gu, hi, kn, ml, mr, or, pa, ta, te
BOTTOMLEFT
Kunchukuttan et al. (2020)
BOTTOMLEFT
Indo4B BPPT
BOTTOMLEFT
id
BOTTOMLEFT
Budiono et al. (2009)
BOTTOMLEFT
Indo4B OPUS JW300
BOTTOMLEFT
id
BOTTOMLEFT
Agi'c and Vuli'c (2019)
BOTTOMLEFT
Indo4B Kompas
BOTTOMLEFT
id
BOTTOMLEFT
Sakti et al. (2008)
BOTTOMLEFT
Indo4B Parallel Corpus
BOTTOMLEFT
id
BOTTOMLEFT
Pisceldo et al. (2009)
BOTTOMLEFT
Dataset
BOTTOMLEFT
Language
BOTTOMLEFT
Source
BOTTOMLEFT
AraBench
BOTTOMLEFT
ar
BOTTOMLEFT
Sajjad et al. (2020)
BOTTOMLEFT
1.5 billion words Arabic Corpus
BOTTOMLEFT
ar
BOTTOMLEFT
El-Khair (2016)
BOTTOMLEFT
BanglaLM
BOTTOMLEFT
bn
BOTTOMLEFT
Kowsher et al. (2021)
BOTTOMLEFT
bangla sentiment classifica- tion datasets
BOTTOMLEFT
bn
BOTTOMLEFT
Rahman et al. (2018)
BOTTOMLEFT
Question answering in Ben- gali
BOTTOMLEFT
bn
BOTTOMLEFT
Mayeesha et al. (2020)
BOTTOMLEFT
Binhvq News Corpus
BOTTOMLEFT
vi
BOTTOMLEFT
Bình (2021)
BOTTOMLEFT
Books by Book Dash
BOTTOMLEFT
en, fr, xh, zu
BOTTOMLEFT
https://bookdash.org/books/
BOTTOMLEFT
Bloom Library
BOTTOMLEFT
ak, bm, fon, ki, lg, ln, nso, rw, st, sw, tn, ts, xh, zu
BOTTOMLEFT
bloomlibrary.org
BOTTOMLEFT
BRAD 2.0
BOTTOMLEFT
ar
BOTTOMLEFT
Elnagar et al. (2018)
BOTTOMLEFT
brWaC Corpus
BOTTOMLEFT
pt
BOTTOMLEFT
EusCrawl
BOTTOMLEFT
eu
BOTTOMLEFT
Artetxe et al. (2022)
BOTTOMLEFT
Catalan General Crawling
BOTTOMLEFT
ca
BOTTOMLEFT
Armengol-Estapé et al. (2021)
BOTTOMLEFT
Catalan Government Crawling
BOTTOMLEFT
ca
BOTTOMLEFT
Armengol-Estapé et al. (2021)
BOTTOMLEFT
Catalan Textual Corpus
BOTTOMLEFT
ca
BOTTOMLEFT
Armengol-Estapé et al. (2021)
BOTTOMLEFT
Data on COVID-19 News Coverage in Vietnam
BOTTOMLEFT
vi
BOTTOMLEFT
Vuong et al. (2021)
BOTTOMLEFT
DuReader
BOTTOMLEFT
zhs
BOTTOMLEFT
He et al. (2018)
BOTTOMLEFT
Enriched CONLLU Ancora for ML training
BOTTOMLEFT
ca
BOTTOMLEFT
Rodriguez-Penagos and Armentano-Oller (2021a)
BOTTOMLEFT
ESTER
BOTTOMLEFT
fr
BOTTOMLEFT
Galliano et al. (2006)
BOTTOMLEFT
Github
BOTTOMLEFT
code
BOTTOMLEFT
Github on BigQuery
BOTTOMLEFT
Habibi
BOTTOMLEFT
ar
BOTTOMLEFT
El-Haj (2020)
BOTTOMLEFT
HAL
BOTTOMLEFT
fr
BOTTOMLEFT
hal.archives-ouvertes.fr/
BOTTOMLEFT
IIT Bombay English-Hindi Parallel Corpus
BOTTOMLEFT
hi
BOTTOMLEFT
Kunchukuttan et al. (2018)
BOTTOMLEFT
IndicNLP Corpus
BOTTOMLEFT
bn, gu, hi, kn, ml, mr, or, pa, ta, te
BOTTOMLEFT
Kunchukuttan et al. (2020)
BOTTOMLEFT
Indo4B BPPT
BOTTOMLEFT
id
BOTTOMLEFT
Budiono et al. (2009)
BOTTOMLEFT
Indo4B OPUS JW300
BOTTOMLEFT
id
BOTTOMLEFT
Agi'c and Vuli'c (2019)
BOTTOMLEFT
Indo4B Kompas
BOTTOMLEFT
id
BOTTOMLEFT
Sakti et al. (2008)
BOTTOMLEFT
Indo4B Parallel Corpus
BOTTOMLEFT
id
BOTTOMLEFT
Pisceldo et al. (2009)
table
#/body
BOTTOMLEFT
#/tables/6
BOTTOMLEFT
Dataset
BOTTOMLEFT
Language
BOTTOMLEFT
Source
BOTTOMLEFT
Indo4B TALPCo
BOTTOMLEFT
id
BOTTOMLEFT
Nomoto et al. (2018)
BOTTOMLEFT
Indo4B Tempo
BOTTOMLEFT
id
BOTTOMLEFT
Sakti et al. (2008)
BOTTOMLEFT
Indonesian Frog Story- telling corpus
BOTTOMLEFT
id
BOTTOMLEFT
Moeljadi (2012)
BOTTOMLEFT
Indonesian News Articles Published at 2017
BOTTOMLEFT
id
BOTTOMLEFT
Ashari (2018)
BOTTOMLEFT
Indonesian News Corpus
BOTTOMLEFT
id
BOTTOMLEFT
Rahutomo and Miqdad Muadz Muzad (2018)
BOTTOMLEFT
IndoNLI
BOTTOMLEFT
id
BOTTOMLEFT
Mahendra et al. (2021)
BOTTOMLEFT
Indosum
BOTTOMLEFT
id
BOTTOMLEFT
Kurniawan and Louvan (2018)
BOTTOMLEFT
KALIMAT
BOTTOMLEFT
ar
BOTTOMLEFT
El-Haj and Koulali (2013)
BOTTOMLEFT
KSUCCA
BOTTOMLEFT
ar
BOTTOMLEFT
Alrabiah et al. (2013)
BOTTOMLEFT
LABR
BOTTOMLEFT
ar
BOTTOMLEFT
Aly and Atiya (2013)
BOTTOMLEFT
Language modeling data for Swahili
BOTTOMLEFT
sw
BOTTOMLEFT
Shikali and Refuoe (2019)
BOTTOMLEFT
Leipzig Corpora Collection
BOTTOMLEFT
ur
BOTTOMLEFT
Goldhahn et al. (2012)
BOTTOMLEFT
Mann Ki Baat
BOTTOMLEFT
bn, gu, hi, ml, mr, or, ta, te, ur
BOTTOMLEFT
Siripragada et al. (2020)
BOTTOMLEFT
Masakhaner
BOTTOMLEFT
ig, lg, rw, sw, wo, yo
BOTTOMLEFT
Adelani et al. (2021)
BOTTOMLEFT
MultiUN v2
BOTTOMLEFT
en, ar, es, fr, zhs
BOTTOMLEFT
Chen and Eisele (2012)
BOTTOMLEFT
Odiencorp
BOTTOMLEFT
en, or
BOTTOMLEFT
Parida et al. (2020)
BOTTOMLEFT
Opensubtitles2016
BOTTOMLEFT
ca, en, ar, es, eu, fr, id, bn, hi, ml, ta, te, ur, pt, vi, zhs
BOTTOMLEFT
Lison and Tiedemann (2016)
BOTTOMLEFT
OpenITI proc
BOTTOMLEFT
ar
BOTTOMLEFT
Belinkov et al. (2019)
BOTTOMLEFT
OPUS-100
BOTTOMLEFT
ca, ar, eu, id, as, bn, gu, hi, ig, kn, ml, mr, or, pa, rw, ta, te, ur, pt, vi, xh, yo, zu
BOTTOMLEFT
Zhang et al. (2020)
BOTTOMLEFT
ParlamentParla
BOTTOMLEFT
ca
BOTTOMLEFT
Külebi (2021)
BOTTOMLEFT
PIB
BOTTOMLEFT
bn, gu, hi, ml, mr, or, pa, ta, te, ur
BOTTOMLEFT
Siripragada et al. (2020)
BOTTOMLEFT
Project Gutenberg
BOTTOMLEFT
en, es, fr, pt, zhs
BOTTOMLEFT
gutenberg.org
BOTTOMLEFT
QED (formely AMARA Corpus)
BOTTOMLEFT
ar, en, es, fr, hi, pt, zhs, zht
BOTTOMLEFT
Abdelali et al. (2014)
BOTTOMLEFT
Recibrew
BOTTOMLEFT
id
BOTTOMLEFT
Wibowo (2020)
BOTTOMLEFT
The Royal Society Corpus
BOTTOMLEFT
en
BOTTOMLEFT
Kermes et al. (2016)
BOTTOMLEFT
S2ORC
BOTTOMLEFT
en
BOTTOMLEFT
Lo et al. (2020)
BOTTOMLEFT
Samanantar
BOTTOMLEFT
as, bn, gu, hi, kn, ml, mr, or, pa, ta, te
BOTTOMLEFT
Ramesh et al. (2021)
BOTTOMLEFT
SANAD
BOTTOMLEFT
ar
BOTTOMLEFT
Einea et al. (2019)
BOTTOMLEFT
SciELO
BOTTOMLEFT
en, es, pt
BOTTOMLEFT
scielo.org
BOTTOMLEFT
Stack Exchange
BOTTOMLEFT
code
BOTTOMLEFT
Gao et al. (2020)
BOTTOMLEFT
Swahili News Classification Dataset
BOTTOMLEFT
sw
BOTTOMLEFT
David (2020)
BOTTOMLEFT
Tashkeela
BOTTOMLEFT
ar
BOTTOMLEFT
Zerrouki and Balla (2017)
BOTTOMLEFT
TeCLa
BOTTOMLEFT
ca
BOTTOMLEFT
Carrino et al. (2021)
BOTTOMLEFT
WIT 3
BOTTOMLEFT
ca, ar, en, es, eu, fr, id, as, bn, gu, hi, kn, ml, mr, pa, sw, ta, te, ur, pt, vi, zhs
BOTTOMLEFT
Cettolo et al. (2012)
BOTTOMLEFT
The Pile: EuroParl
BOTTOMLEFT
en, es, fr, pt
BOTTOMLEFT
Gao et al. (2020)
BOTTOMLEFT
Dataset
BOTTOMLEFT
Language
BOTTOMLEFT
Source
BOTTOMLEFT
Indo4B TALPCo
BOTTOMLEFT
id
BOTTOMLEFT
Nomoto et al. (2018)
BOTTOMLEFT
Indo4B Tempo
BOTTOMLEFT
id
BOTTOMLEFT
Sakti et al. (2008)
BOTTOMLEFT
Indonesian Frog Story- telling corpus
BOTTOMLEFT
id
BOTTOMLEFT
Moeljadi (2012)
BOTTOMLEFT
Indonesian News Articles Published at 2017
BOTTOMLEFT
id
BOTTOMLEFT
Ashari (2018)
BOTTOMLEFT
Indonesian News Corpus
BOTTOMLEFT
id
BOTTOMLEFT
Rahutomo and Miqdad Muadz Muzad (2018)
BOTTOMLEFT
IndoNLI
BOTTOMLEFT
id
BOTTOMLEFT
Mahendra et al. (2021)
BOTTOMLEFT
Indosum
BOTTOMLEFT
id
BOTTOMLEFT
Kurniawan and Louvan (2018)
BOTTOMLEFT
KALIMAT
BOTTOMLEFT
ar
BOTTOMLEFT
El-Haj and Koulali (2013)
BOTTOMLEFT
KSUCCA
BOTTOMLEFT
ar
BOTTOMLEFT
Alrabiah et al. (2013)
BOTTOMLEFT
LABR
BOTTOMLEFT
ar
BOTTOMLEFT
Aly and Atiya (2013)
BOTTOMLEFT
Language modeling data for Swahili
BOTTOMLEFT
sw
BOTTOMLEFT
Shikali and Refuoe (2019)
BOTTOMLEFT
Leipzig Corpora Collection
BOTTOMLEFT
ur
BOTTOMLEFT
Goldhahn et al. (2012)
BOTTOMLEFT
Mann Ki Baat
BOTTOMLEFT
bn, gu, hi, ml, mr, or, ta, te, ur
BOTTOMLEFT
Siripragada et al. (2020)
BOTTOMLEFT
Masakhaner
BOTTOMLEFT
ig, lg, rw, sw, wo, yo
BOTTOMLEFT
Adelani et al. (2021)
BOTTOMLEFT
MultiUN v2
BOTTOMLEFT
en, ar, es, fr, zhs
BOTTOMLEFT
Chen and Eisele (2012)
BOTTOMLEFT
Odiencorp
BOTTOMLEFT
en, or
BOTTOMLEFT
Parida et al. (2020)
BOTTOMLEFT
Opensubtitles2016
BOTTOMLEFT
ca, en, ar, es, eu, fr, id, bn, hi, ml, ta, te, ur, pt, vi, zhs
BOTTOMLEFT
Lison and Tiedemann (2016)
BOTTOMLEFT
OpenITI proc
BOTTOMLEFT
ar
BOTTOMLEFT
Belinkov et al. (2019)
BOTTOMLEFT
OPUS-100
BOTTOMLEFT
ca, ar, eu, id, as, bn, gu, hi, ig, kn, ml, mr, or, pa, rw, ta, te, ur, pt, vi, xh, yo, zu
BOTTOMLEFT
Zhang et al. (2020)
BOTTOMLEFT
ParlamentParla
BOTTOMLEFT
ca
BOTTOMLEFT
Külebi (2021)
BOTTOMLEFT
PIB
BOTTOMLEFT
bn, gu, hi, ml, mr, or, pa, ta, te, ur
BOTTOMLEFT
Siripragada et al. (2020)
BOTTOMLEFT
Project Gutenberg
BOTTOMLEFT
en, es, fr, pt, zhs
BOTTOMLEFT
gutenberg.org
BOTTOMLEFT
QED (formely AMARA Corpus)
BOTTOMLEFT
ar, en, es, fr, hi, pt, zhs, zht
BOTTOMLEFT
Abdelali et al. (2014)
BOTTOMLEFT
Recibrew
BOTTOMLEFT
id
BOTTOMLEFT
Wibowo (2020)
BOTTOMLEFT
The Royal Society Corpus
BOTTOMLEFT
en
BOTTOMLEFT
Kermes et al. (2016)
BOTTOMLEFT
S2ORC
BOTTOMLEFT
en
BOTTOMLEFT
Lo et al. (2020)
BOTTOMLEFT
Samanantar
BOTTOMLEFT
as, bn, gu, hi, kn, ml, mr, or, pa, ta, te
BOTTOMLEFT
Ramesh et al. (2021)
BOTTOMLEFT
SANAD
BOTTOMLEFT
ar
BOTTOMLEFT
Einea et al. (2019)
BOTTOMLEFT
SciELO
BOTTOMLEFT
en, es, pt
BOTTOMLEFT
scielo.org
BOTTOMLEFT
Stack Exchange
BOTTOMLEFT
code
BOTTOMLEFT
Gao et al. (2020)
BOTTOMLEFT
Swahili News Classification Dataset
BOTTOMLEFT
sw
BOTTOMLEFT
David (2020)
BOTTOMLEFT
Tashkeela
BOTTOMLEFT
ar
BOTTOMLEFT
Zerrouki and Balla (2017)
BOTTOMLEFT
TeCLa
BOTTOMLEFT
ca
BOTTOMLEFT
Carrino et al. (2021)
BOTTOMLEFT
WIT 3
BOTTOMLEFT
ca, ar, en, es, eu, fr, id, as, bn, gu, hi, kn, ml, mr, pa, sw, ta, te, ur, pt, vi, zhs
BOTTOMLEFT
Cettolo et al. (2012)
BOTTOMLEFT
The Pile: EuroParl
BOTTOMLEFT
en, es, fr, pt
BOTTOMLEFT
Gao et al. (2020)
table
#/body
BOTTOMLEFT
#/tables/7
BOTTOMLEFT
Dataset
BOTTOMLEFT
Language
BOTTOMLEFT
Source
BOTTOMLEFT
The Pile: USPTO
BOTTOMLEFT
en
BOTTOMLEFT
Gao et al. (2020)
BOTTOMLEFT
UIT-VSMEC
BOTTOMLEFT
vi
BOTTOMLEFT
Ho et al. (2020)
BOTTOMLEFT
United Nations Parallel Corpus
BOTTOMLEFT
ar, en, es, fr, zhs
BOTTOMLEFT
Ziemski et al. (2016)
BOTTOMLEFT
Unsupervised Cross-lingual Representation Learning at Scale Common Crawl Corpus
BOTTOMLEFT
ne
BOTTOMLEFT
Conneau et al. (2020)
BOTTOMLEFT
Urdu Monolingual Corpus
BOTTOMLEFT
ur
BOTTOMLEFT
Jawaid et al. (2014)
BOTTOMLEFT
VietAI SAT
BOTTOMLEFT
vi
BOTTOMLEFT
Ngo and Trinh (2021)
BOTTOMLEFT
Vietnamese Poetry Corpus
BOTTOMLEFT
vi
BOTTOMLEFT
Nguyen et al. (2021)
BOTTOMLEFT
UIT-VSFC
BOTTOMLEFT
vi
BOTTOMLEFT
Nguyen et al. (2018)
BOTTOMLEFT
VilaQuAD
BOTTOMLEFT
ca
BOTTOMLEFT
Rodriguez-Penagos and Armentano-Oller (2021b)
BOTTOMLEFT
VinBigdata-VSLP ASR Challenge 2020
BOTTOMLEFT
vi
BOTTOMLEFT
institute.vinbigdata.org/ev ents/vinbigdata- chia-se-100-gio-du-lieu- tieng-noi-cho-cong-dong/
BOTTOMLEFT
VinBigdata-VSLP Monolin- gual Corpus 2020
BOTTOMLEFT
vi
BOTTOMLEFT
institute.vinbigdata.org/ev ents/vinbigdata- chia-se-100-gio-du-lieu- tieng-noi-cho-cong-dong/
BOTTOMLEFT
VinBigdata-VSLP Bilingual Corpus 2020
BOTTOMLEFT
vi
BOTTOMLEFT
institute.vinbigdata.org/ev ents/vinbigdata- chia-se-100-gio-du-lieu- tieng-noi-cho-cong-dong/
BOTTOMLEFT
ViquiQuAD
BOTTOMLEFT
ca
BOTTOMLEFT
Rodriguez-Penagos and Armentano-Oller (2021c)
BOTTOMLEFT
VNTQcorpus(big)
BOTTOMLEFT
vi
BOTTOMLEFT
http://viet.jnlp.org/download- du-lieu-tu-vung-corpus
BOTTOMLEFT
Wikibooks
BOTTOMLEFT
ca, ar, en, es, eu, fr, id, bn, hi, ml, mr, pa, ta, te, ur, pt, vi, zhs
BOTTOMLEFT
wikibooks.org
BOTTOMLEFT
Wikimedia
BOTTOMLEFT
ca, id, hi, pt
BOTTOMLEFT
wikimedia.org
BOTTOMLEFT
Wikinews
BOTTOMLEFT
ar, ca, en, es, fr, ta, pt, zhs
BOTTOMLEFT
wikinews.org
BOTTOMLEFT
Wikipedia
BOTTOMLEFT
ak, ar, as, bm, bn, ca, en, es, eu, fr, id, ig, gu, hi, ki, kn, lg, ln, ml, mr, nso, ny, or, pa, pt, rn, rw, sn, st, sw, ta, te, tn, ts, tum, tw, ur, vi, wo, yo, zhs,
BOTTOMLEFT
wikipedia.org
BOTTOMLEFT
Wikiquote
BOTTOMLEFT
ar, ca, en, es, eu, fr, id, gu, hi, kn, ml, mr, ta, te, ur, pt, vi, zhs
BOTTOMLEFT
wikiquote.org
BOTTOMLEFT
Wikisource
BOTTOMLEFT
ar, ca, es, eu, fr, id, as, bn, gu, hi, kn, ml, mr, or, pa, ta, te, pt, vi
BOTTOMLEFT
wikisource.org
BOTTOMLEFT
Wikiversity
BOTTOMLEFT
ar, en, es, fr, hi, pt, zhs
BOTTOMLEFT
wikiversity.org
BOTTOMLEFT
Wikivoyage
BOTTOMLEFT
en, es, fr, bn, hi, pt, vi, zhs
BOTTOMLEFT
wikivoyage.org
BOTTOMLEFT
Wiktionary
BOTTOMLEFT
ar, ca, en, es, eu, fr, id, as, bn, gu, hi, kn, ml, mr, or, pa, ta,
BOTTOMLEFT
wiktionary.org
BOTTOMLEFT
WuDaoCorpora
BOTTOMLEFT
te, ur, pt, vi zhs
BOTTOMLEFT
Yuan et al. (2021)
BOTTOMLEFT
XQUAD-ca
BOTTOMLEFT
ca
BOTTOMLEFT
Armengol-Estapé et al. (2021)
BOTTOMLEFT
Dataset
BOTTOMLEFT
Language
BOTTOMLEFT
Source
BOTTOMLEFT
The Pile: USPTO
BOTTOMLEFT
en
BOTTOMLEFT
Gao et al. (2020)
BOTTOMLEFT
UIT-VSMEC
BOTTOMLEFT
vi
BOTTOMLEFT
Ho et al. (2020)
BOTTOMLEFT
United Nations Parallel Corpus
BOTTOMLEFT
ar, en, es, fr, zhs
BOTTOMLEFT
Ziemski et al. (2016)
BOTTOMLEFT
Unsupervised Cross-lingual Representation Learning at Scale Common Crawl Corpus
BOTTOMLEFT
ne
BOTTOMLEFT
Conneau et al. (2020)
BOTTOMLEFT
Urdu Monolingual Corpus
BOTTOMLEFT
ur
BOTTOMLEFT
Jawaid et al. (2014)
BOTTOMLEFT
VietAI SAT
BOTTOMLEFT
vi
BOTTOMLEFT
Ngo and Trinh (2021)
BOTTOMLEFT
Vietnamese Poetry Corpus
BOTTOMLEFT
vi
BOTTOMLEFT
Nguyen et al. (2021)
BOTTOMLEFT
UIT-VSFC
BOTTOMLEFT
vi
BOTTOMLEFT
Nguyen et al. (2018)
BOTTOMLEFT
VilaQuAD
BOTTOMLEFT
ca
BOTTOMLEFT
Rodriguez-Penagos and Armentano-Oller (2021b)
BOTTOMLEFT
VinBigdata-VSLP ASR Challenge 2020
BOTTOMLEFT
vi
BOTTOMLEFT
institute.vinbigdata.org/ev ents/vinbigdata- chia-se-100-gio-du-lieu- tieng-noi-cho-cong-dong/
BOTTOMLEFT
VinBigdata-VSLP Monolin- gual Corpus 2020
BOTTOMLEFT
vi
BOTTOMLEFT
institute.vinbigdata.org/ev ents/vinbigdata- chia-se-100-gio-du-lieu- tieng-noi-cho-cong-dong/
BOTTOMLEFT
VinBigdata-VSLP Bilingual Corpus 2020
BOTTOMLEFT
vi
BOTTOMLEFT
institute.vinbigdata.org/ev ents/vinbigdata- chia-se-100-gio-du-lieu- tieng-noi-cho-cong-dong/
BOTTOMLEFT
ViquiQuAD
BOTTOMLEFT
ca
BOTTOMLEFT
Rodriguez-Penagos and Armentano-Oller (2021c)
BOTTOMLEFT
VNTQcorpus(big)
BOTTOMLEFT
vi
BOTTOMLEFT
http://viet.jnlp.org/download- du-lieu-tu-vung-corpus
BOTTOMLEFT
Wikibooks
BOTTOMLEFT
ca, ar, en, es, eu, fr, id, bn, hi, ml, mr, pa, ta, te, ur, pt, vi, zhs
BOTTOMLEFT
wikibooks.org
BOTTOMLEFT
Wikimedia
BOTTOMLEFT
ca, id, hi, pt
BOTTOMLEFT
wikimedia.org
BOTTOMLEFT
Wikinews
BOTTOMLEFT
ar, ca, en, es, fr, ta, pt, zhs
BOTTOMLEFT
wikinews.org
BOTTOMLEFT
Wikipedia
BOTTOMLEFT
ak, ar, as, bm, bn, ca, en, es, eu, fr, id, ig, gu, hi, ki, kn, lg, ln, ml, mr, nso, ny, or, pa, pt, rn, rw, sn, st, sw, ta, te, tn, ts, tum, tw, ur, vi, wo, yo, zhs,
BOTTOMLEFT
wikipedia.org
BOTTOMLEFT
Wikiquote
BOTTOMLEFT
ar, ca, en, es, eu, fr, id, gu, hi, kn, ml, mr, ta, te, ur, pt, vi, zhs
BOTTOMLEFT
wikiquote.org
BOTTOMLEFT
Wikisource
BOTTOMLEFT
ar, ca, es, eu, fr, id, as, bn, gu, hi, kn, ml, mr, or, pa, ta, te, pt, vi
BOTTOMLEFT
wikisource.org
BOTTOMLEFT
Wikiversity
BOTTOMLEFT
ar, en, es, fr, hi, pt, zhs
BOTTOMLEFT
wikiversity.org
BOTTOMLEFT
Wikivoyage
BOTTOMLEFT
en, es, fr, bn, hi, pt, vi, zhs
BOTTOMLEFT
wikivoyage.org
BOTTOMLEFT
Wiktionary
BOTTOMLEFT
ar, ca, en, es, eu, fr, id, as, bn, gu, hi, kn, ml, mr, or, pa, ta,
BOTTOMLEFT
wiktionary.org
BOTTOMLEFT
WuDaoCorpora
BOTTOMLEFT
te, ur, pt, vi zhs
BOTTOMLEFT
Yuan et al. (2021)
BOTTOMLEFT
XQUAD-ca
BOTTOMLEFT
ca
BOTTOMLEFT
Armengol-Estapé et al. (2021)
table
#/body
BOTTOMLEFT
#/tables/8
#/texts/339
BOTTOMLEFT
Language
BOTTOMLEFT
ISO-639-3 catalog-ref Genus
BOTTOMLEFT
Family
BOTTOMLEFT
Macroarea
BOTTOMLEFT
Size in Bytes
BOTTOMLEFT
Akan
BOTTOMLEFT
aka
BOTTOMLEFT
ak
BOTTOMLEFT
Kwa
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
70,1554
BOTTOMLEFT
Arabic
BOTTOMLEFT
arb
BOTTOMLEFT
ar
BOTTOMLEFT
Semitic
BOTTOMLEFT
Afro-Asiatic
BOTTOMLEFT
Eurasia
BOTTOMLEFT
74,854,900,600
BOTTOMLEFT
Assamese
BOTTOMLEFT
asm
BOTTOMLEFT
as
BOTTOMLEFT
Indic
BOTTOMLEFT
Indo-European Eurasia
BOTTOMLEFT
291,522,098
BOTTOMLEFT
Bambara
BOTTOMLEFT
bam
BOTTOMLEFT
bm
BOTTOMLEFT
Western Mande
BOTTOMLEFT
Mande
BOTTOMLEFT
Africa
BOTTOMLEFT
391,747
BOTTOMLEFT
Basque
BOTTOMLEFT
eus
BOTTOMLEFT
eu
BOTTOMLEFT
Basque
BOTTOMLEFT
Basque
BOTTOMLEFT
Eurasia
BOTTOMLEFT
2,360,470,848
BOTTOMLEFT
Bengali
BOTTOMLEFT
ben
BOTTOMLEFT
bn
BOTTOMLEFT
Indic
BOTTOMLEFT
Indo-European Eurasia
BOTTOMLEFT
18,606,823,104
BOTTOMLEFT
Catalan
BOTTOMLEFT
cat
BOTTOMLEFT
ca
BOTTOMLEFT
Romance
BOTTOMLEFT
Indo-European Eurasia
BOTTOMLEFT
17,792,493,289
BOTTOMLEFT
Chi Chewa
BOTTOMLEFT
nya
BOTTOMLEFT
ny
BOTTOMLEFT
Bantoid
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
1,187,405
BOTTOMLEFT
Chi Shona
BOTTOMLEFT
sna
BOTTOMLEFT
sn
BOTTOMLEFT
Bantoid
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
6,638,639
BOTTOMLEFT
Chi Tumbuka
BOTTOMLEFT
tum
BOTTOMLEFT
tum
BOTTOMLEFT
Bantoid
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
170,360
BOTTOMLEFT
English
BOTTOMLEFT
eng
BOTTOMLEFT
en
BOTTOMLEFT
Germanic
BOTTOMLEFT
Indo-European Eurasia
BOTTOMLEFT
484,953,009,124
BOTTOMLEFT
Fon
BOTTOMLEFT
fon
BOTTOMLEFT
fon
BOTTOMLEFT
Kwa
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
2,478,546
BOTTOMLEFT
French
BOTTOMLEFT
fra
BOTTOMLEFT
fr
BOTTOMLEFT
Romance
BOTTOMLEFT
Indo-European Eurasia
BOTTOMLEFT
Gujarati
BOTTOMLEFT
guj
BOTTOMLEFT
gu
BOTTOMLEFT
Indic
BOTTOMLEFT
Indo-European Eurasia
BOTTOMLEFT
208,242,620,434 1,199,986,460
BOTTOMLEFT
Hindi
BOTTOMLEFT
hin
BOTTOMLEFT
hi
BOTTOMLEFT
Indic
BOTTOMLEFT
Indo-European Eurasia
BOTTOMLEFT
24,622,119,985
BOTTOMLEFT
Igbo
BOTTOMLEFT
ibo
BOTTOMLEFT
ig
BOTTOMLEFT
Igboid
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
14078,521
BOTTOMLEFT
Indonesian
BOTTOMLEFT
ind
BOTTOMLEFT
id
BOTTOMLEFT
Malayo-Sumbawan
BOTTOMLEFT
Austronesian
BOTTOMLEFT
Papunesia
BOTTOMLEFT
19,972,325,222
BOTTOMLEFT
Isi Zulu
BOTTOMLEFT
zul
BOTTOMLEFT
zu
BOTTOMLEFT
Bantoid
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
8,511,561
BOTTOMLEFT
Kannada
BOTTOMLEFT
kan
BOTTOMLEFT
kn
BOTTOMLEFT
Southern Dravidian
BOTTOMLEFT
Dravidian
BOTTOMLEFT
Eurasia
BOTTOMLEFT
2,098,453,560
BOTTOMLEFT
Kikuyu
BOTTOMLEFT
kik
BOTTOMLEFT
ki
BOTTOMLEFT
Bantoid
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
359,615
BOTTOMLEFT
Kinyarwanda
BOTTOMLEFT
kin
BOTTOMLEFT
rw
BOTTOMLEFT
Bantoid
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
40,428,299
BOTTOMLEFT
Kirundi
BOTTOMLEFT
run
BOTTOMLEFT
rn
BOTTOMLEFT
Bantoid
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
3,272,550
BOTTOMLEFT
Lingala
BOTTOMLEFT
lin
BOTTOMLEFT
ln
BOTTOMLEFT
Bantoid
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
1,650,804
BOTTOMLEFT
Luganda
BOTTOMLEFT
lug
BOTTOMLEFT
lg
BOTTOMLEFT
Bantoid
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
4,568,367
BOTTOMLEFT
Malayalam
BOTTOMLEFT
mal
BOTTOMLEFT
ml
BOTTOMLEFT
Southern Dravidian
BOTTOMLEFT
Dravidian
BOTTOMLEFT
Eurasia
BOTTOMLEFT
3,662,571,498
BOTTOMLEFT
Marathi
BOTTOMLEFT
mar
BOTTOMLEFT
mr
BOTTOMLEFT
Indic
BOTTOMLEFT
Indo-European Eurasia
BOTTOMLEFT
1,775,483,122
BOTTOMLEFT
Nepali
BOTTOMLEFT
nep
BOTTOMLEFT
ne
BOTTOMLEFT
Indic
BOTTOMLEFT
Indo-European Eurasia
BOTTOMLEFT
2,551,307,393
BOTTOMLEFT
Northern Sotho
BOTTOMLEFT
nso
BOTTOMLEFT
nso
BOTTOMLEFT
Bantoid
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
1,764,506
BOTTOMLEFT
Odia
BOTTOMLEFT
ori
BOTTOMLEFT
or
BOTTOMLEFT
Indic
BOTTOMLEFT
Indo-European Eurasia
BOTTOMLEFT
1,157,100,133
BOTTOMLEFT
Portuguese
BOTTOMLEFT
por
BOTTOMLEFT
pt
BOTTOMLEFT
Romance
BOTTOMLEFT
Indo-European Eurasia
BOTTOMLEFT
79,277,543,375
BOTTOMLEFT
Punjabi
BOTTOMLEFT
pan
BOTTOMLEFT
pa
BOTTOMLEFT
Indic
BOTTOMLEFT
Indo-European Eurasia
BOTTOMLEFT
1,572,109,752
BOTTOMLEFT
Sesotho
BOTTOMLEFT
sot
BOTTOMLEFT
st
BOTTOMLEFT
Bantoid
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
751,034
BOTTOMLEFT
Setswana
BOTTOMLEFT
tsn
BOTTOMLEFT
tn
BOTTOMLEFT
Bantoid
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
1,502,200
BOTTOMLEFT
Simplified Chinese
BOTTOMLEFT
-
BOTTOMLEFT
zhs
BOTTOMLEFT
Chinese
BOTTOMLEFT
Sino-Tibetan
BOTTOMLEFT
Eurasia
BOTTOMLEFT
261,019,433,892
BOTTOMLEFT
Spanish
BOTTOMLEFT
spa
BOTTOMLEFT
es
BOTTOMLEFT
Romance
BOTTOMLEFT
Indo-European Eurasia
BOTTOMLEFT
175,098,365,045
BOTTOMLEFT
Swahili
BOTTOMLEFT
swh
BOTTOMLEFT
sw
BOTTOMLEFT
Bantoid
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
236,482,543
BOTTOMLEFT
Tamil
BOTTOMLEFT
tam
BOTTOMLEFT
ta
BOTTOMLEFT
Southern Dravidian
BOTTOMLEFT
Dravidian
BOTTOMLEFT
Eurasia
BOTTOMLEFT
7,989,206,220
BOTTOMLEFT
Telugu
BOTTOMLEFT
tel
BOTTOMLEFT
te
BOTTOMLEFT
South-Central Dravidian Dravidian
BOTTOMLEFT
Eurasia
BOTTOMLEFT
2993407,159
BOTTOMLEFT
Traditional Chinese
BOTTOMLEFT
-
BOTTOMLEFT
zht
BOTTOMLEFT
Chinese
BOTTOMLEFT
Sino-Tibetan
BOTTOMLEFT
Eurasia
BOTTOMLEFT
762,489,150
BOTTOMLEFT
Twi
BOTTOMLEFT
twi
BOTTOMLEFT
tw
BOTTOMLEFT
Kwa
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
1,265,041
BOTTOMLEFT
Urdu
BOTTOMLEFT
urd
BOTTOMLEFT
ur
BOTTOMLEFT
Indic
BOTTOMLEFT
Indo-European Eurasia
BOTTOMLEFT
2,781,329,959
BOTTOMLEFT
Vietnamese
BOTTOMLEFT
vie
BOTTOMLEFT
vi
BOTTOMLEFT
Viet-Muong
BOTTOMLEFT
Austro-Asiatic
BOTTOMLEFT
Eurasia
BOTTOMLEFT
43,709,279,959
BOTTOMLEFT
Wolof
BOTTOMLEFT
wol
BOTTOMLEFT
wo
BOTTOMLEFT
Wolof
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
3,606,973
BOTTOMLEFT
Xhosa
BOTTOMLEFT
xho
BOTTOMLEFT
xh
BOTTOMLEFT
Bantoid
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
14,304,074
BOTTOMLEFT
Xitsonga
BOTTOMLEFT
tso
BOTTOMLEFT
ts
BOTTOMLEFT
Bantoid
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
Yoruba
BOTTOMLEFT
yor
BOTTOMLEFT
yo
BOTTOMLEFT
Defoid
BOTTOMLEFT
707,634
BOTTOMLEFT
Programming Languages
BOTTOMLEFT
-
BOTTOMLEFT
-
BOTTOMLEFT
-
BOTTOMLEFT
Niger-Congo -
BOTTOMLEFT
Africa
BOTTOMLEFT
89,695,835 174,700,245,772
BOTTOMLEFT
Language
BOTTOMLEFT
ISO-639-3 catalog-ref Genus
BOTTOMLEFT
Family
BOTTOMLEFT
Macroarea
BOTTOMLEFT
Size in Bytes
BOTTOMLEFT
Akan
BOTTOMLEFT
aka
BOTTOMLEFT
ak
BOTTOMLEFT
Kwa
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
70,1554
BOTTOMLEFT
Arabic
BOTTOMLEFT
arb
BOTTOMLEFT
ar
BOTTOMLEFT
Semitic
BOTTOMLEFT
Afro-Asiatic
BOTTOMLEFT
Eurasia
BOTTOMLEFT
74,854,900,600
BOTTOMLEFT
Assamese
BOTTOMLEFT
asm
BOTTOMLEFT
as
BOTTOMLEFT
Indic
BOTTOMLEFT
Indo-European Eurasia
BOTTOMLEFT
291,522,098
BOTTOMLEFT
Bambara
BOTTOMLEFT
bam
BOTTOMLEFT
bm
BOTTOMLEFT
Western Mande
BOTTOMLEFT
Mande
BOTTOMLEFT
Africa
BOTTOMLEFT
391,747
BOTTOMLEFT
Basque
BOTTOMLEFT
eus
BOTTOMLEFT
eu
BOTTOMLEFT
Basque
BOTTOMLEFT
Basque
BOTTOMLEFT
Eurasia
BOTTOMLEFT
2,360,470,848
BOTTOMLEFT
Bengali
BOTTOMLEFT
ben
BOTTOMLEFT
bn
BOTTOMLEFT
Indic
BOTTOMLEFT
Indo-European Eurasia
BOTTOMLEFT
18,606,823,104
BOTTOMLEFT
Catalan
BOTTOMLEFT
cat
BOTTOMLEFT
ca
BOTTOMLEFT
Romance
BOTTOMLEFT
Indo-European Eurasia
BOTTOMLEFT
17,792,493,289
BOTTOMLEFT
Chi Chewa
BOTTOMLEFT
nya
BOTTOMLEFT
ny
BOTTOMLEFT
Bantoid
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
1,187,405
BOTTOMLEFT
Chi Shona
BOTTOMLEFT
sna
BOTTOMLEFT
sn
BOTTOMLEFT
Bantoid
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
6,638,639
BOTTOMLEFT
Chi Tumbuka
BOTTOMLEFT
tum
BOTTOMLEFT
tum
BOTTOMLEFT
Bantoid
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
170,360
BOTTOMLEFT
English
BOTTOMLEFT
eng
BOTTOMLEFT
en
BOTTOMLEFT
Germanic
BOTTOMLEFT
Indo-European Eurasia
BOTTOMLEFT
484,953,009,124
BOTTOMLEFT
Fon
BOTTOMLEFT
fon
BOTTOMLEFT
fon
BOTTOMLEFT
Kwa
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
2,478,546
BOTTOMLEFT
French
BOTTOMLEFT
fra
BOTTOMLEFT
fr
BOTTOMLEFT
Romance
BOTTOMLEFT
Indo-European Eurasia
BOTTOMLEFT
Gujarati
BOTTOMLEFT
guj
BOTTOMLEFT
gu
BOTTOMLEFT
Indic
BOTTOMLEFT
Indo-European Eurasia
BOTTOMLEFT
208,242,620,434 1,199,986,460
BOTTOMLEFT
Hindi
BOTTOMLEFT
hin
BOTTOMLEFT
hi
BOTTOMLEFT
Indic
BOTTOMLEFT
Indo-European Eurasia
BOTTOMLEFT
24,622,119,985
BOTTOMLEFT
Igbo
BOTTOMLEFT
ibo
BOTTOMLEFT
ig
BOTTOMLEFT
Igboid
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
14078,521
BOTTOMLEFT
Indonesian
BOTTOMLEFT
ind
BOTTOMLEFT
id
BOTTOMLEFT
Malayo-Sumbawan
BOTTOMLEFT
Austronesian
BOTTOMLEFT
Papunesia
BOTTOMLEFT
19,972,325,222
BOTTOMLEFT
Isi Zulu
BOTTOMLEFT
zul
BOTTOMLEFT
zu
BOTTOMLEFT
Bantoid
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
8,511,561
BOTTOMLEFT
Kannada
BOTTOMLEFT
kan
BOTTOMLEFT
kn
BOTTOMLEFT
Southern Dravidian
BOTTOMLEFT
Dravidian
BOTTOMLEFT
Eurasia
BOTTOMLEFT
2,098,453,560
BOTTOMLEFT
Kikuyu
BOTTOMLEFT
kik
BOTTOMLEFT
ki
BOTTOMLEFT
Bantoid
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
359,615
BOTTOMLEFT
Kinyarwanda
BOTTOMLEFT
kin
BOTTOMLEFT
rw
BOTTOMLEFT
Bantoid
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
40,428,299
BOTTOMLEFT
Kirundi
BOTTOMLEFT
run
BOTTOMLEFT
rn
BOTTOMLEFT
Bantoid
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
3,272,550
BOTTOMLEFT
Lingala
BOTTOMLEFT
lin
BOTTOMLEFT
ln
BOTTOMLEFT
Bantoid
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
1,650,804
BOTTOMLEFT
Luganda
BOTTOMLEFT
lug
BOTTOMLEFT
lg
BOTTOMLEFT
Bantoid
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
4,568,367
BOTTOMLEFT
Malayalam
BOTTOMLEFT
mal
BOTTOMLEFT
ml
BOTTOMLEFT
Southern Dravidian
BOTTOMLEFT
Dravidian
BOTTOMLEFT
Eurasia
BOTTOMLEFT
3,662,571,498
BOTTOMLEFT
Marathi
BOTTOMLEFT
mar
BOTTOMLEFT
mr
BOTTOMLEFT
Indic
BOTTOMLEFT
Indo-European Eurasia
BOTTOMLEFT
1,775,483,122
BOTTOMLEFT
Nepali
BOTTOMLEFT
nep
BOTTOMLEFT
ne
BOTTOMLEFT
Indic
BOTTOMLEFT
Indo-European Eurasia
BOTTOMLEFT
2,551,307,393
BOTTOMLEFT
Northern Sotho
BOTTOMLEFT
nso
BOTTOMLEFT
nso
BOTTOMLEFT
Bantoid
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
1,764,506
BOTTOMLEFT
Odia
BOTTOMLEFT
ori
BOTTOMLEFT
or
BOTTOMLEFT
Indic
BOTTOMLEFT
Indo-European Eurasia
BOTTOMLEFT
1,157,100,133
BOTTOMLEFT
Portuguese
BOTTOMLEFT
por
BOTTOMLEFT
pt
BOTTOMLEFT
Romance
BOTTOMLEFT
Indo-European Eurasia
BOTTOMLEFT
79,277,543,375
BOTTOMLEFT
Punjabi
BOTTOMLEFT
pan
BOTTOMLEFT
pa
BOTTOMLEFT
Indic
BOTTOMLEFT
Indo-European Eurasia
BOTTOMLEFT
1,572,109,752
BOTTOMLEFT
Sesotho
BOTTOMLEFT
sot
BOTTOMLEFT
st
BOTTOMLEFT
Bantoid
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
751,034
BOTTOMLEFT
Setswana
BOTTOMLEFT
tsn
BOTTOMLEFT
tn
BOTTOMLEFT
Bantoid
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
1,502,200
BOTTOMLEFT
Simplified Chinese
BOTTOMLEFT
-
BOTTOMLEFT
zhs
BOTTOMLEFT
Chinese
BOTTOMLEFT
Sino-Tibetan
BOTTOMLEFT
Eurasia
BOTTOMLEFT
261,019,433,892
BOTTOMLEFT
Spanish
BOTTOMLEFT
spa
BOTTOMLEFT
es
BOTTOMLEFT
Romance
BOTTOMLEFT
Indo-European Eurasia
BOTTOMLEFT
175,098,365,045
BOTTOMLEFT
Swahili
BOTTOMLEFT
swh
BOTTOMLEFT
sw
BOTTOMLEFT
Bantoid
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
236,482,543
BOTTOMLEFT
Tamil
BOTTOMLEFT
tam
BOTTOMLEFT
ta
BOTTOMLEFT
Southern Dravidian
BOTTOMLEFT
Dravidian
BOTTOMLEFT
Eurasia
BOTTOMLEFT
7,989,206,220
BOTTOMLEFT
Telugu
BOTTOMLEFT
tel
BOTTOMLEFT
te
BOTTOMLEFT
South-Central Dravidian Dravidian
BOTTOMLEFT
Eurasia
BOTTOMLEFT
2993407,159
BOTTOMLEFT
Traditional Chinese
BOTTOMLEFT
-
BOTTOMLEFT
zht
BOTTOMLEFT
Chinese
BOTTOMLEFT
Sino-Tibetan
BOTTOMLEFT
Eurasia
BOTTOMLEFT
762,489,150
BOTTOMLEFT
Twi
BOTTOMLEFT
twi
BOTTOMLEFT
tw
BOTTOMLEFT
Kwa
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
1,265,041
BOTTOMLEFT
Urdu
BOTTOMLEFT
urd
BOTTOMLEFT
ur
BOTTOMLEFT
Indic
BOTTOMLEFT
Indo-European Eurasia
BOTTOMLEFT
2,781,329,959
BOTTOMLEFT
Vietnamese
BOTTOMLEFT
vie
BOTTOMLEFT
vi
BOTTOMLEFT
Viet-Muong
BOTTOMLEFT
Austro-Asiatic
BOTTOMLEFT
Eurasia
BOTTOMLEFT
43,709,279,959
BOTTOMLEFT
Wolof
BOTTOMLEFT
wol
BOTTOMLEFT
wo
BOTTOMLEFT
Wolof
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
3,606,973
BOTTOMLEFT
Xhosa
BOTTOMLEFT
xho
BOTTOMLEFT
xh
BOTTOMLEFT
Bantoid
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
14,304,074
BOTTOMLEFT
Xitsonga
BOTTOMLEFT
tso
BOTTOMLEFT
ts
BOTTOMLEFT
Bantoid
BOTTOMLEFT
Niger-Congo
BOTTOMLEFT
Africa
BOTTOMLEFT
Yoruba
BOTTOMLEFT
yor
BOTTOMLEFT
yo
BOTTOMLEFT
Defoid
BOTTOMLEFT
707,634
BOTTOMLEFT
Programming Languages
BOTTOMLEFT
-
BOTTOMLEFT
-
BOTTOMLEFT
-
BOTTOMLEFT
Niger-Congo -
BOTTOMLEFT
Africa
BOTTOMLEFT
89,695,835 174,700,245,772
table
#/body
BOTTOMLEFT
#/tables/9
#/texts/341
BOTTOMLEFT
Language
BOTTOMLEFT
Number of Pseudocrawled Domains
BOTTOMLEFT
Size in Bytes
BOTTOMLEFT
Spanish
BOTTOMLEFT
108
BOTTOMLEFT
29,440,210,712
BOTTOMLEFT
English
BOTTOMLEFT
22
BOTTOMLEFT
4,537,031,408
BOTTOMLEFT
Swahili
BOTTOMLEFT
5
BOTTOMLEFT
109,110,002
BOTTOMLEFT
Indonesian
BOTTOMLEFT
4
BOTTOMLEFT
770,023,233
BOTTOMLEFT
Basque
BOTTOMLEFT
4
BOTTOMLEFT
281,610,312
BOTTOMLEFT
French
BOTTOMLEFT
3
BOTTOMLEFT
1,416,682,404
BOTTOMLEFT
Hindi
BOTTOMLEFT
2
BOTTOMLEFT
1,536,649,276
BOTTOMLEFT
Simplified Chinese
BOTTOMLEFT
2
BOTTOMLEFT
173,884,238
BOTTOMLEFT
Yoruba
BOTTOMLEFT
2
BOTTOMLEFT
6,198,347
BOTTOMLEFT
Igbo
BOTTOMLEFT
2
BOTTOMLEFT
2,650,116
BOTTOMLEFT
Arabic
BOTTOMLEFT
1
BOTTOMLEFT
694,455,304
BOTTOMLEFT
Portuguese
BOTTOMLEFT
1
BOTTOMLEFT
30,615,557
BOTTOMLEFT
Kinyarwanda
BOTTOMLEFT
1
BOTTOMLEFT
9,301,301
BOTTOMLEFT
Language
BOTTOMLEFT
Number of Pseudocrawled Domains
BOTTOMLEFT
Size in Bytes
BOTTOMLEFT
Spanish
BOTTOMLEFT
108
BOTTOMLEFT
29,440,210,712
BOTTOMLEFT
English
BOTTOMLEFT
22
BOTTOMLEFT
4,537,031,408
BOTTOMLEFT
Swahili
BOTTOMLEFT
5
BOTTOMLEFT
109,110,002
BOTTOMLEFT
Indonesian
BOTTOMLEFT
4
BOTTOMLEFT
770,023,233
BOTTOMLEFT
Basque
BOTTOMLEFT
4
BOTTOMLEFT
281,610,312
BOTTOMLEFT
French
BOTTOMLEFT
3
BOTTOMLEFT
1,416,682,404
BOTTOMLEFT
Hindi
BOTTOMLEFT
2
BOTTOMLEFT
1,536,649,276
BOTTOMLEFT
Simplified Chinese
BOTTOMLEFT
2
BOTTOMLEFT
173,884,238
BOTTOMLEFT
Yoruba
BOTTOMLEFT
2
BOTTOMLEFT
6,198,347
BOTTOMLEFT
Igbo
BOTTOMLEFT
2
BOTTOMLEFT
2,650,116
BOTTOMLEFT
Arabic
BOTTOMLEFT
1
BOTTOMLEFT
694,455,304
BOTTOMLEFT
Portuguese
BOTTOMLEFT
1
BOTTOMLEFT
30,615,557
BOTTOMLEFT
Kinyarwanda
BOTTOMLEFT
1
BOTTOMLEFT
9,301,301
table
#/body
BOTTOMLEFT
#/tables/10