The ACQDIV Corpus Database and Aggregation Pipeline

We present the ACQDIV corpus database and aggregation pipeline, a tool developed as part of the European Research Council (ERC) funded project ACQDIV, which aims to identify the universal cognitive processes that allow children to acquire any language. The corpus database represents 15 corpora from 14 typologically maximally diverse languages. Here we give an overview of the project, database, and our extensible software package for adding more corpora to the current language sample. Lastly, we discuss how we use the corpus database to mine for universal patterns in child language acquisition corpora and we describe avenues for future research.


Overview
In this paper, we present the ACQDIV corpus database and our extensible corpus aggregation pipeline that generates the database from disparate corpus input formats. We begin by describing in Section 2 the motivations behind the database's compilation for the ERC-funded project "ACQuisition processes in maximally DIVerse languages: min(d)ing the ambient language". 1 One of the most pressing questions in cognitive science is: how is a child able to acquire any of the world's 7000 or so extremely diverse languages? Thus, the goal of the five-year ACQDIV project is to identify universal cognitive processes that enable language acquisition despite the substantial cross-linguistic variation found in the world's languages. To create the language sample needed to study worldwide linguistic diversity, ten typologically very different languages were identified to simulate maximal variation in grammar (Stoll and Bickel, 2013). In Section 3, we describe the languages and corpus formats in detail, including additional language acquisition corpora that we have added through open access sources and international collaborations.
Our database aggregation pipeline is called ACQDIV and is written as a PYTHON (Python Software Foundation, 2018) package and available on PYPI. 2 The input data formats that we support are the two most-often used corpus encoding formats: In short, CHILDES CHAT is an encoding defined as part of TalkBank, an open-source system for sharing and studying conversational interactions. 3 In our work, we use several of the corpora from the Child Language Data Exchange 1 https://www.acqdiv.uzh.ch 2 https://pypi.org/project/acqdiv 3 https://talkbank.org System (CHILDES), a subset of TalkBank (MacWhinney, 2000). We also can process input corpora in TOOLBOX, an encoding standard developed by SIL International. 4 In Section 4, we explain how we extract, transform, and load these different corpus formats and corpus-specific annotation schemes into a syntactically and semantically interoperable database (Moran et al., 2016). In Section 5, we provide instructions for how programmers can extend our PYTHON package to include new corpora from TOOLBOX files or from the more than 130 corpora encoded in CHILDES CHAT, which are openly available in TalkBank. In brief, this process consists of creating an INI configuration file to identify the metadata associated with each language and corpus, and then mapping categories, e.g. mappings between corpus-specific part-of-speech labels or morphological categories and glosses, into cross-linguistic standardized annotation sets that include Universal Dependencies (Nivre, 2016) and the Leipzig Glossing Rules (Comrie et al., 2015).
Finally, in Section 6 we briefly discuss some of the research that we have conducted with the ACQDIV corpus database and avenues for future research on one of the most pressing questions in cognitive science: what cognitive mechanisms enable children to learn any of the world's 7000 or so languages?

Background
The goal of the ACQDIV project is to identify universal cognitive processes that enable language acquisition despite the substantial cross-linguistic variation found in the world's languages. To create the language sample needed to study typological diversity, ten grammatically maximally different languages were identified by applying a fuzzy clustering algorithm that takes as input a set of languages and their typological feature values (e.g. grammatical case, inflectional categories, nominal synthesis) (Stoll and Bickel, 2013). The algorithm is applied to language data from thousands of languages encoded in two broad-coverage typological databases: the World Atlas of Linguistic Structures (Dryer and Haspelmath, 2013) and AUTOTYP (Bickel et al., 2017). It outputs five clusters of maximally diverse languages based on a dozen typological variables that are known to be encoded in a variety of different ways cross-linguistically. To ensure diversity, two languages from each cluster were chosen.
For nine of these languages, there exists open-source or privately-owned longitudinal child language acquisition corpora. A longitudinal language acquisition corpus consists of some number of sessions, i.e. a so-called "target child" is recorded in his or her environment, e.g. for an hour a month over several years, and these sessions are then transcribed and linguistically annotated. For the tenth language, our team is in the process of collecting, transcribing and annotating a longitudinal child language acquisition corpus for Dënesųłiné, an endangered language spoken by the Chipewyan people of northwestern Canada. Additionally, we have added several more open source corpora from TalkBank and from new international collaborators, who encode their data in TOOLBOX, to leverage our corpus database pipeline and because researchers are interested in having their data in a format that is interoperable with the existing corpora in ACQDIV. Such a broad and accessible corpus database allows researchers to ask novel questions about child-directed speech in culturally and linguistically diverse languages.

The Language Sample
Currently, we process 15 different child language acquisition corpora, which represent 14 languages. Table 1 shows general information and statistics about these corpora.
Our data sample represents vastly different morphological systems, ranging from isolating to polysynthetic languages. For example, in isolating languages like Indonesian, words and morphemes are mainly in a oneto-one relationship, as shown in Example (1). This is in stark contrast to polysynthetic languages like Cree, where an entire sentence can be analyzed as a single word made up of many morphemes, as illustrated in Example (2).

Data Extraction and Aggregation
Our source code for processing the corpora is available online in a GitHub repository . The ACQDIV aggregation pipeline's workflow follows the fork-and-pull model. We have written our code base in PYTHON   The repository already contains a sample configuration file (see src/acqdiv/config.ini) in which you can adapt the paths to the corpora and for the database file.
We also have a test suite in place to verify that no regression is introduced in the source code and to check the integrity of the database:

pytest tests/unittests pytest tests/systemtests
We release versions of the ACQDIV corpus database pipeline on PYPI and we archive them in Zenodo, which gives us a Digital Object Identifier (DOI) for reference. This allows users to cite particular versions of the pipeline and the database for scientific replicability.
Our data extraction and aggregation pipeline accepts CHILDES CHAT and SIL TOOLBOX as corpus input formats. CHILDES is the child language acquisition component of the TalkBank system, which is an open-source system for sharing and studying conversational interactions. CHAT is the specification used to encode and analyze child-directed and child-surrounding speech from adults. Each file in CHAT represents one recording session. Files are encoded in Unicode UTF-8 plain text according to a set of (semiloose) specifications. 7 An example of a CHAT file is given in Figure 1.
Each speaker utterance is prefixed with an asterisk. Each utterance may then have optional tiers below it that encode information such as the morphological gloss (%gls), morphological annotation (%xcod), and the translation (e.g. %eng for English). Note, however, that these tier labels may differ from corpus creator-to-corpus creator.
The second corpus format that our pipeline accepts as an input format is SIL International's TOOLBOX format. TOOLBOX is a data management tool for collecting and analyzing lexical data and interlinear text. Field linguists 7 The full CHAT specification is given in the TalkBank manual which we discuss below. Figure 1: Example of CHAT format (Demuth, 2015) often use it for creating a morphologically annotated lexicon. Corpus linguists also use this format for its ease of encoding recording sessions through conversational interactions, where each utterance turn is encoded via several user-defined idiosyncratic tiers (e.g. \tx for text; \gw for gloss; \ps for part-of-speech), each of which is separated by a blank line. As in CHAT, every file represents a recording session (file names differ from corpus to corpus, but often include the recording data in the file name). TOOLBOX files are encoded in plain text Unicode UTF-8 format. An example is given in Figure 2.
Both CHAT and TOOLBOX files have accompanying metadata information that summarizes when, where and how the recordings were made, and also provides information about the speakers, such as who is present in the recording, what role do they play (e.g. target child, parent, caregiver) and how old they are. For CHAT, the metadata is provided at the top of each file in terms of key-value pairs prefixed with "@", as shown in Figure 1. For TOOLBOX, metadata is kept in separate files, typically in the XML formats called IMDI (Broeder and Wittenburg, 2006) and more recently its newer version CMDI (Broeder et al., 2012).
With our corpus database aggregation pipeline, illustrated in Figure 3, we bring together the CHAT and TOOLBOX corpora and their metadata into a single unified relational database format. We have chosen a relational database because it allows us to easily analyze parent and child utterances for statistical patterns across morphologically different languages. Also, our aggregation pipeline creates syntactically interoperable data from the different corpus input formats by unifying them into the relational database. During this transformation, we also make the idiosyncratic annotation schemes in the different input corpora semantically interoperable by re-encoding all morphological labels into unified and standardized vocabularies. 8 Our current output formats are a relational SQLITE database and an R data object that includes data frames for each table and view in the database. We chose SQLITE because it requires no special setup or configuration. Since our database is not very large (∼1.5GB), and in most cases will be used by individual users, we do not expect scalability issues. However, as we use object-relational mapping (ORM), users can also easily load the data into other 8 A next step towards semantic interoperability would be to encode the language data, and the relationships within the data, with an ontology. We leave this for future research. database management systems. We also provide an R data object as an output format because many linguists prefer to use R for data analysis (R Core Team, 2018).
Our relational database schema is comprised of seven tables, as shown in Figure 4. The main table is the CORPORA table, which is related in a cascading one-to-many relationship with the SESSIONS (recordings) table, each of which has multiple UTTERANCES, WORDS, and MORPHEMES. 9

Figure 4: ACQDIV Corpus Database Schema
For each corpus, regardless of format, we extend a general corpus parser, written in PYTHON, from which we parse a set of the input data (recording session and speakers metadata, information about the utterances spoken, including attributes such as the utterance duration, its words, morphemes, morphological annotation, parts-of-speech, etc.). However, given the current limitation of tools for accessing multiple CHAT (and TOOLBOX) corpora at the same time, and given the wide range of idiosyncratic morphological annotation schemes encoded in them, our aggregation pipeline provides users the ability to transform CHAT and TOOLBOX files into a single accessible relational database format.
Given the nature of our typologically diverse language sample, it should not be surprising that some languagespecific categories must be broadened or narrowed to attain some level of semantic interoperability. Consider for example the rich noun class system in Sesotho (a Bantu language of South Africa), which has more than 20 different plural strategies, each labeled differently by the corpus collectors (in Figure 1 note the morphological code "nˆ3" that denotes noun class three). We chose to not only keep this rich set of annotations in our database (e.g. MORPHEMES.GLOSS RAW), but to also collapse these different annotations into a single label, NOUN, so that the Sesotho data can be searched (along with all other languages in the database) with the tag set from the Universal Dependencies for nouns, verbs, etc.
This example highlights the most problematic linguistic challenge that we encountered in the development of the ACQDIV database and corpus aggregation pipeline. That is, the same morphological phenomena may be encoded differently per corpus. However, it is not only the labels that differ from corpus to corpus, the actual format for encoding the morphological tier is highly heterogeneous. This requires that we build corpus-specific parsers for the morphology tiers. Particularly problematic are the CHAT corpora because of the flexibility that this corpus encoding format allows its users -basically anything goes. In our TOOLBOX corpora, morphology is much more consistently encoded because the corpus collectors were quite consistent in using the Leipzig Glossing Rules, which describe a standard for segmenting and labeling morphological annotations that is often used by field linguists.
Morpheme, gloss, and part-of-speech information can be stored either on the same or on separate tiers (see Figures  1 and 2 above). TOOLBOX corpora, and to some extent the Cree and Sesotho CHAT corpora, code morphology on separate tiers, which is programmatically easier to parse. However, multiple tiers for morphological analysis introduce a higher probability for misalignments due to user-introduced errors (and thus a higher percentage of the aggregated data being misaligned) between morphemes and their part-of-speech tags and/or glosses.
The V:C is the part-of-speech label, the OS is the morpheme and NEG and PRES are glosses. As the example shows, there is no gloss for the stem OS. And vice versa, there are no morphemes for the suffixes NEG and PRES, only glosses. Unfortunately, we have found that incomplete morpheme data is common in the CHAT corpora. For example, in many CHAT corpora affixes are the only annotated glosses and for stems there may be no gloss at all. Furthermore, morphemes can be segmented in different ways, including dashes (−), equal sign (=), hash (#), colon (:), and plus (+). These delimiters have different meanings depending on where they occur, e.g. ':' often codes gloss subcategories, as shown above, but in other corpora it may have more than one use (see the ACQDIV database corpus manual for a full explication for each corpus (Moran et al., 2019b)).
Another important aspect of our corpus aggregation pipeline involves cleaning the input data. This process includes: • Removing any elements not of interest to our project, e.g. pause annotations, paralinguistic elements like coughing and sneezing, that are only coded in a few of the corpora, so we cannot compare them across the languages in the sample • Reducing misalignments: certain elements only occur on certain tiers but not on others (e.g. punctuation might be encoded on the word tier, but not on the morphology tiers; word repetitions may occur on the word tier, but not morphology tiers) • Unifying glosses, part-of-speech tags, speaker metadata, utterance timestamps, date formats, corpus tier names (e.g. "xtrn" vs. "xmor" in Japanese MiiPro) as to enable a consistent cross-linguistic search • Removing punctuation (and inferring from it what type of utterance it is, e.g. "!" imperatives, "?" interrogatives) During the cleaning procedure, we also undertake a routine for speaker unification. That is, we identify the same speakers across different recording sessions, so that we can populate a UNIQUESPEAKERS table in our database. This involves a certain amount of metadata correction (e.g. fixing typos in the input) and inference: • Speaker name typos, e.g. "Khetheng"' vs. "Khethang" in Sesotho • Speaker labels across corpora are generic, e.g. "CHI" for target children in CHAT corpora, so we must identify each individual • Speaker names may be different from session-tosession, e.g. "Asato" is the same person as "Asatokun" in Japanese MiiPro We must also undertake speaker role unification, which involves: • Target child identification because some recording sessions have assigned the role "Child" to all children including the target child which should receive the role "Target Child" • Text normalization for different annotator labels for the same person, e.g. "Mother" vs. "mother", "Mother's brother" vs. "Uncle" Part-of-speech and gloss unification is also necessary, i.e.: • Many corpora do not use the standardized Leipzig Glossing Rules or Universal Dependency labels, but instead their own idiosyncratic conventions, e.g. "PRS" vs. "PRES", "PRSP", "PRESP", etc., so we identify and unify these different labeling schemes For missing data, we also try to infer as much as information as possible, including but not limited to: • The macrorole of the speaker (i.e. target child, child, adult) as based on age, e.g. speakers over 12 years old are labeled adults • The speaker's gender, which we infer from speaker roles as labeled in the corpora, e.g. mother is mapped to female • A speaker's age is inferred from their birth date and recording session date Finally, we have found that misalignment in the various corpora between the word, morpheme, and annotation tiers poses a serious problem. This is why we link morphemes not only to the word, but also to the complete utterance in our database schema. Again, we document our decisions in the ACQDIV corpus manual, but to sum up, we provide access to the word and morpheme levels, in both cases of when they align and when they do not align.
Running the pipeline on all corpora, which includes 4081 recording sessions and 3580 metadata files, takes 35 minutes and consumes less than 100MB memory on an Intel Core i5-7200U processor with 8GB RAM. Thus, our system processes on average around 2 recording sessions with 1925 utterances and 5218 words per second.

Adding New Corpora
Our extensible approach to coding our corpus aggregation pipeline has been motivated by the desire to make our code base available to other developers and to make our decisions regarding the ACQDIV corpus database clear for users. For example, this includes unifying different annotation schemes, within a database design that unifies syntactically the different corpus input formats. These procedures were an extremely time-consuming endeavor, so we have tried to develop a workflow for easily adding new and diverse corpora. In this section, we describe how to add new corpora to the ACQDIV database by extending the aggregation pipeline. We give users and developers a brief overview of our current corpus input parsers and the components needed to integrate new data sources.
TalkBank contains a range of resources that are transcribed, richly annotated, and aligned with audio and video recordings. Currently, there are over 130 different corpora representing 26 languages. These materials are publicly available online. As such, we have developed the ACQDIV corpus database aggregation pipeline so that developers can add these, as well as corpora encoded in TOOLBOX, to the resulting output database. Detailed instructions on how to add new corpora are described online in the GitHub repository. Here we provide a very brief description of the workflow.
First, create a new section in the configuration file for the corpus being added. A template section for CHAT and TOOLBOX is available in the repository. Second, create a new Python package under parsers/corpora/main/<corpus name> with the following classes: These classes should inherit from already implemented base classes and override any methods that need adaption. Creation of the appropriate objects follows the abstract factory pattern. The main issues to address are the corpus-specific morphological parsing and the mapping of parts-of-speech and morphological glosses to the standardized label schemes (i.e. Universal Dependencies and/or the Leipzig Glossing Rules).
The cost for adding a new corpus depends on many factors, such as whether the developer is already familiar with the pipeline, but also how much cleaning and inference mechanisms have to be implemented for that corpus. Generally speaking, TOOLBOX corpora are faster to integrate because they usually follow the Leipzig Glossing Rules, while CHAT corpora often use their own encoding rules for their morphology. We found that adding a new corpus takes between a couple of hours to several days. We estimate that a developer unfamiliar with the pipeline might need up to one week to add a new corpus.

Research outcomes
Once the ACQDIV corpus database aggregation pipeline and resulting interoperable database were completed, the main goal of the ACQDIV project was to investigate whether there are universal patterns in the input to children cross-linguistically. One area to investigate is child-directed speech because it has been shown to facilitate language learning through various structural features and frequently occurring patterns in the input to children. These include statistical regularities of isolated words (Lew-Williams et al., 2011), adjacent dependencies (Redington et al., 1998), non-adjacent dependencies (Mintz, 2003), and sentence frames (Fernald and Hurtado, 2006).
In fact, using the ACQDIV database we have found striking similarities across morphologically very different languages in child-directed and child-surrounded speech. Furthermore, the cultural settings of languages in our sample vary from WEIRD (western, educated, industrialized, rich and democratic) societies (Henrich et al., 2010), where the norm is directed interaction between caregivers and children (often using so-called motherese), to cultures where children often get more input from their peers than their parents. We highlight in particular three studies that we have published using the ACQDIV corpus database as input for our analyses.
First, we mined the ACQDIV corpus database for discontinuous, but frequently occurring repetitive patterns, and we found language-independent anchor points that predict with high accuracy parts-of-speech. These patterns, known in the literature as frequent frames (Mintz et al., 2002;Mintz, 2003), occur frequently in all languages in our sample (Moran et al., 2018a). An example of a frequent frame is: • I like you • I hate you • I hear you • I love you • I and you • I not you The frame is a nonadjacent dependency between the sequence of three linguistic elements, e.g. the form A B C, in which A and C (here "I" and "you") predict information about B. As shown above, the intervening word is most likely to be a verb in a large corpus of English data. Frames can be calculated at either the word or morpheme levels. For example, consider the morphological frame: • She is sleeping This frame exemplifies morphosyntactic agreement in English and only verbs can appear between the auxiliary verb is and the progressive suffix -ing. This particular dependency signals the grammatical class of the intervening element, i.e. verb. In this case, the intervening morpheme, in English, is always a verb.
We mined the ACQDIV corpus database for discontinuous frames and found that in our diverse language sample, frequently occurring frames are anchor points for accurately identifying the part-of-speech of the intermediate element at the morphology level across all languages in the sample. This shows that there are statistically frequent and reoccurring patterns in the input to children, regardless of the typological characteristics of the language, which may help the learner identify parts-of-speech (Moran et al., 2018a).
In another recent study (Moran et al., 2019a), we identified a cross-linguistic universal pattern in the input to children known in the psycholinguistics literature as variation sets. An example of a variation set in English is (Küntay and Slobin, 2002): • Who did we see when we went out shopping today?
• Who did we see?
• Who did we see in the store?
• Who did we see today?
• When we went out shopping, who did we see?
In this example, the utterances, in close proximity, are anchored around the word see and the question "Who did you see?". To get the message across, the caregiver repeats and rephrases the same question several times, presumably trying to keep the child's attention. Hence, variation sets are repetitions of words, specifically nouns and verbs, in short sequences of interactions. They are thought to provide the child not only with repetitive information, but variation sets also allow children to learn about the different forms that a lexical item can appear in. This is particularly interesting for morphologically-rich languages, e.g. Turkish (Küntay and Slobin, 2002): (3) Ver el-ler-in-i. To understand the actual distribution of these variation sets, we again mined the ACQDIV database and found that all child directed speech (for all languages in our sample) contains variation sets (Moran et al., 2019a). These repetitions of words in short sequences of interaction provide children not only with lexical repetition, but they are also believed to assist them in learning different lexical forms in morphologically-rich languages.
Lastly, in recent work we model language specific childdirected speech as lexical adjacency network graphs. We then identified that in child-directed speech, regardless of the language in our sample, each corpus shows the so-called small world property associated with networks, i.e. low average path length and high clustering coefficient in the network. We postulate that these global structural characteristics reflect principles of self-organization of the lexicon and they may facilitate cognitive processing and learning (Moran et al., 2018b). Our findings support the emergent view of language development, i.e. language acquisition is a piecemeal process that takes several stages and many years to master, on which the scaffolding of learning words and constructions and their interrelations is paramount.

Future directions
For future research, we plan to increase the coverage of the ACQDIV corpus database by extending it with more corpora from TalkBank and from private corpus owners. For example, we have started to extend the aggregation pipeline with several of the phonetically transcribed corpora in PhonBank (Durand et al., 2013). PhonBank includes child language acquisition corpora with phonetic transcriptions. We are converting the individual transcription practices into a single unified International Phonetic Alphabet encoded in the Unicode Standard via orthography profiles (Moran and Cysouw, 2018). We are now investigating whether there are universal patterns in the input to children at the phonological level (Moran and Stoll, 2018).
As we described above, the ACQDIV corpus database aggregation pipeline can be extended to include any CHAT or TOOLBOX formatted corpus. Additionally, linguistic annotations -to the extent that they can be, cf. Haspelmath (2010a; 2010b) and Newmeyer (2010) -are made semantically comparable by using our mappings tables with annotation labels from the Universal Dependencies and the Leipzig Glossing Rules. As such, TalkBank also includes language data from aphasics, second language acquisition, conversation analysis, and classroom language learning. Since these resources are encoded in CHAT, they can be readily parsed into the ACQDIV unified database format for analysis and research on topics other than child language acquisition. Additionally, given that the Universal Dependencies framework provides NLP tools for parsing and analyzing treebanks, we imagine such tools being applied to (at least some of) the corpora in our database and in CHILDES, so that these datasets can be lemmatized and extended with dependency parses -for example with SPACY (Honnibal and Montani, 2017). This information can then be integrated back into the ACQDIV corpus database, so that new research questions can be asked.
Lastly we note that there are several other tools and software packages available for interacting with the CHILDES data, including TalkBank's CLAN program (allows regular expression search and does basic statistics like mean length of utterance); the NLTK (Loper and Bird, 2002) which has an API for querying the CHILDES XML files; and recently there is CHILDES-DB which has a browsable web application and an API in R (Sanchez et al., 2018).
These tools provide exciting opportunities for crosslinguistic research into language acquisition. Our aim here has also been to provide a rich set of accessible data for research on child language acquisition, both qualitative and quantitative in nature. We believe the data formats that we produce provide the basis for integration with existing projects and the opportunity for creating new and exciting tools for future research.

Summary
In this paper, we have given an overview of the ACQDIV corpus database and aggregation pipeline, which is available as a PYTHON package via PYPI. Our ACQDIV software package integrates 15 corpora from 14 typologically maximally diverse languages into a single syntactically and semantically interoperable database. Our aggregation pipeline is designed to be extensible, and as such, can be extended to include corpora from both CHILDES CHAT and SIL TOOLBOX input formats. Here, we described the technological challenges and architectural decisions we have made so that we have been able to create the tools needed to be able to work with corpora encoded in different input formats and with different annotation schemes and encodings.
Given the vast typologically diversity in the language sample that we currently work with, we have developed a workflow that allows users to map morphological labels and annotations into a unified cross-linguistic format, using Universal Dependencies and the Leipzig Glossing Rules. Given this rich resource of longitudinal child language acquisition corpora from a diverse set of the world's languages, we discussed briefly the kinds of research on universal patterns in the input to children that we have identified. Using the ACQDIV corpus database, our research shows that there is a repertoire of universal distributional patterns in the input to children, that it is a cross-linguistic phenomenon, and that it needs further investigation to help identify the cognitive processes that underlie child language acquisition.

Acknowledgements
The research leading to this paper was part of the project 'Acquisition processes in maximally diverse languages: Min(ding) the ambient languages (ACQDIV)' that has received funding from the European Research Council (ERC) under the European Union's Seventh Framework Programme (FP7-2007(FP7- -2013 (Grant agreement No. 615988;PI Sabine Stoll). We gratefully acknowledge Robert Forkel and Sebastian Bank for comments and feedback on the ACQDIV software package and Janis Goldzycher for contributing to the code base. Also many thanks to three anonymous reviewers.

Author Contributions
SM & AJ designed the database, developed the software package, and wrote the paper. SST designed the ACQDIV project and supervised the project. All authors read and approved the final manuscript.