Killkan: The Automatic Speech Recognition Dataset for Kichwa with Morphosyntactic Information
Chihiro Taguchi Jefferson Saransig Dayana Velásquez David Chiang






§ INTRODUCTION

Language endangerment has been one of the world's cultural crises, by which many of the world's languages are losing their speakers at an unprecedented pace <cit.>.
Among efforts to document and revitalize languages, recent years have seen growing attention and work to incorporate digital technologies and media to this end <cit.>.
In the same spirit, this study presents the Killkan[Killkan stands for Kichwa uyashkata payllatak killkak anta (Kichwa automatic speech recognizer) in Kichwa. The word killkan also means “it writes”.] corpus, the first dataset for automatic speech recognition (ASR) for the Kichwa language.
Though Kichwa is estimated to have a few hundred thousand speakers in Ecuador, it is considered endangered, as the society is undergoing a language shift to Spanish only.
In natural language processing (NLP), Kichwa is an extremely low-resource language, as there have been no datasets available for either building language models or conducting computational linguistic research of Kichwa.

Our dataset consists of  hours of audio with its orthographic transcription containing  tokens.
Furthermore, each sentence is annotated with its Spanish translation and morphosyntactic information in the CoNLL-U format of Universal Dependencies (UD) <cit.>.

To evaluate the utility of the dataset, we train ASR models on it by fine-tuning the pretrained model .
The experiment shows that the fine-tuned model's performance was 2.04% Character Error Rate (CER), which is comparable to Wav2Vec2 models fine-tuned on high-resource languages.

In the following section, we provide a linguistic overview of the Kichwa language.
Then, Section <ref> surveys previous related work done in the field of NLP for Quechuan languages.
Section <ref> describes the details of our dataset, including the data source, the annotation process, and a brief analysis of the dataset.
Section <ref> reports the experimental results of training ASR models on our dataset, followed by concluding remarks in Section <ref>.

Our contributions in this work are the following:

    
  * We publish the first dataset for Kichwa containing manually annotated audio, transcription, its Spanish translation, and morphosyntactic information;
    
  * We develop the first ASR models for Kichwa;
    
  * We present a new UD Treebank for Kichwa incorporated in ELAN annotation;
    
  * Our dataset, the ASR models, and the code used to develop them are publicly available.[The dataset and the code are available in <https://github.com/ctaguchi/killkan>, and the model is available in <https://huggingface.co/ctaguchi/killkan_asr>.]




§ BACKGROUND



  
The Kichwa language.
Kichwa is the most widely spoken indigenous language in the Republic of Ecuador, particularly along the Andean mountain range in the middle and the Amazonian region to the east of the country.
Though the number of speakers greatly varies among different statistics, the language is estimated to have at least 300,000 speakers <cit.>.
Kichwa is classified in the Northern Quechua branch of the Quechua II group in the Quechuan language family.
Though the Quechua II group also includes more widely spoken varieties such as Cuzco Quechua and Ayacucho Quechua of Peru, Kichwa shows a number of differences from them in phonology and morphosyntax.
For example, Kichwa has lost ejective consonants, possessive suffixes, the inclusive/exclusive distinction in the first-person plural pronoun, has a reduced system of evidentiality <cit.>.



Kichwa is in fact an umbrella term that involves several regional varieties of Northern Quechua.
The Endangered Language Project elp lists Highland Ecuadorian Kichwa and Lowland Ecuadorian Kichwa, under which several subvarieties are further categorized.
See Figure <ref> for a summary of the classification of Quechuan varieties, and see Figure <ref> for a map of their distribution.

With regard to typological aspects, like other Quechuan varieties, Kichwa is an agglutinative language, where verbal and nominal suffixes and discourse clitics are attached to the root to mark verbal features, cases, and information structure.
The example in (<ref>) shows the agglutination of voice, tense, case, and topic morphemes on the verb root llamka “to work”.


    []
    llamka-naku-nka-kaman=ka
    work-rcp-prosp-ter=top
    Until (someone) works together[rcp: reciprocal voice, prosp: prospective aspect, ter: terminative case, top: topic.]







  
Language contact and code-switching.
Since the arrival of Spanish colonizers in the 16th century, Quechuan languages have had language contact with Spanish <cit.>.
The centuries of bilingualism in the Andean region have strongly influenced the lexicon of Quechuan languages, and it is common for Kichwa speakers to code-switch to Spanish in daily speech, which is a language variety sometimes referred to as Media Lengua <cit.>.
The spoken samples in our dataset also contain code-switched speech with Spanish.
The code-switched segment can range from a morpheme (also called intra-word code-switching <cit.>) to a whole phrase;
examples from the dataset are shown in (<ref>) and (<ref>), respectively, where code-switched parts are in green and underlined.


    []
    Ñuka=rak vacuna-ri-kri-ni.
    1sg=cont vaccinate-refl-prosp-prs.1sg
    I am going to get vaccinated first.[1sg: first person singular, refl: reflexive voice, prs: present tense.]



    []
    Consulta popular alli=mi ri-ku-n.
    inquiry popular good=foc go-prog-prs.3
    The referendum is going well.[foc: focus, prog: progressive aspect, 3: third person.]


Most Kichwa speakers are bilingual with Spanish and speak Spanish with non-Kichwa speakers.
Though still estimated to have a few hundred thousand speakers, Kichwa is an endangered language that younger generations often do not inherit, speaking only Spanish instead <cit.>.
In addition, Kichwa is both politically and socially marginalized, as suggested by the pejorative term for Kichwa, yanka shimi “useless language” <cit.>.
Unlike Quechua in Peru and Bolivia, Kichwa is merely a recognized language and is not granted an official status in Ecuador.
These factors add to the ongoing endangerment of the language, and resource building for language technologies is indispensable for both documentation and revitalization of the language.
Yet, it is worth mentioning that there are ongoing revitalization activities and Kichwa–Spanish bilingual schooling in Ecuador.



  
Orthography.
The orthography of Kichwa is based on the Latin alphabet.
The modern orthographic standardization of Kichwa has undergone two crucial modifications in the late 20th century.
The first attempt to standardize the Kichwa orthography was proposed in 1980.
This orthography exhibits several influences from the Spanish orthography, such as the use of <c> and <q> for the phoneme /k/ (e.g., <quillca> /kilka/ `writing') and the use of <hu> to represent the phoneme /w/ (e.g., <huahua> /wawa/ `child').
In 1998, the orthography was revised again and has been the standard since then <cit.>.
The major modifications are phonology-based simplification, where redundant graphemes such as <qu>/<c> for /k/ and <hu> for /w/ were changed to <k> and <w>, respectively.
Though the old orthography is still sometimes informally used, the transcription in our dataset is in the new orthography since the latter orthography is officially and widely used in today's writing.

The modern Kichwa orthography has 18 letters including three digraphs:
<a>, <ch>, <h>, <i>, <k>, <l>, <ll>, <m>, <n>, <ñ>, <p>, <r>, <s>, <sh>, <t>, <u>, <w>, <y>.
On top of this, two graphemes, <ts> and <z>, may also be used for a small number of words depending on dialects.
For code-switched Spanish words, Spanish orthography is used, though Kichwa orthography may also be used for old loanwords such as <ura> `time' from Spanish hora.
Though the correspondence between the orthography and pronunciation is more or less regular, there are slight dialectal differences in the actual phonetic value for each grapheme.
For example, the word alli “good” is pronounced as /ali/ in Imbabura Kichwa and /a1.2i/ in Chimborazo Kichwa.



§ RELATED WORK

Although there is no previous dataset for Ecuadorian Kichwa, there have been several efforts to create datasets and NLP applications for other related Quechuan languages, especially Southern Quechua varieties such as Cuzco Quechua of Peru.
<cit.> developed a text normalization pipeline and a morphological analyzer for Cuzco Quechua, to which a machine translation system and a dependency treebank are added in their later work <cit.>.
<cit.> is a speech corpus for Ayacucho (Chanca) Quechua and Puno (Collao) Quechua, which are both Southern Quechuan languages of the Quechua II group spoken in Peru.
<cit.> introduces a new parallel text corpus and trains a neural machine translation system for Quechua from Peru and Bolivia, though it does not mention which specific Quechuan variety the text is written in.
Since Quechuan languages are highly agglutinative, they have been sometimes used in morphology-related tasks in NLP.
For example, <cit.> investigates the effect of morphology-aware segmentation instead of Byte-Pair Encoding (BPE) on Quechua.[
The paper does not mention what variety of Quechua was used in their experiments. Their dataset description implies that some Peruvian varieties were used.
]
More recently, another speech dataset for Peruvian indigenous language was released that includes ∼180 hours of Southern Quechua audio <cit.>.
All in all, NLP research and applications for Quechuan languages have centered around the varieties spoken in Peru and Bolivia, and other varieties like Ecuadorian Kichwa have yet to be included in language technologies.



§ DATASET

This section describes the details of our dataset.



 §.§ Source

The source of the audio in the dataset is a radio program “Jaboneropak Ayllullaktapi” (In the neighborhood of Jabonero) provided by Radialistas Apasionadas y Apasionados,[
<https://radialistas.net>.
]
an Ecuador-based non-profit radio station.
It is a compilation of fictional stories related to life during the COVID-19 pandemic.
The program is published under a Creative Commons BY-SA license, permitting re-use and re-distribution of the work.
The acted characters include male and female with various voice qualities and with both adult and child roles.
Though the detailed demographic information of the voice actors is unavailable, it is certain that the speech contains several regional varieties of Highland Kichwa.
The radio program contains 20 episodes in total, each of which has a length of ∼12 minutes approximately.
The total audio length of the whole dataset is ∼234.86 minutes (∼3.91 hours).
The dataset contains  samples, where each audio sample corresponds to a sentence.
The average length of a sample is ∼ seconds.
The transcription contains  tokens, and the average length of a token was ∼ characters.
The average sentence length was ∼ tokens.



 §.§ Annotation

The annotation of the dataset contains the following elements:
time-aligned sentence-level transcriptions, their translation in Spanish, and morphosyntactic annotation compatible with UD.
All of these annotations were done in ELAN <cit.>, and the annotated data are saved as XML-based EAF (ELAN Annotation Format) files.
ELAN is software commonly used to annotate spoken audio and video clips collected during linguistic fieldwork.
A screenshot of the annotation interface for the dataset building in this study is shown in Figure <ref>.
To create an ASR dataset containing pairs of an audio sample and its transcription, the original audio files were segmented into sentence-level audio files based on the timestamps logged in the EAF files.
To process the annotation document with Python, the annotated EAF files were parsed into Python objects by the  library <cit.>.
Though there are UD treebanks that were converted from ELAN-native annotation <cit.>, our dataset is the first attempt to directly incorporate UD annotation in the CoNLL-U format into ELAN to our knowledge.





  §.§.§ Transcription

The source website provides transcriptions in Kichwa for each episode.
However, there were three problems in using the provided transcriptions.
First, words that are actually said by the actors often differ from the transcriptions (token-level inconsistencies).
Second, the actors often insert short sentences or interjections that do not appear in the transcriptions (utterance-level inconsistencies).
Third, the provided transcriptions have inconsistencies in the orthography (orthographic inconsistencies).
Table <ref> summarizes the errors that the original transcription had compared to manually corrected transcriptions.
The metrics used in the Table, Character Error Rate (CER), Word Error Rate (WER), and Word Information Lost (WIL), are defined as follows:

    CER, WER = S + D + IN


    WIL = 1 - CN + CP,

where S, D, I are the numbers of necessary substitutions, deletions, and insertions, respectively, to match the reference text, and N is the number of characters (for CER) or words (for WER and WIL).
P is the number of words in the prediction, and C is the number of correctly predicted words.
As the Table shows, the original transcription had 22.7% CER compared to the corrected transcriptions, meaning that approximately one in five characters was either missing, wrong, or unnecessary, and 54.6% WER, meaning that more than half of the originally transcribed words required some correction.
Furthermore, 7.2% of the actual utterances was missing from the original transcriptions.
Because these discrepancies make it difficult to automatically align the transcriptions with the audio segments, every sentence was manually checked and aligned.

.
2






  §.§.§ Translation

The Spanish translations of the transcriptions are also given by Radialistas.
However, they tend to be free translations that depend on the surrounding contexts and sometimes deviate from the information expressed in Kichwa.
For this reason, the translations were manually checked by a Kichwa–Spanish bilingual speaker and were corrected if necessary.




  §.§.§ Morphosyntactic annotation

The morphosyntactic annotation of this dataset follows the CoNLL-U format of UD that annotates the lemma (), part-of-speech (), morphological features (), syntactic head (), and dependency relation () for each token.



Since Kichwa is highly agglutinative and employs a number of suffixes to express functional meanings, there are several morphological features that are absent in the standard UD guidelines and are newly introduced in this dataset.
A list of newly introduced morphological features are summarized in Table <ref>.
The feature  stands for the ventive (cislocative) morpheme that expresses the “coming” motion in the action expressed by the verb.
The feature key  corresponds to focus-sensitive morphemes.
Kichwa has the additive focus marker =pash “also” and the restrictive focus marker =lla “only”.
The feature key  is used to mark the switch-reference features <cit.> that co-occur with converbs[A converb is “a nonfinite verb form whose main function is to mark
adverbial subordination” <cit.>.].
Switch-reference in Kichwa specifies whether the subject of the subordinate clause is the same as or different from that of the main clause.
For example, in (<ref>), the subject is the same, while in (<ref>), the subject is different:


    
  *  []
    miku-nkapak muna-ni.
    eat-cnv.prp.ss want-prs.1sg
    I want to eat.

    
  *  []
    miku-chun muna-ni.
    eat-cnv.prp.ds want-prs.1sg
    I want (somebody else) to eat.[cnv: converb, prp: purposive mood, ss: same subject, ds: different subject]


Another significant modification from the standard UD guidelines is that this dataset annotates topic and focus in Kichwa as morphological features.
In current UD, morphological features cannot express grammatically marked topic and focus, because the guidelines do not have any features for them.
One reason for this treatment is that markers like topic and focus are syntactically less selective and can be attached to both nominal and verbal expressions.
Because UD's morphological features only allow for lexical, nominal (e.g., case), and verbal features (e.g., tense), features that have to do with the information structure cannot fit in the framework.
Indeed, unlike canonical affixes, the morphemes listed in Table <ref> have less syntactic restrictions as to which syntactic category they can be attached to; therefore, previous studies call them morfemas independientes “independent morphemes” <cit.> or enclíticos “enclitics” <cit.>.

In the standard UD guidelines, clitics are usually treated as independent tokens and are not represented as morphological features of the head token.
However, in this approach, it is impossible to annotate the topic and focus features as morphological features, and therefore information structure remains underrepresented in current UD for topic-prominent languages <cit.> that mark topic and focus morphologically like Kichwa.
For this reason, we tentatively added those information-structural features, which can be automatically converted to separate tokens if necessary.

Given the frequent code-mixing with Spanish in spoken Kichwa, the annotation in the dataset also includes the language code and the intra-word code-switching boundary for each token.
The code-switching annotation is listed in the  column, following the format in other code-switching UD treebanks <cit.>.



 §.§ Analysis



This subsection provides a brief analysis of our dataset with a focus on agglutinativity and code-switching of Kichwa.



  
Morphological complexity.
Table <ref> reports the morphological complexity scores of our Kichwa dataset based on the measures proposed in <cit.>.
The table demonstrates that the morphological complexity of the Kichwa dataset is the highest for MSP (mean size of paradigm), WS (information in word structure), IS (inflectional synthesis), MFH (morphological feature entropy).
This shows the extremely high agglutinativity of Kichwa morphology, because MSP, IS, and MFH are calculated based on the diversity of inflected forms and morphological features.
On the other hand, our dataset did not show a high degree of complexity in terms of TTR (type–token ratio), WH (word entropy), and LH (lemma entropy).
This implies that there is not much diversity in the vocabulary of the dataset, since the dataset consists of a series of stories and has common topics and characters throughout the radio program.



  
Code-switching.
Table <ref> shows the distribution of languages in the dataset.
Code-switched tokens comprise ∼11.19% of the entire dataset, and approximately half of them are word-internally code-switched tokens.
It is empirically known that agglutinative languages in language contact tend to derive morpheme-level code-switching such as in Turkish–German <cit.>, and the code-switching distribution in Kichwa also follows this tendency.
As <cit.> pointed out, code-switched Spanish words appear either in an uninflected form (root) or in a fully inflected form, which can be followed by Kichwa morphemes, and, on the contrary, Kichwa stems are not followed by Spanish morphemes.
In terms of the selectivity of parts-of-speech, various syntactic categories can be code-switched.
Though open-class categories such as nouns and verbs commonly exhibit code-switching, closed-class categories like conjunctions also employ Spanish words, particularly in spoken varieties, as exemplified in the underlined word in (<ref>).
Other colored segments are open-class Spanish words.


    []
    kay=ka gasto=chu o inversión=chu ka-n.
    this=top expense=foc.plq or investment=foc.plq be-prs.3
    Are these expenses or investments?[top: topic, plq: polar question]


.
2





§ EXPERIMENTS: ASR

This section reports the results of training the first ASR models for Kichwa based on our proposed dataset.



 §.§ Setup

We developed a Kichwa ASR model by fine-tuning  with the Kichwa dataset.
Wav2Vec2 is a framework for pretraining a self-supervised ASR model that learns contextualized speech representations <cit.>.
Wav2Vec2 first segments the raw speech input into frames with 16kHz sampling rate and encodes into 512-dimensional features through 7 convolution blocks.
The feature vectors are then quantized into discrete values using Gumbel-softmax <cit.>, and these quantized values are used as the labels later during the pretraining.
For the training step, some parts of the input values are masked, and the same feature vectors are fed into Transformer layers to predict the discrete labels of the masked frames, through which the model learns generalized speech representations.
In this way, Wav2Vec2 does not require manually labeled datasets for training and is able to be flexibly fine-tuned to a wide range of speech-related downstream tasks.
In particular, its offset pretrained model Wav2Vec2-XLSR-53 is trained on 53 languages, and it has been empirically shown that it has a strong adaptability to various languages by fine-tuning with small datasets <cit.>.

For our purpose, the training, validation, and test sets were generated by an 8:1:1 split, respectively.
During the preprocessing, we removed samples shorter than 1 second and longer than 15 seconds to ensure that frame masking is correctly done and to prevent the out-of-memory error, respectively.
The learning rate was set to 10^-4.
We also trained models with smaller training sizes with 500, 1k, and 2k samples to imitate various degrees of low-resource settings.
The training was run for 30 epochs on 1 NVIDIA A10 GPU with a 24GB RAM.
The training took about 6 hours to complete, and the average power usage during the training was about 102W.
For the evaluation metrics, we used CER, WER, and WIL.



 §.§ Results





The experimental results are shown in Table <ref>.
It compares four ASR models trained on different numbers of training samples: all samples (), 2k samples, 1k samples, and 500 samples, with the same hyperparameters.
The best model was the one trained with the most training data, which conforms with the general trend in machine learning.

For comparison, Table <ref> also lists the CER score of the Southern Quechua ASR model (Huqariq) reported in <cit.>; Huqariq was fine-tuned on Spanish monolingual Wav2Vec2 with 144-hour Southern Kichwa training data.
Though the test datasets and the pretrained models are different between our studies and theirs, the clear contrast in CER (28.73% and 2.94%) shows the relatively successful performance of the Kichwa ASR model that was only trained on less than 3% of the Southern Quechua training data.
Importantly, even the extremely low-resource scenario with only 500 training samples achieved 7.35% CER.
Note that WER in Kichwa can be higher than WER in analytic languages like English, as tokens in Kichwa tend to consist of more characters with multiple agglutinated suffixes.
For example, the average length of English tokens in the GUM corpus <cit.> is 4.08 while that of Kichwa tokens in our dataset is 6.04.



 §.§ Error analysis



For an error analysis, we prepared seven error types (, , , , , , ) and categorized the errors found in the dev set.
, , and  are errors in transcribing tokens in those languages.
 is an error where unnecessary spacing is inserted or a necessary spacing is omitted.
 is an error in choosing a punctuation symbol or capitalization.
 is an error where the spellings in both the reference and prediction texts are acceptable.
In other words, this type of error is not a wrong transcription in practice.
 is an error in transcribing an interjection tokens.
Table <ref> lists an actual prediction given by the model for each error type.

Table <ref> provides the distribution of the transcription error types found in the dev set.
The most common errors were punctuation errors, which took up more than one-third of the errors.
Given the fact that 67.81% of the dataset is Kichwa tokens and 8.19% either Spanish or code-switched as shown in Table <ref>, it can be observed that Spanish and code-switching tokens tend to cause errors relatively more often than Kichwa tokens.
Because code-switched Spanish words tend to be either technical words, proper nouns, or other relatively uncommon words, it is difficult to train the model to be able to predict such corner cases correctly in this monolingual fine-tuning.
The prediction examples also exhibit the model's confusion in different Spanish spellings that share the same phoneme, such as <ho>/<o> and <v>/<b>.
Investigation of the methods to improve the transcription of low-frequency code-switched segments is beyond the scope of this study and is left for future work.



Considering the fact that the most common errors were mere punctuation errors and capitalization errors, we also measured the metrics after normalizing texts by lowercasing and removing punctuation.
As summarized in Table <ref>, without casing and punctuation errors, CER was 2.04% and WER 13.41% for the best performing model.



§ CONCLUSION

This study presented Killkan, the first linguistic dataset for Kichwa.
It contains speech and manually annotated transcription, Spanish translation, and morphosyntactic parsing information in UD's CoNLL-U format.
Our dataset also annotates morpheme-level code-switching with Spanish, which enabled us to conduct linguistic analyses related to code-switching such as measuring code-switching frequency.

Our study showcased the process of resource building and ASR model development for an extremely low-resource language.
The experimental results demonstrated 2.04% CER for the speech recognition task by the ASR model trained on less than 4 hours of audio data from our Killkan dataset.
Though this is a promising result for the extremely low-resource language, the analysis of the predicted output highlighted the difficulty for the model to correctly predict uncommon code-switched words.
Since code-switching is a common linguistic activity found across all over the world, especially among endangered languages in contact with other prestige languages, it is an important remaining task to improve prediction accuracy of code-switched words.
Also, the experimental results suggested that having more training samples is likely to contribute to improving the performance of Kichwa ASR, calling for more active resource building for low-resource languages.



§ ETHICAL CONSIDERATIONS

As our dataset has been developed only based on publicly available audio data, there is no direct concern of copyright infringement in this work.
However, there are several potential ethical concerns pertaining to technologies for low-resource languages in general.



  
Accessibility.
Though our dataset and model are publicly available, the mode of the distribution is primarily in English, which might be an obstacle for the non-English-speaking users.
We will try to mitigate the disproportionate accessibility by adding descriptions in Kichwa and Spanish.



  
Demand by the community.
Although our project was positively regarded by several native speakers during the first author's fieldwork in Quito, it does not mean that the technology should be embraced unconditionally by all speakers.



  
Language standardization.
As described in Section <ref>, Ecuadorian Kichwa has a number of subdialects that have slightly different vocabulary, phonology, and morphology from each other.
Since our dataset and our ASR model are based on the standardized writing system, they might become an implicit force to use linguistic expressions of standardized Kichwa.
While this could be a positive effect on the literacy, it could also negatively affect the linguistic diversity of the Kichwa-speaking world.



§ ACKNOWLEDGEMENTS

This material is based upon work supported by the National Science Foundation under Grant No. BCS-2109709 and by the University of Notre Dame under the Summer Language Abroad Grant (Quechua).
We are grateful to Luis Santillán and Lourdes Perugachi for the feedback on the project and the support during the linguistic fieldwork.
We also thank the feedback given at the VI Seminario Internacional Revitalizando Ando and at the Tecnologías Digitales y Lenguas Indígenas Workshop.



§ BIBLIOGRAPHICAL REFERENCES


lrec-coling2024-natbib




§ LANGUAGE RESOURCE REFERENCES


lrec-coling2024-natbib
languageresource



§ APPENDIX A. GLOSSING ABBREVIATIONS




