IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languages
Harman Singh Nitish Gupta Shikhar Bharadwaj Dinesh Tewari Partha Talukdar

V^2A-Mark: Versatile Deep Visual-Audio Watermarking for Manipulation Localization and Copyright Protection
    
Xuanyu Zhang1,2, Youmin Xu1, Runyi Li1, Jiwen Yu1, Weiqi Li1, Zhipei Xu1, Jian Zhang1,2 

1 School of Electronic and Computer Engineering, Peking University

2 Peking University Shenzhen Graduate School-Rabbitpre AIGC Joint Research Laboratory

    April 29, 2024
=========================================================================================================================================================================================================================================================




As large language models (LLMs) see increasing adoption across the globe, it is imperative for LLMs to be representative of the linguistic diversity of the world. India is a linguistically diverse country of 1.4 Billion people. To facilitate research on multilingual LLM evaluation, we release  — the largest benchmark for evaluating LLMs on user-facing generation tasks across a diverse set 29 of Indic languages covering 13 scripts and 4 language families.  
 is composed of diverse generation tasks like cross-lingual summarization, machine translation, and cross-lingual question answering. 
 extends existing benchmarks to many Indic languages through human curation providing multi-way parallel evaluation data for many under-represented Indic languages for the first time. We evaluate a wide range of proprietary and open-source LLMs including GPT-3.5, GPT-4, , , ,  and  on  in a variety of settings. The largest  models performs the best on most tasks, however, there is a significant performance gap in all languages compared to English showing that further research is needed for the development of more inclusive multilingual language models.  is available at https://www.github.com/google-research-datasets/indic-gen-bench/www.github.com/google-research-datasets/indic-gen-bench







§ INTRODUCTION

With the advances in generative language technologies powered by Large Language Models <cit.>, there has been a surge of interest in evaluating the multilingual capabilities of these models.
Recent work <cit.> shows a consistent performance gap between high resource languages and languages with lower amounts of web resources available.
To develop highly multilingual generative LLMs which should work equally well for 100s of languages spoken by billions of people in the world, it is crucial to evaluate their capabilities across a variety of languages to uncover performance gaps and guide future research.

In this work we focus on India, a country with 
1369 rationalized mother tongues spoken by more than a billion people.[https://en.wikipedia.org/wiki/Languages_of_Indiahttps://en.wikipedia.org/wiki/Languages_of_India] Making progress on language technologies for Indic languages will not only improve the state of affairs in this region, but will also provide valuable learning to the NLP community which will be applicable to other geographical regions and language families. 
There are has been much work from the community in building natural language understanding (NLU) models for Indic languages <cit.>, as well as evaluation datasets <cit.> to support such models. In this work, our focus is to develop a high-quality benchmark for evaluating generative language capabilities in a variety of Indic languages across various levels of resourcefulness. 

We release , a multilingual, multi-way parallel benchmark for measuring language generation capabilities across diverse user-facing tasks in 29 Indic languages across 4 language families (tab:lang_resource_based_classification).
 extends existing benchmarks such as CrossSum <cit.>, XQuAD <cit.>, XorQA <cit.>, and FLORES <cit.> for additional Indic languages and is composed of tasks like cross-lingual summarization (), machine translation (), cross-lingual reading comprehension ( and ) and multilingual reading comprehension ().
Each dataset consists of parallel examples in up to 29 low to comparatively higher resource Indic languages; and for some tasks (e.g. ),  provides the first-ever evaluation datasets for as many as 18 of these languages.
We also release a small training set in all tasks for efficient adaptation of LLMs. 
Our comprehensive evaluation of various state-of-the-art proprietary and open-source LLMs on  shows that there is a significant gap in performance between English and Indic languages (see Figure <ref>).
Our contributions are as follows:

    
  * Created and released , a high quality text benchmark in diverse language generation tasks like summarization, question-answering, and translation across 29 Indic languages.  is the largest generation benchmark for Indic languages.

    
  * Comprehensive experimentation on SoTA LLMs (, , , , 3.5, 4, ) across various model sizes and training settings to benchmark their Indic language generation capabilities.

    
  * A qualitative analysis for assessing the gaps in current language technologies and define potential directions of future research.



§ 



 is a high-quality, human-curated benchmark to evaluate text generation capabilities of multilingual models on Indic languages. 
Our benchmark consists of 5 user-facing tasks (viz., summarization, machine translation, and question answering) across 29 Indic languages spanning 13 writing scripts and 4 language families. For certain tasks,  provides the first-ever evaluation dataset for up to 18 Indic languages.
tab:task_examples provides summary of  and examples of instances across tasks present in it.


Languages in  are divided into (relatively) Higher, Medium, and Low resource categories based on the availability of web text resources (see appendix <ref> for details).[We note that the languages called relatively higher resource in this paper,  e.g., Hindi or Bengali, are in fact mid-low Web resource when compared to English and other truly high resource languages. For example, using Wikipedia as a proxy for language resources, compared to 6.6M+ Wikipedia articles in English, there are only 160K Hindi Wikipedia articles.]



0.95
Higher (9): Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Tamil, Telugu, Urdu 

Medium (7): Assamese, Bhojpuri, Nepali, Odia, Punjabi, Pashto, Sanskrit 

Low (13): Awadhi, Haryanvi, Tibetan, Garhwali, Konkani, Chhattisgarhi, Rajasthani, Maithili, Manipuri, Malvi, Marwari, Santali, Bodo 






As evident from the lists above, our benchmark provides a broad-coverage over languages with respect to their resourcedness, allowing users to evaluate language models on relatively high-resource languages such as Hindi and extremely low-resource languages such as Manipuri in Meitei script on a single benchmark.

To curate the evaluation datasets for our benchmark, we use the following existing datasets as the source:
CrossSum <cit.> for cross-lingual summarization, FLORES <cit.> for machine translation, 
XQuAD <cit.> for multilingual QA, and
XoRQA <cit.> for cross-lingual QA.
From each of these datasets we select a subset of English examples to be a part of our benchmark, and then collect professional human translations for these examples in all target Indic languages. 
Some target languages are already covered by the source datasets in which case we re-purpose this existing data and only collect translations for the remaining languages. We also collect and release a small amount of training and validation data making possible evaluation of training techniques like fine-tuning, parameter-efficient training, in-context learning, and others.

Why extend existing benchmarks? We chose to collect human translations of existing benchmarks as opposed to creating evaluation data from scratch due to various reasons:

    
  * Translation-based extension of existing benchmark results in multi-way parallel data, allowing researchers to attribute performance due to task knowledge vs. language understanding, and measure cross-lingual generalization
    
  * For many low-resource languages in , clean text knowledge corpus (e.g., Wikipedia) is not available making it difficult to acquire source data for annotation
    
  * By focusing only on translation quality in the target Indic languages, we are able to leverage the quality control that went into designing the source benchmarks.


Annotators were professional data labelers working as contractors at our organization and with a vendor. Annotators were paid competitive rates in compliance with applicable labor laws and prevailing market rates. Our pay rate to annotators varied across languages, ranging from USD 2.80 per hour for Pashto to USD 15.90 per hour for Tibetan. 



  §.§.§ Cross-Lingual Summarization: 

We create  based on CrossSum <cit.>, a dataset for cross-lingual summarization, which in turn is derived from XL-Sum <cit.>. CrossSum contains multi-way parallel data in 45 languages where BBC news articles as source in a language are paired with corresponding summaries in other languages. Based on their matching criteria, different languages have different amount of source-target pairs. 

We sample 700 English article-summary pairs (100 each from train/dev and 500 from test) and ask human translators to translate the English summary into the target Indic languages. CrossSum already contains data for 9 of our 29 target languages; for these languages we sample 100/100/500 examples from the original dataset to maintain equity with other languages we collect data for.  contains a total of 20.3k examples across 29 Indic languages in our benchmark.



  §.§.§ Machine Translation: 

FLORES-200 <cit.> is a human-annotated multi-way parallel machine translation (MT) benchmark for 200 languages where the same source English sentences are translated by humans into the target 200 languages. It contains data in 22 of our 29 target languages; we extend this by collecting human translations for the remaining 7 new languages leading to a MT benchmark in 29 Indic languages which we call .

FLORES-200 is divided into three splits: dev (997), devtest (1012), test (992), of which the test set it not public. We collect translations for all 997 dev and 1012 devtest sentences, yielding 2009 sentences per language. Collectively,  contains 58.2k examples across 29 Indic languages.



  §.§.§ Multilingual Question-Answering: 
 
We create an Indic Multilingual Question Answering task  based on the multilingual reading comprehension dataset XQuAD <cit.>. XQuAD is in turn derived from the SQuAD dataset <cit.>, in which an English Wikipedia passage is paired with multiple question-answer (QA) pairs where the answers are short spans for the given passage.
The authors of XQuAD collected human translations for 240 passages and 1190 QA pairs from the SQuAD v1.1 development set into 10 higher resource languages (Hindi being the only Indic language). 

To create , we use the 240 passages and 1190 QA pairs from XQuAD as our test set. 
We additionally selected 20 passages and 100 QA pairs from the original SQuAD v1.1 training and development sets each to create our training and development set. 
For all the 280 passages and 1390 QA pairs we collect professional human translations in 12 Indic languages.[ contains all 9 higher-resource languages (see <ref>) and 3 medium-resources languages, namely, Assamese, Odia, and Punjabi.]
Overall,  contains 3.3k passages and 16.6k QA pairs in 12 Indic languages. 




  §.§.§ Cross-Lingual Question-Answering: 

We create Indic Cross-lingual Question-Answering dataset  based on the Xor-TyDi QA dataset <cit.>. Xor-TyDi contains questions in non-English languages paired with English evidence passages and short span answers from those passages (similar to SQuAD). It was created with the idea of developing NLP systems that can answer questions in users' native language by refering to sources in a high-resource language, such as English, which was more likely to contain the answer due to the information scarcity of low-resources languages on the web. The original Xor-TyDi contains data in 7 languages out of which Bengali and Telugu are the two Indic languages.

To create , we select the 302 Bengali and 237 Telugu examples (Bn/Te-question, En-passage, En-answer) from the Xor-TyDi dev set as our test data.[Xor-TyDi has not publicly released its test set.]
Additionally, we sample 600 examples (equally from Bengali and Telugu) from the training set of Xor-TyDi to create our training (100) and development (500) set. 
We then follow a two-staged translation process, where we first ask the human translators to translate the Bengali or Telugu question (Bn/Te-question) into English (En-question). 
In the second stage, we collect translations for these English questions (En-question) into target languages (Xx-question) and translations for the English answers (En-answer) into the target languages (Xx-answer). 

We create two tasks from this translated data:
1. : Each example contains (Xx-question, En-passage, En-answer). This task is similar to the Xor-TyDi dataset in additional Indic languages. 

2. : Each example contains (Xx-question, En-passage, Xx-answer), where the task is to generate the answer in the same language as the question. 

We collect data for 28 Indic languages resulting in 32k examples.[We do not collect translations for Nepali.]







§ EXPERIMENTS AND ANALYSIS

We use  to benchmark multilingual and cross-lingual language generation capabilities of various LLMs on Indic languages.
We perform experiments with a variety of 
open-source LLMs —  <cit.>,  <cit.>,[-2 could not be used due to a restrictive licence],  <cit.>,  <cit.>;
and proprietary LLMs — 3.5, 4 <cit.>, and  <cit.>. 


We compare and analyze the performance of these LLMs and their variants in terms of model sizes under different learning paradigms and settings. We first evaluate model performance on one-shot prompting (<ref>) and also measure performance across language categories based on resourcedness (<ref>). We then evaluate the effect of number of in-context examples shown to the model as supervised data (<ref>) and the effect of prompting in a higher-resource language such an English or Hindi (<ref>). Using the training data contained in , we measure how the performance of LLMs after fine-tuning compares with few-shot prompting (<ref>). Finally, we perform qualitative analysis of models on  and highlight some areas of improvement for future model development (<ref>).



  
Evaluation Metrics For the cross-lingual summarization and translation tasks,  and , we report Character-F1 (ChrF) metric <cit.> since token-level metrics like ROUGE and BLEU are not reliable for low-resource languages <cit.>. To stay consistent with existing literature on QA tasks, we report SQuAD-style Token-F1 on our  and  QA tasks.

On , we report translation performance in both directions—translating from English to the target language () and vice-versa ().





 §.§ Comparison of LLMs on 




In Table <ref> we evaluate , , ,  and  family of models on all tasks of  in a one-shot prompted setting. Numbers are averaged across all languages in the evaluation data. To compare, we also report English performance for 4 and -L. 

We see across tasks that larger models from the same LLM family perform better.
-L performs the best among all LLMs considered, except for the  task where -S performs slightly better.
We find that open source  models perform much worse compared to proprietary models; even the largest -65B model significantly underperforms the smallest -XXS model. -7B instruction tuned model performs better than -13B as well as -65B on most tasks. , which is an instruction tuned version of  <cit.>, pre-trained on large-scale multilingual data, works the best on three out of five tasks in . On  and  it falls behind  and .
Compared to English, we see significant room for improvement (20+ ChrF or Token-F1 points) across all tasks.



 §.§ Performance across language categories


In Table <ref> we report one-shot performance across language categories defined in Section <ref>. We only show performance for -7B-IT, -7B, -65B, 4 and -L models here and report performance for the other models in appendix <ref>. We find that there is a significant performance drop going from higher resourced languages to medium resourced ones, and further drop in lower resourced languages.

We would like to point out two observations here: 

(a) In , the performance for translating English to the target language () drops significantly from higher to lower resourced languages (56.9 → 41.9 for -L) whereas the performance in the  direction does not fall this drastically (68.2 → 62.6). A similar trend is seen when comparing  and . This highlights that current LLMs are better at understanding these lower-resourced languages than generating fluent text in them.  

(b) In few cases, we see smaller performance deltas between medium and lower resourced languages compared to higher and medium categories. From our analysis, this can mainly be attributed to many languages in the lower category being similar to Hindi and written in the same Devanagari script.







 §.§ In-context learning on 



In this section we aim to understand the impact of the number of in-context examples shown to the LLM during few-shot prompting. 

Since  and  input passages are long, we are only able to perform 0-and-1-shot prompting. For  and  we perform 0-to-3-shot prompting, and for  we perform 0, 1 and 5-shot prompting.

We show performance for  and  in Table <ref>. Other results are shown in appendix <ref> due to space limitations. Across model families and sizes we observe that increasing the amount of supervision in terms of the in-context examples improves performance. 








 §.§ Transfer from high-resource languages


For languages with no supervised data, one option to improve performance is utilizing existing supervised data another language as in-context exemplars. In this section we aim to study if the language in which the model is prompted plays a role in performance. 

In Table <ref> we show performance when the model is prompted in English vs. Hindi, a representative higher resourced Indic language. For comparison, we also show performance when the in-context exemplar is in the same language as the test instance. 
We find that Hindi in-context exemplars are much more useful for all models as compared to their English counterparts. Surprisingly, for smaller models, performance with Hindi exemplars comes extremely close to prompting in the test language, even better sometimes.




 §.§ Fine-tuning LLMs on  and Comparison with In-Context Learning


As outlined in Section <ref>, we also release a small, high-quality training set for all tasks in  (except  which only has dev and test sets). This training data can be used to adapt LLMs to downstream tasks in Indic languages via fine-tuning and other training techniques.

Table <ref> shows our results of fine-tuning  and  models and their comparison with in-context learning using . We fine-tune each model on training data from all available languages including English, use the development set for early stopping, and report numbers on the test set.
For question-answering tasks that require generating short spans as answers, we find that older generation  models significantly outperform smaller  models in most cases.[Since the parameter count for  models is not public, we cannot attribute this performance difference to model sizes.]
On  which requires generating a longer summary, we find that  models are more effective.

For Question-Answering tasks, as the model size increases from -XXS to -XS, we see that in-context learning yields equal or better performance compared to fine-tuning the model. For example, in , as the model size increases from XXS to XS, we see that the gap between few-shot prompting and fine-tuning significantly increases from 2-4% (in XXS) to 9-10% (in XS). In the case of , we see that for the larger -XS model, its much better to perform in-context learning as compared to fine-tuning, for both medium and high resource Indic languages. For , in-context learning reaches the fine-tuning performance as model size increases to -XS. For the , the gap between fine-tuning and in-context learning is reducing as model size increases, which reinforces that for even larger model sizes, it might be better to learn in-context.





 §.§ Analyzing Tokenizer across Indic languages







In Figure <ref>, we compare the token fertility (average number of sub-words that a word is broken down into by the tokenizer) across all Indic langugaes in .[We use OpenAI's BPE tokenizer https://platform.openai.com/tokenizer(platform.openai.com/tokenizer).  tokenizer is not publicly available.]
We find that the token fertility varies significantly across languages; from 4.1 for Pashto to 19.9 for Tibetan. 

A high token fertility is undesirable and can disproportionately effect a particular language's performance.
For languages where text is broken into more number of tokens, fewer in-context examples can be input to the LLM during inference. This can negatively impact performance (see Table <ref>). In Figure <ref>, we show how the percentage of data that fits in a particular context length changes with number of in-context examples for various languages. For example, we see in Figure <ref> that for medium resource languages with high token-fertility like Oriya and Punjabi we can in-corporate much fewer in-context examples, compared to Indic languages with lower token-fertility like Hindi and Marathi.



 §.§ Qualitative Analysis



We manually analyze predictions from the best performing model -L with the aim to understand the shortcomings of current LLMs and highlight areas of improvements for future research. We randomly select 20 examples each in the  and  tasks for the following languages which are reviewed by native speakers: Awadhi, Haryanvi, Chhatisgarhi, Konkani, and Assamese.
We found the following patterns of errors:



  
Generation in a related language
The languages Awadhi, Haryanvi, and Chhatisgarhi are related to a higher resource language Hindi and written in the same script Devanagari. We find that the model generates mixed-language output with words mixed from Hindi and also outputs incorrectly inflected forms of the main verbs in the output. 
We show couple of examples of this phenomena in Figure <ref> in the appendix.



  
Hallucination and Missing Information
In the cross-lingual summarization task , we find that the model often outputs extra information that is not present in the source article. In translation, we have observed examples where come crucial information from the source sentence is missing from the generated output. Also, in some cases, the model fails to understand polysemous English words and generates translation for the incorrect sense.
We show examples of these phenomena in Figures <ref>, <ref>, and <ref> in the appendix.


§ RELATED WORK


In the last few years, many multilingual LLMs have been developed—starting from mBART <cit.> trained on 25 languages to LLMs that are pre-trained on hundreds of languages, such as 5 <cit.>,  <cit.>, 4 <cit.>, Gemini <cit.>, and others.
These LLMs are typically evaluated on individual multilingual tasks for Translation: WMT <cit.>, FLORES <cit.>; Question-Answering: XQuAD <cit.>, TyDiQA <cit.>, XorQA <cit.>; Summarization: XLSUM <cit.>, CrossSum <cit.>; Reasoning: MGSM <cit.>, XCOPA <cit.> to name a few, or on  multilingual benchmarks such as, XTREME <cit.> and XTREME-UP <cit.>.
However, most of these evaluation resources contain only a handful of languages or do not contain data for low resource languages, especially Indics. Besides, cross-lingual evaluation data is even more sparse. This work is an effort to bridge these gaps by releasing , a suite of datasets covering diverse cross-lingual and multilingual generation tasks in Indic languages.

Most work on creating evaluation data on Indic languages  have focused on natural language understanding (NLU) tasks.
<cit.> and <cit.> have released NLU test sets in Indic languages for a wide variety of tasks such as QA and NLI.
Naamapadam <cit.> is a named entity recognition dataset specifically for Indic languages, MASSIVE <cit.> is a slot-filling and intent classification dataset available in 7 Indic languages, IndicGLUE <cit.> is an NLU benchmark for 11 Indic languages, whereas GLUECoS <cit.> is a Hindi-English code-mixed benchmark, containing various NLU tasks. The Belebele Benchmark <cit.> is a multiple-choice machine reading comprehension dataset for 122 languages of which 17 are Indic.
On the other hand,  is a natural language generation (NLG) benchmark.

Recently, there has been work in creating evaluation benchmarks for natural language generation (NLG) on Indic languages.
IndicNLG Suite <cit.>, consisting of 5 NLG taks in 11 Indic languages, is a leap in this direction.
These datasets in this suite are automatically created, either using data from the web (e.g., Wikipedia) or using translation systems. There are few works which create evaluation data for individual tasks in Indic languages. For example,  IndicTrans2 <cit.> creates an n-way parallel dataset for machine translation in 22 scheduled Indian Languages, Mukhyansh <cit.> and PMIndiaSum <cit.> are headline generation datasets for 8 and 14 Indic languages respectively, and TeSum <cit.> is an abstractive summarization dataset in the Telugu language. <cit.> introduced Samanantar, a large translation dataset covering 11 Indic languages.
Our work complements IndicNLGSuite and the other datasets in multiple ways.
 is manually annotated ensuring high-quality, noise-free text which is not typically found on the web. Our benchmark contains evaluation data for a much larger set of languages spanning low, medium and high resource.
Our datasets are multi-language parallel enabling better comparison among different languages.
Lastly, we focus on a complementary and challenging set of tasks, including cross-lingual summarization, cross-lingual and multilingual question answering, and translation.



§ CONCLUSION

We release , the largest benchmark for evaluating LLMs on 5 user-facing generation tasks across 29 Indic languages, providing evaluation data for many under-represented Indic languages for the first time.  is broad coverage along many dimensions – it covers 13 writing scripts, 4 language families, and spans languages across the available web resource  spectrum. We carry out extensive comparison of current SoTA LLMs on  and highlight areas for future improvement. We are hopeful  will play an important role in further development of LLMs in Indic languages ultimately benefiting a billion-plus population.


§ LIMITATIONS

Since  extends existing benchmarks to new Indic languages through human translation, it may miss some  India-specific entities and linguistic nuances. Future work can explore trans-localization for creating improved evaluation and fine-tuning.  doesn't cover long-form generation and reasoning tasks. Creating such datasets is part of our future work. 


§ ACKNOWLEDGMENTS

We thank Aditi Chaudhury, Ashok Popat, Shachi Dave, Sagar Gubbi, Megh Umekar and members of the Languages team at Google Research India (GRI) for providing feedback on this work. The authors would like to thank Manish Gupta and Divy Thakkar for their support and guidance.



Appendix









§ RESOURCE-WISE LANGUAGE CLASSIFICATION


See Table <ref> for the language resource based classification of the languages studied in our work. This classification is done on the basis of multiple criteria, such as whether google translate supports the particular language, the classification of Indic languages suggested by <cit.> which in turn uses <cit.>.



§ DETAILED AND ADDITONAL EXPERIMENTAL RESULTS

In this section, we report 
(a) one-shot results across all models and tasks averaged over different language categories (<ref>);
(b) performance for different tasks and models by varying the number of in-context exemplars shown in few-shot prompting (<ref>);
(c) one-shot performance for the largest model in each LLM family across all languages (<ref>); 
(d) fine-tuning performance for the largest models across all languages (<ref>).




 §.§ Resource wise one-shot results


In Table <ref> we show performance for all LLMs considered in this paper on the one-shot prompting method across different language categories.



 §.§ Performance by varying number of in-context examples


In Tables <ref>, <ref>, <ref>, and <ref> we show additional results on how performance changes with the number of in-context examples across various tasks in .



 §.§ Language wise one-shot performance across models


In Tables <ref>, <ref>, <ref>, <ref> we show language wise breakdown of performance for largest/best models – -65B, 4, -L in one-shot setting, across all  tasks.



 §.§ Language wise fine-tuning performance across models


In Tables <ref>, <ref>, <ref> we show language wise breakdown of performance for largest  and  models that we fine-tune on , , , .



 §.§ Indic Specific LLMs


To the best of our knowledge, there is no LLM that is pre-trained or fine-tuned on a broad set of Indic languages, or a majority of the 29 languages we study in this work. Some Indic LLMs such as OpenHathi (or its instruction-tuned variant  <cit.>), Tamil-LLama <cit.>, or Kan-LLama are primarily trained/fine-tuned on a single Indic language (Hindi, Telugu, and Kannada, respectively).

In this section we evaluate an Indic specific LLM, Airavata which is primarily pre-trained and fine-tuned on English and Hindi language data. We evaluate the model on 6 Indic languages: Hindi (hi), Bengali (bn), Punjabi (pa), Assamese (as), Manipuri (mni) and Santali (sat) (2 each from higher, medium and low resource), and English. See Table <ref> for the results.

We find that Airavata performs reasonably well only on the Hindi language, and its performance on other Indic languages is much worse (as expected since it is only trained for Hindi). Airavata performs significantly better than LLaMA on the Hindi language, across tasks, except for CSSUM-IN summarization tasks. BLOOMZ outperforms Airavata on most tasks in the Hindi language.


















§ HYPERPARAMETERS


For Fine-tuning experiments on , we hyperparameter search for batch size in the range {16, 32, 64} and learning rate in {5e-3, 1e-3, 5e-4, 1e-4, 5e-5, 1e-5} and select the best hyperparameters based on the Token-F1 (for Question answering datasets) score or ChrF (for ,  experiments) score on the validation set of the dataset. For  fine-tuning experiments, we hyperparameter search for batch size in the range {16, 32, 64} and learning rate in {5e-4, 1e-4, 5e-5, 1e-5} and select the best hyperparameters based on the Token-F1 score or ChrF score, as in the case of  experiments.
We keep temperature=0 as the default setting while decoding using  and  family of models. For  models we increased the temperature to 0.1, since otherwise, the performance of  were coming out to be close to 0 for many of our datasets.
For each dataset, we fix a prompt for all models since searching for a prompt would have led to a much larger compute cost for training and evaluation. We also choose in-context exemplars randomly from the training set (from the dev set in the case of ). This decision is a limitation of this work, since it has been shown that prompt and in-context exemplars can significantly impact an LLMs performance <cit.>. The prompts we use are provided in <ref>. Searching for better prompts and exemplars is left as future work.
For all fine-tuning runs, we just perform one run (as opposed to taking average of multiple runs) due to prohibitive costs of fine-tuning LLMs multiple times.



§ COMPUTING INFRA


For  fine-tuning runs, we use a combination of 4 to 64 TPU-V3, and for  we use up to 256 TPU-V4. All experiments take about 6 hours to  1-2 days of time depending upon the resources and size of the model being trained. We use NVIDIA-V100 16GB GPUs for evaluating open-source models like , ,  on .



§ DATASET LICENSES

In compliance with licenses of the original datasets, we release our evaluation datasets under the following licenses:
(a)  and under the CC-BY-SA 4.0 license; (b)  under the CC BY-NC-SA 4.0 license; and (d)  under the MIT license.



§ PROMPTS


The set of prompts used for evaluations on all datasets are shown in Table <ref>.




























