Setting up the Data Printer with Improved English to Ukrainian Machine Translation
Yurii Paniv Dmytro Chaplynskyi Nikita Trynus Volodymyr Kyrylov






§ INTRODUCTION


The availability of the data is the most important ingredient when one needs to pretrain general-purpose large language models for a specific natural language task or a set of tasks. While it is relatively easy to obtain a good and balanced dataset under specific domain for the English language, it is much harder to do the same for other under-resourced languages such as Ukrainian.


Since curating a corpus of tasks in Ukrainian is a large endeavor, and given a large body of work done for English, we consider existing instruction tuning datasets as a source of tasks to reuse in Ukrainian using automatic machine translation.

This work focuses on improving the current state of machine translation from English to Ukrainian.

We contribute a recipe for finetuning a large pretrained language model with publicly available data to build a translation system (<ref>, <ref>). This  matches state of the art performance of the best encoder-decoder model on a common multilingual benchmark using a consumer GPU with 24 GiB of VRAM.  We release training, evaluation code, datasets, and model at <https://github.com/lang-uk/dragoman>. Our main results are summarized in <ref>. We provide examples of the top-5 best and worst translations on the FLORES devtest set in the <ref>.

We base pretrained model selection on evaluation in few-shot learning setting (<ref>). We find that it's a promising method to design tasks without training, and the model can perform comparably to specialized systems given increased inference budget and auxiliary translation scoring functions, yet still underperforms our finetuned recipe.






§ SUPERVISED FINETUNING
 


We cast machine translation as a likelihood maximization of a density p of Ukrainian sentences Y = "перекладене речення" ∈𝒴 conditioned on their English sources with quasi-instruction formatting: X  = "[INST] translated sentence [/INST]" ∈𝒳.

The density is parametrized using a neural network with frozen pretrained weights θ:

    argmax_ϕp_θ,ϕ(Y|X)


We implement the conditional language modeling objective by masking out tokens of X when computing token-wise cross entropy of shifted targets.
We only optimize extra low rank  adapter <cit.>  parameters ϕ after nf4 quantization <cit.>. In practice we use large rank values and adapter mixture weights. All training runs proceed for one epoch and we use dropout <cit.> for regularization against data noise. 


We use Mistral-7B-v0.1 <cit.> as a base pretrained decoder-only transformer, as it performs favorably in our few-shot experiments (<ref>). 




§ FIRST PHASE: HEURISTIC FILTERING OF PARACRAWL
 

We use the publicly available Paracrawl dataset <cit.>. This dataset contains 13,354,365 English-Ukrainian sentence pairs, collected by automatically matching similar sentences in large corpora of internet text.

We have identified issues with translation pairs, including a significant number of repetitive or incorrect examples. We encounter a large subset of repetitive weather forecasts following the template “The temperature in <x> is <y> degrees,” and sentences from site navigation menus. Additionally, many texts appear to be scraped from adult websites, containing low-quality, machine-translated samples. We have spotted numerous instances of incomplete or significantly incorrect translation pairs. Some target sentences were written in languages other than Ukrainian.

To control the quality of the sentences, we apply multiple heuristics. 





  
*Language filtering gcld3 library[<https://github.com/google/cld3>] provides language detection capabilities. We remove all sentences that failed to verify as Ukrainian or English.





  
*Perplexity thresholding We score source and target sentences using two decoder-only models trained on different monolingual datasets <cit.> and sum their bits per character measures.



  
*Translation mismatch filtering LaBSE <cit.> embeds sentences into a space, where similar sentences in different languages are close together. We use it to filter out badly aligned sentence pairs. 




  
*Length filtering The lengths of the original and translated sentences reveal examples that are too short or too long. Absolute differences of lengths point to pairs with long target for the short source and vice versa.


We arbitrarily choose joint values of filtering thresholds to get the desired approximate example counts: 1 million, 3 million and 8 million. We perform multiple experiments with these splits while searching for optimal hyperparameters. We list threshold values in <ref> and best results for each subset.




§ SECOND PHASE: UNSUPERVISED DATA SELECTION ON EXTENDED MULTI30K



We use the best checkpoint from the previous finetuning phase to train on a high-quality dataset: Extended Multi30K from <cit.>. Switching datasets gives us a performance boost of 1.97 BLEU. We additionally delete 11600 sentences from the dataset using unsupervised perplexity filtering pipeline gaining 0.35 on the dev set that translates to 0.3 BLEU on the devtest subset of FLORES.




We use perplexity as a data selection criterion to calculate thresholds to filter out highly surprising sentences.
We apply the k-fold cross-validation technique to make the perplexity evaluation in-domain. We split the training data into k=5 folds and train k models withholding one of the folds from each run. Then we score every sentence using the model that has not seen that sentence in training. Next, we sweep for acceptable threshold values by minimizing BLEU on the development set and report results in <ref>. We plot the distribution of scores in <ref>. We also provide threshold sweep results for training from base Mistral-7B-v0.1 checkpoint in <ref>. By comparing finetuned results, we demonstrate that data from the second phase alone is not enough to match the performance of our best checkpoint.






§ FEW-SHOT TRANSLATION


Conditioning the model on a sequence of demonstrations of performing some task allows the model to learn this task in-context, also known as “few-shot learning” (e.g. <cit.>), thanks to the ability of the Transformers to modulate representations of its future tokens using past context, implementing a specialized internal context-dependent learning algorithm inside its weights <cit.>.

While few-shot learning allows to quickly try any task with a low number of demonstrations,  <cit.> have shown that parameter-efficient finetuning allows smaller models achieve better performance, effectively spending less floating point operations per test example at inference time. 

Setting up the model for finetuning requires a lot of work, and in-context learning allows to quickly probe capability of a large model using inference software that performs efficient management of key-value cache for speed <cit.>.

To test backbone models before finetuning, we attempt decoding translations with a basic prompt shown in <ref>.



We find that the model significantly underperforms compared to current state of the art translation models when using beam search <cit.>.

This decoding algorithm performs pruned breadth-first expansion, scoring target sentence prefixes using model's own log probability, approximating maximum a-posteriori estimation of the best translation.

Inspection of the n-best list of translation candidates (beams) reveals that the models can produce high-quality translations, however assign low probabilities to them. We find the best possible translation by rescoring beams using the BLEU score as a loss function <cit.> with respect to the reference translation (the so-called “oracle”).

We employ this oracle rescoring strategy to gauge the potential capability of the model to produce good translations without finetuning, and find that in a regime of increased computation (large width of the beam) and assuming perfect selection capability, a base model is competitive with specialized alternatives. We sweep over a grid of multiple beam widths and report highest attainable BLEU scores in <ref>.



Consecutive sentences in FLORES are samples from the same document. We hypothesize, dynamically adjusting the prompt by inserting previous translations will improve results. We observe that the model indeed improves translation of certain words such as proper nouns through access to correct definitions provided in the context (<ref>), however its overall performance degrades in other examples.



We additionally attempt basic 0-shot with a system prompt , and 10-shot prompting using automatic prompt selection based on similarity between source sentences experiments with GPT-4 and GPT-4 Turbo and find that commerical systems perform similarly to other open source systems, as shown in <ref>.



§ DISCUSSION AND LIMITATIONS




  
*Single-sentence translation Our system is trained on demonstrations of standalone sentence pairs.




  
*Decoder-only models with long context windows We choose to finetune existing decoder-only models since the choice of models with almost the same architecture but different massive pretraining data is abundant. The number of open-source models released recently and their constant improvement offers a good prospective for the machine translation tasks.

These models receive gradient from all outputs during pretraining, and the self-attention mechanism can see the input, the partial output, and access past examples of translations in its context window using induction heads <cit.>.

For efficiency, we only train on examples with single short sentence pairs and do not pack context windows full of tokens as done in pretraining. In our early experiments, we find that our models still generalize to inputs longer that what is seen in training. This generalization behavior is often attributed to relative position embeddings <cit.>. We leave evaluation of long context attention stability under these conditions for future work.




  
*Training on the noisy dataset
Data cleaning has a positive effect on the resulting metrics. However, our models trained on 8 million filtered, examples perform worse than models trained on 3 million examples (<ref>).




  
*Tokenizer performance 

We used the LLaMA and Mistral tokenizers during our experiments, which use at least twice as many tokens to compress a sentence in Ukrainian of the same length as an English sentence in character. In practice, that means that generating a sentence in Ukrainian takes at least twice as many steps to generate. We show a distribution of sentence token lengths in <ref>. 






  
*Evaluation We choose BLEU-4 score <cit.> as our core evaluation metric and model selection criterion. BLEU-4 measures 4-gram precisions, where grams are defined as words. We use the implementation and rely on tokenization decisions of <cit.>. This metric is sensitive to minor  differences that do not affect the meaning of the sentence, for example case inflections that tend to cascade to multiple adjacent words. BLEU is known to poorly correlate with human judgement of translation quality, and <cit.> recommend learned metrics.

Choosing an appropriate learned metric for judgements of translation quality of Ukrainian requires careful consideration, and incorporation of data informed by the language community, such as a curated corpus of grammar corrections that reflects proper modern use of language <cit.>.

Regardless of limitations of BLEU, improvement in BLEU still signals improvement in translation quality in our regime.



  
*WMT22 Our reviewers have pointed out that WMT22 benchmark <cit.> includes a test set for Ukrainian. Our model achieves 24.72 on the WMT22 test set without any postprocessing, ranking behind the best result of <cit.> at 25.2 BLEU. We note that the submission that scores relatively low on the WMT22 test, scores comparably to our results on FLORES. These data distribution properties require closer exploration.




§ RELATED WORK




  
*Translation to Ukrainian <cit.> explore translation controllability by conditioning the model on text embeddings that encode style by finetuning an encoder-decoder model. They claim high quality translations on a private test set.






  
*Instruction-tuned language models <cit.> explore large-scale translation efforts to produce a multilingual instruction-tuned language model Aya. This work translates large datasets like the Flan Collection <cit.> using the NLLB-3B model  <cit.>.





  
*Translation systems 
<cit.> provide an iterated backtranslation recipe to bootstrap neural machine translation systems using generative models: zero-shot translation ability is used to produce candidates for few-shot demonstrations. Filtered few-shot demonstrations are used to sample new sentences for further finetuning for translation in two directions.




  
*Translation benchmarks Besides FLORES-101 (<cit.>, or FLORES-200 <cit.>, both include the same data for Ukrainian) dataset used in this work, <cit.> provides an additional dataset for multilingual evaluation.





  
*Data selection techniques
<cit.> propose a perplexity filtering pipeline, in which the data is split into k folds to classify low quality augmentation generations produced by surrogate language models. <cit.> provide recipes on curating data for language models by directly asking language models to score examples.  




§ CONCLUSION

In this work, we build a translation system using a two-phase data cleaning pipeline. We demonstrate matching performance to state-of-the-art encoder-decoder models for English-Ukrainian translation task. Notably, our system exhibits superior performance compared to the NLLB model, which was instrumental in generating the Aya dataset and contributed significantly to the advancement of multilingual language models. Improved machine translation could bring new capabilities to the next generation of large language models trained for the Ukrainian language. The recent improvements made for decoder-only backbones and the general dynamics of this process encourages us: we firmly believe that recipes we propose in this paper can be used to improve the quality of the translation by simply upgrading the backbone model.


§ CONTRIBUTIONS


Yurii Paniv worked on unsupervised data selection on extended Multi30K dataset, Dmytro Chaplynskyi performed initial training using heuristic filtering of Paracrawl, Nikita Trynus worked on evaluation, Volodymyr Kyrylov designed few-shot learning experiments and evaluation.




§ ACKNOWLEDGEMENTS

The authors would like to thank Talents for Ukraine project of Kyiv School of Economics for the grant on compute, used for this paper, Ukrainian Catholic University, Roman Kyslyi, and Oleksii Turuta for the fruitful discussions and inspiration. We would also like to thank paper reviewers for their comments.




§ REFERENCES

lrec-coling2024-natbib












§ APPENDIX




§ TRANSLATION EXAMPLES





  
Sample of top 5 worst examples by BLEU from FLORES devtest set
Source: While most of their food would be familiar to us, Romans did have their share of strange or unusual feast items, including wild boar, peacock, snails, and a type of rodent called a dormouse

Hypothesis: Хоча більшість їхніх продуктів харчування було б знайомим для нас, римляни мали свою частку дивних або незвичайних страв, включаючи кабана, павича, равликів та гризунів, званих куницями.

Reference: Переважно римляни харчувались тим, що знайоме і нам, але були у них і свої дивні та незвичні бенкетні частування, серед яких дикий кабан, павич, слимаки і гризун під назвою вовчок.

BLEU: 1.72



Source: "I was moved every time we did a rehearsal on this, from the bottom of my heart."

Hypothesis: "Я був зворушений кожного разу, коли ми робили репетицію з цього приводу, від щирого серця".

Reference: Мене це торкало кожен раз під час репетиції до глибини душі.

BLEU: 1.91



Source: Feral children may have experienced severe child abuse or trauma before being abandoned or running away.

Hypothesis: Дитина-дикун, можливо, пережила жорстоке насильство над дітьми або травму, перш ніж бути кинутою або втекти.

Reference: Може таке бути, що дикарські діти були об'єктами жорстокого поводження або отримали серйозні травми перед тим, як їх полишили або вони втекли.

BLEU: 2.31



Source: Workers must often get their superiors' approval for any decisions they make, and are expected to obey their superiors' instructions without question.

Hypothesis: Працівники часто повинні отримувати схвалення своїх начальників на будь-які рішення, які вони приймають, і повинні беззаперечно виконувати вказівки своїх начальників.

Reference: Працівникам часто потрібно отримати схвалення керівника щодо будь-яких рішень, а також від них очікують беззаперечного дотримання настанов керівників.

BLEU: 2.32



Source: Typically there will be a tuition fee to enroll in these educational programs.

Hypothesis: Як правило, буде плата за навчання, щоб зареєструватися в цих освітніх програмах.

Reference: Зазвичай такі освітні програми платні.

BLEU: 2.62



  
Sample of top 5 best examples by BLEU from FLORES devtest set
Source: The East African Islands are in the Indian Ocean off the eastern coast of Africa.

Hypothesis: Східноафриканські острови знаходяться в Індійському океані біля східного узбережжя Африки.

Reference: Східноафриканські острови знаходяться в Індійському океані біля східного узбережжя Африки.

BLEU: 100.00



Source: Earlier the Chinese news agency Xinhua reported a plane to be hijacked.

Hypothesis: Раніше китайське інформаційне агентство Сіньхуа повідомило про викрадення літака.

Reference: Раніше китайське інформаційне агентство Сіньхуа повідомило про викрадення літака.

BLEU: 100.00



Source: For instance, they didn't have corn, nor tomatoes, nor potatoes, nor cocoa, and no ancient Roman ever tasted a turkey.

Hypothesis: Наприклад, у них не було ні кукурудзи, ні помідорів, ні картоплі, ні какао, і жоден стародавній римлянин ніколи не скуштував індичку.

Reference: Наприклад, у них не було ні кукурудзи, ні помідорів, ні картоплі, ні какао, і жоден стародавній римлянин ніколи не куштував індичку.

BLEU: 90.95



Source: The luminosity and rotation are used together to determine a star's Rossby number, which is related to plasma flow.

Hypothesis: Світність і обертання використовуються разом для визначення числа Россбі зірки, яке пов'язане з потоком плазми.

Reference: Світність і обертання використовуються разом для визначення числа Россбі зірки, яке пов'язане з плазмовим потоком.

BLEU: 83.26



Source: But being placed in the "high tropics" just a few degrees north of equator you will need to deal with both heat (always) and strong sun (when the sky is clear, more rarely).

Hypothesis: Але перебуваючи в "високих тропіках" всього в декількох градусах на північ від екватора, вам доведеться мати справу як з спекою (завжди), так і з сильним сонцем (коли небо чисте, рідше).

Reference: Але, перебуваючи в "високих тропіках" всього в декількох градусах на північ від екватора, вам доведеться мати справу як зі спекою (завжди), так і з палючим сонцем (коли небо чисте, рідше).

BLEU: 82.47




