
# Zabanshenas - Language Detector

Zabanshenas is a Transformer-based solution for identifying the most likely language of a written document/text. Zabanshenas is a Persian word that has two meanings:

- A person who studies linguistics.
- A way to identify the type of written language.


## How to use

Follow [Zabanshenas repo](https://github.com/m3hrdadfi/zabanshenas) for more information!


## Evaluation
The following tables summarize the scores obtained by model overall and per each class.


### By Paragraph

|                language                | precision |  recall  | f1-score |
|:--------------------------------------:|:---------:|:--------:|:--------:|
|             Achinese (ace)             |  1.000000 | 0.982143 | 0.990991 |
|             Afrikaans (afr)            |  1.000000 | 1.000000 | 1.000000 |
|         Alemannic German (als)         |  1.000000 | 0.946429 | 0.972477 |
|              Amharic (amh)             |  1.000000 | 0.982143 | 0.990991 |
|            Old English (ang)           |  0.981818 | 0.964286 | 0.972973 |
|              Arabic (ara)              |  0.846154 | 0.982143 | 0.909091 |
|             Aragonese (arg)            |  1.000000 | 1.000000 | 1.000000 |
|          Egyptian Arabic (arz)         |  0.979592 | 0.857143 | 0.914286 |
|             Assamese (asm)             |  0.981818 | 0.964286 | 0.972973 |
|             Asturian (ast)             |  0.964912 | 0.982143 | 0.973451 |
|               Avar (ava)               |  0.941176 | 0.905660 | 0.923077 |
|              Aymara (aym)              |  0.964912 | 0.982143 | 0.973451 |
|         South Azerbaijani (azb)        |  0.965517 | 1.000000 | 0.982456 |
|            Azerbaijani (aze)           |  1.000000 | 1.000000 | 1.000000 |
|              Bashkir (bak)             |  1.000000 | 0.978261 | 0.989011 |
|             Bavarian (bar)             |  0.843750 | 0.964286 | 0.900000 |
|           Central Bikol (bcl)          |  1.000000 | 0.982143 | 0.990991 |
| Belarusian (Taraschkewiza) (be-tarask) |  1.000000 | 0.875000 | 0.933333 |
|            Belarusian (bel)            |  0.870968 | 0.964286 | 0.915254 |
|              Bengali (ben)             |  0.982143 | 0.982143 | 0.982143 |
|             Bhojpuri (bho)             |  1.000000 | 0.928571 | 0.962963 |
|              Banjar (bjn)              |  0.981132 | 0.945455 | 0.962963 |
|              Tibetan (bod)             |  1.000000 | 0.982143 | 0.990991 |
|              Bosnian (bos)             |  0.552632 | 0.375000 | 0.446809 |
|            Bishnupriya (bpy)           |  1.000000 | 0.982143 | 0.990991 |
|              Breton (bre)              |  1.000000 | 0.964286 | 0.981818 |
|             Bulgarian (bul)            |  1.000000 | 0.964286 | 0.981818 |
|              Buryat (bxr)              |  0.946429 | 0.946429 | 0.946429 |
|              Catalan (cat)             |  0.982143 | 0.982143 | 0.982143 |
|             Chavacano (cbk)            |  0.914894 | 0.767857 | 0.834951 |
|             Min Dong (cdo)             |  1.000000 | 0.982143 | 0.990991 |
|              Cebuano (ceb)             |  1.000000 | 1.000000 | 1.000000 |
|               Czech (ces)              |  1.000000 | 1.000000 | 1.000000 |
|              Chechen (che)             |  1.000000 | 1.000000 | 1.000000 |
|             Cherokee (chr)             |  1.000000 | 0.963636 | 0.981481 |
|              Chuvash (chv)             |  0.938776 | 0.958333 | 0.948454 |
|          Central Kurdish (ckb)         |  1.000000 | 1.000000 | 1.000000 |
|              Cornish (cor)             |  1.000000 | 1.000000 | 1.000000 |
|             Corsican (cos)             |  1.000000 | 0.982143 | 0.990991 |
|           Crimean Tatar (crh)          |  1.000000 | 0.946429 | 0.972477 |
|             Kashubian (csb)            |  1.000000 | 0.963636 | 0.981481 |
|               Welsh (cym)              |  1.000000 | 1.000000 | 1.000000 |
|              Danish (dan)              |  1.000000 | 1.000000 | 1.000000 |
|              German (deu)              |  0.828125 | 0.946429 | 0.883333 |
|               Dimli (diq)              |  0.964912 | 0.982143 | 0.973451 |
|              Dhivehi (div)             |  1.000000 | 1.000000 | 1.000000 |
|           Lower Sorbian (dsb)          |  1.000000 | 0.982143 | 0.990991 |
|              Doteli (dty)              |  0.940000 | 0.854545 | 0.895238 |
|              Emilian (egl)             |  1.000000 | 0.928571 | 0.962963 |
|           Modern Greek (ell)           |  1.000000 | 1.000000 | 1.000000 |
|              English (eng)             |  0.588889 | 0.946429 | 0.726027 |
|             Esperanto (epo)            |  1.000000 | 0.982143 | 0.990991 |
|             Estonian (est)             |  0.963636 | 0.946429 | 0.954955 |
|              Basque (eus)              |  1.000000 | 0.982143 | 0.990991 |
|           Extremaduran (ext)           |  0.982143 | 0.982143 | 0.982143 |
|              Faroese (fao)             |  1.000000 | 1.000000 | 1.000000 |
|              Persian (fas)             |  0.948276 | 0.982143 | 0.964912 |
|              Finnish (fin)             |  1.000000 | 1.000000 | 1.000000 |
|              French (fra)              |  0.710145 | 0.875000 | 0.784000 |
|              Arpitan (frp)             |  1.000000 | 0.946429 | 0.972477 |
|          Western Frisian (fry)         |  0.982143 | 0.982143 | 0.982143 |
|             Friulian (fur)             |  1.000000 | 0.982143 | 0.990991 |
|              Gagauz (gag)              |  0.981132 | 0.945455 | 0.962963 |
|          Scottish Gaelic (gla)         |  0.982143 | 0.982143 | 0.982143 |
|               Irish (gle)              |  0.949153 | 1.000000 | 0.973913 |
|             Galician (glg)             |  1.000000 | 1.000000 | 1.000000 |
|              Gilaki (glk)              |  0.981132 | 0.945455 | 0.962963 |
|               Manx (glv)               |  1.000000 | 1.000000 | 1.000000 |
|              Guarani (grn)             |  1.000000 | 0.964286 | 0.981818 |
|             Gujarati (guj)             |  1.000000 | 0.982143 | 0.990991 |
|           Hakka Chinese (hak)          |  0.981818 | 0.964286 | 0.972973 |
|          Haitian Creole (hat)          |  1.000000 | 1.000000 | 1.000000 |
|               Hausa (hau)              |  1.000000 | 0.945455 | 0.971963 |
|          Serbo-Croatian (hbs)          |  0.448276 | 0.464286 | 0.456140 |
|              Hebrew (heb)              |  1.000000 | 0.982143 | 0.990991 |
|            Fiji Hindi (hif)            |  0.890909 | 0.890909 | 0.890909 |
|               Hindi (hin)              |  0.981481 | 0.946429 | 0.963636 |
|             Croatian (hrv)             |  0.500000 | 0.636364 | 0.560000 |
|           Upper Sorbian (hsb)          |  0.955556 | 1.000000 | 0.977273 |
|             Hungarian (hun)            |  1.000000 | 1.000000 | 1.000000 |
|             Armenian (hye)             |  1.000000 | 0.981818 | 0.990826 |
|               Igbo (ibo)               |  0.918033 | 1.000000 | 0.957265 |
|                Ido (ido)               |  1.000000 | 1.000000 | 1.000000 |
|            Interlingue (ile)           |  1.000000 | 0.962264 | 0.980769 |
|               Iloko (ilo)              |  0.947368 | 0.964286 | 0.955752 |
|            Interlingua (ina)           |  1.000000 | 1.000000 | 1.000000 |
|            Indonesian (ind)            |  0.761905 | 0.872727 | 0.813559 |
|             Icelandic (isl)            |  1.000000 | 1.000000 | 1.000000 |
|              Italian (ita)             |  0.861538 | 1.000000 | 0.925620 |
|          Jamaican Patois (jam)         |  1.000000 | 0.946429 | 0.972477 |
|             Javanese (jav)             |  0.964912 | 0.982143 | 0.973451 |
|              Lojban (jbo)              |  1.000000 | 1.000000 | 1.000000 |
|             Japanese (jpn)             |  1.000000 | 1.000000 | 1.000000 |
|            Karakalpak (kaa)            |  0.965517 | 1.000000 | 0.982456 |
|              Kabyle (kab)              |  1.000000 | 0.964286 | 0.981818 |
|              Kannada (kan)             |  0.982143 | 0.982143 | 0.982143 |
|             Georgian (kat)             |  1.000000 | 0.964286 | 0.981818 |
|              Kazakh (kaz)              |  0.980769 | 0.980769 | 0.980769 |
|             Kabardian (kbd)            |  1.000000 | 0.982143 | 0.990991 |
|           Central Khmer (khm)          |  0.960784 | 0.875000 | 0.915888 |
|            Kinyarwanda (kin)           |  0.981132 | 0.928571 | 0.954128 |
|              Kirghiz (kir)             |  1.000000 | 1.000000 | 1.000000 |
|           Komi-Permyak (koi)           |  0.962264 | 0.910714 | 0.935780 |
|              Konkani (kok)             |  0.964286 | 0.981818 | 0.972973 |
|               Komi (kom)               |  1.000000 | 0.962264 | 0.980769 |
|              Korean (kor)              |  1.000000 | 1.000000 | 1.000000 |
|          Karachay-Balkar (krc)         |  1.000000 | 0.982143 | 0.990991 |
|            Ripuarisch (ksh)            |  1.000000 | 0.964286 | 0.981818 |
|              Kurdish (kur)             |  1.000000 | 0.964286 | 0.981818 |
|              Ladino (lad)              |  1.000000 | 1.000000 | 1.000000 |
|                Lao (lao)               |  0.961538 | 0.909091 | 0.934579 |
|               Latin (lat)              |  0.877193 | 0.943396 | 0.909091 |
|              Latvian (lav)             |  0.963636 | 0.946429 | 0.954955 |
|             Lezghian (lez)             |  1.000000 | 0.964286 | 0.981818 |
|             Ligurian (lij)             |  1.000000 | 0.964286 | 0.981818 |
|             Limburgan (lim)            |  0.938776 | 1.000000 | 0.968421 |
|              Lingala (lin)             |  0.980769 | 0.927273 | 0.953271 |
|            Lithuanian (lit)            |  0.982456 | 1.000000 | 0.991150 |
|              Lombard (lmo)             |  1.000000 | 1.000000 | 1.000000 |
|           Northern Luri (lrc)          |  1.000000 | 0.928571 | 0.962963 |
|             Latgalian (ltg)            |  1.000000 | 0.982143 | 0.990991 |
|           Luxembourgish (ltz)          |  0.949153 | 1.000000 | 0.973913 |
|              Luganda (lug)             |  1.000000 | 1.000000 | 1.000000 |
|         Literary Chinese (lzh)         |  1.000000 | 1.000000 | 1.000000 |
|             Maithili (mai)             |  0.931034 | 0.964286 | 0.947368 |
|             Malayalam (mal)            |  1.000000 | 0.982143 | 0.990991 |
|          Banyumasan (map-bms)          |  0.977778 | 0.785714 | 0.871287 |
|              Marathi (mar)             |  0.949153 | 1.000000 | 0.973913 |
|              Moksha (mdf)              |  0.980000 | 0.890909 | 0.933333 |
|           Eastern Mari (mhr)           |  0.981818 | 0.964286 | 0.972973 |
|            Minangkabau (min)           |  1.000000 | 1.000000 | 1.000000 |
|            Macedonian (mkd)            |  1.000000 | 0.981818 | 0.990826 |
|             Malagasy (mlg)             |  0.981132 | 1.000000 | 0.990476 |
|              Maltese (mlt)             |  0.982456 | 1.000000 | 0.991150 |
|          Min Nan Chinese (nan)         |  1.000000 | 1.000000 | 1.000000 |
|             Mongolian (mon)            |  1.000000 | 0.981818 | 0.990826 |
|               Maori (mri)              |  1.000000 | 1.000000 | 1.000000 |
|           Western Mari (mrj)           |  0.982456 | 1.000000 | 0.991150 |
|               Malay (msa)              |  0.862069 | 0.892857 | 0.877193 |
|             Mirandese (mwl)            |  1.000000 | 0.982143 | 0.990991 |
|              Burmese (mya)             |  1.000000 | 1.000000 | 1.000000 |
|               Erzya (myv)              |  0.818182 | 0.964286 | 0.885246 |
|            Mazanderani (mzn)           |  0.981481 | 1.000000 | 0.990654 |
|            Neapolitan (nap)            |  1.000000 | 0.981818 | 0.990826 |
|              Navajo (nav)              |  1.000000 | 1.000000 | 1.000000 |
|         Classical Nahuatl (nci)        |  0.981481 | 0.946429 | 0.963636 |
|            Low German (nds)            |  0.982143 | 0.982143 | 0.982143 |
|        West Low German (nds-nl)        |  1.000000 | 1.000000 | 1.000000 |
|      Nepali (macrolanguage) (nep)      |  0.881356 | 0.928571 | 0.904348 |
|              Newari (new)              |  1.000000 | 0.909091 | 0.952381 |
|               Dutch (nld)              |  0.982143 | 0.982143 | 0.982143 |
|         Norwegian Nynorsk (nno)        |  1.000000 | 1.000000 | 1.000000 |
|              Bokmål (nob)              |  1.000000 | 1.000000 | 1.000000 |
|               Narom (nrm)              |  0.981818 | 0.964286 | 0.972973 |
|          Northern Sotho (nso)          |  1.000000 | 1.000000 | 1.000000 |
|              Occitan (oci)             |  0.903846 | 0.839286 | 0.870370 |
|          Livvi-Karelian (olo)          |  0.982456 | 1.000000 | 0.991150 |
|               Oriya (ori)              |  0.964912 | 0.982143 | 0.973451 |
|               Oromo (orm)              |  0.982143 | 0.982143 | 0.982143 |
|             Ossetian (oss)             |  0.982143 | 1.000000 | 0.990991 |
|            Pangasinan (pag)            |  0.980000 | 0.875000 | 0.924528 |
|             Pampanga (pam)             |  0.928571 | 0.896552 | 0.912281 |
|              Panjabi (pan)             |  1.000000 | 1.000000 | 1.000000 |
|            Papiamento (pap)            |  1.000000 | 0.964286 | 0.981818 |
|              Picard (pcd)              |  0.849057 | 0.849057 | 0.849057 |
|        Pennsylvania German (pdc)       |  0.854839 | 0.946429 | 0.898305 |
|          Palatine German (pfl)         |  0.946429 | 0.946429 | 0.946429 |
|          Western Panjabi (pnb)         |  0.981132 | 0.962963 | 0.971963 |
|              Polish (pol)              |  0.933333 | 1.000000 | 0.965517 |
|            Portuguese (por)            |  0.774648 | 0.982143 | 0.866142 |
|              Pushto (pus)              |  1.000000 | 0.910714 | 0.953271 |
|              Quechua (que)             |  0.962963 | 0.928571 | 0.945455 |
|      Tarantino dialect (roa-tara)      |  1.000000 | 0.964286 | 0.981818 |
|              Romansh (roh)             |  1.000000 | 0.928571 | 0.962963 |
|             Romanian (ron)             |  0.965517 | 1.000000 | 0.982456 |
|               Rusyn (rue)              |  0.946429 | 0.946429 | 0.946429 |
|             Aromanian (rup)            |  0.962963 | 0.928571 | 0.945455 |
|              Russian (rus)             |  0.859375 | 0.982143 | 0.916667 |
|               Yakut (sah)              |  1.000000 | 0.982143 | 0.990991 |
|             Sanskrit (san)             |  0.982143 | 0.982143 | 0.982143 |
|             Sicilian (scn)             |  1.000000 | 1.000000 | 1.000000 |
|               Scots (sco)              |  0.982143 | 0.982143 | 0.982143 |
|            Samogitian (sgs)            |  1.000000 | 0.982143 | 0.990991 |
|              Sinhala (sin)             |  0.964912 | 0.982143 | 0.973451 |
|              Slovak (slk)              |  1.000000 | 0.982143 | 0.990991 |
|              Slovene (slv)             |  1.000000 | 0.981818 | 0.990826 |
|           Northern Sami (sme)          |  0.962264 | 0.962264 | 0.962264 |
|               Shona (sna)              |  0.933333 | 1.000000 | 0.965517 |
|              Sindhi (snd)              |  1.000000 | 1.000000 | 1.000000 |
|              Somali (som)              |  0.948276 | 1.000000 | 0.973451 |
|              Spanish (spa)             |  0.739130 | 0.910714 | 0.816000 |
|             Albanian (sqi)             |  0.982143 | 0.982143 | 0.982143 |
|             Sardinian (srd)            |  1.000000 | 0.982143 | 0.990991 |
|              Sranan (srn)              |  1.000000 | 1.000000 | 1.000000 |
|              Serbian (srp)             |  1.000000 | 0.946429 | 0.972477 |
|          Saterfriesisch (stq)          |  1.000000 | 0.964286 | 0.981818 |
|             Sundanese (sun)            |  1.000000 | 0.977273 | 0.988506 |
|      Swahili (macrolanguage) (swa)     |  1.000000 | 1.000000 | 1.000000 |
|              Swedish (swe)             |  1.000000 | 1.000000 | 1.000000 |
|             Silesian (szl)             |  1.000000 | 0.981481 | 0.990654 |
|               Tamil (tam)              |  0.982143 | 1.000000 | 0.990991 |
|               Tatar (tat)              |  1.000000 | 1.000000 | 1.000000 |
|               Tulu (tcy)               |  0.982456 | 1.000000 | 0.991150 |
|              Telugu (tel)              |  1.000000 | 0.920000 | 0.958333 |
|               Tetum (tet)              |  1.000000 | 0.964286 | 0.981818 |
|               Tajik (tgk)              |  1.000000 | 1.000000 | 1.000000 |
|              Tagalog (tgl)             |  1.000000 | 1.000000 | 1.000000 |
|               Thai (tha)               |  0.932203 | 0.982143 | 0.956522 |
|              Tongan (ton)              |  1.000000 | 0.964286 | 0.981818 |
|              Tswana (tsn)              |  1.000000 | 1.000000 | 1.000000 |
|              Turkmen (tuk)             |  1.000000 | 0.982143 | 0.990991 |
|              Turkish (tur)             |  0.901639 | 0.982143 | 0.940171 |
|               Tuvan (tyv)              |  1.000000 | 0.964286 | 0.981818 |
|              Udmurt (udm)              |  1.000000 | 0.982143 | 0.990991 |
|              Uighur (uig)              |  1.000000 | 0.982143 | 0.990991 |
|             Ukrainian (ukr)            |  0.963636 | 0.946429 | 0.954955 |
|               Urdu (urd)               |  1.000000 | 0.982143 | 0.990991 |
|               Uzbek (uzb)              |  1.000000 | 1.000000 | 1.000000 |
|             Venetian (vec)             |  1.000000 | 0.982143 | 0.990991 |
|               Veps (vep)               |  0.982456 | 1.000000 | 0.991150 |
|            Vietnamese (vie)            |  0.964912 | 0.982143 | 0.973451 |
|              Vlaams (vls)              |  1.000000 | 0.982143 | 0.990991 |
|              Volapük (vol)             |  1.000000 | 1.000000 | 1.000000 |
|               Võro (vro)               |  0.964286 | 0.964286 | 0.964286 |
|               Waray (war)              |  1.000000 | 0.982143 | 0.990991 |
|              Walloon (wln)             |  1.000000 | 1.000000 | 1.000000 |
|               Wolof (wol)              |  0.981481 | 0.963636 | 0.972477 |
|            Wu Chinese (wuu)            |  0.981481 | 0.946429 | 0.963636 |
|               Xhosa (xho)              |  1.000000 | 0.964286 | 0.981818 |
|            Mingrelian (xmf)            |  1.000000 | 0.964286 | 0.981818 |
|              Yiddish (yid)             |  1.000000 | 1.000000 | 1.000000 |
|              Yoruba (yor)              |  0.964912 | 0.982143 | 0.973451 |
|              Zeeuws (zea)              |  1.000000 | 0.982143 | 0.990991 |
|           Cantonese (zh-yue)           |  0.981481 | 0.946429 | 0.963636 |
|         Standard Chinese (zho)         |  0.932203 | 0.982143 | 0.956522 |
|                accuracy                |  0.963055 | 0.963055 | 0.963055 |
|                macro avg               |  0.966424 | 0.963216 | 0.963891 |
|              weighted avg              |  0.966040 | 0.963055 | 0.963606 |

### By Sentence

|                language                | precision |  recall  | f1-score |
|:--------------------------------------:|:---------:|:--------:|:--------:|
|             Achinese (ace)             |  0.754545 | 0.873684 | 0.809756 |
|             Afrikaans (afr)            |  0.708955 | 0.940594 | 0.808511 |
|         Alemannic German (als)         |  0.870130 | 0.752809 | 0.807229 |
|              Amharic (amh)             |  1.000000 | 0.820000 | 0.901099 |
|            Old English (ang)           |  0.966667 | 0.906250 | 0.935484 |
|              Arabic (ara)              |  0.907692 | 0.967213 | 0.936508 |
|             Aragonese (arg)            |  0.921569 | 0.959184 | 0.940000 |
|          Egyptian Arabic (arz)         |  0.964286 | 0.843750 | 0.900000 |
|             Assamese (asm)             |  0.964286 | 0.870968 | 0.915254 |
|             Asturian (ast)             |  0.880000 | 0.795181 | 0.835443 |
|               Avar (ava)               |  0.864198 | 0.843373 | 0.853659 |
|              Aymara (aym)              |  1.000000 | 0.901961 | 0.948454 |
|         South Azerbaijani (azb)        |  0.979381 | 0.989583 | 0.984456 |
|            Azerbaijani (aze)           |  0.989899 | 0.960784 | 0.975124 |
|              Bashkir (bak)             |  0.837209 | 0.857143 | 0.847059 |
|             Bavarian (bar)             |  0.741935 | 0.766667 | 0.754098 |
|           Central Bikol (bcl)          |  0.962963 | 0.928571 | 0.945455 |
| Belarusian (Taraschkewiza) (be-tarask) |  0.857143 | 0.733333 | 0.790419 |
|            Belarusian (bel)            |  0.775510 | 0.752475 | 0.763819 |
|              Bengali (ben)             |  0.861111 | 0.911765 | 0.885714 |
|             Bhojpuri (bho)             |  0.965517 | 0.933333 | 0.949153 |
|              Banjar (bjn)              |  0.891566 | 0.880952 | 0.886228 |
|              Tibetan (bod)             |  1.000000 | 1.000000 | 1.000000 |
|              Bosnian (bos)             |  0.375000 | 0.323077 | 0.347107 |
|            Bishnupriya (bpy)           |  0.986301 | 1.000000 | 0.993103 |
|              Breton (bre)              |  0.951613 | 0.893939 | 0.921875 |
|             Bulgarian (bul)            |  0.945055 | 0.877551 | 0.910053 |
|              Buryat (bxr)              |  0.955556 | 0.843137 | 0.895833 |
|              Catalan (cat)             |  0.692308 | 0.750000 | 0.720000 |
|             Chavacano (cbk)            |  0.842857 | 0.641304 | 0.728395 |
|             Min Dong (cdo)             |  0.972973 | 1.000000 | 0.986301 |
|              Cebuano (ceb)             |  0.981308 | 0.954545 | 0.967742 |
|               Czech (ces)              |  0.944444 | 0.915385 | 0.929687 |
|              Chechen (che)             |  0.875000 | 0.700000 | 0.777778 |
|             Cherokee (chr)             |  1.000000 | 0.970588 | 0.985075 |
|              Chuvash (chv)             |  0.875000 | 0.836957 | 0.855556 |
|          Central Kurdish (ckb)         |  1.000000 | 0.983051 | 0.991453 |
|              Cornish (cor)             |  0.979592 | 0.969697 | 0.974619 |
|             Corsican (cos)             |  0.986842 | 0.925926 | 0.955414 |
|           Crimean Tatar (crh)          |  0.958333 | 0.907895 | 0.932432 |
|             Kashubian (csb)            |  0.920354 | 0.904348 | 0.912281 |
|               Welsh (cym)              |  0.971014 | 0.943662 | 0.957143 |
|              Danish (dan)              |  0.865169 | 0.777778 | 0.819149 |
|              German (deu)              |  0.721311 | 0.822430 | 0.768559 |
|               Dimli (diq)              |  0.915966 | 0.923729 | 0.919831 |
|              Dhivehi (div)             |  1.000000 | 0.991228 | 0.995595 |
|           Lower Sorbian (dsb)          |  0.898876 | 0.879121 | 0.888889 |
|              Doteli (dty)              |  0.821429 | 0.638889 | 0.718750 |
|              Emilian (egl)             |  0.988095 | 0.922222 | 0.954023 |
|           Modern Greek (ell)           |  0.988636 | 0.966667 | 0.977528 |
|              English (eng)             |  0.522727 | 0.784091 | 0.627273 |
|             Esperanto (epo)            |  0.963855 | 0.930233 | 0.946746 |
|             Estonian (est)             |  0.922222 | 0.873684 | 0.897297 |
|              Basque (eus)              |  1.000000 | 0.941176 | 0.969697 |
|           Extremaduran (ext)           |  0.925373 | 0.885714 | 0.905109 |
|              Faroese (fao)             |  0.855072 | 0.887218 | 0.870849 |
|              Persian (fas)             |  0.879630 | 0.979381 | 0.926829 |
|              Finnish (fin)             |  0.952830 | 0.943925 | 0.948357 |
|              French (fra)              |  0.676768 | 0.943662 | 0.788235 |
|              Arpitan (frp)             |  0.867925 | 0.807018 | 0.836364 |
|          Western Frisian (fry)         |  0.956989 | 0.890000 | 0.922280 |
|             Friulian (fur)             |  1.000000 | 0.857143 | 0.923077 |
|              Gagauz (gag)              |  0.939024 | 0.802083 | 0.865169 |
|          Scottish Gaelic (gla)         |  1.000000 | 0.879121 | 0.935673 |
|               Irish (gle)              |  0.989247 | 0.958333 | 0.973545 |
|             Galician (glg)             |  0.910256 | 0.922078 | 0.916129 |
|              Gilaki (glk)              |  0.964706 | 0.872340 | 0.916201 |
|               Manx (glv)               |  1.000000 | 0.965517 | 0.982456 |
|              Guarani (grn)             |  0.983333 | 1.000000 | 0.991597 |
|             Gujarati (guj)             |  1.000000 | 0.991525 | 0.995745 |
|           Hakka Chinese (hak)          |  0.955224 | 0.955224 | 0.955224 |
|          Haitian Creole (hat)          |  0.833333 | 0.666667 | 0.740741 |
|               Hausa (hau)              |  0.936709 | 0.913580 | 0.925000 |
|          Serbo-Croatian (hbs)          |  0.452830 | 0.410256 | 0.430493 |
|              Hebrew (heb)              |  0.988235 | 0.976744 | 0.982456 |
|            Fiji Hindi (hif)            |  0.936709 | 0.840909 | 0.886228 |
|               Hindi (hin)              |  0.965517 | 0.756757 | 0.848485 |
|             Croatian (hrv)             |  0.443820 | 0.537415 | 0.486154 |
|           Upper Sorbian (hsb)          |  0.951613 | 0.830986 | 0.887218 |
|             Hungarian (hun)            |  0.854701 | 0.909091 | 0.881057 |
|             Armenian (hye)             |  1.000000 | 0.816327 | 0.898876 |
|               Igbo (ibo)               |  0.974359 | 0.926829 | 0.950000 |
|                Ido (ido)               |  0.975000 | 0.987342 | 0.981132 |
|            Interlingue (ile)           |  0.880597 | 0.921875 | 0.900763 |
|               Iloko (ilo)              |  0.882353 | 0.821918 | 0.851064 |
|            Interlingua (ina)           |  0.952381 | 0.895522 | 0.923077 |
|            Indonesian (ind)            |  0.606383 | 0.695122 | 0.647727 |
|             Icelandic (isl)            |  0.978261 | 0.882353 | 0.927835 |
|              Italian (ita)             |  0.910448 | 0.910448 | 0.910448 |
|          Jamaican Patois (jam)         |  0.988764 | 0.967033 | 0.977778 |
|             Javanese (jav)             |  0.903614 | 0.862069 | 0.882353 |
|              Lojban (jbo)              |  0.943878 | 0.929648 | 0.936709 |
|             Japanese (jpn)             |  1.000000 | 0.764706 | 0.866667 |
|            Karakalpak (kaa)            |  0.940171 | 0.901639 | 0.920502 |
|              Kabyle (kab)              |  0.985294 | 0.837500 | 0.905405 |
|              Kannada (kan)             |  0.975806 | 0.975806 | 0.975806 |
|             Georgian (kat)             |  0.953704 | 0.903509 | 0.927928 |
|              Kazakh (kaz)              |  0.934579 | 0.877193 | 0.904977 |
|             Kabardian (kbd)            |  0.987952 | 0.953488 | 0.970414 |
|           Central Khmer (khm)          |  0.928571 | 0.829787 | 0.876404 |
|            Kinyarwanda (kin)           |  0.953125 | 0.938462 | 0.945736 |
|              Kirghiz (kir)             |  0.927632 | 0.881250 | 0.903846 |
|           Komi-Permyak (koi)           |  0.750000 | 0.776786 | 0.763158 |
|              Konkani (kok)             |  0.893491 | 0.872832 | 0.883041 |
|               Komi (kom)               |  0.734177 | 0.690476 | 0.711656 |
|              Korean (kor)              |  0.989899 | 0.989899 | 0.989899 |
|          Karachay-Balkar (krc)         |  0.928571 | 0.917647 | 0.923077 |
|            Ripuarisch (ksh)            |  0.915789 | 0.896907 | 0.906250 |
|              Kurdish (kur)             |  0.977528 | 0.935484 | 0.956044 |
|              Ladino (lad)              |  0.985075 | 0.904110 | 0.942857 |
|                Lao (lao)               |  0.896552 | 0.812500 | 0.852459 |
|               Latin (lat)              |  0.741935 | 0.831325 | 0.784091 |
|              Latvian (lav)             |  0.710526 | 0.878049 | 0.785455 |
|             Lezghian (lez)             |  0.975309 | 0.877778 | 0.923977 |
|             Ligurian (lij)             |  0.951807 | 0.897727 | 0.923977 |
|             Limburgan (lim)            |  0.909091 | 0.921053 | 0.915033 |
|              Lingala (lin)             |  0.942857 | 0.814815 | 0.874172 |
|            Lithuanian (lit)            |  0.892857 | 0.925926 | 0.909091 |
|              Lombard (lmo)             |  0.766234 | 0.951613 | 0.848921 |
|           Northern Luri (lrc)          |  0.972222 | 0.875000 | 0.921053 |
|             Latgalian (ltg)            |  0.895349 | 0.865169 | 0.880000 |
|           Luxembourgish (ltz)          |  0.882353 | 0.750000 | 0.810811 |
|              Luganda (lug)             |  0.946429 | 0.883333 | 0.913793 |
|         Literary Chinese (lzh)         |  1.000000 | 1.000000 | 1.000000 |
|             Maithili (mai)             |  0.893617 | 0.823529 | 0.857143 |
|             Malayalam (mal)            |  1.000000 | 0.975000 | 0.987342 |
|          Banyumasan (map-bms)          |  0.924242 | 0.772152 | 0.841379 |
|              Marathi (mar)             |  0.874126 | 0.919118 | 0.896057 |
|              Moksha (mdf)              |  0.771242 | 0.830986 | 0.800000 |
|           Eastern Mari (mhr)           |  0.820000 | 0.860140 | 0.839590 |
|            Minangkabau (min)           |  0.973684 | 0.973684 | 0.973684 |
|            Macedonian (mkd)            |  0.895652 | 0.953704 | 0.923767 |
|             Malagasy (mlg)             |  1.000000 | 0.966102 | 0.982759 |
|              Maltese (mlt)             |  0.987952 | 0.964706 | 0.976190 |
|          Min Nan Chinese (nan)         |  0.975000 | 1.000000 | 0.987342 |
|             Mongolian (mon)            |  0.954545 | 0.933333 | 0.943820 |
|               Maori (mri)              |  0.985294 | 1.000000 | 0.992593 |
|           Western Mari (mrj)           |  0.966292 | 0.914894 | 0.939891 |
|               Malay (msa)              |  0.770270 | 0.695122 | 0.730769 |
|             Mirandese (mwl)            |  0.970588 | 0.891892 | 0.929577 |
|              Burmese (mya)             |  1.000000 | 0.964286 | 0.981818 |
|               Erzya (myv)              |  0.535714 | 0.681818 | 0.600000 |
|            Mazanderani (mzn)           |  0.968750 | 0.898551 | 0.932331 |
|            Neapolitan (nap)            |  0.892308 | 0.865672 | 0.878788 |
|              Navajo (nav)              |  0.984375 | 0.984375 | 0.984375 |
|         Classical Nahuatl (nci)        |  0.901408 | 0.761905 | 0.825806 |
|            Low German (nds)            |  0.896226 | 0.913462 | 0.904762 |
|        West Low German (nds-nl)        |  0.873563 | 0.835165 | 0.853933 |
|      Nepali (macrolanguage) (nep)      |  0.704545 | 0.861111 | 0.775000 |
|              Newari (new)              |  0.920000 | 0.741935 | 0.821429 |
|               Dutch (nld)              |  0.925926 | 0.872093 | 0.898204 |
|         Norwegian Nynorsk (nno)        |  0.847059 | 0.808989 | 0.827586 |
|              Bokmål (nob)              |  0.861386 | 0.852941 | 0.857143 |
|               Narom (nrm)              |  0.966667 | 0.983051 | 0.974790 |
|          Northern Sotho (nso)          |  0.897436 | 0.921053 | 0.909091 |
|              Occitan (oci)             |  0.958333 | 0.696970 | 0.807018 |
|          Livvi-Karelian (olo)          |  0.967742 | 0.937500 | 0.952381 |
|               Oriya (ori)              |  0.933333 | 1.000000 | 0.965517 |
|               Oromo (orm)              |  0.977528 | 0.915789 | 0.945652 |
|             Ossetian (oss)             |  0.958333 | 0.841463 | 0.896104 |
|            Pangasinan (pag)            |  0.847328 | 0.909836 | 0.877470 |
|             Pampanga (pam)             |  0.969697 | 0.780488 | 0.864865 |
|              Panjabi (pan)             |  1.000000 | 1.000000 | 1.000000 |
|            Papiamento (pap)            |  0.876190 | 0.920000 | 0.897561 |
|              Picard (pcd)              |  0.707317 | 0.568627 | 0.630435 |
|        Pennsylvania German (pdc)       |  0.827273 | 0.827273 | 0.827273 |
|          Palatine German (pfl)         |  0.882353 | 0.914634 | 0.898204 |
|          Western Panjabi (pnb)         |  0.964286 | 0.931034 | 0.947368 |
|              Polish (pol)              |  0.859813 | 0.910891 | 0.884615 |
|            Portuguese (por)            |  0.535714 | 0.833333 | 0.652174 |
|              Pushto (pus)              |  0.989362 | 0.902913 | 0.944162 |
|              Quechua (que)             |  0.979167 | 0.903846 | 0.940000 |
|      Tarantino dialect (roa-tara)      |  0.964912 | 0.901639 | 0.932203 |
|              Romansh (roh)             |  0.914894 | 0.895833 | 0.905263 |
|             Romanian (ron)             |  0.880597 | 0.880597 | 0.880597 |
|               Rusyn (rue)              |  0.932584 | 0.805825 | 0.864583 |
|             Aromanian (rup)            |  0.783333 | 0.758065 | 0.770492 |
|              Russian (rus)             |  0.517986 | 0.765957 | 0.618026 |
|               Yakut (sah)              |  0.954023 | 0.922222 | 0.937853 |
|             Sanskrit (san)             |  0.866667 | 0.951220 | 0.906977 |
|             Sicilian (scn)             |  0.984375 | 0.940299 | 0.961832 |
|               Scots (sco)              |  0.851351 | 0.900000 | 0.875000 |
|            Samogitian (sgs)            |  0.977011 | 0.876289 | 0.923913 |
|              Sinhala (sin)             |  0.406154 | 0.985075 | 0.575163 |
|              Slovak (slk)              |  0.956989 | 0.872549 | 0.912821 |
|              Slovene (slv)             |  0.907216 | 0.854369 | 0.880000 |
|           Northern Sami (sme)          |  0.949367 | 0.892857 | 0.920245 |
|               Shona (sna)              |  0.936508 | 0.855072 | 0.893939 |
|              Sindhi (snd)              |  0.984962 | 0.992424 | 0.988679 |
|              Somali (som)              |  0.949153 | 0.848485 | 0.896000 |
|              Spanish (spa)             |  0.584158 | 0.746835 | 0.655556 |
|             Albanian (sqi)             |  0.988095 | 0.912088 | 0.948571 |
|             Sardinian (srd)            |  0.957746 | 0.931507 | 0.944444 |
|              Sranan (srn)              |  0.985714 | 0.945205 | 0.965035 |
|              Serbian (srp)             |  0.950980 | 0.889908 | 0.919431 |
|          Saterfriesisch (stq)          |  0.962500 | 0.875000 | 0.916667 |
|             Sundanese (sun)            |  0.778846 | 0.910112 | 0.839378 |
|      Swahili (macrolanguage) (swa)     |  0.915493 | 0.878378 | 0.896552 |
|              Swedish (swe)             |  0.989247 | 0.958333 | 0.973545 |
|             Silesian (szl)             |  0.944444 | 0.904255 | 0.923913 |
|               Tamil (tam)              |  0.990000 | 0.970588 | 0.980198 |
|               Tatar (tat)              |  0.942029 | 0.902778 | 0.921986 |
|               Tulu (tcy)               |  0.980519 | 0.967949 | 0.974194 |
|              Telugu (tel)              |  0.965986 | 0.965986 | 0.965986 |
|               Tetum (tet)              |  0.898734 | 0.855422 | 0.876543 |
|               Tajik (tgk)              |  0.974684 | 0.939024 | 0.956522 |
|              Tagalog (tgl)             |  0.965909 | 0.934066 | 0.949721 |
|               Thai (tha)               |  0.923077 | 0.882353 | 0.902256 |
|              Tongan (ton)              |  0.970149 | 0.890411 | 0.928571 |
|              Tswana (tsn)              |  0.888889 | 0.926316 | 0.907216 |
|              Turkmen (tuk)             |  0.968000 | 0.889706 | 0.927203 |
|              Turkish (tur)             |  0.871287 | 0.926316 | 0.897959 |
|               Tuvan (tyv)              |  0.948454 | 0.859813 | 0.901961 |
|              Udmurt (udm)              |  0.989362 | 0.894231 | 0.939394 |
|              Uighur (uig)              |  1.000000 | 0.953333 | 0.976109 |
|             Ukrainian (ukr)            |  0.893617 | 0.875000 | 0.884211 |
|               Urdu (urd)               |  1.000000 | 1.000000 | 1.000000 |
|               Uzbek (uzb)              |  0.636042 | 0.886700 | 0.740741 |
|             Venetian (vec)             |  1.000000 | 0.941176 | 0.969697 |
|               Veps (vep)               |  0.858586 | 0.965909 | 0.909091 |
|            Vietnamese (vie)            |  1.000000 | 0.940476 | 0.969325 |
|              Vlaams (vls)              |  0.885714 | 0.898551 | 0.892086 |
|              Volapük (vol)             |  0.975309 | 0.975309 | 0.975309 |
|               Võro (vro)               |  0.855670 | 0.864583 | 0.860104 |
|               Waray (war)              |  0.972222 | 0.909091 | 0.939597 |
|              Walloon (wln)             |  0.742138 | 0.893939 | 0.810997 |
|               Wolof (wol)              |  0.882979 | 0.954023 | 0.917127 |
|            Wu Chinese (wuu)            |  0.961538 | 0.833333 | 0.892857 |
|               Xhosa (xho)              |  0.934066 | 0.867347 | 0.899471 |
|            Mingrelian (xmf)            |  0.958333 | 0.929293 | 0.943590 |
|              Yiddish (yid)             |  0.984375 | 0.875000 | 0.926471 |
|              Yoruba (yor)              |  0.868421 | 0.857143 | 0.862745 |
|              Zeeuws (zea)              |  0.879518 | 0.793478 | 0.834286 |
|           Cantonese (zh-yue)           |  0.896552 | 0.812500 | 0.852459 |
|         Standard Chinese (zho)         |  0.906250 | 0.935484 | 0.920635 |
|                accuracy                |  0.881051 | 0.881051 | 0.881051 |
|                macro avg               |  0.903245 | 0.880618 | 0.888996 |
|              weighted avg              |  0.894174 | 0.881051 | 0.884520 |

### By Token (3 to 5)

|                language                | precision |  recall  | f1-score |
|:--------------------------------------:|:---------:|:--------:|:--------:|
|             Achinese (ace)             |  0.873846 | 0.827988 | 0.850299 |
|             Afrikaans (afr)            |  0.638060 | 0.732334 | 0.681954 |
|         Alemannic German (als)         |  0.673780 | 0.547030 | 0.603825 |
|              Amharic (amh)             |  0.997743 | 0.954644 | 0.975717 |
|            Old English (ang)           |  0.840816 | 0.693603 | 0.760148 |
|              Arabic (ara)              |  0.768737 | 0.840749 | 0.803132 |
|             Aragonese (arg)            |  0.493671 | 0.505181 | 0.499360 |
|          Egyptian Arabic (arz)         |  0.823529 | 0.741935 | 0.780606 |
|             Assamese (asm)             |  0.948454 | 0.893204 | 0.920000 |
|             Asturian (ast)             |  0.490000 | 0.508299 | 0.498982 |
|               Avar (ava)               |  0.813636 | 0.655678 | 0.726166 |
|              Aymara (aym)              |  0.795833 | 0.779592 | 0.787629 |
|         South Azerbaijani (azb)        |  0.832836 | 0.863777 | 0.848024 |
|            Azerbaijani (aze)           |  0.867470 | 0.800000 | 0.832370 |
|              Bashkir (bak)             |  0.851852 | 0.750000 | 0.797688 |
|             Bavarian (bar)             |  0.560897 | 0.522388 | 0.540958 |
|           Central Bikol (bcl)          |  0.708229 | 0.668235 | 0.687651 |
| Belarusian (Taraschkewiza) (be-tarask) |  0.615635 | 0.526462 | 0.567568 |
|            Belarusian (bel)            |  0.539952 | 0.597855 | 0.567430 |
|              Bengali (ben)             |  0.830275 | 0.885086 | 0.856805 |
|             Bhojpuri (bho)             |  0.723118 | 0.691517 | 0.706965 |
|              Banjar (bjn)              |  0.619586 | 0.726269 | 0.668699 |
|              Tibetan (bod)             |  0.999537 | 0.991728 | 0.995617 |
|              Bosnian (bos)             |  0.330849 | 0.403636 | 0.363636 |
|            Bishnupriya (bpy)           |  0.941634 | 0.949020 | 0.945312 |
|              Breton (bre)              |  0.772222 | 0.745308 | 0.758527 |
|             Bulgarian (bul)            |  0.771505 | 0.706897 | 0.737789 |
|              Buryat (bxr)              |  0.741935 | 0.753149 | 0.747500 |
|              Catalan (cat)             |  0.528716 | 0.610136 | 0.566516 |
|             Chavacano (cbk)            |  0.409449 | 0.312625 | 0.354545 |
|             Min Dong (cdo)             |  0.951264 | 0.936057 | 0.943599 |
|              Cebuano (ceb)             |  0.888298 | 0.876640 | 0.882431 |
|               Czech (ces)              |  0.806045 | 0.758294 | 0.781441 |
|              Chechen (che)             |  0.857143 | 0.600000 | 0.705882 |
|             Cherokee (chr)             |  0.997840 | 0.952577 | 0.974684 |
|              Chuvash (chv)             |  0.874346 | 0.776744 | 0.822660 |
|          Central Kurdish (ckb)         |  0.984848 | 0.953545 | 0.968944 |
|              Cornish (cor)             |  0.747596 | 0.807792 | 0.776529 |
|             Corsican (cos)             |  0.673913 | 0.708571 | 0.690808 |
|           Crimean Tatar (crh)          |  0.498801 | 0.700337 | 0.582633 |
|             Kashubian (csb)            |  0.797059 | 0.794721 | 0.795888 |
|               Welsh (cym)              |  0.829609 | 0.841360 | 0.835443 |
|              Danish (dan)              |  0.649789 | 0.622222 | 0.635707 |
|              German (deu)              |  0.559406 | 0.763514 | 0.645714 |
|               Dimli (diq)              |  0.835580 | 0.763547 | 0.797941 |
|              Dhivehi (div)             |  1.000000 | 0.980645 | 0.990228 |
|           Lower Sorbian (dsb)          |  0.740484 | 0.694805 | 0.716918 |
|              Doteli (dty)              |  0.616314 | 0.527132 | 0.568245 |
|              Emilian (egl)             |  0.822993 | 0.769625 | 0.795414 |
|           Modern Greek (ell)           |  0.972043 | 0.963753 | 0.967880 |
|              English (eng)             |  0.260492 | 0.724346 | 0.383183 |
|             Esperanto (epo)            |  0.766764 | 0.716621 | 0.740845 |
|             Estonian (est)             |  0.698885 | 0.673835 | 0.686131 |
|              Basque (eus)              |  0.882716 | 0.841176 | 0.861446 |
|           Extremaduran (ext)           |  0.570605 | 0.511628 | 0.539510 |
|              Faroese (fao)             |  0.773987 | 0.784017 | 0.778970 |
|              Persian (fas)             |  0.709836 | 0.809346 | 0.756332 |
|              Finnish (fin)             |  0.866261 | 0.796089 | 0.829694 |
|              French (fra)              |  0.496263 | 0.700422 | 0.580927 |
|              Arpitan (frp)             |  0.663366 | 0.584302 | 0.621329 |
|          Western Frisian (fry)         |  0.750000 | 0.756148 | 0.753061 |
|             Friulian (fur)             |  0.713555 | 0.675545 | 0.694030 |
|              Gagauz (gag)              |  0.728125 | 0.677326 | 0.701807 |
|          Scottish Gaelic (gla)         |  0.831601 | 0.817996 | 0.824742 |
|               Irish (gle)              |  0.868852 | 0.801296 | 0.833708 |
|             Galician (glg)             |  0.469816 | 0.454315 | 0.461935 |
|              Gilaki (glk)              |  0.703883 | 0.687204 | 0.695444 |
|               Manx (glv)               |  0.873047 | 0.886905 | 0.879921 |
|              Guarani (grn)             |  0.848580 | 0.793510 | 0.820122 |
|             Gujarati (guj)             |  0.995643 | 0.926978 | 0.960084 |
|           Hakka Chinese (hak)          |  0.898403 | 0.904971 | 0.901675 |
|          Haitian Creole (hat)          |  0.719298 | 0.518987 | 0.602941 |
|               Hausa (hau)              |  0.815353 | 0.829114 | 0.822176 |
|          Serbo-Croatian (hbs)          |  0.343465 | 0.244589 | 0.285714 |
|              Hebrew (heb)              |  0.891304 | 0.933941 | 0.912125 |
|            Fiji Hindi (hif)            |  0.662577 | 0.664615 | 0.663594 |
|               Hindi (hin)              |  0.782301 | 0.778169 | 0.780229 |
|             Croatian (hrv)             |  0.360308 | 0.374000 | 0.367026 |
|           Upper Sorbian (hsb)          |  0.745763 | 0.611111 | 0.671756 |
|             Hungarian (hun)            |  0.876812 | 0.846154 | 0.861210 |
|             Armenian (hye)             |  0.988201 | 0.917808 | 0.951705 |
|               Igbo (ibo)               |  0.825397 | 0.696429 | 0.755448 |
|                Ido (ido)               |  0.760479 | 0.814103 | 0.786378 |
|            Interlingue (ile)           |  0.701299 | 0.580645 | 0.635294 |
|               Iloko (ilo)              |  0.688356 | 0.844538 | 0.758491 |
|            Interlingua (ina)           |  0.577889 | 0.588235 | 0.583016 |
|            Indonesian (ind)            |  0.415879 | 0.514019 | 0.459770 |
|             Icelandic (isl)            |  0.855263 | 0.790754 | 0.821745 |
|              Italian (ita)             |  0.474576 | 0.561247 | 0.514286 |
|          Jamaican Patois (jam)         |  0.826087 | 0.791667 | 0.808511 |
|             Javanese (jav)             |  0.670130 | 0.658163 | 0.664093 |
|              Lojban (jbo)              |  0.896861 | 0.917431 | 0.907029 |
|             Japanese (jpn)             |  0.931373 | 0.848214 | 0.887850 |
|            Karakalpak (kaa)            |  0.790393 | 0.827744 | 0.808637 |
|              Kabyle (kab)              |  0.828571 | 0.759162 | 0.792350 |
|              Kannada (kan)             |  0.879357 | 0.847545 | 0.863158 |
|             Georgian (kat)             |  0.916399 | 0.907643 | 0.912000 |
|              Kazakh (kaz)              |  0.900901 | 0.819672 | 0.858369 |
|             Kabardian (kbd)            |  0.923345 | 0.892256 | 0.907534 |
|           Central Khmer (khm)          |  0.976667 | 0.816156 | 0.889226 |
|            Kinyarwanda (kin)           |  0.824324 | 0.726190 | 0.772152 |
|              Kirghiz (kir)             |  0.674766 | 0.779698 | 0.723447 |
|           Komi-Permyak (koi)           |  0.652830 | 0.633700 | 0.643123 |
|              Konkani (kok)             |  0.778865 | 0.728938 | 0.753075 |
|               Komi (kom)               |  0.737374 | 0.572549 | 0.644592 |
|              Korean (kor)              |  0.984615 | 0.967603 | 0.976035 |
|          Karachay-Balkar (krc)         |  0.869416 | 0.857627 | 0.863481 |
|            Ripuarisch (ksh)            |  0.709859 | 0.649485 | 0.678331 |
|              Kurdish (kur)             |  0.883777 | 0.862884 | 0.873206 |
|              Ladino (lad)              |  0.660920 | 0.576441 | 0.615797 |
|                Lao (lao)               |  0.986175 | 0.918455 | 0.951111 |
|               Latin (lat)              |  0.581250 | 0.636986 | 0.607843 |
|              Latvian (lav)             |  0.824513 | 0.797844 | 0.810959 |
|             Lezghian (lez)             |  0.898955 | 0.793846 | 0.843137 |
|             Ligurian (lij)             |  0.662903 | 0.677100 | 0.669927 |
|             Limburgan (lim)            |  0.615385 | 0.581818 | 0.598131 |
|              Lingala (lin)             |  0.836207 | 0.763780 | 0.798354 |
|            Lithuanian (lit)            |  0.756329 | 0.804714 | 0.779772 |
|              Lombard (lmo)             |  0.556818 | 0.536986 | 0.546722 |
|           Northern Luri (lrc)          |  0.838574 | 0.753296 | 0.793651 |
|             Latgalian (ltg)            |  0.759531 | 0.755102 | 0.757310 |
|           Luxembourgish (ltz)          |  0.645062 | 0.614706 | 0.629518 |
|              Luganda (lug)             |  0.787535 | 0.805797 | 0.796562 |
|         Literary Chinese (lzh)         |  0.921951 | 0.949749 | 0.935644 |
|             Maithili (mai)             |  0.777778 | 0.761658 | 0.769634 |
|             Malayalam (mal)            |  0.993377 | 0.949367 | 0.970874 |
|          Banyumasan (map-bms)          |  0.531429 | 0.453659 | 0.489474 |
|              Marathi (mar)             |  0.748744 | 0.818681 | 0.782152 |
|              Moksha (mdf)              |  0.728745 | 0.800000 | 0.762712 |
|           Eastern Mari (mhr)           |  0.790323 | 0.760870 | 0.775316 |
|            Minangkabau (min)           |  0.953271 | 0.886957 | 0.918919 |
|            Macedonian (mkd)            |  0.816399 | 0.849722 | 0.832727 |
|             Malagasy (mlg)             |  0.925187 | 0.918317 | 0.921739 |
|              Maltese (mlt)             |  0.869421 | 0.890017 | 0.879599 |
|          Min Nan Chinese (nan)         |  0.743707 | 0.820707 | 0.780312 |
|             Mongolian (mon)            |  0.852194 | 0.838636 | 0.845361 |
|               Maori (mri)              |  0.934726 | 0.937173 | 0.935948 |
|           Western Mari (mrj)           |  0.818792 | 0.827119 | 0.822934 |
|               Malay (msa)              |  0.508065 | 0.376119 | 0.432247 |
|             Mirandese (mwl)            |  0.650407 | 0.685225 | 0.667362 |
|              Burmese (mya)             |  0.995968 | 0.972441 | 0.984064 |
|               Erzya (myv)              |  0.475783 | 0.503012 | 0.489019 |
|            Mazanderani (mzn)           |  0.775362 | 0.701639 | 0.736661 |
|            Neapolitan (nap)            |  0.628993 | 0.595349 | 0.611708 |
|              Navajo (nav)              |  0.955882 | 0.937500 | 0.946602 |
|         Classical Nahuatl (nci)        |  0.679758 | 0.589005 | 0.631136 |
|            Low German (nds)            |  0.669789 | 0.690821 | 0.680143 |
|        West Low German (nds-nl)        |  0.513889 | 0.504545 | 0.509174 |
|      Nepali (macrolanguage) (nep)      |  0.640476 | 0.649758 | 0.645084 |
|              Newari (new)              |  0.928571 | 0.745902 | 0.827273 |
|               Dutch (nld)              |  0.553763 | 0.553763 | 0.553763 |
|         Norwegian Nynorsk (nno)        |  0.569277 | 0.519231 | 0.543103 |
|              Bokmål (nob)              |  0.519856 | 0.562500 | 0.540338 |
|               Narom (nrm)              |  0.691275 | 0.605882 | 0.645768 |
|          Northern Sotho (nso)          |  0.950276 | 0.815166 | 0.877551 |
|              Occitan (oci)             |  0.483444 | 0.366834 | 0.417143 |
|          Livvi-Karelian (olo)          |  0.816850 | 0.790780 | 0.803604 |
|               Oriya (ori)              |  0.981481 | 0.963636 | 0.972477 |
|               Oromo (orm)              |  0.885714 | 0.829218 | 0.856536 |
|             Ossetian (oss)             |  0.822006 | 0.855219 | 0.838284 |
|            Pangasinan (pag)            |  0.842105 | 0.715655 | 0.773748 |
|             Pampanga (pam)             |  0.770000 | 0.435028 | 0.555957 |
|              Panjabi (pan)             |  0.996154 | 0.984791 | 0.990440 |
|            Papiamento (pap)            |  0.674672 | 0.661670 | 0.668108 |
|              Picard (pcd)              |  0.407895 | 0.356322 | 0.380368 |
|        Pennsylvania German (pdc)       |  0.487047 | 0.509485 | 0.498013 |
|          Palatine German (pfl)         |  0.614173 | 0.570732 | 0.591656 |
|          Western Panjabi (pnb)         |  0.926267 | 0.887417 | 0.906426 |
|              Polish (pol)              |  0.797059 | 0.734417 | 0.764457 |
|            Portuguese (por)            |  0.500914 | 0.586724 | 0.540434 |
|              Pushto (pus)              |  0.941489 | 0.898477 | 0.919481 |
|              Quechua (que)             |  0.854167 | 0.797665 | 0.824950 |
|      Tarantino dialect (roa-tara)      |  0.669794 | 0.724138 | 0.695906 |
|              Romansh (roh)             |  0.745527 | 0.760649 | 0.753012 |
|             Romanian (ron)             |  0.805486 | 0.769048 | 0.786845 |
|               Rusyn (rue)              |  0.718543 | 0.645833 | 0.680251 |
|             Aromanian (rup)            |  0.288482 | 0.730245 | 0.413580 |
|              Russian (rus)             |  0.530120 | 0.690583 | 0.599805 |
|               Yakut (sah)              |  0.853521 | 0.865714 | 0.859574 |
|             Sanskrit (san)             |  0.931343 | 0.896552 | 0.913616 |
|             Sicilian (scn)             |  0.734139 | 0.618321 | 0.671271 |
|               Scots (sco)              |  0.571429 | 0.540816 | 0.555701 |
|            Samogitian (sgs)            |  0.829167 | 0.748120 | 0.786561 |
|              Sinhala (sin)             |  0.909474 | 0.935065 | 0.922092 |
|              Slovak (slk)              |  0.738235 | 0.665782 | 0.700139 |
|              Slovene (slv)             |  0.671123 | 0.662269 | 0.666667 |
|           Northern Sami (sme)          |  0.800676 | 0.825784 | 0.813036 |
|               Shona (sna)              |  0.761702 | 0.724696 | 0.742739 |
|              Sindhi (snd)              |  0.950172 | 0.946918 | 0.948542 |
|              Somali (som)              |  0.849462 | 0.802030 | 0.825065 |
|              Spanish (spa)             |  0.325234 | 0.413302 | 0.364017 |
|             Albanian (sqi)             |  0.875899 | 0.832479 | 0.853637 |
|             Sardinian (srd)            |  0.750000 | 0.711061 | 0.730012 |
|              Sranan (srn)              |  0.888889 | 0.771084 | 0.825806 |
|              Serbian (srp)             |  0.824561 | 0.814356 | 0.819427 |
|          Saterfriesisch (stq)          |  0.790087 | 0.734417 | 0.761236 |
|             Sundanese (sun)            |  0.764192 | 0.631769 | 0.691700 |
|      Swahili (macrolanguage) (swa)     |  0.763496 | 0.796247 | 0.779528 |
|              Swedish (swe)             |  0.838284 | 0.723647 | 0.776758 |
|             Silesian (szl)             |  0.819788 | 0.750809 | 0.783784 |
|               Tamil (tam)              |  0.985765 | 0.955172 | 0.970228 |
|               Tatar (tat)              |  0.469780 | 0.795349 | 0.590674 |
|               Tulu (tcy)               |  0.893300 | 0.873786 | 0.883436 |
|              Telugu (tel)              |  1.000000 | 0.913690 | 0.954899 |
|               Tetum (tet)              |  0.765116 | 0.744344 | 0.754587 |
|               Tajik (tgk)              |  0.828418 | 0.813158 | 0.820717 |
|              Tagalog (tgl)             |  0.751468 | 0.757396 | 0.754420 |
|               Thai (tha)               |  0.933884 | 0.807143 | 0.865900 |
|              Tongan (ton)              |  0.920245 | 0.923077 | 0.921659 |
|              Tswana (tsn)              |  0.873397 | 0.889070 | 0.881164 |
|              Turkmen (tuk)             |  0.898438 | 0.837887 | 0.867107 |
|              Turkish (tur)             |  0.666667 | 0.716981 | 0.690909 |
|               Tuvan (tyv)              |  0.857143 | 0.805063 | 0.830287 |
|              Udmurt (udm)              |  0.865517 | 0.756024 | 0.807074 |
|              Uighur (uig)              |  0.991597 | 0.967213 | 0.979253 |
|             Ukrainian (ukr)            |  0.771341 | 0.702778 | 0.735465 |
|               Urdu (urd)               |  0.877647 | 0.855505 | 0.866434 |
|               Uzbek (uzb)              |  0.655652 | 0.797040 | 0.719466 |
|             Venetian (vec)             |  0.611111 | 0.527233 | 0.566082 |
|               Veps (vep)               |  0.672862 | 0.688213 | 0.680451 |
|            Vietnamese (vie)            |  0.932406 | 0.914230 | 0.923228 |
|              Vlaams (vls)              |  0.594427 | 0.501305 | 0.543909 |
|              Volapük (vol)             |  0.765625 | 0.942308 | 0.844828 |
|               Võro (vro)               |  0.797203 | 0.740260 | 0.767677 |
|               Waray (war)              |  0.930876 | 0.930876 | 0.930876 |
|              Walloon (wln)             |  0.636804 | 0.693931 | 0.664141 |
|               Wolof (wol)              |  0.864220 | 0.845601 | 0.854809 |
|            Wu Chinese (wuu)            |  0.848921 | 0.830986 | 0.839858 |
|               Xhosa (xho)              |  0.837398 | 0.759214 | 0.796392 |
|            Mingrelian (xmf)            |  0.943396 | 0.874126 | 0.907441 |
|              Yiddish (yid)             |  0.955729 | 0.897311 | 0.925599 |
|              Yoruba (yor)              |  0.812010 | 0.719907 | 0.763190 |
|              Zeeuws (zea)              |  0.617737 | 0.550409 | 0.582133 |
|           Cantonese (zh-yue)           |  0.859649 | 0.649007 | 0.739623 |
|         Standard Chinese (zho)         |  0.845528 | 0.781955 | 0.812500 |
|                accuracy                |  0.749527 | 0.749527 | 0.749527 |
|                macro avg               |  0.762866 | 0.742101 | 0.749261 |
|              weighted avg              |  0.762006 | 0.749527 | 0.752910 |


## Questions?
Post a Github issue from [HERE](https://github.com/m3hrdadfi/zabanshenas/issues).
{"language": ["multilingual", "ace", "afr", "als", "amh", "ang", "ara", "arg", "arz", "asm", "ast", "ava", "aym", "azb", "aze", "bak", "bar", "bcl", "bel", "ben", "bho", "bjn", "bod", "bos", "bpy", "bre", "bul", "bxr", "cat", "cbk", "cdo", "ceb", "ces", "che", "chr", "chv", "ckb", "cor", "cos", "crh", "csb", "cym", "dan", "deu", "diq", "div", "dsb", "dty", "egl", "ell", "eng", "epo", "est", "eus", "ext", "fao", "fas", "fin", "fra", "frp", "fry", "fur", "gag", "gla", "gle", "glg", "glk", "glv", "grn", "guj", "hak", "hat", "hau", "hbs", "heb", "hif", "hin", "hrv", "hsb", "hun", "hye", "ibo", "ido", "ile", "ilo", "ina", "ind", "isl", "ita", "jam", "jav", "jbo", "jpn", "kaa", "kab", "kan", "kat", "kaz", "kbd", "khm", "kin", "kir", "koi", "kok", "kom", "kor", "krc", "ksh", "kur", "lad", "lao", "lat", "lav", "lez", "lij", "lim", "lin", "lit", "lmo", "lrc", "ltg", "ltz", "lug", "lzh", "mai", "mal", "mar", "mdf", "mhr", "min", "mkd", "mlg", "mlt", "nan", "mon", "mri", "mrj", "msa", "mwl", "mya", "myv", "mzn", "nap", "nav", "nci", "nds", "nep", "new", "nld", "nno", "nob", "nrm", "nso", "oci", "olo", "ori", "orm", "oss", "pag", "pam", "pan", "pap", "pcd", "pdc", "pfl", "pnb", "pol", "por", "pus", "que", "roh", "ron", "rue", "rup", "rus", "sah", "san", "scn", "sco", "sgs", "sin", "slk", "slv", "sme", "sna", "snd", "som", "spa", "sqi", "srd", "srn", "srp", "stq", "sun", "swa", "swe", "szl", "tam", "tat", "tcy", "tel", "tet", "tgk", "tgl", "tha", "ton", "tsn", "tuk", "tur", "tyv", "udm", "uig", "ukr", "urd", "uzb", "vec", "vep", "vie", "vls", "vol", "vro", "war", "wln", "wol", "wuu", "xho", "xmf", "yid", "yor", "zea", "zho"], "license": "apache-2.0", "datasets": ["wili_2018"], "language_bcp47": ["be-tarask", "map-bms", "nds-nl", "roa-tara", "zh-yue"]}
transformers pytorch tf roberta text-classification multilingual ace afr als amh ang ara arg arz asm ast ava aym azb aze bak bar bcl bel ben bho bjn bod bos bpy bre bul bxr cat cbk cdo ceb ces che chr chv ckb cor cos crh csb cym dan deu diq div dsb dty egl ell eng epo est eus ext fao fas fin fra frp fry fur gag gla gle glg glk glv grn guj hak hat hau hbs heb hif hin hrv hsb hun hye ibo ido ile ilo ina ind isl ita jam jav jbo jpn kaa kab kan kat kaz kbd khm kin kir koi kok kom kor krc ksh kur lad lao lat lav lez lij lim lin lit lmo lrc ltg ltz lug lzh mai mal mar mdf mhr min mkd mlg mlt nan mon mri mrj msa mwl mya myv mzn nap nav nci nds nep new nld nno nob nrm nso oci olo ori orm oss pag pam pan pap pcd pdc pfl pnb pol por pus que roh ron rue rup rus sah san scn sco sgs sin slk slv sme sna snd som spa sqi srd srn srp stq sun swa swe szl tam tat tcy tel tet tgk tgl tha ton tsn tuk tur tyv udm uig ukr urd uzb vec vep vie vls vol vro war wln wol wuu xho xmf yid yor zea zho dataset:wili_2018 license:apache-2.0 autotrain_compatible endpoints_compatible has_space region:us
TAGS
#transformers #pytorch #tf #roberta #text-classification #multilingual #ace #afr #als #amh #ang #ara #arg #arz #asm #ast #ava #aym #azb #aze #bak #bar #bcl #bel #ben #bho #bjn #bod #bos #bpy #bre #bul #bxr #cat #cbk #cdo #ceb #ces #che #chr #chv #ckb #cor #cos #crh #csb #cym #dan #deu #diq #div #dsb #dty #egl #ell #eng #epo #est #eus #ext #fao #fas #fin #fra #frp #fry #fur #gag #gla #gle #glg #glk #glv #grn #guj #hak #hat #hau #hbs #heb #hif #hin #hrv #hsb #hun #hye #ibo #ido #ile #ilo #ina #ind #isl #ita #jam #jav #jbo #jpn #kaa #kab #kan #kat #kaz #kbd #khm #kin #kir #koi #kok #kom #kor #krc #ksh #kur #lad #lao #lat #lav #lez #lij #lim #lin #lit #lmo #lrc #ltg #ltz #lug #lzh #mai #mal #mar #mdf #mhr #min #mkd #mlg #mlt #nan #mon #mri #mrj #msa #mwl #mya #myv #mzn #nap #nav #nci #nds #nep #new #nld #nno #nob #nrm #nso #oci #olo #ori #orm #oss #pag #pam #pan #pap #pcd #pdc #pfl #pnb #pol #por #pus #que #roh #ron #rue #rup #rus #sah #san #scn #sco #sgs #sin #slk #slv #sme #sna #snd #som #spa #sqi #srd #srn #srp #stq #sun #swa #swe #szl #tam #tat #tcy #tel #tet #tgk #tgl #tha #ton #tsn #tuk #tur #tyv #udm #uig #ukr #urd #uzb #vec #vep #vie #vls #vol #vro #war #wln #wol #wuu #xho #xmf #yid #yor #zea #zho #dataset-wili_2018 #license-apache-2.0 #autotrain_compatible #endpoints_compatible #has_space #region-us 

Zabanshenas - Language Detector
===============================


Zabanshenas is a Transformer-based solution for identifying the most likely language of a written document/text. Zabanshenas is a Persian word that has two meanings:


* A person who studies linguistics.
* A way to identify the type of written language.


How to use
----------


Follow Zabanshenas repo for more information!


Evaluation
----------


The following tables summarize the scores obtained by model overall and per each class.


### By Paragraph



### By Sentence



### By Token (3 to 5)



Questions?
----------


Post a Github issue from HERE.



### By Paragraph ### By Sentence ### By Token (3 to 5)



Questions?
----------


Post a Github issue from HERE.
TAGS
#transformers #pytorch #tf #roberta #text-classification #multilingual #ace #afr #als #amh #ang #ara #arg #arz #asm #ast #ava #aym #azb #aze #bak #bar #bcl #bel #ben #bho #bjn #bod #bos #bpy #bre #bul #bxr #cat #cbk #cdo #ceb #ces #che #chr #chv #ckb #cor #cos #crh #csb #cym #dan #deu #diq #div #dsb #dty #egl #ell #eng #epo #est #eus #ext #fao #fas #fin #fra #frp #fry #fur #gag #gla #gle #glg #glk #glv #grn #guj #hak #hat #hau #hbs #heb #hif #hin #hrv #hsb #hun #hye #ibo #ido #ile #ilo #ina #ind #isl #ita #jam #jav #jbo #jpn #kaa #kab #kan #kat #kaz #kbd #khm #kin #kir #koi #kok #kom #kor #krc #ksh #kur #lad #lao #lat #lav #lez #lij #lim #lin #lit #lmo #lrc #ltg #ltz #lug #lzh #mai #mal #mar #mdf #mhr #min #mkd #mlg #mlt #nan #mon #mri #mrj #msa #mwl #mya #myv #mzn #nap #nav #nci #nds #nep #new #nld #nno #nob #nrm #nso #oci #olo #ori #orm #oss #pag #pam #pan #pap #pcd #pdc #pfl #pnb #pol #por #pus #que #roh #ron #rue #rup #rus #sah #san #scn #sco #sgs #sin #slk #slv #sme #sna #snd #som #spa #sqi #srd #srn #srp #stq #sun #swa #swe #szl #tam #tat #tcy #tel #tet #tgk #tgl #tha #ton #tsn #tuk #tur #tyv #udm #uig #ukr #urd #uzb #vec #vep #vie #vls #vol #vro #war #wln #wol #wuu #xho #xmf #yid #yor #zea #zho #dataset-wili_2018 #license-apache-2.0 #autotrain_compatible #endpoints_compatible #has_space #region-us 
 ### By Paragraph ### By Sentence ### By Token (3 to 5)



Questions?
----------


Post a Github issue from HERE.
passage: 