Hugging Face's logo

DeepPavlov
/
bert-base-bg-cs-pl-ru-cased

Feature Extraction
Transformers
PyTorch
JAX
bert
Inference Endpoints
Model card Files Community
1
bert-base-bg-cs-pl-ru-cased / README.md
julien-c's picture
julien-c HF staff
Migrate model card from transformers-repo
dba4c95
preview code
|
raw
history blame
597 Bytes
metadata
language:
  - bg
  - cs
  - pl
  - ru

bert-base-bg-cs-pl-ru-cased

SlavicBERT[1] (Slavic (bg, cs, pl, ru), cased, 12‑layer, 768‑hidden, 12‑heads, 180M parameters) was trained on Russian News and four Wikipedias: Bulgarian, Czech, Polish, and Russian. Subtoken vocabulary was built using this data. Multilingual BERT was used as an initialization for SlavicBERT.

[1]: Arkhipov M., Trofimova M., Kuratov Y., Sorokin A. (2019). Tuning Multilingual Transformers for Language-Specific Named Entity Recognition. ACL anthology W19-3712.