
# HerBERT tokenizer

**[HerBERT](https://en.wikipedia.org/wiki/Zbigniew_Herbert)** tokenizer is a character level byte-pair encoding with
vocabulary size of 50k tokens. The tokenizer was trained on [Wolne Lektury](https://wolnelektury.pl/) and a publicly available subset of
[National Corpus of Polish](http://nkjp.pl/index.php?page=14&lang=0) with [fastBPE](https://github.com/glample/fastBPE) library.
Tokenizer utilize `XLMTokenizer` implementation from [transformers](https://github.com/huggingface/transformers).

## Tokenizer usage
Herbert tokenizer should be used together with [HerBERT model](https://huggingface.co/allegro/herbert-klej-cased-v1):
```python
from transformers import XLMTokenizer, RobertaModel

tokenizer = XLMTokenizer.from_pretrained("allegro/herbert-klej-cased-tokenizer-v1")
model = RobertaModel.from_pretrained("allegro/herbert-klej-cased-v1")

encoded_input = tokenizer.encode("Kto ma lepszą sztukę, ma lepszy rząd – to jasne.", return_tensors='pt')
outputs = model(encoded_input)
```

## License
CC BY-SA 4.0

## Citation
If you use this tokenizer, please cite the following paper:
```
@inproceedings{rybak-etal-2020-klej,
    title = "{KLEJ}: Comprehensive Benchmark for {P}olish Language Understanding",
    author = "Rybak, Piotr  and
      Mroczkowski, Robert  and
      Tracz, Janusz  and
      Gawlik, Ireneusz",
    booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics",
    month = jul,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.acl-main.111",
    doi = "10.18653/v1/2020.acl-main.111",
    pages = "1191--1201",
}
```

## Authors
Tokenizer was created by **Allegro Machine Learning Research** team.

You can contact us at: <a href="mailto:klejbenchmark@allegro.pl">klejbenchmark@allegro.pl</a>

{"language": "pl"}
transformers xlm pl endpoints_compatible region:us
TAGS
#transformers #xlm #pl #endpoints_compatible #region-us 


# HerBERT tokenizer

HerBERT tokenizer is a character level byte-pair encoding with
vocabulary size of 50k tokens. The tokenizer was trained on Wolne Lektury and a publicly available subset of
National Corpus of Polish with fastBPE library.
Tokenizer utilize 'XLMTokenizer' implementation from transformers.

## Tokenizer usage
Herbert tokenizer should be used together with HerBERT model:


## License
CC BY-SA 4.0

If you use this tokenizer, please cite the following paper:


## Authors
Tokenizer was created by Allegro Machine Learning Research team.

You can contact us at: <a href="mailto:klejbenchmark@URL">klejbenchmark@URL</a>

# HerBERT tokenizer

HerBERT tokenizer is a character level byte-pair encoding with
vocabulary size of 50k tokens. The tokenizer was trained on Wolne Lektury and a publicly available subset of
National Corpus of Polish with fastBPE library.
Tokenizer utilize 'XLMTokenizer' implementation from transformers. ## Tokenizer usage
Herbert tokenizer should be used together with HerBERT model: ## License
CC BY-SA 4.0

If you use this tokenizer, please cite the following paper: ## Authors
Tokenizer was created by Allegro Machine Learning Research team.

You can contact us at: <a href="mailto:klejbenchmark@URL">klejbenchmark@URL</a>
TAGS
#transformers #xlm #pl #endpoints_compatible #region-us 
 # HerBERT tokenizer

HerBERT tokenizer is a character level byte-pair encoding with
vocabulary size of 50k tokens. The tokenizer was trained on Wolne Lektury and a publicly available subset of
National Corpus of Polish with fastBPE library.
Tokenizer utilize 'XLMTokenizer' implementation from transformers. ## Tokenizer usage
Herbert tokenizer should be used together with HerBERT model: ## License
CC BY-SA 4.0

If you use this tokenizer, please cite the following paper: ## Authors
Tokenizer was created by Allegro Machine Learning Research team.

You can contact us at: <a href="mailto:klejbenchmark@URL">klejbenchmark@URL</a>
passage: TAGS
#transformers #xlm #pl #endpoints_compatible #region-us 
# HerBERT tokenizer

HerBERT tokenizer is a character level byte-pair encoding with
vocabulary size of 50k tokens. The tokenizer was trained on Wolne Lektury and a publicly available subset of
National Corpus of Polish with fastBPE library.
Tokenizer utilize 'XLMTokenizer' implementation from transformers.## Tokenizer usage
Herbert tokenizer should be used together with HerBERT model:## License
CC BY-SA 4.0

If you use this tokenizer, please cite the following paper:## Authors
Tokenizer was created by Allegro Machine Learning Research team.

You can contact us at: <a href="mailto:klejbenchmark@URL">klejbenchmark@URL</a>