
# Japanese Dummy Tokenizer

Repository containing a dummy Japanese Tokenizer trained on ```snow_simplified_japanese_corpus``` dataset. The tokenizer has been trained using Hugging Face datasets in a streaming manner. 

## Intended uses & limitations

You can use this tokenizer to tokenize Japanese sentences.

## How to use it

```
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("ybelkada/japanese-dummy-tokenizer")
```

## How to train the tokenizer

Check the file ```tokenizer.py```, you can freely adapt it to other datasets. This tokenizer is based on the tokenizer from ```csebuetnlp/mT5_multilingual_XLSum```.
{"language": ["en", "ja"], "license": "mit", "tags": ["ja", "japanese", "tokenizer"], "datasets": ["snow_simplified_japanese_corpus"], "widget": [{"text": "\u8ab0\u304c\u4e00\u756a\u306b\u7740\u304f\u304b\u79c1\u306b\u306f\u5206\u304b\u308a\u307e\u305b\u3093\u3002"}]}
transformers ja japanese tokenizer en dataset:snow_simplified_japanese_corpus license:mit endpoints_compatible region:us
TAGS
#transformers #ja #japanese #tokenizer #en #dataset-snow_simplified_japanese_corpus #license-mit #endpoints_compatible #region-us 


# Japanese Dummy Tokenizer

Repository containing a dummy Japanese Tokenizer trained on  dataset. The tokenizer has been trained using Hugging Face datasets in a streaming manner. 

## Intended uses & limitations

You can use this tokenizer to tokenize Japanese sentences.

## How to use it



## How to train the tokenizer

Check the file , you can freely adapt it to other datasets. This tokenizer is based on the tokenizer from .
# Japanese Dummy Tokenizer

Repository containing a dummy Japanese Tokenizer trained on  dataset. The tokenizer has been trained using Hugging Face datasets in a streaming manner. ## Intended uses & limitations

You can use this tokenizer to tokenize Japanese sentences. ## How to use it ## How to train the tokenizer

Check the file , you can freely adapt it to other datasets. This tokenizer is based on the tokenizer from .
TAGS
#transformers #ja #japanese #tokenizer #en #dataset-snow_simplified_japanese_corpus #license-mit #endpoints_compatible #region-us 
 # Japanese Dummy Tokenizer

Repository containing a dummy Japanese Tokenizer trained on  dataset. The tokenizer has been trained using Hugging Face datasets in a streaming manner. ## Intended uses & limitations

You can use this tokenizer to tokenize Japanese sentences. ## How to use it ## How to train the tokenizer

Check the file , you can freely adapt it to other datasets. This tokenizer is based on the tokenizer from .
passage: TAGS
#transformers #ja #japanese #tokenizer #en #dataset-snow_simplified_japanese_corpus #license-mit #endpoints_compatible #region-us 
# Japanese Dummy Tokenizer

Repository containing a dummy Japanese Tokenizer trained on  dataset. The tokenizer has been trained using Hugging Face datasets in a streaming manner.## Intended uses & limitations

You can use this tokenizer to tokenize Japanese sentences.## How to use it## How to train the tokenizer

Check the file , you can freely adapt it to other datasets. This tokenizer is based on the tokenizer from .