hiroshi-matsuda-rit
/

bert-base-sudachitra-v11

Model card Files Files and versions Community

hiroshi-matsuda-rit commited on Jan 14, 2024

Commit

a9045c1

·

verified ·

1 Parent(s): 03305da

use sudachi_config_path

Files changed (1) hide show

tokenizer_config.json +2 -2

tokenizer_config.json CHANGED Viewed

@@ -12,8 +12,8 @@
   "mask_token": "[MASK]",
   "sudachi_kwargs": {
     "sudachi_split_mode": "A",
-    "projection": "normalized_nouns",
-    "sudachi_dict_type": "core"
   },
   "model_max_length": 512,
   "tokenizer_class": "BertJapaneseTokenizer"

   "mask_token": "[MASK]",
   "sudachi_kwargs": {
     "sudachi_split_mode": "A",
+    "sudachi_dict_type": "core",
+    "sudachi_config_path": "sudachi.cfg"
   },
   "model_max_length": 512,
   "tokenizer_class": "BertJapaneseTokenizer"