iszoke
/

LELD_cz_0102_v3_tokenizer_BPE_tc_punc_unk_VOC-lng-spkchng_500

Model card Files Files and versions Community

iszoke commited on Jan 16

Commit

dfce87b

·

verified ·

1 Parent(s): fb1abf0

Update tokenizer.json

Files changed (1) hide show

tokenizer.json +2 -2

tokenizer.json CHANGED Viewed

@@ -225,7 +225,7 @@
       "¨": 88,
       "©": 89,
       "ª": 90,
-      "«": 91,
       "®": 92,
       "¯": 93,
       "±": 94,
@@ -233,7 +233,7 @@
       "´": 96,
       "¶": 97,
       "º": 98,
-      "»": 99,
       "¼": 100,
       "½": 101,
       "¾": 102,

       "¨": 88,
       "©": 89,
       "ª": 90,
+      "<lng>": 91,
       "®": 92,
       "¯": 93,
       "±": 94,
       "´": 96,
       "¶": 97,
       "º": 98,
+      "<spkchng>": 99,
       "¼": 100,
       "½": 101,
       "¾": 102,