stabilityai
/

arcade100k

Model card Files Files and versions Community

arcade100k / tokenization_arcade100k.py

Commit History

fix: add missing `errors`

b5b7b34

jon-tow commited on Jan 25, 2024

feat: add pickle support

cd5298a

jon-tow commited on Jan 23, 2024

fix: make `eos_token`/`pad_token` overridable

dfe1f6f

jon-tow commited on Jan 23, 2024

feat: expose tiktoken tokenizer merge ranks and special tokens

ee2a4ae

jon-tow commited on Jan 19, 2024

fix: check surface form against special tokens in `_add_tokens`

9ada655

jon-tow commited on Dec 21, 2023

fix: update license

7e4138f

jon-tow commited on Dec 19, 2023

fix: re-ordering special tokens

f221007

jon-tow commited on Nov 10, 2023

fix: remove `print` debug statements

3c66e0d

jon-tow commited on Nov 7, 2023

Update tokenization_arcade100k.py

8593ce9

jon-tow commited on Nov 3, 2023

fix: create final list of special tokens

9b47601

jon-tow commited on Nov 3, 2023

init: first commit

aceea5b

jon-tow commited on Oct 28, 2023