stabilityai
/

arcade100k

Model card Files Files and versions Community

Commit History

Update README.md

627be45
verified

hassanzay commited on Apr 12, 2024

Update README.md

e921c0e
verified

hassanzay commited on Apr 12, 2024

Update README.md

611d7a4
verified

hassanzay commited on Apr 12, 2024

Update README.md

b6c6300
verified

jon-tow commited on Apr 9, 2024

fix: add missing `errors`

b5b7b34

jon-tow commited on Jan 25, 2024

feat: add pickle support

cd5298a

jon-tow commited on Jan 23, 2024

fix: make `eos_token`/`pad_token` overridable

dfe1f6f

jon-tow commited on Jan 23, 2024

feat: expose tiktoken tokenizer merge ranks and special tokens

ee2a4ae

jon-tow commited on Jan 19, 2024

fix: check surface form against special tokens in `_add_tokens`

9ada655

jon-tow commited on Dec 21, 2023

fix: update license

7e4138f

jon-tow commited on Dec 19, 2023

fix: store `cl100k_base` mergeable ranks in `arcade100k.tiktoken`

1587587

jon-tow commited on Nov 10, 2023

Merge branch 'main' of hf.co:jon-tow/stablelm-tokenizer-v1 into main

d8301d9

jon-tow commited on Nov 10, 2023

fix: re-ordering special tokens

f221007

jon-tow commited on Nov 10, 2023

Create tokenizer_config.json

af51a49

jon-tow commited on Nov 10, 2023

Delete tokenizer_config.json

beeb2b0

jon-tow commited on Nov 10, 2023

Delete special_tokens_map.json

83aaca7

jon-tow commited on Nov 10, 2023

fix: pass in `vocab_file` to tiktoken loader

a06afc6

jon-tow commited on Nov 10, 2023

Upload tokenizer

f3e546a

jon-tow commited on Nov 10, 2023

fix: remove `print` debug statements

3c66e0d

jon-tow commited on Nov 7, 2023

Update tokenization_arcade100k.py

8593ce9

jon-tow commited on Nov 3, 2023

Update README.md

be6493b

jon-tow commited on Nov 3, 2023

fix: create final list of special tokens

9b47601

jon-tow commited on Nov 3, 2023

init: first commit

aceea5b

jon-tow commited on Oct 28, 2023

initial commit

99cb326

jon-tow commited on Oct 27, 2023