Understanding vocab.txt of vinai/bertweet-base
While looking at vocab.txt here, I was left wondering why the vocabulary for Bertweet is not continuous.
For instance, see the below sample:
...
dice 63328
)@@ 63327
struggled 63326
wraps 63324
Investors 63312
#summer@@ 63305
...

As you can see, after 63305, we have 63312, followed by 63324... what about the numbers in between?

Also, it feels a bit strange why vocabulary starts at around 3800.

르@@ 3800
utory 3798
...

Any explanations will be really appreciated.


Each number denotes the frequency count that the corresponding word appears in the pre-training corpus.

Only top 64k words are included in the vocab.


