PrajwalS's picture
add tokenizer
ca5a4cc
raw
history blame
278 Bytes
{"y": 0, "u": 1, "q": 3, "r": 4, "i": 5, "f": 6, "o": 7, "a": 8, "k": 9, "l": 10, "b": 11, "t": 12, "w": 13, "z": 14, "x": 15, "g": 16, "m": 17, "h": 18, "'": 19, "d": 20, "c": 21, "v": 22, "j": 23, "s": 24, "â": 25, "p": 26, "e": 27, "n": 28, "|": 2, "[UNK]": 29, "[PAD]": 30}