How is falcon able to generate indic token , when it is not trained on Indic languages. even tokenizer.json doest have any Indic tokens.

#90

by Sibadatta - opened Oct 3, 2023

Oct 3, 2023

@sanchit-gandhi boss can you please answer this doubt.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Your need to confirm your account before you can post a new comment.

· Sign up or log in to comment