Regarding Tokeniser

#19

by ashesblue - opened Jul 8, 2023

Jul 8, 2023

Did you guys use EOT TOKEN while training the model ?

rooa

Salesforce org Jul 10, 2023

Yes, <|endoftext|> is used as the document boundary during training.

rooa changed discussion status to closed Jul 10, 2023

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Your need to confirm your account before you can post a new comment.

· Sign up or log in to comment