I could be wrong, but I think the <think> tag needs to be removed from the last bit of the jinja template in the tokenizer_config.json

#1
by jth01 - opened

As is, interfaces like open-webui don't display the opening tag, but changing {{- '<|im_start|>assistant\\n<think>\\n' }}\n{%- endif %}\n to {{- '<|im_start|>assistant\\n'}}\n{%- endif %}\n fixes this. I could be missing something important though.

Edit: And I'm getting better results removing the new line, {{- '<|im_start|>assistant\\n' }}{%- endif %}\n

Unsloth AI org

It's part of the add_generation_prompt so I think it default adds during the assistant turn.

Are you saying you get better results if we don add it by default?

With it added, open-webui and other frontends don't display the opening think tag, when I remove it, they do. Figured it's worth mentioning even if it's not worth changing.

Also, thank you, Unsloth is amazing.

With it added, open-webui and other frontends don't display the opening think tag, when I remove it, they do. Figured it's worth mentioning even if it's not worth changing.

Also, thank you, Unsloth is amazing.

You can check qwen model page, if you don't add <think>\n before generation, there's a chance the model will skip reasoning and perform worse, so they add it in chat template.

jth01 changed discussion status to closed
Your need to confirm your account before you can post a new comment.

Sign up or log in to comment