ibm-granite

/

granite-vision-3.1-2b-preview

like 43

Follow

IBM Granite 982

This PR adds the system prompt as the default in the chat template; it will be rendered when there are no messages with the system role.

Chat template changes are a bit horrible to read; the main change is shown below; this is added before the main message loop to check if there is a system role, and render the default template if there isn't.

{%- for message in messages if message['role'] == 'system'%}{% else %}<|system|>\nA chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.\n{% endfor %}

Test case to verify (run against my local transformers build):

def test_default_system_prompt():
    """
    Ensure that we apply a default for the system prompt if there are no system role messages found.
    """
    default_system_prompt = "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions."
    t_system_prompt = {"role": "system", "content": default_system_prompt}
    p_system_prompt = {"role": "system", "content": [{"type": "text", "text": default_system_prompt}]}
    

    tokenizer_messages = [
        {"role": "user", "content": "<image>\n<image>\nhello"},
    ]
    processor_messages = [
        {
            "role": "user",
            "content": [
                {"type": "image"},
                {"type": "text", "text": "hello"},
                {"type": "image"},
            ]
        },
    ]

    tokenizer = transformers.AutoTokenizer.from_pretrained(MODEL_PATH)
    processor = transformers.AutoProcessor.from_pretrained(MODEL_PATH)
    # Pass the messages with / without an explicit system prompt
    tok_result = tokenizer.apply_chat_template(tokenizer_messages, tokenize=False, add_generation_prompt=True)
    override_tok_result = tokenizer.apply_chat_template([t_system_prompt] + tokenizer_messages, tokenize=False, add_generation_prompt=True)

    proc_result = processor.apply_chat_template(processor_messages, tokenizer=False, add_generation_prompt=True)
    override_proc_result = processor.apply_chat_template([p_system_prompt] + processor_messages, tokenizer=False, add_generation_prompt=True)

    # Since the system prompt pass as an override matches the default,
    # we should get the same result everywhere
    assert tok_result == override_tok_result == proc_result == override_proc_result

I also updated the transformers example to remove the system prompt, since after this change, the template it is using is applied by default.

Thanks @abrooks9944 !