autogen/website/docs/topics/non-openai-models/local-vllm.md
Chat Template  vLLM uses a pre-defined chat template, unless the model has a chat template defined in its config file on Hugging Face. This can cause an issue if the chat template doesn't allow `'role' : 'system'` messages, as used in AutoGen.  Therefore, we will create a chat template for the Mistral.AI Mistral 7B model we are using that allows roles of 'user', 'assistant', and 'system'.  Create a file name `autogenmistraltemplate.jinja` with the following content: ```` text {{ bos_token }} {% for message in messages %}     {% if ((message['role'] == 'user' or message['role'] == 'system') != (loop.index0 % 2 == 0)) %}         {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}     {% endif %}      {% if (message['role'] == 'user' or message['role'] == 'system') %}         {{ '[INST] ' + message['content'] + ' [/INST]' }}     {% elif message['role'] == 'assistant' %}         {{ message['content'] + eos_token}}     {% else %}         {{ raise_exception('Only system, user and assistant roles are supported!') }}     {% endif %} {% endfor %} ````  ````mdx-code-block :::warning Chat Templates are specific to the model/model family. The example shown here is for Mistral-based models like Mistral 7B and Mixtral 8x7B.  vLLM has a number of [example templates](https://github.com/vllm-project/vllm/tree/main/examples) for models that can be a starting point for your chat template. Just remember, the template may need to be adjusted to support 'system' role messages. ::: ```` 