File size: 3,266 Bytes
0bd62e5 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 |
# Using the Messages data format
In the previous guides, we built chatbots where the conversation history was stored in a list of tuple pairs.
It is also possible to use the more flexible [Messages API](https://huggingface.co/docs/text-generation-inference/en/messages_api#messages-api), which is fully compatible with LLM API providers such as Hugging Face Text Generation Inference, Llama.cpp server, and OpenAI's chat completions API.
To use this format, set the `type` parameter of `gr.Chatbot` or `gr.ChatInterface` to `'messages'`. This expects a list of dictionaries with content and role keys.
The `role` key should be `'assistant'` for the bot/llm and `user` for the human.
The `content` key can be one of three things:
1. A string (markdown supported) to display a simple text message
2. A dictionary (or `gr.FileData`) to display a file. At minimum this dictionary should contain a `path` key corresponding to the path to the file. Full documenation of this dictionary is in the appendix of this guide.
3. A gradio component - at present `gr.Plot`, `gr.Image`, `gr.Gallery`, `gr.Video`, `gr.Audio` are supported.
For better type hinting and auto-completion in your IDE, you can use the `gr.ChatMessage` dataclass:
```python
from gradio import ChatMessage
def chat_function(message, history):
history.append(ChatMessage(role="user", content=message))
history.append(ChatMessage(role="assistant", content="Hello, how can I help you?"))
return history
```
## Examples
The following chatbot will always greet the user with "Hello"
```python
import gradio as gr
def chat_greeter(msg, history):
history.append({"role": "assistant", "content": "Hello!"})
return history
with gr.Blocks() as demo:
chatbot = gr.Chatbot(type="messages")
msg = gr.Textbox()
clear = gr.ClearButton([msg, chatbot])
msg.submit(chat_greeter, [msg, chatbot], [chatbot])
demo.launch()
```
The messages format lets us seemlessly stream from the Hugging Face Inference API -
```python
import gradio as gr
from huggingface_hub import InferenceClient
client = InferenceClient("HuggingFaceH4/zephyr-7b-beta")
def respond(message, history: list[dict]):
messages = history + [{"role": "user", "content": message}]
print(messages)
response = {"role": "assistant", "content": ""}
for message in client.chat_completion(
messages,
max_tokens=512,
stream=True,
temperature=0.7,
top_p=0.95,
):
token = message.choices[0].delta.content
response['content'] += token
yield response
demo = gr.ChatInterface(respond, type="messages")
if __name__ == "__main__":
demo.launch()
```
### Appendix
The full contents of the dictionary format for files is documented here
```python
class FileDataDict(TypedDict):
path: str # server filepath
url: NotRequired[Optional[str]] # normalised server url
size: NotRequired[Optional[int]] # size in bytes
orig_name: NotRequired[Optional[str]] # original filename
mime_type: NotRequired[Optional[str]]
is_stream: NotRequired[bool]
meta: dict[Literal["_type"], Literal["gradio.FileData"]]
``` |