# embeddings_queue

created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: c7171c25-753f-467b-ba58-d4907c4da862
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_gpu_many", "title": "Efficient Training on Multiple GPUs", "tokens": 9074, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"c7171c25-753f-467b-ba58-d4907c4da862\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ea042a4b-12ab-4251-a5f9-a94e6ac726d4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"66ac7fce0977224d623be0586b848b5609072b1bac1dfb9f30c513043b9b707d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8896779a-3f18-425b-905b-208e5458bad9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_gpu_many\", \"title\": \"Efficient Training on Multiple GPUs\", \"tokens\": 9074, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"f19c9d3c93d37e938a5569698ab3b47d1084f3f9c6ba8910b2247aee442c86f2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 34249, \"end_char_idx\": 36278, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "ref_doc_id": "ea042a4b-12ab-4251-a5f9-a94e6ac726d4", "chroma:document": "For example, if you have 4 GPUs (0, 1, 2, 3) and you only want to run GPUs 0 and 2:\n\n```bash\nCUDA_VISIBLE_DEVICES=0,2 torchrun trainer-program.py ...\n```\n\nOnly the 2 physical GPUs (0 and 2) are \"visible\" to PyTorch and these are mapped to `cuda:0` and `cuda:1` respectively. You can also reverse the order of the GPUs to use 2 first. Now, the mapping is `cuda:1` for GPU 0 and `cuda:0` for GPU 2.\n\n```bash\nCUDA_VISIBLE_DEVICES=2,0 torchrun trainer-program.py ...\n```\n\nYou can also set the `CUDA_VISIBLE_DEVICES` environment variable to an empty value to create an environment without GPUs.\n\n```bash\nCUDA_VISIBLE_DEVICES= python trainer-program.py ...\n```\n\n<Tip warning={true}>\n\nAs with any environment variable, they can be exported instead of being added to the command line. However, this is not recommended because it can be confusing if you forget how the environment variable was setup and you end up using the wrong GPUs. Instead, it is common practice to set the environment variable for a specific training run on the same command line.\n\n</Tip>\n\n`CUDA_DEVICE_ORDER` is an alternative environment variable you can use to control how the GPUs are ordered. You can either order them by:\n\n1. PCIe bus ID's that matches the order of [`nvidia-smi`](https://developer.nvidia.com/nvidia-system-management-interface) and [`rocm-smi`](https://rocm.docs.amd.com/projects/rocm_smi_lib/en/latest/.doxygen/docBin/html/index.html) for NVIDIA and AMD GPUs respectively\n\n```bash\nexport CUDA_DEVICE_ORDER=PCI_BUS_ID\n```\n\n2. GPU compute ability\n\n```bash\nexport CUDA_DEVICE_ORDER=FASTEST_FIRST\n```\n\nThe `CUDA_DEVICE_ORDER` is especially useful if your training setup consists of an older and newer GPU, where the older GPU appears first, but you cannot physically swap the cards to make the newer GPU appear first. In this case, set `CUDA_DEVICE_ORDER=FASTEST_FIRST` to always use the newer and faster GPU first (`nvidia-smi` or `rocm-smi` still reports the GPUs in their PCIe order). Or you could also set `export CUDA_VISIBLE_DEVICES=1,0`."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: ddb9a2b0-da44-48ec-8f43-b5ce2ef18918
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"ddb9a2b0-da44-48ec-8f43-b5ce2ef18918\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"544b1cdb-2cc9-4645-adb9-558d399cc3e1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"54374f90-6c4a-4a8a-8db6-43b2111b020d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"aaf2f9718a6a3c761152d24cd7272130958591333fb5cb963ccc19c90d0ddb3b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3074, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "ref_doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "chroma:document": "# Templates for Chat Models\n\n## Introduction\n\nAn increasingly common use case for LLMs is **chat**. In a chat context, rather than continuing a single string\nof text (as is the case with a standard language model), the model instead continues a conversation that consists\nof one or more **messages**, each of which includes a **role**, like \"user\" or \"assistant\", as well as message text.\n\nMuch like tokenization, different models expect very different input formats for chat. This is the reason we added\n**chat templates** as a feature. Chat templates are part of the tokenizer. They specify how to convert conversations, \nrepresented as lists of messages, into a single tokenizable string in the format that the model expects. \n\nLet's make this concrete with a quick example using the `BlenderBot` model. BlenderBot has an extremely simple default \ntemplate, which mostly just adds whitespace between rounds of dialogue:\n\n```python\n>>> from transformers import AutoTokenizer\n>>> tokenizer = AutoTokenizer.from_pretrained(\"facebook/blenderbot-400M-distill\")\n\n>>> chat = [\n...    {\"role\": \"user\", \"content\": \"Hello, how are you?\"},\n...    {\"role\": \"assistant\", \"content\": \"I'm doing great. How can I help you today?\"},\n...    {\"role\": \"user\", \"content\": \"I'd like to show off how chat templating works!\"},\n... ]\n\n>>> tokenizer.apply_chat_template(chat, tokenize=False)\n\" Hello, how are you?  I'm doing great. How can I help you today?   I'd like to show off how chat templating works!</s>\"\n```\n\nNotice how the entire chat is condensed into a single string. If we use `tokenize=True`, which is the default setting,\nthat string will also be tokenized for us. To see a more complex template in action, though, let's use the \n`mistralai/Mistral-7B-Instruct-v0.1` model.\n\n```python\n>>> from transformers import AutoTokenizer\n>>> tokenizer = AutoTokenizer.from_pretrained(\"mistralai/Mistral-7B-Instruct-v0.1\")\n\n>>> chat = [\n...   {\"role\": \"user\", \"content\": \"Hello, how are you?\"},\n...   {\"role\": \"assistant\", \"content\": \"I'm doing great. How can I help you today?\"},\n...   {\"role\": \"user\", \"content\": \"I'd like to show off how chat templating works!\"},\n... ]\n\n>>> tokenizer.apply_chat_template(chat, tokenize=False)\n\"<s>[INST] Hello, how are you? [/INST]I'm doing great. How can I help you today?</s> [INST] I'd like to show off how chat templating works! [/INST]\"\n```\n\nNote that this time, the tokenizer has added the control tokens [INST] and [/INST] to indicate the start and end of \nuser messages (but not assistant messages!). Mistral-instruct was trained with these tokens, but BlenderBot was not.\n\n## How do I use chat templates?\n\nAs you can see in the example above, chat templates are easy to use. Simply build a list of messages, with `role`\nand `content` keys, and then pass it to the [`~PreTrainedTokenizer.apply_chat_template`] method. Once you do that,\nyou'll get output that's ready to go! When using chat templates as input for model generation, it's also a good idea\nto use `add_generation_prompt=True` to add a [generation prompt](#what-are-generation-prompts)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 54374f90-6c4a-4a8a-8db6-43b2111b020d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"54374f90-6c4a-4a8a-8db6-43b2111b020d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"544b1cdb-2cc9-4645-adb9-558d399cc3e1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ddb9a2b0-da44-48ec-8f43-b5ce2ef18918\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"1d277dccbd88ca7d48b0da5d8e53973f43876e0bce175cb78c7da90b33238930\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"027683d4-8f1c-4607-90d7-056e9efc3a3f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e25898941bcd9e254bdf0854b82d2f2b42af8802a836f50eca862bc3e83d739b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1657, \"end_char_idx\": 4667, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "ref_doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "chroma:document": "To see a more complex template in action, though, let's use the \n`mistralai/Mistral-7B-Instruct-v0.1` model.\n\n```python\n>>> from transformers import AutoTokenizer\n>>> tokenizer = AutoTokenizer.from_pretrained(\"mistralai/Mistral-7B-Instruct-v0.1\")\n\n>>> chat = [\n...   {\"role\": \"user\", \"content\": \"Hello, how are you?\"},\n...   {\"role\": \"assistant\", \"content\": \"I'm doing great. How can I help you today?\"},\n...   {\"role\": \"user\", \"content\": \"I'd like to show off how chat templating works!\"},\n... ]\n\n>>> tokenizer.apply_chat_template(chat, tokenize=False)\n\"<s>[INST] Hello, how are you? [/INST]I'm doing great. How can I help you today?</s> [INST] I'd like to show off how chat templating works! [/INST]\"\n```\n\nNote that this time, the tokenizer has added the control tokens [INST] and [/INST] to indicate the start and end of \nuser messages (but not assistant messages!). Mistral-instruct was trained with these tokens, but BlenderBot was not.\n\n## How do I use chat templates?\n\nAs you can see in the example above, chat templates are easy to use. Simply build a list of messages, with `role`\nand `content` keys, and then pass it to the [`~PreTrainedTokenizer.apply_chat_template`] method. Once you do that,\nyou'll get output that's ready to go! When using chat templates as input for model generation, it's also a good idea\nto use `add_generation_prompt=True` to add a [generation prompt](#what-are-generation-prompts). \n\nHere's an example of preparing input for `model.generate()`, using the `Zephyr` assistant model:\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\ncheckpoint = \"HuggingFaceH4/zephyr-7b-beta\"\ntokenizer = AutoTokenizer.from_pretrained(checkpoint)\nmodel = AutoModelForCausalLM.from_pretrained(checkpoint)  # You may want to use bfloat16 and/or move to GPU here\n\nmessages = [\n    {\n        \"role\": \"system\",\n        \"content\": \"You are a friendly chatbot who always responds in the style of a pirate\",\n    },\n    {\"role\": \"user\", \"content\": \"How many helicopters can a human eat in one sitting?\"},\n ]\ntokenized_chat = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors=\"pt\")\nprint(tokenizer.decode(tokenized_chat[0]))\n```\nThis will yield a string in the input format that Zephyr expects. \n```text\n<|system|>\nYou are a friendly chatbot who always responds in the style of a pirate</s> \n<|user|>\nHow many helicopters can a human eat in one sitting?</s> \n<|assistant|>\n```\n\nNow that our input is formatted correctly for Zephyr, we can use the model to generate a response to the user's question:\n\n```python\noutputs = model.generate(tokenized_chat, max_new_tokens=128) \nprint(tokenizer.decode(outputs[0]))\n```\n\nThis will yield:\n\n```text\n<|system|>\nYou are a friendly chatbot who always responds in the style of a pirate</s> \n<|user|>\nHow many helicopters can a human eat in one sitting?</s> \n<|assistant|>\nMatey, I'm afraid I must inform ye that humans cannot eat helicopters. Helicopters are not food, they are flying machines."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 027683d4-8f1c-4607-90d7-056e9efc3a3f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"027683d4-8f1c-4607-90d7-056e9efc3a3f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"544b1cdb-2cc9-4645-adb9-558d399cc3e1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"54374f90-6c4a-4a8a-8db6-43b2111b020d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"15655ea7b342721aa2b8a6168f703ea5a7b97a78cdff0ff043779cc5e0258263\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a94fee9d-0f1f-4032-8f09-5fa3510ab848\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f9f544acf6838481cccf8e0cb3f215a2b7dd1199c83fd4f9a26b4699fe3f0e14\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3077, \"end_char_idx\": 6139, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "ref_doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "chroma:document": "Here's an example of preparing input for `model.generate()`, using the `Zephyr` assistant model:\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\ncheckpoint = \"HuggingFaceH4/zephyr-7b-beta\"\ntokenizer = AutoTokenizer.from_pretrained(checkpoint)\nmodel = AutoModelForCausalLM.from_pretrained(checkpoint)  # You may want to use bfloat16 and/or move to GPU here\n\nmessages = [\n    {\n        \"role\": \"system\",\n        \"content\": \"You are a friendly chatbot who always responds in the style of a pirate\",\n    },\n    {\"role\": \"user\", \"content\": \"How many helicopters can a human eat in one sitting?\"},\n ]\ntokenized_chat = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors=\"pt\")\nprint(tokenizer.decode(tokenized_chat[0]))\n```\nThis will yield a string in the input format that Zephyr expects. \n```text\n<|system|>\nYou are a friendly chatbot who always responds in the style of a pirate</s> \n<|user|>\nHow many helicopters can a human eat in one sitting?</s> \n<|assistant|>\n```\n\nNow that our input is formatted correctly for Zephyr, we can use the model to generate a response to the user's question:\n\n```python\noutputs = model.generate(tokenized_chat, max_new_tokens=128) \nprint(tokenizer.decode(outputs[0]))\n```\n\nThis will yield:\n\n```text\n<|system|>\nYou are a friendly chatbot who always responds in the style of a pirate</s> \n<|user|>\nHow many helicopters can a human eat in one sitting?</s> \n<|assistant|>\nMatey, I'm afraid I must inform ye that humans cannot eat helicopters. Helicopters are not food, they are flying machines. Food is meant to be eaten, like a hearty plate o' grog, a savory bowl o' stew, or a delicious loaf o' bread. But helicopters, they be for transportin' and movin' around, not for eatin'. So, I'd say none, me hearties. None at all.\n```\n\nArr, 'twas easy after all!\n\n## Is there an automated pipeline for chat?\n\nYes, there is! Our text generation pipelines support chat inputs, which makes it easy to use chat models. In the past,\nwe used to use a dedicated \"ConversationalPipeline\" class, but this has now been deprecated and its functionality\nhas been merged into the [`TextGenerationPipeline`]. Let's try the `Zephyr` example again, but this time using \na pipeline:\n\n```python\nfrom transformers import pipeline\n\npipe = pipeline(\"text-generation\", \"HuggingFaceH4/zephyr-7b-beta\")\nmessages = [\n    {\n        \"role\": \"system\",\n        \"content\": \"You are a friendly chatbot who always responds in the style of a pirate\",\n    },\n    {\"role\": \"user\", \"content\": \"How many helicopters can a human eat in one sitting?\"},\n]\nprint(pipe(messages, max_new_tokens=128)[0]['generated_text'][-1])  # Print the assistant's response\n```\n\n```text\n{'role': 'assistant', 'content': \"Matey, I'm afraid I must inform ye that humans cannot eat helicopters. Helicopters are not food, they are flying machines. Food is meant to be eaten, like a hearty plate o' grog, a savory bowl o' stew, or a delicious loaf o' bread. But helicopters, they be for transportin' and movin' around, not for eatin'."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: a94fee9d-0f1f-4032-8f09-5fa3510ab848
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"a94fee9d-0f1f-4032-8f09-5fa3510ab848\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"544b1cdb-2cc9-4645-adb9-558d399cc3e1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"027683d4-8f1c-4607-90d7-056e9efc3a3f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"b60584dc5b44c40d2a48eb8f1a82e2b23facc385e05070881257d6f34dbb77e1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"320802b0-03b6-4a74-ae00-0ca9b36a5d8a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b7f27af8fe5a32cfaa70506fe28a9d578d588fb697a2d7c25045332f6db5780b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4616, \"end_char_idx\": 6867, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "ref_doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "chroma:document": "Helicopters are not food, they are flying machines. Food is meant to be eaten, like a hearty plate o' grog, a savory bowl o' stew, or a delicious loaf o' bread. But helicopters, they be for transportin' and movin' around, not for eatin'. So, I'd say none, me hearties. None at all.\n```\n\nArr, 'twas easy after all!\n\n## Is there an automated pipeline for chat?\n\nYes, there is! Our text generation pipelines support chat inputs, which makes it easy to use chat models. In the past,\nwe used to use a dedicated \"ConversationalPipeline\" class, but this has now been deprecated and its functionality\nhas been merged into the [`TextGenerationPipeline`]. Let's try the `Zephyr` example again, but this time using \na pipeline:\n\n```python\nfrom transformers import pipeline\n\npipe = pipeline(\"text-generation\", \"HuggingFaceH4/zephyr-7b-beta\")\nmessages = [\n    {\n        \"role\": \"system\",\n        \"content\": \"You are a friendly chatbot who always responds in the style of a pirate\",\n    },\n    {\"role\": \"user\", \"content\": \"How many helicopters can a human eat in one sitting?\"},\n]\nprint(pipe(messages, max_new_tokens=128)[0]['generated_text'][-1])  # Print the assistant's response\n```\n\n```text\n{'role': 'assistant', 'content': \"Matey, I'm afraid I must inform ye that humans cannot eat helicopters. Helicopters are not food, they are flying machines. Food is meant to be eaten, like a hearty plate o' grog, a savory bowl o' stew, or a delicious loaf o' bread. But helicopters, they be for transportin' and movin' around, not for eatin'. So, I'd say none, me hearties. None at all.\"}\n```\n\nThe pipeline will take care of all the details of tokenization and calling `apply_chat_template` for you -\nonce the model has a chat template, all you need to do is initialize the pipeline and pass it the list of messages!\n\n## What are \"generation prompts\"?\n\nYou may have noticed that the `apply_chat_template` method has an `add_generation_prompt` argument. This argument tells\nthe template to add tokens that indicate the start of a bot response. For example, consider the following chat:\n\n```python\nmessages = [\n    {\"role\": \"user\", \"content\": \"Hi there!\"},\n    {\"role\": \"assistant\", \"content\": \"Nice to meet you!\"},\n    {\"role\": \"user\", \"content\": \"Can I ask a question?\"}"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 320802b0-03b6-4a74-ae00-0ca9b36a5d8a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"320802b0-03b6-4a74-ae00-0ca9b36a5d8a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"544b1cdb-2cc9-4645-adb9-558d399cc3e1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a94fee9d-0f1f-4032-8f09-5fa3510ab848\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"2b5ec3881d0222ea9fa2886ec58cd52d111b7b475ae795f5087d4245ed978dfc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"faf0c33d-6929-49b6-8c6b-f302dd41066e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f0649922fc5861e201401aeab73c8d44558eb136d0cb43825cb924faad9f89ab\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5677, \"end_char_idx\": 8829, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "ref_doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "chroma:document": "\"},\n]\nprint(pipe(messages, max_new_tokens=128)[0]['generated_text'][-1])  # Print the assistant's response\n```\n\n```text\n{'role': 'assistant', 'content': \"Matey, I'm afraid I must inform ye that humans cannot eat helicopters. Helicopters are not food, they are flying machines. Food is meant to be eaten, like a hearty plate o' grog, a savory bowl o' stew, or a delicious loaf o' bread. But helicopters, they be for transportin' and movin' around, not for eatin'. So, I'd say none, me hearties. None at all.\"}\n```\n\nThe pipeline will take care of all the details of tokenization and calling `apply_chat_template` for you -\nonce the model has a chat template, all you need to do is initialize the pipeline and pass it the list of messages!\n\n## What are \"generation prompts\"?\n\nYou may have noticed that the `apply_chat_template` method has an `add_generation_prompt` argument. This argument tells\nthe template to add tokens that indicate the start of a bot response. For example, consider the following chat:\n\n```python\nmessages = [\n    {\"role\": \"user\", \"content\": \"Hi there!\"},\n    {\"role\": \"assistant\", \"content\": \"Nice to meet you!\"},\n    {\"role\": \"user\", \"content\": \"Can I ask a question?\"}\n]\n```\n\nHere's what this will look like without a generation prompt, using the ChatML template we saw in the Zephyr example:\n\n```python\ntokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=False)\n\"\"\"<|im_start|>user\nHi there!<|im_end|>\n<|im_start|>assistant\nNice to meet you!<|im_end|>\n<|im_start|>user\nCan I ask a question?<|im_end|>\n\"\"\"\n```\n\nAnd here's what it looks like **with** a generation prompt:\n\n```python\ntokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)\n\"\"\"<|im_start|>user\nHi there!<|im_end|>\n<|im_start|>assistant\nNice to meet you!<|im_end|>\n<|im_start|>user\nCan I ask a question?<|im_end|>\n<|im_start|>assistant\n\"\"\"\n```\n\nNote that this time, we've added the tokens that indicate the start of a bot response. This ensures that when the model\ngenerates text it will write a bot response instead of doing something unexpected, like continuing the user's \nmessage. Remember, chat models are still just language models - they're trained to continue text, and chat is just a \nspecial kind of text to them! You need to guide them with appropriate control tokens, so they know what they're \nsupposed to be doing.\n\nNot all models require generation prompts. Some models, like BlenderBot and LLaMA, don't have any\nspecial tokens before bot responses. In these cases, the `add_generation_prompt` argument will have no effect. The exact\neffect that `add_generation_prompt` has will depend on the template being used.\n\n## Can I use chat templates in training?\n\nYes! This is a good way to ensure that the chat template matches the tokens the model sees during training.\nWe recommend that you apply the chat template as a preprocessing step for your dataset. After this, you\ncan simply continue like any other language model training task. When training, you should usually set \n`add_generation_prompt=False`, because the added tokens to prompt an assistant response will not be helpful during \ntraining."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: faf0c33d-6929-49b6-8c6b-f302dd41066e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"faf0c33d-6929-49b6-8c6b-f302dd41066e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"544b1cdb-2cc9-4645-adb9-558d399cc3e1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"320802b0-03b6-4a74-ae00-0ca9b36a5d8a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8d46a33000306c2fa7627e4fc929570a9288c3b1569c82172c5577503d63afa2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a04f2683-e3ef-43c1-8b6f-81a07bb9051d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f08c3ff1288ed11d4fdabe5de208e5e1203b3d1df44d1cc93033b09803551eb2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7648, \"end_char_idx\": 11113, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "ref_doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "chroma:document": "This ensures that when the model\ngenerates text it will write a bot response instead of doing something unexpected, like continuing the user's \nmessage. Remember, chat models are still just language models - they're trained to continue text, and chat is just a \nspecial kind of text to them! You need to guide them with appropriate control tokens, so they know what they're \nsupposed to be doing.\n\nNot all models require generation prompts. Some models, like BlenderBot and LLaMA, don't have any\nspecial tokens before bot responses. In these cases, the `add_generation_prompt` argument will have no effect. The exact\neffect that `add_generation_prompt` has will depend on the template being used.\n\n## Can I use chat templates in training?\n\nYes! This is a good way to ensure that the chat template matches the tokens the model sees during training.\nWe recommend that you apply the chat template as a preprocessing step for your dataset. After this, you\ncan simply continue like any other language model training task. When training, you should usually set \n`add_generation_prompt=False`, because the added tokens to prompt an assistant response will not be helpful during \ntraining. Let's see an example:\n\n```python\nfrom transformers import AutoTokenizer\nfrom datasets import Dataset\n\ntokenizer = AutoTokenizer.from_pretrained(\"HuggingFaceH4/zephyr-7b-beta\")\n\nchat1 = [\n    {\"role\": \"user\", \"content\": \"Which is bigger, the moon or the sun?\"},\n    {\"role\": \"assistant\", \"content\": \"The sun.\"}\n]\nchat2 = [\n    {\"role\": \"user\", \"content\": \"Which is bigger, a virus or a bacterium?\"},\n    {\"role\": \"assistant\", \"content\": \"A bacterium.\"}\n]\n\ndataset = Dataset.from_dict({\"chat\": [chat1, chat2]})\ndataset = dataset.map(lambda x: {\"formatted_chat\": tokenizer.apply_chat_template(x[\"chat\"], tokenize=False, add_generation_prompt=False)})\nprint(dataset['formatted_chat'][0])\n```\nAnd we get:\n```text\n<|user|>\nWhich is bigger, the moon or the sun?</s>\n<|assistant|>\nThe sun.</s>\n```\n\nFrom here, just continue training like you would with a standard language modelling task, using the `formatted_chat` column.\n\n<Tip>\nIf you format text with `apply_chat_template(tokenize=False)` and then tokenize it in a separate step, you should set the argument\n`add_special_tokens=False`. If you use `apply_chat_template(tokenize=True)`, you don't need to worry about this!\n\nBy default, some tokenizers add special tokens like `<bos>` and `<eos>` to text they tokenize. Chat templates should \nalways include all of the special tokens they need, and so adding extra special tokens with\nthe default `add_special_tokens=True` can result in incorrect or duplicated special tokens, which will hurt model\nperformance.\n</Tip>\n\n## Advanced: Extra inputs to chat templates\n\nThe only argument that `apply_chat_template` requires is `messages`. However, you can pass any keyword\nargument to `apply_chat_template` and it will be accessible inside the template. This gives you a lot of freedom to use\nchat templates for many things. There are no restrictions on the names or the format of these arguments - you can pass\nstrings, lists, dicts or whatever else you want. \n\nThat said, there are some common use-cases for these extra arguments,\nsuch as passing tools for function calling, or documents for retrieval-augmented generation. In these common cases,\nwe have some opinionated recommendations about what the names and formats of these arguments should be, which are\ndescribed in the sections below."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: a04f2683-e3ef-43c1-8b6f-81a07bb9051d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"a04f2683-e3ef-43c1-8b6f-81a07bb9051d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"544b1cdb-2cc9-4645-adb9-558d399cc3e1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"faf0c33d-6929-49b6-8c6b-f302dd41066e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8911e65bf29946376b8191e029838d0ca16609ef84ef53a2784f8033488f0cc0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"968ef10a-9063-48e1-a5c4-515679695989\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"078d3159ccb378f9113bf75b886b87054273edcb8a4d98ec3596a8194a692e90\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9747, \"end_char_idx\": 13181, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "ref_doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "chroma:document": "<Tip>\nIf you format text with `apply_chat_template(tokenize=False)` and then tokenize it in a separate step, you should set the argument\n`add_special_tokens=False`. If you use `apply_chat_template(tokenize=True)`, you don't need to worry about this!\n\nBy default, some tokenizers add special tokens like `<bos>` and `<eos>` to text they tokenize. Chat templates should \nalways include all of the special tokens they need, and so adding extra special tokens with\nthe default `add_special_tokens=True` can result in incorrect or duplicated special tokens, which will hurt model\nperformance.\n</Tip>\n\n## Advanced: Extra inputs to chat templates\n\nThe only argument that `apply_chat_template` requires is `messages`. However, you can pass any keyword\nargument to `apply_chat_template` and it will be accessible inside the template. This gives you a lot of freedom to use\nchat templates for many things. There are no restrictions on the names or the format of these arguments - you can pass\nstrings, lists, dicts or whatever else you want. \n\nThat said, there are some common use-cases for these extra arguments,\nsuch as passing tools for function calling, or documents for retrieval-augmented generation. In these common cases,\nwe have some opinionated recommendations about what the names and formats of these arguments should be, which are\ndescribed in the sections below. We encourage model authors to make their chat templates compatible with this format,\nto make it easy to transfer tool-calling code between models.\n\n## Advanced: Tool use / function calling\n\n\"Tool use\" LLMs can choose to call functions as external tools before generating an answer. When passing tools\nto a tool-use model, you can simply pass a list of functions to the `tools` argument:\n\n```python\nimport datetime\n\ndef current_time():\n    \"\"\"Get the current local time as a string.\"\"\"\n    return str(datetime.now())\n\ndef multiply(a: float, b: float):\n    \"\"\"\n    A function that multiplies two numbers\n    \n    Args:\n        a: The first number to multiply\n        b: The second number to multiply\n    \"\"\"\n    return a * b\n\ntools = [current_time, multiply]\n\nmodel_input = tokenizer.apply_chat_template(\n    messages,\n    tools=tools\n)\n```\n\nIn order for this to work correctly, you should write your functions in the format above, so that they can be parsed\ncorrectly as tools. Specifically, you should follow these rules:\n\n- The function should have a descriptive name\n- Every argument must have a type hint\n- The function must have a docstring in the standard Google style (in other words, an initial function description  \n  followed by an `Args:` block that describes the arguments, unless the function does not have any arguments. \n- Do not include types in the `Args:` block. In other words, write `a: The first number to multiply`, not\n  `a (int): The first number to multiply`. Type hints should go in the function header instead.\n- The function can have a return type and a `Returns:` block in the docstring. However, these are optional\n  because most tool-use models ignore them.\n\n### Passing tool results to the model\n\nThe sample code above is enough to list the available tools for your model, but what happens if it wants to actually use\none? If that happens, you should:\n\n1. Parse the model's output to get the tool name(s) and arguments.\n2. Add the model's tool call(s) to the conversation.\n3. Call the corresponding function(s) with those arguments.\n4."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 968ef10a-9063-48e1-a5c4-515679695989
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"968ef10a-9063-48e1-a5c4-515679695989\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"544b1cdb-2cc9-4645-adb9-558d399cc3e1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a04f2683-e3ef-43c1-8b6f-81a07bb9051d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"5c02aa5cbf37f2b03e31a69067e7b8041a342cf3bd006a4f3bf03bb7fc777de6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ec746d55-c0f6-419e-b8c9-b174ad8c087c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d78e5b774a9839cde82bd2dae5430ad22c6bbc77bcfa035ce773a51d3b34ea5f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11603, \"end_char_idx\": 14801, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "ref_doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "chroma:document": "return str(datetime.now())\n\ndef multiply(a: float, b: float):\n    \"\"\"\n    A function that multiplies two numbers\n    \n    Args:\n        a: The first number to multiply\n        b: The second number to multiply\n    \"\"\"\n    return a * b\n\ntools = [current_time, multiply]\n\nmodel_input = tokenizer.apply_chat_template(\n    messages,\n    tools=tools\n)\n```\n\nIn order for this to work correctly, you should write your functions in the format above, so that they can be parsed\ncorrectly as tools. Specifically, you should follow these rules:\n\n- The function should have a descriptive name\n- Every argument must have a type hint\n- The function must have a docstring in the standard Google style (in other words, an initial function description  \n  followed by an `Args:` block that describes the arguments, unless the function does not have any arguments. \n- Do not include types in the `Args:` block. In other words, write `a: The first number to multiply`, not\n  `a (int): The first number to multiply`. Type hints should go in the function header instead.\n- The function can have a return type and a `Returns:` block in the docstring. However, these are optional\n  because most tool-use models ignore them.\n\n### Passing tool results to the model\n\nThe sample code above is enough to list the available tools for your model, but what happens if it wants to actually use\none? If that happens, you should:\n\n1. Parse the model's output to get the tool name(s) and arguments.\n2. Add the model's tool call(s) to the conversation.\n3. Call the corresponding function(s) with those arguments.\n4. Add the result(s) to the conversation\n\n### A complete tool use example\n\nLet's walk through a tool use example, step by step. For this example, we will use an 8B `Hermes-2-Pro` model,\nas it is one of the highest-performing tool-use models in its size category at the time of writing. If you have the\nmemory, you can consider using a larger model instead like [Command-R](https://huggingface.co/CohereForAI/c4ai-command-r-v01)\nor [Mixtral-8x22B](https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1), both of which also support tool use\nand offer even stronger performance.\n\nFirst, let's load our model and tokenizer:\n\n```python\nimport torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\ncheckpoint = \"NousResearch/Hermes-2-Pro-Llama-3-8B\"\n\ntokenizer = AutoTokenizer.from_pretrained(checkpoint, revision=\"pr/13\")\nmodel = AutoModelForCausalLM.from_pretrained(checkpoint, torch_dtype=torch.bfloat16, device_map=\"auto\")\n```\n\nNext, let's define a list of tools:\n\n```python\ndef get_current_temperature(location: str, unit: str) -> float:\n    \"\"\"\n    Get the current temperature at a location.\n    \n    Args:\n        location: The location to get the temperature for, in the format \"City, Country\"\n        unit: The unit to return the temperature in. (choices: [\"celsius\", \"fahrenheit\"])\n    Returns:\n        The current temperature at the specified location in the specified units, as a float.\n    \"\"\"\n    return 22.  # A real function should probably actually get the temperature!\n\ndef get_current_wind_speed(location: str) -> float:\n    \"\"\"\n    Get the current wind speed in km/h at a given location."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: ec746d55-c0f6-419e-b8c9-b174ad8c087c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"ec746d55-c0f6-419e-b8c9-b174ad8c087c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"544b1cdb-2cc9-4645-adb9-558d399cc3e1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"968ef10a-9063-48e1-a5c4-515679695989\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"10278b1e9fb039aa5901b75a910a9def302bec7e499b509bf8ce42d10b53fdfb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b0a18978-a762-4bb0-baaf-517897c124f7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a7bc31e4dbe6b809646f0e3d0fa24d48d8bda94641c720dc3f94d18cfc4b9827\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13307, \"end_char_idx\": 16415, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "ref_doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "chroma:document": "For this example, we will use an 8B `Hermes-2-Pro` model,\nas it is one of the highest-performing tool-use models in its size category at the time of writing. If you have the\nmemory, you can consider using a larger model instead like [Command-R](https://huggingface.co/CohereForAI/c4ai-command-r-v01)\nor [Mixtral-8x22B](https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1), both of which also support tool use\nand offer even stronger performance.\n\nFirst, let's load our model and tokenizer:\n\n```python\nimport torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\ncheckpoint = \"NousResearch/Hermes-2-Pro-Llama-3-8B\"\n\ntokenizer = AutoTokenizer.from_pretrained(checkpoint, revision=\"pr/13\")\nmodel = AutoModelForCausalLM.from_pretrained(checkpoint, torch_dtype=torch.bfloat16, device_map=\"auto\")\n```\n\nNext, let's define a list of tools:\n\n```python\ndef get_current_temperature(location: str, unit: str) -> float:\n    \"\"\"\n    Get the current temperature at a location.\n    \n    Args:\n        location: The location to get the temperature for, in the format \"City, Country\"\n        unit: The unit to return the temperature in. (choices: [\"celsius\", \"fahrenheit\"])\n    Returns:\n        The current temperature at the specified location in the specified units, as a float.\n    \"\"\"\n    return 22.  # A real function should probably actually get the temperature!\n\ndef get_current_wind_speed(location: str) -> float:\n    \"\"\"\n    Get the current wind speed in km/h at a given location.\n    \n    Args:\n        location: The location to get the temperature for, in the format \"City, Country\"\n    Returns:\n        The current wind speed at the given location in km/h, as a float.\n    \"\"\"\n    return 6.  # A real function should probably actually get the wind speed!\n\ntools = [get_current_temperature, get_current_wind_speed]\n```\n\nNow, let's set up a conversation for our bot:\n\n```python\nmessages = [\n  {\"role\": \"system\", \"content\": \"You are a bot that responds to weather queries. You should reply with the unit used in the queried location.\"},\n  {\"role\": \"user\", \"content\": \"Hey, what's the temperature in Paris right now?\"}\n]\n```\n\nNow, let's apply the chat template and generate a response:\n\n```python\ninputs = tokenizer.apply_chat_template(messages, chat_template=\"tool_use\", tools=tools, add_generation_prompt=True, return_dict=True, return_tensors=\"pt\")\ninputs = {k: v.to(model.device) for k, v in inputs.items()}\nout = model.generate(**inputs, max_new_tokens=128)\nprint(tokenizer.decode(out[0][len(inputs[\"input_ids\"][0]):]))\n```\n\nAnd we get:\n\n```text\n<tool_call>\n{\"arguments\": {\"location\": \"Paris, France\", \"unit\": \"celsius\"}, \"name\": \"get_current_temperature\"}\n</tool_call><|im_end|>\n```\n\nThe model has called the function with valid arguments, in the format requested by the function docstring. It has\ninferred that we're most likely referring to the Paris in France, and it remembered that, as the home of SI units,\nthe temperature in France should certainly be displayed in Celsius.\n\nLet's append the model's tool call to the conversation. Note that we generate a random `tool_call_id` here."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: b0a18978-a762-4bb0-baaf-517897c124f7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"b0a18978-a762-4bb0-baaf-517897c124f7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"544b1cdb-2cc9-4645-adb9-558d399cc3e1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ec746d55-c0f6-419e-b8c9-b174ad8c087c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"38b7fcdf7c28ca895a9de8fff8d6bba2b5b1ca6f8e3ebcae859ab9ef2c9a645f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"46239ce9-0666-45f3-9e39-f6310134ac82\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b34d5bffbabd22b89741ac038fdc025dbbb8b2ac212943994a3efcce839b85a0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14811, \"end_char_idx\": 17314, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "ref_doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "chroma:document": "Args:\n        location: The location to get the temperature for, in the format \"City, Country\"\n    Returns:\n        The current wind speed at the given location in km/h, as a float.\n    \"\"\"\n    return 6.  # A real function should probably actually get the wind speed!\n\ntools = [get_current_temperature, get_current_wind_speed]\n```\n\nNow, let's set up a conversation for our bot:\n\n```python\nmessages = [\n  {\"role\": \"system\", \"content\": \"You are a bot that responds to weather queries. You should reply with the unit used in the queried location.\"},\n  {\"role\": \"user\", \"content\": \"Hey, what's the temperature in Paris right now?\"}\n]\n```\n\nNow, let's apply the chat template and generate a response:\n\n```python\ninputs = tokenizer.apply_chat_template(messages, chat_template=\"tool_use\", tools=tools, add_generation_prompt=True, return_dict=True, return_tensors=\"pt\")\ninputs = {k: v.to(model.device) for k, v in inputs.items()}\nout = model.generate(**inputs, max_new_tokens=128)\nprint(tokenizer.decode(out[0][len(inputs[\"input_ids\"][0]):]))\n```\n\nAnd we get:\n\n```text\n<tool_call>\n{\"arguments\": {\"location\": \"Paris, France\", \"unit\": \"celsius\"}, \"name\": \"get_current_temperature\"}\n</tool_call><|im_end|>\n```\n\nThe model has called the function with valid arguments, in the format requested by the function docstring. It has\ninferred that we're most likely referring to the Paris in France, and it remembered that, as the home of SI units,\nthe temperature in France should certainly be displayed in Celsius.\n\nLet's append the model's tool call to the conversation. Note that we generate a random `tool_call_id` here. These IDs\nare not used by all models, but they allow models to issue multiple tool calls at once and keep track of which response\ncorresponds to which call. You can generate them any way you like, but they should be unique within each chat.\n\n```python\ntool_call_id = \"vAHdf3\"  # Random ID, should be unique for each tool call\ntool_call = {\"name\": \"get_current_temperature\", \"arguments\": {\"location\": \"Paris, France\", \"unit\": \"celsius\"}}\nmessages.append({\"role\": \"assistant\", \"tool_calls\": [{\"id\": tool_call_id, \"type\": \"function\", \"function\": tool_call}]})\n```\n\n\nNow that we've added the tool call to the conversation, we can call the function and append the result to the\nconversation. Since we're just using a dummy function for this example that always returns 22.0, we can just append \nthat result directly. Again, note the `tool_call_id` - this should match the ID used in the tool call above."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 46239ce9-0666-45f3-9e39-f6310134ac82
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"46239ce9-0666-45f3-9e39-f6310134ac82\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"544b1cdb-2cc9-4645-adb9-558d399cc3e1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b0a18978-a762-4bb0-baaf-517897c124f7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"04906b7739c50d99bd1f45ff58d86746eb06229317587e5d5c89c40d96a039be\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4180a3b7-ffe3-4ccd-bdf2-d08af12ed4f1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5260ffdc090236272bcf58e7e83915169d4a9b462596dcbcb7f9733747a64dc5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 16117, \"end_char_idx\": 19434, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "ref_doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "chroma:document": "It has\ninferred that we're most likely referring to the Paris in France, and it remembered that, as the home of SI units,\nthe temperature in France should certainly be displayed in Celsius.\n\nLet's append the model's tool call to the conversation. Note that we generate a random `tool_call_id` here. These IDs\nare not used by all models, but they allow models to issue multiple tool calls at once and keep track of which response\ncorresponds to which call. You can generate them any way you like, but they should be unique within each chat.\n\n```python\ntool_call_id = \"vAHdf3\"  # Random ID, should be unique for each tool call\ntool_call = {\"name\": \"get_current_temperature\", \"arguments\": {\"location\": \"Paris, France\", \"unit\": \"celsius\"}}\nmessages.append({\"role\": \"assistant\", \"tool_calls\": [{\"id\": tool_call_id, \"type\": \"function\", \"function\": tool_call}]})\n```\n\n\nNow that we've added the tool call to the conversation, we can call the function and append the result to the\nconversation. Since we're just using a dummy function for this example that always returns 22.0, we can just append \nthat result directly. Again, note the `tool_call_id` - this should match the ID used in the tool call above.\n\n```python\nmessages.append({\"role\": \"tool\", \"tool_call_id\": tool_call_id, \"name\": \"get_current_temperature\", \"content\": \"22.0\"})\n```\n\nFinally, let's let the assistant read the function outputs and continue chatting with the user:\n\n```python\ninputs = tokenizer.apply_chat_template(messages, chat_template=\"tool_use\", tools=tools, add_generation_prompt=True, return_dict=True, return_tensors=\"pt\")\ninputs = {k: v.to(model.device) for k, v in inputs.items()}\nout = model.generate(**inputs, max_new_tokens=128)\nprint(tokenizer.decode(out[0][len(inputs[\"input_ids\"][0]):]))\n```\n\nAnd we get:\n\n```text\nThe current temperature in Paris, France is 22.0 \u00b0 Celsius.<|im_end|>\n```\n\nAlthough this was a simple demo with dummy tools and a single call, the same technique works with \nmultiple real tools and longer conversations. This can be a powerful way to extend the capabilities of conversational\nagents with real-time information, computational tools like calculators, or access to large databases.\n\n<Tip>\nNot all of the tool-calling features shown above are used by all models. Some use tool call IDs, others simply use the function name and\nmatch tool calls to results using the ordering, and there are several models that use neither and only issue one tool \ncall at a time to avoid confusion. If you want your code to be compatible across as many models as possible, we \nrecommend structuring your tools calls like we've shown here, and returning tool results in the order that\nthey were issued by the model. The chat templates on each model should handle the rest.\n</Tip>\n\n### Understanding tool schemas\n\nEach function you pass to the `tools` argument of `apply_chat_template` is converted into a \n[JSON schema](https://json-schema.org/learn/getting-started-step-by-step). These schemas\nare then passed to the model chat template. In other words, tool-use models do not see your functions directly, and they\nnever see the actual code inside them. What they care about is the function **definitions** and the **arguments** they\nneed to pass to them - they care about what the tools do and how to use them, not how they work!"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 4180a3b7-ffe3-4ccd-bdf2-d08af12ed4f1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"4180a3b7-ffe3-4ccd-bdf2-d08af12ed4f1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"544b1cdb-2cc9-4645-adb9-558d399cc3e1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"46239ce9-0666-45f3-9e39-f6310134ac82\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"ebb68cf5ecf88abb4e22c20b2a7aedd95cd4f5a04c82718e05791c6b7d8e8784\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"24d9ed0e-5ffc-4f93-93c2-8162a8b7d833\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"04b122433c2f617b76cf3cc481e1e23fefc0e56c46cf8ad4fddc04dfb3e46efd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 18130, \"end_char_idx\": 21582, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "ref_doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "chroma:document": "This can be a powerful way to extend the capabilities of conversational\nagents with real-time information, computational tools like calculators, or access to large databases.\n\n<Tip>\nNot all of the tool-calling features shown above are used by all models. Some use tool call IDs, others simply use the function name and\nmatch tool calls to results using the ordering, and there are several models that use neither and only issue one tool \ncall at a time to avoid confusion. If you want your code to be compatible across as many models as possible, we \nrecommend structuring your tools calls like we've shown here, and returning tool results in the order that\nthey were issued by the model. The chat templates on each model should handle the rest.\n</Tip>\n\n### Understanding tool schemas\n\nEach function you pass to the `tools` argument of `apply_chat_template` is converted into a \n[JSON schema](https://json-schema.org/learn/getting-started-step-by-step). These schemas\nare then passed to the model chat template. In other words, tool-use models do not see your functions directly, and they\nnever see the actual code inside them. What they care about is the function **definitions** and the **arguments** they\nneed to pass to them - they care about what the tools do and how to use them, not how they work! It is up to you\nto read their outputs, detect if they have requested to use a tool, pass their arguments to the tool function, and\nreturn the response in the chat.\n\nGenerating JSON schemas to pass to the template should be automatic and invisible as long as your functions\nfollow the specification above, but if you encounter problems, or you simply want more control over the conversion, \nyou can handle the conversion manually. Here is an example of a manual schema conversion.\n\n```python\nfrom transformers.utils import get_json_schema\n\ndef multiply(a: float, b: float):\n    \"\"\"\n    A function that multiplies two numbers\n    \n    Args:\n        a: The first number to multiply\n        b: The second number to multiply\n    \"\"\"\n    return a * b\n\nschema = get_json_schema(multiply)\nprint(schema)\n```\n\nThis will yield:\n\n```json\n{\n  \"type\": \"function\", \n  \"function\": {\n    \"name\": \"multiply\", \n    \"description\": \"A function that multiplies two numbers\", \n    \"parameters\": {\n      \"type\": \"object\", \n      \"properties\": {\n        \"a\": {\n          \"type\": \"number\", \n          \"description\": \"The first number to multiply\"\n        }, \n        \"b\": {\n          \"type\": \"number\",\n          \"description\": \"The second number to multiply\"\n        }\n      }, \n      \"required\": [\"a\", \"b\"]\n    }\n  }\n}\n```\n\nIf you wish, you can edit these schemas, or even write them from scratch yourself without using `get_json_schema` at \nall. JSON schemas can be passed directly to the `tools` argument of \n`apply_chat_template` - this gives you a lot of power to define precise schemas for more complex functions. Be careful,\nthough - the more complex your schemas, the more likely the model is to get confused when dealing with them! We \nrecommend simple function signatures where possible, keeping arguments (and especially complex, nested arguments) \nto a minimum.\n\nHere is an example of defining schemas by hand, and passing them directly to `apply_chat_template`:\n\n```python\n# A simple function that takes no arguments\ncurrent_time = {\n  \"type\": \"function\", \n  \"function\": {\n    \"name\": \"current_time\",\n    \"description\": \"Get the current local time as a string."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 24d9ed0e-5ffc-4f93-93c2-8162a8b7d833
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"24d9ed0e-5ffc-4f93-93c2-8162a8b7d833\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"544b1cdb-2cc9-4645-adb9-558d399cc3e1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4180a3b7-ffe3-4ccd-bdf2-d08af12ed4f1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8d69a388d2a3eb2001035dd623088e42d7d913fe70c0180c95d3800083eaff42\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"04002efc-e1db-4a20-a7df-93ac132f0be2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4cca5d89125ec3c144f66865f56ebd168b4054814bb4ea2ce9f279916e52aedc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 19916, \"end_char_idx\": 23079, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "ref_doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "chroma:document": "```python\nfrom transformers.utils import get_json_schema\n\ndef multiply(a: float, b: float):\n    \"\"\"\n    A function that multiplies two numbers\n    \n    Args:\n        a: The first number to multiply\n        b: The second number to multiply\n    \"\"\"\n    return a * b\n\nschema = get_json_schema(multiply)\nprint(schema)\n```\n\nThis will yield:\n\n```json\n{\n  \"type\": \"function\", \n  \"function\": {\n    \"name\": \"multiply\", \n    \"description\": \"A function that multiplies two numbers\", \n    \"parameters\": {\n      \"type\": \"object\", \n      \"properties\": {\n        \"a\": {\n          \"type\": \"number\", \n          \"description\": \"The first number to multiply\"\n        }, \n        \"b\": {\n          \"type\": \"number\",\n          \"description\": \"The second number to multiply\"\n        }\n      }, \n      \"required\": [\"a\", \"b\"]\n    }\n  }\n}\n```\n\nIf you wish, you can edit these schemas, or even write them from scratch yourself without using `get_json_schema` at \nall. JSON schemas can be passed directly to the `tools` argument of \n`apply_chat_template` - this gives you a lot of power to define precise schemas for more complex functions. Be careful,\nthough - the more complex your schemas, the more likely the model is to get confused when dealing with them! We \nrecommend simple function signatures where possible, keeping arguments (and especially complex, nested arguments) \nto a minimum.\n\nHere is an example of defining schemas by hand, and passing them directly to `apply_chat_template`:\n\n```python\n# A simple function that takes no arguments\ncurrent_time = {\n  \"type\": \"function\", \n  \"function\": {\n    \"name\": \"current_time\",\n    \"description\": \"Get the current local time as a string.\",\n    \"parameters\": {\n      'type': 'object',\n      'properties': {}\n    }\n  }\n}\n\n# A more complete function that takes two numerical arguments\nmultiply = {\n  'type': 'function',\n  'function': {\n    'name': 'multiply',\n    'description': 'A function that multiplies two numbers', \n    'parameters': {\n      'type': 'object', \n      'properties': {\n        'a': {\n          'type': 'number',\n          'description': 'The first number to multiply'\n        }, \n        'b': {\n          'type': 'number', 'description': 'The second number to multiply'\n        }\n      }, \n      'required': ['a', 'b']\n    }\n  }\n}\n\nmodel_input = tokenizer.apply_chat_template(\n    messages,\n    tools = [current_time, multiply]\n)\n```\n\n## Advanced: Retrieval-augmented generation\n\n\"Retrieval-augmented generation\" or \"RAG\" LLMs can search a corpus of documents for information before responding\nto a query. This allows models to vastly expand their knowledge base beyond their limited context size. Our \nrecommendation for RAG models is that their template\nshould accept a `documents` argument. This should be a list of documents, where each \"document\"\nis a single dict with `title` and `contents` keys, both of which are strings. Because this format is much simpler\nthan the JSON schemas used for tools, no helper functions are necessary.\n\nHere's an example of a RAG template in action:\n\n```python\ndocument1 = {\n    \"title\": \"The Moon: Our Age-Old Foe\",\n    \"contents\": \"Man has always dreamed of destroying the moon."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 04002efc-e1db-4a20-a7df-93ac132f0be2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"04002efc-e1db-4a20-a7df-93ac132f0be2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"544b1cdb-2cc9-4645-adb9-558d399cc3e1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"24d9ed0e-5ffc-4f93-93c2-8162a8b7d833\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"42470a7ddbcf2e1b6bd05add1770ea8e63514d4f424a04e3f83509680643d71c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0e5d5497-fb87-44f8-8fae-a3ee4b67526c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b9957649c918f82549f1288887196479431035b7b872f14ae4124ec8b731dc7c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 21582, \"end_char_idx\": 24637, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "ref_doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "chroma:document": "\",\n    \"parameters\": {\n      'type': 'object',\n      'properties': {}\n    }\n  }\n}\n\n# A more complete function that takes two numerical arguments\nmultiply = {\n  'type': 'function',\n  'function': {\n    'name': 'multiply',\n    'description': 'A function that multiplies two numbers', \n    'parameters': {\n      'type': 'object', \n      'properties': {\n        'a': {\n          'type': 'number',\n          'description': 'The first number to multiply'\n        }, \n        'b': {\n          'type': 'number', 'description': 'The second number to multiply'\n        }\n      }, \n      'required': ['a', 'b']\n    }\n  }\n}\n\nmodel_input = tokenizer.apply_chat_template(\n    messages,\n    tools = [current_time, multiply]\n)\n```\n\n## Advanced: Retrieval-augmented generation\n\n\"Retrieval-augmented generation\" or \"RAG\" LLMs can search a corpus of documents for information before responding\nto a query. This allows models to vastly expand their knowledge base beyond their limited context size. Our \nrecommendation for RAG models is that their template\nshould accept a `documents` argument. This should be a list of documents, where each \"document\"\nis a single dict with `title` and `contents` keys, both of which are strings. Because this format is much simpler\nthan the JSON schemas used for tools, no helper functions are necessary.\n\nHere's an example of a RAG template in action:\n\n```python\ndocument1 = {\n    \"title\": \"The Moon: Our Age-Old Foe\",\n    \"contents\": \"Man has always dreamed of destroying the moon. In this essay, I shall...\"\n}\n\ndocument2 = {\n    \"title\": \"The Sun: Our Age-Old Friend\",\n    \"contents\": \"Although often underappreciated, the sun provides several notable benefits...\"\n}\n\nmodel_input = tokenizer.apply_chat_template(\n    messages,\n    documents=[document1, document2]\n)\n```\n\n## Advanced: How do chat templates work?\n\nThe chat template for a model is stored on the `tokenizer.chat_template` attribute. If no chat template is set, the\ndefault template for that model class is used instead. Let's take a look at the template for `BlenderBot`:\n\n```python\n\n>>> from transformers import AutoTokenizer\n>>> tokenizer = AutoTokenizer.from_pretrained(\"facebook/blenderbot-400M-distill\")\n\n>>> tokenizer.chat_template\n\"{% for message in messages %}{% if message['role'] == 'user' %}{{ ' ' }}{% endif %}{{ message['content'] }}{% if not loop.last %}{{ '  ' }}{% endif %}{% endfor %}{{ eos_token }}\"\n```\n\nThat's kind of intimidating. Let's clean it up a little to make it more readable. In the process, though, we also make\nsure that the newlines and indentation we add don't end up being included in the template output - see the tip on\n[trimming whitespace](#trimming-whitespace) below!\n\n```\n{%- for message in messages %}\n    {%- if message['role'] == 'user' %}\n        {{- ' ' }}\n    {%- endif %}\n    {{- message['content'] }}\n    {%- if not loop.last %}\n        {{- '  ' }}\n    {%- endif %}\n{%- endfor %}\n{{- eos_token }}\n```\n\nIf you've never seen one of these before, this is a [Jinja template](https://jinja.palletsprojects.com/en/3.1.x/templates/)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 0e5d5497-fb87-44f8-8fae-a3ee4b67526c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"0e5d5497-fb87-44f8-8fae-a3ee4b67526c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"544b1cdb-2cc9-4645-adb9-558d399cc3e1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"04002efc-e1db-4a20-a7df-93ac132f0be2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"86567b7f34060d5dc025df9fce5e6764a92f3bf962da161bc573753a503fb2d9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"270d6860-0401-4bd9-9fca-f9ce13927614\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"758f1b8e3dcb8fc9afe129f6507242ba3ccb97b08e8454439838e7227e8aae66\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 23412, \"end_char_idx\": 25957, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "ref_doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "chroma:document": "The chat template for a model is stored on the `tokenizer.chat_template` attribute. If no chat template is set, the\ndefault template for that model class is used instead. Let's take a look at the template for `BlenderBot`:\n\n```python\n\n>>> from transformers import AutoTokenizer\n>>> tokenizer = AutoTokenizer.from_pretrained(\"facebook/blenderbot-400M-distill\")\n\n>>> tokenizer.chat_template\n\"{% for message in messages %}{% if message['role'] == 'user' %}{{ ' ' }}{% endif %}{{ message['content'] }}{% if not loop.last %}{{ '  ' }}{% endif %}{% endfor %}{{ eos_token }}\"\n```\n\nThat's kind of intimidating. Let's clean it up a little to make it more readable. In the process, though, we also make\nsure that the newlines and indentation we add don't end up being included in the template output - see the tip on\n[trimming whitespace](#trimming-whitespace) below!\n\n```\n{%- for message in messages %}\n    {%- if message['role'] == 'user' %}\n        {{- ' ' }}\n    {%- endif %}\n    {{- message['content'] }}\n    {%- if not loop.last %}\n        {{- '  ' }}\n    {%- endif %}\n{%- endfor %}\n{{- eos_token }}\n```\n\nIf you've never seen one of these before, this is a [Jinja template](https://jinja.palletsprojects.com/en/3.1.x/templates/).\nJinja is a templating language that allows you to write simple code that generates text. In many ways, the code and\nsyntax resembles Python. In pure Python, this template would look something like this:\n\n```python\nfor idx, message in enumerate(messages):\n    if message['role'] == 'user':\n        print(' ')\n    print(message['content'])\n    if not idx == len(messages) - 1:  # Check for the last message in the conversation\n        print('  ')\nprint(eos_token)\n```\n\nEffectively, the template does three things:\n1. For each message, if the message is a user message, add a blank space before it, otherwise print nothing.\n2. Add the message content\n3. If the message is not the last message, add two spaces after it. After the final message, print the EOS token.\n\nThis is a pretty simple template - it doesn't add any control tokens, and it doesn't support \"system\" messages, which \nare a common way to give the model directives about how it should behave in the subsequent conversation.\nBut Jinja gives you a lot of flexibility to do those things! Let's see a Jinja template that can format inputs\nsimilarly to the way LLaMA formats them (note that the real LLaMA template includes handling for default system\nmessages and slightly different system message handling in general - don't use this one in your actual code!)"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 270d6860-0401-4bd9-9fca-f9ce13927614
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"270d6860-0401-4bd9-9fca-f9ce13927614\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"544b1cdb-2cc9-4645-adb9-558d399cc3e1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0e5d5497-fb87-44f8-8fae-a3ee4b67526c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"bb536d209752276417c458a7b387306673ff94d0aa72e208ae142e0e2cf62f52\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1ab0421c-bfae-413a-8ff2-658bda3ad5c0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"270533b1c4006fc61fde43555ea9cbabb15a108a44d25cae80851f304a81defd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 24638, \"end_char_idx\": 27623, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "ref_doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "chroma:document": "Jinja is a templating language that allows you to write simple code that generates text. In many ways, the code and\nsyntax resembles Python. In pure Python, this template would look something like this:\n\n```python\nfor idx, message in enumerate(messages):\n    if message['role'] == 'user':\n        print(' ')\n    print(message['content'])\n    if not idx == len(messages) - 1:  # Check for the last message in the conversation\n        print('  ')\nprint(eos_token)\n```\n\nEffectively, the template does three things:\n1. For each message, if the message is a user message, add a blank space before it, otherwise print nothing.\n2. Add the message content\n3. If the message is not the last message, add two spaces after it. After the final message, print the EOS token.\n\nThis is a pretty simple template - it doesn't add any control tokens, and it doesn't support \"system\" messages, which \nare a common way to give the model directives about how it should behave in the subsequent conversation.\nBut Jinja gives you a lot of flexibility to do those things! Let's see a Jinja template that can format inputs\nsimilarly to the way LLaMA formats them (note that the real LLaMA template includes handling for default system\nmessages and slightly different system message handling in general - don't use this one in your actual code!)\n\n```\n{%- for message in messages %}\n    {%- if message['role'] == 'user' %}\n        {{- bos_token + '[INST] ' + message['content'] + ' [/INST]' }}\n    {%- elif message['role'] == 'system' %}\n        {{- '<<SYS>>\\\\n' + message['content'] + '\\\\n<</SYS>>\\\\n\\\\n' }}\n    {%- elif message['role'] == 'assistant' %}\n        {{- ' '  + message['content'] + ' ' + eos_token }}\n    {%- endif %}\n{%- endfor %}\n```\n\nHopefully if you stare at this for a little bit you can see what this template is doing - it adds specific tokens based\non the \"role\" of each message, which represents who sent it. User, assistant and system messages are clearly\ndistinguishable to the model because of the tokens they're wrapped in.\n\n## Advanced: Adding and editing chat templates\n\n### How do I create a chat template?\n\nSimple, just write a jinja template and set `tokenizer.chat_template`. You may find it easier to start with an \nexisting template from another model and simply edit it for your needs! For example, we could take the LLaMA template\nabove and add \"[ASST]\" and \"[/ASST]\" to assistant messages:\n\n```\n{%- for message in messages %}\n    {%- if message['role'] == 'user' %}\n        {{- bos_token + '[INST] ' + message['content'].strip() + ' [/INST]' }}\n    {%- elif message['role'] == 'system' %}\n        {{- '<<SYS>>\\\\n' + message['content'].strip() + '\\\\n<</SYS>>\\\\n\\\\n' }}\n    {%- elif message['role'] == 'assistant' %}\n        {{- '[ASST] '  + message['content'] + ' [/ASST]' + eos_token }}\n    {%- endif %}\n{%- endfor %}\n```\n\nNow, simply set the `tokenizer.chat_template` attribute. Next time you use [`~PreTrainedTokenizer.apply_chat_template`], it will\nuse your new template!"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 1ab0421c-bfae-413a-8ff2-658bda3ad5c0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"1ab0421c-bfae-413a-8ff2-658bda3ad5c0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"544b1cdb-2cc9-4645-adb9-558d399cc3e1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"270d6860-0401-4bd9-9fca-f9ce13927614\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"459e7a4e51a5532b4593e55be584025d65edd1f7dd1ea77c03dadb8805d0ed63\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b67c8ce1-49f3-4640-acf3-5d34aee2c364\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"043e4963ac31fab00c5ce61db3f70b730089dbe3815a0f4332c0450a859f16e2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 26543, \"end_char_idx\": 29910, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "ref_doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "chroma:document": "User, assistant and system messages are clearly\ndistinguishable to the model because of the tokens they're wrapped in.\n\n## Advanced: Adding and editing chat templates\n\n### How do I create a chat template?\n\nSimple, just write a jinja template and set `tokenizer.chat_template`. You may find it easier to start with an \nexisting template from another model and simply edit it for your needs! For example, we could take the LLaMA template\nabove and add \"[ASST]\" and \"[/ASST]\" to assistant messages:\n\n```\n{%- for message in messages %}\n    {%- if message['role'] == 'user' %}\n        {{- bos_token + '[INST] ' + message['content'].strip() + ' [/INST]' }}\n    {%- elif message['role'] == 'system' %}\n        {{- '<<SYS>>\\\\n' + message['content'].strip() + '\\\\n<</SYS>>\\\\n\\\\n' }}\n    {%- elif message['role'] == 'assistant' %}\n        {{- '[ASST] '  + message['content'] + ' [/ASST]' + eos_token }}\n    {%- endif %}\n{%- endfor %}\n```\n\nNow, simply set the `tokenizer.chat_template` attribute. Next time you use [`~PreTrainedTokenizer.apply_chat_template`], it will\nuse your new template! This attribute will be saved in the `tokenizer_config.json` file, so you can use\n[`~utils.PushToHubMixin.push_to_hub`] to upload your new template to the Hub and make sure everyone's using the right\ntemplate for your model!\n\n```python\ntemplate = tokenizer.chat_template\ntemplate = template.replace(\"SYS\", \"SYSTEM\")  # Change the system token\ntokenizer.chat_template = template  # Set the new template\ntokenizer.push_to_hub(\"model_name\")  # Upload your new template to the Hub!\n```\n\nThe method [`~PreTrainedTokenizer.apply_chat_template`] which uses your chat template is called by the [`TextGenerationPipeline`] class, so \nonce you set the correct chat template, your model will automatically become compatible with [`TextGenerationPipeline`].\n\n<Tip>\nIf you're fine-tuning a model for chat, in addition to setting a chat template, you should probably add any new chat\ncontrol tokens as special tokens in the tokenizer. Special tokens are never split, \nensuring that your control tokens are always handled as single tokens rather than being tokenized in pieces. You \nshould also set the tokenizer's `eos_token` attribute to the token that marks the end of assistant generations in your\ntemplate. This will ensure that text generation tools can correctly figure out when to stop generating text.\n</Tip>\n\n\n### Why do some models have multiple templates?\n\nSome models use different templates for different use cases. For example, they might use one template for normal chat\nand another for tool-use, or retrieval-augmented generation. In these cases, `tokenizer.chat_template` is a dictionary.\nThis can cause some confusion, and where possible, we recommend using a single template for all use-cases. You can use\nJinja statements like `if tools is defined` and `{% macro %}` definitions to easily wrap multiple code paths in a\nsingle template.\n\nWhen a tokenizer has multiple templates, `tokenizer.chat_template` will be a `dict`, where each key is the name\nof a template. The `apply_chat_template` method has special handling for certain template names: Specifically, it will\nlook for a template named `default` in most cases, and will raise an error if it can't find one. However, if a template\nnamed `tool_use` exists when the user has passed a `tools` argument, it will use that instead."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: b67c8ce1-49f3-4640-acf3-5d34aee2c364
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"b67c8ce1-49f3-4640-acf3-5d34aee2c364\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"544b1cdb-2cc9-4645-adb9-558d399cc3e1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1ab0421c-bfae-413a-8ff2-658bda3ad5c0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"c27a414fd6bcccbe5a5159fb6e9fcf35cb2b68994b6bc836941b60cd9d27cde9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c0ef51a0-800f-4846-b5be-0848993154b7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"59985f350bc611905b37d72e60b0e15cdc0e7e24a242f61ec8c5bba498e44a4a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 28101, \"end_char_idx\": 31557, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "ref_doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "chroma:document": "```\n\nThe method [`~PreTrainedTokenizer.apply_chat_template`] which uses your chat template is called by the [`TextGenerationPipeline`] class, so \nonce you set the correct chat template, your model will automatically become compatible with [`TextGenerationPipeline`].\n\n<Tip>\nIf you're fine-tuning a model for chat, in addition to setting a chat template, you should probably add any new chat\ncontrol tokens as special tokens in the tokenizer. Special tokens are never split, \nensuring that your control tokens are always handled as single tokens rather than being tokenized in pieces. You \nshould also set the tokenizer's `eos_token` attribute to the token that marks the end of assistant generations in your\ntemplate. This will ensure that text generation tools can correctly figure out when to stop generating text.\n</Tip>\n\n\n### Why do some models have multiple templates?\n\nSome models use different templates for different use cases. For example, they might use one template for normal chat\nand another for tool-use, or retrieval-augmented generation. In these cases, `tokenizer.chat_template` is a dictionary.\nThis can cause some confusion, and where possible, we recommend using a single template for all use-cases. You can use\nJinja statements like `if tools is defined` and `{% macro %}` definitions to easily wrap multiple code paths in a\nsingle template.\n\nWhen a tokenizer has multiple templates, `tokenizer.chat_template` will be a `dict`, where each key is the name\nof a template. The `apply_chat_template` method has special handling for certain template names: Specifically, it will\nlook for a template named `default` in most cases, and will raise an error if it can't find one. However, if a template\nnamed `tool_use` exists when the user has passed a `tools` argument, it will use that instead. To access templates\nwith other names, pass the name of the template you want to the `chat_template` argument of\n`apply_chat_template()`.\n\nWe find that this can be a bit confusing for users, though - so if you're writing a template yourself, we recommend\ntrying to put it all in a single template where possible!\n\n### What template should I use?\n\nWhen setting the template for a model that's already been trained for chat, you should ensure that the template\nexactly matches the message formatting that the model saw during training, or else you will probably experience\nperformance degradation. This is true even if you're training the model further - you will probably get the best \nperformance if you keep the chat tokens constant. This is very analogous to tokenization - you generally get the\nbest performance for inference or fine-tuning when you precisely match the tokenization used during training.\n\nIf you're training a model from scratch, or fine-tuning a base language model for chat, on the other hand,\nyou have a lot of freedom to choose an appropriate template! LLMs are smart enough to learn to handle lots of different\ninput formats. One popular choice is the `ChatML` format, and this is a good, flexible choice for many use-cases. \nIt looks like this:\n\n```\n{%- for message in messages %}\n    {{- '<|im_start|>' + message['role'] + '\\n' + message['content'] + '<|im_end|>' + '\\n' }}\n{%- endfor %}\n```\n\nIf you like this one, here it is in one-liner form, ready to copy into your code. The one-liner also includes\nhandy support for [generation prompts](#what-are-generation-prompts), but note that it doesn't add BOS or EOS tokens!"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: c0ef51a0-800f-4846-b5be-0848993154b7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"c0ef51a0-800f-4846-b5be-0848993154b7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"544b1cdb-2cc9-4645-adb9-558d399cc3e1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b67c8ce1-49f3-4640-acf3-5d34aee2c364\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"a26fe0adbf9a4ba284d1cb7afb550dcd7e9439d963fb7c5324c0982d083adb59\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c5872dda-31ea-4617-83eb-190e22a3121a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"caa34dc4a20f5c85f2bec8c6d87a48d24083b0f354900c893c3fc57722c003e6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 29911, \"end_char_idx\": 33053, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "ref_doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "chroma:document": "To access templates\nwith other names, pass the name of the template you want to the `chat_template` argument of\n`apply_chat_template()`.\n\nWe find that this can be a bit confusing for users, though - so if you're writing a template yourself, we recommend\ntrying to put it all in a single template where possible!\n\n### What template should I use?\n\nWhen setting the template for a model that's already been trained for chat, you should ensure that the template\nexactly matches the message formatting that the model saw during training, or else you will probably experience\nperformance degradation. This is true even if you're training the model further - you will probably get the best \nperformance if you keep the chat tokens constant. This is very analogous to tokenization - you generally get the\nbest performance for inference or fine-tuning when you precisely match the tokenization used during training.\n\nIf you're training a model from scratch, or fine-tuning a base language model for chat, on the other hand,\nyou have a lot of freedom to choose an appropriate template! LLMs are smart enough to learn to handle lots of different\ninput formats. One popular choice is the `ChatML` format, and this is a good, flexible choice for many use-cases. \nIt looks like this:\n\n```\n{%- for message in messages %}\n    {{- '<|im_start|>' + message['role'] + '\\n' + message['content'] + '<|im_end|>' + '\\n' }}\n{%- endfor %}\n```\n\nIf you like this one, here it is in one-liner form, ready to copy into your code. The one-liner also includes\nhandy support for [generation prompts](#what-are-generation-prompts), but note that it doesn't add BOS or EOS tokens!\nIf your model expects those, they won't be added automatically by `apply_chat_template` - in other words, the\ntext will be tokenized with `add_special_tokens=False`. This is to avoid potential conflicts between the template and\nthe `add_special_tokens` logic. If your model expects special tokens, make sure to add them to the template!\n\n```python\ntokenizer.chat_template = \"{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% for message in messages %}{{'<|im_start|>' + message['role'] + '\\n' + message['content'] + '<|im_end|>' + '\\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\\n' }}{% endif %}\"\n```\n\nThis template wraps each message in `<|im_start|>` and `<|im_end|>` tokens, and simply writes the role as a string, which\nallows for flexibility in the roles you train with. The output looks like this:\n\n```text\n<|im_start|>system\nYou are a helpful chatbot that will do its best not to say anything so stupid that people tweet about it.<|im_end|>\n<|im_start|>user\nHow are you?<|im_end|>\n<|im_start|>assistant\nI'm doing great!<|im_end|>\n```\n\nThe \"user\", \"system\" and \"assistant\" roles are the standard for chat, and we recommend using them when it makes sense,\nparticularly if you want your model to operate well with [`TextGenerationPipeline`]. However, you are not limited\nto these roles - templating is extremely flexible, and any string can be a role.\n\n### I want to add some chat templates! How should I get started?"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: c5872dda-31ea-4617-83eb-190e22a3121a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"c5872dda-31ea-4617-83eb-190e22a3121a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"544b1cdb-2cc9-4645-adb9-558d399cc3e1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c0ef51a0-800f-4846-b5be-0848993154b7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"762750e5441b0d976cb1c1787948d6b6eaa610ddb720ec71bd18f869886f8b40\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ce0146dc-854c-4098-ad92-f07a121aca5b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a4484dbb6246d68a70b63b9bc7fb67485edf7831dddc0ffe1589ef0594075406\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 31558, \"end_char_idx\": 34824, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "ref_doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "chroma:document": "If your model expects those, they won't be added automatically by `apply_chat_template` - in other words, the\ntext will be tokenized with `add_special_tokens=False`. This is to avoid potential conflicts between the template and\nthe `add_special_tokens` logic. If your model expects special tokens, make sure to add them to the template!\n\n```python\ntokenizer.chat_template = \"{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% for message in messages %}{{'<|im_start|>' + message['role'] + '\\n' + message['content'] + '<|im_end|>' + '\\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\\n' }}{% endif %}\"\n```\n\nThis template wraps each message in `<|im_start|>` and `<|im_end|>` tokens, and simply writes the role as a string, which\nallows for flexibility in the roles you train with. The output looks like this:\n\n```text\n<|im_start|>system\nYou are a helpful chatbot that will do its best not to say anything so stupid that people tweet about it.<|im_end|>\n<|im_start|>user\nHow are you?<|im_end|>\n<|im_start|>assistant\nI'm doing great!<|im_end|>\n```\n\nThe \"user\", \"system\" and \"assistant\" roles are the standard for chat, and we recommend using them when it makes sense,\nparticularly if you want your model to operate well with [`TextGenerationPipeline`]. However, you are not limited\nto these roles - templating is extremely flexible, and any string can be a role.\n\n### I want to add some chat templates! How should I get started?\n\nIf you have any chat models, you should set their `tokenizer.chat_template` attribute and test it using\n[`~PreTrainedTokenizer.apply_chat_template`], then push the updated tokenizer to the Hub. This applies even if you're\nnot the model owner - if you're using a model with an empty chat template, or one that's still using the default class\ntemplate, please open a [pull request](https://huggingface.co/docs/hub/repositories-pull-requests-discussions) to the model repository so that this attribute can be set properly!\n\nOnce the attribute is set, that's it, you're done! `tokenizer.apply_chat_template` will now work correctly for that\nmodel, which means it is also automatically supported in places like `TextGenerationPipeline`!\n\nBy ensuring that models have this attribute, we can make sure that the whole community gets to use the full power of\nopen-source models. Formatting mismatches have been haunting the field and silently harming performance for too long - \nit's time to put an end to them!\n\n## Advanced: Template writing tips\n\nIf you're unfamiliar with Jinja, we generally find that the easiest way to write a chat template is to first\nwrite a short Python script that formats messages the way you want, and then convert that script into a template.\n\nRemember that the template handler will receive the conversation history as a variable called `messages`.  \nYou will be able to access `messages` in your template just like you can in Python, which means you can loop over \nit with `{% for message in messages %}` or access individual messages with `{{ messages[0] }}`, for example.\n\nYou can also use the following tips to convert your code to Jinja:\n\n### Trimming whitespace\n\nBy default, Jinja will print any whitespace that comes before or after a block."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: ce0146dc-854c-4098-ad92-f07a121aca5b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"ce0146dc-854c-4098-ad92-f07a121aca5b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"544b1cdb-2cc9-4645-adb9-558d399cc3e1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c5872dda-31ea-4617-83eb-190e22a3121a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"089f19319148e2d0700edc7a45e3651011cb6b9a32942213f7678f598e3de312\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"77e2291e-d56f-45ff-ac08-414d284f4fde\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a1f056b7fc590e7523e5eee2012e788efb33d72eaeee36e8f7bda138ec72cdbc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 32989, \"end_char_idx\": 36341, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "ref_doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "chroma:document": "### I want to add some chat templates! How should I get started?\n\nIf you have any chat models, you should set their `tokenizer.chat_template` attribute and test it using\n[`~PreTrainedTokenizer.apply_chat_template`], then push the updated tokenizer to the Hub. This applies even if you're\nnot the model owner - if you're using a model with an empty chat template, or one that's still using the default class\ntemplate, please open a [pull request](https://huggingface.co/docs/hub/repositories-pull-requests-discussions) to the model repository so that this attribute can be set properly!\n\nOnce the attribute is set, that's it, you're done! `tokenizer.apply_chat_template` will now work correctly for that\nmodel, which means it is also automatically supported in places like `TextGenerationPipeline`!\n\nBy ensuring that models have this attribute, we can make sure that the whole community gets to use the full power of\nopen-source models. Formatting mismatches have been haunting the field and silently harming performance for too long - \nit's time to put an end to them!\n\n## Advanced: Template writing tips\n\nIf you're unfamiliar with Jinja, we generally find that the easiest way to write a chat template is to first\nwrite a short Python script that formats messages the way you want, and then convert that script into a template.\n\nRemember that the template handler will receive the conversation history as a variable called `messages`.  \nYou will be able to access `messages` in your template just like you can in Python, which means you can loop over \nit with `{% for message in messages %}` or access individual messages with `{{ messages[0] }}`, for example.\n\nYou can also use the following tips to convert your code to Jinja:\n\n### Trimming whitespace\n\nBy default, Jinja will print any whitespace that comes before or after a block. This can be a problem for chat\ntemplates, which generally want to be very precise with whitespace! To avoid this, we strongly recommend writing\nyour templates like this:\n\n```\n{%- for message in messages %}\n    {{- message['role'] + message['content'] }}\n{%- endfor %}\n```\n\nrather than like this:\n\n```\n{% for message in messages %}\n    {{ message['role'] + message['content'] }}\n{% endfor %}\n```\n\nAdding `-` will strip any whitespace that comes before the block. The second example looks innocent, but the newline\nand indentation may end up being included in the output, which is probably not what you want!\n\n### For loops\n\nFor loops in Jinja look like this:\n\n```\n{%- for message in messages %}\n    {{- message['content'] }}\n{%- endfor %}\n```\n\nNote that whatever's inside the {{ expression block }} will be printed to the output. You can use operators like\n`+` to combine strings inside expression blocks.\n\n### If statements\n\nIf statements in Jinja look like this:\n\n```\n{%- if message['role'] == 'user' %}\n    {{- message['content'] }}\n{%- endif %}\n```\n\nNote how where Python uses whitespace to mark the beginnings and ends of `for` and `if` blocks, Jinja requires you\nto explicitly end them with `{% endfor %}` and `{% endif %}`.\n\n### Special variables\n\nInside your template, you will have access to the list of `messages`, but you can also access several other special\nvariables. These include special tokens like `bos_token` and `eos_token`, as well as the `add_generation_prompt`\nvariable that we discussed above."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 77e2291e-d56f-45ff-ac08-414d284f4fde
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"77e2291e-d56f-45ff-ac08-414d284f4fde\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"544b1cdb-2cc9-4645-adb9-558d399cc3e1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ce0146dc-854c-4098-ad92-f07a121aca5b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"6e70a068347b2dff569c2b5f85186c276b6c6e2725defc8ee131aba97db12480\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b185b548-ac1a-4679-861e-39269998cec0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"53cecd8a1f18eaf777d7b45aa6e9c175cf17d232a3a2e8a48e85ac563cb9a228\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 34652, \"end_char_idx\": 37941, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "ref_doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "chroma:document": "You can also use the following tips to convert your code to Jinja:\n\n### Trimming whitespace\n\nBy default, Jinja will print any whitespace that comes before or after a block. This can be a problem for chat\ntemplates, which generally want to be very precise with whitespace! To avoid this, we strongly recommend writing\nyour templates like this:\n\n```\n{%- for message in messages %}\n    {{- message['role'] + message['content'] }}\n{%- endfor %}\n```\n\nrather than like this:\n\n```\n{% for message in messages %}\n    {{ message['role'] + message['content'] }}\n{% endfor %}\n```\n\nAdding `-` will strip any whitespace that comes before the block. The second example looks innocent, but the newline\nand indentation may end up being included in the output, which is probably not what you want!\n\n### For loops\n\nFor loops in Jinja look like this:\n\n```\n{%- for message in messages %}\n    {{- message['content'] }}\n{%- endfor %}\n```\n\nNote that whatever's inside the {{ expression block }} will be printed to the output. You can use operators like\n`+` to combine strings inside expression blocks.\n\n### If statements\n\nIf statements in Jinja look like this:\n\n```\n{%- if message['role'] == 'user' %}\n    {{- message['content'] }}\n{%- endif %}\n```\n\nNote how where Python uses whitespace to mark the beginnings and ends of `for` and `if` blocks, Jinja requires you\nto explicitly end them with `{% endfor %}` and `{% endif %}`.\n\n### Special variables\n\nInside your template, you will have access to the list of `messages`, but you can also access several other special\nvariables. These include special tokens like `bos_token` and `eos_token`, as well as the `add_generation_prompt`\nvariable that we discussed above. You can also use the `loop` variable to access information about the current loop\niteration, for example  using `{% if loop.last %}` to check if the current message is the last message in the \nconversation. Here's an example that puts these ideas together to add a generation prompt at the end of the\nconversation if add_generation_prompt is `True`:\n\n```\n{%- if loop.last and add_generation_prompt %}\n    {{- bos_token + 'Assistant:\\n' }}\n{%- endif %}\n```\n\n### Compatibility with non-Python Jinja\n\nThere are multiple implementations of Jinja in various languages. They generally have the same syntax,\nbut a key difference is that when you're writing a template in Python you can use Python methods, such as\n`.lower()` on strings or `.items()` on dicts. This will break if someone tries to use your template on a non-Python\nimplementation of Jinja. Non-Python implementations are particularly common in deployment environments, where JS\nand Rust are very popular. \n\nDon't panic, though! There are a few easy changes you can make to your templates to ensure they're compatible across\nall implementations of Jinja:\n\n- Replace Python methods with Jinja filters. These usually have the same name, for example `string.lower()` becomes\n  `string|lower`, and `dict.items()` becomes `dict|items`. One notable change is that `string.strip()` becomes `string|trim`.\n  See the [list of built-in filters](https://jinja.palletsprojects.com/en/3.1.x/templates/#builtin-filters)\n  in the Jinja documentation for more.\n- Replace `True`, `False` and `None`, which are Python-specific, with `true`, `false` and `none`."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: b185b548-ac1a-4679-861e-39269998cec0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/chat_templating", "title": "Templates for Chat Models", "tokens": 9151, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"b185b548-ac1a-4679-861e-39269998cec0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"544b1cdb-2cc9-4645-adb9-558d399cc3e1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8a51bac97932a8f644e752a1c3743ccb4186c8b7d41066c6288e84987566fff2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"77e2291e-d56f-45ff-ac08-414d284f4fde\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/chat_templating\", \"title\": \"Templates for Chat Models\", \"tokens\": 9151, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"6e79d51c615d7821d1cda5cdf3e3f93723bc5b87f0913e82d26cce4f06d01a8b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 36342, \"end_char_idx\": 38175, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "ref_doc_id": "544b1cdb-2cc9-4645-adb9-558d399cc3e1", "chroma:document": "You can also use the `loop` variable to access information about the current loop\niteration, for example  using `{% if loop.last %}` to check if the current message is the last message in the \nconversation. Here's an example that puts these ideas together to add a generation prompt at the end of the\nconversation if add_generation_prompt is `True`:\n\n```\n{%- if loop.last and add_generation_prompt %}\n    {{- bos_token + 'Assistant:\\n' }}\n{%- endif %}\n```\n\n### Compatibility with non-Python Jinja\n\nThere are multiple implementations of Jinja in various languages. They generally have the same syntax,\nbut a key difference is that when you're writing a template in Python you can use Python methods, such as\n`.lower()` on strings or `.items()` on dicts. This will break if someone tries to use your template on a non-Python\nimplementation of Jinja. Non-Python implementations are particularly common in deployment environments, where JS\nand Rust are very popular. \n\nDon't panic, though! There are a few easy changes you can make to your templates to ensure they're compatible across\nall implementations of Jinja:\n\n- Replace Python methods with Jinja filters. These usually have the same name, for example `string.lower()` becomes\n  `string|lower`, and `dict.items()` becomes `dict|items`. One notable change is that `string.strip()` becomes `string|trim`.\n  See the [list of built-in filters](https://jinja.palletsprojects.com/en/3.1.x/templates/#builtin-filters)\n  in the Jinja documentation for more.\n- Replace `True`, `False` and `None`, which are Python-specific, with `true`, `false` and `none`.\n- Directly rendering a dict or list may give different results in other implementations (for example, string entries\n  might change from single-quoted to double-quoted). Adding the `tojson` filter can help to ensure consistency here."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: e09ed79f-1c91-4cac-80f8-30248477b485
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"e09ed79f-1c91-4cac-80f8-30248477b485\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9b0f4da-8c49-44a8-963a-251ef8b99637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d58a77e8-5ea6-4ec1-988a-8b8566c60a0b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b226a18b18a0ff2715c0422af7dd5128eb99b351bdf1dbb5c8c2fce313f98236\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3234, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "ref_doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "chroma:document": "# How to add a model to \ud83e\udd17 Transformers?\n\nThe \ud83e\udd17 Transformers library is often able to offer new models thanks to community contributors. But this can be a challenging project and requires an in-depth knowledge of the \ud83e\udd17 Transformers library and the model to implement. At Hugging Face, we're trying to empower more of the community to actively add models and we've put together this guide to walk you through the process of adding a PyTorch model (make sure you have [PyTorch installed](https://pytorch.org/get-started/locally/)).\n\nAlong the way, you'll:\n\n- get insights into open-source best practices\n- understand the design principles behind one of the most popular deep learning libraries\n- learn how to efficiently test large models\n- learn how to integrate Python utilities like `black`, `ruff`, and `make fix-copies` to ensure clean and readable code\n\nA Hugging Face team member will be available to help you along the way so you'll never be alone. \ud83e\udd17 \u2764\ufe0f\n\nTo get started, open a [New model addition](https://github.com/huggingface/transformers/issues/new?assignees=&labels=New+model&template=new-model-addition.yml) issue for the model you want to see in \ud83e\udd17 Transformers. If you're not especially picky about contributing a specific model, you can filter by the [New model label](https://github.com/huggingface/transformers/labels/New%20model) to see if there are any unclaimed model requests and work on it.\n\nOnce you've opened a new model request, the first step is to get familiar with \ud83e\udd17 Transformers if you aren't already!\n\n## General overview of \ud83e\udd17 Transformers\n\nFirst, you should get a general overview of \ud83e\udd17 Transformers. \ud83e\udd17 Transformers is a very opinionated library, so there is a\nchance that you don't agree with some of the library's philosophies or design choices. From our experience, however, we\nfound that the fundamental design choices and philosophies of the library are crucial to efficiently scale \ud83e\udd17\nTransformers while keeping maintenance costs at a reasonable level.\n\nA good first starting point to better understand the library is to read the [documentation of our philosophy](philosophy). As a result of our way of working, there are some choices that we try to apply to all models:\n\n- Composition is generally favored over-abstraction\n- Duplicating code is not always bad if it strongly improves the readability or accessibility of a model\n- Model files are as self-contained as possible so that when you read the code of a specific model, you ideally only\n  have to look into the respective `modeling_....py` file.\n\nIn our opinion, the library's code is not just a means to provide a product, *e.g.* the ability to use BERT for\ninference, but also as the very product that we want to improve. Hence, when adding a model, the user is not only the\nperson who will use your model, but also everybody who will read, try to understand, and possibly tweak your code.\n\nWith this in mind, let's go a bit deeper into the general library design.\n\n### Overview of models\n\nTo successfully add a model, it is important to understand the interaction between your model and its config,\n[`PreTrainedModel`], and [`PretrainedConfig`]. For exemplary purposes, we will\ncall the model to be added to \ud83e\udd17 Transformers `BrandNewBert`."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: d58a77e8-5ea6-4ec1-988a-8b8566c60a0b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"d58a77e8-5ea6-4ec1-988a-8b8566c60a0b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9b0f4da-8c49-44a8-963a-251ef8b99637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e09ed79f-1c91-4cac-80f8-30248477b485\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"9fb421124cef8f0c531b3296d23d73749bc33d19139382c97cde2be697555702\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"40f86fda-ee6f-4227-b96f-03abb0727b0d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"55e340fc7b813885869bdc0a62d5f54ce62ce1b90e1ffc6c2fece981c6c0e784\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1531, \"end_char_idx\": 4527, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "ref_doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "chroma:document": "## General overview of \ud83e\udd17 Transformers\n\nFirst, you should get a general overview of \ud83e\udd17 Transformers. \ud83e\udd17 Transformers is a very opinionated library, so there is a\nchance that you don't agree with some of the library's philosophies or design choices. From our experience, however, we\nfound that the fundamental design choices and philosophies of the library are crucial to efficiently scale \ud83e\udd17\nTransformers while keeping maintenance costs at a reasonable level.\n\nA good first starting point to better understand the library is to read the [documentation of our philosophy](philosophy). As a result of our way of working, there are some choices that we try to apply to all models:\n\n- Composition is generally favored over-abstraction\n- Duplicating code is not always bad if it strongly improves the readability or accessibility of a model\n- Model files are as self-contained as possible so that when you read the code of a specific model, you ideally only\n  have to look into the respective `modeling_....py` file.\n\nIn our opinion, the library's code is not just a means to provide a product, *e.g.* the ability to use BERT for\ninference, but also as the very product that we want to improve. Hence, when adding a model, the user is not only the\nperson who will use your model, but also everybody who will read, try to understand, and possibly tweak your code.\n\nWith this in mind, let's go a bit deeper into the general library design.\n\n### Overview of models\n\nTo successfully add a model, it is important to understand the interaction between your model and its config,\n[`PreTrainedModel`], and [`PretrainedConfig`]. For exemplary purposes, we will\ncall the model to be added to \ud83e\udd17 Transformers `BrandNewBert`.\n\nLet's take a look:\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers_overview.png\"/>\n\nAs you can see, we do make use of inheritance in \ud83e\udd17 Transformers, but we keep the level of abstraction to an absolute\nminimum. There are never more than two levels of abstraction for any model in the library. `BrandNewBertModel`\ninherits from `BrandNewBertPreTrainedModel` which in turn inherits from [`PreTrainedModel`] and\nthat's it. As a general rule, we want to make sure that a new model only depends on\n[`PreTrainedModel`]. The important functionalities that are automatically provided to every new\nmodel are [`~PreTrainedModel.from_pretrained`] and\n[`~PreTrainedModel.save_pretrained`], which are used for serialization and deserialization. All of the\nother important functionalities, such as `BrandNewBertModel.forward` should be completely defined in the new\n`modeling_brand_new_bert.py` script. Next, we want to make sure that a model with a specific head layer, such as\n`BrandNewBertForMaskedLM` does not inherit from `BrandNewBertModel`, but rather uses `BrandNewBertModel`\nas a component that can be called in its forward pass to keep the level of abstraction low. Every new model requires a\nconfiguration class, called `BrandNewBertConfig`."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 40f86fda-ee6f-4227-b96f-03abb0727b0d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"40f86fda-ee6f-4227-b96f-03abb0727b0d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9b0f4da-8c49-44a8-963a-251ef8b99637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d58a77e8-5ea6-4ec1-988a-8b8566c60a0b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"599d4cb4920a032486754f52278809d177cab7546e7ccd0005e7afb96865be20\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4d5cd700-5f79-4a87-b1e9-1205338c1439\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"db2e62acb5a2af71f77dce749b331e11674c5a3f6f81328e01718c3a19140510\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2886, \"end_char_idx\": 6173, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "ref_doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "chroma:document": "With this in mind, let's go a bit deeper into the general library design.\n\n### Overview of models\n\nTo successfully add a model, it is important to understand the interaction between your model and its config,\n[`PreTrainedModel`], and [`PretrainedConfig`]. For exemplary purposes, we will\ncall the model to be added to \ud83e\udd17 Transformers `BrandNewBert`.\n\nLet's take a look:\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers_overview.png\"/>\n\nAs you can see, we do make use of inheritance in \ud83e\udd17 Transformers, but we keep the level of abstraction to an absolute\nminimum. There are never more than two levels of abstraction for any model in the library. `BrandNewBertModel`\ninherits from `BrandNewBertPreTrainedModel` which in turn inherits from [`PreTrainedModel`] and\nthat's it. As a general rule, we want to make sure that a new model only depends on\n[`PreTrainedModel`]. The important functionalities that are automatically provided to every new\nmodel are [`~PreTrainedModel.from_pretrained`] and\n[`~PreTrainedModel.save_pretrained`], which are used for serialization and deserialization. All of the\nother important functionalities, such as `BrandNewBertModel.forward` should be completely defined in the new\n`modeling_brand_new_bert.py` script. Next, we want to make sure that a model with a specific head layer, such as\n`BrandNewBertForMaskedLM` does not inherit from `BrandNewBertModel`, but rather uses `BrandNewBertModel`\nas a component that can be called in its forward pass to keep the level of abstraction low. Every new model requires a\nconfiguration class, called `BrandNewBertConfig`. This configuration is always stored as an attribute in\n[`PreTrainedModel`], and thus can be accessed via the `config` attribute for all classes\ninheriting from `BrandNewBertPreTrainedModel`:\n\n```python\nmodel = BrandNewBertModel.from_pretrained(\"brandy/brand_new_bert\")\nmodel.config  # model has access to its config\n```\n\nSimilar to the model, the configuration inherits basic serialization and deserialization functionalities from\n[`PretrainedConfig`]. Note that the configuration and the model are always serialized into two\ndifferent formats - the model to a *pytorch_model.bin* file and the configuration to a *config.json* file. Calling\nthe model's [`~PreTrainedModel.save_pretrained`] will automatically call\nthe config's [`~PretrainedConfig.save_pretrained`], so that both model and configuration are saved.\n\n\n### Code style\n\nWhen coding your new model, keep in mind that Transformers is an opinionated library and we have a few quirks of our\nown regarding how code should be written :-)\n\n1. The forward pass of your model should be fully written in the modeling file while being fully independent of other\n   models in the library. If you want to reuse a block from another model, copy the code and paste it with a\n   `# Copied from` comment on top (see [here](https://github.com/huggingface/transformers/blob/v4.17.0/src/transformers/models/roberta/modeling_roberta.py#L160)\n   for a good example and [there](pr_checks#check-copies) for more documentation on Copied from). \n2. The code should be fully understandable, even by a non-native English speaker. This means you should pick\n   descriptive variable names and avoid abbreviations."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 4d5cd700-5f79-4a87-b1e9-1205338c1439
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"4d5cd700-5f79-4a87-b1e9-1205338c1439\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9b0f4da-8c49-44a8-963a-251ef8b99637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"40f86fda-ee6f-4227-b96f-03abb0727b0d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8dffee4b0209628a0344fbbe9c6e72d3c0e2fefa62190277169a323e00892b15\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3f14874a-6d9b-4660-b2aa-b42ca6fd4cc6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"aebd6107d9b984589933cf8bb5c321d05161a68552ef15f02410b8268f3e1539\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4528, \"end_char_idx\": 7719, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "ref_doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "chroma:document": "This configuration is always stored as an attribute in\n[`PreTrainedModel`], and thus can be accessed via the `config` attribute for all classes\ninheriting from `BrandNewBertPreTrainedModel`:\n\n```python\nmodel = BrandNewBertModel.from_pretrained(\"brandy/brand_new_bert\")\nmodel.config  # model has access to its config\n```\n\nSimilar to the model, the configuration inherits basic serialization and deserialization functionalities from\n[`PretrainedConfig`]. Note that the configuration and the model are always serialized into two\ndifferent formats - the model to a *pytorch_model.bin* file and the configuration to a *config.json* file. Calling\nthe model's [`~PreTrainedModel.save_pretrained`] will automatically call\nthe config's [`~PretrainedConfig.save_pretrained`], so that both model and configuration are saved.\n\n\n### Code style\n\nWhen coding your new model, keep in mind that Transformers is an opinionated library and we have a few quirks of our\nown regarding how code should be written :-)\n\n1. The forward pass of your model should be fully written in the modeling file while being fully independent of other\n   models in the library. If you want to reuse a block from another model, copy the code and paste it with a\n   `# Copied from` comment on top (see [here](https://github.com/huggingface/transformers/blob/v4.17.0/src/transformers/models/roberta/modeling_roberta.py#L160)\n   for a good example and [there](pr_checks#check-copies) for more documentation on Copied from). \n2. The code should be fully understandable, even by a non-native English speaker. This means you should pick\n   descriptive variable names and avoid abbreviations. As an example, `activation` is preferred to `act`.\n   One-letter variable names are strongly discouraged unless it's an index in a for loop.\n3. More generally we prefer longer explicit code to short magical one.\n4. Avoid subclassing `nn.Sequential` in PyTorch but subclass `nn.Module` and write the forward pass, so that anyone\n   using your code can quickly debug it by adding print statements or breaking points.\n5. Your function signature should be type-annotated. For the rest, good variable names are way more readable and\n   understandable than type annotations.\n\n### Overview of tokenizers\n\nNot quite ready yet :-( This section will be added soon!\n\n## Step-by-step recipe to add a model to \ud83e\udd17 Transformers\n\nEveryone has different preferences of how to port a model so it can be very helpful for you to take a look at summaries\nof how other contributors ported models to Hugging Face. Here is a list of community blog posts on how to port a model:\n\n1. [Porting GPT2 Model](https://medium.com/huggingface/from-tensorflow-to-pytorch-265f40ef2a28) by [Thomas](https://huggingface.co/thomwolf)\n2. [Porting WMT19 MT Model](https://huggingface.co/blog/porting-fsmt) by [Stas](https://huggingface.co/stas)\n\nFrom experience, we can tell you that the most important things to keep in mind when adding a model are:\n\n-  Don't reinvent the wheel! Most parts of the code you will add for the new \ud83e\udd17 Transformers model already exist\n  somewhere in \ud83e\udd17 Transformers. Take some time to find similar, already existing models and tokenizers you can copy\n  from."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 3f14874a-6d9b-4660-b2aa-b42ca6fd4cc6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"3f14874a-6d9b-4660-b2aa-b42ca6fd4cc6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9b0f4da-8c49-44a8-963a-251ef8b99637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4d5cd700-5f79-4a87-b1e9-1205338c1439\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"efcadc0aba4f766d6b9d518f95092c142e69d9a44ef09a72cea5ef5c91eb86cb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"685e6d2b-53b9-4677-b9b3-914889880efd\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c236486c6ce9da7fa02c53e36b0c14b99062c69816a971f539be47e6c58dc569\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6092, \"end_char_idx\": 8662, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "ref_doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "chroma:document": "This means you should pick\n   descriptive variable names and avoid abbreviations. As an example, `activation` is preferred to `act`.\n   One-letter variable names are strongly discouraged unless it's an index in a for loop.\n3. More generally we prefer longer explicit code to short magical one.\n4. Avoid subclassing `nn.Sequential` in PyTorch but subclass `nn.Module` and write the forward pass, so that anyone\n   using your code can quickly debug it by adding print statements or breaking points.\n5. Your function signature should be type-annotated. For the rest, good variable names are way more readable and\n   understandable than type annotations.\n\n### Overview of tokenizers\n\nNot quite ready yet :-( This section will be added soon!\n\n## Step-by-step recipe to add a model to \ud83e\udd17 Transformers\n\nEveryone has different preferences of how to port a model so it can be very helpful for you to take a look at summaries\nof how other contributors ported models to Hugging Face. Here is a list of community blog posts on how to port a model:\n\n1. [Porting GPT2 Model](https://medium.com/huggingface/from-tensorflow-to-pytorch-265f40ef2a28) by [Thomas](https://huggingface.co/thomwolf)\n2. [Porting WMT19 MT Model](https://huggingface.co/blog/porting-fsmt) by [Stas](https://huggingface.co/stas)\n\nFrom experience, we can tell you that the most important things to keep in mind when adding a model are:\n\n-  Don't reinvent the wheel! Most parts of the code you will add for the new \ud83e\udd17 Transformers model already exist\n  somewhere in \ud83e\udd17 Transformers. Take some time to find similar, already existing models and tokenizers you can copy\n  from. [grep](https://www.gnu.org/software/grep/) and [rg](https://github.com/BurntSushi/ripgrep) are your\n  friends. Note that it might very well happen that your model's tokenizer is based on one model implementation, and\n  your model's modeling code on another one. *E.g.* FSMT's modeling code is based on BART, while FSMT's tokenizer code\n  is based on XLM.\n-  It's more of an engineering challenge than a scientific challenge. You should spend more time creating an\n  efficient debugging environment rather than trying to understand all theoretical aspects of the model in the paper.\n-  Ask for help, when you're stuck! Models are the core component of \ud83e\udd17 Transformers so we at Hugging Face are more\n  than happy to help you at every step to add your model. Don't hesitate to ask if you notice you are not making\n  progress.\n\nIn the following, we try to give you a general recipe that we found most useful when porting a model to \ud83e\udd17 Transformers."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 685e6d2b-53b9-4677-b9b3-914889880efd
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"685e6d2b-53b9-4677-b9b3-914889880efd\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9b0f4da-8c49-44a8-963a-251ef8b99637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3f14874a-6d9b-4660-b2aa-b42ca6fd4cc6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"85ee1b28729c1ab320a1c6423134bb1cad525cd2b58c01dbb959dcc684614600\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"aae5bb19-b3d6-40c1-b4d8-7701a6c5ed5f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"af9de8a25317467879b8e1cc32a798ce305c8c10626afd614c2ab9f35bb60a8e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7131, \"end_char_idx\": 10359, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "ref_doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "chroma:document": "[Porting GPT2 Model](https://medium.com/huggingface/from-tensorflow-to-pytorch-265f40ef2a28) by [Thomas](https://huggingface.co/thomwolf)\n2. [Porting WMT19 MT Model](https://huggingface.co/blog/porting-fsmt) by [Stas](https://huggingface.co/stas)\n\nFrom experience, we can tell you that the most important things to keep in mind when adding a model are:\n\n-  Don't reinvent the wheel! Most parts of the code you will add for the new \ud83e\udd17 Transformers model already exist\n  somewhere in \ud83e\udd17 Transformers. Take some time to find similar, already existing models and tokenizers you can copy\n  from. [grep](https://www.gnu.org/software/grep/) and [rg](https://github.com/BurntSushi/ripgrep) are your\n  friends. Note that it might very well happen that your model's tokenizer is based on one model implementation, and\n  your model's modeling code on another one. *E.g.* FSMT's modeling code is based on BART, while FSMT's tokenizer code\n  is based on XLM.\n-  It's more of an engineering challenge than a scientific challenge. You should spend more time creating an\n  efficient debugging environment rather than trying to understand all theoretical aspects of the model in the paper.\n-  Ask for help, when you're stuck! Models are the core component of \ud83e\udd17 Transformers so we at Hugging Face are more\n  than happy to help you at every step to add your model. Don't hesitate to ask if you notice you are not making\n  progress.\n\nIn the following, we try to give you a general recipe that we found most useful when porting a model to \ud83e\udd17 Transformers.\n\nThe following list is a summary of everything that has to be done to add a model and can be used by you as a To-Do\nList:\n\n\u2610 (Optional) Understood the model's theoretical aspects<br>\n\u2610 Prepared \ud83e\udd17 Transformers dev environment<br>\n\u2610 Set up debugging environment of the original repository<br>\n\u2610 Created script that successfully runs the `forward()` pass using the original repository and checkpoint<br>\n\u2610 Successfully added the model skeleton to \ud83e\udd17 Transformers<br>\n\u2610 Successfully converted original checkpoint to \ud83e\udd17 Transformers checkpoint<br>\n\u2610 Successfully ran `forward()` pass in \ud83e\udd17 Transformers that gives identical output to original checkpoint<br>\n\u2610 Finished model tests in \ud83e\udd17 Transformers<br>\n\u2610 Successfully added tokenizer in \ud83e\udd17 Transformers<br>\n\u2610 Run end-to-end integration tests<br>\n\u2610 Finished docs<br>\n\u2610 Uploaded model weights to the Hub<br>\n\u2610 Submitted the pull request<br>\n\u2610 (Optional) Added a demo notebook\n\nTo begin with, we usually recommend starting by getting a good theoretical understanding of `BrandNewBert`. However,\nif you prefer to understand the theoretical aspects of the model *on-the-job*, then it is totally fine to directly dive\ninto the `BrandNewBert`'s code-base. This option might suit you better if your engineering skills are better than\nyour theoretical skill, if you have trouble understanding `BrandNewBert`'s paper, or if you just enjoy programming\nmuch more than reading scientific papers.\n\n### 1. (Optional) Theoretical aspects of BrandNewBert\n\nYou should take some time to read *BrandNewBert's* paper, if such descriptive work exists. There might be large\nsections of the paper that are difficult to understand. If this is the case, this is fine - don't worry!"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: aae5bb19-b3d6-40c1-b4d8-7701a6c5ed5f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"aae5bb19-b3d6-40c1-b4d8-7701a6c5ed5f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9b0f4da-8c49-44a8-963a-251ef8b99637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"685e6d2b-53b9-4677-b9b3-914889880efd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"6353f44da589d0c2fbfcb84e7c2ee7a2c78ac1d05b2759e14e683f7178d08244\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"80885097-dab2-4eb6-982d-2b0cefd29b9c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"948af9405745c9f24d617e415809f2cfbd55ec33e9c9e52eac5b8fb3add1476d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8664, \"end_char_idx\": 11945, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "ref_doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "chroma:document": "The following list is a summary of everything that has to be done to add a model and can be used by you as a To-Do\nList:\n\n\u2610 (Optional) Understood the model's theoretical aspects<br>\n\u2610 Prepared \ud83e\udd17 Transformers dev environment<br>\n\u2610 Set up debugging environment of the original repository<br>\n\u2610 Created script that successfully runs the `forward()` pass using the original repository and checkpoint<br>\n\u2610 Successfully added the model skeleton to \ud83e\udd17 Transformers<br>\n\u2610 Successfully converted original checkpoint to \ud83e\udd17 Transformers checkpoint<br>\n\u2610 Successfully ran `forward()` pass in \ud83e\udd17 Transformers that gives identical output to original checkpoint<br>\n\u2610 Finished model tests in \ud83e\udd17 Transformers<br>\n\u2610 Successfully added tokenizer in \ud83e\udd17 Transformers<br>\n\u2610 Run end-to-end integration tests<br>\n\u2610 Finished docs<br>\n\u2610 Uploaded model weights to the Hub<br>\n\u2610 Submitted the pull request<br>\n\u2610 (Optional) Added a demo notebook\n\nTo begin with, we usually recommend starting by getting a good theoretical understanding of `BrandNewBert`. However,\nif you prefer to understand the theoretical aspects of the model *on-the-job*, then it is totally fine to directly dive\ninto the `BrandNewBert`'s code-base. This option might suit you better if your engineering skills are better than\nyour theoretical skill, if you have trouble understanding `BrandNewBert`'s paper, or if you just enjoy programming\nmuch more than reading scientific papers.\n\n### 1. (Optional) Theoretical aspects of BrandNewBert\n\nYou should take some time to read *BrandNewBert's* paper, if such descriptive work exists. There might be large\nsections of the paper that are difficult to understand. If this is the case, this is fine - don't worry! The goal is\nnot to get a deep theoretical understanding of the paper, but to extract the necessary information required to\neffectively re-implement the model in \ud83e\udd17 Transformers. That being said, you don't have to spend too much time on the\ntheoretical aspects, but rather focus on the practical ones, namely:\n\n-  What type of model is *brand_new_bert*? BERT-like encoder-only model? GPT2-like decoder-only model? BART-like\n  encoder-decoder model? Look at the [model_summary](model_summary) if you're not familiar with the differences between those.\n-  What are the applications of *brand_new_bert*? Text classification? Text generation? Seq2Seq tasks, *e.g.,*\n  summarization?\n-  What is the novel feature of the model that makes it different from BERT/GPT-2/BART?\n-  Which of the already existing [\ud83e\udd17 Transformers models](https://huggingface.co/transformers/#contents) is most\n  similar to *brand_new_bert*?\n-  What type of tokenizer is used? A sentencepiece tokenizer? Word piece tokenizer? Is it the same tokenizer as used\n  for BERT or BART?\n\nAfter you feel like you have gotten a good overview of the architecture of the model, you might want to write to the\nHugging Face team with any questions you might have. This might include questions regarding the model's architecture,\nits attention layer, etc. We will be more than happy to help you.\n\n### 2. Next prepare your environment\n\n1. Fork the [repository](https://github.com/huggingface/transformers) by clicking on the \u2018Fork' button on the\n   repository's page. This creates a copy of the code under your GitHub user account.\n\n2."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 80885097-dab2-4eb6-982d-2b0cefd29b9c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"80885097-dab2-4eb6-982d-2b0cefd29b9c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9b0f4da-8c49-44a8-963a-251ef8b99637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"aae5bb19-b3d6-40c1-b4d8-7701a6c5ed5f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"2fac60ef1bb3fbc91fe99cde393329444534c62e92f21744733b4bdb802a637d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0d44b683-5ef8-4991-86df-3dcc8d57601e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9aaf36e639c7816d40b8eef8fdb0d0c60da77962cd551f7e49fd55fde9a094c1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10360, \"end_char_idx\": 13466, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "ref_doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "chroma:document": "The goal is\nnot to get a deep theoretical understanding of the paper, but to extract the necessary information required to\neffectively re-implement the model in \ud83e\udd17 Transformers. That being said, you don't have to spend too much time on the\ntheoretical aspects, but rather focus on the practical ones, namely:\n\n-  What type of model is *brand_new_bert*? BERT-like encoder-only model? GPT2-like decoder-only model? BART-like\n  encoder-decoder model? Look at the [model_summary](model_summary) if you're not familiar with the differences between those.\n-  What are the applications of *brand_new_bert*? Text classification? Text generation? Seq2Seq tasks, *e.g.,*\n  summarization?\n-  What is the novel feature of the model that makes it different from BERT/GPT-2/BART?\n-  Which of the already existing [\ud83e\udd17 Transformers models](https://huggingface.co/transformers/#contents) is most\n  similar to *brand_new_bert*?\n-  What type of tokenizer is used? A sentencepiece tokenizer? Word piece tokenizer? Is it the same tokenizer as used\n  for BERT or BART?\n\nAfter you feel like you have gotten a good overview of the architecture of the model, you might want to write to the\nHugging Face team with any questions you might have. This might include questions regarding the model's architecture,\nits attention layer, etc. We will be more than happy to help you.\n\n### 2. Next prepare your environment\n\n1. Fork the [repository](https://github.com/huggingface/transformers) by clicking on the \u2018Fork' button on the\n   repository's page. This creates a copy of the code under your GitHub user account.\n\n2. Clone your `transformers` fork to your local disk, and add the base repository as a remote:\n\n   ```bash\n   git clone https://github.com/[your Github handle]/transformers.git\n   cd transformers\n   git remote add upstream https://github.com/huggingface/transformers.git\n   ```\n\n3. Set up a development environment, for instance by running the following command:\n\n   ```bash\n   python -m venv .env\n   source .env/bin/activate\n   pip install -e \".[dev]\"\n   ```\n\n   Depending on your OS, and since the number of optional dependencies of Transformers is growing, you might get a\n   failure with this command. If that's the case make sure to install the Deep Learning framework you are working with\n   (PyTorch, TensorFlow and/or Flax) then do:\n\n   ```bash\n   pip install -e \".[quality]\"\n   ```\n\n   which should be enough for most use cases. You can then return to the parent directory\n\n   ```bash\n   cd ..\n   ```\n\n4. We recommend adding the PyTorch version of *brand_new_bert* to Transformers. To install PyTorch, please follow the\n   instructions on https://pytorch.org/get-started/locally/.\n\n   **Note:** You don't need to have CUDA installed. Making the new model work on CPU is sufficient.\n\n5. To port *brand_new_bert*, you will also need access to its original repository:\n\n   ```bash\n   git clone https://github.com/org_that_created_brand_new_bert_org/brand_new_bert.git\n   cd brand_new_bert\n   pip install -e .\n   ```\n\nNow you have set up a development environment to port *brand_new_bert* to \ud83e\udd17 Transformers.\n\n### 3.-4."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 0d44b683-5ef8-4991-86df-3dcc8d57601e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"0d44b683-5ef8-4991-86df-3dcc8d57601e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9b0f4da-8c49-44a8-963a-251ef8b99637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"80885097-dab2-4eb6-982d-2b0cefd29b9c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"5a8a41d87fc34e6c79f100d3bf25cb9aacd7079f54e017ab85f663bb37975914\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"88ed9cb4-6412-4b2a-88d2-3a281e12f349\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0349d8d3844d8364eac2e77a6c75e52a4af6bbcbb068e8172aa1b0e6fc1b9d54\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11943, \"end_char_idx\": 15221, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "ref_doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "chroma:document": "2. Clone your `transformers` fork to your local disk, and add the base repository as a remote:\n\n   ```bash\n   git clone https://github.com/[your Github handle]/transformers.git\n   cd transformers\n   git remote add upstream https://github.com/huggingface/transformers.git\n   ```\n\n3. Set up a development environment, for instance by running the following command:\n\n   ```bash\n   python -m venv .env\n   source .env/bin/activate\n   pip install -e \".[dev]\"\n   ```\n\n   Depending on your OS, and since the number of optional dependencies of Transformers is growing, you might get a\n   failure with this command. If that's the case make sure to install the Deep Learning framework you are working with\n   (PyTorch, TensorFlow and/or Flax) then do:\n\n   ```bash\n   pip install -e \".[quality]\"\n   ```\n\n   which should be enough for most use cases. You can then return to the parent directory\n\n   ```bash\n   cd ..\n   ```\n\n4. We recommend adding the PyTorch version of *brand_new_bert* to Transformers. To install PyTorch, please follow the\n   instructions on https://pytorch.org/get-started/locally/.\n\n   **Note:** You don't need to have CUDA installed. Making the new model work on CPU is sufficient.\n\n5. To port *brand_new_bert*, you will also need access to its original repository:\n\n   ```bash\n   git clone https://github.com/org_that_created_brand_new_bert_org/brand_new_bert.git\n   cd brand_new_bert\n   pip install -e .\n   ```\n\nNow you have set up a development environment to port *brand_new_bert* to \ud83e\udd17 Transformers.\n\n### 3.-4. Run a pretrained checkpoint using the original repository\n\nAt first, you will work on the original *brand_new_bert* repository. Often, the original implementation is very\n\u201cresearchy\u201d. Meaning that documentation might be lacking and the code can be difficult to understand. But this should\nbe exactly your motivation to reimplement *brand_new_bert*. At Hugging Face, one of our main goals is to *make people\nstand on the shoulders of giants* which translates here very well into taking a working model and rewriting it to make\nit as **accessible, user-friendly, and beautiful** as possible. This is the number-one motivation to re-implement\nmodels into \ud83e\udd17 Transformers - trying to make complex new NLP technology accessible to **everybody**.\n\nYou should start thereby by diving into the original repository.\n\nSuccessfully running the official pretrained model in the original repository is often **the most difficult** step.\nFrom our experience, it is very important to spend some time getting familiar with the original code-base. You need to\nfigure out the following:\n\n- Where to find the pretrained weights?\n- How to load the pretrained weights into the corresponding model?\n- How to run the tokenizer independently from the model?\n- Trace one forward pass so that you know which classes and functions are required for a simple forward pass. Usually,\n  you only have to reimplement those functions.\n- Be able to locate the important components of the model: Where is the model's class? Are there model sub-classes,\n  *e.g.* EncoderModel, DecoderModel? Where is the self-attention layer? Are there multiple different attention layers,\n  *e.g.* *self-attention*, *cross-attention*...?\n- How can you debug the model in the original environment of the repo?"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 88ed9cb4-6412-4b2a-88d2-3a281e12f349
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"88ed9cb4-6412-4b2a-88d2-3a281e12f349\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9b0f4da-8c49-44a8-963a-251ef8b99637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0d44b683-5ef8-4991-86df-3dcc8d57601e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"6780076fc18998b851cc70a5773957296d8892ca2057df54effde0563728ab6a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2ccbbb71-0cdf-4ca2-a53b-3acadfbfcf18\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8b70049ff604f8f5d1e707e669783f73920110b664ecfbd0eb610b5e53a2b950\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13457, \"end_char_idx\": 16973, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "ref_doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "chroma:document": "### 3.-4. Run a pretrained checkpoint using the original repository\n\nAt first, you will work on the original *brand_new_bert* repository. Often, the original implementation is very\n\u201cresearchy\u201d. Meaning that documentation might be lacking and the code can be difficult to understand. But this should\nbe exactly your motivation to reimplement *brand_new_bert*. At Hugging Face, one of our main goals is to *make people\nstand on the shoulders of giants* which translates here very well into taking a working model and rewriting it to make\nit as **accessible, user-friendly, and beautiful** as possible. This is the number-one motivation to re-implement\nmodels into \ud83e\udd17 Transformers - trying to make complex new NLP technology accessible to **everybody**.\n\nYou should start thereby by diving into the original repository.\n\nSuccessfully running the official pretrained model in the original repository is often **the most difficult** step.\nFrom our experience, it is very important to spend some time getting familiar with the original code-base. You need to\nfigure out the following:\n\n- Where to find the pretrained weights?\n- How to load the pretrained weights into the corresponding model?\n- How to run the tokenizer independently from the model?\n- Trace one forward pass so that you know which classes and functions are required for a simple forward pass. Usually,\n  you only have to reimplement those functions.\n- Be able to locate the important components of the model: Where is the model's class? Are there model sub-classes,\n  *e.g.* EncoderModel, DecoderModel? Where is the self-attention layer? Are there multiple different attention layers,\n  *e.g.* *self-attention*, *cross-attention*...?\n- How can you debug the model in the original environment of the repo? Do you have to add *print* statements, can you\n  work with an interactive debugger like *ipdb*, or should you use an efficient IDE to debug the model, like PyCharm?\n\nIt is very important that before you start the porting process, you can **efficiently** debug code in the original\nrepository! Also, remember that you are working with an open-source library, so do not hesitate to open an issue, or\neven a pull request in the original repository. The maintainers of this repository are most likely very happy about\nsomeone looking into their code!\n\nAt this point, it is really up to you which debugging environment and strategy you prefer to use to debug the original\nmodel. We strongly advise against setting up a costly GPU environment, but simply work on a CPU both when starting to\ndive into the original repository and also when starting to write the \ud83e\udd17 Transformers implementation of the model. Only\nat the very end, when the model has already been successfully ported to \ud83e\udd17 Transformers, one should verify that the\nmodel also works as expected on GPU.\n\nIn general, there are two possible debugging environments for running the original model\n\n-  [Jupyter notebooks](https://jupyter.org/) / [google colab](https://colab.research.google.com/notebooks/intro.ipynb)\n-  Local python scripts.\n\nJupyter notebooks have the advantage that they allow for cell-by-cell execution which can be helpful to better split\nlogical components from one another and to have faster debugging cycles as intermediate results can be stored. Also,\nnotebooks are often easier to share with other contributors, which might be very helpful if you want to ask the Hugging\nFace team for help. If you are familiar with Jupyter notebooks, we strongly recommend you work with them."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 2ccbbb71-0cdf-4ca2-a53b-3acadfbfcf18
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"2ccbbb71-0cdf-4ca2-a53b-3acadfbfcf18\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9b0f4da-8c49-44a8-963a-251ef8b99637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"88ed9cb4-6412-4b2a-88d2-3a281e12f349\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"dd05effb5d9216329253572d6d1cffec7b63f0f9f895b582d6503f0b47531030\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2e70cdbc-55c0-4249-a9fe-c607453b2c7c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ae1f10527656926adfd0bd2fab62ba4aa1cd3f58df2f7edfea75b640d7b87db6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15151, \"end_char_idx\": 18407, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "ref_doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "chroma:document": "- How can you debug the model in the original environment of the repo? Do you have to add *print* statements, can you\n  work with an interactive debugger like *ipdb*, or should you use an efficient IDE to debug the model, like PyCharm?\n\nIt is very important that before you start the porting process, you can **efficiently** debug code in the original\nrepository! Also, remember that you are working with an open-source library, so do not hesitate to open an issue, or\neven a pull request in the original repository. The maintainers of this repository are most likely very happy about\nsomeone looking into their code!\n\nAt this point, it is really up to you which debugging environment and strategy you prefer to use to debug the original\nmodel. We strongly advise against setting up a costly GPU environment, but simply work on a CPU both when starting to\ndive into the original repository and also when starting to write the \ud83e\udd17 Transformers implementation of the model. Only\nat the very end, when the model has already been successfully ported to \ud83e\udd17 Transformers, one should verify that the\nmodel also works as expected on GPU.\n\nIn general, there are two possible debugging environments for running the original model\n\n-  [Jupyter notebooks](https://jupyter.org/) / [google colab](https://colab.research.google.com/notebooks/intro.ipynb)\n-  Local python scripts.\n\nJupyter notebooks have the advantage that they allow for cell-by-cell execution which can be helpful to better split\nlogical components from one another and to have faster debugging cycles as intermediate results can be stored. Also,\nnotebooks are often easier to share with other contributors, which might be very helpful if you want to ask the Hugging\nFace team for help. If you are familiar with Jupyter notebooks, we strongly recommend you work with them.\n\nThe obvious disadvantage of Jupyter notebooks is that if you are not used to working with them you will have to spend\nsome time adjusting to the new programming environment and you might not be able to use your known debugging tools\nanymore, like `ipdb`.\n\nFor each code-base, a good first step is always to load a **small** pretrained checkpoint and to be able to reproduce a\nsingle forward pass using a dummy integer vector of input IDs as an input. Such a script could look like this (in\npseudocode):\n\n```python\nmodel = BrandNewBertModel.load_pretrained_checkpoint(\"/path/to/checkpoint/\")\ninput_ids = [0, 4, 5, 2, 3, 7, 9]  # vector of input ids\noriginal_output = model.predict(input_ids)\n```\n\nNext, regarding the debugging strategy, there are generally a few from which to choose from:\n\n- Decompose the original model into many small testable components and run a forward pass on each of those for\n  verification\n- Decompose the original model only into the original *tokenizer* and the original *model*, run a forward pass on\n  those, and use intermediate print statements or breakpoints for verification\n\nAgain, it is up to you which strategy to choose. Often, one or the other is advantageous depending on the original code\nbase.\n\nIf the original code-base allows you to decompose the model into smaller sub-components, *e.g.* if the original\ncode-base can easily be run in eager mode, it is usually worth the effort to do so."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 2e70cdbc-55c0-4249-a9fe-c607453b2c7c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"2e70cdbc-55c0-4249-a9fe-c607453b2c7c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9b0f4da-8c49-44a8-963a-251ef8b99637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2ccbbb71-0cdf-4ca2-a53b-3acadfbfcf18\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"321653e0f8b1395e19610df2d871d32b4a7d5c6a4488c856923333ceffd663b2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f5b4c0b5-98f1-4482-86a7-c6e152a48b07\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a736c2ca4d0fdfeec83a5fc5a01243831daeabceef8f2c8e58b908fc4bdce857\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 16742, \"end_char_idx\": 20335, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "ref_doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "chroma:document": "Also,\nnotebooks are often easier to share with other contributors, which might be very helpful if you want to ask the Hugging\nFace team for help. If you are familiar with Jupyter notebooks, we strongly recommend you work with them.\n\nThe obvious disadvantage of Jupyter notebooks is that if you are not used to working with them you will have to spend\nsome time adjusting to the new programming environment and you might not be able to use your known debugging tools\nanymore, like `ipdb`.\n\nFor each code-base, a good first step is always to load a **small** pretrained checkpoint and to be able to reproduce a\nsingle forward pass using a dummy integer vector of input IDs as an input. Such a script could look like this (in\npseudocode):\n\n```python\nmodel = BrandNewBertModel.load_pretrained_checkpoint(\"/path/to/checkpoint/\")\ninput_ids = [0, 4, 5, 2, 3, 7, 9]  # vector of input ids\noriginal_output = model.predict(input_ids)\n```\n\nNext, regarding the debugging strategy, there are generally a few from which to choose from:\n\n- Decompose the original model into many small testable components and run a forward pass on each of those for\n  verification\n- Decompose the original model only into the original *tokenizer* and the original *model*, run a forward pass on\n  those, and use intermediate print statements or breakpoints for verification\n\nAgain, it is up to you which strategy to choose. Often, one or the other is advantageous depending on the original code\nbase.\n\nIf the original code-base allows you to decompose the model into smaller sub-components, *e.g.* if the original\ncode-base can easily be run in eager mode, it is usually worth the effort to do so. There are some important advantages\nto taking the more difficult road in the beginning:\n\n- at a later stage when comparing the original model to the Hugging Face implementation, you can verify automatically\n  for each component individually that the corresponding component of the \ud83e\udd17 Transformers implementation matches instead\n  of relying on visual comparison via print statements\n- it can give you some rope to decompose the big problem of porting a model into smaller problems of just porting\n  individual components and thus structure your work better\n- separating the model into logical meaningful components will help you to get a better overview of the model's design\n  and thus to better understand the model\n- at a later stage those component-by-component tests help you to ensure that no regression occurs as you continue\n  changing your code\n\n[Lysandre's](https://gist.github.com/LysandreJik/db4c948f6b4483960de5cbac598ad4ed) integration checks for ELECTRA\ngives a nice example of how this can be done.\n\nHowever, if the original code-base is very complex or only allows intermediate components to be run in a compiled mode,\nit might be too time-consuming or even impossible to separate the model into smaller testable sub-components. A good\nexample is [T5's MeshTensorFlow](https://github.com/tensorflow/mesh/tree/master/mesh_tensorflow) library which is\nvery complex and does not offer a simple way to decompose the model into its sub-components. For such libraries, one\noften relies on verifying print statements.\n\nNo matter which strategy you choose, the recommended procedure is often the same that you should start to debug the\nstarting layers first and the ending layers last.\n\nIt is recommended that you retrieve the output, either by print statements or sub-component functions, of the following\nlayers in the following order:\n\n1. Retrieve the input IDs passed to the model\n2. Retrieve the word embeddings\n3."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: f5b4c0b5-98f1-4482-86a7-c6e152a48b07
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"f5b4c0b5-98f1-4482-86a7-c6e152a48b07\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9b0f4da-8c49-44a8-963a-251ef8b99637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2e70cdbc-55c0-4249-a9fe-c607453b2c7c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"cde417571e1b6facd61edf08699322359edbe83ab4a20ee473e59733c8733024\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"594bc362-1b95-4fb8-b421-79738ccfe2a7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fb0c9573da64959e8a525f2ee61e1e6326cf6c6c04dc8cc78370563f589f92ae\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 19423, \"end_char_idx\": 22179, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "ref_doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "chroma:document": "However, if the original code-base is very complex or only allows intermediate components to be run in a compiled mode,\nit might be too time-consuming or even impossible to separate the model into smaller testable sub-components. A good\nexample is [T5's MeshTensorFlow](https://github.com/tensorflow/mesh/tree/master/mesh_tensorflow) library which is\nvery complex and does not offer a simple way to decompose the model into its sub-components. For such libraries, one\noften relies on verifying print statements.\n\nNo matter which strategy you choose, the recommended procedure is often the same that you should start to debug the\nstarting layers first and the ending layers last.\n\nIt is recommended that you retrieve the output, either by print statements or sub-component functions, of the following\nlayers in the following order:\n\n1. Retrieve the input IDs passed to the model\n2. Retrieve the word embeddings\n3. Retrieve the input of the first Transformer layer\n4. Retrieve the output of the first Transformer layer\n5. Retrieve the output of the following n - 1 Transformer layers\n6. Retrieve the output of the whole BrandNewBert Model\n\nInput IDs should thereby consists of an array of integers, *e.g.* `input_ids = [0, 4, 4, 3, 2, 4, 1, 7, 19]`\n\nThe outputs of the following layers often consist of multi-dimensional float arrays and can look like this:\n\n```\n[[\n [-0.1465, -0.6501,  0.1993,  ...,  0.1451,  0.3430,  0.6024],\n [-0.4417, -0.5920,  0.3450,  ..., -0.3062,  0.6182,  0.7132],\n [-0.5009, -0.7122,  0.4548,  ..., -0.3662,  0.6091,  0.7648],\n ...,\n [-0.5613, -0.6332,  0.4324,  ..., -0.3792,  0.7372,  0.9288],\n [-0.5416, -0.6345,  0.4180,  ..., -0.3564,  0.6992,  0.9191],\n [-0.5334, -0.6403,  0.4271,  ..., -0.3339,  0.6533,  0.8694]]],\n```\n\nWe expect that every model added to \ud83e\udd17 Transformers passes a couple of integration tests, meaning that the original\nmodel and the reimplemented version in \ud83e\udd17 Transformers have to give the exact same output up to a precision of 0.001!\nSince it is normal that the exact same model written in different libraries can give a slightly different output\ndepending on the library framework, we accept an error tolerance of 1e-3 (0.001). It is not enough if the model gives\nnearly the same output, they have to be almost identical. Therefore, you will certainly compare the intermediate\noutputs of the \ud83e\udd17 Transformers version multiple times against the intermediate outputs of the original implementation of\n*brand_new_bert* in which case an **efficient** debugging environment of the original repository is absolutely\nimportant. Here is some advice to make your debugging environment as efficient as possible.\n\n- Find the best way of debugging intermediate results. Is the original repository written in PyTorch?"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 594bc362-1b95-4fb8-b421-79738ccfe2a7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"594bc362-1b95-4fb8-b421-79738ccfe2a7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9b0f4da-8c49-44a8-963a-251ef8b99637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f5b4c0b5-98f1-4482-86a7-c6e152a48b07\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"2e8c025d4eebfd604ab599cbaedc37b9d2f3e011e92ce8113bad947ca548e503\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7c239810-7002-40f1-a682-f6b67a46acf5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"eaff508018d10e9ea1cd7cc466d9d0b30ca83da4f9f8d714ff6c7e3c3a855dde\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 21410, \"end_char_idx\": 24711, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "ref_doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "chroma:document": "Since it is normal that the exact same model written in different libraries can give a slightly different output\ndepending on the library framework, we accept an error tolerance of 1e-3 (0.001). It is not enough if the model gives\nnearly the same output, they have to be almost identical. Therefore, you will certainly compare the intermediate\noutputs of the \ud83e\udd17 Transformers version multiple times against the intermediate outputs of the original implementation of\n*brand_new_bert* in which case an **efficient** debugging environment of the original repository is absolutely\nimportant. Here is some advice to make your debugging environment as efficient as possible.\n\n- Find the best way of debugging intermediate results. Is the original repository written in PyTorch? Then you should\n  probably take the time to write a longer script that decomposes the original model into smaller sub-components to\n  retrieve intermediate values. Is the original repository written in Tensorflow 1? Then you might have to rely on\n  TensorFlow print operations like [tf.print](https://www.tensorflow.org/api_docs/python/tf/print) to output\n  intermediate values. Is the original repository written in Jax? Then make sure that the model is **not jitted** when\n  running the forward pass, *e.g.* check-out [this link](https://github.com/google/jax/issues/196).\n- Use the smallest pretrained checkpoint you can find. The smaller the checkpoint, the faster your debug cycle\n  becomes. It is not efficient if your pretrained model is so big that your forward pass takes more than 10 seconds.\n  In case only very large checkpoints are available, it might make more sense to create a dummy model in the new\n  environment with randomly initialized weights and save those weights for comparison with the \ud83e\udd17 Transformers version\n  of your model\n- Make sure you are using the easiest way of calling a forward pass in the original repository. Ideally, you want to\n  find the function in the original repository that **only** calls a single forward pass, *i.e.* that is often called\n  `predict`, `evaluate`, `forward` or `__call__`. You don't want to debug a function that calls `forward`\n  multiple times, *e.g.* to generate text, like `autoregressive_sample`, `generate`.\n- Try to separate the tokenization from the model's *forward* pass. If the original repository shows examples where\n  you have to input a string, then try to find out where in the forward call the string input is changed to input ids\n  and start from this point. This might mean that you have to possibly write a small script yourself or change the\n  original code so that you can directly input the ids instead of an input string.\n- Make sure that the model in your debugging setup is **not** in training mode, which often causes the model to yield\n  random outputs due to multiple dropout layers in the model. Make sure that the forward pass in your debugging\n  environment is **deterministic** so that the dropout layers are not used. Or use *transformers.utils.set_seed*\n  if the old and new implementations are in the same framework.\n\nThe following section gives you more specific details/tips on how you can do this for *brand_new_bert*.\n\n### 5.-14. Port BrandNewBert to \ud83e\udd17 Transformers\n\nNext, you can finally start adding new code to \ud83e\udd17 Transformers."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 7c239810-7002-40f1-a682-f6b67a46acf5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"7c239810-7002-40f1-a682-f6b67a46acf5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9b0f4da-8c49-44a8-963a-251ef8b99637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"594bc362-1b95-4fb8-b421-79738ccfe2a7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"9ada35ed9444acbc9b74a97bd73c7dae6316f40cb8d39dc6d525b6f8665ce218\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dd4afb93-e2ef-45d4-a0bb-9e9321e9a561\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e921c7fee3dbba93ca2ac34a7691c11fa78995341a54135017835230b585ecbe\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 22985, \"end_char_idx\": 26304, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "ref_doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "chroma:document": "In case only very large checkpoints are available, it might make more sense to create a dummy model in the new\n  environment with randomly initialized weights and save those weights for comparison with the \ud83e\udd17 Transformers version\n  of your model\n- Make sure you are using the easiest way of calling a forward pass in the original repository. Ideally, you want to\n  find the function in the original repository that **only** calls a single forward pass, *i.e.* that is often called\n  `predict`, `evaluate`, `forward` or `__call__`. You don't want to debug a function that calls `forward`\n  multiple times, *e.g.* to generate text, like `autoregressive_sample`, `generate`.\n- Try to separate the tokenization from the model's *forward* pass. If the original repository shows examples where\n  you have to input a string, then try to find out where in the forward call the string input is changed to input ids\n  and start from this point. This might mean that you have to possibly write a small script yourself or change the\n  original code so that you can directly input the ids instead of an input string.\n- Make sure that the model in your debugging setup is **not** in training mode, which often causes the model to yield\n  random outputs due to multiple dropout layers in the model. Make sure that the forward pass in your debugging\n  environment is **deterministic** so that the dropout layers are not used. Or use *transformers.utils.set_seed*\n  if the old and new implementations are in the same framework.\n\nThe following section gives you more specific details/tips on how you can do this for *brand_new_bert*.\n\n### 5.-14. Port BrandNewBert to \ud83e\udd17 Transformers\n\nNext, you can finally start adding new code to \ud83e\udd17 Transformers. Go into the clone of your \ud83e\udd17 Transformers' fork:\n\n```bash\ncd transformers\n```\n\nIn the special case that you are adding a model whose architecture exactly matches the model architecture of an\nexisting model you only have to add a conversion script as described in [this section](#write-a-conversion-script).\nIn this case, you can just re-use the whole model architecture of the already existing model.\n\nOtherwise, let's start generating a new model. We recommend using the following script to add a model starting from\nan existing model:\n\n```bash\ntransformers-cli add-new-model-like\n```\n\nYou will be prompted with a questionnaire to fill in the basic information of your model.\n\n**Open a Pull Request on the main huggingface/transformers repo**\n\nBefore starting to adapt the automatically generated code, now is the time to open a \u201cWork in progress (WIP)\u201d pull\nrequest, *e.g.* \u201c[WIP] Add *brand_new_bert*\u201d, in \ud83e\udd17 Transformers so that you and the Hugging Face team can work\nside-by-side on integrating the model into \ud83e\udd17 Transformers.\n\nYou should do the following:\n\n1. Create a branch with a descriptive name from your main branch\n\n   ```bash\n   git checkout -b add_brand_new_bert\n   ```\n\n2. Commit the automatically generated code:\n\n   ```bash\n   git add .\n   git commit\n   ```\n\n3. Fetch and rebase to current main\n\n   ```bash\n   git fetch upstream\n   git rebase upstream/main\n   ```\n\n4. Push the changes to your account using:\n\n   ```bash\n   git push -u origin a-descriptive-name-for-my-changes\n   ```\n\n5. Once you are satisfied, go to the webpage of your fork on GitHub. Click on \u201cPull request\u201d."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: dd4afb93-e2ef-45d4-a0bb-9e9321e9a561
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"dd4afb93-e2ef-45d4-a0bb-9e9321e9a561\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9b0f4da-8c49-44a8-963a-251ef8b99637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7c239810-7002-40f1-a682-f6b67a46acf5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"aed4a4fa678cc069f832a5ca2526338f94a8deb9021ac1c10d13f746c6e35a4a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"80afb18d-3d2e-45aa-a61c-461d149c4132\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"20f149cb0d6b0f8977e675d2a7146ad47e9912ee7cc77612ac04c2e637a54a32\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 24712, \"end_char_idx\": 28028, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "ref_doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "chroma:document": "Go into the clone of your \ud83e\udd17 Transformers' fork:\n\n```bash\ncd transformers\n```\n\nIn the special case that you are adding a model whose architecture exactly matches the model architecture of an\nexisting model you only have to add a conversion script as described in [this section](#write-a-conversion-script).\nIn this case, you can just re-use the whole model architecture of the already existing model.\n\nOtherwise, let's start generating a new model. We recommend using the following script to add a model starting from\nan existing model:\n\n```bash\ntransformers-cli add-new-model-like\n```\n\nYou will be prompted with a questionnaire to fill in the basic information of your model.\n\n**Open a Pull Request on the main huggingface/transformers repo**\n\nBefore starting to adapt the automatically generated code, now is the time to open a \u201cWork in progress (WIP)\u201d pull\nrequest, *e.g.* \u201c[WIP] Add *brand_new_bert*\u201d, in \ud83e\udd17 Transformers so that you and the Hugging Face team can work\nside-by-side on integrating the model into \ud83e\udd17 Transformers.\n\nYou should do the following:\n\n1. Create a branch with a descriptive name from your main branch\n\n   ```bash\n   git checkout -b add_brand_new_bert\n   ```\n\n2. Commit the automatically generated code:\n\n   ```bash\n   git add .\n   git commit\n   ```\n\n3. Fetch and rebase to current main\n\n   ```bash\n   git fetch upstream\n   git rebase upstream/main\n   ```\n\n4. Push the changes to your account using:\n\n   ```bash\n   git push -u origin a-descriptive-name-for-my-changes\n   ```\n\n5. Once you are satisfied, go to the webpage of your fork on GitHub. Click on \u201cPull request\u201d. Make sure to add the\n   GitHub handle of some members of the Hugging Face team as reviewers, so that the Hugging Face team gets notified for\n   future changes.\n\n6. Change the PR into a draft by clicking on \u201cConvert to draft\u201d on the right of the GitHub pull request web page.\n\nIn the following, whenever you have made some progress, don't forget to commit your work and push it to your account so\nthat it shows in the pull request. Additionally, you should make sure to update your work with the current main from\ntime to time by doing:\n\n```bash\ngit fetch upstream\ngit merge upstream/main\n```\n\nIn general, all questions you might have regarding the model or your implementation should be asked in your PR and\ndiscussed/solved in the PR. This way, the Hugging Face team will always be notified when you are committing new code or\nif you have a question. It is often very helpful to point the Hugging Face team to your added code so that the Hugging\nFace team can efficiently understand your problem or question.\n\nTo do so, you can go to the \u201cFiles changed\u201d tab where you see all of your changes, go to a line regarding which you\nwant to ask a question, and click on the \u201c+\u201d symbol to add a comment. Whenever a question or problem has been solved,\nyou can click on the \u201cResolve\u201d button of the created comment.\n\nIn the same way, the Hugging Face team will open comments when reviewing your code. We recommend asking most questions\non GitHub on your PR. For some very general questions that are not very useful for the public, feel free to ping the\nHugging Face team by Slack or email.\n\n**5. Adapt the generated models code for brand_new_bert**\n\nAt first, we will focus only on the model itself and not care about the tokenizer."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 80afb18d-3d2e-45aa-a61c-461d149c4132
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"80afb18d-3d2e-45aa-a61c-461d149c4132\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9b0f4da-8c49-44a8-963a-251ef8b99637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dd4afb93-e2ef-45d4-a0bb-9e9321e9a561\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"248c8d63a5f70a05125001442207460dd278df6b942fd7743725879f01aa69ca\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cc86e993-1b99-4b42-b268-36d898a5d378\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8e2212f255b0b3d4ec2a1f39de3541255cf8cdbf9dfcde027eba571fa7784bd2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 26305, \"end_char_idx\": 29599, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "ref_doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "chroma:document": "Make sure to add the\n   GitHub handle of some members of the Hugging Face team as reviewers, so that the Hugging Face team gets notified for\n   future changes.\n\n6. Change the PR into a draft by clicking on \u201cConvert to draft\u201d on the right of the GitHub pull request web page.\n\nIn the following, whenever you have made some progress, don't forget to commit your work and push it to your account so\nthat it shows in the pull request. Additionally, you should make sure to update your work with the current main from\ntime to time by doing:\n\n```bash\ngit fetch upstream\ngit merge upstream/main\n```\n\nIn general, all questions you might have regarding the model or your implementation should be asked in your PR and\ndiscussed/solved in the PR. This way, the Hugging Face team will always be notified when you are committing new code or\nif you have a question. It is often very helpful to point the Hugging Face team to your added code so that the Hugging\nFace team can efficiently understand your problem or question.\n\nTo do so, you can go to the \u201cFiles changed\u201d tab where you see all of your changes, go to a line regarding which you\nwant to ask a question, and click on the \u201c+\u201d symbol to add a comment. Whenever a question or problem has been solved,\nyou can click on the \u201cResolve\u201d button of the created comment.\n\nIn the same way, the Hugging Face team will open comments when reviewing your code. We recommend asking most questions\non GitHub on your PR. For some very general questions that are not very useful for the public, feel free to ping the\nHugging Face team by Slack or email.\n\n**5. Adapt the generated models code for brand_new_bert**\n\nAt first, we will focus only on the model itself and not care about the tokenizer. All the relevant code should be\nfound in the generated files `src/transformers/models/brand_new_bert/modeling_brand_new_bert.py` and\n`src/transformers/models/brand_new_bert/configuration_brand_new_bert.py`.\n\nNow you can finally start coding :). The generated code in\n`src/transformers/models/brand_new_bert/modeling_brand_new_bert.py` will either have the same architecture as BERT if\nit's an encoder-only model or BART if it's an encoder-decoder model. At this point, you should remind yourself what\nyou've learned in the beginning about the theoretical aspects of the model: *How is the model different from BERT or\nBART?*\". Implement those changes which often means changing the *self-attention* layer, the order of the normalization\nlayer, etc\u2026 Again, it is often useful to look at the similar architecture of already existing models in Transformers to\nget a better feeling of how your model should be implemented.\n\n**Note** that at this point, you don't have to be very sure that your code is fully correct or clean. Rather, it is\nadvised to add a first *unclean*, copy-pasted version of the original code to\n`src/transformers/models/brand_new_bert/modeling_brand_new_bert.py` until you feel like all the necessary code is\nadded. From our experience, it is much more efficient to quickly add a first version of the required code and\nimprove/correct the code iteratively with the conversion script as described in the next section. The only thing that\nhas to work at this point is that you can instantiate the \ud83e\udd17 Transformers implementation of *brand_new_bert*, *i.e."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: cc86e993-1b99-4b42-b268-36d898a5d378
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"cc86e993-1b99-4b42-b268-36d898a5d378\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9b0f4da-8c49-44a8-963a-251ef8b99637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"80afb18d-3d2e-45aa-a61c-461d149c4132\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"d5829f36c3c4008a64a785966f778ccadf56e69a2c6ff5e6ec209efd67957489\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"33ad78e4-91cd-4b5c-8128-8fac169101e5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"db18b816a8bb6f0bb1a4f3592d6a3b6c7ece174defcf84fdd6e00208b8fe390e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 27892, \"end_char_idx\": 31126, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "ref_doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "chroma:document": "Adapt the generated models code for brand_new_bert**\n\nAt first, we will focus only on the model itself and not care about the tokenizer. All the relevant code should be\nfound in the generated files `src/transformers/models/brand_new_bert/modeling_brand_new_bert.py` and\n`src/transformers/models/brand_new_bert/configuration_brand_new_bert.py`.\n\nNow you can finally start coding :). The generated code in\n`src/transformers/models/brand_new_bert/modeling_brand_new_bert.py` will either have the same architecture as BERT if\nit's an encoder-only model or BART if it's an encoder-decoder model. At this point, you should remind yourself what\nyou've learned in the beginning about the theoretical aspects of the model: *How is the model different from BERT or\nBART?*\". Implement those changes which often means changing the *self-attention* layer, the order of the normalization\nlayer, etc\u2026 Again, it is often useful to look at the similar architecture of already existing models in Transformers to\nget a better feeling of how your model should be implemented.\n\n**Note** that at this point, you don't have to be very sure that your code is fully correct or clean. Rather, it is\nadvised to add a first *unclean*, copy-pasted version of the original code to\n`src/transformers/models/brand_new_bert/modeling_brand_new_bert.py` until you feel like all the necessary code is\nadded. From our experience, it is much more efficient to quickly add a first version of the required code and\nimprove/correct the code iteratively with the conversion script as described in the next section. The only thing that\nhas to work at this point is that you can instantiate the \ud83e\udd17 Transformers implementation of *brand_new_bert*, *i.e.* the\nfollowing command should work:\n\n```python\nfrom transformers import BrandNewBertModel, BrandNewBertConfig\n\nmodel = BrandNewBertModel(BrandNewBertConfig())\n```\n\nThe above command will create a model according to the default parameters as defined in `BrandNewBertConfig()` with\nrandom weights, thus making sure that the `init()` methods of all components works.\n\nNote that all random initialization should happen in the `_init_weights` method of your `BrandnewBertPreTrainedModel`\nclass. It should initialize all leaf modules depending on the variables of the config. Here is an example with the\nBERT `_init_weights` method:\n\n```py\ndef _init_weights(self, module):\n    \"\"\"Initialize the weights\"\"\"\n    if isinstance(module, nn.Linear):\n        module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)\n        if module.bias is not None:\n            module.bias.data.zero_()\n    elif isinstance(module, nn.Embedding):\n        module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)\n        if module.padding_idx is not None:\n            module.weight.data[module.padding_idx].zero_()\n    elif isinstance(module, nn.LayerNorm):\n        module.bias.data.zero_()\n        module.weight.data.fill_(1.0)\n```\n\nYou can have some more custom schemes if you need a special initialization for some modules. For instance, in\n`Wav2Vec2ForPreTraining`, the last two linear layers need to have the initialization of the regular PyTorch `nn.Linear`\nbut all the other ones should use an initialization as above."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 33ad78e4-91cd-4b5c-8128-8fac169101e5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"33ad78e4-91cd-4b5c-8128-8fac169101e5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9b0f4da-8c49-44a8-963a-251ef8b99637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cc86e993-1b99-4b42-b268-36d898a5d378\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"ae0114515928a2e185b6a6b2ff16e1a6e6a4449445949b82c72b5cfd86b5c0c4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6d82b6a4-d526-478c-95c1-18300c697686\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6b72d901339824be5511fecce72586feb08fa401f616b29675add2a5ded0bf0e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 29465, \"end_char_idx\": 32734, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "ref_doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "chroma:document": "The only thing that\nhas to work at this point is that you can instantiate the \ud83e\udd17 Transformers implementation of *brand_new_bert*, *i.e.* the\nfollowing command should work:\n\n```python\nfrom transformers import BrandNewBertModel, BrandNewBertConfig\n\nmodel = BrandNewBertModel(BrandNewBertConfig())\n```\n\nThe above command will create a model according to the default parameters as defined in `BrandNewBertConfig()` with\nrandom weights, thus making sure that the `init()` methods of all components works.\n\nNote that all random initialization should happen in the `_init_weights` method of your `BrandnewBertPreTrainedModel`\nclass. It should initialize all leaf modules depending on the variables of the config. Here is an example with the\nBERT `_init_weights` method:\n\n```py\ndef _init_weights(self, module):\n    \"\"\"Initialize the weights\"\"\"\n    if isinstance(module, nn.Linear):\n        module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)\n        if module.bias is not None:\n            module.bias.data.zero_()\n    elif isinstance(module, nn.Embedding):\n        module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)\n        if module.padding_idx is not None:\n            module.weight.data[module.padding_idx].zero_()\n    elif isinstance(module, nn.LayerNorm):\n        module.bias.data.zero_()\n        module.weight.data.fill_(1.0)\n```\n\nYou can have some more custom schemes if you need a special initialization for some modules. For instance, in\n`Wav2Vec2ForPreTraining`, the last two linear layers need to have the initialization of the regular PyTorch `nn.Linear`\nbut all the other ones should use an initialization as above. This is coded like this:\n\n```py\ndef _init_weights(self, module):\n    \"\"\"Initialize the weights\"\"\"\n    if isinstance(module, Wav2Vec2ForPreTraining):\n        module.project_hid.reset_parameters()\n        module.project_q.reset_parameters()\n        module.project_hid._is_hf_initialized = True\n        module.project_q._is_hf_initialized = True\n    elif isinstance(module, nn.Linear):\n        module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)\n        if module.bias is not None:\n            module.bias.data.zero_()\n```\n\nThe `_is_hf_initialized` flag is internally used to make sure we only initialize a submodule once. By setting it to\n`True` for `module.project_q` and `module.project_hid`, we make sure the custom initialization we did is not overridden later on,\nthe `_init_weights` function won't be applied to them.\n\n**6. Write a conversion script**\n\nNext, you should write a conversion script that lets you convert the checkpoint you used to debug *brand_new_bert* in\nthe original repository to a checkpoint compatible with your just created \ud83e\udd17 Transformers implementation of\n*brand_new_bert*. It is not advised to write the conversion script from scratch, but rather to look through already\nexisting conversion scripts in \ud83e\udd17 Transformers for one that has been used to convert a similar model that was written in\nthe same framework as *brand_new_bert*. Usually, it is enough to copy an already existing conversion script and\nslightly adapt it for your use case. Don't hesitate to ask the Hugging Face team to point you to a similar already\nexisting conversion script for your model."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 6d82b6a4-d526-478c-95c1-18300c697686
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"6d82b6a4-d526-478c-95c1-18300c697686\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9b0f4da-8c49-44a8-963a-251ef8b99637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"33ad78e4-91cd-4b5c-8128-8fac169101e5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"51f094ff6ee5d474b9b26600a36717f2b47f9f47bf2f756cd54aeeea3652f962\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"836f47a5-e9a8-4748-81df-e9ba4c761a48\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2ccc8f19db1200e2188c86c7db5e40ba74075a88a1a2956a5fae250f51c6e58a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 30928, \"end_char_idx\": 33917, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "ref_doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "chroma:document": "For instance, in\n`Wav2Vec2ForPreTraining`, the last two linear layers need to have the initialization of the regular PyTorch `nn.Linear`\nbut all the other ones should use an initialization as above. This is coded like this:\n\n```py\ndef _init_weights(self, module):\n    \"\"\"Initialize the weights\"\"\"\n    if isinstance(module, Wav2Vec2ForPreTraining):\n        module.project_hid.reset_parameters()\n        module.project_q.reset_parameters()\n        module.project_hid._is_hf_initialized = True\n        module.project_q._is_hf_initialized = True\n    elif isinstance(module, nn.Linear):\n        module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)\n        if module.bias is not None:\n            module.bias.data.zero_()\n```\n\nThe `_is_hf_initialized` flag is internally used to make sure we only initialize a submodule once. By setting it to\n`True` for `module.project_q` and `module.project_hid`, we make sure the custom initialization we did is not overridden later on,\nthe `_init_weights` function won't be applied to them.\n\n**6. Write a conversion script**\n\nNext, you should write a conversion script that lets you convert the checkpoint you used to debug *brand_new_bert* in\nthe original repository to a checkpoint compatible with your just created \ud83e\udd17 Transformers implementation of\n*brand_new_bert*. It is not advised to write the conversion script from scratch, but rather to look through already\nexisting conversion scripts in \ud83e\udd17 Transformers for one that has been used to convert a similar model that was written in\nthe same framework as *brand_new_bert*. Usually, it is enough to copy an already existing conversion script and\nslightly adapt it for your use case. Don't hesitate to ask the Hugging Face team to point you to a similar already\nexisting conversion script for your model.\n\n- If you are porting a model from TensorFlow to PyTorch, a good starting point might be BERT's conversion script [here](https://github.com/huggingface/transformers/blob/7acfa95afb8194f8f9c1f4d2c6028224dbed35a2/src/transformers/models/bert/modeling_bert.py#L91)\n- If you are porting a model from PyTorch to PyTorch, a good starting point might be BART's conversion script [here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/bart/convert_bart_original_pytorch_checkpoint_to_pytorch.py)\n\nIn the following, we'll quickly explain how PyTorch models store layer weights and define layer names. In PyTorch, the\nname of a layer is defined by the name of the class attribute you give the layer. Let's define a dummy model in\nPyTorch, called `SimpleModel` as follows:\n\n```python\nfrom torch import nn\n\n\nclass SimpleModel(nn.Module):\n    def __init__(self):\n        super().__init__()\n        self.dense = nn.Linear(10, 10)\n        self.intermediate = nn.Linear(10, 10)\n        self.layer_norm = nn.LayerNorm(10)\n```\n\nNow we can create an instance of this model definition which will fill all weights: `dense`, `intermediate`,\n`layer_norm` with random weights."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 836f47a5-e9a8-4748-81df-e9ba4c761a48
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"836f47a5-e9a8-4748-81df-e9ba4c761a48\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9b0f4da-8c49-44a8-963a-251ef8b99637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6d82b6a4-d526-478c-95c1-18300c697686\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"50658d17d1e104ffb45ae39b0bdd7bd16bcfb99a8be36400a1e6f5a05a8ab61c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7e44d8db-ad92-46e8-bc48-445875cbb0c0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"dcc459908fc9931a611151b70f769b6dc407c463e9f82a6a53b2d35720959c46\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 32505, \"end_char_idx\": 34367, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "ref_doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "chroma:document": "Usually, it is enough to copy an already existing conversion script and\nslightly adapt it for your use case. Don't hesitate to ask the Hugging Face team to point you to a similar already\nexisting conversion script for your model.\n\n- If you are porting a model from TensorFlow to PyTorch, a good starting point might be BERT's conversion script [here](https://github.com/huggingface/transformers/blob/7acfa95afb8194f8f9c1f4d2c6028224dbed35a2/src/transformers/models/bert/modeling_bert.py#L91)\n- If you are porting a model from PyTorch to PyTorch, a good starting point might be BART's conversion script [here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/bart/convert_bart_original_pytorch_checkpoint_to_pytorch.py)\n\nIn the following, we'll quickly explain how PyTorch models store layer weights and define layer names. In PyTorch, the\nname of a layer is defined by the name of the class attribute you give the layer. Let's define a dummy model in\nPyTorch, called `SimpleModel` as follows:\n\n```python\nfrom torch import nn\n\n\nclass SimpleModel(nn.Module):\n    def __init__(self):\n        super().__init__()\n        self.dense = nn.Linear(10, 10)\n        self.intermediate = nn.Linear(10, 10)\n        self.layer_norm = nn.LayerNorm(10)\n```\n\nNow we can create an instance of this model definition which will fill all weights: `dense`, `intermediate`,\n`layer_norm` with random weights. We can print the model to see its architecture\n\n```python\nmodel = SimpleModel()\n\nprint(model)\n```\n\nThis will print out the following:\n\n```\nSimpleModel(\n  (dense): Linear(in_features=10, out_features=10, bias=True)\n  (intermediate): Linear(in_features=10, out_features=10, bias=True)\n  (layer_norm): LayerNorm((10,), eps=1e-05, elementwise_affine=True)\n)\n```\n\nWe can see that the layer names are defined by the name of the class attribute in PyTorch."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 7e44d8db-ad92-46e8-bc48-445875cbb0c0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"7e44d8db-ad92-46e8-bc48-445875cbb0c0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9b0f4da-8c49-44a8-963a-251ef8b99637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"836f47a5-e9a8-4748-81df-e9ba4c761a48\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"3beec97ae0ec67f8e1ebedb2404ffc24efe524b51848143cc43ca0ecd5f9907a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"276b2f38-97ee-49d5-9488-b974f3c867c5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"85865ed2b2bb37e4c963177732ba9640fd0c81396e8aa0c9a6463ce10c00cbb3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 33357, \"end_char_idx\": 35618, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "ref_doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "chroma:document": "In PyTorch, the\nname of a layer is defined by the name of the class attribute you give the layer. Let's define a dummy model in\nPyTorch, called `SimpleModel` as follows:\n\n```python\nfrom torch import nn\n\n\nclass SimpleModel(nn.Module):\n    def __init__(self):\n        super().__init__()\n        self.dense = nn.Linear(10, 10)\n        self.intermediate = nn.Linear(10, 10)\n        self.layer_norm = nn.LayerNorm(10)\n```\n\nNow we can create an instance of this model definition which will fill all weights: `dense`, `intermediate`,\n`layer_norm` with random weights. We can print the model to see its architecture\n\n```python\nmodel = SimpleModel()\n\nprint(model)\n```\n\nThis will print out the following:\n\n```\nSimpleModel(\n  (dense): Linear(in_features=10, out_features=10, bias=True)\n  (intermediate): Linear(in_features=10, out_features=10, bias=True)\n  (layer_norm): LayerNorm((10,), eps=1e-05, elementwise_affine=True)\n)\n```\n\nWe can see that the layer names are defined by the name of the class attribute in PyTorch. You can print out the weight\nvalues of a specific layer:\n\n```python\nprint(model.dense.weight.data)\n```\n\nto see that the weights were randomly initialized\n\n```\ntensor([[-0.0818,  0.2207, -0.0749, -0.0030,  0.0045, -0.1569, -0.1598,  0.0212,\n         -0.2077,  0.2157],\n        [ 0.1044,  0.0201,  0.0990,  0.2482,  0.3116,  0.2509,  0.2866, -0.2190,\n          0.2166, -0.0212],\n        [-0.2000,  0.1107, -0.1999, -0.3119,  0.1559,  0.0993,  0.1776, -0.1950,\n         -0.1023, -0.0447],\n        [-0.0888, -0.1092,  0.2281,  0.0336,  0.1817, -0.0115,  0.2096,  0.1415,\n         -0.1876, -0.2467],\n        [ 0.2208, -0.2352, -0.1426, -0.2636, -0.2889, -0.2061, -0.2849, -0.0465,\n          0.2577,  0.0402],\n        [ 0.1502,  0.2465,  0.2566,  0.0693,  0.2352, -0.0530,  0.1859, -0.0604,\n          0.2132,  0.1680],\n        [ 0.1733, -0.2407, -0.1721,  0.1484,  0.0358, -0.0633, -0.0721, -0.0090,\n          0.2707, -0.2509],\n        [-0.1173,  0.1561,  0.2945,  0.0595, -0.1996,  0.2988, -0.0802,  0.0407,\n          0.1829, -0.1568],\n        [-0.1164, -0.2228, -0.0403,  0.0428,  0.1339,  0.0047,  0.1967,  0.2923,\n          0.0333, -0.0536],\n        [-0.1492, -0.1616,  0.1057,  0.1950, -0.2807, -0.2710, -0.1586,  0.0739,\n          0.2220,  0.2358]])."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 276b2f38-97ee-49d5-9488-b974f3c867c5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"276b2f38-97ee-49d5-9488-b974f3c867c5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9b0f4da-8c49-44a8-963a-251ef8b99637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7e44d8db-ad92-46e8-bc48-445875cbb0c0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"1653a302ce09457180eb3ee3d403a09912e2c6447e20111723d5a1e8d3c00884\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e6c0a3c7-10d0-4c91-a914-be3e0ecfd55a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1eb8bdbc68f31d016c44971931cb0e0b77d67b3197d8f4d9164c75635f50ff1e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 35619, \"end_char_idx\": 38901, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "ref_doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "chroma:document": "```\n\nIn the conversion script, you should fill those randomly initialized weights with the exact weights of the\ncorresponding layer in the checkpoint. *E.g.*\n\n```python\n# retrieve matching layer weights, e.g. by\n# recursive algorithm\nlayer_name = \"dense\"\npretrained_weight = array_of_dense_layer\n\nmodel_pointer = getattr(model, \"dense\")\n\nmodel_pointer.weight.data = torch.from_numpy(pretrained_weight)\n```\n\nWhile doing so, you must verify that each randomly initialized weight of your PyTorch model and its corresponding\npretrained checkpoint weight exactly match in both **shape and name**. To do so, it is **necessary** to add assert\nstatements for the shape and print out the names of the checkpoints weights. E.g. you should add statements like:\n\n```python\nassert (\n    model_pointer.weight.shape == pretrained_weight.shape\n), f\"Pointer shape of random weight {model_pointer.shape} and array shape of checkpoint weight {pretrained_weight.shape} mismatched\"\n```\n\nBesides, you should also print out the names of both weights to make sure they match, *e.g.*\n\n```python\nlogger.info(f\"Initialize PyTorch weight {layer_name} from {pretrained_weight.name}\")\n```\n\nIf either the shape or the name doesn't match, you probably assigned the wrong checkpoint weight to a randomly\ninitialized layer of the \ud83e\udd17 Transformers implementation.\n\nAn incorrect shape is most likely due to an incorrect setting of the config parameters in `BrandNewBertConfig()` that\ndo not exactly match those that were used for the checkpoint you want to convert. However, it could also be that\nPyTorch's implementation of a layer requires the weight to be transposed beforehand.\n\nFinally, you should also check that **all** required weights are initialized and print out all checkpoint weights that\nwere not used for initialization to make sure the model is correctly converted. It is completely normal, that the\nconversion trials fail with either a wrong shape statement or a wrong name assignment. This is most likely because either\nyou used incorrect parameters in `BrandNewBertConfig()`, have a wrong architecture in the \ud83e\udd17 Transformers\nimplementation, you have a bug in the `init()` functions of one of the components of the \ud83e\udd17 Transformers\nimplementation or you need to transpose one of the checkpoint weights.\n\nThis step should be iterated with the previous step until all weights of the checkpoint are correctly loaded in the\nTransformers model. Having correctly loaded the checkpoint into the \ud83e\udd17 Transformers implementation, you can then save\nthe model under a folder of your choice `/path/to/converted/checkpoint/folder` that should then contain both a\n`pytorch_model.bin` file and a `config.json` file:\n\n```python\nmodel.save_pretrained(\"/path/to/converted/checkpoint/folder\")\n```\n\n**7. Implement the forward pass**\n\nHaving managed to correctly load the pretrained weights into the \ud83e\udd17 Transformers implementation, you should now make\nsure that the forward pass is correctly implemented. In [Get familiar with the original repository](#3-4-run-a-pretrained-checkpoint-using-the-original-repository), you have already created a script that runs a forward\npass of the model using the original repository. Now you should write an analogous script using the \ud83e\udd17 Transformers\nimplementation instead of the original one."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: e6c0a3c7-10d0-4c91-a914-be3e0ecfd55a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"e6c0a3c7-10d0-4c91-a914-be3e0ecfd55a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9b0f4da-8c49-44a8-963a-251ef8b99637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"276b2f38-97ee-49d5-9488-b974f3c867c5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"b2c0164edb9c4a63a308faab12e4616eebd84072d12d80a9b83c10f00983a1c0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7afbc7a8-b1d8-4622-8178-ffb2d2e6dcd5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5334815d10b82c623f1d6822a7028d94bfda04723221d7bd352af05b898a215a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 37147, \"end_char_idx\": 40667, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "ref_doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "chroma:document": "However, it could also be that\nPyTorch's implementation of a layer requires the weight to be transposed beforehand.\n\nFinally, you should also check that **all** required weights are initialized and print out all checkpoint weights that\nwere not used for initialization to make sure the model is correctly converted. It is completely normal, that the\nconversion trials fail with either a wrong shape statement or a wrong name assignment. This is most likely because either\nyou used incorrect parameters in `BrandNewBertConfig()`, have a wrong architecture in the \ud83e\udd17 Transformers\nimplementation, you have a bug in the `init()` functions of one of the components of the \ud83e\udd17 Transformers\nimplementation or you need to transpose one of the checkpoint weights.\n\nThis step should be iterated with the previous step until all weights of the checkpoint are correctly loaded in the\nTransformers model. Having correctly loaded the checkpoint into the \ud83e\udd17 Transformers implementation, you can then save\nthe model under a folder of your choice `/path/to/converted/checkpoint/folder` that should then contain both a\n`pytorch_model.bin` file and a `config.json` file:\n\n```python\nmodel.save_pretrained(\"/path/to/converted/checkpoint/folder\")\n```\n\n**7. Implement the forward pass**\n\nHaving managed to correctly load the pretrained weights into the \ud83e\udd17 Transformers implementation, you should now make\nsure that the forward pass is correctly implemented. In [Get familiar with the original repository](#3-4-run-a-pretrained-checkpoint-using-the-original-repository), you have already created a script that runs a forward\npass of the model using the original repository. Now you should write an analogous script using the \ud83e\udd17 Transformers\nimplementation instead of the original one. It should look as follows:\n\n```python\nmodel = BrandNewBertModel.from_pretrained(\"/path/to/converted/checkpoint/folder\")\ninput_ids = [0, 4, 4, 3, 2, 4, 1, 7, 19]\noutput = model(input_ids).last_hidden_states\n```\n\nIt is very likely that the \ud83e\udd17 Transformers implementation and the original model implementation don't give the exact\nsame output the very first time or that the forward pass throws an error. Don't be disappointed - it's expected! First,\nyou should make sure that the forward pass doesn't throw any errors. It often happens that the wrong dimensions are\nused leading to a *Dimensionality mismatch* error or that the wrong data type object is used, *e.g.* `torch.long`\ninstead of `torch.float32`. Don't hesitate to ask the Hugging Face team for help, if you don't manage to solve\ncertain errors.\n\nThe final part to make sure the \ud83e\udd17 Transformers implementation works correctly is to ensure that the outputs are\nequivalent to a precision of `1e-3`. First, you should ensure that the output shapes are identical, *i.e.*\n`outputs.shape` should yield the same value for the script of the \ud83e\udd17 Transformers implementation and the original\nimplementation. Next, you should make sure that the output values are identical as well. This one of the most difficult\nparts of adding a new model. Common mistakes why the outputs are not identical are:\n\n- Some layers were not added, *i.e.* an *activation* layer was not added, or the residual connection was forgotten\n- The word embedding matrix was not tied\n- The wrong positional embeddings are used because the original implementation uses on offset\n- Dropout is applied during the forward pass. To fix this make sure *model.training is False* and that no dropout\n  layer is falsely activated during the forward pass, *i.e."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 7afbc7a8-b1d8-4622-8178-ffb2d2e6dcd5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"7afbc7a8-b1d8-4622-8178-ffb2d2e6dcd5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9b0f4da-8c49-44a8-963a-251ef8b99637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e6c0a3c7-10d0-4c91-a914-be3e0ecfd55a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"af050468cd3d7414bc1b8766e57295785252abf8d44d229e8e1d025adf7e32b6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"72bf632b-4866-43fb-90b4-30c2c50c5922\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9b16a2c9b53d32006f57f6fadb5eb275d23da687f1e3cdb18e4db61108186782\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 39303, \"end_char_idx\": 42701, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "ref_doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "chroma:document": "Don't be disappointed - it's expected! First,\nyou should make sure that the forward pass doesn't throw any errors. It often happens that the wrong dimensions are\nused leading to a *Dimensionality mismatch* error or that the wrong data type object is used, *e.g.* `torch.long`\ninstead of `torch.float32`. Don't hesitate to ask the Hugging Face team for help, if you don't manage to solve\ncertain errors.\n\nThe final part to make sure the \ud83e\udd17 Transformers implementation works correctly is to ensure that the outputs are\nequivalent to a precision of `1e-3`. First, you should ensure that the output shapes are identical, *i.e.*\n`outputs.shape` should yield the same value for the script of the \ud83e\udd17 Transformers implementation and the original\nimplementation. Next, you should make sure that the output values are identical as well. This one of the most difficult\nparts of adding a new model. Common mistakes why the outputs are not identical are:\n\n- Some layers were not added, *i.e.* an *activation* layer was not added, or the residual connection was forgotten\n- The word embedding matrix was not tied\n- The wrong positional embeddings are used because the original implementation uses on offset\n- Dropout is applied during the forward pass. To fix this make sure *model.training is False* and that no dropout\n  layer is falsely activated during the forward pass, *i.e.* pass *self.training* to [PyTorch's functional dropout](https://pytorch.org/docs/stable/nn.functional.html?highlight=dropout#torch.nn.functional.dropout)\n\nThe best way to fix the problem is usually to look at the forward pass of the original implementation and the \ud83e\udd17\nTransformers implementation side-by-side and check if there are any differences. Ideally, you should debug/print out\nintermediate outputs of both implementations of the forward pass to find the exact position in the network where the \ud83e\udd17\nTransformers implementation shows a different output than the original implementation. First, make sure that the\nhard-coded `input_ids` in both scripts are identical. Next, verify that the outputs of the first transformation of\nthe `input_ids` (usually the word embeddings) are identical. And then work your way up to the very last layer of the\nnetwork. At some point, you will notice a difference between the two implementations, which should point you to the bug\nin the \ud83e\udd17 Transformers implementation. From our experience, a simple and efficient way is to add many print statements\nin both the original implementation and \ud83e\udd17 Transformers implementation, at the same positions in the network\nrespectively, and to successively remove print statements showing the same values for intermediate presentations.\n\nWhen you're confident that both implementations yield the same output, verify the outputs with\n`torch.allclose(original_output, output, atol=1e-3)`, you're done with the most difficult part! Congratulations - the\nwork left to be done should be a cakewalk \ud83d\ude0a.\n\n**8. Adding all necessary model tests**\n\nAt this point, you have successfully added a new model. However, it is very much possible that the model does not yet\nfully comply with the required design. To make sure, the implementation is fully compatible with \ud83e\udd17 Transformers, all\ncommon tests should pass. The Cookiecutter should have automatically added a test file for your model, probably under\nthe same `tests/models/brand_new_bert/test_modeling_brand_new_bert.py`."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 72bf632b-4866-43fb-90b4-30c2c50c5922
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"72bf632b-4866-43fb-90b4-30c2c50c5922\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9b0f4da-8c49-44a8-963a-251ef8b99637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7afbc7a8-b1d8-4622-8178-ffb2d2e6dcd5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"10f3f215a80510be07cafb8cbd83f12ee4cd5c97db065681ec82d60bfe04847f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"28fd4950-45bd-4fd1-9a9a-f527ed72ac63\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"05d9ba81c31cc571cddc095781768c24359b4547ab1a90e63ac426d8383af90f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 41016, \"end_char_idx\": 44440, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "ref_doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "chroma:document": "Ideally, you should debug/print out\nintermediate outputs of both implementations of the forward pass to find the exact position in the network where the \ud83e\udd17\nTransformers implementation shows a different output than the original implementation. First, make sure that the\nhard-coded `input_ids` in both scripts are identical. Next, verify that the outputs of the first transformation of\nthe `input_ids` (usually the word embeddings) are identical. And then work your way up to the very last layer of the\nnetwork. At some point, you will notice a difference between the two implementations, which should point you to the bug\nin the \ud83e\udd17 Transformers implementation. From our experience, a simple and efficient way is to add many print statements\nin both the original implementation and \ud83e\udd17 Transformers implementation, at the same positions in the network\nrespectively, and to successively remove print statements showing the same values for intermediate presentations.\n\nWhen you're confident that both implementations yield the same output, verify the outputs with\n`torch.allclose(original_output, output, atol=1e-3)`, you're done with the most difficult part! Congratulations - the\nwork left to be done should be a cakewalk \ud83d\ude0a.\n\n**8. Adding all necessary model tests**\n\nAt this point, you have successfully added a new model. However, it is very much possible that the model does not yet\nfully comply with the required design. To make sure, the implementation is fully compatible with \ud83e\udd17 Transformers, all\ncommon tests should pass. The Cookiecutter should have automatically added a test file for your model, probably under\nthe same `tests/models/brand_new_bert/test_modeling_brand_new_bert.py`. Run this test file to verify that all common\ntests pass:\n\n```bash\npytest tests/models/brand_new_bert/test_modeling_brand_new_bert.py\n```\n\nHaving fixed all common tests, it is now crucial to ensure that all the nice work you have done is well tested, so that\n\n- a) The community can easily understand your work by looking at specific tests of *brand_new_bert*\n- b) Future changes to your model will not break any important feature of the model.\n\nAt first, integration tests should be added. Those integration tests essentially do the same as the debugging scripts\nyou used earlier to implement the model to \ud83e\udd17 Transformers. A template of those model tests has already added by the\nCookiecutter, called `BrandNewBertModelIntegrationTests` and only has to be filled out by you. To ensure that those\ntests are passing, run\n\n```bash\nRUN_SLOW=1 pytest -sv tests/models/brand_new_bert/test_modeling_brand_new_bert.py::BrandNewBertModelIntegrationTests\n```\n\n<Tip>\n\nIn case you are using Windows, you should replace `RUN_SLOW=1` with `SET RUN_SLOW=1`\n\n</Tip>\n\nSecond, all features that are special to *brand_new_bert* should be tested additionally in a separate test under\n`BrandNewBertModelTester`/`BrandNewBertModelTest`. This part is often forgotten but is extremely useful in two\nways:\n\n- It helps to transfer the knowledge you have acquired during the model addition to the community by showing how the\n  special features of *brand_new_bert* should work.\n- Future contributors can quickly test changes to the model by running those special tests.\n\n\n**9. Implement the tokenizer**\n\nNext, we should add the tokenizer of *brand_new_bert*. Usually, the tokenizer is equivalent to or very similar to an\nalready existing tokenizer of \ud83e\udd17 Transformers."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 28fd4950-45bd-4fd1-9a9a-f527ed72ac63
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"28fd4950-45bd-4fd1-9a9a-f527ed72ac63\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9b0f4da-8c49-44a8-963a-251ef8b99637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"72bf632b-4866-43fb-90b4-30c2c50c5922\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"ed2d002ed644782732385207609108e33ad3d608d7a3e573bd630d14f96d3848\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3f02203b-f2ce-4f0b-a18b-409452d16182\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b1f7182610ff5e7000eb4d67ae7bf2b76a40d9415a374d494774d07bb476ccb8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 43147, \"end_char_idx\": 46452, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "ref_doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "chroma:document": "At first, integration tests should be added. Those integration tests essentially do the same as the debugging scripts\nyou used earlier to implement the model to \ud83e\udd17 Transformers. A template of those model tests has already added by the\nCookiecutter, called `BrandNewBertModelIntegrationTests` and only has to be filled out by you. To ensure that those\ntests are passing, run\n\n```bash\nRUN_SLOW=1 pytest -sv tests/models/brand_new_bert/test_modeling_brand_new_bert.py::BrandNewBertModelIntegrationTests\n```\n\n<Tip>\n\nIn case you are using Windows, you should replace `RUN_SLOW=1` with `SET RUN_SLOW=1`\n\n</Tip>\n\nSecond, all features that are special to *brand_new_bert* should be tested additionally in a separate test under\n`BrandNewBertModelTester`/`BrandNewBertModelTest`. This part is often forgotten but is extremely useful in two\nways:\n\n- It helps to transfer the knowledge you have acquired during the model addition to the community by showing how the\n  special features of *brand_new_bert* should work.\n- Future contributors can quickly test changes to the model by running those special tests.\n\n\n**9. Implement the tokenizer**\n\nNext, we should add the tokenizer of *brand_new_bert*. Usually, the tokenizer is equivalent to or very similar to an\nalready existing tokenizer of \ud83e\udd17 Transformers.\n\nIt is very important to find/extract the original tokenizer file and to manage to load this file into the \ud83e\udd17\nTransformers' implementation of the tokenizer.\n\nTo ensure that the tokenizer works correctly, it is recommended to first create a script in the original repository\nthat inputs a string and returns the `input_ids`. It could look similar to this (in pseudo-code):\n\n```python\ninput_str = \"This is a long example input string containing special characters .$?-, numbers 2872 234 12 and words.\"\nmodel = BrandNewBertModel.load_pretrained_checkpoint(\"/path/to/checkpoint/\")\ninput_ids = model.tokenize(input_str)\n```\n\nYou might have to take a deeper look again into the original repository to find the correct tokenizer function or you\nmight even have to do changes to your clone of the original repository to only output the `input_ids`. Having written\na functional tokenization script that uses the original repository, an analogous script for \ud83e\udd17 Transformers should be\ncreated. It should look similar to this:\n\n```python\nfrom transformers import BrandNewBertTokenizer\n\ninput_str = \"This is a long example input string containing special characters .$?-, numbers 2872 234 12 and words.\"\n\ntokenizer = BrandNewBertTokenizer.from_pretrained(\"/path/to/tokenizer/folder/\")\n\ninput_ids = tokenizer(input_str).input_ids\n```\n\nWhen both `input_ids` yield the same values, as a final step a tokenizer test file should also be added.\n\nAnalogous to the modeling test files of *brand_new_bert*, the tokenization test files of *brand_new_bert* should\ncontain a couple of hard-coded integration tests.\n\n**10. Run End-to-end integration tests**\n\nHaving added the tokenizer, you should also add a couple of end-to-end integration tests using both the model and the\ntokenizer to `tests/models/brand_new_bert/test_modeling_brand_new_bert.py` in \ud83e\udd17 Transformers.\nSuch a test should show on a meaningful\ntext-to-text sample that the \ud83e\udd17 Transformers implementation works as expected. A meaningful text-to-text sample can\ninclude *e.g."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 3f02203b-f2ce-4f0b-a18b-409452d16182
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"3f02203b-f2ce-4f0b-a18b-409452d16182\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9b0f4da-8c49-44a8-963a-251ef8b99637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"28fd4950-45bd-4fd1-9a9a-f527ed72ac63\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8537ed84ac95a2b5f9263ec9ecf6e4a9d3a3e2c51c631386db93bd79a1589911\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8ff909a1-9e71-4394-b22f-4e6442d060b8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b578a32d139f003213b9aeadf7461667bf2e8219bd4a3dba8834d879e11cf3f9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 44764, \"end_char_idx\": 48147, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "ref_doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "chroma:document": "It could look similar to this (in pseudo-code):\n\n```python\ninput_str = \"This is a long example input string containing special characters .$?-, numbers 2872 234 12 and words.\"\nmodel = BrandNewBertModel.load_pretrained_checkpoint(\"/path/to/checkpoint/\")\ninput_ids = model.tokenize(input_str)\n```\n\nYou might have to take a deeper look again into the original repository to find the correct tokenizer function or you\nmight even have to do changes to your clone of the original repository to only output the `input_ids`. Having written\na functional tokenization script that uses the original repository, an analogous script for \ud83e\udd17 Transformers should be\ncreated. It should look similar to this:\n\n```python\nfrom transformers import BrandNewBertTokenizer\n\ninput_str = \"This is a long example input string containing special characters .$?-, numbers 2872 234 12 and words.\"\n\ntokenizer = BrandNewBertTokenizer.from_pretrained(\"/path/to/tokenizer/folder/\")\n\ninput_ids = tokenizer(input_str).input_ids\n```\n\nWhen both `input_ids` yield the same values, as a final step a tokenizer test file should also be added.\n\nAnalogous to the modeling test files of *brand_new_bert*, the tokenization test files of *brand_new_bert* should\ncontain a couple of hard-coded integration tests.\n\n**10. Run End-to-end integration tests**\n\nHaving added the tokenizer, you should also add a couple of end-to-end integration tests using both the model and the\ntokenizer to `tests/models/brand_new_bert/test_modeling_brand_new_bert.py` in \ud83e\udd17 Transformers.\nSuch a test should show on a meaningful\ntext-to-text sample that the \ud83e\udd17 Transformers implementation works as expected. A meaningful text-to-text sample can\ninclude *e.g.* a source-to-target-translation pair, an article-to-summary pair, a question-to-answer pair, etc\u2026 If none\nof the ported checkpoints has been fine-tuned on a downstream task it is enough to simply rely on the model tests. In a\nfinal step to ensure that the model is fully functional, it is advised that you also run all tests on GPU. It can\nhappen that you forgot to add some `.to(self.device)` statements to internal tensors of the model, which in such a\ntest would show in an error. In case you have no access to a GPU, the Hugging Face team can take care of running those\ntests for you.\n\n**11. Add Docstring**\n\nNow, all the necessary functionality for *brand_new_bert* is added - you're almost done! The only thing left to add is\na nice docstring and a doc page. The Cookiecutter should have added a template file called\n`docs/source/model_doc/brand_new_bert.md` that you should fill out. Users of your model will usually first look at\nthis page before using your model. Hence, the documentation must be understandable and concise. It is very useful for\nthe community to add some *Tips* to show how the model should be used. Don't hesitate to ping the Hugging Face team\nregarding the docstrings.\n\nNext, make sure that the docstring added to `src/transformers/models/brand_new_bert/modeling_brand_new_bert.py` is\ncorrect and included all necessary inputs and outputs. We have a detailed guide about writing documentation and our docstring format [here](writing-documentation). It is always good to remind oneself that documentation should\nbe treated at least as carefully as the code in \ud83e\udd17 Transformers since the documentation is usually the first contact\npoint of the community with the model."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 8ff909a1-9e71-4394-b22f-4e6442d060b8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"8ff909a1-9e71-4394-b22f-4e6442d060b8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9b0f4da-8c49-44a8-963a-251ef8b99637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3f02203b-f2ce-4f0b-a18b-409452d16182\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"3a8ddf295a129d62aca02068c3e3131dc9fc02afa088b413dac735ba5be75c30\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0012d14a-a3d3-4c60-8372-e2165b7d7b81\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9f82f4b7d877fc9637c4cfef639a74321f1dffdea649a7151c866df39e43e489\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 46452, \"end_char_idx\": 49833, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "ref_doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "chroma:document": "* a source-to-target-translation pair, an article-to-summary pair, a question-to-answer pair, etc\u2026 If none\nof the ported checkpoints has been fine-tuned on a downstream task it is enough to simply rely on the model tests. In a\nfinal step to ensure that the model is fully functional, it is advised that you also run all tests on GPU. It can\nhappen that you forgot to add some `.to(self.device)` statements to internal tensors of the model, which in such a\ntest would show in an error. In case you have no access to a GPU, the Hugging Face team can take care of running those\ntests for you.\n\n**11. Add Docstring**\n\nNow, all the necessary functionality for *brand_new_bert* is added - you're almost done! The only thing left to add is\na nice docstring and a doc page. The Cookiecutter should have added a template file called\n`docs/source/model_doc/brand_new_bert.md` that you should fill out. Users of your model will usually first look at\nthis page before using your model. Hence, the documentation must be understandable and concise. It is very useful for\nthe community to add some *Tips* to show how the model should be used. Don't hesitate to ping the Hugging Face team\nregarding the docstrings.\n\nNext, make sure that the docstring added to `src/transformers/models/brand_new_bert/modeling_brand_new_bert.py` is\ncorrect and included all necessary inputs and outputs. We have a detailed guide about writing documentation and our docstring format [here](writing-documentation). It is always good to remind oneself that documentation should\nbe treated at least as carefully as the code in \ud83e\udd17 Transformers since the documentation is usually the first contact\npoint of the community with the model.\n\n**Code refactor**\n\nGreat, now you have added all the necessary code for *brand_new_bert*. At this point, you should correct some potential\nincorrect code style by running:\n\n```bash\nmake style\n```\n\nand verify that your coding style passes the quality check:\n\n```bash\nmake quality\n```\n\nThere are a couple of other very strict design tests in \ud83e\udd17 Transformers that might still be failing, which shows up in\nthe tests of your pull request. This is often because of some missing information in the docstring or some incorrect\nnaming. The Hugging Face team will surely help you if you're stuck here.\n\nLastly, it is always a good idea to refactor one's code after having ensured that the code works correctly. With all\ntests passing, now it's a good time to go over the added code again and do some refactoring.\n\nYou have now finished the coding part, congratulation! \ud83c\udf89 You are Awesome! \ud83d\ude0e\n\n**12. Upload the models to the model hub**\n\nIn this final part, you should convert and upload all checkpoints to the model hub and add a model card for each\nuploaded model checkpoint. You can get familiar with the hub functionalities by reading our [Model sharing and uploading Page](model_sharing). You should work alongside the Hugging Face team here to decide on a fitting name for each\ncheckpoint and to get the required access rights to be able to upload the model under the author's organization of\n*brand_new_bert*. The `push_to_hub` method, present in all models in `transformers`, is a quick and efficient way to push your checkpoint to the hub. A little snippet is pasted below:\n\n```python\nbrand_new_bert.push_to_hub(\"brand_new_bert\")\n# Uncomment the following line to push to an organization."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 0012d14a-a3d3-4c60-8372-e2165b7d7b81
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/add_new_model", "title": "How to add a model to \ud83e\udd17 Transformers?", "tokens": 12286, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"0012d14a-a3d3-4c60-8372-e2165b7d7b81\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9b0f4da-8c49-44a8-963a-251ef8b99637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"63fca19672448cac39a89af4724ae7ee8bbd98232aa6e5da325e271c70dd3701\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8ff909a1-9e71-4394-b22f-4e6442d060b8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/add_new_model\", \"title\": \"How to add a model to \\ud83e\\udd17 Transformers?\", \"tokens\": 12286, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"779c0ee32fc50d0bc1f2450b15ded0fe0f4e28e70ce0dd5bc5dcb7e2d156ccbc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 48149, \"end_char_idx\": 51422, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "ref_doc_id": "d9b0f4da-8c49-44a8-963a-251ef8b99637", "chroma:document": "**Code refactor**\n\nGreat, now you have added all the necessary code for *brand_new_bert*. At this point, you should correct some potential\nincorrect code style by running:\n\n```bash\nmake style\n```\n\nand verify that your coding style passes the quality check:\n\n```bash\nmake quality\n```\n\nThere are a couple of other very strict design tests in \ud83e\udd17 Transformers that might still be failing, which shows up in\nthe tests of your pull request. This is often because of some missing information in the docstring or some incorrect\nnaming. The Hugging Face team will surely help you if you're stuck here.\n\nLastly, it is always a good idea to refactor one's code after having ensured that the code works correctly. With all\ntests passing, now it's a good time to go over the added code again and do some refactoring.\n\nYou have now finished the coding part, congratulation! \ud83c\udf89 You are Awesome! \ud83d\ude0e\n\n**12. Upload the models to the model hub**\n\nIn this final part, you should convert and upload all checkpoints to the model hub and add a model card for each\nuploaded model checkpoint. You can get familiar with the hub functionalities by reading our [Model sharing and uploading Page](model_sharing). You should work alongside the Hugging Face team here to decide on a fitting name for each\ncheckpoint and to get the required access rights to be able to upload the model under the author's organization of\n*brand_new_bert*. The `push_to_hub` method, present in all models in `transformers`, is a quick and efficient way to push your checkpoint to the hub. A little snippet is pasted below:\n\n```python\nbrand_new_bert.push_to_hub(\"brand_new_bert\")\n# Uncomment the following line to push to an organization.\n# brand_new_bert.push_to_hub(\"<organization>/brand_new_bert\")\n```\n\nIt is worth spending some time to create fitting model cards for each checkpoint. The model cards should highlight the\nspecific characteristics of this particular checkpoint, *e.g.* On which dataset was the checkpoint\npretrained/fine-tuned on? On what down-stream task should the model be used? And also include some code on how to\ncorrectly use the model.\n\n**13. (Optional) Add notebook**\n\nIt is very helpful to add a notebook that showcases in-detail how *brand_new_bert* can be used for inference and/or\nfine-tuned on a downstream task. This is not mandatory to merge your PR, but very useful for the community.\n\n**14. Submit your finished PR**\n\nYou're done programming now and can move to the last step, which is getting your PR merged into main. Usually, the\nHugging Face team should have helped you already at this point, but it is worth taking some time to give your finished\nPR a nice description and eventually add comments to your code, if you want to point out certain design choices to your\nreviewer.\n\n### Share your work!!\n\nNow, it's time to get some credit from the community for your work! Having completed a model addition is a major\ncontribution to Transformers and the whole NLP community. Your code and the ported pre-trained models will certainly be\nused by hundreds and possibly even thousands of developers and researchers. You should be proud of your work and share\nyour achievements with the community.\n\n**You have made another model that is super easy to access for everyone in the community! \ud83e\udd2f**"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 73973398-ec38-4163-a9a5-2d2dce97e5b2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/pipeline_webserver", "title": "Using pipelines for a webserver", "tokens": 1467, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"73973398-ec38-4163-a9a5-2d2dce97e5b2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/pipeline_webserver\", \"title\": \"Using pipelines for a webserver\", \"tokens\": 1467, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"44a12a3f-a485-4ecc-afa8-96c15c2c0080\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/pipeline_webserver\", \"title\": \"Using pipelines for a webserver\", \"tokens\": 1467, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4833f52239d8ecf22c2fa5e74690dbc8e6cee4387648bcf70bbe622fd6edc308\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b022ed78-4866-4212-a949-22713d4b1e39\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"31874c9931bb2a115e2638d7c5b9a26e10bc5f1f8f2cd2db3824b9ae774bdac2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2361, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "44a12a3f-a485-4ecc-afa8-96c15c2c0080", "doc_id": "44a12a3f-a485-4ecc-afa8-96c15c2c0080", "ref_doc_id": "44a12a3f-a485-4ecc-afa8-96c15c2c0080", "chroma:document": "<!--\u26a0\ufe0f Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be\nrendered properly in your Markdown viewer.\n-->\n\n# Using pipelines for a webserver\n\n<Tip>\nCreating an inference engine is a complex topic, and the \"best\" solution \nwill most likely depend on your problem space. Are you on CPU or GPU? Do\nyou want the lowest latency, the highest throughput, support for\nmany models, or just highly optimize 1 specific model?\nThere are many ways to tackle this topic, so what we are going to present is a good default\nto get started which may not necessarily be the most optimal solution for you.\n</Tip>\n\n\nThe key thing to understand is that we can use an iterator, just like you would [on a\ndataset](pipeline_tutorial#using-pipelines-on-a-dataset), since a webserver is basically a system that waits for requests and\ntreats them as they come in.\n\nUsually webservers are multiplexed (multithreaded, async, etc..) to handle various\nrequests concurrently. Pipelines on the other hand (and mostly the underlying models)\nare not really great for parallelism; they take up a lot of RAM, so it's best to give them all the available resources when they are running or it's a compute-intensive job.\n\nWe are going to solve that by having the webserver handle the light load of receiving\nand sending requests, and having a single thread handling the actual work.\nThis example is going to use `starlette`. The actual framework is not really\nimportant, but you might have to tune or change the code if you are using another\none to achieve the same effect.\n\nCreate `server.py`:\n\n```py\nfrom starlette.applications import Starlette\nfrom starlette.responses import JSONResponse\nfrom starlette.routing import Route\nfrom transformers import pipeline\nimport asyncio\n\n\nasync def homepage(request):\n    payload = await request.body()\n    string = payload.decode(\"utf-8\")\n    response_q = asyncio.Queue()\n    await request.app.model_queue.put((string, response_q))\n    output = await response_q.get()\n    return JSONResponse(output)\n\n\nasync def server_loop(q):\n    pipe = pipeline(model=\"google-bert/bert-base-uncased\")\n    while True:\n        (string, response_q) = await q.get()\n        out = pipe(string)\n        await response_q.put(out)\n\n\napp = Starlette(\n    routes=[\n        Route(\"/\", homepage, methods=[\"POST\"]),\n    ],\n)"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: b022ed78-4866-4212-a949-22713d4b1e39
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/pipeline_webserver", "title": "Using pipelines for a webserver", "tokens": 1467, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b022ed78-4866-4212-a949-22713d4b1e39\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/pipeline_webserver\", \"title\": \"Using pipelines for a webserver\", \"tokens\": 1467, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"44a12a3f-a485-4ecc-afa8-96c15c2c0080\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/pipeline_webserver\", \"title\": \"Using pipelines for a webserver\", \"tokens\": 1467, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4833f52239d8ecf22c2fa5e74690dbc8e6cee4387648bcf70bbe622fd6edc308\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"73973398-ec38-4163-a9a5-2d2dce97e5b2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/pipeline_webserver\", \"title\": \"Using pipelines for a webserver\", \"tokens\": 1467, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7ed842c217e658042d2f6531b9e7e292bc7198a3ca29657bd8f6b80c84e42389\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f2499c1d-ba6f-4e35-82a0-ead4e003e950\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"86fe1179852aefe265940043d5a2d81a7ff6bf32971e1850fd31cc2fdaea0569\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 659, \"end_char_idx\": 4351, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "44a12a3f-a485-4ecc-afa8-96c15c2c0080", "doc_id": "44a12a3f-a485-4ecc-afa8-96c15c2c0080", "ref_doc_id": "44a12a3f-a485-4ecc-afa8-96c15c2c0080", "chroma:document": "The key thing to understand is that we can use an iterator, just like you would [on a\ndataset](pipeline_tutorial#using-pipelines-on-a-dataset), since a webserver is basically a system that waits for requests and\ntreats them as they come in.\n\nUsually webservers are multiplexed (multithreaded, async, etc..) to handle various\nrequests concurrently. Pipelines on the other hand (and mostly the underlying models)\nare not really great for parallelism; they take up a lot of RAM, so it's best to give them all the available resources when they are running or it's a compute-intensive job.\n\nWe are going to solve that by having the webserver handle the light load of receiving\nand sending requests, and having a single thread handling the actual work.\nThis example is going to use `starlette`. The actual framework is not really\nimportant, but you might have to tune or change the code if you are using another\none to achieve the same effect.\n\nCreate `server.py`:\n\n```py\nfrom starlette.applications import Starlette\nfrom starlette.responses import JSONResponse\nfrom starlette.routing import Route\nfrom transformers import pipeline\nimport asyncio\n\n\nasync def homepage(request):\n    payload = await request.body()\n    string = payload.decode(\"utf-8\")\n    response_q = asyncio.Queue()\n    await request.app.model_queue.put((string, response_q))\n    output = await response_q.get()\n    return JSONResponse(output)\n\n\nasync def server_loop(q):\n    pipe = pipeline(model=\"google-bert/bert-base-uncased\")\n    while True:\n        (string, response_q) = await q.get()\n        out = pipe(string)\n        await response_q.put(out)\n\n\napp = Starlette(\n    routes=[\n        Route(\"/\", homepage, methods=[\"POST\"]),\n    ],\n)\n\n\n@app.on_event(\"startup\")\nasync def startup_event():\n    q = asyncio.Queue()\n    app.model_queue = q\n    asyncio.create_task(server_loop(q))\n```\n\nNow you can start it with:\n```bash\nuvicorn server:app\n```\n\nAnd you can query it:\n```bash\ncurl -X POST -d \"test [MASK]\" http://localhost:8000/\n#[{\"score\":0.7742936015129089,\"token\":1012,\"token_str\":\".\",\"sequence\":\"test.\"},...]\n```\n\nAnd there you go, now you have a good idea of how to create a webserver!\n\nWhat is really important is that we load the model only **once**, so there are no copies\nof the model on the webserver. This way, no unnecessary RAM is being used.\nThen the queuing mechanism allows you to do fancy stuff like maybe accumulating a few\nitems before inferring to use dynamic batching:\n\n<Tip warning={true}>\n\nThe code sample below is intentionally written like pseudo-code for readability.\nDo not run this without checking if it makes sense for your system resources!\n\n</Tip>\n\n```py\n(string, rq) = await q.get()\nstrings = []\nqueues = []\nwhile True:\n    try:\n        (string, rq) = await asyncio.wait_for(q.get(), timeout=0.001)  # 1ms\n    except asyncio.exceptions.TimeoutError:\n        break\n    strings.append(string)\n    queues.append(rq)\nstrings\nouts = pipe(strings, batch_size=len(strings))\nfor rq, out in zip(queues, outs):\n    await rq.put(out)\n```\n\nAgain, the proposed code is optimized for readability, not for being the best code.\nFirst of all, there's no batch size limit which is usually not a \ngreat idea. Next, the timeout is reset on every queue fetch, meaning you could\nwait much more than 1ms before running the inference (delaying the first request \nby that much). \n\nIt would be better to have a single 1ms deadline.\n\nThis will always wait for 1ms even if the queue is empty, which might not be the\nbest since you probably want to start doing inference if there's nothing in the queue.\nBut maybe it does make sense if batching is really crucial for your use case.\nAgain, there's really no one best solution."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: f2499c1d-ba6f-4e35-82a0-ead4e003e950
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/pipeline_webserver", "title": "Using pipelines for a webserver", "tokens": 1467, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f2499c1d-ba6f-4e35-82a0-ead4e003e950\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/pipeline_webserver\", \"title\": \"Using pipelines for a webserver\", \"tokens\": 1467, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"44a12a3f-a485-4ecc-afa8-96c15c2c0080\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/pipeline_webserver\", \"title\": \"Using pipelines for a webserver\", \"tokens\": 1467, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4833f52239d8ecf22c2fa5e74690dbc8e6cee4387648bcf70bbe622fd6edc308\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b022ed78-4866-4212-a949-22713d4b1e39\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/pipeline_webserver\", \"title\": \"Using pipelines for a webserver\", \"tokens\": 1467, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ccbe9e805dc59f5a009ae137bb6daed2dd5679278091b5a7e55afad0766cc661\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4354, \"end_char_idx\": 6456, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "44a12a3f-a485-4ecc-afa8-96c15c2c0080", "doc_id": "44a12a3f-a485-4ecc-afa8-96c15c2c0080", "ref_doc_id": "44a12a3f-a485-4ecc-afa8-96c15c2c0080", "chroma:document": "## Few things you might want to consider\n\n### Error checking\n\nThere's a lot that can go wrong in production: out of memory, out of space,\nloading the model might fail, the query might be wrong, the query might be\ncorrect but still fail to run because of a model misconfiguration, and so on.\n\nGenerally, it's good if the server outputs the errors to the user, so\nadding a lot of `try..except` statements to show those errors is a good\nidea. But keep in mind it may also be a security risk to reveal all those errors depending \non your security context.\n\n### Circuit breaking\n\nWebservers usually look better when they do circuit breaking. It means they \nreturn proper errors when they're overloaded instead of just waiting for the query indefinitely. Return a 503 error instead of waiting for a super long time or a 504 after a long time.\n\nThis is relatively easy to implement in the proposed code since there is a single queue.\nLooking at the queue size is a basic way to start returning errors before your \nwebserver fails under load.\n\n### Blocking the main thread\n\nCurrently PyTorch is not async aware, and computation will block the main\nthread while running. That means it would be better if PyTorch was forced to run\non its own thread/process. This wasn't done here because the code is a lot more\ncomplex (mostly because threads and async and queues don't play nice together).\nBut ultimately it does the same thing.\n\nThis would be important if the inference of single items were long (> 1s) because \nin this case, it means every query during inference would have to wait for 1s before\neven receiving an error.\n\n### Dynamic batching\n\nIn general, batching is not necessarily an improvement over passing 1 item at \na time (see [batching details](./main_classes/pipelines#pipeline-batching) for more information). But it can be very effective\nwhen used in the correct setting. In the API, there is no dynamic\nbatching by default (too much opportunity for a slowdown). But for BLOOM inference -\nwhich is a very large model - dynamic batching is **essential** to provide a decent experience for everyone."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 31d92740-761e-4ac1-85d8-b0160d8a0627
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"31d92740-761e-4ac1-85d8-b0160d8a0627\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tokenizer_summary\", \"title\": \"Summary of the tokenizers\", \"tokens\": 4431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb1e6a04-2c55-43c0-8265-84fe092a5176\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tokenizer_summary\", \"title\": \"Summary of the tokenizers\", \"tokens\": 4431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"42019efa99141d419d86ef0bac547c2ebd2bec88434d903f5446fd01d4e002d2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"30c15642-c071-4ffc-a6c0-e6f93f3b1c97\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"db3529fe6abeda9c5d241bad9e2a829cc3ea91c3fdedb579edc6c756ecd10fb9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2987, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb1e6a04-2c55-43c0-8265-84fe092a5176", "doc_id": "cb1e6a04-2c55-43c0-8265-84fe092a5176", "ref_doc_id": "cb1e6a04-2c55-43c0-8265-84fe092a5176", "chroma:document": "# Summary of the tokenizers\n\n[[open-in-colab]]\n\nOn this page, we will have a closer look at tokenization.\n\n<Youtube id=\"VFp38yj8h3A\"/>\n\nAs we saw in [the preprocessing tutorial](preprocessing), tokenizing a text is splitting it into words or\nsubwords, which then are converted to ids through a look-up table. Converting words or subwords to ids is\nstraightforward, so in this summary, we will focus on splitting a text into words or subwords (i.e. tokenizing a text).\nMore specifically, we will look at the three main types of tokenizers used in \ud83e\udd17 Transformers: [Byte-Pair Encoding\n(BPE)](#byte-pair-encoding), [WordPiece](#wordpiece), and [SentencePiece](#sentencepiece), and show examples\nof which tokenizer type is used by which model.\n\nNote that on each model page, you can look at the documentation of the associated tokenizer to know which tokenizer\ntype was used by the pretrained model. For instance, if we look at [`BertTokenizer`], we can see\nthat the model uses [WordPiece](#wordpiece).\n\n## Introduction\n\nSplitting a text into smaller chunks is a task that is harder than it looks, and there are multiple ways of doing so.\nFor instance, let's look at the sentence `\"Don't you love \ud83e\udd17 Transformers? We sure do.\"`\n\n<Youtube id=\"nhJxYji1aho\"/>\n\nA simple way of tokenizing this text is to split it by spaces, which would give:\n\n```\n[\"Don't\", \"you\", \"love\", \"\ud83e\udd17\", \"Transformers?\", \"We\", \"sure\", \"do.\"]\n```\n\nThis is a sensible first step, but if we look at the tokens `\"Transformers?\"` and `\"do.\"`, we notice that the\npunctuation is attached to the words `\"Transformer\"` and `\"do\"`, which is suboptimal. We should take the\npunctuation into account so that a model does not have to learn a different representation of a word and every possible\npunctuation symbol that could follow it, which would explode the number of representations the model has to learn.\nTaking punctuation into account, tokenizing our exemplary text would give:\n\n```\n[\"Don\", \"'\", \"t\", \"you\", \"love\", \"\ud83e\udd17\", \"Transformers\", \"?\", \"We\", \"sure\", \"do\", \".\"]\n```\n\nBetter. However, it is disadvantageous, how the tokenization dealt with the word `\"Don't\"`. `\"Don't\"` stands for\n`\"do not\"`, so it would be better tokenized as `[\"Do\", \"n't\"]`. This is where things start getting complicated, and\npart of the reason each model has its own tokenizer type. Depending on the rules we apply for tokenizing a text, a\ndifferent tokenized output is generated for the same text. A pretrained model only performs properly if you feed it an\ninput that was tokenized with the same rules that were used to tokenize its training data.\n\n[spaCy](https://spacy.io/) and [Moses](http://www.statmt.org/moses/?n=Development.GetStarted) are two popular\nrule-based tokenizers. Applying them on our example, *spaCy* and *Moses* would output something like:\n\n```\n[\"Do\", \"n't\", \"you\", \"love\", \"\ud83e\udd17\", \"Transformers\", \"?\", \"We\", \"sure\", \"do\", \".\"]\n```\n\nAs can be seen space and punctuation tokenization, as well as rule-based tokenization, is used here."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 30c15642-c071-4ffc-a6c0-e6f93f3b1c97
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"30c15642-c071-4ffc-a6c0-e6f93f3b1c97\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tokenizer_summary\", \"title\": \"Summary of the tokenizers\", \"tokens\": 4431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb1e6a04-2c55-43c0-8265-84fe092a5176\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tokenizer_summary\", \"title\": \"Summary of the tokenizers\", \"tokens\": 4431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"42019efa99141d419d86ef0bac547c2ebd2bec88434d903f5446fd01d4e002d2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"31d92740-761e-4ac1-85d8-b0160d8a0627\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tokenizer_summary\", \"title\": \"Summary of the tokenizers\", \"tokens\": 4431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b4107365c574a29fd3ecf651cd1c9ea038b3132b0411c805270aef3c8a552123\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4b021940-d938-480a-a4fb-a0d8fb6a9429\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"12952e2eeb2e588dcf770a55cc4742f6cb495c4da0b3005a36eed67e9427fe04\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1486, \"end_char_idx\": 4839, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb1e6a04-2c55-43c0-8265-84fe092a5176", "doc_id": "cb1e6a04-2c55-43c0-8265-84fe092a5176", "ref_doc_id": "cb1e6a04-2c55-43c0-8265-84fe092a5176", "chroma:document": "\"` and `\"do.\"`, we notice that the\npunctuation is attached to the words `\"Transformer\"` and `\"do\"`, which is suboptimal. We should take the\npunctuation into account so that a model does not have to learn a different representation of a word and every possible\npunctuation symbol that could follow it, which would explode the number of representations the model has to learn.\nTaking punctuation into account, tokenizing our exemplary text would give:\n\n```\n[\"Don\", \"'\", \"t\", \"you\", \"love\", \"\ud83e\udd17\", \"Transformers\", \"?\", \"We\", \"sure\", \"do\", \".\"]\n```\n\nBetter. However, it is disadvantageous, how the tokenization dealt with the word `\"Don't\"`. `\"Don't\"` stands for\n`\"do not\"`, so it would be better tokenized as `[\"Do\", \"n't\"]`. This is where things start getting complicated, and\npart of the reason each model has its own tokenizer type. Depending on the rules we apply for tokenizing a text, a\ndifferent tokenized output is generated for the same text. A pretrained model only performs properly if you feed it an\ninput that was tokenized with the same rules that were used to tokenize its training data.\n\n[spaCy](https://spacy.io/) and [Moses](http://www.statmt.org/moses/?n=Development.GetStarted) are two popular\nrule-based tokenizers. Applying them on our example, *spaCy* and *Moses* would output something like:\n\n```\n[\"Do\", \"n't\", \"you\", \"love\", \"\ud83e\udd17\", \"Transformers\", \"?\", \"We\", \"sure\", \"do\", \".\"]\n```\n\nAs can be seen space and punctuation tokenization, as well as rule-based tokenization, is used here. Space and\npunctuation tokenization and rule-based tokenization are both examples of word tokenization, which is loosely defined\nas splitting sentences into words. While it's the most intuitive way to split texts into smaller chunks, this\ntokenization method can lead to problems for massive text corpora. In this case, space and punctuation tokenization\nusually generates a very big vocabulary (the set of all unique words and tokens used). *E.g.*, [Transformer XL](model_doc/transfo-xl) uses space and punctuation tokenization, resulting in a vocabulary size of 267,735!\n\nSuch a big vocabulary size forces the model to have an enormous embedding matrix as the input and output layer, which\ncauses both an increased memory and time complexity. In general, transformers models rarely have a vocabulary size\ngreater than 50,000, especially if they are pretrained only on a single language.\n\nSo if simple space and punctuation tokenization is unsatisfactory, why not simply tokenize on characters?\n\n<Youtube id=\"ssLq_EK2jLE\"/>\n\nWhile character tokenization is very simple and would greatly reduce memory and time complexity it makes it much harder\nfor the model to learn meaningful input representations. *E.g.* learning a meaningful context-independent\nrepresentation for the letter `\"t\"` is much harder than learning a context-independent representation for the word\n`\"today\"`. Therefore, character tokenization is often accompanied by a loss of performance. So to get the best of\nboth worlds, transformers models use a hybrid between word-level and character-level tokenization called **subword**\ntokenization.\n\n## Subword tokenization\n\n<Youtube id=\"zHvTiHr506c\"/>\n\nSubword tokenization algorithms rely on the principle that frequently used words should not be split into smaller\nsubwords, but rare words should be decomposed into meaningful subwords."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 4b021940-d938-480a-a4fb-a0d8fb6a9429
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4b021940-d938-480a-a4fb-a0d8fb6a9429\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tokenizer_summary\", \"title\": \"Summary of the tokenizers\", \"tokens\": 4431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb1e6a04-2c55-43c0-8265-84fe092a5176\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tokenizer_summary\", \"title\": \"Summary of the tokenizers\", \"tokens\": 4431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"42019efa99141d419d86ef0bac547c2ebd2bec88434d903f5446fd01d4e002d2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"30c15642-c071-4ffc-a6c0-e6f93f3b1c97\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tokenizer_summary\", \"title\": \"Summary of the tokenizers\", \"tokens\": 4431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4d8384c600f509ba2d625704447b36a31adb723c5a02af4cc0d5c4f3f221dca6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5351c2c4-216b-4157-b28f-62c4d7e85034\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6cd62ebc4905a3ea204968a163181fe8fd52ef3f38fb71d2404c14e9a275a605\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2988, \"end_char_idx\": 6346, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb1e6a04-2c55-43c0-8265-84fe092a5176", "doc_id": "cb1e6a04-2c55-43c0-8265-84fe092a5176", "ref_doc_id": "cb1e6a04-2c55-43c0-8265-84fe092a5176", "chroma:document": "Space and\npunctuation tokenization and rule-based tokenization are both examples of word tokenization, which is loosely defined\nas splitting sentences into words. While it's the most intuitive way to split texts into smaller chunks, this\ntokenization method can lead to problems for massive text corpora. In this case, space and punctuation tokenization\nusually generates a very big vocabulary (the set of all unique words and tokens used). *E.g.*, [Transformer XL](model_doc/transfo-xl) uses space and punctuation tokenization, resulting in a vocabulary size of 267,735!\n\nSuch a big vocabulary size forces the model to have an enormous embedding matrix as the input and output layer, which\ncauses both an increased memory and time complexity. In general, transformers models rarely have a vocabulary size\ngreater than 50,000, especially if they are pretrained only on a single language.\n\nSo if simple space and punctuation tokenization is unsatisfactory, why not simply tokenize on characters?\n\n<Youtube id=\"ssLq_EK2jLE\"/>\n\nWhile character tokenization is very simple and would greatly reduce memory and time complexity it makes it much harder\nfor the model to learn meaningful input representations. *E.g.* learning a meaningful context-independent\nrepresentation for the letter `\"t\"` is much harder than learning a context-independent representation for the word\n`\"today\"`. Therefore, character tokenization is often accompanied by a loss of performance. So to get the best of\nboth worlds, transformers models use a hybrid between word-level and character-level tokenization called **subword**\ntokenization.\n\n## Subword tokenization\n\n<Youtube id=\"zHvTiHr506c\"/>\n\nSubword tokenization algorithms rely on the principle that frequently used words should not be split into smaller\nsubwords, but rare words should be decomposed into meaningful subwords. For instance `\"annoyingly\"` might be\nconsidered a rare word and could be decomposed into `\"annoying\"` and `\"ly\"`. Both `\"annoying\"` and `\"ly\"` as\nstand-alone subwords would appear more frequently while at the same time the meaning of `\"annoyingly\"` is kept by the\ncomposite meaning of `\"annoying\"` and `\"ly\"`. This is especially useful in agglutinative languages such as Turkish,\nwhere you can form (almost) arbitrarily long complex words by stringing together subwords.\n\nSubword tokenization allows the model to have a reasonable vocabulary size while being able to learn meaningful\ncontext-independent representations. In addition, subword tokenization enables the model to process words it has never\nseen before, by decomposing them into known subwords. For instance, the [`~transformers.BertTokenizer`] tokenizes\n`\"I have a new GPU!\"` as follows:\n\n```py\n>>> from transformers import BertTokenizer\n\n>>> tokenizer = BertTokenizer.from_pretrained(\"google-bert/bert-base-uncased\")\n>>> tokenizer.tokenize(\"I have a new GPU!\")\n[\"i\", \"have\", \"a\", \"new\", \"gp\", \"##u\", \"!\"]\n```\n\nBecause we are considering the uncased model, the sentence was lowercased first. We can see that the words `[\"i\", \"have\", \"a\", \"new\"]` are present in the tokenizer's vocabulary, but the word `\"gpu\"` is not. Consequently, the\ntokenizer splits `\"gpu\"` into known subwords: `[\"gp\" and \"##u\"]`. `\"##\"` means that the rest of the token should\nbe attached to the previous one, without space (for decoding or reversal of the tokenization)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 5351c2c4-216b-4157-b28f-62c4d7e85034
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5351c2c4-216b-4157-b28f-62c4d7e85034\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tokenizer_summary\", \"title\": \"Summary of the tokenizers\", \"tokens\": 4431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb1e6a04-2c55-43c0-8265-84fe092a5176\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tokenizer_summary\", \"title\": \"Summary of the tokenizers\", \"tokens\": 4431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"42019efa99141d419d86ef0bac547c2ebd2bec88434d903f5446fd01d4e002d2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4b021940-d938-480a-a4fb-a0d8fb6a9429\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tokenizer_summary\", \"title\": \"Summary of the tokenizers\", \"tokens\": 4431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e45a10d24608efaec5d2c9638fceaa8eba9cf9f815d22dc42c3d699af61707b2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b64277e1-174a-4c6f-b78a-7c8f7ef08e7c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c6c5e8c3eea22dd0b14b73f8fa92e985fd47836293203981f8d77ceb9d6a0938\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4840, \"end_char_idx\": 7687, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb1e6a04-2c55-43c0-8265-84fe092a5176", "doc_id": "cb1e6a04-2c55-43c0-8265-84fe092a5176", "ref_doc_id": "cb1e6a04-2c55-43c0-8265-84fe092a5176", "chroma:document": "For instance `\"annoyingly\"` might be\nconsidered a rare word and could be decomposed into `\"annoying\"` and `\"ly\"`. Both `\"annoying\"` and `\"ly\"` as\nstand-alone subwords would appear more frequently while at the same time the meaning of `\"annoyingly\"` is kept by the\ncomposite meaning of `\"annoying\"` and `\"ly\"`. This is especially useful in agglutinative languages such as Turkish,\nwhere you can form (almost) arbitrarily long complex words by stringing together subwords.\n\nSubword tokenization allows the model to have a reasonable vocabulary size while being able to learn meaningful\ncontext-independent representations. In addition, subword tokenization enables the model to process words it has never\nseen before, by decomposing them into known subwords. For instance, the [`~transformers.BertTokenizer`] tokenizes\n`\"I have a new GPU!\"` as follows:\n\n```py\n>>> from transformers import BertTokenizer\n\n>>> tokenizer = BertTokenizer.from_pretrained(\"google-bert/bert-base-uncased\")\n>>> tokenizer.tokenize(\"I have a new GPU!\")\n[\"i\", \"have\", \"a\", \"new\", \"gp\", \"##u\", \"!\"]\n```\n\nBecause we are considering the uncased model, the sentence was lowercased first. We can see that the words `[\"i\", \"have\", \"a\", \"new\"]` are present in the tokenizer's vocabulary, but the word `\"gpu\"` is not. Consequently, the\ntokenizer splits `\"gpu\"` into known subwords: `[\"gp\" and \"##u\"]`. `\"##\"` means that the rest of the token should\nbe attached to the previous one, without space (for decoding or reversal of the tokenization).\n\nAs another example, [`~transformers.XLNetTokenizer`] tokenizes our previously exemplary text as follows:\n\n```py\n>>> from transformers import XLNetTokenizer\n\n>>> tokenizer = XLNetTokenizer.from_pretrained(\"xlnet/xlnet-base-cased\")\n>>> tokenizer.tokenize(\"Don't you love \ud83e\udd17 Transformers? We sure do.\")\n[\"\u2581Don\", \"'\", \"t\", \"\u2581you\", \"\u2581love\", \"\u2581\", \"\ud83e\udd17\", \"\u2581\", \"Transform\", \"ers\", \"?\", \"\u2581We\", \"\u2581sure\", \"\u2581do\", \".\"]\n```\n\nWe'll get back to the meaning of those `\"\u2581\"` when we look at [SentencePiece](#sentencepiece). As one can see,\nthe rare word `\"Transformers\"` has been split into the more frequent subwords `\"Transform\"` and `\"ers\"`.\n\nLet's now look at how the different subword tokenization algorithms work. Note that all of those tokenization\nalgorithms rely on some form of training which is usually done on the corpus the corresponding model will be trained\non.\n\n<a id='byte-pair-encoding'></a>\n\n### Byte-Pair Encoding (BPE)\n\nByte-Pair Encoding (BPE) was introduced in [Neural Machine Translation of Rare Words with Subword Units (Sennrich et\nal., 2015)](https://arxiv.org/abs/1508.07909). BPE relies on a pre-tokenizer that splits the training data into\nwords. Pretokenization can be as simple as space tokenization, e.g. [GPT-2](model_doc/gpt2), [RoBERTa](model_doc/roberta). More advanced pre-tokenization include rule-based tokenization, e.g."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: b64277e1-174a-4c6f-b78a-7c8f7ef08e7c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b64277e1-174a-4c6f-b78a-7c8f7ef08e7c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tokenizer_summary\", \"title\": \"Summary of the tokenizers\", \"tokens\": 4431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb1e6a04-2c55-43c0-8265-84fe092a5176\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tokenizer_summary\", \"title\": \"Summary of the tokenizers\", \"tokens\": 4431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"42019efa99141d419d86ef0bac547c2ebd2bec88434d903f5446fd01d4e002d2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5351c2c4-216b-4157-b28f-62c4d7e85034\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tokenizer_summary\", \"title\": \"Summary of the tokenizers\", \"tokens\": 4431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1e831cb8f1e51f6ed45baf89755398201ec0d03f7739c2c6ac3ab61da2baefc1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a1383ae9-b084-4dad-ad10-4a6ef4a484ee\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b463e05c505295089811a339bbdd796d62a84bc1986c11f4ea6b017a8e413dde\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6348, \"end_char_idx\": 8974, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb1e6a04-2c55-43c0-8265-84fe092a5176", "doc_id": "cb1e6a04-2c55-43c0-8265-84fe092a5176", "ref_doc_id": "cb1e6a04-2c55-43c0-8265-84fe092a5176", "chroma:document": "As another example, [`~transformers.XLNetTokenizer`] tokenizes our previously exemplary text as follows:\n\n```py\n>>> from transformers import XLNetTokenizer\n\n>>> tokenizer = XLNetTokenizer.from_pretrained(\"xlnet/xlnet-base-cased\")\n>>> tokenizer.tokenize(\"Don't you love \ud83e\udd17 Transformers? We sure do.\")\n[\"\u2581Don\", \"'\", \"t\", \"\u2581you\", \"\u2581love\", \"\u2581\", \"\ud83e\udd17\", \"\u2581\", \"Transform\", \"ers\", \"?\", \"\u2581We\", \"\u2581sure\", \"\u2581do\", \".\"]\n```\n\nWe'll get back to the meaning of those `\"\u2581\"` when we look at [SentencePiece](#sentencepiece). As one can see,\nthe rare word `\"Transformers\"` has been split into the more frequent subwords `\"Transform\"` and `\"ers\"`.\n\nLet's now look at how the different subword tokenization algorithms work. Note that all of those tokenization\nalgorithms rely on some form of training which is usually done on the corpus the corresponding model will be trained\non.\n\n<a id='byte-pair-encoding'></a>\n\n### Byte-Pair Encoding (BPE)\n\nByte-Pair Encoding (BPE) was introduced in [Neural Machine Translation of Rare Words with Subword Units (Sennrich et\nal., 2015)](https://arxiv.org/abs/1508.07909). BPE relies on a pre-tokenizer that splits the training data into\nwords. Pretokenization can be as simple as space tokenization, e.g. [GPT-2](model_doc/gpt2), [RoBERTa](model_doc/roberta). More advanced pre-tokenization include rule-based tokenization, e.g. [XLM](model_doc/xlm),\n[FlauBERT](model_doc/flaubert) which uses Moses for most languages, or [GPT](model_doc/openai-gpt) which uses\nspaCy and ftfy, to count the frequency of each word in the training corpus.\n\nAfter pre-tokenization, a set of unique words has been created and the frequency with which each word occurred in the\ntraining data has been determined. Next, BPE creates a base vocabulary consisting of all symbols that occur in the set\nof unique words and learns merge rules to form a new symbol from two symbols of the base vocabulary. It does so until\nthe vocabulary has attained the desired vocabulary size. Note that the desired vocabulary size is a hyperparameter to\ndefine before training the tokenizer.\n\nAs an example, let's assume that after pre-tokenization, the following set of words including their frequency has been\ndetermined:\n\n```\n(\"hug\", 10), (\"pug\", 5), (\"pun\", 12), (\"bun\", 4), (\"hugs\", 5)\n```\n\nConsequently, the base vocabulary is `[\"b\", \"g\", \"h\", \"n\", \"p\", \"s\", \"u\"]`. Splitting all words into symbols of the\nbase vocabulary, we obtain:\n\n```\n(\"h\" \"u\" \"g\", 10), (\"p\" \"u\" \"g\", 5), (\"p\" \"u\" \"n\", 12), (\"b\" \"u\" \"n\", 4), (\"h\" \"u\" \"g\" \"s\", 5)\n```\n\nBPE then counts the frequency of each possible symbol pair and picks the symbol pair that occurs most frequently."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: a1383ae9-b084-4dad-ad10-4a6ef4a484ee
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a1383ae9-b084-4dad-ad10-4a6ef4a484ee\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tokenizer_summary\", \"title\": \"Summary of the tokenizers\", \"tokens\": 4431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb1e6a04-2c55-43c0-8265-84fe092a5176\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tokenizer_summary\", \"title\": \"Summary of the tokenizers\", \"tokens\": 4431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"42019efa99141d419d86ef0bac547c2ebd2bec88434d903f5446fd01d4e002d2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b64277e1-174a-4c6f-b78a-7c8f7ef08e7c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tokenizer_summary\", \"title\": \"Summary of the tokenizers\", \"tokens\": 4431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f52978d4a85a86594798709a74bd6d7e42892c06664611ff2c3051db5853e477\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"78d8bbf4-dfa4-4376-9e80-644df262c17e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a14ad6ae7ead94441aaf990b0210e8524805459183184d162385a0094aa1ff39\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7619, \"end_char_idx\": 9852, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb1e6a04-2c55-43c0-8265-84fe092a5176", "doc_id": "cb1e6a04-2c55-43c0-8265-84fe092a5176", "ref_doc_id": "cb1e6a04-2c55-43c0-8265-84fe092a5176", "chroma:document": "More advanced pre-tokenization include rule-based tokenization, e.g. [XLM](model_doc/xlm),\n[FlauBERT](model_doc/flaubert) which uses Moses for most languages, or [GPT](model_doc/openai-gpt) which uses\nspaCy and ftfy, to count the frequency of each word in the training corpus.\n\nAfter pre-tokenization, a set of unique words has been created and the frequency with which each word occurred in the\ntraining data has been determined. Next, BPE creates a base vocabulary consisting of all symbols that occur in the set\nof unique words and learns merge rules to form a new symbol from two symbols of the base vocabulary. It does so until\nthe vocabulary has attained the desired vocabulary size. Note that the desired vocabulary size is a hyperparameter to\ndefine before training the tokenizer.\n\nAs an example, let's assume that after pre-tokenization, the following set of words including their frequency has been\ndetermined:\n\n```\n(\"hug\", 10), (\"pug\", 5), (\"pun\", 12), (\"bun\", 4), (\"hugs\", 5)\n```\n\nConsequently, the base vocabulary is `[\"b\", \"g\", \"h\", \"n\", \"p\", \"s\", \"u\"]`. Splitting all words into symbols of the\nbase vocabulary, we obtain:\n\n```\n(\"h\" \"u\" \"g\", 10), (\"p\" \"u\" \"g\", 5), (\"p\" \"u\" \"n\", 12), (\"b\" \"u\" \"n\", 4), (\"h\" \"u\" \"g\" \"s\", 5)\n```\n\nBPE then counts the frequency of each possible symbol pair and picks the symbol pair that occurs most frequently. In\nthe example above `\"h\"` followed by `\"u\"` is present _10 + 5 = 15_ times (10 times in the 10 occurrences of\n`\"hug\"`, 5 times in the 5 occurrences of `\"hugs\"`). However, the most frequent symbol pair is `\"u\"` followed by\n`\"g\"`, occurring _10 + 5 + 5 = 20_ times in total. Thus, the first merge rule the tokenizer learns is to group all\n`\"u\"` symbols followed by a `\"g\"` symbol together. Next, `\"ug\"` is added to the vocabulary. The set of words then\nbecomes\n\n```\n(\"h\" \"ug\", 10), (\"p\" \"ug\", 5), (\"p\" \"u\" \"n\", 12), (\"b\" \"u\" \"n\", 4), (\"h\" \"ug\" \"s\", 5)\n```\n\nBPE then identifies the next most common symbol pair. It's `\"u\"` followed by `\"n\"`, which occurs 16 times. `\"u\"`,\n`\"n\"` is merged to `\"un\"` and added to the vocabulary. The next most frequent symbol pair is `\"h\"` followed by\n`\"ug\"`, occurring 15 times. Again the pair is merged and `\"hug\"` can be added to the vocabulary."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 78d8bbf4-dfa4-4376-9e80-644df262c17e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"78d8bbf4-dfa4-4376-9e80-644df262c17e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tokenizer_summary\", \"title\": \"Summary of the tokenizers\", \"tokens\": 4431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb1e6a04-2c55-43c0-8265-84fe092a5176\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tokenizer_summary\", \"title\": \"Summary of the tokenizers\", \"tokens\": 4431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"42019efa99141d419d86ef0bac547c2ebd2bec88434d903f5446fd01d4e002d2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a1383ae9-b084-4dad-ad10-4a6ef4a484ee\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tokenizer_summary\", \"title\": \"Summary of the tokenizers\", \"tokens\": 4431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"adad7afc05d3efd318b649694e6daca16233e4fb0527b69072ef7ffaa5ef3b53\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"86958e44-ffea-4ea7-a86e-8fc3a1ba9bf6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f03416eb1ee2ab84639acbb41e711ba0a065200d3dcd0f889f0354857f0fec62\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8688, \"end_char_idx\": 11159, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb1e6a04-2c55-43c0-8265-84fe092a5176", "doc_id": "cb1e6a04-2c55-43c0-8265-84fe092a5176", "ref_doc_id": "cb1e6a04-2c55-43c0-8265-84fe092a5176", "chroma:document": "Splitting all words into symbols of the\nbase vocabulary, we obtain:\n\n```\n(\"h\" \"u\" \"g\", 10), (\"p\" \"u\" \"g\", 5), (\"p\" \"u\" \"n\", 12), (\"b\" \"u\" \"n\", 4), (\"h\" \"u\" \"g\" \"s\", 5)\n```\n\nBPE then counts the frequency of each possible symbol pair and picks the symbol pair that occurs most frequently. In\nthe example above `\"h\"` followed by `\"u\"` is present _10 + 5 = 15_ times (10 times in the 10 occurrences of\n`\"hug\"`, 5 times in the 5 occurrences of `\"hugs\"`). However, the most frequent symbol pair is `\"u\"` followed by\n`\"g\"`, occurring _10 + 5 + 5 = 20_ times in total. Thus, the first merge rule the tokenizer learns is to group all\n`\"u\"` symbols followed by a `\"g\"` symbol together. Next, `\"ug\"` is added to the vocabulary. The set of words then\nbecomes\n\n```\n(\"h\" \"ug\", 10), (\"p\" \"ug\", 5), (\"p\" \"u\" \"n\", 12), (\"b\" \"u\" \"n\", 4), (\"h\" \"ug\" \"s\", 5)\n```\n\nBPE then identifies the next most common symbol pair. It's `\"u\"` followed by `\"n\"`, which occurs 16 times. `\"u\"`,\n`\"n\"` is merged to `\"un\"` and added to the vocabulary. The next most frequent symbol pair is `\"h\"` followed by\n`\"ug\"`, occurring 15 times. Again the pair is merged and `\"hug\"` can be added to the vocabulary.\n\nAt this stage, the vocabulary is `[\"b\", \"g\", \"h\", \"n\", \"p\", \"s\", \"u\", \"ug\", \"un\", \"hug\"]` and our set of unique words\nis represented as\n\n```\n(\"hug\", 10), (\"p\" \"ug\", 5), (\"p\" \"un\", 12), (\"b\" \"un\", 4), (\"hug\" \"s\", 5)\n```\n\nAssuming, that the Byte-Pair Encoding training would stop at this point, the learned merge rules would then be applied\nto new words (as long as those new words do not include symbols that were not in the base vocabulary). For instance,\nthe word `\"bug\"` would be tokenized to `[\"b\", \"ug\"]` but `\"mug\"` would be tokenized as `[\"<unk>\", \"ug\"]` since\nthe symbol `\"m\"` is not in the base vocabulary. In general, single letters such as `\"m\"` are not replaced by the\n`\"<unk>\"` symbol because the training data usually includes at least one occurrence of each letter, but it is likely\nto happen for very special characters like emojis.\n\nAs mentioned earlier, the vocabulary size, *i.e.* the base vocabulary size + the number of merges, is a hyperparameter\nto choose. For instance [GPT](model_doc/openai-gpt) has a vocabulary size of 40,478 since they have 478 base characters\nand chose to stop training after 40,000 merges.\n\n#### Byte-level BPE\n\nA base vocabulary that includes all possible base characters can be quite large if *e.g.* all unicode characters are\nconsidered as base characters."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 86958e44-ffea-4ea7-a86e-8fc3a1ba9bf6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"86958e44-ffea-4ea7-a86e-8fc3a1ba9bf6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tokenizer_summary\", \"title\": \"Summary of the tokenizers\", \"tokens\": 4431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb1e6a04-2c55-43c0-8265-84fe092a5176\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tokenizer_summary\", \"title\": \"Summary of the tokenizers\", \"tokens\": 4431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"42019efa99141d419d86ef0bac547c2ebd2bec88434d903f5446fd01d4e002d2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"78d8bbf4-dfa4-4376-9e80-644df262c17e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tokenizer_summary\", \"title\": \"Summary of the tokenizers\", \"tokens\": 4431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d1f10c997b145f86030f7d8991e45f3b36bf10ecce4fe3471949928fc5ff6bfa\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cdb5403a-5230-4801-93e8-c394aaef241e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"096dc6670b28c3da7686e7d8d73d9c20b0fe65f87dc6a27c11a5a797cfc34441\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9784, \"end_char_idx\": 12819, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb1e6a04-2c55-43c0-8265-84fe092a5176", "doc_id": "cb1e6a04-2c55-43c0-8265-84fe092a5176", "ref_doc_id": "cb1e6a04-2c55-43c0-8265-84fe092a5176", "chroma:document": "Again the pair is merged and `\"hug\"` can be added to the vocabulary.\n\nAt this stage, the vocabulary is `[\"b\", \"g\", \"h\", \"n\", \"p\", \"s\", \"u\", \"ug\", \"un\", \"hug\"]` and our set of unique words\nis represented as\n\n```\n(\"hug\", 10), (\"p\" \"ug\", 5), (\"p\" \"un\", 12), (\"b\" \"un\", 4), (\"hug\" \"s\", 5)\n```\n\nAssuming, that the Byte-Pair Encoding training would stop at this point, the learned merge rules would then be applied\nto new words (as long as those new words do not include symbols that were not in the base vocabulary). For instance,\nthe word `\"bug\"` would be tokenized to `[\"b\", \"ug\"]` but `\"mug\"` would be tokenized as `[\"<unk>\", \"ug\"]` since\nthe symbol `\"m\"` is not in the base vocabulary. In general, single letters such as `\"m\"` are not replaced by the\n`\"<unk>\"` symbol because the training data usually includes at least one occurrence of each letter, but it is likely\nto happen for very special characters like emojis.\n\nAs mentioned earlier, the vocabulary size, *i.e.* the base vocabulary size + the number of merges, is a hyperparameter\nto choose. For instance [GPT](model_doc/openai-gpt) has a vocabulary size of 40,478 since they have 478 base characters\nand chose to stop training after 40,000 merges.\n\n#### Byte-level BPE\n\nA base vocabulary that includes all possible base characters can be quite large if *e.g.* all unicode characters are\nconsidered as base characters. To have a better base vocabulary, [GPT-2](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) uses bytes\nas the base vocabulary, which is a clever trick to force the base vocabulary to be of size 256 while ensuring that\nevery base character is included in the vocabulary. With some additional rules to deal with punctuation, the GPT2's\ntokenizer can tokenize every text without the need for the <unk> symbol. [GPT-2](model_doc/gpt) has a vocabulary\nsize of 50,257, which corresponds to the 256 bytes base tokens, a special end-of-text token and the symbols learned\nwith 50,000 merges.\n\n<a id='wordpiece'></a>\n\n### WordPiece\n\nWordPiece is the subword tokenization algorithm used for [BERT](model_doc/bert), [DistilBERT](model_doc/distilbert), and [Electra](model_doc/electra). The algorithm was outlined in [Japanese and Korean\nVoice Search (Schuster et al., 2012)](https://static.googleusercontent.com/media/research.google.com/ja//pubs/archive/37842.pdf) and is very similar to\nBPE. WordPiece first initializes the vocabulary to include every character present in the training data and\nprogressively learns a given number of merge rules. In contrast to BPE, WordPiece does not choose the most frequent\nsymbol pair, but the one that maximizes the likelihood of the training data once added to the vocabulary.\n\nSo what does this mean exactly? Referring to the previous example, maximizing the likelihood of the training data is\nequivalent to finding the symbol pair, whose probability divided by the probabilities of its first symbol followed by\nits second symbol is the greatest among all symbol pairs. *E.g."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: cdb5403a-5230-4801-93e8-c394aaef241e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"cdb5403a-5230-4801-93e8-c394aaef241e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tokenizer_summary\", \"title\": \"Summary of the tokenizers\", \"tokens\": 4431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb1e6a04-2c55-43c0-8265-84fe092a5176\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tokenizer_summary\", \"title\": \"Summary of the tokenizers\", \"tokens\": 4431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"42019efa99141d419d86ef0bac547c2ebd2bec88434d903f5446fd01d4e002d2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"86958e44-ffea-4ea7-a86e-8fc3a1ba9bf6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tokenizer_summary\", \"title\": \"Summary of the tokenizers\", \"tokens\": 4431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"899d2eccc2ced12bbdbca304f2c8caed6a1f4fec961684e48ff25cd22d77d606\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"402d3aa4-69ea-4346-928e-4843937ae944\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a199022d28c511731e84149e966ab1cece3b7efff923c849eefe3ece8bc35744\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11160, \"end_char_idx\": 14493, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb1e6a04-2c55-43c0-8265-84fe092a5176", "doc_id": "cb1e6a04-2c55-43c0-8265-84fe092a5176", "ref_doc_id": "cb1e6a04-2c55-43c0-8265-84fe092a5176", "chroma:document": "To have a better base vocabulary, [GPT-2](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) uses bytes\nas the base vocabulary, which is a clever trick to force the base vocabulary to be of size 256 while ensuring that\nevery base character is included in the vocabulary. With some additional rules to deal with punctuation, the GPT2's\ntokenizer can tokenize every text without the need for the <unk> symbol. [GPT-2](model_doc/gpt) has a vocabulary\nsize of 50,257, which corresponds to the 256 bytes base tokens, a special end-of-text token and the symbols learned\nwith 50,000 merges.\n\n<a id='wordpiece'></a>\n\n### WordPiece\n\nWordPiece is the subword tokenization algorithm used for [BERT](model_doc/bert), [DistilBERT](model_doc/distilbert), and [Electra](model_doc/electra). The algorithm was outlined in [Japanese and Korean\nVoice Search (Schuster et al., 2012)](https://static.googleusercontent.com/media/research.google.com/ja//pubs/archive/37842.pdf) and is very similar to\nBPE. WordPiece first initializes the vocabulary to include every character present in the training data and\nprogressively learns a given number of merge rules. In contrast to BPE, WordPiece does not choose the most frequent\nsymbol pair, but the one that maximizes the likelihood of the training data once added to the vocabulary.\n\nSo what does this mean exactly? Referring to the previous example, maximizing the likelihood of the training data is\nequivalent to finding the symbol pair, whose probability divided by the probabilities of its first symbol followed by\nits second symbol is the greatest among all symbol pairs. *E.g.* `\"u\"`, followed by `\"g\"` would have only been\nmerged if the probability of `\"ug\"` divided by `\"u\"`, `\"g\"` would have been greater than for any other symbol\npair. Intuitively, WordPiece is slightly different to BPE in that it evaluates what it _loses_ by merging two symbols\nto ensure it's _worth it_.\n\n<a id='unigram'></a>\n\n### Unigram\n\nUnigram is a subword tokenization algorithm introduced in [Subword Regularization: Improving Neural Network Translation\nModels with Multiple Subword Candidates (Kudo, 2018)](https://arxiv.org/pdf/1804.10959.pdf). In contrast to BPE or\nWordPiece, Unigram initializes its base vocabulary to a large number of symbols and progressively trims down each\nsymbol to obtain a smaller vocabulary. The base vocabulary could for instance correspond to all pre-tokenized words and\nthe most common substrings. Unigram is not used directly for any of the models in the transformers, but it's used in\nconjunction with [SentencePiece](#sentencepiece).\n\nAt each training step, the Unigram algorithm defines a loss (often defined as the log-likelihood) over the training\ndata given the current vocabulary and a unigram language model. Then, for each symbol in the vocabulary, the algorithm\ncomputes how much the overall loss would increase if the symbol was to be removed from the vocabulary. Unigram then\nremoves p (with p usually being 10% or 20%) percent of the symbols whose loss increase is the lowest, *i.e.* those\nsymbols that least affect the overall loss over the training data. This process is repeated until the vocabulary has\nreached the desired size. The Unigram algorithm always keeps the base characters so that any word can be tokenized."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 402d3aa4-69ea-4346-928e-4843937ae944
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"402d3aa4-69ea-4346-928e-4843937ae944\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tokenizer_summary\", \"title\": \"Summary of the tokenizers\", \"tokens\": 4431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb1e6a04-2c55-43c0-8265-84fe092a5176\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tokenizer_summary\", \"title\": \"Summary of the tokenizers\", \"tokens\": 4431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"42019efa99141d419d86ef0bac547c2ebd2bec88434d903f5446fd01d4e002d2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cdb5403a-5230-4801-93e8-c394aaef241e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tokenizer_summary\", \"title\": \"Summary of the tokenizers\", \"tokens\": 4431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1427fca741d28fbb9f7a2bd9fb37288a178cbbe71bdcca5e2e4a21af1c433d41\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"714b1be9-2b7d-4176-8f6f-b73e6f651b3b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f1e792802583c60c44ef6f9ed6afd3ea851b1f0d8c7af63f931be4901c2b39d6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12814, \"end_char_idx\": 15869, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb1e6a04-2c55-43c0-8265-84fe092a5176", "doc_id": "cb1e6a04-2c55-43c0-8265-84fe092a5176", "ref_doc_id": "cb1e6a04-2c55-43c0-8265-84fe092a5176", "chroma:document": "*E.g.* `\"u\"`, followed by `\"g\"` would have only been\nmerged if the probability of `\"ug\"` divided by `\"u\"`, `\"g\"` would have been greater than for any other symbol\npair. Intuitively, WordPiece is slightly different to BPE in that it evaluates what it _loses_ by merging two symbols\nto ensure it's _worth it_.\n\n<a id='unigram'></a>\n\n### Unigram\n\nUnigram is a subword tokenization algorithm introduced in [Subword Regularization: Improving Neural Network Translation\nModels with Multiple Subword Candidates (Kudo, 2018)](https://arxiv.org/pdf/1804.10959.pdf). In contrast to BPE or\nWordPiece, Unigram initializes its base vocabulary to a large number of symbols and progressively trims down each\nsymbol to obtain a smaller vocabulary. The base vocabulary could for instance correspond to all pre-tokenized words and\nthe most common substrings. Unigram is not used directly for any of the models in the transformers, but it's used in\nconjunction with [SentencePiece](#sentencepiece).\n\nAt each training step, the Unigram algorithm defines a loss (often defined as the log-likelihood) over the training\ndata given the current vocabulary and a unigram language model. Then, for each symbol in the vocabulary, the algorithm\ncomputes how much the overall loss would increase if the symbol was to be removed from the vocabulary. Unigram then\nremoves p (with p usually being 10% or 20%) percent of the symbols whose loss increase is the lowest, *i.e.* those\nsymbols that least affect the overall loss over the training data. This process is repeated until the vocabulary has\nreached the desired size. The Unigram algorithm always keeps the base characters so that any word can be tokenized.\n\nBecause Unigram is not based on merge rules (in contrast to BPE and WordPiece), the algorithm has several ways of\ntokenizing new text after training. As an example, if a trained Unigram tokenizer exhibits the vocabulary:\n\n```\n[\"b\", \"g\", \"h\", \"n\", \"p\", \"s\", \"u\", \"ug\", \"un\", \"hug\"],\n```\n\n`\"hugs\"` could be tokenized both as `[\"hug\", \"s\"]`, `[\"h\", \"ug\", \"s\"]` or `[\"h\", \"u\", \"g\", \"s\"]`. So which one\nto choose? Unigram saves the probability of each token in the training corpus on top of saving the vocabulary so that\nthe probability of each possible tokenization can be computed after training. The algorithm simply picks the most\nlikely tokenization in practice, but also offers the possibility to sample a possible tokenization according to their\nprobabilities.\n\nThose probabilities are defined by the loss the tokenizer is trained on. Assuming that the training data consists of\nthe words \\\\(x_{1}, \\dots, x_{N}\\\\) and that the set of all possible tokenizations for a word \\\\(x_{i}\\\\) is\ndefined as \\\\(S(x_{i})\\\\), then the overall loss is defined as\n\n$$\\mathcal{L} = -\\sum_{i=1}^{N} \\log \\left ( \\sum_{x \\in S(x_{i})} p(x) \\right )$$\n\n<a id='sentencepiece'></a>\n\n### SentencePiece\n\nAll tokenization algorithms described so far have the same problem: It is assumed that the input text uses spaces to\nseparate words. However, not all languages use spaces to separate words."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 714b1be9-2b7d-4176-8f6f-b73e6f651b3b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tokenizer_summary", "title": "Summary of the tokenizers", "tokens": 4431, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"714b1be9-2b7d-4176-8f6f-b73e6f651b3b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tokenizer_summary\", \"title\": \"Summary of the tokenizers\", \"tokens\": 4431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb1e6a04-2c55-43c0-8265-84fe092a5176\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tokenizer_summary\", \"title\": \"Summary of the tokenizers\", \"tokens\": 4431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"42019efa99141d419d86ef0bac547c2ebd2bec88434d903f5446fd01d4e002d2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"402d3aa4-69ea-4346-928e-4843937ae944\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tokenizer_summary\", \"title\": \"Summary of the tokenizers\", \"tokens\": 4431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6e6609527a2ac9f496a884fce72d1bb3ecb47b199d9e36befb2b06197f9e2b9a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14404, \"end_char_idx\": 16925, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb1e6a04-2c55-43c0-8265-84fe092a5176", "doc_id": "cb1e6a04-2c55-43c0-8265-84fe092a5176", "ref_doc_id": "cb1e6a04-2c55-43c0-8265-84fe092a5176", "chroma:document": "The Unigram algorithm always keeps the base characters so that any word can be tokenized.\n\nBecause Unigram is not based on merge rules (in contrast to BPE and WordPiece), the algorithm has several ways of\ntokenizing new text after training. As an example, if a trained Unigram tokenizer exhibits the vocabulary:\n\n```\n[\"b\", \"g\", \"h\", \"n\", \"p\", \"s\", \"u\", \"ug\", \"un\", \"hug\"],\n```\n\n`\"hugs\"` could be tokenized both as `[\"hug\", \"s\"]`, `[\"h\", \"ug\", \"s\"]` or `[\"h\", \"u\", \"g\", \"s\"]`. So which one\nto choose? Unigram saves the probability of each token in the training corpus on top of saving the vocabulary so that\nthe probability of each possible tokenization can be computed after training. The algorithm simply picks the most\nlikely tokenization in practice, but also offers the possibility to sample a possible tokenization according to their\nprobabilities.\n\nThose probabilities are defined by the loss the tokenizer is trained on. Assuming that the training data consists of\nthe words \\\\(x_{1}, \\dots, x_{N}\\\\) and that the set of all possible tokenizations for a word \\\\(x_{i}\\\\) is\ndefined as \\\\(S(x_{i})\\\\), then the overall loss is defined as\n\n$$\\mathcal{L} = -\\sum_{i=1}^{N} \\log \\left ( \\sum_{x \\in S(x_{i})} p(x) \\right )$$\n\n<a id='sentencepiece'></a>\n\n### SentencePiece\n\nAll tokenization algorithms described so far have the same problem: It is assumed that the input text uses spaces to\nseparate words. However, not all languages use spaces to separate words. One possible solution is to use language\nspecific pre-tokenizers, *e.g.* [XLM](model_doc/xlm) uses a specific Chinese, Japanese, and Thai pre-tokenizer.\nTo solve this problem more generally, [SentencePiece: A simple and language independent subword tokenizer and\ndetokenizer for Neural Text Processing (Kudo et al., 2018)](https://arxiv.org/pdf/1808.06226.pdf) treats the input\nas a raw input stream, thus including the space in the set of characters to use. It then uses the BPE or unigram\nalgorithm to construct the appropriate vocabulary.\n\nThe [`XLNetTokenizer`] uses SentencePiece for example, which is also why in the example earlier the\n`\"\u2581\"` character was included in the vocabulary. Decoding with SentencePiece is very easy since all tokens can just be\nconcatenated and `\"\u2581\"` is replaced by a space.\n\nAll transformers models in the library that use SentencePiece use it in combination with unigram. Examples of models\nusing SentencePiece are [ALBERT](model_doc/albert), [XLNet](model_doc/xlnet), [Marian](model_doc/marian), and [T5](model_doc/t5)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: b4ddb1c2-2825-43f9-ae39-0f1307f1f6a9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"b4ddb1c2-2825-43f9-ae39-0f1307f1f6a9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/deepspeed\", \"title\": \"DeepSpeed\", \"tokens\": 14117, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"afaba1b1-fa3b-417a-b720-ff69bd7e6bd7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/deepspeed\", \"title\": \"DeepSpeed\", \"tokens\": 14117, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8d7c725c-8867-44bb-925e-5822f47f3654\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b259ba8d0f4db412c3051f2fe4331a9609ef34d1d9270717ff5e2bc90f5991a5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2883, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "afaba1b1-fa3b-417a-b720-ff69bd7e6bd7", "doc_id": "afaba1b1-fa3b-417a-b720-ff69bd7e6bd7", "ref_doc_id": "afaba1b1-fa3b-417a-b720-ff69bd7e6bd7", "chroma:document": "# DeepSpeed\n\n[DeepSpeed](https://www.deepspeed.ai/) is a PyTorch optimization library that makes distributed training memory-efficient and fast. At its core is the [Zero Redundancy Optimizer (ZeRO)](https://hf.co/papers/1910.02054) which enables training large models at scale. ZeRO works in several stages:\n\n* ZeRO-1, optimizer state partitioning across GPUs\n* ZeRO-2, gradient partitioning across GPUs\n* ZeRO-3, parameter partitioning across GPUs\n\nIn GPU-limited environments, ZeRO also enables offloading optimizer memory and computation from the GPU to the CPU to fit and train really large models on a single GPU. DeepSpeed is integrated with the Transformers [`Trainer`] class for all ZeRO stages and offloading. All you need to do is provide a config file or you can use a provided template. For inference, Transformers support ZeRO-3 and offloading since it allows loading huge models.\n\nThis guide will walk you through how to deploy DeepSpeed training, the features you can enable, how to setup the config files for different ZeRO stages, offloading, inference, and using DeepSpeed without the [`Trainer`].\n\n## Installation\n\nDeepSpeed is available to install from PyPI or Transformers (for more detailed installation options, take a look at the DeepSpeed [installation details](https://www.deepspeed.ai/tutorials/advanced-install/) or the GitHub [README](https://github.com/microsoft/deepspeed#installation)).\n\n<Tip>\n\nIf you're having difficulties installing DeepSpeed, check the [DeepSpeed CUDA installation](../debugging#deepspeed-cuda-installation) guide. While DeepSpeed has a pip installable PyPI package, it is highly recommended to [install it from source](https://www.deepspeed.ai/tutorials/advanced-install/#install-deepspeed-from-source) to best match your hardware and to support certain features, like 1-bit Adam, which aren\u2019t available in the PyPI distribution.\n\n</Tip>\n\n<hfoptions id=\"install\">\n<hfoption id=\"PyPI\">\n\n```bash\npip install deepspeed\n```\n\n</hfoption>\n<hfoption id=\"Transformers\">\n\n```bash\npip install transformers[deepspeed]\n```\n\n</hfoption>\n</hfoptions>\n\n## Memory requirements\n\nBefore you begin, it is a good idea to check whether you have enough GPU and CPU memory to fit your model. DeepSpeed provides a tool for estimating the required CPU/GPU memory. For example, to estimate the memory requirements for the [bigscience/T0_3B](bigscience/T0_3B) model on a single GPU:\n\n```bash\n$ python -c 'from transformers import AutoModel; \\\nfrom deepspeed.runtime.zero.stage3 import estimate_zero3_model_states_mem_needs_all_live; \\\nmodel = AutoModel.from_pretrained(\"bigscience/T0_3B\"); \\\nestimate_zero3_model_states_mem_needs_all_live(model, num_gpus_per_node=1, num_nodes=1)'\n[...]\nEstimated memory needed for params, optim states and gradients for a:\nHW: Setup with 1 node, 1 GPU per node.\nSW: Model with 2783M total params, 65M largest layer params."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 8d7c725c-8867-44bb-925e-5822f47f3654
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"8d7c725c-8867-44bb-925e-5822f47f3654\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/deepspeed\", \"title\": \"DeepSpeed\", \"tokens\": 14117, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"afaba1b1-fa3b-417a-b720-ff69bd7e6bd7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/deepspeed\", \"title\": \"DeepSpeed\", \"tokens\": 14117, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b4ddb1c2-2825-43f9-ae39-0f1307f1f6a9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/deepspeed\", \"title\": \"DeepSpeed\", \"tokens\": 14117, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"99f2398d62a4251882d0fc1e5a1038425db3984644438f7c0b6612876dbd112c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ac4d615c-7236-4f48-a95d-ec9039242d78\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d426d2dbab6d384d3da2fd4eaae004d80281ba432f770744071b926c91a644de\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1420, \"end_char_idx\": 4081, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "afaba1b1-fa3b-417a-b720-ff69bd7e6bd7", "doc_id": "afaba1b1-fa3b-417a-b720-ff69bd7e6bd7", "ref_doc_id": "afaba1b1-fa3b-417a-b720-ff69bd7e6bd7", "chroma:document": "<Tip>\n\nIf you're having difficulties installing DeepSpeed, check the [DeepSpeed CUDA installation](../debugging#deepspeed-cuda-installation) guide. While DeepSpeed has a pip installable PyPI package, it is highly recommended to [install it from source](https://www.deepspeed.ai/tutorials/advanced-install/#install-deepspeed-from-source) to best match your hardware and to support certain features, like 1-bit Adam, which aren\u2019t available in the PyPI distribution.\n\n</Tip>\n\n<hfoptions id=\"install\">\n<hfoption id=\"PyPI\">\n\n```bash\npip install deepspeed\n```\n\n</hfoption>\n<hfoption id=\"Transformers\">\n\n```bash\npip install transformers[deepspeed]\n```\n\n</hfoption>\n</hfoptions>\n\n## Memory requirements\n\nBefore you begin, it is a good idea to check whether you have enough GPU and CPU memory to fit your model. DeepSpeed provides a tool for estimating the required CPU/GPU memory. For example, to estimate the memory requirements for the [bigscience/T0_3B](bigscience/T0_3B) model on a single GPU:\n\n```bash\n$ python -c 'from transformers import AutoModel; \\\nfrom deepspeed.runtime.zero.stage3 import estimate_zero3_model_states_mem_needs_all_live; \\\nmodel = AutoModel.from_pretrained(\"bigscience/T0_3B\"); \\\nestimate_zero3_model_states_mem_needs_all_live(model, num_gpus_per_node=1, num_nodes=1)'\n[...]\nEstimated memory needed for params, optim states and gradients for a:\nHW: Setup with 1 node, 1 GPU per node.\nSW: Model with 2783M total params, 65M largest layer params.\n  per CPU  |  per GPU |   Options\n   70.00GB |   0.25GB | offload_param=cpu , offload_optimizer=cpu , zero_init=1\n   70.00GB |   0.25GB | offload_param=cpu , offload_optimizer=cpu , zero_init=0\n   62.23GB |   5.43GB | offload_param=none, offload_optimizer=cpu , zero_init=1\n   62.23GB |   5.43GB | offload_param=none, offload_optimizer=cpu , zero_init=0\n    0.37GB |  46.91GB | offload_param=none, offload_optimizer=none, zero_init=1\n   15.56GB |  46.91GB | offload_param=none, offload_optimizer=none, zero_init=0\n```\n\nThis means you either need a single 80GB GPU without CPU offload or a 8GB GPU and a ~60GB CPU to offload to (these are just the memory requirements for the parameters, optimizer states and gradients, and you'll need a bit more for the CUDA kernels and activations). You should also consider the tradeoff between cost and speed because it'll be cheaper to rent or buy a smaller GPU but it'll take longer to train your model.\n\nIf you have enough GPU memory make sure you disable CPU/NVMe offload to make everything faster.\n\n## Select a ZeRO stage\n\nAfter you've installed DeepSpeed and have a better idea of your memory requirements, the next step is selecting a ZeRO stage to use."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: ac4d615c-7236-4f48-a95d-ec9039242d78
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"ac4d615c-7236-4f48-a95d-ec9039242d78\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/deepspeed\", \"title\": \"DeepSpeed\", \"tokens\": 14117, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"afaba1b1-fa3b-417a-b720-ff69bd7e6bd7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/deepspeed\", \"title\": \"DeepSpeed\", \"tokens\": 14117, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8d7c725c-8867-44bb-925e-5822f47f3654\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/deepspeed\", \"title\": \"DeepSpeed\", \"tokens\": 14117, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"e1ff953fc47b9cd37de0e3fde8f8612cf634650fd12bc09c7863e1d46d8c783d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f022b3d5-d7b1-4e4f-a9cf-91ad69971c50\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"20474397d829d08f4ff0fa535341607a156dc33dc7635e05ebd4f86163aa1664\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2823, \"end_char_idx\": 5722, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "afaba1b1-fa3b-417a-b720-ff69bd7e6bd7", "doc_id": "afaba1b1-fa3b-417a-b720-ff69bd7e6bd7", "ref_doc_id": "afaba1b1-fa3b-417a-b720-ff69bd7e6bd7", "chroma:document": "SW: Model with 2783M total params, 65M largest layer params.\n  per CPU  |  per GPU |   Options\n   70.00GB |   0.25GB | offload_param=cpu , offload_optimizer=cpu , zero_init=1\n   70.00GB |   0.25GB | offload_param=cpu , offload_optimizer=cpu , zero_init=0\n   62.23GB |   5.43GB | offload_param=none, offload_optimizer=cpu , zero_init=1\n   62.23GB |   5.43GB | offload_param=none, offload_optimizer=cpu , zero_init=0\n    0.37GB |  46.91GB | offload_param=none, offload_optimizer=none, zero_init=1\n   15.56GB |  46.91GB | offload_param=none, offload_optimizer=none, zero_init=0\n```\n\nThis means you either need a single 80GB GPU without CPU offload or a 8GB GPU and a ~60GB CPU to offload to (these are just the memory requirements for the parameters, optimizer states and gradients, and you'll need a bit more for the CUDA kernels and activations). You should also consider the tradeoff between cost and speed because it'll be cheaper to rent or buy a smaller GPU but it'll take longer to train your model.\n\nIf you have enough GPU memory make sure you disable CPU/NVMe offload to make everything faster.\n\n## Select a ZeRO stage\n\nAfter you've installed DeepSpeed and have a better idea of your memory requirements, the next step is selecting a ZeRO stage to use. In order of fastest and most memory-efficient:\n\n| Fastest          | Memory efficient |\n|------------------|------------------|\n| ZeRO-1           | ZeRO-3 + offload |\n| ZeRO-2           | ZeRO-3           |\n| ZeRO-2 + offload | ZeRO-2 + offload |\n| ZeRO-3           | ZeRO-2           |\n| ZeRO-3 + offload | ZeRO-1           |\n\nTo find what works best for you, start with the fastest approach and if you run out of memory, try the next stage which is slower but more memory efficient. Feel free to work in whichever direction you prefer (starting with the most memory efficient or fastest) to discover the appropriate balance between speed and memory usage.\n\nA general process you can use is (start with batch size of 1):\n\n1. enable gradient checkpointing\n2. try ZeRO-2\n3. try ZeRO-2 and offload the optimizer\n4. try ZeRO-3\n5. try ZeRO-3 and offload parameters to the CPU\n6. try ZeRO-3 and offload parameters and the optimizer to the CPU\n7. try lowering various default values like a narrower search beam if you're using the [`~GenerationMixin.generate`] method\n8. try mixed half-precision (fp16 on older GPU architectures and bf16 on Ampere) over full-precision weights\n9. add more hardware if possible or enable Infinity to offload parameters and the optimizer to a NVMe\n10. once you're not running out of memory, measure effective throughput and then try to increase the batch size as large as you can to maximize GPU efficiency\n11. lastly, try to optimize your training setup by disabling some offload features or use a faster ZeRO stage and increasing/decreasing the batch size to find the best tradeoff between speed and memory usage"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: f022b3d5-d7b1-4e4f-a9cf-91ad69971c50
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"f022b3d5-d7b1-4e4f-a9cf-91ad69971c50\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/deepspeed\", \"title\": \"DeepSpeed\", \"tokens\": 14117, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"afaba1b1-fa3b-417a-b720-ff69bd7e6bd7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/deepspeed\", \"title\": \"DeepSpeed\", \"tokens\": 14117, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ac4d615c-7236-4f48-a95d-ec9039242d78\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/deepspeed\", \"title\": \"DeepSpeed\", \"tokens\": 14117, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"90629852709f297cbd0a8aed89b2f74e4d7969cadb9c74a6ff6873e794f5a83a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"065b6f8a-9922-4f7e-80fe-2c3089651eca\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"57c662e2d5b645555a19baa75f67d852ec634028cd166a6c796b28d19e0a6a19\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4082, \"end_char_idx\": 7404, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "afaba1b1-fa3b-417a-b720-ff69bd7e6bd7", "doc_id": "afaba1b1-fa3b-417a-b720-ff69bd7e6bd7", "ref_doc_id": "afaba1b1-fa3b-417a-b720-ff69bd7e6bd7", "chroma:document": "In order of fastest and most memory-efficient:\n\n| Fastest          | Memory efficient |\n|------------------|------------------|\n| ZeRO-1           | ZeRO-3 + offload |\n| ZeRO-2           | ZeRO-3           |\n| ZeRO-2 + offload | ZeRO-2 + offload |\n| ZeRO-3           | ZeRO-2           |\n| ZeRO-3 + offload | ZeRO-1           |\n\nTo find what works best for you, start with the fastest approach and if you run out of memory, try the next stage which is slower but more memory efficient. Feel free to work in whichever direction you prefer (starting with the most memory efficient or fastest) to discover the appropriate balance between speed and memory usage.\n\nA general process you can use is (start with batch size of 1):\n\n1. enable gradient checkpointing\n2. try ZeRO-2\n3. try ZeRO-2 and offload the optimizer\n4. try ZeRO-3\n5. try ZeRO-3 and offload parameters to the CPU\n6. try ZeRO-3 and offload parameters and the optimizer to the CPU\n7. try lowering various default values like a narrower search beam if you're using the [`~GenerationMixin.generate`] method\n8. try mixed half-precision (fp16 on older GPU architectures and bf16 on Ampere) over full-precision weights\n9. add more hardware if possible or enable Infinity to offload parameters and the optimizer to a NVMe\n10. once you're not running out of memory, measure effective throughput and then try to increase the batch size as large as you can to maximize GPU efficiency\n11. lastly, try to optimize your training setup by disabling some offload features or use a faster ZeRO stage and increasing/decreasing the batch size to find the best tradeoff between speed and memory usage\n\n\n## DeepSpeed configuration file\n\nDeepSpeed works with the [`Trainer`] class by way of a config file containing all the parameters for configuring how you want setup your training run. When you execute your training script, DeepSpeed logs the configuration it received from [`Trainer`] to the console so you can see exactly what configuration was used.\n\n<Tip>\n\nFind a complete list of DeepSpeed configuration options on the [DeepSpeed Configuration JSON](https://www.deepspeed.ai/docs/config-json/) reference. You can also find more practical examples of various DeepSpeed configuration examples on the [DeepSpeedExamples](https://github.com/microsoft/DeepSpeedExamples) repository or the main [DeepSpeed](https://github.com/microsoft/DeepSpeed) repository. To quickly find specific examples, you can:\n\n```bash\ngit clone https://github.com/microsoft/DeepSpeedExamples\ncd DeepSpeedExamples\nfind . -name '*json'\n# find examples with the Lamb optimizer\ngrep -i Lamb $(find . -name '*json')\n```\n\n</Tip>\n\nThe DeepSpeed configuration file is passed as a path to a JSON file if you're training from the command line interface or as a nested `dict` object if you're using the [`Trainer`] in a notebook setting.\n\n<hfoptions id=\"pass-config\">\n<hfoption id=\"path to file\">\n\n```py\nTrainingArguments(..., deepspeed=\"path/to/deepspeed_config.json\")\n```\n\n</hfoption>\n<hfoption id=\"nested dict\">\n\n```py\nds_config_dict = dict(scheduler=scheduler_params, optimizer=optimizer_params)\nargs = TrainingArguments(..., deepspeed=ds_config_dict)\ntrainer = Trainer(model, args, ...)\n```\n\n</hfoption>\n</hfoptions>\n\n### DeepSpeed and Trainer parameters\n\nThere are three types of configuration parameters:\n\n1."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 065b6f8a-9922-4f7e-80fe-2c3089651eca
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"065b6f8a-9922-4f7e-80fe-2c3089651eca\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/deepspeed\", \"title\": \"DeepSpeed\", \"tokens\": 14117, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"afaba1b1-fa3b-417a-b720-ff69bd7e6bd7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/deepspeed\", \"title\": \"DeepSpeed\", \"tokens\": 14117, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f022b3d5-d7b1-4e4f-a9cf-91ad69971c50\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/deepspeed\", \"title\": \"DeepSpeed\", \"tokens\": 14117, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"6ca4d0ae303ef0e2dda560f5089ba14f7745228a4a48a8ab94ff402980c92e8d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2d0a7776-1819-47bb-9a21-67ac2291816a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6a484ff399c53116700d6212e3896df029a3b4418281542312ec2188cd9fe06b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5725, \"end_char_idx\": 9333, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "afaba1b1-fa3b-417a-b720-ff69bd7e6bd7", "doc_id": "afaba1b1-fa3b-417a-b720-ff69bd7e6bd7", "ref_doc_id": "afaba1b1-fa3b-417a-b720-ff69bd7e6bd7", "chroma:document": "## DeepSpeed configuration file\n\nDeepSpeed works with the [`Trainer`] class by way of a config file containing all the parameters for configuring how you want setup your training run. When you execute your training script, DeepSpeed logs the configuration it received from [`Trainer`] to the console so you can see exactly what configuration was used.\n\n<Tip>\n\nFind a complete list of DeepSpeed configuration options on the [DeepSpeed Configuration JSON](https://www.deepspeed.ai/docs/config-json/) reference. You can also find more practical examples of various DeepSpeed configuration examples on the [DeepSpeedExamples](https://github.com/microsoft/DeepSpeedExamples) repository or the main [DeepSpeed](https://github.com/microsoft/DeepSpeed) repository. To quickly find specific examples, you can:\n\n```bash\ngit clone https://github.com/microsoft/DeepSpeedExamples\ncd DeepSpeedExamples\nfind . -name '*json'\n# find examples with the Lamb optimizer\ngrep -i Lamb $(find . -name '*json')\n```\n\n</Tip>\n\nThe DeepSpeed configuration file is passed as a path to a JSON file if you're training from the command line interface or as a nested `dict` object if you're using the [`Trainer`] in a notebook setting.\n\n<hfoptions id=\"pass-config\">\n<hfoption id=\"path to file\">\n\n```py\nTrainingArguments(..., deepspeed=\"path/to/deepspeed_config.json\")\n```\n\n</hfoption>\n<hfoption id=\"nested dict\">\n\n```py\nds_config_dict = dict(scheduler=scheduler_params, optimizer=optimizer_params)\nargs = TrainingArguments(..., deepspeed=ds_config_dict)\ntrainer = Trainer(model, args, ...)\n```\n\n</hfoption>\n</hfoptions>\n\n### DeepSpeed and Trainer parameters\n\nThere are three types of configuration parameters:\n\n1. Some of the configuration parameters are shared by [`Trainer`] and DeepSpeed, and it can be difficult to identify errors when there are conflicting definitions. To make it easier, these shared configuration parameters are configured from the [`Trainer`] command line arguments.\n\n2. Some configuration parameters that are automatically derived from the model configuration so you don't need to manually adjust these values. The [`Trainer`] uses a configuration value `auto` to determine set the most correct or efficient value. You could set your own configuration parameters explicitly, but you must take care to ensure the [`Trainer`] arguments and DeepSpeed configuration parameters agree. Mismatches may cause the training to fail in very difficult to detect ways!\n\n3. Some configuration parameters specific to DeepSpeed only which need to be manually set based on your training needs.\n\nYou could also modify the DeepSpeed configuration and edit [`TrainingArguments`] from it:\n\n1. Create or load a DeepSpeed configuration to use as the main configuration\n2. Create a [`TrainingArguments`] object based on these DeepSpeed configuration values\n\nSome values, such as `scheduler.params.total_num_steps` are calculated by the [`Trainer`] during training.\n\n### ZeRO configuration\n\nThere are three configurations, each corresponding to a different ZeRO stage. Stage 1 is not as interesting for scalability, and this guide focuses on stages 2 and 3. The `zero_optimization` configuration contains all the options for what to enable and how to configure them. For a more detailed explanation of each parameter, take a look at the [DeepSpeed Configuration JSON](https://www.deepspeed.ai/docs/config-json/) reference.\n\n<Tip warning={true}>\nDeepSpeed doesn\u2019t validate parameter names and any typos fallback on the parameter's default setting. You can watch the DeepSpeed engine startup log messages to see what values it is going to use."}


