# embeddings_queue

created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 242a285b-537a-46d4-80ca-2276c3dc15cc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"242a285b-537a-46d4-80ca-2276c3dc15cc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d0be5bbc-3da9-40b5-86cd-6e745b701b3d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3e60ba6acb9b94b0caa16c4dcf8fd1e857d79b0cb4554071b24b5fbe5185a8c9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"59e12aa6-e3a9-4979-b7cc-7727cee8386b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b8695312d2ea15464b66d0686cc53131a07384dc40bb8ed7ee2c65462f60e135\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"419af2bb-b4d9-4000-85b2-c0ae8c67d43f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6314744aa142dec6c5699d8efa4a2c87b252929610d77a0eb3e9fa5fd71c0a4b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3068, \"end_char_idx\": 5366, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "doc_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "ref_doc_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "chroma:document": "```py\n>>> import torch\n\n>>> from transformers import IdeficsForVisionText2Text, AutoProcessor\n\n>>> processor = AutoProcessor.from_pretrained(checkpoint)\n\n>>> model = IdeficsForVisionText2Text.from_pretrained(checkpoint, torch_dtype=torch.bfloat16, device_map=\"auto\")\n```\n\nSetting `device_map` to `\"auto\"` will automatically determine how to load and store the model weights in the most optimized \nmanner given existing devices.\n\n### Quantized model\n\nIf high-memory GPU availability is an issue, you can load the quantized version of the model. To load the model and the \nprocessor in 4bit precision, pass a `BitsAndBytesConfig` to the `from_pretrained` method and the model will be compressed \non the fly while loading.\n\n```py\n>>> import torch\n>>> from transformers import IdeficsForVisionText2Text, AutoProcessor, BitsAndBytesConfig\n\n>>> quantization_config = BitsAndBytesConfig(\n...     load_in_4bit=True,\n...     bnb_4bit_compute_dtype=torch.float16,\n... )\n\n>>> processor = AutoProcessor.from_pretrained(checkpoint)\n\n>>> model = IdeficsForVisionText2Text.from_pretrained(\n...     checkpoint,\n...     quantization_config=quantization_config,\n...     device_map=\"auto\"\n... )\n```\n\nNow that you have the model loaded in one of the suggested ways, let's move on to exploring tasks that you can use IDEFICS for.\n\n## Image captioning\nImage captioning is the task of predicting a caption for a given image. A common application is to aid visually impaired \npeople navigate through different situations, for instance, explore image content online. \n\nTo illustrate the task, get an image to be captioned, e.g.:\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-im-captioning.jpg\" alt=\"Image of a puppy in a flower bed\"/>\n</div>\n\nPhoto by [Hendo Wang](https://unsplash.com/@hendoo). \n\nIDEFICS accepts text and image prompts. However, to caption an image, you do not have to provide a text prompt to the \nmodel, only the preprocessed input image. Without a text prompt, the model will start generating text from the \nBOS (beginning-of-sequence) token thus creating a caption.\n\nAs image input to the model, you can use either an image object (`PIL.Image`) or a url from which the image can be retrieved."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 419af2bb-b4d9-4000-85b2-c0ae8c67d43f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"419af2bb-b4d9-4000-85b2-c0ae8c67d43f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d0be5bbc-3da9-40b5-86cd-6e745b701b3d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3e60ba6acb9b94b0caa16c4dcf8fd1e857d79b0cb4554071b24b5fbe5185a8c9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"242a285b-537a-46d4-80ca-2276c3dc15cc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5b4f5570c180f7bf576e64b5461c69788eb727d15f0b2b85151b95c023ab0b7d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5ea25d55-fa0f-40e7-bc84-3e228eb14587\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"65ea3011d628dee36c517d306deb9deeaf604bef2ac68d605ca3af5917cb66a1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3789, \"end_char_idx\": 6581, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "doc_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "ref_doc_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "chroma:document": "```py\n>>> import torch\n>>> from transformers import IdeficsForVisionText2Text, AutoProcessor, BitsAndBytesConfig\n\n>>> quantization_config = BitsAndBytesConfig(\n...     load_in_4bit=True,\n...     bnb_4bit_compute_dtype=torch.float16,\n... )\n\n>>> processor = AutoProcessor.from_pretrained(checkpoint)\n\n>>> model = IdeficsForVisionText2Text.from_pretrained(\n...     checkpoint,\n...     quantization_config=quantization_config,\n...     device_map=\"auto\"\n... )\n```\n\nNow that you have the model loaded in one of the suggested ways, let's move on to exploring tasks that you can use IDEFICS for.\n\n## Image captioning\nImage captioning is the task of predicting a caption for a given image. A common application is to aid visually impaired \npeople navigate through different situations, for instance, explore image content online. \n\nTo illustrate the task, get an image to be captioned, e.g.:\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-im-captioning.jpg\" alt=\"Image of a puppy in a flower bed\"/>\n</div>\n\nPhoto by [Hendo Wang](https://unsplash.com/@hendoo). \n\nIDEFICS accepts text and image prompts. However, to caption an image, you do not have to provide a text prompt to the \nmodel, only the preprocessed input image. Without a text prompt, the model will start generating text from the \nBOS (beginning-of-sequence) token thus creating a caption.\n\nAs image input to the model, you can use either an image object (`PIL.Image`) or a url from which the image can be retrieved.\n\n```py\n>>> prompt = [\n...     \"https://images.unsplash.com/photo-1583160247711-2191776b4b91?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3542&q=80\",\n... ]\n\n>>> inputs = processor(prompt, return_tensors=\"pt\").to(\"cuda\")\n>>> bad_words_ids = processor.tokenizer([\"<image>\", \"<fake_token_around_image>\"], add_special_tokens=False).input_ids\n\n>>> generated_ids = model.generate(**inputs, max_new_tokens=10, bad_words_ids=bad_words_ids)\n>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)\n>>> print(generated_text[0])\nA puppy in a flower bed\n```\n\n<Tip>\n\nIt is a good idea to include the `bad_words_ids` in the call to `generate` to avoid errors arising when increasing \nthe `max_new_tokens`: the model will want to generate a new `<image>` or `<fake_token_around_image>` token when there \nis no image being generated by the model.\nYou can set it on-the-fly as in this guide, or store in the `GenerationConfig` as described in the [Text generation strategies](../generation_strategies) guide.\n</Tip>\n\n## Prompted image captioning\n\nYou can extend image captioning by providing a text prompt, which the model will continue given the image."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 5ea25d55-fa0f-40e7-bc84-3e228eb14587
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5ea25d55-fa0f-40e7-bc84-3e228eb14587\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d0be5bbc-3da9-40b5-86cd-6e745b701b3d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3e60ba6acb9b94b0caa16c4dcf8fd1e857d79b0cb4554071b24b5fbe5185a8c9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"419af2bb-b4d9-4000-85b2-c0ae8c67d43f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8b21268d119b02729689bc0fee05003f69fe679e255c0c1cac8f5ea07c3c3daa\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"88a4bb26-1cf9-4c89-8339-1aa57dc1dd6c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"654548ee4c21d47b1c2af7c733295e93ef9e27c3a9323b1a09b738509de0df7a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5111, \"end_char_idx\": 7991, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "doc_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "ref_doc_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "chroma:document": "Without a text prompt, the model will start generating text from the \nBOS (beginning-of-sequence) token thus creating a caption.\n\nAs image input to the model, you can use either an image object (`PIL.Image`) or a url from which the image can be retrieved.\n\n```py\n>>> prompt = [\n...     \"https://images.unsplash.com/photo-1583160247711-2191776b4b91?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3542&q=80\",\n... ]\n\n>>> inputs = processor(prompt, return_tensors=\"pt\").to(\"cuda\")\n>>> bad_words_ids = processor.tokenizer([\"<image>\", \"<fake_token_around_image>\"], add_special_tokens=False).input_ids\n\n>>> generated_ids = model.generate(**inputs, max_new_tokens=10, bad_words_ids=bad_words_ids)\n>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)\n>>> print(generated_text[0])\nA puppy in a flower bed\n```\n\n<Tip>\n\nIt is a good idea to include the `bad_words_ids` in the call to `generate` to avoid errors arising when increasing \nthe `max_new_tokens`: the model will want to generate a new `<image>` or `<fake_token_around_image>` token when there \nis no image being generated by the model.\nYou can set it on-the-fly as in this guide, or store in the `GenerationConfig` as described in the [Text generation strategies](../generation_strategies) guide.\n</Tip>\n\n## Prompted image captioning\n\nYou can extend image captioning by providing a text prompt, which the model will continue given the image. Let's take \nanother image to illustrate:\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-prompted-im-captioning.jpg\" alt=\"Image of the Eiffel Tower at night\"/>\n</div>\n\nPhoto by [Denys Nevozhai](https://unsplash.com/@dnevozhai).\n   \nTextual and image prompts can be passed to the model's processor as a single list to create appropriate inputs.\n\n```py\n>>> prompt = [\n...     \"https://images.unsplash.com/photo-1543349689-9a4d426bee8e?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3501&q=80\",\n...     \"This is an image of \",\n... ]\n\n>>> inputs = processor(prompt, return_tensors=\"pt\").to(\"cuda\")\n>>> bad_words_ids = processor.tokenizer([\"<image>\", \"<fake_token_around_image>\"], add_special_tokens=False).input_ids\n\n>>> generated_ids = model.generate(**inputs, max_new_tokens=10, bad_words_ids=bad_words_ids)\n>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)\n>>> print(generated_text[0])\nThis is an image of the Eiffel Tower in Paris, France.\n```\n\n## Few-shot prompting\n\nWhile IDEFICS demonstrates great zero-shot results, your task may require a certain format of the caption, or come with \nother restrictions or requirements that increase task's complexity. Few-shot prompting can be used to enable in-context learning."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 88a4bb26-1cf9-4c89-8339-1aa57dc1dd6c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"88a4bb26-1cf9-4c89-8339-1aa57dc1dd6c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d0be5bbc-3da9-40b5-86cd-6e745b701b3d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3e60ba6acb9b94b0caa16c4dcf8fd1e857d79b0cb4554071b24b5fbe5185a8c9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5ea25d55-fa0f-40e7-bc84-3e228eb14587\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fa539d43a77412145ceab596d8890de2036b120262a1fa2bb51ad0eacce7e8c2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9765e6c5-dd4c-4977-867c-d0f3f7ab9232\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"da22b7ba743231e97503e1c29f8db4dea794aabfdcdcc7948f3d4cd4e7218d15\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6582, \"end_char_idx\": 9041, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "doc_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "ref_doc_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "chroma:document": "Let's take \nanother image to illustrate:\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-prompted-im-captioning.jpg\" alt=\"Image of the Eiffel Tower at night\"/>\n</div>\n\nPhoto by [Denys Nevozhai](https://unsplash.com/@dnevozhai).\n   \nTextual and image prompts can be passed to the model's processor as a single list to create appropriate inputs.\n\n```py\n>>> prompt = [\n...     \"https://images.unsplash.com/photo-1543349689-9a4d426bee8e?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3501&q=80\",\n...     \"This is an image of \",\n... ]\n\n>>> inputs = processor(prompt, return_tensors=\"pt\").to(\"cuda\")\n>>> bad_words_ids = processor.tokenizer([\"<image>\", \"<fake_token_around_image>\"], add_special_tokens=False).input_ids\n\n>>> generated_ids = model.generate(**inputs, max_new_tokens=10, bad_words_ids=bad_words_ids)\n>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)\n>>> print(generated_text[0])\nThis is an image of the Eiffel Tower in Paris, France.\n```\n\n## Few-shot prompting\n\nWhile IDEFICS demonstrates great zero-shot results, your task may require a certain format of the caption, or come with \nother restrictions or requirements that increase task's complexity. Few-shot prompting can be used to enable in-context learning.\nBy providing examples in the prompt, you can steer the model to generate results that mimic the format of given examples. \n\nLet's use the previous image of the Eiffel Tower as an example for the model and build a prompt that demonstrates to the model \nthat in addition to learning what the object in an image is, we would also like to get some interesting information about it. \nThen, let's see, if we can get the same response format for an image of the Statue of Liberty:\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-few-shot.jpg\" alt=\"Image of the Statue of Liberty\"/>\n</div>\n\nPhoto by [Juan Mayobre](https://unsplash.com/@jmayobres).\n  \n```py\n>>> prompt = [\"User:\",\n...            \"https://images.unsplash.com/photo-1543349689-9a4d426bee8e?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3501&q=80\",\n...            \"Describe this image.\\nAssistant: An image of the Eiffel Tower at night."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 9765e6c5-dd4c-4977-867c-d0f3f7ab9232
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9765e6c5-dd4c-4977-867c-d0f3f7ab9232\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d0be5bbc-3da9-40b5-86cd-6e745b701b3d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3e60ba6acb9b94b0caa16c4dcf8fd1e857d79b0cb4554071b24b5fbe5185a8c9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"88a4bb26-1cf9-4c89-8339-1aa57dc1dd6c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"44e1da43cb969f2d6fa55a275cae6f6acbbe691e613624877436973cc4b988f2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"79cbe94a-a98e-47f6-8cab-e97329935038\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8ab4a099d58b7dd269830a0ff351a502f2e7bd7ef074339997e740a7193538d3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7713, \"end_char_idx\": 10618, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "doc_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "ref_doc_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "chroma:document": "```\n\n## Few-shot prompting\n\nWhile IDEFICS demonstrates great zero-shot results, your task may require a certain format of the caption, or come with \nother restrictions or requirements that increase task's complexity. Few-shot prompting can be used to enable in-context learning.\nBy providing examples in the prompt, you can steer the model to generate results that mimic the format of given examples. \n\nLet's use the previous image of the Eiffel Tower as an example for the model and build a prompt that demonstrates to the model \nthat in addition to learning what the object in an image is, we would also like to get some interesting information about it. \nThen, let's see, if we can get the same response format for an image of the Statue of Liberty:\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-few-shot.jpg\" alt=\"Image of the Statue of Liberty\"/>\n</div>\n\nPhoto by [Juan Mayobre](https://unsplash.com/@jmayobres).\n  \n```py\n>>> prompt = [\"User:\",\n...            \"https://images.unsplash.com/photo-1543349689-9a4d426bee8e?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3501&q=80\",\n...            \"Describe this image.\\nAssistant: An image of the Eiffel Tower at night. Fun fact: the Eiffel Tower is the same height as an 81-storey building.\\n\",\n...            \"User:\",\n...            \"https://images.unsplash.com/photo-1524099163253-32b7f0256868?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3387&q=80\",\n...            \"Describe this image.\\nAssistant:\"\n...            ]\n\n>>> inputs = processor(prompt, return_tensors=\"pt\").to(\"cuda\")\n>>> bad_words_ids = processor.tokenizer([\"<image>\", \"<fake_token_around_image>\"], add_special_tokens=False).input_ids\n\n>>> generated_ids = model.generate(**inputs, max_new_tokens=30, bad_words_ids=bad_words_ids)\n>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)\n>>> print(generated_text[0])\nUser: Describe this image.\nAssistant: An image of the Eiffel Tower at night. Fun fact: the Eiffel Tower is the same height as an 81-storey building. \nUser: Describe this image.\nAssistant: An image of the Statue of Liberty. Fun fact: the Statue of Liberty is 151 feet tall.\n```\n\nNotice that just from a single example (i.e., 1-shot) the model has learned how to perform the task. For more complex tasks, \nfeel free to experiment with a larger number of examples (e.g., 3-shot, 5-shot, etc.).\n\n## Visual question answering\n\nVisual Question Answering (VQA) is the task of answering open-ended questions based on an image. Similar to image \ncaptioning it can be used in accessibility applications, but also in education (reasoning about visual materials), customer \nservice (questions about products based on images), and image retrieval."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 79cbe94a-a98e-47f6-8cab-e97329935038
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"79cbe94a-a98e-47f6-8cab-e97329935038\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d0be5bbc-3da9-40b5-86cd-6e745b701b3d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3e60ba6acb9b94b0caa16c4dcf8fd1e857d79b0cb4554071b24b5fbe5185a8c9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9765e6c5-dd4c-4977-867c-d0f3f7ab9232\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b5fdca2836360a87e76a373fd7328ee9774efd6b7cf9ef04a9ef3697c3ffcd3a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"95361241-f724-462a-b6f0-1bbbccc4b3ca\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c8f66bfa86885f3915a179717b6d0338d4942e643b5475e337780cba910cd25b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9811, \"end_char_idx\": 12703, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "doc_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "ref_doc_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "chroma:document": "Assistant: An image of the Eiffel Tower at night. Fun fact: the Eiffel Tower is the same height as an 81-storey building. \nUser: Describe this image.\nAssistant: An image of the Statue of Liberty. Fun fact: the Statue of Liberty is 151 feet tall.\n```\n\nNotice that just from a single example (i.e., 1-shot) the model has learned how to perform the task. For more complex tasks, \nfeel free to experiment with a larger number of examples (e.g., 3-shot, 5-shot, etc.).\n\n## Visual question answering\n\nVisual Question Answering (VQA) is the task of answering open-ended questions based on an image. Similar to image \ncaptioning it can be used in accessibility applications, but also in education (reasoning about visual materials), customer \nservice (questions about products based on images), and image retrieval.\n\nLet's get a new image for this task: \n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-vqa.jpg\" alt=\"Image of a couple having a picnic\"/>\n</div>\n\nPhoto by [Jarritos Mexican Soda](https://unsplash.com/@jarritos). \n\nYou can steer the model from image captioning to visual question answering by prompting it with appropriate instructions: \n\n```py\n>>> prompt = [\n...     \"Instruction: Provide an answer to the question. Use the image to answer.\\n\",\n...     \"https://images.unsplash.com/photo-1623944889288-cd147dbb517c?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3540&q=80\",\n...     \"Question: Where are these people and what's the weather like? Answer:\"\n... ]\n\n>>> inputs = processor(prompt, return_tensors=\"pt\").to(\"cuda\")\n>>> bad_words_ids = processor.tokenizer([\"<image>\", \"<fake_token_around_image>\"], add_special_tokens=False).input_ids\n\n>>> generated_ids = model.generate(**inputs, max_new_tokens=20, bad_words_ids=bad_words_ids)\n>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)\n>>> print(generated_text[0])\nInstruction: Provide an answer to the question. Use the image to answer.\n Question: Where are these people and what's the weather like? Answer: They're in a park in New York City, and it's a beautiful day.\n```\n\n## Image classification\n\nIDEFICS is capable of classifying images into different categories without being explicitly trained on data containing \nlabeled examples from those specific categories. Given a list of categories and using its image and text understanding \ncapabilities, the model can infer which category the image likely belongs to. \n\nSay, we have this image of a vegetable stand: \n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-classification.jpg\" alt=\"Image of a vegetable stand\"/>\n</div>\n\nPhoto by [Peter Wendt](https://unsplash.com/@peterwendt)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 95361241-f724-462a-b6f0-1bbbccc4b3ca
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"95361241-f724-462a-b6f0-1bbbccc4b3ca\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d0be5bbc-3da9-40b5-86cd-6e745b701b3d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3e60ba6acb9b94b0caa16c4dcf8fd1e857d79b0cb4554071b24b5fbe5185a8c9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"79cbe94a-a98e-47f6-8cab-e97329935038\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"89c58d9b593d46fcebc83a01668db25b75b2a65689be2029916086dc24b2e06d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6fda5281-4693-4f63-9332-ec5f94eb56ad\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"20c5eabf06d276d9cda6a51d857c804a7c0d6e1057a6c11ff9896d4510ae0363\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11420, \"end_char_idx\": 14556, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "doc_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "ref_doc_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "chroma:document": "Answer:\"\n... ]\n\n>>> inputs = processor(prompt, return_tensors=\"pt\").to(\"cuda\")\n>>> bad_words_ids = processor.tokenizer([\"<image>\", \"<fake_token_around_image>\"], add_special_tokens=False).input_ids\n\n>>> generated_ids = model.generate(**inputs, max_new_tokens=20, bad_words_ids=bad_words_ids)\n>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)\n>>> print(generated_text[0])\nInstruction: Provide an answer to the question. Use the image to answer.\n Question: Where are these people and what's the weather like? Answer: They're in a park in New York City, and it's a beautiful day.\n```\n\n## Image classification\n\nIDEFICS is capable of classifying images into different categories without being explicitly trained on data containing \nlabeled examples from those specific categories. Given a list of categories and using its image and text understanding \ncapabilities, the model can infer which category the image likely belongs to. \n\nSay, we have this image of a vegetable stand: \n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-classification.jpg\" alt=\"Image of a vegetable stand\"/>\n</div>\n\nPhoto by [Peter Wendt](https://unsplash.com/@peterwendt).\n\nWe can instruct the model to classify the image into one of the categories that we have:\n\n```py\n>>> categories = ['animals','vegetables', 'city landscape', 'cars', 'office']\n>>> prompt = [f\"Instruction: Classify the following image into a single category from the following list: {categories}.\\n\",\n...     \"https://images.unsplash.com/photo-1471193945509-9ad0617afabf?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3540&q=80\",    \n...     \"Category: \"\n... ]\n\n>>> inputs = processor(prompt, return_tensors=\"pt\").to(\"cuda\")\n>>> bad_words_ids = processor.tokenizer([\"<image>\", \"<fake_token_around_image>\"], add_special_tokens=False).input_ids\n\n>>> generated_ids = model.generate(**inputs, max_new_tokens=6, bad_words_ids=bad_words_ids)\n>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)\n>>> print(generated_text[0])\nInstruction: Classify the following image into a single category from the following list: ['animals', 'vegetables', 'city landscape', 'cars', 'office'].\nCategory: Vegetables\n```  \n\nIn the example above we instruct the model to classify the image into a single category, however, you can also prompt the model to do rank classification.\n\n## Image-guided text generation\n\nFor more creative applications, you can use image-guided text generation to generate text based on an image. This can be \nuseful to create descriptions of products, ads, descriptions of a scene, etc. \n\nLet's prompt IDEFICS to write a story based on a simple image of a red door: \n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-story-generation.jpg\" alt=\"Image of a red door with a pumpkin on the steps\"/>\n</div>\n\nPhoto by [Craig Tidball](https://unsplash.com/@devonshiremedia)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6fda5281-4693-4f63-9332-ec5f94eb56ad
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6fda5281-4693-4f63-9332-ec5f94eb56ad\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d0be5bbc-3da9-40b5-86cd-6e745b701b3d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3e60ba6acb9b94b0caa16c4dcf8fd1e857d79b0cb4554071b24b5fbe5185a8c9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"95361241-f724-462a-b6f0-1bbbccc4b3ca\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d23047bac1019360c1aeec82f0d48ac8cd687c761c4198ce78551b8b84152fd8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"36d6d607-3d26-4d35-9762-e16ebdab0351\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"76fcaeea0553411f5c081b1fc41752115607bd0eefda7001fc7ce7e42967a99c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13754, \"end_char_idx\": 16529, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "doc_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "ref_doc_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "chroma:document": "Category: Vegetables\n```  \n\nIn the example above we instruct the model to classify the image into a single category, however, you can also prompt the model to do rank classification.\n\n## Image-guided text generation\n\nFor more creative applications, you can use image-guided text generation to generate text based on an image. This can be \nuseful to create descriptions of products, ads, descriptions of a scene, etc. \n\nLet's prompt IDEFICS to write a story based on a simple image of a red door: \n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-story-generation.jpg\" alt=\"Image of a red door with a pumpkin on the steps\"/>\n</div>\n\nPhoto by [Craig Tidball](https://unsplash.com/@devonshiremedia).\n  \n```py\n>>> prompt = [\"Instruction: Use the image to write a story. \\n\",\n...     \"https://images.unsplash.com/photo-1517086822157-2b0358e7684a?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=2203&q=80\",\n...     \"Story: \\n\"]\n\n>>> inputs = processor(prompt, return_tensors=\"pt\").to(\"cuda\")\n>>> bad_words_ids = processor.tokenizer([\"<image>\", \"<fake_token_around_image>\"], add_special_tokens=False).input_ids\n\n>>> generated_ids = model.generate(**inputs, num_beams=2, max_new_tokens=200, bad_words_ids=bad_words_ids)\n>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)\n>>> print(generated_text[0]) \nInstruction: Use the image to write a story. \n Story: \nOnce upon a time, there was a little girl who lived in a house with a red door.  She loved her red door.  It was the prettiest door in the whole world.\n\nOne day, the little girl was playing in her yard when she noticed a man standing on her doorstep.  He was wearing a long black coat and a top hat.\n\nThe little girl ran inside and told her mother about the man.\n\nHer mother said, \u201cDon\u2019t worry, honey.  He\u2019s just a friendly ghost.\u201d\n\nThe little girl wasn\u2019t sure if she believed her mother, but she went outside anyway.\n\nWhen she got to the door, the man was gone.\n\nThe next day, the little girl was playing in her yard again when she noticed the man standing on her doorstep.\n\nHe was wearing a long black coat and a top hat.\n\nThe little girl ran\n```\n\nLooks like IDEFICS noticed the pumpkin on the doorstep and went with a spooky Halloween story about a ghost.\n\n<Tip>\n\nFor longer outputs like this, you will greatly benefit from tweaking the text generation strategy. This can help \nyou significantly improve the quality of the generated output. Check out [Text generation strategies](../generation_strategies) \nto learn more. \n</Tip>\n\n## Running inference in batch mode\n\nAll of the earlier sections illustrated IDEFICS for a single example."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 36d6d607-3d26-4d35-9762-e16ebdab0351
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"36d6d607-3d26-4d35-9762-e16ebdab0351\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d0be5bbc-3da9-40b5-86cd-6e745b701b3d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3e60ba6acb9b94b0caa16c4dcf8fd1e857d79b0cb4554071b24b5fbe5185a8c9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6fda5281-4693-4f63-9332-ec5f94eb56ad\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8ea45a9dd57cb1c0dc7350805d9fe4b41cf003e23f85455d49f30db6fc51861f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1a6afd39-6f53-46c6-b1c0-1d27f5d7c7fb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3caf6a68132596513471071bf1e0996a13ea2fc60dab38e30eb2d8189643694c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15282, \"end_char_idx\": 17893, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "doc_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "ref_doc_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "chroma:document": "Story: \nOnce upon a time, there was a little girl who lived in a house with a red door.  She loved her red door.  It was the prettiest door in the whole world.\n\nOne day, the little girl was playing in her yard when she noticed a man standing on her doorstep.  He was wearing a long black coat and a top hat.\n\nThe little girl ran inside and told her mother about the man.\n\nHer mother said, \u201cDon\u2019t worry, honey.  He\u2019s just a friendly ghost.\u201d\n\nThe little girl wasn\u2019t sure if she believed her mother, but she went outside anyway.\n\nWhen she got to the door, the man was gone.\n\nThe next day, the little girl was playing in her yard again when she noticed the man standing on her doorstep.\n\nHe was wearing a long black coat and a top hat.\n\nThe little girl ran\n```\n\nLooks like IDEFICS noticed the pumpkin on the doorstep and went with a spooky Halloween story about a ghost.\n\n<Tip>\n\nFor longer outputs like this, you will greatly benefit from tweaking the text generation strategy. This can help \nyou significantly improve the quality of the generated output. Check out [Text generation strategies](../generation_strategies) \nto learn more. \n</Tip>\n\n## Running inference in batch mode\n\nAll of the earlier sections illustrated IDEFICS for a single example. In a very similar fashion, you can run inference \nfor a batch of examples by passing a list of prompts:\n\n```py\n>>> prompts = [\n...     [   \"https://images.unsplash.com/photo-1543349689-9a4d426bee8e?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3501&q=80\",\n...         \"This is an image of \",\n...     ],\n...     [   \"https://images.unsplash.com/photo-1623944889288-cd147dbb517c?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3540&q=80\",\n...         \"This is an image of \",\n...     ],\n...     [   \"https://images.unsplash.com/photo-1471193945509-9ad0617afabf?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3540&q=80\",\n...         \"This is an image of \",\n...     ],\n... ]\n\n>>> inputs = processor(prompts, return_tensors=\"pt\").to(\"cuda\")\n>>> bad_words_ids = processor.tokenizer([\"<image>\", \"<fake_token_around_image>\"], add_special_tokens=False).input_ids\n\n>>> generated_ids = model.generate(**inputs, max_new_tokens=10, bad_words_ids=bad_words_ids)\n>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)\n>>> for i,t in enumerate(generated_text):\n...     print(f\"{i}:\\n{t}\\n\") \n0:\nThis is an image of the Eiffel Tower in Paris, France.\n\n1:\nThis is an image of a couple on a picnic blanket."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 1a6afd39-6f53-46c6-b1c0-1d27f5d7c7fb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1a6afd39-6f53-46c6-b1c0-1d27f5d7c7fb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d0be5bbc-3da9-40b5-86cd-6e745b701b3d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3e60ba6acb9b94b0caa16c4dcf8fd1e857d79b0cb4554071b24b5fbe5185a8c9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"36d6d607-3d26-4d35-9762-e16ebdab0351\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/idefics\", \"title\": \"Image tasks with IDEFICS\", \"tokens\": 5432, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c67678b28c19b6750e75084ff1a0c524c9cc087c53a18025d1d4555a08ae8cd2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 17841, \"end_char_idx\": 20184, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "doc_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "ref_doc_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "chroma:document": "1:\nThis is an image of a couple on a picnic blanket.\n\n2:\nThis is an image of a vegetable stand.\n```\n\n## IDEFICS instruct for conversational use\n\nFor conversational use cases, you can find fine-tuned instructed versions of the model on the \ud83e\udd17 Hub: \n`HuggingFaceM4/idefics-80b-instruct` and `HuggingFaceM4/idefics-9b-instruct`.\n\nThese checkpoints are the result of fine-tuning the respective base models on a mixture of supervised and instruction \nfine-tuning datasets, which boosts the downstream performance while making the models more usable in conversational settings.\n\nThe use and prompting for the conversational use is very similar to using the base models: \n\n```py\n>>> import torch\n>>> from transformers import IdeficsForVisionText2Text, AutoProcessor\n\n>>> device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n\n>>> checkpoint = \"HuggingFaceM4/idefics-9b-instruct\"\n>>> model = IdeficsForVisionText2Text.from_pretrained(checkpoint, torch_dtype=torch.bfloat16).to(device)\n>>> processor = AutoProcessor.from_pretrained(checkpoint)\n\n>>> prompts = [\n...     [\n...         \"User: What is in this image?\",\n...         \"https://upload.wikimedia.org/wikipedia/commons/8/86/Id%C3%A9fix.JPG\",\n...         \"<end_of_utterance>\",\n\n...         \"\\nAssistant: This picture depicts Idefix, the dog of Obelix in Asterix and Obelix. Idefix is running on the ground.<end_of_utterance>\",\n\n...         \"\\nUser:\",\n...         \"https://static.wikia.nocookie.net/asterix/images/2/25/R22b.gif/revision/latest?cb=20110815073052\",\n...         \"And who is that?<end_of_utterance>\",\n\n...         \"\\nAssistant:\",\n...     ],\n... ]\n\n>>> # --batched mode\n>>> inputs = processor(prompts, add_end_of_utterance_token=False, return_tensors=\"pt\").to(device)\n>>> # --single sample mode\n>>> # inputs = processor(prompts[0], return_tensors=\"pt\").to(device)\n\n>>> # Generation args\n>>> exit_condition = processor.tokenizer(\"<end_of_utterance>\", add_special_tokens=False).input_ids\n>>> bad_words_ids = processor.tokenizer([\"<image>\", \"<fake_token_around_image>\"], add_special_tokens=False).input_ids\n\n>>> generated_ids = model.generate(**inputs, eos_token_id=exit_condition, bad_words_ids=bad_words_ids, max_length=100)\n>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)\n>>> for i, t in enumerate(generated_text):\n...     print(f\"{i}:\\n{t}\\n\")\n```"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: c3607953-6210-4713-977d-434be26c14f8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c3607953-6210-4713-977d-434be26c14f8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/summarization\", \"title\": \"Summarization\", \"tokens\": 5008, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8b9ed6f8-100f-4853-af25-09cf928ce987\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/summarization\", \"title\": \"Summarization\", \"tokens\": 5008, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c3d648d8a928b2b8b0f3f62054d010ad49f16e95df5d6868f3d8266a0d71666e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"25c28460-c9a2-417f-95bd-963db38738a4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d69b9fbd8f133968faf60fad49c0cf076cb76dd8d11684fbd15c54d237a1639c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3520, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "doc_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "ref_doc_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "chroma:document": "# Summarization\n\n[[open-in-colab]]\n\n<Youtube id=\"yHnr5Dk2zCI\"/>\n\nSummarization creates a shorter version of a document or an article that captures all the important information. Along with translation, it is another example of a task that can be formulated as a sequence-to-sequence task. Summarization can be:\n\n- Extractive: extract the most relevant information from a document.\n- Abstractive: generate new text that captures the most relevant information.\n\nThis guide will show you how to:\n\n1. Finetune [T5](https://huggingface.co/google-t5/t5-small) on the California state bill subset of the [BillSum](https://huggingface.co/datasets/billsum) dataset for abstractive summarization.\n2. Use your finetuned model for inference.\n\n<Tip>\n\nTo see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/summarization)\n\n</Tip>\n\nBefore you begin, make sure you have all the necessary libraries installed:\n\n```bash\npip install transformers datasets evaluate rouge_score\n```\n\nWe encourage you to login to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to login:\n\n```py\n>>> from huggingface_hub import notebook_login\n\n>>> notebook_login()\n```\n\n## Load BillSum dataset\n\nStart by loading the smaller California state bill subset of the BillSum dataset from the \ud83e\udd17 Datasets library:\n\n```py\n>>> from datasets import load_dataset\n\n>>> billsum = load_dataset(\"billsum\", split=\"ca_test\")\n```\n\nSplit the dataset into a train and test set with the [`~datasets.Dataset.train_test_split`] method:\n\n```py\n>>> billsum = billsum.train_test_split(test_size=0.2)\n```\n\nThen take a look at an example:\n\n```py\n>>> billsum[\"train\"][0]\n{'summary': 'Existing law authorizes state agencies to enter into contracts for the acquisition of goods or services upon approval by the Department of General Services. Existing law sets forth various requirements and prohibitions for those contracts, including, but not limited to, a prohibition on entering into contracts for the acquisition of goods or services of $100,000 or more with a contractor that discriminates between spouses and domestic partners or same-sex and different-sex couples in the provision of benefits. Existing law provides that a contract entered into in violation of those requirements and prohibitions is void and authorizes the state or any person acting on behalf of the state to bring a civil action seeking a determination that a contract is in violation and therefore void. Under existing law, a willful violation of those requirements and prohibitions is a misdemeanor.\\nThis bill would also prohibit a state agency from entering into contracts for the acquisition of goods or services of $100,000 or more with a contractor that discriminates between employees on the basis of gender identity in the provision of benefits, as specified. By expanding the scope of a crime, this bill would impose a state-mandated local program.\\nThe California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.\\nThis bill would provide that no reimbursement is required by this act for a specified reason.',\n 'text': 'The people of the State of California do enact as follows:\\n\\n\\nSECTION 1.\\nSection 10295.35 is added to the Public Contract Code, to read:\\n10295.35.\\n(a) (1) Notwithstanding any other law,"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 25c28460-c9a2-417f-95bd-963db38738a4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"25c28460-c9a2-417f-95bd-963db38738a4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/summarization\", \"title\": \"Summarization\", \"tokens\": 5008, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8b9ed6f8-100f-4853-af25-09cf928ce987\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/summarization\", \"title\": \"Summarization\", \"tokens\": 5008, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c3d648d8a928b2b8b0f3f62054d010ad49f16e95df5d6868f3d8266a0d71666e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c3607953-6210-4713-977d-434be26c14f8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/summarization\", \"title\": \"Summarization\", \"tokens\": 5008, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c7158908fd4c06f7316c01779fa4d818caec8dbdd842f445f658e9a49991f34e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f4dc6a5e-7db8-46dc-bcf8-165cc2dbeeda\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c9ac8e7815f0574e35d09a6dd26f2828a1c4909300a12220195a21dade7bd7da\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1919, \"end_char_idx\": 5868, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "doc_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "ref_doc_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "chroma:document": "Existing law sets forth various requirements and prohibitions for those contracts, including, but not limited to, a prohibition on entering into contracts for the acquisition of goods or services of $100,000 or more with a contractor that discriminates between spouses and domestic partners or same-sex and different-sex couples in the provision of benefits. Existing law provides that a contract entered into in violation of those requirements and prohibitions is void and authorizes the state or any person acting on behalf of the state to bring a civil action seeking a determination that a contract is in violation and therefore void. Under existing law, a willful violation of those requirements and prohibitions is a misdemeanor.\\nThis bill would also prohibit a state agency from entering into contracts for the acquisition of goods or services of $100,000 or more with a contractor that discriminates between employees on the basis of gender identity in the provision of benefits, as specified. By expanding the scope of a crime, this bill would impose a state-mandated local program.\\nThe California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.\\nThis bill would provide that no reimbursement is required by this act for a specified reason.',\n 'text': 'The people of the State of California do enact as follows:\\n\\n\\nSECTION 1.\\nSection 10295.35 is added to the Public Contract Code, to read:\\n10295.35.\\n(a) (1) Notwithstanding any other law, a state agency shall not enter into any contract for the acquisition of goods or services in the amount of one hundred thousand dollars ($100,000) or more with a contractor that, in the provision of benefits, discriminates between employees on the basis of an employee\u2019s or dependent\u2019s actual or perceived gender identity, including, but not limited to, the employee\u2019s or dependent\u2019s identification as transgender.\\n(2) For purposes of this section, \u201ccontract\u201d includes contracts with a cumulative amount of one hundred thousand dollars ($100,000) or more per contractor in each fiscal year.\\n(3) For purposes of this section, an employee health plan is discriminatory if the plan is not consistent with Section 1365.5 of the Health and Safety Code and Section 10140 of the Insurance Code.\\n(4) The requirements of this section shall apply only to those portions of a contractor\u2019s operations that occur under any of the following conditions:\\n(A) Within the state.\\n(B) On real property outside the state if the property is owned by the state or if the state has a right to occupy the property, and if the contractor\u2019s presence at that location is connected to a contract with the state.\\n(C) Elsewhere in the United States where work related to a state contract is being performed.\\n(b) Contractors shall treat as confidential, to the maximum extent allowed by law or by the requirement of the contractor\u2019s insurance provider, any request by an employee or applicant for employment benefits or any documentation of eligibility for benefits submitted by an employee or applicant for employment.\\n(c) After taking all reasonable measures to find a contractor that complies with this section, as determined by the state agency, the requirements of this section may be waived under any of the following circumstances:\\n(1) There is only one prospective contractor willing to enter into a specific contract with the state agency.\\n(2) The contract is necessary to respond to an emergency, as determined by the state agency, that endangers the public health, welfare, or safety, or the contract is necessary for the provision of essential services, and no entity that complies with the requirements of this section capable of responding to the emergency is immediately available.\\n(3) The requirements of this section violate, or are inconsistent with,"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f4dc6a5e-7db8-46dc-bcf8-165cc2dbeeda
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f4dc6a5e-7db8-46dc-bcf8-165cc2dbeeda\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/summarization\", \"title\": \"Summarization\", \"tokens\": 5008, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8b9ed6f8-100f-4853-af25-09cf928ce987\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/summarization\", \"title\": \"Summarization\", \"tokens\": 5008, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c3d648d8a928b2b8b0f3f62054d010ad49f16e95df5d6868f3d8266a0d71666e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"25c28460-c9a2-417f-95bd-963db38738a4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/summarization\", \"title\": \"Summarization\", \"tokens\": 5008, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"92b25a146fdf7a84d567208b5a108d27dc64c3515bc360a477842cc7b18abf2f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"62a9bc2a-f3d8-46b2-b577-24c791606df5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d06e8648aeacf19c052162a6d4ea3dd77baecd5b96e99a2427822bf7e0b9c80b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3855, \"end_char_idx\": 7575, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "doc_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "ref_doc_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "chroma:document": "but not limited to, the employee\u2019s or dependent\u2019s identification as transgender.\\n(2) For purposes of this section, \u201ccontract\u201d includes contracts with a cumulative amount of one hundred thousand dollars ($100,000) or more per contractor in each fiscal year.\\n(3) For purposes of this section, an employee health plan is discriminatory if the plan is not consistent with Section 1365.5 of the Health and Safety Code and Section 10140 of the Insurance Code.\\n(4) The requirements of this section shall apply only to those portions of a contractor\u2019s operations that occur under any of the following conditions:\\n(A) Within the state.\\n(B) On real property outside the state if the property is owned by the state or if the state has a right to occupy the property, and if the contractor\u2019s presence at that location is connected to a contract with the state.\\n(C) Elsewhere in the United States where work related to a state contract is being performed.\\n(b) Contractors shall treat as confidential, to the maximum extent allowed by law or by the requirement of the contractor\u2019s insurance provider, any request by an employee or applicant for employment benefits or any documentation of eligibility for benefits submitted by an employee or applicant for employment.\\n(c) After taking all reasonable measures to find a contractor that complies with this section, as determined by the state agency, the requirements of this section may be waived under any of the following circumstances:\\n(1) There is only one prospective contractor willing to enter into a specific contract with the state agency.\\n(2) The contract is necessary to respond to an emergency, as determined by the state agency, that endangers the public health, welfare, or safety, or the contract is necessary for the provision of essential services, and no entity that complies with the requirements of this section capable of responding to the emergency is immediately available.\\n(3) The requirements of this section violate, or are inconsistent with, the terms or conditions of a grant, subvention, or agreement, if the agency has made a good faith attempt to change the terms or conditions of any grant, subvention, or agreement to authorize application of this section.\\n(4) The contractor is providing wholesale or bulk water, power, or natural gas, the conveyance or transmission of the same, or ancillary services, as required for ensuring reliable services in accordance with good utility practice, if the purchase of the same cannot practically be accomplished through the standard competitive bidding procedures and the contractor is not providing direct retail services to end users.\\n(d) (1) A contractor shall not be deemed to discriminate in the provision of benefits if the contractor, in providing the benefits, pays the actual costs incurred in obtaining the benefit.\\n(2) If a contractor is unable to provide a certain benefit, despite taking reasonable measures to do so, the contractor shall not be deemed to discriminate in the provision of benefits.\\n(e) (1) Every contract subject to this chapter shall contain a statement by which the contractor certifies that the contractor is in compliance with this section.\\n(2) The department or other contracting agency shall enforce this section pursuant to its existing enforcement powers.\\n(3) (A) If a contractor falsely certifies that it is in compliance with this section, the contract with that contractor shall be subject to Article 9 (commencing with Section 10420), unless, within a time period specified by the department or other contracting agency, the contractor provides to the department or agency proof that it has complied, or is in the process of complying, with this section."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 62a9bc2a-f3d8-46b2-b577-24c791606df5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"62a9bc2a-f3d8-46b2-b577-24c791606df5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/summarization\", \"title\": \"Summarization\", \"tokens\": 5008, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8b9ed6f8-100f-4853-af25-09cf928ce987\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/summarization\", \"title\": \"Summarization\", \"tokens\": 5008, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c3d648d8a928b2b8b0f3f62054d010ad49f16e95df5d6868f3d8266a0d71666e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f4dc6a5e-7db8-46dc-bcf8-165cc2dbeeda\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/summarization\", \"title\": \"Summarization\", \"tokens\": 5008, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"915f288c389674783d90f69bae2a86f2b199aadf4cb9a2869cf4103229d01776\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"108e24d3-3364-42a2-b344-b54672ff6863\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e184f27f91ed33251037adbf26dbdde4ba8d1122c0f70d760ea4c9dc986bdfb1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5541, \"end_char_idx\": 9360, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "doc_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "ref_doc_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "chroma:document": "that endangers the public health, welfare, or safety, or the contract is necessary for the provision of essential services, and no entity that complies with the requirements of this section capable of responding to the emergency is immediately available.\\n(3) The requirements of this section violate, or are inconsistent with, the terms or conditions of a grant, subvention, or agreement, if the agency has made a good faith attempt to change the terms or conditions of any grant, subvention, or agreement to authorize application of this section.\\n(4) The contractor is providing wholesale or bulk water, power, or natural gas, the conveyance or transmission of the same, or ancillary services, as required for ensuring reliable services in accordance with good utility practice, if the purchase of the same cannot practically be accomplished through the standard competitive bidding procedures and the contractor is not providing direct retail services to end users.\\n(d) (1) A contractor shall not be deemed to discriminate in the provision of benefits if the contractor, in providing the benefits, pays the actual costs incurred in obtaining the benefit.\\n(2) If a contractor is unable to provide a certain benefit, despite taking reasonable measures to do so, the contractor shall not be deemed to discriminate in the provision of benefits.\\n(e) (1) Every contract subject to this chapter shall contain a statement by which the contractor certifies that the contractor is in compliance with this section.\\n(2) The department or other contracting agency shall enforce this section pursuant to its existing enforcement powers.\\n(3) (A) If a contractor falsely certifies that it is in compliance with this section, the contract with that contractor shall be subject to Article 9 (commencing with Section 10420), unless, within a time period specified by the department or other contracting agency, the contractor provides to the department or agency proof that it has complied, or is in the process of complying, with this section.\\n(B) The application of the remedies or penalties contained in Article 9 (commencing with Section 10420) to a contract subject to this chapter shall not preclude the application of any existing remedies otherwise available to the department or other contracting agency under its existing enforcement powers.\\n(f) Nothing in this section is intended to regulate the contracting practices of any local jurisdiction.\\n(g) This section shall be construed so as not to conflict with applicable federal laws, rules, or regulations. In the event that a court or agency of competent jurisdiction holds that federal law, rule, or regulation invalidates any clause, sentence, paragraph, or section of this code or the application thereof to any person or circumstances, it is the intent of the state that the court or agency sever that clause, sentence, paragraph, or section so that the remainder of this section shall remain in effect.\\nSEC. 2.\\nSection 10295.35 of the Public Contract Code shall not be construed to create any new enforcement authority or responsibility in the Department of General Services or any other contracting agency.\\nSEC. 3.\\nNo reimbursement is required by this act pursuant to Section 6 of Article XIII\\u2009B of the California Constitution because the only costs that may be incurred by a local agency or school district will be incurred because this act creates a new crime or infraction, eliminates a crime or infraction, or changes the penalty for a crime or infraction, within the meaning of Section 17556 of the Government Code, or changes the definition of a crime within the meaning of Section 6 of Article XIII\\u2009B of the California Constitution.',\n 'title': 'An act to add Section 10295.35 to the Public Contract Code, relating to public contracts.'}"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 108e24d3-3364-42a2-b344-b54672ff6863
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"108e24d3-3364-42a2-b344-b54672ff6863\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/summarization\", \"title\": \"Summarization\", \"tokens\": 5008, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8b9ed6f8-100f-4853-af25-09cf928ce987\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/summarization\", \"title\": \"Summarization\", \"tokens\": 5008, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c3d648d8a928b2b8b0f3f62054d010ad49f16e95df5d6868f3d8266a0d71666e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"62a9bc2a-f3d8-46b2-b577-24c791606df5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/summarization\", \"title\": \"Summarization\", \"tokens\": 5008, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cc5df8dfdb3f11e8041b33b70a4d4429c81cf1fc345eea286c0d315a6c41b24b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a79ab83f-ba93-4d89-ac75-06fe51fe7cee\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bdd516d048cb095cdf53b9aa952df7a6cbe211995fb48bd543965af445200623\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7442, \"end_char_idx\": 10203, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "doc_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "ref_doc_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "chroma:document": "the contractor provides to the department or agency proof that it has complied, or is in the process of complying, with this section.\\n(B) The application of the remedies or penalties contained in Article 9 (commencing with Section 10420) to a contract subject to this chapter shall not preclude the application of any existing remedies otherwise available to the department or other contracting agency under its existing enforcement powers.\\n(f) Nothing in this section is intended to regulate the contracting practices of any local jurisdiction.\\n(g) This section shall be construed so as not to conflict with applicable federal laws, rules, or regulations. In the event that a court or agency of competent jurisdiction holds that federal law, rule, or regulation invalidates any clause, sentence, paragraph, or section of this code or the application thereof to any person or circumstances, it is the intent of the state that the court or agency sever that clause, sentence, paragraph, or section so that the remainder of this section shall remain in effect.\\nSEC. 2.\\nSection 10295.35 of the Public Contract Code shall not be construed to create any new enforcement authority or responsibility in the Department of General Services or any other contracting agency.\\nSEC. 3.\\nNo reimbursement is required by this act pursuant to Section 6 of Article XIII\\u2009B of the California Constitution because the only costs that may be incurred by a local agency or school district will be incurred because this act creates a new crime or infraction, eliminates a crime or infraction, or changes the penalty for a crime or infraction, within the meaning of Section 17556 of the Government Code, or changes the definition of a crime within the meaning of Section 6 of Article XIII\\u2009B of the California Constitution.',\n 'title': 'An act to add Section 10295.35 to the Public Contract Code, relating to public contracts.'}\n```\n\nThere are two fields that you'll want to use:\n\n- `text`: the text of the bill which'll be the input to the model.\n- `summary`: a condensed version of `text` which'll be the model target.\n\n## Preprocess\n\nThe next step is to load a T5 tokenizer to process `text` and `summary`:\n\n```py\n>>> from transformers import AutoTokenizer\n\n>>> checkpoint = \"google-t5/t5-small\"\n>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)\n```\n\nThe preprocessing function you want to create needs to:\n\n1. Prefix the input with a prompt so T5 knows this is a summarization task. Some models capable of multiple NLP tasks require prompting for specific tasks.\n2. Use the keyword `text_target` argument when tokenizing labels.\n3. Truncate sequences to be no longer than the maximum length set by the `max_length` parameter.\n\n```py\n>>> prefix = \"summarize: \""}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a79ab83f-ba93-4d89-ac75-06fe51fe7cee
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a79ab83f-ba93-4d89-ac75-06fe51fe7cee\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/summarization\", \"title\": \"Summarization\", \"tokens\": 5008, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8b9ed6f8-100f-4853-af25-09cf928ce987\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/summarization\", \"title\": \"Summarization\", \"tokens\": 5008, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c3d648d8a928b2b8b0f3f62054d010ad49f16e95df5d6868f3d8266a0d71666e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"108e24d3-3364-42a2-b344-b54672ff6863\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/summarization\", \"title\": \"Summarization\", \"tokens\": 5008, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"dd0dda660f36abd526a9a23cfe6282da7d88765cd50f136883e97c27ed26c46d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6ae5f4c4-7b08-4996-be0c-7b354a61deb8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c67ea74fee14e2d3caffe00d7a6ea3311b1f5f8c114cbef7d0fef2f8de4a0fe6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8510, \"end_char_idx\": 12162, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "doc_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "ref_doc_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "chroma:document": "2.\\nSection 10295.35 of the Public Contract Code shall not be construed to create any new enforcement authority or responsibility in the Department of General Services or any other contracting agency.\\nSEC. 3.\\nNo reimbursement is required by this act pursuant to Section 6 of Article XIII\\u2009B of the California Constitution because the only costs that may be incurred by a local agency or school district will be incurred because this act creates a new crime or infraction, eliminates a crime or infraction, or changes the penalty for a crime or infraction, within the meaning of Section 17556 of the Government Code, or changes the definition of a crime within the meaning of Section 6 of Article XIII\\u2009B of the California Constitution.',\n 'title': 'An act to add Section 10295.35 to the Public Contract Code, relating to public contracts.'}\n```\n\nThere are two fields that you'll want to use:\n\n- `text`: the text of the bill which'll be the input to the model.\n- `summary`: a condensed version of `text` which'll be the model target.\n\n## Preprocess\n\nThe next step is to load a T5 tokenizer to process `text` and `summary`:\n\n```py\n>>> from transformers import AutoTokenizer\n\n>>> checkpoint = \"google-t5/t5-small\"\n>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)\n```\n\nThe preprocessing function you want to create needs to:\n\n1. Prefix the input with a prompt so T5 knows this is a summarization task. Some models capable of multiple NLP tasks require prompting for specific tasks.\n2. Use the keyword `text_target` argument when tokenizing labels.\n3. Truncate sequences to be no longer than the maximum length set by the `max_length` parameter.\n\n```py\n>>> prefix = \"summarize: \"\n\n\n>>> def preprocess_function(examples):\n...     inputs = [prefix + doc for doc in examples[\"text\"]]\n...     model_inputs = tokenizer(inputs, max_length=1024, truncation=True)\n\n...     labels = tokenizer(text_target=examples[\"summary\"], max_length=128, truncation=True)\n\n...     model_inputs[\"labels\"] = labels[\"input_ids\"]\n...     return model_inputs\n```\n\nTo apply the preprocessing function over the entire dataset, use \ud83e\udd17 Datasets [`~datasets.Dataset.map`] method. You can speed up the `map` function by setting `batched=True` to process multiple elements of the dataset at once:\n\n```py\n>>> tokenized_billsum = billsum.map(preprocess_function, batched=True)\n```\n\nNow create a batch of examples using [`DataCollatorForSeq2Seq`]. It's more efficient to *dynamically pad* the sentences to the longest length in a batch during collation, instead of padding the whole dataset to the maximum length.\n\n<frameworkcontent>\n<pt>\n\n```py\n>>> from transformers import DataCollatorForSeq2Seq\n\n>>> data_collator = DataCollatorForSeq2Seq(tokenizer=tokenizer, model=checkpoint)\n```\n</pt>\n<tf>\n\n```py\n>>> from transformers import DataCollatorForSeq2Seq\n\n>>> data_collator = DataCollatorForSeq2Seq(tokenizer=tokenizer, model=checkpoint, return_tensors=\"tf\")\n```\n</tf>\n</frameworkcontent>\n\n## Evaluate\n\nIncluding a metric during training is often helpful for evaluating your model's performance. You can quickly load a evaluation method with the \ud83e\udd17 [Evaluate](https://huggingface.co/docs/evaluate/index) library. For this task, load the [ROUGE](https://huggingface.co/spaces/evaluate-metric/rouge) metric (see the \ud83e\udd17 Evaluate [quick tour](https://huggingface.co/docs/evaluate/a_quick_tour) to learn more about how to load and compute a metric):\n\n```py\n>>> import evaluate\n\n>>> rouge = evaluate.load(\"rouge\")\n```\n\nThen create a function that passes your predictions and labels to [`~evaluate.EvaluationModule.compute`] to calculate the ROUGE metric:\n\n```py\n>>> import numpy as np"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6ae5f4c4-7b08-4996-be0c-7b354a61deb8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6ae5f4c4-7b08-4996-be0c-7b354a61deb8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/summarization\", \"title\": \"Summarization\", \"tokens\": 5008, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8b9ed6f8-100f-4853-af25-09cf928ce987\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/summarization\", \"title\": \"Summarization\", \"tokens\": 5008, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c3d648d8a928b2b8b0f3f62054d010ad49f16e95df5d6868f3d8266a0d71666e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a79ab83f-ba93-4d89-ac75-06fe51fe7cee\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/summarization\", \"title\": \"Summarization\", \"tokens\": 5008, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a9bcc77ad663fe4b59591c2ec2f105bfa391d78dd501d87a6fc882956688feae\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7b798ecf-ee45-4623-83ec-d0a104df1619\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1fc095916afa675f55c71c898912f5e8a12a05820a51c9f84f0ce208ff8be328\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12165, \"end_char_idx\": 15070, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "doc_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "ref_doc_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "chroma:document": ">>> def compute_metrics(eval_pred):\n...     predictions, labels = eval_pred\n...     decoded_preds = tokenizer.batch_decode(predictions, skip_special_tokens=True)\n...     labels = np.where(labels != -100, labels, tokenizer.pad_token_id)\n...     decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)\n\n...     result = rouge.compute(predictions=decoded_preds, references=decoded_labels, use_stemmer=True)\n\n...     prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in predictions]\n...     result[\"gen_len\"] = np.mean(prediction_lens)\n\n...     return {k: round(v, 4) for k, v in result.items()}\n```\n\nYour `compute_metrics` function is ready to go now, and you'll return to it when you setup your training.\n\n## Train\n\n<frameworkcontent>\n<pt>\n<Tip>\n\nIf you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!\n\n</Tip>\n\nYou're ready to start training your model now! Load T5 with [`AutoModelForSeq2SeqLM`]:\n\n```py\n>>> from transformers import AutoModelForSeq2SeqLM, Seq2SeqTrainingArguments, Seq2SeqTrainer\n\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint)\n```\n\nAt this point, only three steps remain:\n\n1. Define your training hyperparameters in [`Seq2SeqTrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the ROUGE metric and save the training checkpoint.\n2. Pass the training arguments to [`Seq2SeqTrainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.\n3. Call [`~Trainer.train`] to finetune your model.\n\n```py\n>>> training_args = Seq2SeqTrainingArguments(\n...     output_dir=\"my_awesome_billsum_model\",\n...     eval_strategy=\"epoch\",\n...     learning_rate=2e-5,\n...     per_device_train_batch_size=16,\n...     per_device_eval_batch_size=16,\n...     weight_decay=0.01,\n...     save_total_limit=3,\n...     num_train_epochs=4,\n...     predict_with_generate=True,\n...     fp16=True,\n...     push_to_hub=True,\n... )\n\n>>> trainer = Seq2SeqTrainer(\n...     model=model,\n...     args=training_args,\n...     train_dataset=tokenized_billsum[\"train\"],\n...     eval_dataset=tokenized_billsum[\"test\"],\n...     tokenizer=tokenizer,\n...     data_collator=data_collator,\n...     compute_metrics=compute_metrics,\n... )\n\n>>> trainer.train()\n```\n\nOnce training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:\n\n```py\n>>> trainer.push_to_hub()\n```\n</pt>\n<tf>\n<Tip>\n\nIf you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 7b798ecf-ee45-4623-83ec-d0a104df1619
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7b798ecf-ee45-4623-83ec-d0a104df1619\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/summarization\", \"title\": \"Summarization\", \"tokens\": 5008, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8b9ed6f8-100f-4853-af25-09cf928ce987\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/summarization\", \"title\": \"Summarization\", \"tokens\": 5008, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c3d648d8a928b2b8b0f3f62054d010ad49f16e95df5d6868f3d8266a0d71666e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6ae5f4c4-7b08-4996-be0c-7b354a61deb8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/summarization\", \"title\": \"Summarization\", \"tokens\": 5008, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7cfbbe55fa36f9f4290779c9e7e000f9cd289982a015cbcd362139e0c9df294a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"682ae8b1-c9fc-4879-ab43-2c537c8894db\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"76f7488be7188048a3312e3b809ec8bb3c8e382229592e80113100cfe3b302de\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13570, \"end_char_idx\": 16590, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "doc_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "ref_doc_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "chroma:document": "You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the ROUGE metric and save the training checkpoint.\n2. Pass the training arguments to [`Seq2SeqTrainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.\n3. Call [`~Trainer.train`] to finetune your model.\n\n```py\n>>> training_args = Seq2SeqTrainingArguments(\n...     output_dir=\"my_awesome_billsum_model\",\n...     eval_strategy=\"epoch\",\n...     learning_rate=2e-5,\n...     per_device_train_batch_size=16,\n...     per_device_eval_batch_size=16,\n...     weight_decay=0.01,\n...     save_total_limit=3,\n...     num_train_epochs=4,\n...     predict_with_generate=True,\n...     fp16=True,\n...     push_to_hub=True,\n... )\n\n>>> trainer = Seq2SeqTrainer(\n...     model=model,\n...     args=training_args,\n...     train_dataset=tokenized_billsum[\"train\"],\n...     eval_dataset=tokenized_billsum[\"test\"],\n...     tokenizer=tokenizer,\n...     data_collator=data_collator,\n...     compute_metrics=compute_metrics,\n... )\n\n>>> trainer.train()\n```\n\nOnce training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:\n\n```py\n>>> trainer.push_to_hub()\n```\n</pt>\n<tf>\n<Tip>\n\nIf you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!\n\n</Tip>\nTo finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:\n\n```py\n>>> from transformers import create_optimizer, AdamWeightDecay\n\n>>> optimizer = AdamWeightDecay(learning_rate=2e-5, weight_decay_rate=0.01)\n```\n\nThen you can load T5 with [`TFAutoModelForSeq2SeqLM`]:\n\n```py\n>>> from transformers import TFAutoModelForSeq2SeqLM\n\n>>> model = TFAutoModelForSeq2SeqLM.from_pretrained(checkpoint)\n```\n\nConvert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:\n\n```py\n>>> tf_train_set = model.prepare_tf_dataset(\n...     tokenized_billsum[\"train\"],\n...     shuffle=True,\n...     batch_size=16,\n...     collate_fn=data_collator,\n... )\n\n>>> tf_test_set = model.prepare_tf_dataset(\n...     tokenized_billsum[\"test\"],\n...     shuffle=False,\n...     batch_size=16,\n...     collate_fn=data_collator,\n... )\n```\n\nConfigure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method). Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:\n\n```py\n>>> import tensorflow as tf\n\n>>> model.compile(optimizer=optimizer)  # No loss argument!\n```\n\nThe last two things to setup before you start training is to compute the ROUGE score from the predictions, and provide a way to push your model to the Hub. Both are done by using [Keras callbacks](../main_classes/keras_callbacks)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 682ae8b1-c9fc-4879-ab43-2c537c8894db
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"682ae8b1-c9fc-4879-ab43-2c537c8894db\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/summarization\", \"title\": \"Summarization\", \"tokens\": 5008, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8b9ed6f8-100f-4853-af25-09cf928ce987\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/summarization\", \"title\": \"Summarization\", \"tokens\": 5008, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c3d648d8a928b2b8b0f3f62054d010ad49f16e95df5d6868f3d8266a0d71666e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7b798ecf-ee45-4623-83ec-d0a104df1619\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/summarization\", \"title\": \"Summarization\", \"tokens\": 5008, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9e018253aeb8aa9f16bce63257b76d4b5819840f2864aa3e4a37b1a7ca8198e3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4c3865de-2a35-4b27-a657-b422679485cb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"063f3f076384dff4a8b28e20d5c36e2d8758bfd57f5946327c7d3786d17d860d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15072, \"end_char_idx\": 18303, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "doc_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "ref_doc_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "chroma:document": "</Tip>\nTo finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:\n\n```py\n>>> from transformers import create_optimizer, AdamWeightDecay\n\n>>> optimizer = AdamWeightDecay(learning_rate=2e-5, weight_decay_rate=0.01)\n```\n\nThen you can load T5 with [`TFAutoModelForSeq2SeqLM`]:\n\n```py\n>>> from transformers import TFAutoModelForSeq2SeqLM\n\n>>> model = TFAutoModelForSeq2SeqLM.from_pretrained(checkpoint)\n```\n\nConvert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:\n\n```py\n>>> tf_train_set = model.prepare_tf_dataset(\n...     tokenized_billsum[\"train\"],\n...     shuffle=True,\n...     batch_size=16,\n...     collate_fn=data_collator,\n... )\n\n>>> tf_test_set = model.prepare_tf_dataset(\n...     tokenized_billsum[\"test\"],\n...     shuffle=False,\n...     batch_size=16,\n...     collate_fn=data_collator,\n... )\n```\n\nConfigure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method). Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:\n\n```py\n>>> import tensorflow as tf\n\n>>> model.compile(optimizer=optimizer)  # No loss argument!\n```\n\nThe last two things to setup before you start training is to compute the ROUGE score from the predictions, and provide a way to push your model to the Hub. Both are done by using [Keras callbacks](../main_classes/keras_callbacks).\n\nPass your `compute_metrics` function to [`~transformers.KerasMetricCallback`]:\n\n```py\n>>> from transformers.keras_callbacks import KerasMetricCallback\n\n>>> metric_callback = KerasMetricCallback(metric_fn=compute_metrics, eval_dataset=tf_validation_set)\n```\n\nSpecify where to push your model and tokenizer in the [`~transformers.PushToHubCallback`]:\n\n```py\n>>> from transformers.keras_callbacks import PushToHubCallback\n\n>>> push_to_hub_callback = PushToHubCallback(\n...     output_dir=\"my_awesome_billsum_model\",\n...     tokenizer=tokenizer,\n... )\n```\n\nThen bundle your callbacks together:\n\n```py\n>>> callbacks = [metric_callback, push_to_hub_callback]\n```\n\nFinally, you're ready to start training your model! Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callbacks to finetune the model:\n\n```py\n>>> model.fit(x=tf_train_set, validation_data=tf_test_set, epochs=3, callbacks=callbacks)\n```\n\nOnce training is completed, your model is automatically uploaded to the Hub so everyone can use it!\n</tf>\n</frameworkcontent>\n\n<Tip>\n\nFor a more in-depth example of how to finetune a model for summarization, take a look at the corresponding\n[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization.ipynb)\nor [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization-tf.ipynb).\n\n</Tip>\n\n## Inference\n\nGreat, now that you've finetuned a model, you can use it for inference!\n\nCome up with some text you'd like to summarize. For T5, you need to prefix your input depending on the task you're working on."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 4c3865de-2a35-4b27-a657-b422679485cb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4c3865de-2a35-4b27-a657-b422679485cb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/summarization\", \"title\": \"Summarization\", \"tokens\": 5008, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8b9ed6f8-100f-4853-af25-09cf928ce987\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/summarization\", \"title\": \"Summarization\", \"tokens\": 5008, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c3d648d8a928b2b8b0f3f62054d010ad49f16e95df5d6868f3d8266a0d71666e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"682ae8b1-c9fc-4879-ab43-2c537c8894db\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/summarization\", \"title\": \"Summarization\", \"tokens\": 5008, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"74a71bb1dedafba0f2f7cc50a5826c97194ab0fac0e06be36528742aaf47aa71\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"423b4164-731a-48da-9bec-979f3e4e010f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"16dcb01e3c780f50e688b261f9a493f76dbd2d759ed0de3a5afbbca6c2c676fe\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 17302, \"end_char_idx\": 19550, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "doc_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "ref_doc_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "chroma:document": "Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callbacks to finetune the model:\n\n```py\n>>> model.fit(x=tf_train_set, validation_data=tf_test_set, epochs=3, callbacks=callbacks)\n```\n\nOnce training is completed, your model is automatically uploaded to the Hub so everyone can use it!\n</tf>\n</frameworkcontent>\n\n<Tip>\n\nFor a more in-depth example of how to finetune a model for summarization, take a look at the corresponding\n[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization.ipynb)\nor [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization-tf.ipynb).\n\n</Tip>\n\n## Inference\n\nGreat, now that you've finetuned a model, you can use it for inference!\n\nCome up with some text you'd like to summarize. For T5, you need to prefix your input depending on the task you're working on. For summarization you should prefix your input as shown below:\n\n```py\n>>> text = \"summarize: The Inflation Reduction Act lowers prescription drug costs, health care costs, and energy costs. It's the most aggressive action on tackling the climate crisis in American history, which will lift up American workers and create good-paying, union jobs across the country. It'll lower the deficit and ask the ultra-wealthy and corporations to pay their fair share. And no one making under $400,000 per year will pay a penny more in taxes.\"\n```\n\nThe simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for summarization with your model, and pass your text to it:\n\n```py\n>>> from transformers import pipeline\n\n>>> summarizer = pipeline(\"summarization\", model=\"stevhliu/my_awesome_billsum_model\")\n>>> summarizer(text)\n[{\"summary_text\": \"The Inflation Reduction Act lowers prescription drug costs, health care costs, and energy costs. It's the most aggressive action on tackling the climate crisis in American history, which will lift up American workers and create good-paying, union jobs across the country.\"}]\n```\n\nYou can also manually replicate the results of the `pipeline` if you'd like:"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 423b4164-731a-48da-9bec-979f3e4e010f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"423b4164-731a-48da-9bec-979f3e4e010f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/summarization\", \"title\": \"Summarization\", \"tokens\": 5008, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8b9ed6f8-100f-4853-af25-09cf928ce987\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/summarization\", \"title\": \"Summarization\", \"tokens\": 5008, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c3d648d8a928b2b8b0f3f62054d010ad49f16e95df5d6868f3d8266a0d71666e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4c3865de-2a35-4b27-a657-b422679485cb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/summarization\", \"title\": \"Summarization\", \"tokens\": 5008, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"914e98619a7325698aadb8469f32e55408e0f3f687eb31c221fde0bca0549fad\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 18082, \"end_char_idx\": 21890, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "doc_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "ref_doc_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "chroma:document": "</Tip>\n\n## Inference\n\nGreat, now that you've finetuned a model, you can use it for inference!\n\nCome up with some text you'd like to summarize. For T5, you need to prefix your input depending on the task you're working on. For summarization you should prefix your input as shown below:\n\n```py\n>>> text = \"summarize: The Inflation Reduction Act lowers prescription drug costs, health care costs, and energy costs. It's the most aggressive action on tackling the climate crisis in American history, which will lift up American workers and create good-paying, union jobs across the country. It'll lower the deficit and ask the ultra-wealthy and corporations to pay their fair share. And no one making under $400,000 per year will pay a penny more in taxes.\"\n```\n\nThe simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for summarization with your model, and pass your text to it:\n\n```py\n>>> from transformers import pipeline\n\n>>> summarizer = pipeline(\"summarization\", model=\"stevhliu/my_awesome_billsum_model\")\n>>> summarizer(text)\n[{\"summary_text\": \"The Inflation Reduction Act lowers prescription drug costs, health care costs, and energy costs. It's the most aggressive action on tackling the climate crisis in American history, which will lift up American workers and create good-paying, union jobs across the country.\"}]\n```\n\nYou can also manually replicate the results of the `pipeline` if you'd like:\n\n\n<frameworkcontent>\n<pt>\nTokenize the text and return the `input_ids` as PyTorch tensors:\n\n```py\n>>> from transformers import AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"stevhliu/my_awesome_billsum_model\")\n>>> inputs = tokenizer(text, return_tensors=\"pt\").input_ids\n```\n\nUse the [`~generation.GenerationMixin.generate`] method to create the summarization. For more details about the different text generation strategies and parameters for controlling generation, check out the [Text Generation](../main_classes/text_generation) API.\n\n```py\n>>> from transformers import AutoModelForSeq2SeqLM\n\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"stevhliu/my_awesome_billsum_model\")\n>>> outputs = model.generate(inputs, max_new_tokens=100, do_sample=False)\n```\n\nDecode the generated token ids back into text:\n\n```py\n>>> tokenizer.decode(outputs[0], skip_special_tokens=True)\n'the inflation reduction act lowers prescription drug costs, health care costs, and energy costs. it's the most aggressive action on tackling the climate crisis in american history. it will ask the ultra-wealthy and corporations to pay their fair share.'\n```\n</pt>\n<tf>\nTokenize the text and return the `input_ids` as TensorFlow tensors:\n\n```py\n>>> from transformers import AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"stevhliu/my_awesome_billsum_model\")\n>>> inputs = tokenizer(text, return_tensors=\"tf\").input_ids\n```\n\nUse the [`~transformers.generation_tf_utils.TFGenerationMixin.generate`] method to create the summarization. For more details about the different text generation strategies and parameters for controlling generation, check out the [Text Generation](../main_classes/text_generation) API.\n\n```py\n>>> from transformers import TFAutoModelForSeq2SeqLM\n\n>>> model = TFAutoModelForSeq2SeqLM.from_pretrained(\"stevhliu/my_awesome_billsum_model\")\n>>> outputs = model.generate(inputs, max_new_tokens=100, do_sample=False)\n```\n\nDecode the generated token ids back into text:\n\n```py\n>>> tokenizer.decode(outputs[0], skip_special_tokens=True)\n'the inflation reduction act lowers prescription drug costs, health care costs, and energy costs. it's the most aggressive action on tackling the climate crisis in american history. it will ask the ultra-wealthy and corporations to pay their fair share.'\n```\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 65891829-a1aa-4a50-bcc6-f9a3e35da8e6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/mask_generation", "title": "Mask Generation", "tokens": 2117, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"65891829-a1aa-4a50-bcc6-f9a3e35da8e6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/mask_generation\", \"title\": \"Mask Generation\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d17f7d20-b435-4559-8f4c-b31a07de6ee9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/mask_generation\", \"title\": \"Mask Generation\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"385f9b234ba753f61a1095d2d3cde2181f866dca68ccaf42a2dcf54ca0fb65b2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a67a987a-bddb-4978-99cf-136202525203\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6ec150254f33a00b4c0197617151c016cd3de138c6dfe16f684383b42f7ecf07\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3002, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d17f7d20-b435-4559-8f4c-b31a07de6ee9", "doc_id": "d17f7d20-b435-4559-8f4c-b31a07de6ee9", "ref_doc_id": "d17f7d20-b435-4559-8f4c-b31a07de6ee9", "chroma:document": "# Mask Generation\n\nMask generation is the task of generating semantically meaningful masks for an image. \nThis task is very similar to [image segmentation](semantic_segmentation), but many differences exist. Image segmentation models are trained on labeled datasets and are limited to the classes they have seen during training; they return a set of masks and corresponding classes, given an image. \n\nMask generation models are trained on large amounts of data and operate in two modes. \n- Prompting mode: In this mode, the model takes in an image and a prompt, where a prompt can be a 2D point location (XY coordinates) in the image within an object or a bounding box surrounding an object. In prompting mode, the model only returns the mask over the object \nthat the prompt is pointing out. \n- Segment Everything mode: In segment everything, given an image, the model generates every mask in the image. To do so, a grid of points is generated and overlaid on the image for inference. \n\nMask generation task is supported by [Segment Anything Model (SAM)](model_doc/sam). It's a powerful model that consists of a Vision Transformer-based image encoder, a prompt encoder, and a two-way transformer mask decoder. Images and prompts are encoded, and the decoder takes these embeddings and generates valid masks. \n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/sam.png\" alt=\"SAM Architecture\"/>\n</div>\n\nSAM serves as a powerful foundation model for segmentation as it has large data coverage. It is trained on \n[SA-1B](https://ai.meta.com/datasets/segment-anything/), a dataset with 1 million images and 1.1 billion masks. \n\nIn this guide, you will learn how to:\n- Infer in segment everything mode with batching,\n- Infer in point prompting mode,\n- Infer in box prompting mode.\n\nFirst, let's install `transformers`:\n\n```bash\npip install -q transformers\n```\n\n## Mask Generation Pipeline\n\nThe easiest way to infer mask generation models is to use the `mask-generation` pipeline.\n\n```python\n>>> from transformers import pipeline\n\n>>> checkpoint = \"facebook/sam-vit-base\"\n>>> mask_generator = pipeline(model=checkpoint, task=\"mask-generation\")\n```\n\nLet's see the image.\n\n```python\nfrom PIL import Image\nimport requests\n\nimg_url = \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg\"\nimage = Image.open(requests.get(img_url, stream=True).raw).convert(\"RGB\")\n```\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg\" alt=\"Example Image\"/>\n</div>\n\nLet's segment everything. `points-per-batch` enables parallel inference of points in segment everything mode. This enables faster inference, but consumes more memory. Moreover, SAM only enables batching over points and not the images. `pred_iou_thresh` is the IoU confidence threshold where only the masks above that certain threshold are returned."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a67a987a-bddb-4978-99cf-136202525203
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/mask_generation", "title": "Mask Generation", "tokens": 2117, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a67a987a-bddb-4978-99cf-136202525203\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/mask_generation\", \"title\": \"Mask Generation\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d17f7d20-b435-4559-8f4c-b31a07de6ee9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/mask_generation\", \"title\": \"Mask Generation\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"385f9b234ba753f61a1095d2d3cde2181f866dca68ccaf42a2dcf54ca0fb65b2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"65891829-a1aa-4a50-bcc6-f9a3e35da8e6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/mask_generation\", \"title\": \"Mask Generation\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1d4c7c8d31efdaf6fbe969e46d7b92aceb94ca80b9b1f8fdd25159f8178c7eaf\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a7c8ee15-04e4-47f1-9da6-28769de1f85d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c099248979b7f82b2d3fc917922ca409f250c854bb01029223269d4b40c174c5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1311, \"end_char_idx\": 4262, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d17f7d20-b435-4559-8f4c-b31a07de6ee9", "doc_id": "d17f7d20-b435-4559-8f4c-b31a07de6ee9", "ref_doc_id": "d17f7d20-b435-4559-8f4c-b31a07de6ee9", "chroma:document": "<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/sam.png\" alt=\"SAM Architecture\"/>\n</div>\n\nSAM serves as a powerful foundation model for segmentation as it has large data coverage. It is trained on \n[SA-1B](https://ai.meta.com/datasets/segment-anything/), a dataset with 1 million images and 1.1 billion masks. \n\nIn this guide, you will learn how to:\n- Infer in segment everything mode with batching,\n- Infer in point prompting mode,\n- Infer in box prompting mode.\n\nFirst, let's install `transformers`:\n\n```bash\npip install -q transformers\n```\n\n## Mask Generation Pipeline\n\nThe easiest way to infer mask generation models is to use the `mask-generation` pipeline.\n\n```python\n>>> from transformers import pipeline\n\n>>> checkpoint = \"facebook/sam-vit-base\"\n>>> mask_generator = pipeline(model=checkpoint, task=\"mask-generation\")\n```\n\nLet's see the image.\n\n```python\nfrom PIL import Image\nimport requests\n\nimg_url = \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg\"\nimage = Image.open(requests.get(img_url, stream=True).raw).convert(\"RGB\")\n```\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg\" alt=\"Example Image\"/>\n</div>\n\nLet's segment everything. `points-per-batch` enables parallel inference of points in segment everything mode. This enables faster inference, but consumes more memory. Moreover, SAM only enables batching over points and not the images. `pred_iou_thresh` is the IoU confidence threshold where only the masks above that certain threshold are returned.\n\n```python\nmasks = mask_generator(image, points_per_batch=128, pred_iou_thresh=0.88)\n```\n\nThe `masks` looks like the following:\n\n```bash\n{'masks': [array([[False, False, False, ...,  True,  True,  True],\n         [False, False, False, ...,  True,  True,  True],\n         [False, False, False, ...,  True,  True,  True],\n         ...,\n         [False, False, False, ..., False, False, False],\n         [False, False, False, ..., False, False, False],\n         [False, False, False, ..., False, False, False]]),\n  array([[False, False, False, ..., False, False, False],\n         [False, False, False, ..., False, False, False],\n         [False, False, False, ..., False, False, False],\n         ...,\n'scores': tensor([0.9972, 0.9917,\n        ...,\n}\n```\n\nWe can visualize them like this:\n\n```python\nimport matplotlib.pyplot as plt\n\nplt.imshow(image, cmap='gray')\n\nfor i, mask in enumerate(masks[\"masks\"]):\n    plt.imshow(mask, cmap='viridis', alpha=0.1, vmin=0, vmax=1)\n\nplt.axis('off')\nplt.show()\n```\n\nBelow is the original image in grayscale with colorful maps overlaid. Very impressive.\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee_segmented.png\" alt=\"Visualized\"/>\n</div>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a7c8ee15-04e4-47f1-9da6-28769de1f85d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/mask_generation", "title": "Mask Generation", "tokens": 2117, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a7c8ee15-04e4-47f1-9da6-28769de1f85d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/mask_generation\", \"title\": \"Mask Generation\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d17f7d20-b435-4559-8f4c-b31a07de6ee9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/mask_generation\", \"title\": \"Mask Generation\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"385f9b234ba753f61a1095d2d3cde2181f866dca68ccaf42a2dcf54ca0fb65b2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a67a987a-bddb-4978-99cf-136202525203\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/mask_generation\", \"title\": \"Mask Generation\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"41fcdf816c00785d00457985ae6f1e79a4aeb18f36cbce7ee4df1c31342bdc17\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8c3e66b8-503e-48dc-b360-3ca289e8dc59\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f0ea727a0aa4d79b68a673172a4fea6e31fea884671222f24da93b79c0fc4de1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2764, \"end_char_idx\": 6701, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d17f7d20-b435-4559-8f4c-b31a07de6ee9", "doc_id": "d17f7d20-b435-4559-8f4c-b31a07de6ee9", "ref_doc_id": "d17f7d20-b435-4559-8f4c-b31a07de6ee9", "chroma:document": "This enables faster inference, but consumes more memory. Moreover, SAM only enables batching over points and not the images. `pred_iou_thresh` is the IoU confidence threshold where only the masks above that certain threshold are returned.\n\n```python\nmasks = mask_generator(image, points_per_batch=128, pred_iou_thresh=0.88)\n```\n\nThe `masks` looks like the following:\n\n```bash\n{'masks': [array([[False, False, False, ...,  True,  True,  True],\n         [False, False, False, ...,  True,  True,  True],\n         [False, False, False, ...,  True,  True,  True],\n         ...,\n         [False, False, False, ..., False, False, False],\n         [False, False, False, ..., False, False, False],\n         [False, False, False, ..., False, False, False]]),\n  array([[False, False, False, ..., False, False, False],\n         [False, False, False, ..., False, False, False],\n         [False, False, False, ..., False, False, False],\n         ...,\n'scores': tensor([0.9972, 0.9917,\n        ...,\n}\n```\n\nWe can visualize them like this:\n\n```python\nimport matplotlib.pyplot as plt\n\nplt.imshow(image, cmap='gray')\n\nfor i, mask in enumerate(masks[\"masks\"]):\n    plt.imshow(mask, cmap='viridis', alpha=0.1, vmin=0, vmax=1)\n\nplt.axis('off')\nplt.show()\n```\n\nBelow is the original image in grayscale with colorful maps overlaid. Very impressive.\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee_segmented.png\" alt=\"Visualized\"/>\n</div>\n\n\n## Model Inference\n\n### Point Prompting\n\nYou can also use the model without the pipeline. To do so, initialize the model and\nthe processor.\n\n```python\nfrom transformers import SamModel, SamProcessor\nimport torch\n\ndevice = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n\nmodel = SamModel.from_pretrained(\"facebook/sam-vit-base\").to(device)\nprocessor = SamProcessor.from_pretrained(\"facebook/sam-vit-base\")\n```\n\nTo do point prompting, pass the input point to the processor, then take the processor output\nand pass it to the model for inference. To post-process the model output, pass the outputs and\n`original_sizes` and `reshaped_input_sizes` we take from the processor's initial output. We need to pass these \nsince the processor resizes the image, and the output needs to be extrapolated.\n\n```python\ninput_points = [[[2592, 1728]]] # point location of the bee\n\ninputs = processor(image, input_points=input_points, return_tensors=\"pt\").to(device)\nwith torch.no_grad():\n    outputs = model(**inputs)\nmasks = processor.image_processor.post_process_masks(outputs.pred_masks.cpu(), inputs[\"original_sizes\"].cpu(), inputs[\"reshaped_input_sizes\"].cpu())\n```\nWe can visualize the three masks in the `masks`\u00a0output.\n\n```python\nimport matplotlib.pyplot as plt\nimport numpy as np\n\nfig, axes = plt.subplots(1, 4, figsize=(15, 5))\n\naxes[0].imshow(image)\naxes[0].set_title('Original Image')\nmask_list = [masks[0][0][0].numpy(), masks[0][0][1].numpy(), masks[0][0][2].numpy()]\n\nfor i, mask in enumerate(mask_list, start=1):\n    overlayed_image = np.array(image).copy()\n\n    overlayed_image[:,:,0] = np.where(mask == 1, 255, overlayed_image[:,:,0])\n    overlayed_image[:,:,1] = np.where(mask == 1, 0, overlayed_image[:,:,1])\n    overlayed_image[:,:,2] = np.where(mask == 1, 0, overlayed_image[:,:,2])\n    \n    axes[i].imshow(overlayed_image)\n    axes[i].set_title(f'Mask {i}')\nfor ax in axes:\n    ax.axis('off')\n\nplt.show()\n```\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/masks.png\" alt=\"Visualized\"/>\n</div>\n\n### Box Prompting\n\nYou can also do box prompting in a similar fashion to point prompting. You can simply pass the input box in the format of a list\n`[x_min, y_min, x_max, y_max]` format along with the image to the `processor`. Take the processor output and directly pass it \nto the model, then post-process the output again."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 8c3e66b8-503e-48dc-b360-3ca289e8dc59
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/mask_generation", "title": "Mask Generation", "tokens": 2117, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8c3e66b8-503e-48dc-b360-3ca289e8dc59\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/mask_generation\", \"title\": \"Mask Generation\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d17f7d20-b435-4559-8f4c-b31a07de6ee9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/mask_generation\", \"title\": \"Mask Generation\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"385f9b234ba753f61a1095d2d3cde2181f866dca68ccaf42a2dcf54ca0fb65b2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a7c8ee15-04e4-47f1-9da6-28769de1f85d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/mask_generation\", \"title\": \"Mask Generation\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1496c6cfa0a677066af684ad48ac8af018ea5f1bfc479890e6b473a0edab61e1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6704, \"end_char_idx\": 7995, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d17f7d20-b435-4559-8f4c-b31a07de6ee9", "doc_id": "d17f7d20-b435-4559-8f4c-b31a07de6ee9", "ref_doc_id": "d17f7d20-b435-4559-8f4c-b31a07de6ee9", "chroma:document": "```python\n# bounding box around the bee\nbox = [2350, 1600, 2850, 2100]\n\ninputs = processor(\n        image,\n        input_boxes=[[[box]]],\n        return_tensors=\"pt\"\n    ).to(\"cuda\")\n\nwith torch.no_grad():\n    outputs = model(**inputs)\n\nmask = processor.image_processor.post_process_masks(\n    outputs.pred_masks.cpu(),\n    inputs[\"original_sizes\"].cpu(),\n    inputs[\"reshaped_input_sizes\"].cpu()\n)[0][0][0].numpy()\n```\n\nYou can visualize the bounding box around the bee as shown below.\n\n```python\nimport matplotlib.patches as patches\n\nfig, ax = plt.subplots()\nax.imshow(image)\n\nrectangle = patches.Rectangle((2350, 1600), 500, 500, linewidth=2, edgecolor='r', facecolor='none')\nax.add_patch(rectangle)\nax.axis(\"off\")\nplt.show()\n```\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/bbox.png\" alt=\"Visualized Bbox\"/>\n</div>\n\nYou can see the inference output below. \n\n```python\nfig, ax = plt.subplots()\nax.imshow(image)\nax.imshow(mask, cmap='viridis', alpha=0.4)\n\nax.axis(\"off\")\nplt.show()\n```\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/box_inference.png\" alt=\"Visualized Inference\"/>\n</div>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f0e39e31-50eb-4daa-92aa-6cc6ff2b1bbe
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f0e39e31-50eb-4daa-92aa-6cc6ff2b1bbe\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection\", \"title\": \"Zero-shot object detection\", \"tokens\": 2993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"63f38317-e197-49f6-a801-741703eeaa50\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection\", \"title\": \"Zero-shot object detection\", \"tokens\": 2993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7e8db6ca78f64dab105a404ac6bb403a05c857981260dd057dd944018eb8cee3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"50745a6d-daa6-4802-aa98-33895d2194c4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"57f0f676a9f4b24e604e3b97d0e1f70e4861a8cb761b5e925fbca7da301c5826\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2898, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "63f38317-e197-49f6-a801-741703eeaa50", "doc_id": "63f38317-e197-49f6-a801-741703eeaa50", "ref_doc_id": "63f38317-e197-49f6-a801-741703eeaa50", "chroma:document": "# Zero-shot object detection\n\n[[open-in-colab]]\n\nTraditionally, models used for [object detection](object_detection) require labeled image datasets for training,\nand are limited to detecting the set of classes from the training data.\n\nZero-shot object detection is supported by the [OWL-ViT](../model_doc/owlvit) model which uses a different approach. OWL-ViT\nis an open-vocabulary object detector. It means that it can detect objects in images based on free-text queries without\nthe need to fine-tune the model on labeled datasets.\n\nOWL-ViT leverages multi-modal representations to perform open-vocabulary detection. It combines [CLIP](../model_doc/clip) with\nlightweight object classification and localization heads. Open-vocabulary detection is achieved by embedding free-text queries with the text encoder of CLIP and using them as input to the object classification and localization heads.\nassociate images and their corresponding textual descriptions, and ViT processes image patches as inputs. The authors\nof OWL-ViT first trained CLIP from scratch and then fine-tuned OWL-ViT end to end on standard object detection datasets using\na bipartite matching loss.\n\nWith this approach, the model can detect objects based on textual descriptions without prior training on labeled datasets.\n\nIn this guide, you will learn how to use OWL-ViT:\n- to detect objects based on text prompts\n- for batch object detection\n- for image-guided object detection\n\nBefore you begin, make sure you have all the necessary libraries installed:\n\n```bash\npip install -q transformers\n```\n\n## Zero-shot object detection pipeline\n\nThe simplest way to try out inference with OWL-ViT is to use it in a [`pipeline`]. Instantiate a pipeline\nfor zero-shot object detection from a [checkpoint on the Hugging Face Hub](https://huggingface.co/models?other=owlvit):\n\n```python\n>>> from transformers import pipeline\n\n>>> checkpoint = \"google/owlv2-base-patch16-ensemble\"\n>>> detector = pipeline(model=checkpoint, task=\"zero-shot-object-detection\")\n```\n\nNext, choose an image you'd like to detect objects in. Here we'll use the image of astronaut Eileen Collins that is\na part of the [NASA](https://www.nasa.gov/multimedia/imagegallery/index.html) Great Images dataset.\n\n```py\n>>> import skimage\n>>> import numpy as np\n>>> from PIL import Image\n\n>>> image = skimage.data.astronaut()\n>>> image = Image.fromarray(np.uint8(image)).convert(\"RGB\")\n\n>>> image\n```\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_1.png\" alt=\"Astronaut Eileen Collins\"/>\n</div>\n\nPass the image and the candidate object labels to look for to the pipeline.\nHere we pass the image directly; other suitable options include a local path to an image or an image url. We also pass text descriptions for all items we want to query the image for."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 50745a6d-daa6-4802-aa98-33895d2194c4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"50745a6d-daa6-4802-aa98-33895d2194c4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection\", \"title\": \"Zero-shot object detection\", \"tokens\": 2993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"63f38317-e197-49f6-a801-741703eeaa50\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection\", \"title\": \"Zero-shot object detection\", \"tokens\": 2993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7e8db6ca78f64dab105a404ac6bb403a05c857981260dd057dd944018eb8cee3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f0e39e31-50eb-4daa-92aa-6cc6ff2b1bbe\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection\", \"title\": \"Zero-shot object detection\", \"tokens\": 2993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a96066c26dc7602891a516c6e79e22e57ee235d6cbf4bd552a5698c765c0bded\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"31ff1cf1-586b-4a08-8566-edc5d167368d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0bedb9152c57476bafe862cc8c56f580398b1ab2552580308d8d30a181821169\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1167, \"end_char_idx\": 4636, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "63f38317-e197-49f6-a801-741703eeaa50", "doc_id": "63f38317-e197-49f6-a801-741703eeaa50", "ref_doc_id": "63f38317-e197-49f6-a801-741703eeaa50", "chroma:document": "With this approach, the model can detect objects based on textual descriptions without prior training on labeled datasets.\n\nIn this guide, you will learn how to use OWL-ViT:\n- to detect objects based on text prompts\n- for batch object detection\n- for image-guided object detection\n\nBefore you begin, make sure you have all the necessary libraries installed:\n\n```bash\npip install -q transformers\n```\n\n## Zero-shot object detection pipeline\n\nThe simplest way to try out inference with OWL-ViT is to use it in a [`pipeline`]. Instantiate a pipeline\nfor zero-shot object detection from a [checkpoint on the Hugging Face Hub](https://huggingface.co/models?other=owlvit):\n\n```python\n>>> from transformers import pipeline\n\n>>> checkpoint = \"google/owlv2-base-patch16-ensemble\"\n>>> detector = pipeline(model=checkpoint, task=\"zero-shot-object-detection\")\n```\n\nNext, choose an image you'd like to detect objects in. Here we'll use the image of astronaut Eileen Collins that is\na part of the [NASA](https://www.nasa.gov/multimedia/imagegallery/index.html) Great Images dataset.\n\n```py\n>>> import skimage\n>>> import numpy as np\n>>> from PIL import Image\n\n>>> image = skimage.data.astronaut()\n>>> image = Image.fromarray(np.uint8(image)).convert(\"RGB\")\n\n>>> image\n```\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_1.png\" alt=\"Astronaut Eileen Collins\"/>\n</div>\n\nPass the image and the candidate object labels to look for to the pipeline.\nHere we pass the image directly; other suitable options include a local path to an image or an image url. We also pass text descriptions for all items we want to query the image for. \n\n```py\n>>> predictions = detector(\n...     image,\n...     candidate_labels=[\"human face\", \"rocket\", \"nasa badge\", \"star-spangled banner\"],\n... )\n>>> predictions\n[{'score': 0.3571370542049408,\n  'label': 'human face',\n  'box': {'xmin': 180, 'ymin': 71, 'xmax': 271, 'ymax': 178}},\n {'score': 0.28099656105041504,\n  'label': 'nasa badge',\n  'box': {'xmin': 129, 'ymin': 348, 'xmax': 206, 'ymax': 427}},\n {'score': 0.2110239565372467,\n  'label': 'rocket',\n  'box': {'xmin': 350, 'ymin': -1, 'xmax': 468, 'ymax': 288}},\n {'score': 0.13790413737297058,\n  'label': 'star-spangled banner',\n  'box': {'xmin': 1, 'ymin': 1, 'xmax': 105, 'ymax': 509}},\n {'score': 0.11950037628412247,\n  'label': 'nasa badge',\n  'box': {'xmin': 277, 'ymin': 338, 'xmax': 327, 'ymax': 380}},\n {'score': 0.10649408400058746,\n  'label': 'rocket',\n  'box': {'xmin': 358, 'ymin': 64, 'xmax': 424, 'ymax': 280}}]\n```\n\nLet's visualize the predictions:\n\n```py\n>>> from PIL import ImageDraw\n\n>>> draw = ImageDraw.Draw(image)\n\n>>> for prediction in predictions:\n...     box = prediction[\"box\"]\n...     label = prediction[\"label\"]\n...     score = prediction[\"score\"]\n\n...     xmin, ymin, xmax, ymax = box.values()\n...     draw.rectangle((xmin, ymin, xmax, ymax), outline=\"red\", width=1)\n...     draw.text((xmin, ymin), f\"{label}: {round(score,2)}\", fill=\"white\")\n\n>>> image\n```\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_2.png\" alt=\"Visualized predictions on NASA image\"/>\n</div>\n\n## Text-prompted zero-shot object detection by hand\n\nNow that you've seen how to use the zero-shot object detection pipeline, let's replicate the same\nresult manually."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 31ff1cf1-586b-4a08-8566-edc5d167368d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"31ff1cf1-586b-4a08-8566-edc5d167368d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection\", \"title\": \"Zero-shot object detection\", \"tokens\": 2993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"63f38317-e197-49f6-a801-741703eeaa50\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection\", \"title\": \"Zero-shot object detection\", \"tokens\": 2993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7e8db6ca78f64dab105a404ac6bb403a05c857981260dd057dd944018eb8cee3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"50745a6d-daa6-4802-aa98-33895d2194c4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection\", \"title\": \"Zero-shot object detection\", \"tokens\": 2993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"59ba5ca7ff9b5f78b61c152e547936316f203ccbc49a52eea786b8c3333af528\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2a91e4f9-b76c-4013-8b92-5418a6470afa\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f2679b3e1283b8389acede742e3aa00aca7adae05978ae60bf292065ec2d21f9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4638, \"end_char_idx\": 7506, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "63f38317-e197-49f6-a801-741703eeaa50", "doc_id": "63f38317-e197-49f6-a801-741703eeaa50", "ref_doc_id": "63f38317-e197-49f6-a801-741703eeaa50", "chroma:document": "Start by loading the model and associated processor from a [checkpoint on the Hugging Face Hub](https://huggingface.co/models?other=owlvit).\nHere we'll use the same checkpoint as before:\n\n```py\n>>> from transformers import AutoProcessor, AutoModelForZeroShotObjectDetection\n\n>>> model = AutoModelForZeroShotObjectDetection.from_pretrained(checkpoint)\n>>> processor = AutoProcessor.from_pretrained(checkpoint)\n```\n\nLet's take a different image to switch things up.\n\n```py\n>>> import requests\n\n>>> url = \"https://unsplash.com/photos/oj0zeY2Ltk4/download?ixid=MnwxMjA3fDB8MXxzZWFyY2h8MTR8fHBpY25pY3xlbnwwfHx8fDE2Nzc0OTE1NDk&force=true&w=640\"\n>>> im = Image.open(requests.get(url, stream=True).raw)\n>>> im\n```\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_3.png\" alt=\"Beach photo\"/>\n</div>\n\nUse the processor to prepare the inputs for the model. The processor combines an image processor that prepares the\nimage for the model by resizing and normalizing it, and a [`CLIPTokenizer`] that takes care of the text inputs.\n\n```py\n>>> text_queries = [\"hat\", \"book\", \"sunglasses\", \"camera\"]\n>>> inputs = processor(text=text_queries, images=im, return_tensors=\"pt\")\n```\n\nPass the inputs through the model, post-process, and visualize the results. Since the image processor resized images before\nfeeding them to the model, you need to use the [`~OwlViTImageProcessor.post_process_object_detection`] method to make sure the predicted bounding\nboxes have the correct coordinates relative to the original image:\n\n```py\n>>> import torch\n\n>>> with torch.no_grad():\n...     outputs = model(**inputs)\n...     target_sizes = torch.tensor([im.size[::-1]])\n...     results = processor.post_process_object_detection(outputs, threshold=0.1, target_sizes=target_sizes)[0]\n\n>>> draw = ImageDraw.Draw(im)\n\n>>> scores = results[\"scores\"].tolist()\n>>> labels = results[\"labels\"].tolist()\n>>> boxes = results[\"boxes\"].tolist()\n\n>>> for box, score, label in zip(boxes, scores, labels):\n...     xmin, ymin, xmax, ymax = box\n...     draw.rectangle((xmin, ymin, xmax, ymax), outline=\"red\", width=1)\n...     draw.text((xmin, ymin), f\"{text_queries[label]}: {round(score,2)}\", fill=\"white\")\n\n>>> im\n```\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_4.png\" alt=\"Beach photo with detected objects\"/>\n</div>\n\n## Batch processing\n\nYou can pass multiple sets of images and text queries to search for different (or same) objects in several images.\nLet's use both an astronaut image and the beach image together.\nFor batch processing, you should pass text queries as a nested list to the processor and images as lists of PIL images,\nPyTorch tensors, or NumPy arrays."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 2a91e4f9-b76c-4013-8b92-5418a6470afa
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2a91e4f9-b76c-4013-8b92-5418a6470afa\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection\", \"title\": \"Zero-shot object detection\", \"tokens\": 2993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"63f38317-e197-49f6-a801-741703eeaa50\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection\", \"title\": \"Zero-shot object detection\", \"tokens\": 2993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7e8db6ca78f64dab105a404ac6bb403a05c857981260dd057dd944018eb8cee3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"31ff1cf1-586b-4a08-8566-edc5d167368d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection\", \"title\": \"Zero-shot object detection\", \"tokens\": 2993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"74ea051195fba5a8758d9926433f244481b93edc6ab1cf3aa5d61e59e9cea028\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"15353950-3760-4509-aa89-781c39225fc3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7ec40982ef3dc2111c501cdfbc6684f788849a2fc270c6be455be431fb0c863c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5996, \"end_char_idx\": 9215, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "63f38317-e197-49f6-a801-741703eeaa50", "doc_id": "63f38317-e197-49f6-a801-741703eeaa50", "ref_doc_id": "63f38317-e197-49f6-a801-741703eeaa50", "chroma:document": "Since the image processor resized images before\nfeeding them to the model, you need to use the [`~OwlViTImageProcessor.post_process_object_detection`] method to make sure the predicted bounding\nboxes have the correct coordinates relative to the original image:\n\n```py\n>>> import torch\n\n>>> with torch.no_grad():\n...     outputs = model(**inputs)\n...     target_sizes = torch.tensor([im.size[::-1]])\n...     results = processor.post_process_object_detection(outputs, threshold=0.1, target_sizes=target_sizes)[0]\n\n>>> draw = ImageDraw.Draw(im)\n\n>>> scores = results[\"scores\"].tolist()\n>>> labels = results[\"labels\"].tolist()\n>>> boxes = results[\"boxes\"].tolist()\n\n>>> for box, score, label in zip(boxes, scores, labels):\n...     xmin, ymin, xmax, ymax = box\n...     draw.rectangle((xmin, ymin, xmax, ymax), outline=\"red\", width=1)\n...     draw.text((xmin, ymin), f\"{text_queries[label]}: {round(score,2)}\", fill=\"white\")\n\n>>> im\n```\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_4.png\" alt=\"Beach photo with detected objects\"/>\n</div>\n\n## Batch processing\n\nYou can pass multiple sets of images and text queries to search for different (or same) objects in several images.\nLet's use both an astronaut image and the beach image together.\nFor batch processing, you should pass text queries as a nested list to the processor and images as lists of PIL images,\nPyTorch tensors, or NumPy arrays.\n\n```py\n>>> images = [image, im]\n>>> text_queries = [\n...     [\"human face\", \"rocket\", \"nasa badge\", \"star-spangled banner\"],\n...     [\"hat\", \"book\", \"sunglasses\", \"camera\"],\n... ]\n>>> inputs = processor(text=text_queries, images=images, return_tensors=\"pt\")\n```\n\nPreviously for post-processing you passed the single image's size as a tensor, but you can also pass a tuple, or, in case\nof several images, a list of tuples. Let's create predictions for the two examples, and visualize the second one (`image_idx = 1`).\n\n```py\n>>> with torch.no_grad():\n...     outputs = model(**inputs)\n...     target_sizes = [x.size[::-1] for x in images]\n...     results = processor.post_process_object_detection(outputs, threshold=0.1, target_sizes=target_sizes)\n\n>>> image_idx = 1\n>>> draw = ImageDraw.Draw(images[image_idx])\n\n>>> scores = results[image_idx][\"scores\"].tolist()\n>>> labels = results[image_idx][\"labels\"].tolist()\n>>> boxes = results[image_idx][\"boxes\"].tolist()\n\n>>> for box, score, label in zip(boxes, scores, labels):\n...     xmin, ymin, xmax, ymax = box\n...     draw.rectangle((xmin, ymin, xmax, ymax), outline=\"red\", width=1)\n...     draw.text((xmin, ymin), f\"{text_queries[image_idx][label]}: {round(score,2)}\", fill=\"white\")\n\n>>> images[image_idx]\n```\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_4.png\" alt=\"Beach photo with detected objects\"/>\n</div>\n\n## Image-guided object detection\n\nIn addition to zero-shot object detection with text queries, OWL-ViT offers image-guided object detection. This means\nyou can use an image query to find similar objects in the target image."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 15353950-3760-4509-aa89-781c39225fc3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"15353950-3760-4509-aa89-781c39225fc3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection\", \"title\": \"Zero-shot object detection\", \"tokens\": 2993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"63f38317-e197-49f6-a801-741703eeaa50\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection\", \"title\": \"Zero-shot object detection\", \"tokens\": 2993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7e8db6ca78f64dab105a404ac6bb403a05c857981260dd057dd944018eb8cee3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2a91e4f9-b76c-4013-8b92-5418a6470afa\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection\", \"title\": \"Zero-shot object detection\", \"tokens\": 2993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"270f0c224ab48347ff571c1b6ad5e00539daa4382277a026662cc95b58b4d108\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4a952c14-7a76-4301-9bbf-0ee8a74adf36\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"579fbd9f59083f4ea1aafad225d23c489c474f135b633e9077cf401783306302\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7929, \"end_char_idx\": 10432, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "63f38317-e197-49f6-a801-741703eeaa50", "doc_id": "63f38317-e197-49f6-a801-741703eeaa50", "ref_doc_id": "63f38317-e197-49f6-a801-741703eeaa50", "chroma:document": "Let's create predictions for the two examples, and visualize the second one (`image_idx = 1`).\n\n```py\n>>> with torch.no_grad():\n...     outputs = model(**inputs)\n...     target_sizes = [x.size[::-1] for x in images]\n...     results = processor.post_process_object_detection(outputs, threshold=0.1, target_sizes=target_sizes)\n\n>>> image_idx = 1\n>>> draw = ImageDraw.Draw(images[image_idx])\n\n>>> scores = results[image_idx][\"scores\"].tolist()\n>>> labels = results[image_idx][\"labels\"].tolist()\n>>> boxes = results[image_idx][\"boxes\"].tolist()\n\n>>> for box, score, label in zip(boxes, scores, labels):\n...     xmin, ymin, xmax, ymax = box\n...     draw.rectangle((xmin, ymin, xmax, ymax), outline=\"red\", width=1)\n...     draw.text((xmin, ymin), f\"{text_queries[image_idx][label]}: {round(score,2)}\", fill=\"white\")\n\n>>> images[image_idx]\n```\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_4.png\" alt=\"Beach photo with detected objects\"/>\n</div>\n\n## Image-guided object detection\n\nIn addition to zero-shot object detection with text queries, OWL-ViT offers image-guided object detection. This means\nyou can use an image query to find similar objects in the target image.\nUnlike text queries, only a single example image is allowed.\n\nLet's take an image with two cats on a couch as a target image, and an image of a single cat\nas a query:\n\n```py\n>>> url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image_target = Image.open(requests.get(url, stream=True).raw)\n\n>>> query_url = \"http://images.cocodataset.org/val2017/000000524280.jpg\"\n>>> query_image = Image.open(requests.get(query_url, stream=True).raw)\n```\n\nLet's take a quick look at the images:\n\n```py\n>>> import matplotlib.pyplot as plt\n\n>>> fig, ax = plt.subplots(1, 2)\n>>> ax[0].imshow(image_target)\n>>> ax[1].imshow(query_image)\n```\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_5.png\" alt=\"Cats\"/>\n</div>\n\nIn the preprocessing step, instead of text queries, you now need to use `query_images`:\n\n```py\n>>> inputs = processor(images=image_target, query_images=query_image, return_tensors=\"pt\")\n```\n\nFor predictions, instead of passing the inputs to the model, pass them to [`~OwlViTForObjectDetection.image_guided_detection`]. Draw the predictions\nas before except now there are no labels."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 4a952c14-7a76-4301-9bbf-0ee8a74adf36
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4a952c14-7a76-4301-9bbf-0ee8a74adf36\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection\", \"title\": \"Zero-shot object detection\", \"tokens\": 2993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"63f38317-e197-49f6-a801-741703eeaa50\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection\", \"title\": \"Zero-shot object detection\", \"tokens\": 2993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7e8db6ca78f64dab105a404ac6bb403a05c857981260dd057dd944018eb8cee3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"15353950-3760-4509-aa89-781c39225fc3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection\", \"title\": \"Zero-shot object detection\", \"tokens\": 2993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2041335e14cf1710e017cb57d3a6b32fc465d0fb5a16145e7fc042e1f682fbdd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9133, \"end_char_idx\": 11224, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "63f38317-e197-49f6-a801-741703eeaa50", "doc_id": "63f38317-e197-49f6-a801-741703eeaa50", "ref_doc_id": "63f38317-e197-49f6-a801-741703eeaa50", "chroma:document": "This means\nyou can use an image query to find similar objects in the target image.\nUnlike text queries, only a single example image is allowed.\n\nLet's take an image with two cats on a couch as a target image, and an image of a single cat\nas a query:\n\n```py\n>>> url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image_target = Image.open(requests.get(url, stream=True).raw)\n\n>>> query_url = \"http://images.cocodataset.org/val2017/000000524280.jpg\"\n>>> query_image = Image.open(requests.get(query_url, stream=True).raw)\n```\n\nLet's take a quick look at the images:\n\n```py\n>>> import matplotlib.pyplot as plt\n\n>>> fig, ax = plt.subplots(1, 2)\n>>> ax[0].imshow(image_target)\n>>> ax[1].imshow(query_image)\n```\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_5.png\" alt=\"Cats\"/>\n</div>\n\nIn the preprocessing step, instead of text queries, you now need to use `query_images`:\n\n```py\n>>> inputs = processor(images=image_target, query_images=query_image, return_tensors=\"pt\")\n```\n\nFor predictions, instead of passing the inputs to the model, pass them to [`~OwlViTForObjectDetection.image_guided_detection`]. Draw the predictions\nas before except now there are no labels.\n\n```py\n>>> with torch.no_grad():\n...     outputs = model.image_guided_detection(**inputs)\n...     target_sizes = torch.tensor([image_target.size[::-1]])\n...     results = processor.post_process_image_guided_detection(outputs=outputs, target_sizes=target_sizes)[0]\n\n>>> draw = ImageDraw.Draw(image_target)\n\n>>> scores = results[\"scores\"].tolist()\n>>> boxes = results[\"boxes\"].tolist()\n\n>>> for box, score, label in zip(boxes, scores, labels):\n...     xmin, ymin, xmax, ymax = box\n...     draw.rectangle((xmin, ymin, xmax, ymax), outline=\"white\", width=4)\n\n>>> image_target\n```\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_6.png\" alt=\"Cats with bounding boxes\"/>\n</div>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 045fa8fb-62f4-49bf-81e5-f6f6a30c0e6b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"045fa8fb-62f4-49bf-81e5-f6f6a30c0e6b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2c93072b-42b4-4c50-bac8-8f2f8d5f938a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"54c13277-7cb7-452d-973b-aa5b7468bde5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6124ef1c5d9dbf649af3c2f030fd53cd1b84bcb0c53bccb420bccefdfb79a230\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2327, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "ref_doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "chroma:document": "# Image Segmentation\n\n[[open-in-colab]]\n\n<Youtube id=\"dKE8SIt9C-w\"/>\n\nImage segmentation models separate areas corresponding to different areas of interest in an image. These models work by assigning a label to each pixel. There are several types of segmentation: semantic segmentation, instance segmentation, and panoptic segmentation.\n\nIn this guide, we will:\n1. [Take a look at different types of segmentation](#types-of-segmentation).\n2. [Have an end-to-end fine-tuning example for semantic segmentation](#fine-tuning-a-model-for-segmentation).\n\nBefore you begin, make sure you have all the necessary libraries installed:\n\n```py\n# uncomment to install the necessary libraries\n!pip install -q datasets transformers evaluate accelerate\n```\n\nWe encourage you to log in to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to log in:\n\n```py\n>>> from huggingface_hub import notebook_login\n\n>>> notebook_login()\n```\n\n## Types of Segmentation\n\nSemantic segmentation assigns a label or class to every single pixel in an image. Let's take a look at a semantic segmentation model output. It will assign the same class to every instance of an object it comes across in an image, for example, all cats will be labeled as \"cat\" instead of \"cat-1\", \"cat-2\".\nWe can use transformers' image segmentation pipeline to quickly infer a semantic segmentation model. Let's take a look at the example image.\n\n```python\nfrom transformers import pipeline\nfrom PIL import Image\nimport requests\n\nurl = \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/segmentation_input.jpg\"\nimage = Image.open(requests.get(url, stream=True).raw)\nimage\n```\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/segmentation_input.jpg\" alt=\"Segmentation Input\"/>\n</div>\n\nWe will use [nvidia/segformer-b1-finetuned-cityscapes-1024-1024](https://huggingface.co/nvidia/segformer-b1-finetuned-cityscapes-1024-1024).\n\n```python\nsemantic_segmentation = pipeline(\"image-segmentation\", \"nvidia/segformer-b1-finetuned-cityscapes-1024-1024\")\nresults = semantic_segmentation(image)\nresults\n```\n\nThe segmentation pipeline output includes a mask for every predicted class."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 54c13277-7cb7-452d-973b-aa5b7468bde5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"54c13277-7cb7-452d-973b-aa5b7468bde5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2c93072b-42b4-4c50-bac8-8f2f8d5f938a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"045fa8fb-62f4-49bf-81e5-f6f6a30c0e6b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"5ba99c88e389ec3cde8471d8e4d55cc3cbe451f531d7ef37b11b4175f1e1b630\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6bb4781b-812c-40c2-900d-8d6df47ad19e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"332e480efc41cae58485573f76a0402fbc065de4b38e5b72408088e6d4f58cff\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 858, \"end_char_idx\": 3383, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "ref_doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "chroma:document": "When prompted, enter your token to log in:\n\n```py\n>>> from huggingface_hub import notebook_login\n\n>>> notebook_login()\n```\n\n## Types of Segmentation\n\nSemantic segmentation assigns a label or class to every single pixel in an image. Let's take a look at a semantic segmentation model output. It will assign the same class to every instance of an object it comes across in an image, for example, all cats will be labeled as \"cat\" instead of \"cat-1\", \"cat-2\".\nWe can use transformers' image segmentation pipeline to quickly infer a semantic segmentation model. Let's take a look at the example image.\n\n```python\nfrom transformers import pipeline\nfrom PIL import Image\nimport requests\n\nurl = \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/segmentation_input.jpg\"\nimage = Image.open(requests.get(url, stream=True).raw)\nimage\n```\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/segmentation_input.jpg\" alt=\"Segmentation Input\"/>\n</div>\n\nWe will use [nvidia/segformer-b1-finetuned-cityscapes-1024-1024](https://huggingface.co/nvidia/segformer-b1-finetuned-cityscapes-1024-1024).\n\n```python\nsemantic_segmentation = pipeline(\"image-segmentation\", \"nvidia/segformer-b1-finetuned-cityscapes-1024-1024\")\nresults = semantic_segmentation(image)\nresults\n```\n\nThe segmentation pipeline output includes a mask for every predicted class.\n```bash\n[{'score': None,\n  'label': 'road',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': None,\n  'label': 'sidewalk',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': None,\n  'label': 'building',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': None,\n  'label': 'wall',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': None,\n  'label': 'pole',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': None,\n  'label': 'traffic sign',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': None,\n  'label': 'vegetation',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': None,\n  'label': 'terrain',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': None,\n  'label': 'sky',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': None,\n  'label': 'car',\n  'mask': <PIL.Image.Image image mode=L size=612x415>}]\n```\n\nTaking a look at the mask for the car class, we can see every car is classified with the same mask."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6bb4781b-812c-40c2-900d-8d6df47ad19e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"6bb4781b-812c-40c2-900d-8d6df47ad19e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2c93072b-42b4-4c50-bac8-8f2f8d5f938a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"54c13277-7cb7-452d-973b-aa5b7468bde5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"e8f03516ae1599e5d062fef92a3b681da7cf8e67f1c8ecda8a046f45860c268c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f7503ae5-9844-4df0-899e-63f9a9eb1761\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d19353615ce3434092b174583dda05256f54f985c7c13b5ac4fbb23fc4319713\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2328, \"end_char_idx\": 4836, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "ref_doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "chroma:document": "```bash\n[{'score': None,\n  'label': 'road',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': None,\n  'label': 'sidewalk',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': None,\n  'label': 'building',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': None,\n  'label': 'wall',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': None,\n  'label': 'pole',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': None,\n  'label': 'traffic sign',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': None,\n  'label': 'vegetation',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': None,\n  'label': 'terrain',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': None,\n  'label': 'sky',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': None,\n  'label': 'car',\n  'mask': <PIL.Image.Image image mode=L size=612x415>}]\n```\n\nTaking a look at the mask for the car class, we can see every car is classified with the same mask.\n\n```python\nresults[-1][\"mask\"]\n```\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/semantic_segmentation_output.png\" alt=\"Semantic Segmentation Output\"/>\n</div>\n\nIn instance segmentation, the goal is not to classify every pixel, but to predict a mask for **every instance of an object** in a given image. It works very similar to object detection, where there is a bounding box for every instance, there's a segmentation mask instead. We will use [facebook/mask2former-swin-large-cityscapes-instance](https://huggingface.co/facebook/mask2former-swin-large-cityscapes-instance) for this.\n\n```python\ninstance_segmentation = pipeline(\"image-segmentation\", \"facebook/mask2former-swin-large-cityscapes-instance\")\nresults = instance_segmentation(image)\nresults\n```\n\nAs you can see below, there are multiple cars classified, and there's no classification for pixels other than pixels that belong to car and person instances.\n\n```bash\n[{'score': 0.999944,\n  'label': 'car',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': 0.999945,\n  'label': 'car',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': 0.999652,\n  'label': 'car',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': 0.903529,\n  'label': 'person',\n  'mask': <PIL.Image.Image image mode=L size=612x415>}]\n```\nChecking out one of the car masks below."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f7503ae5-9844-4df0-899e-63f9a9eb1761
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"f7503ae5-9844-4df0-899e-63f9a9eb1761\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2c93072b-42b4-4c50-bac8-8f2f8d5f938a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6bb4781b-812c-40c2-900d-8d6df47ad19e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"1f825b0efe8e261f6f3bde900d9d62ebdec615d5fa4be2cb43f08d9b1750307a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e4b3efff-2542-4b78-ada3-5df37495ef37\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e48832e7c3c106d2f31feb8510cce7b1d819460b8dfeda004749a8a783d24f53\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3385, \"end_char_idx\": 5765, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "ref_doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "chroma:document": "```python\nresults[-1][\"mask\"]\n```\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/semantic_segmentation_output.png\" alt=\"Semantic Segmentation Output\"/>\n</div>\n\nIn instance segmentation, the goal is not to classify every pixel, but to predict a mask for **every instance of an object** in a given image. It works very similar to object detection, where there is a bounding box for every instance, there's a segmentation mask instead. We will use [facebook/mask2former-swin-large-cityscapes-instance](https://huggingface.co/facebook/mask2former-swin-large-cityscapes-instance) for this.\n\n```python\ninstance_segmentation = pipeline(\"image-segmentation\", \"facebook/mask2former-swin-large-cityscapes-instance\")\nresults = instance_segmentation(image)\nresults\n```\n\nAs you can see below, there are multiple cars classified, and there's no classification for pixels other than pixels that belong to car and person instances.\n\n```bash\n[{'score': 0.999944,\n  'label': 'car',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': 0.999945,\n  'label': 'car',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': 0.999652,\n  'label': 'car',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': 0.903529,\n  'label': 'person',\n  'mask': <PIL.Image.Image image mode=L size=612x415>}]\n```\nChecking out one of the car masks below.\n\n```python\nresults[2][\"mask\"]\n```\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/instance_segmentation_output.png\" alt=\"Semantic Segmentation Output\"/>\n</div>\n\nPanoptic segmentation combines semantic segmentation and instance segmentation, where every pixel is classified into a class and an instance of that class, and there are multiple masks for each instance of a class. We can use [facebook/mask2former-swin-large-cityscapes-panoptic](https://huggingface.co/facebook/mask2former-swin-large-cityscapes-panoptic) for this.\n\n```python\npanoptic_segmentation = pipeline(\"image-segmentation\", \"facebook/mask2former-swin-large-cityscapes-panoptic\")\nresults = panoptic_segmentation(image)\nresults\n```\nAs you can see below, we have more classes. We will later illustrate to see that every pixel is classified into one of the classes."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e4b3efff-2542-4b78-ada3-5df37495ef37
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"e4b3efff-2542-4b78-ada3-5df37495ef37\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2c93072b-42b4-4c50-bac8-8f2f8d5f938a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f7503ae5-9844-4df0-899e-63f9a9eb1761\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"9781af54940e88f586b3d844232b5c8dfda380abc9d858caf034ed50da57bb99\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c12511ca-acbe-4cb0-8bda-7921462dd3d0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"27b255274b441150644516f516d93eea7eb1c96885cdeb37882b514bd088ccc1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4401, \"end_char_idx\": 7045, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "ref_doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "chroma:document": "```bash\n[{'score': 0.999944,\n  'label': 'car',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': 0.999945,\n  'label': 'car',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': 0.999652,\n  'label': 'car',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': 0.903529,\n  'label': 'person',\n  'mask': <PIL.Image.Image image mode=L size=612x415>}]\n```\nChecking out one of the car masks below.\n\n```python\nresults[2][\"mask\"]\n```\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/instance_segmentation_output.png\" alt=\"Semantic Segmentation Output\"/>\n</div>\n\nPanoptic segmentation combines semantic segmentation and instance segmentation, where every pixel is classified into a class and an instance of that class, and there are multiple masks for each instance of a class. We can use [facebook/mask2former-swin-large-cityscapes-panoptic](https://huggingface.co/facebook/mask2former-swin-large-cityscapes-panoptic) for this.\n\n```python\npanoptic_segmentation = pipeline(\"image-segmentation\", \"facebook/mask2former-swin-large-cityscapes-panoptic\")\nresults = panoptic_segmentation(image)\nresults\n```\nAs you can see below, we have more classes. We will later illustrate to see that every pixel is classified into one of the classes.\n\n```bash\n[{'score': 0.999981,\n  'label': 'car',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': 0.999958,\n  'label': 'car',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': 0.99997,\n  'label': 'vegetation',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': 0.999575,\n  'label': 'pole',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': 0.999958,\n  'label': 'building',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': 0.999634,\n  'label': 'road',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': 0.996092,\n  'label': 'sidewalk',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': 0.999221,\n  'label': 'car',\n  'mask': <PIL.Image.Image image mode=L size=612x415>},\n {'score': 0.99987,\n  'label': 'sky',\n  'mask': <PIL.Image.Image image mode=L size=612x415>}]\n```\n\nLet's have a side by side comparison for all types of segmentation.\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/segmentation-comparison.png\" alt=\"Segmentation Maps Compared\"/>\n</div>\n\nSeeing all types of segmentation, let's have a deep dive on fine-tuning a model for semantic segmentation."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: c12511ca-acbe-4cb0-8bda-7921462dd3d0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"c12511ca-acbe-4cb0-8bda-7921462dd3d0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2c93072b-42b4-4c50-bac8-8f2f8d5f938a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e4b3efff-2542-4b78-ada3-5df37495ef37\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"43c8a8cd4ee7f004e637d86c2997a65ee5ed3d66279d994c9d554840111c7649\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"44e5c046-1784-4fe3-8bda-cece1b68d30b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"889e3b3d6625a79644a3da7340abdbaa71aa3916a57d49cd34bb75056a3950e2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6721, \"end_char_idx\": 9110, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "ref_doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "chroma:document": "<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/segmentation-comparison.png\" alt=\"Segmentation Maps Compared\"/>\n</div>\n\nSeeing all types of segmentation, let's have a deep dive on fine-tuning a model for semantic segmentation.\n\nCommon real-world applications of semantic segmentation include training self-driving cars to identify pedestrians and important traffic information, identifying cells and abnormalities in medical imagery, and monitoring environmental changes from satellite imagery.\n\n## Fine-tuning a Model for Segmentation\n\nWe will now:\n\n1. Finetune [SegFormer](https://huggingface.co/docs/transformers/main/en/model_doc/segformer#segformer) on the [SceneParse150](https://huggingface.co/datasets/scene_parse_150) dataset.\n2. Use your fine-tuned model for inference.\n\n<Tip>\n\nTo see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/image-segmentation)\n\n</Tip>\n\n\n### Load SceneParse150 dataset\n\nStart by loading a smaller subset of the SceneParse150 dataset from the \ud83e\udd17 Datasets library. This'll give you a chance to experiment and make sure everything works before spending more time training on the full dataset.\n\n```py\n>>> from datasets import load_dataset\n\n>>> ds = load_dataset(\"scene_parse_150\", split=\"train[:50]\")\n```\n\nSplit the dataset's `train` split into a train and test set with the [`~datasets.Dataset.train_test_split`] method:\n\n```py\n>>> ds = ds.train_test_split(test_size=0.2)\n>>> train_ds = ds[\"train\"]\n>>> test_ds = ds[\"test\"]\n```\n\nThen take a look at an example:\n\n```py\n>>> train_ds[0]\n{'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=512x683 at 0x7F9B0C201F90>,\n 'annotation': <PIL.PngImagePlugin.PngImageFile image mode=L size=512x683 at 0x7F9B0C201DD0>,\n 'scene_category': 368}\n\n# view the image\n>>> train_ds[0][\"image\"]\n```\n\n- `image`: a PIL image of the scene.\n- `annotation`: a PIL image of the segmentation map, which is also the model's target.\n- `scene_category`: a category id that describes the image scene like \"kitchen\" or \"office\". In this guide, you'll only need `image` and `annotation`, both of which are PIL images.\n\nYou'll also want to create a dictionary that maps a label id to a label class which will be useful when you set up the model later."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 44e5c046-1784-4fe3-8bda-cece1b68d30b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"44e5c046-1784-4fe3-8bda-cece1b68d30b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2c93072b-42b4-4c50-bac8-8f2f8d5f938a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c12511ca-acbe-4cb0-8bda-7921462dd3d0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"997df885267a99b92a30e5b060c6a9a7b95111e27c1d5a44784f8932016f6fc7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6c747df0-bcae-4263-9f19-9c6d3c09a8f9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1f9d8ee40e2d2fbf13dd9983ec089b2be97e662ce07b844c9119aa24aba5ec27\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7600, \"end_char_idx\": 9916, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "ref_doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "chroma:document": "<Tip>\n\nTo see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/image-segmentation)\n\n</Tip>\n\n\n### Load SceneParse150 dataset\n\nStart by loading a smaller subset of the SceneParse150 dataset from the \ud83e\udd17 Datasets library. This'll give you a chance to experiment and make sure everything works before spending more time training on the full dataset.\n\n```py\n>>> from datasets import load_dataset\n\n>>> ds = load_dataset(\"scene_parse_150\", split=\"train[:50]\")\n```\n\nSplit the dataset's `train` split into a train and test set with the [`~datasets.Dataset.train_test_split`] method:\n\n```py\n>>> ds = ds.train_test_split(test_size=0.2)\n>>> train_ds = ds[\"train\"]\n>>> test_ds = ds[\"test\"]\n```\n\nThen take a look at an example:\n\n```py\n>>> train_ds[0]\n{'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=512x683 at 0x7F9B0C201F90>,\n 'annotation': <PIL.PngImagePlugin.PngImageFile image mode=L size=512x683 at 0x7F9B0C201DD0>,\n 'scene_category': 368}\n\n# view the image\n>>> train_ds[0][\"image\"]\n```\n\n- `image`: a PIL image of the scene.\n- `annotation`: a PIL image of the segmentation map, which is also the model's target.\n- `scene_category`: a category id that describes the image scene like \"kitchen\" or \"office\". In this guide, you'll only need `image` and `annotation`, both of which are PIL images.\n\nYou'll also want to create a dictionary that maps a label id to a label class which will be useful when you set up the model later. Download the mappings from the Hub and create the `id2label` and `label2id` dictionaries:\n\n```py\n>>> import json\n>>> from pathlib import Path\n>>> from huggingface_hub import hf_hub_download\n\n>>> repo_id = \"huggingface/label-files\"\n>>> filename = \"ade20k-id2label.json\"\n>>> id2label = json.loads(Path(hf_hub_download(repo_id, filename, repo_type=\"dataset\")).read_text())\n>>> id2label = {int(k): v for k, v in id2label.items()}\n>>> label2id = {v: k for k, v in id2label.items()}\n>>> num_labels = len(id2label)\n```\n\n#### Custom dataset\n\nYou could also create and use your own dataset if you prefer to train with the [run_semantic_segmentation.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/semantic-segmentation/run_semantic_segmentation.py) script instead of a notebook instance."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6c747df0-bcae-4263-9f19-9c6d3c09a8f9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"6c747df0-bcae-4263-9f19-9c6d3c09a8f9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2c93072b-42b4-4c50-bac8-8f2f8d5f938a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"44e5c046-1784-4fe3-8bda-cece1b68d30b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"6b6290eec4cf3f4dcf021af475cdd0db68fafac2a081cea325d85bac3780327b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"969eadb6-7288-4ae8-87b6-e917446fc07d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"42d94ed0b47c326fd19a6ff0cea5f0ad6679e4b4386896470be65edb34dbec02\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8710, \"end_char_idx\": 11901, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "ref_doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "chroma:document": "- `annotation`: a PIL image of the segmentation map, which is also the model's target.\n- `scene_category`: a category id that describes the image scene like \"kitchen\" or \"office\". In this guide, you'll only need `image` and `annotation`, both of which are PIL images.\n\nYou'll also want to create a dictionary that maps a label id to a label class which will be useful when you set up the model later. Download the mappings from the Hub and create the `id2label` and `label2id` dictionaries:\n\n```py\n>>> import json\n>>> from pathlib import Path\n>>> from huggingface_hub import hf_hub_download\n\n>>> repo_id = \"huggingface/label-files\"\n>>> filename = \"ade20k-id2label.json\"\n>>> id2label = json.loads(Path(hf_hub_download(repo_id, filename, repo_type=\"dataset\")).read_text())\n>>> id2label = {int(k): v for k, v in id2label.items()}\n>>> label2id = {v: k for k, v in id2label.items()}\n>>> num_labels = len(id2label)\n```\n\n#### Custom dataset\n\nYou could also create and use your own dataset if you prefer to train with the [run_semantic_segmentation.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/semantic-segmentation/run_semantic_segmentation.py) script instead of a notebook instance. The script requires:\n\n1. a [`~datasets.DatasetDict`] with two [`~datasets.Image`] columns, \"image\" and \"label\"\n\n     ```py\n     from datasets import Dataset, DatasetDict, Image\n\n     image_paths_train = [\"path/to/image_1.jpg/jpg\", \"path/to/image_2.jpg/jpg\", ..., \"path/to/image_n.jpg/jpg\"]\n     label_paths_train = [\"path/to/annotation_1.png\", \"path/to/annotation_2.png\", ..., \"path/to/annotation_n.png\"]\n\n     image_paths_validation = [...]\n     label_paths_validation = [...]\n\n     def create_dataset(image_paths, label_paths):\n         dataset = Dataset.from_dict({\"image\": sorted(image_paths),\n                                     \"label\": sorted(label_paths)})\n         dataset = dataset.cast_column(\"image\", Image())\n         dataset = dataset.cast_column(\"label\", Image())\n         return dataset\n\n     # step 1: create Dataset objects\n     train_dataset = create_dataset(image_paths_train, label_paths_train)\n     validation_dataset = create_dataset(image_paths_validation, label_paths_validation)\n\n     # step 2: create DatasetDict\n     dataset = DatasetDict({\n          \"train\": train_dataset,\n          \"validation\": validation_dataset,\n          }\n     )\n\n     # step 3: push to Hub (assumes you have ran the huggingface-cli login command in a terminal/notebook)\n     dataset.push_to_hub(\"your-name/dataset-repo\")\n\n     # optionally, you can push to a private repo on the Hub\n     # dataset.push_to_hub(\"name of repo on the hub\", private=True)\n     ```\n\n2. an id2label dictionary mapping the class integers to their class names\n\n     ```py\n     import json\n     # simple example\n     id2label = {0: 'cat', 1: 'dog'}\n     with open('id2label.json', 'w') as fp:\n     json.dump(id2label, fp)\n     ```\n\nAs an example, take a look at this [example dataset](https://huggingface.co/datasets/nielsr/ade20k-demo) which was created with the steps shown above.\n\n### Preprocess\n\nThe next step is to load a SegFormer image processor to prepare the images and annotations for the model."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 969eadb6-7288-4ae8-87b6-e917446fc07d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"969eadb6-7288-4ae8-87b6-e917446fc07d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2c93072b-42b4-4c50-bac8-8f2f8d5f938a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6c747df0-bcae-4263-9f19-9c6d3c09a8f9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"aa7869a99e353b4c272ba282649499def73dfcb786cb358bcc096639f9bfc973\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3b8f1056-23d6-4b14-9469-f281df739699\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0c9d533a75316f71ec98e4ecc3c9aa81d70fea132a66dc3e87d91e1ed3be6646\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11780, \"end_char_idx\": 14827, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "ref_doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "chroma:document": "### Preprocess\n\nThe next step is to load a SegFormer image processor to prepare the images and annotations for the model. Some datasets, like this one, use the zero-index as the background class. However, the background class isn't actually included in the 150 classes, so you'll need to set `do_reduce_labels=True` to subtract one from all the labels. The zero-index is replaced by `255` so it's ignored by SegFormer's loss function:\n\n```py\n>>> from transformers import AutoImageProcessor\n\n>>> checkpoint = \"nvidia/mit-b0\"\n>>> image_processor = AutoImageProcessor.from_pretrained(checkpoint, do_reduce_labels=True)\n```\n\n<frameworkcontent>\n<pt>\n\nIt is common to apply some data augmentations to an image dataset to make a model more robust against overfitting. In this guide, you'll use the [`ColorJitter`](https://pytorch.org/vision/stable/generated/torchvision.transforms.ColorJitter.html) function from [torchvision](https://pytorch.org/vision/stable/index.html) to randomly change the color properties of an image, but you can also use any image library you like.\n\n```py\n>>> from torchvision.transforms import ColorJitter\n\n>>> jitter = ColorJitter(brightness=0.25, contrast=0.25, saturation=0.25, hue=0.1)\n```\n\nNow create two preprocessing functions to prepare the images and annotations for the model. These functions convert the images into `pixel_values` and annotations to `labels`. For the training set, `jitter` is applied before providing the images to the image processor. For the test set, the image processor crops and normalizes the `images`, and only crops the `labels` because no data augmentation is applied during testing.\n\n```py\n>>> def train_transforms(example_batch):\n...     images = [jitter(x) for x in example_batch[\"image\"]]\n...     labels = [x for x in example_batch[\"annotation\"]]\n...     inputs = image_processor(images, labels)\n...     return inputs\n\n\n>>> def val_transforms(example_batch):\n...     images = [x for x in example_batch[\"image\"]]\n...     labels = [x for x in example_batch[\"annotation\"]]\n...     inputs = image_processor(images, labels)\n...     return inputs\n```\n\nTo apply the `jitter` over the entire dataset, use the \ud83e\udd17 Datasets [`~datasets.Dataset.set_transform`] function. The transform is applied on the fly which is faster and consumes less disk space:\n\n```py\n>>> train_ds.set_transform(train_transforms)\n>>> test_ds.set_transform(val_transforms)\n```\n\n</pt>\n</frameworkcontent>\n\n<frameworkcontent>\n<tf>\nIt is common to apply some data augmentations to an image dataset to make a model more robust against overfitting.\nIn this guide, you'll use [`tf.image`](https://www.tensorflow.org/api_docs/python/tf/image) to randomly change the color properties of an image, but you can also use any image\nlibrary you like.\nDefine two separate transformation functions:\n- training data transformations that include image augmentation\n- validation data transformations that only transpose the images, since computer vision models in \ud83e\udd17 Transformers expect channels-first layout\n\n```py\n>>> import tensorflow as tf"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 3b8f1056-23d6-4b14-9469-f281df739699
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"3b8f1056-23d6-4b14-9469-f281df739699\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2c93072b-42b4-4c50-bac8-8f2f8d5f938a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"969eadb6-7288-4ae8-87b6-e917446fc07d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"0d08270beb704ce04c29d4470ee78a61335bad2e1bfb8310d66a8a615c20a488\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5126e8a7-accd-477e-9b14-70f8add1840d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"32889705aca16e332ef49ef69a1e05e88bdfef77fdf4ac66a6fa3b27f58b8bb9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13087, \"end_char_idx\": 15955, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "ref_doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "chroma:document": "These functions convert the images into `pixel_values` and annotations to `labels`. For the training set, `jitter` is applied before providing the images to the image processor. For the test set, the image processor crops and normalizes the `images`, and only crops the `labels` because no data augmentation is applied during testing.\n\n```py\n>>> def train_transforms(example_batch):\n...     images = [jitter(x) for x in example_batch[\"image\"]]\n...     labels = [x for x in example_batch[\"annotation\"]]\n...     inputs = image_processor(images, labels)\n...     return inputs\n\n\n>>> def val_transforms(example_batch):\n...     images = [x for x in example_batch[\"image\"]]\n...     labels = [x for x in example_batch[\"annotation\"]]\n...     inputs = image_processor(images, labels)\n...     return inputs\n```\n\nTo apply the `jitter` over the entire dataset, use the \ud83e\udd17 Datasets [`~datasets.Dataset.set_transform`] function. The transform is applied on the fly which is faster and consumes less disk space:\n\n```py\n>>> train_ds.set_transform(train_transforms)\n>>> test_ds.set_transform(val_transforms)\n```\n\n</pt>\n</frameworkcontent>\n\n<frameworkcontent>\n<tf>\nIt is common to apply some data augmentations to an image dataset to make a model more robust against overfitting.\nIn this guide, you'll use [`tf.image`](https://www.tensorflow.org/api_docs/python/tf/image) to randomly change the color properties of an image, but you can also use any image\nlibrary you like.\nDefine two separate transformation functions:\n- training data transformations that include image augmentation\n- validation data transformations that only transpose the images, since computer vision models in \ud83e\udd17 Transformers expect channels-first layout\n\n```py\n>>> import tensorflow as tf\n\n\n>>> def aug_transforms(image):\n...     image = tf.keras.utils.img_to_array(image)\n...     image = tf.image.random_brightness(image, 0.25)\n...     image = tf.image.random_contrast(image, 0.5, 2.0)\n...     image = tf.image.random_saturation(image, 0.75, 1.25)\n...     image = tf.image.random_hue(image, 0.1)\n...     image = tf.transpose(image, (2, 0, 1))\n...     return image\n\n\n>>> def transforms(image):\n...     image = tf.keras.utils.img_to_array(image)\n...     image = tf.transpose(image, (2, 0, 1))\n...     return image\n```\n\nNext, create two preprocessing functions to prepare batches of images and annotations for the model. These functions apply\nthe image transformations and use the earlier loaded `image_processor` to convert the images into `pixel_values` and\nannotations to `labels`. `ImageProcessor` also takes care of resizing and normalizing the images.\n\n```py\n>>> def train_transforms(example_batch):\n...     images = [aug_transforms(x.convert(\"RGB\")) for x in example_batch[\"image\"]]\n...     labels = [x for x in example_batch[\"annotation\"]]\n...     inputs = image_processor(images, labels)\n...     return inputs"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 5126e8a7-accd-477e-9b14-70f8add1840d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"5126e8a7-accd-477e-9b14-70f8add1840d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2c93072b-42b4-4c50-bac8-8f2f8d5f938a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3b8f1056-23d6-4b14-9469-f281df739699\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"24a96b9c6e47a6a34ce21abf93dc701d6e2e255c230f69f59bc6608b6e77b5c3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"365fa7ba-97a3-4db0-92d7-050cb6ce1582\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f64919519d54ee46dcc789813149f5d5cf857cdbd88e3dde70f071cc4d69c7f8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14830, \"end_char_idx\": 18339, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "ref_doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "chroma:document": ">>> def aug_transforms(image):\n...     image = tf.keras.utils.img_to_array(image)\n...     image = tf.image.random_brightness(image, 0.25)\n...     image = tf.image.random_contrast(image, 0.5, 2.0)\n...     image = tf.image.random_saturation(image, 0.75, 1.25)\n...     image = tf.image.random_hue(image, 0.1)\n...     image = tf.transpose(image, (2, 0, 1))\n...     return image\n\n\n>>> def transforms(image):\n...     image = tf.keras.utils.img_to_array(image)\n...     image = tf.transpose(image, (2, 0, 1))\n...     return image\n```\n\nNext, create two preprocessing functions to prepare batches of images and annotations for the model. These functions apply\nthe image transformations and use the earlier loaded `image_processor` to convert the images into `pixel_values` and\nannotations to `labels`. `ImageProcessor` also takes care of resizing and normalizing the images.\n\n```py\n>>> def train_transforms(example_batch):\n...     images = [aug_transforms(x.convert(\"RGB\")) for x in example_batch[\"image\"]]\n...     labels = [x for x in example_batch[\"annotation\"]]\n...     inputs = image_processor(images, labels)\n...     return inputs\n\n\n>>> def val_transforms(example_batch):\n...     images = [transforms(x.convert(\"RGB\")) for x in example_batch[\"image\"]]\n...     labels = [x for x in example_batch[\"annotation\"]]\n...     inputs = image_processor(images, labels)\n...     return inputs\n```\n\nTo apply the preprocessing transformations over the entire dataset, use the \ud83e\udd17 Datasets [`~datasets.Dataset.set_transform`] function.\nThe transform is applied on the fly which is faster and consumes less disk space:\n\n```py\n>>> train_ds.set_transform(train_transforms)\n>>> test_ds.set_transform(val_transforms)\n```\n</tf>\n</frameworkcontent>\n\n### Evaluate\n\nIncluding a metric during training is often helpful for evaluating your model's performance. You can quickly load an evaluation method with the \ud83e\udd17 [Evaluate](https://huggingface.co/docs/evaluate/index) library. For this task, load the [mean Intersection over Union](https://huggingface.co/spaces/evaluate-metric/accuracy) (IoU) metric (see the \ud83e\udd17 Evaluate [quick tour](https://huggingface.co/docs/evaluate/a_quick_tour) to learn more about how to load and compute a metric):\n\n```py\n>>> import evaluate\n\n>>> metric = evaluate.load(\"mean_iou\")\n```\n\nThen create a function to [`~evaluate.EvaluationModule.compute`] the metrics. Your predictions need to be converted to\nlogits first, and then reshaped to match the size of the labels before you can call [`~evaluate.EvaluationModule.compute`]:\n\n<frameworkcontent>\n<pt>\n\n```py\n>>> import numpy as np\n>>> import torch\n>>> from torch import nn\n\n>>> def compute_metrics(eval_pred):\n...     with torch.no_grad():\n...         logits, labels = eval_pred\n...         logits_tensor = torch.from_numpy(logits)\n...         logits_tensor = nn.functional.interpolate(\n...             logits_tensor,\n...             size=labels.shape[-2:],\n...             mode=\"bilinear\",\n...             align_corners=False,\n...         ).argmax(dim=1)\n\n...         pred_labels = logits_tensor.detach().cpu().numpy()\n...         metrics = metric.compute(\n...             predictions=pred_labels,\n...             references=labels,\n...             num_labels=num_labels,\n...             ignore_index=255,\n...             reduce_labels=False,\n...         )\n...         for key, value in metrics.items():\n...             if isinstance(value, np.ndarray):\n...                 metrics[key] = value.tolist()\n...         return metrics\n```\n\n</pt>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 365fa7ba-97a3-4db0-92d7-050cb6ce1582
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"365fa7ba-97a3-4db0-92d7-050cb6ce1582\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2c93072b-42b4-4c50-bac8-8f2f8d5f938a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5126e8a7-accd-477e-9b14-70f8add1840d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"33c991f84f091f412b0d91f7bb70a35f8ca8b77f7d1a2221faa1253eb1db12ff\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"592e9db3-5761-4cda-9d06-735559b62a94\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ee45fd45fe330e33ea671d3741c48858a1725e958bad49742727131c00d3fac7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 18342, \"end_char_idx\": 20884, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "ref_doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "chroma:document": "<frameworkcontent>\n<tf>\n\n```py\n>>> def compute_metrics(eval_pred):\n...     logits, labels = eval_pred\n...     logits = tf.transpose(logits, perm=[0, 2, 3, 1])\n...     logits_resized = tf.image.resize(\n...         logits,\n...         size=tf.shape(labels)[1:],\n...         method=\"bilinear\",\n...     )\n\n...     pred_labels = tf.argmax(logits_resized, axis=-1)\n...     metrics = metric.compute(\n...         predictions=pred_labels,\n...         references=labels,\n...         num_labels=num_labels,\n...         ignore_index=-1,\n...         reduce_labels=image_processor.do_reduce_labels,\n...     )\n\n...     per_category_accuracy = metrics.pop(\"per_category_accuracy\").tolist()\n...     per_category_iou = metrics.pop(\"per_category_iou\").tolist()\n\n...     metrics.update({f\"accuracy_{id2label[i]}\": v for i, v in enumerate(per_category_accuracy)})\n...     metrics.update({f\"iou_{id2label[i]}\": v for i, v in enumerate(per_category_iou)})\n...     return {\"val_\" + k: v for k, v in metrics.items()}\n```\n\n</tf>\n</frameworkcontent>\n\nYour `compute_metrics` function is ready to go now, and you'll return to it when you setup your training.\n\n### Train\n<frameworkcontent>\n<pt>\n<Tip>\n\nIf you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#finetune-with-trainer)!\n\n</Tip>\n\nYou're ready to start training your model now! Load SegFormer with [`AutoModelForSemanticSegmentation`], and pass the model the mapping between label ids and label classes:\n\n```py\n>>> from transformers import AutoModelForSemanticSegmentation, TrainingArguments, Trainer\n\n>>> model = AutoModelForSemanticSegmentation.from_pretrained(checkpoint, id2label=id2label, label2id=label2id)\n```\n\nAt this point, only three steps remain:\n\n1. Define your training hyperparameters in [`TrainingArguments`]. It is important you don't remove unused columns because this'll drop the `image` column. Without the `image` column, you can't create `pixel_values`. Set `remove_unused_columns=False` to prevent this behavior! The only other required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the IoU metric and save the training checkpoint.\n2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.\n3. Call [`~Trainer.train`] to finetune your model."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 592e9db3-5761-4cda-9d06-735559b62a94
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"592e9db3-5761-4cda-9d06-735559b62a94\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2c93072b-42b4-4c50-bac8-8f2f8d5f938a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"365fa7ba-97a3-4db0-92d7-050cb6ce1582\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"aab139918b1c21b544f60d4e39a5acc5b3ad675b1896b056747981c6fa69c115\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"87d4fb6e-1ba3-43f3-a707-f4c6d3b98974\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"99650b7cf77a92208f6a77c57a65980fa003762a9a1fe65c26224f0e021c9099\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 19473, \"end_char_idx\": 22350, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "ref_doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "chroma:document": "### Train\n<frameworkcontent>\n<pt>\n<Tip>\n\nIf you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#finetune-with-trainer)!\n\n</Tip>\n\nYou're ready to start training your model now! Load SegFormer with [`AutoModelForSemanticSegmentation`], and pass the model the mapping between label ids and label classes:\n\n```py\n>>> from transformers import AutoModelForSemanticSegmentation, TrainingArguments, Trainer\n\n>>> model = AutoModelForSemanticSegmentation.from_pretrained(checkpoint, id2label=id2label, label2id=label2id)\n```\n\nAt this point, only three steps remain:\n\n1. Define your training hyperparameters in [`TrainingArguments`]. It is important you don't remove unused columns because this'll drop the `image` column. Without the `image` column, you can't create `pixel_values`. Set `remove_unused_columns=False` to prevent this behavior! The only other required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the IoU metric and save the training checkpoint.\n2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.\n3. Call [`~Trainer.train`] to finetune your model.\n\n```py\n>>> training_args = TrainingArguments(\n...     output_dir=\"segformer-b0-scene-parse-150\",\n...     learning_rate=6e-5,\n...     num_train_epochs=50,\n...     per_device_train_batch_size=2,\n...     per_device_eval_batch_size=2,\n...     save_total_limit=3,\n...     eval_strategy=\"steps\",\n...     save_strategy=\"steps\",\n...     save_steps=20,\n...     eval_steps=20,\n...     logging_steps=1,\n...     eval_accumulation_steps=5,\n...     remove_unused_columns=False,\n...     push_to_hub=True,\n... )\n\n>>> trainer = Trainer(\n...     model=model,\n...     args=training_args,\n...     train_dataset=train_ds,\n...     eval_dataset=test_ds,\n...     compute_metrics=compute_metrics,\n... )\n\n>>> trainer.train()\n```\n\nOnce training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:\n\n```py\n>>> trainer.push_to_hub()\n```\n</pt>\n</frameworkcontent>\n\n<frameworkcontent>\n<tf>\n<Tip>\n\nIf you are unfamiliar with fine-tuning a model with Keras, check out the [basic tutorial](./training#train-a-tensorflow-model-with-keras) first!\n\n</Tip>\n\nTo fine-tune a model in TensorFlow, follow these steps:\n1. Define the training hyperparameters, and set up an optimizer and a learning rate schedule.\n2. Instantiate a pretrained model.\n3. Convert a \ud83e\udd17 Dataset to a `tf.data.Dataset`.\n4. Compile your model.\n5. Add callbacks to calculate metrics and upload your model to \ud83e\udd17 Hub\n6. Use the `fit()` method to run the training."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 87d4fb6e-1ba3-43f3-a707-f4c6d3b98974
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"87d4fb6e-1ba3-43f3-a707-f4c6d3b98974\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2c93072b-42b4-4c50-bac8-8f2f8d5f938a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"592e9db3-5761-4cda-9d06-735559b62a94\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"e8d9d759fea8cd60ef4148356d4bba788fe62b2a841f312e0f3f0d9fe1937586\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f0327594-ce4d-40d9-88cd-0411c25f6d3d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7029ad31362defd4ebe54fe57f2d3d563e0cab9abf5c0c5e242e680f5e19f722\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 20834, \"end_char_idx\": 23366, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "ref_doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "chroma:document": "3. Call [`~Trainer.train`] to finetune your model.\n\n```py\n>>> training_args = TrainingArguments(\n...     output_dir=\"segformer-b0-scene-parse-150\",\n...     learning_rate=6e-5,\n...     num_train_epochs=50,\n...     per_device_train_batch_size=2,\n...     per_device_eval_batch_size=2,\n...     save_total_limit=3,\n...     eval_strategy=\"steps\",\n...     save_strategy=\"steps\",\n...     save_steps=20,\n...     eval_steps=20,\n...     logging_steps=1,\n...     eval_accumulation_steps=5,\n...     remove_unused_columns=False,\n...     push_to_hub=True,\n... )\n\n>>> trainer = Trainer(\n...     model=model,\n...     args=training_args,\n...     train_dataset=train_ds,\n...     eval_dataset=test_ds,\n...     compute_metrics=compute_metrics,\n... )\n\n>>> trainer.train()\n```\n\nOnce training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:\n\n```py\n>>> trainer.push_to_hub()\n```\n</pt>\n</frameworkcontent>\n\n<frameworkcontent>\n<tf>\n<Tip>\n\nIf you are unfamiliar with fine-tuning a model with Keras, check out the [basic tutorial](./training#train-a-tensorflow-model-with-keras) first!\n\n</Tip>\n\nTo fine-tune a model in TensorFlow, follow these steps:\n1. Define the training hyperparameters, and set up an optimizer and a learning rate schedule.\n2. Instantiate a pretrained model.\n3. Convert a \ud83e\udd17 Dataset to a `tf.data.Dataset`.\n4. Compile your model.\n5. Add callbacks to calculate metrics and upload your model to \ud83e\udd17 Hub\n6. Use the `fit()` method to run the training.\n\nStart by defining the hyperparameters, optimizer and learning rate schedule:\n\n```py\n>>> from transformers import create_optimizer\n\n>>> batch_size = 2\n>>> num_epochs = 50\n>>> num_train_steps = len(train_ds) * num_epochs\n>>> learning_rate = 6e-5\n>>> weight_decay_rate = 0.01\n\n>>> optimizer, lr_schedule = create_optimizer(\n...     init_lr=learning_rate,\n...     num_train_steps=num_train_steps,\n...     weight_decay_rate=weight_decay_rate,\n...     num_warmup_steps=0,\n... )\n```\n\nThen, load SegFormer with [`TFAutoModelForSemanticSegmentation`] along with the label mappings, and compile it with the\noptimizer. Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:\n\n```py\n>>> from transformers import TFAutoModelForSemanticSegmentation\n\n>>> model = TFAutoModelForSemanticSegmentation.from_pretrained(\n...     checkpoint,\n...     id2label=id2label,\n...     label2id=label2id,\n... )\n>>> model.compile(optimizer=optimizer)  # No loss argument!"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f0327594-ce4d-40d9-88cd-0411c25f6d3d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"f0327594-ce4d-40d9-88cd-0411c25f6d3d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2c93072b-42b4-4c50-bac8-8f2f8d5f938a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"87d4fb6e-1ba3-43f3-a707-f4c6d3b98974\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"65218c2ea38b83146cb36a8927bb2d78a8d5a1d2f373600b1943939059f8e5d4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e0e0a6f8-2d17-469c-95d0-8219f61660d5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9a49d4c667aab8e8cff7116217554b6f2008b61206e7c039915cb2d739b64817\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 21972, \"end_char_idx\": 25213, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "ref_doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "chroma:document": "</Tip>\n\nTo fine-tune a model in TensorFlow, follow these steps:\n1. Define the training hyperparameters, and set up an optimizer and a learning rate schedule.\n2. Instantiate a pretrained model.\n3. Convert a \ud83e\udd17 Dataset to a `tf.data.Dataset`.\n4. Compile your model.\n5. Add callbacks to calculate metrics and upload your model to \ud83e\udd17 Hub\n6. Use the `fit()` method to run the training.\n\nStart by defining the hyperparameters, optimizer and learning rate schedule:\n\n```py\n>>> from transformers import create_optimizer\n\n>>> batch_size = 2\n>>> num_epochs = 50\n>>> num_train_steps = len(train_ds) * num_epochs\n>>> learning_rate = 6e-5\n>>> weight_decay_rate = 0.01\n\n>>> optimizer, lr_schedule = create_optimizer(\n...     init_lr=learning_rate,\n...     num_train_steps=num_train_steps,\n...     weight_decay_rate=weight_decay_rate,\n...     num_warmup_steps=0,\n... )\n```\n\nThen, load SegFormer with [`TFAutoModelForSemanticSegmentation`] along with the label mappings, and compile it with the\noptimizer. Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:\n\n```py\n>>> from transformers import TFAutoModelForSemanticSegmentation\n\n>>> model = TFAutoModelForSemanticSegmentation.from_pretrained(\n...     checkpoint,\n...     id2label=id2label,\n...     label2id=label2id,\n... )\n>>> model.compile(optimizer=optimizer)  # No loss argument!\n```\n\nConvert your datasets to the `tf.data.Dataset` format using the [`~datasets.Dataset.to_tf_dataset`] and the [`DefaultDataCollator`]:\n\n```py\n>>> from transformers import DefaultDataCollator\n\n>>> data_collator = DefaultDataCollator(return_tensors=\"tf\")\n\n>>> tf_train_dataset = train_ds.to_tf_dataset(\n...     columns=[\"pixel_values\", \"label\"],\n...     shuffle=True,\n...     batch_size=batch_size,\n...     collate_fn=data_collator,\n... )\n\n>>> tf_eval_dataset = test_ds.to_tf_dataset(\n...     columns=[\"pixel_values\", \"label\"],\n...     shuffle=True,\n...     batch_size=batch_size,\n...     collate_fn=data_collator,\n... )\n```\n\nTo compute the accuracy from the predictions and push your model to the \ud83e\udd17 Hub, use [Keras callbacks](../main_classes/keras_callbacks).\nPass your `compute_metrics` function to [`KerasMetricCallback`],\nand use the [`PushToHubCallback`] to upload the model:\n\n```py\n>>> from transformers.keras_callbacks import KerasMetricCallback, PushToHubCallback\n\n>>> metric_callback = KerasMetricCallback(\n...     metric_fn=compute_metrics, eval_dataset=tf_eval_dataset, batch_size=batch_size, label_cols=[\"labels\"]\n... )\n\n>>> push_to_hub_callback = PushToHubCallback(output_dir=\"scene_segmentation\", tokenizer=image_processor)\n\n>>> callbacks = [metric_callback, push_to_hub_callback]\n```\n\nFinally, you are ready to train your model! Call `fit()` with your training and validation datasets, the number of epochs,\nand your callbacks to fine-tune the model:\n\n```py\n>>> model.fit(\n...     tf_train_dataset,\n...     validation_data=tf_eval_dataset,\n...     callbacks=callbacks,\n...     epochs=num_epochs,\n... )\n```\n\nCongratulations! You have fine-tuned your model and shared it on the \ud83e\udd17 Hub. You can now use it for inference!\n</tf>\n</frameworkcontent>\n\n### Inference\n\nGreat, now that you've finetuned a model, you can use it for inference!"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e0e0a6f8-2d17-469c-95d0-8219f61660d5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"e0e0a6f8-2d17-469c-95d0-8219f61660d5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2c93072b-42b4-4c50-bac8-8f2f8d5f938a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f0327594-ce4d-40d9-88cd-0411c25f6d3d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"fe5eefbf1a32d7c8903d578a103ed34e84bb98770f76cd1ff9d5d6f4af66f6d7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bda047c9-652b-4393-8a19-1a625d7acb2f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7b6c332f760af9acedcc19b651959283d44cfbfdb2f81cdbc404410fce0a7921\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 24129, \"end_char_idx\": 25749, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "ref_doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "chroma:document": "Pass your `compute_metrics` function to [`KerasMetricCallback`],\nand use the [`PushToHubCallback`] to upload the model:\n\n```py\n>>> from transformers.keras_callbacks import KerasMetricCallback, PushToHubCallback\n\n>>> metric_callback = KerasMetricCallback(\n...     metric_fn=compute_metrics, eval_dataset=tf_eval_dataset, batch_size=batch_size, label_cols=[\"labels\"]\n... )\n\n>>> push_to_hub_callback = PushToHubCallback(output_dir=\"scene_segmentation\", tokenizer=image_processor)\n\n>>> callbacks = [metric_callback, push_to_hub_callback]\n```\n\nFinally, you are ready to train your model! Call `fit()` with your training and validation datasets, the number of epochs,\nand your callbacks to fine-tune the model:\n\n```py\n>>> model.fit(\n...     tf_train_dataset,\n...     validation_data=tf_eval_dataset,\n...     callbacks=callbacks,\n...     epochs=num_epochs,\n... )\n```\n\nCongratulations! You have fine-tuned your model and shared it on the \ud83e\udd17 Hub. You can now use it for inference!\n</tf>\n</frameworkcontent>\n\n### Inference\n\nGreat, now that you've finetuned a model, you can use it for inference!\n\nReload the dataset and load an image for inference.\n\n```py\n>>> from datasets import load_dataset\n\n>>> ds = load_dataset(\"scene_parse_150\", split=\"train[:50]\")\n>>> ds = ds.train_test_split(test_size=0.2)\n>>> test_ds = ds[\"test\"]\n>>> image = ds[\"test\"][0][\"image\"]\n>>> image\n```\n\n<div class=\"flex justify-center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/semantic-seg-image.png\" alt=\"Image of bedroom\"/>\n</div>\n\n<frameworkcontent>\n<pt>\n\nWe will now see how to infer without a pipeline."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: bda047c9-652b-4393-8a19-1a625d7acb2f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"bda047c9-652b-4393-8a19-1a625d7acb2f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2c93072b-42b4-4c50-bac8-8f2f8d5f938a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e0e0a6f8-2d17-469c-95d0-8219f61660d5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"c5ad54cd33f49a77c0a9c777d832b8d5cc1519c815b4031a48d750a2c273684a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c99cb049-7b42-4a40-bb1a-d6f981b7dac5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"654a3cc059ce368316d40b0afbe0907eb3c10a56d33a23321eaed3c70be3c3ea\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 24129, \"end_char_idx\": 27390, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "ref_doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "chroma:document": "Pass your `compute_metrics` function to [`KerasMetricCallback`],\nand use the [`PushToHubCallback`] to upload the model:\n\n```py\n>>> from transformers.keras_callbacks import KerasMetricCallback, PushToHubCallback\n\n>>> metric_callback = KerasMetricCallback(\n...     metric_fn=compute_metrics, eval_dataset=tf_eval_dataset, batch_size=batch_size, label_cols=[\"labels\"]\n... )\n\n>>> push_to_hub_callback = PushToHubCallback(output_dir=\"scene_segmentation\", tokenizer=image_processor)\n\n>>> callbacks = [metric_callback, push_to_hub_callback]\n```\n\nFinally, you are ready to train your model! Call `fit()` with your training and validation datasets, the number of epochs,\nand your callbacks to fine-tune the model:\n\n```py\n>>> model.fit(\n...     tf_train_dataset,\n...     validation_data=tf_eval_dataset,\n...     callbacks=callbacks,\n...     epochs=num_epochs,\n... )\n```\n\nCongratulations! You have fine-tuned your model and shared it on the \ud83e\udd17 Hub. You can now use it for inference!\n</tf>\n</frameworkcontent>\n\n### Inference\n\nGreat, now that you've finetuned a model, you can use it for inference!\n\nReload the dataset and load an image for inference.\n\n```py\n>>> from datasets import load_dataset\n\n>>> ds = load_dataset(\"scene_parse_150\", split=\"train[:50]\")\n>>> ds = ds.train_test_split(test_size=0.2)\n>>> test_ds = ds[\"test\"]\n>>> image = ds[\"test\"][0][\"image\"]\n>>> image\n```\n\n<div class=\"flex justify-center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/semantic-seg-image.png\" alt=\"Image of bedroom\"/>\n</div>\n\n<frameworkcontent>\n<pt>\n\nWe will now see how to infer without a pipeline. Process the image with an image processor and place the `pixel_values` on a GPU:\n\n```py\n>>> device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")  # use GPU if available, otherwise use a CPU\n>>> encoding = image_processor(image, return_tensors=\"pt\")\n>>> pixel_values = encoding.pixel_values.to(device)\n```\n\nPass your input to the model and return the `logits`:\n\n```py\n>>> outputs = model(pixel_values=pixel_values)\n>>> logits = outputs.logits.cpu()\n```\n\nNext, rescale the logits to the original image size:\n\n```py\n>>> upsampled_logits = nn.functional.interpolate(\n...     logits,\n...     size=image.size[::-1],\n...     mode=\"bilinear\",\n...     align_corners=False,\n... )\n\n>>> pred_seg = upsampled_logits.argmax(dim=1)[0]\n```\n\n</pt>\n</frameworkcontent>\n\n<frameworkcontent>\n<tf>\nLoad an image processor to preprocess the image and return the input as TensorFlow tensors:\n\n```py\n>>> from transformers import AutoImageProcessor\n\n>>> image_processor = AutoImageProcessor.from_pretrained(\"MariaK/scene_segmentation\")\n>>> inputs = image_processor(image, return_tensors=\"tf\")\n```\n\nPass your input to the model and return the `logits`:\n\n```py\n>>> from transformers import TFAutoModelForSemanticSegmentation\n\n>>> model = TFAutoModelForSemanticSegmentation.from_pretrained(\"MariaK/scene_segmentation\")\n>>> logits = model(**inputs).logits\n```\n\nNext, rescale the logits to the original image size and apply argmax on the class dimension:\n```py\n>>> logits = tf.transpose(logits, [0, 2, 3, 1])\n\n>>> upsampled_logits = tf.image.resize(\n...     logits,\n...     # We reverse the shape of `image` because `image.size` returns width and height."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: c99cb049-7b42-4a40-bb1a-d6f981b7dac5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"c99cb049-7b42-4a40-bb1a-d6f981b7dac5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2c93072b-42b4-4c50-bac8-8f2f8d5f938a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bda047c9-652b-4393-8a19-1a625d7acb2f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"7081e74c336b02ae379a944be528f76fd07cad6ca41171517c8a11ba526095e0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"20b55705-bff3-47f4-8e53-5310b10cbed1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c76b62c8021d262578030a4d6397245ab30aeafd02a7d38df60b9c3d9e7f4d1c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 25750, \"end_char_idx\": 28315, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "ref_doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "chroma:document": "Process the image with an image processor and place the `pixel_values` on a GPU:\n\n```py\n>>> device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")  # use GPU if available, otherwise use a CPU\n>>> encoding = image_processor(image, return_tensors=\"pt\")\n>>> pixel_values = encoding.pixel_values.to(device)\n```\n\nPass your input to the model and return the `logits`:\n\n```py\n>>> outputs = model(pixel_values=pixel_values)\n>>> logits = outputs.logits.cpu()\n```\n\nNext, rescale the logits to the original image size:\n\n```py\n>>> upsampled_logits = nn.functional.interpolate(\n...     logits,\n...     size=image.size[::-1],\n...     mode=\"bilinear\",\n...     align_corners=False,\n... )\n\n>>> pred_seg = upsampled_logits.argmax(dim=1)[0]\n```\n\n</pt>\n</frameworkcontent>\n\n<frameworkcontent>\n<tf>\nLoad an image processor to preprocess the image and return the input as TensorFlow tensors:\n\n```py\n>>> from transformers import AutoImageProcessor\n\n>>> image_processor = AutoImageProcessor.from_pretrained(\"MariaK/scene_segmentation\")\n>>> inputs = image_processor(image, return_tensors=\"tf\")\n```\n\nPass your input to the model and return the `logits`:\n\n```py\n>>> from transformers import TFAutoModelForSemanticSegmentation\n\n>>> model = TFAutoModelForSemanticSegmentation.from_pretrained(\"MariaK/scene_segmentation\")\n>>> logits = model(**inputs).logits\n```\n\nNext, rescale the logits to the original image size and apply argmax on the class dimension:\n```py\n>>> logits = tf.transpose(logits, [0, 2, 3, 1])\n\n>>> upsampled_logits = tf.image.resize(\n...     logits,\n...     # We reverse the shape of `image` because `image.size` returns width and height.\n...     image.size[::-1],\n... )\n\n>>> pred_seg = tf.math.argmax(upsampled_logits, axis=-1)[0]\n```\n\n</tf>\n</frameworkcontent>\n\nTo visualize the results, load the [dataset color palette](https://github.com/tensorflow/models/blob/3f1ca33afe3c1631b733ea7e40c294273b9e406d/research/deeplab/utils/get_dataset_colormap.py#L51) as `ade_palette()` that maps each class to their RGB values.\n\n```py\ndef ade_palette():\n  return np.asarray([\n      [0, 0, 0],\n      [120, 120, 120],\n      [180, 120, 120],\n      [6, 230, 230],\n      [80, 50, 50],\n      [4, 200, 3],\n      [120, 120, 80],\n      [140, 140, 140],\n      [204, 5, 255],\n      [230, 230, 230],\n      [4, 250, 7],\n      [224, 5, 255],\n      [235, 255, 7],\n      [150, 5, 61],\n      [120, 120, 70],\n      [8, 255, 51],\n      [255, 6, 82],\n      [143, 255, 140],\n      [204, 255, 4],\n      [255, 51, 7],\n      [204, 70, 3],\n      [0, 102, 200],\n      [61, 230, 250],\n      [255, 6,"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 20b55705-bff3-47f4-8e53-5310b10cbed1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"20b55705-bff3-47f4-8e53-5310b10cbed1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2c93072b-42b4-4c50-bac8-8f2f8d5f938a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c99cb049-7b42-4a40-bb1a-d6f981b7dac5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"b210651afc2275c70c55697fe703c97d5ca985c691b2515dcf5b30ecad6d9245\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c2623d96-a2fa-47f0-b5ad-5d02885ee04b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8a2243df478a5ee3610ff7bb2c6dfe62393290c9fe20d73c944b8f4849da0583\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 27391, \"end_char_idx\": 29063, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "ref_doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "chroma:document": "...     image.size[::-1],\n... )\n\n>>> pred_seg = tf.math.argmax(upsampled_logits, axis=-1)[0]\n```\n\n</tf>\n</frameworkcontent>\n\nTo visualize the results, load the [dataset color palette](https://github.com/tensorflow/models/blob/3f1ca33afe3c1631b733ea7e40c294273b9e406d/research/deeplab/utils/get_dataset_colormap.py#L51) as `ade_palette()` that maps each class to their RGB values.\n\n```py\ndef ade_palette():\n  return np.asarray([\n      [0, 0, 0],\n      [120, 120, 120],\n      [180, 120, 120],\n      [6, 230, 230],\n      [80, 50, 50],\n      [4, 200, 3],\n      [120, 120, 80],\n      [140, 140, 140],\n      [204, 5, 255],\n      [230, 230, 230],\n      [4, 250, 7],\n      [224, 5, 255],\n      [235, 255, 7],\n      [150, 5, 61],\n      [120, 120, 70],\n      [8, 255, 51],\n      [255, 6, 82],\n      [143, 255, 140],\n      [204, 255, 4],\n      [255, 51, 7],\n      [204, 70, 3],\n      [0, 102, 200],\n      [61, 230, 250],\n      [255, 6, 51],\n      [11, 102, 255],\n      [255, 7, 71],\n      [255, 9, 224],\n      [9, 7, 230],\n      [220, 220, 220],\n      [255, 9, 92],\n      [112, 9, 255],\n      [8, 255, 214],\n      [7, 255, 224],\n      [255, 184, 6],\n      [10, 255, 71],\n      [255, 41, 10],\n      [7, 255, 255],\n      [224, 255, 8],\n      [102, 8, 255],\n      [255, 61, 6],\n      [255, 194, 7],\n      [255, 122, 8],\n      [0, 255, 20],\n      [255, 8, 41],\n      [255, 5, 153],\n      [6, 51, 255],\n      [235, 12, 255],\n      [160, 150, 20],\n      [0, 163, 255],\n      [140, 140, 140],\n      [250, 10, 15],\n      [20, 255, 0],\n      [31, 255, 0],\n      [255, 31, 0],\n      [255, 224, 0],\n      [153, 255, 0],\n      [0, 0, 255],\n      [255, 71, 0],\n      [0, 235, 255],\n      [0, 173,"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: c2623d96-a2fa-47f0-b5ad-5d02885ee04b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"c2623d96-a2fa-47f0-b5ad-5d02885ee04b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2c93072b-42b4-4c50-bac8-8f2f8d5f938a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"20b55705-bff3-47f4-8e53-5310b10cbed1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"4d233c881ff7a7be0af0e1bfda61a66d05e8d1a5465a510ebb03d40931a169dc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"23032ed7-c8f4-49b8-85f8-95b3acd358c5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"65527d728b9d77eff1556d1fdd0bdec48b564fcdf2bf2e970072adae620c89b2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 28313, \"end_char_idx\": 29781, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "ref_doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "chroma:document": "6, 51],\n      [11, 102, 255],\n      [255, 7, 71],\n      [255, 9, 224],\n      [9, 7, 230],\n      [220, 220, 220],\n      [255, 9, 92],\n      [112, 9, 255],\n      [8, 255, 214],\n      [7, 255, 224],\n      [255, 184, 6],\n      [10, 255, 71],\n      [255, 41, 10],\n      [7, 255, 255],\n      [224, 255, 8],\n      [102, 8, 255],\n      [255, 61, 6],\n      [255, 194, 7],\n      [255, 122, 8],\n      [0, 255, 20],\n      [255, 8, 41],\n      [255, 5, 153],\n      [6, 51, 255],\n      [235, 12, 255],\n      [160, 150, 20],\n      [0, 163, 255],\n      [140, 140, 140],\n      [250, 10, 15],\n      [20, 255, 0],\n      [31, 255, 0],\n      [255, 31, 0],\n      [255, 224, 0],\n      [153, 255, 0],\n      [0, 0, 255],\n      [255, 71, 0],\n      [0, 235, 255],\n      [0, 173, 255],\n      [31, 0, 255],\n      [11, 200, 200],\n      [255, 82, 0],\n      [0, 255, 245],\n      [0, 61, 255],\n      [0, 255, 112],\n      [0, 255, 133],\n      [255, 0, 0],\n      [255, 163, 0],\n      [255, 102, 0],\n      [194, 255, 0],\n      [0, 143, 255],\n      [51, 255, 0],\n      [0, 82, 255],\n      [0, 255, 41],\n      [0, 255, 173],\n      [10, 0, 255],\n      [173, 255, 0],\n      [0, 255, 153],\n      [255, 92, 0],\n      [255, 0, 255],\n      [255, 0, 245],\n      [255, 0, 102],\n      [255, 173, 0],\n      [255, 0, 20],\n      [255, 184, 184],\n      [0, 31, 255],\n      [0, 255, 61],\n      [0, 71, 255],\n      [255, 0, 204],\n      [0, 255, 194],\n      [0, 255, 82],\n      [0, 10, 255],\n      [0, 112, 255],\n      [51,"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 23032ed7-c8f4-49b8-85f8-95b3acd358c5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"23032ed7-c8f4-49b8-85f8-95b3acd358c5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2c93072b-42b4-4c50-bac8-8f2f8d5f938a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c2623d96-a2fa-47f0-b5ad-5d02885ee04b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"a7eea018a59e9df652e1d36c2338730de37257cdd20f842f82c2bea79a299056\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"46e74bac-9f70-4b12-ab7c-6df655878c5c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a1e219b327f2f232ebb37965203f0aa2e13799f883996ce398724fc32606ed52\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 29038, \"end_char_idx\": 30512, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "ref_doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "chroma:document": "235, 255],\n      [0, 173, 255],\n      [31, 0, 255],\n      [11, 200, 200],\n      [255, 82, 0],\n      [0, 255, 245],\n      [0, 61, 255],\n      [0, 255, 112],\n      [0, 255, 133],\n      [255, 0, 0],\n      [255, 163, 0],\n      [255, 102, 0],\n      [194, 255, 0],\n      [0, 143, 255],\n      [51, 255, 0],\n      [0, 82, 255],\n      [0, 255, 41],\n      [0, 255, 173],\n      [10, 0, 255],\n      [173, 255, 0],\n      [0, 255, 153],\n      [255, 92, 0],\n      [255, 0, 255],\n      [255, 0, 245],\n      [255, 0, 102],\n      [255, 173, 0],\n      [255, 0, 20],\n      [255, 184, 184],\n      [0, 31, 255],\n      [0, 255, 61],\n      [0, 71, 255],\n      [255, 0, 204],\n      [0, 255, 194],\n      [0, 255, 82],\n      [0, 10, 255],\n      [0, 112, 255],\n      [51, 0, 255],\n      [0, 194, 255],\n      [0, 122, 255],\n      [0, 255, 163],\n      [255, 153, 0],\n      [0, 255, 10],\n      [255, 112, 0],\n      [143, 255, 0],\n      [82, 0, 255],\n      [163, 255, 0],\n      [255, 235, 0],\n      [8, 184, 170],\n      [133, 0, 255],\n      [0, 255, 92],\n      [184, 0, 255],\n      [255, 0, 31],\n      [0, 184, 255],\n      [0, 214, 255],\n      [255, 0, 112],\n      [92, 255, 0],\n      [0, 224, 255],\n      [112, 224, 255],\n      [70, 184, 160],\n      [163, 0, 255],\n      [153, 0, 255],\n      [71, 255, 0],\n      [255, 0, 163],\n      [255, 204, 0],\n      [255, 0, 143],\n      [0, 255, 235],\n      [133, 255, 0],\n      [255, 0, 235],\n      [245, 0, 255],\n      [255, 0, 122],\n      [255, 245, 0],\n      [10,"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 46e74bac-9f70-4b12-ab7c-6df655878c5c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"46e74bac-9f70-4b12-ab7c-6df655878c5c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2c93072b-42b4-4c50-bac8-8f2f8d5f938a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"23032ed7-c8f4-49b8-85f8-95b3acd358c5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"419c9751cc3eea09f8fb17102abd2990196f60f772d5328fbcbfe85cf96742a7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ab9ca066-8e54-4b9d-a067-70dbcccb842e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8820c311ab122c4c017562b695f18cc3f2cf85f41d7f71e754c9f88f6b5c3036\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "ref_doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "chroma:document": "112, 255],\n      [51, 0, 255],\n      [0, 194, 255],\n      [0, 122, 255],\n      [0, 255, 163],\n      [255, 153, 0],\n      [0, 255, 10],\n      [255, 112, 0],\n      [143, 255, 0],\n      [82, 0, 255],\n      [163, 255, 0],\n      [255, 235, 0],\n      [8, 184, 170],\n      [133, 0, 255],\n      [0, 255, 92],\n      [184, 0, 255],\n      [255, 0, 31],\n      [0, 184, 255],\n      [0, 214, 255],\n      [255, 0, 112],\n      [92, 255, 0],\n      [0, 224, 255],\n      [112, 224, 255],\n      [70, 184, 160],\n      [163, 0, 255],\n      [153, 0, 255],\n      [71, 255, 0],\n      [255, 0, 163],\n      [255, 204, 0],\n      [255, 0, 143],\n      [0, 255, 235],\n      [133, 255, 0],\n      [255, 0, 235],\n      [245, 0, 255],\n      [255, 0, 122],\n      [255, 245, 0],\n      [10, 190, 212],\n      [214, 255, 0],\n      [0, 204, 255],\n      [20, 0, 255],\n      [255, 255, 0],\n      [0, 153, 255],\n      [0, 41, 255],\n      [0, 255, 204],\n      [41, 0, 255],\n      [41, 255, 0],\n      [173, 0, 255],\n      [0, 245, 255],\n      [71, 0, 255],\n      [122, 0, 255],\n      [0, 255, 184],\n      [0, 92, 255],\n      [184, 255, 0],\n      [0, 133, 255],\n      [255, 214, 0],\n      [25, 194, 194],\n      [102, 255, 0],\n      [92, 0, 255],\n  ])\n```\n\nThen you can combine and plot your image and the predicted segmentation map:\n\n```py\n>>> import matplotlib.pyplot as plt\n>>> import numpy as np\n\n>>> color_seg = np.zeros((pred_seg.shape[0], pred_seg.shape[1], 3), dtype=np.uint8)\n>>> palette = np.array(ade_palette())\n>>> for label, color in enumerate(palette):\n.     color_seg[pred_seg == label, :] = color\n>>> color_seg = color_seg[. ::-1]  # convert to BGR\n\n>>> img = np.array(image) * 0.5 + color_seg * 0."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: ab9ca066-8e54-4b9d-a067-70dbcccb842e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"ab9ca066-8e54-4b9d-a067-70dbcccb842e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2c93072b-42b4-4c50-bac8-8f2f8d5f938a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"46e74bac-9f70-4b12-ab7c-6df655878c5c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/semantic_segmentation\", \"title\": \"Image Segmentation\", \"tokens\": 8705, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"9489197698d5a35cd49a44d8cf073705044b39c44004ca34f708e170daa19e87\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "ref_doc_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "chroma:document": "0, 122],\n      [255, 245, 0],\n      [10, 190, 212],\n      [214, 255, 0],\n      [0, 204, 255],\n      [20, 0, 255],\n      [255, 255, 0],\n      [0, 153, 255],\n      [0, 41, 255],\n      [0, 255, 204],\n      [41, 0, 255],\n      [41, 255, 0],\n      [173, 0, 255],\n      [0, 245, 255],\n      [71, 0, 255],\n      [122, 0, 255],\n      [0, 255, 184],\n      [0, 92, 255],\n      [184, 255, 0],\n      [0, 133, 255],\n      [255, 214, 0],\n      [25, 194, 194],\n      [102, 255, 0],\n      [92, 0, 255],\n  ])\n```\n\nThen you can combine and plot your image and the predicted segmentation map:\n\n```py\n>>> import matplotlib.pyplot as plt\n>>> import numpy as np\n\n>>> color_seg = np.zeros((pred_seg.shape[0], pred_seg.shape[1], 3), dtype=np.uint8)\n>>> palette = np.array(ade_palette())\n>>> for label, color in enumerate(palette):\n.     color_seg[pred_seg == label, :] = color\n>>> color_seg = color_seg[. ::-1]  # convert to BGR\n\n>>> img = np.array(image) * 0.5 + color_seg * 0.5  # plot the image with the segmentation map\n>>> img = img.astype(np.uint8)\n\n>>> plt.figure(figsize=(15, 10))\n>>> plt.imshow(img)\n>>> plt.show()\n```\n\n<div class=\"flex justify-center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/semantic-seg-preds.png\" alt=\"Image of bedroom overlaid with segmentation map\"/>\n</div>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: ac30ffe2-b3d5-4792-901f-ba1f72bb80cb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/image_captioning", "title": "Image captioning", "tokens": 1900, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ac30ffe2-b3d5-4792-901f-ba1f72bb80cb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_captioning\", \"title\": \"Image captioning\", \"tokens\": 1900, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"32a9f40d-0fe2-4e78-90a6-12c4c4bd5c22\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_captioning\", \"title\": \"Image captioning\", \"tokens\": 1900, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"91c7f9bca67878ef01d86e644d523e7e994743923414f046881da32189729c0f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f77dd0eb-f1f0-4d37-94fd-de6ddf4f841e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c469c64c686b38b6063bb340fd071e6777e3cc9aa72c916bf687c08dc232085d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2354, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "32a9f40d-0fe2-4e78-90a6-12c4c4bd5c22", "doc_id": "32a9f40d-0fe2-4e78-90a6-12c4c4bd5c22", "ref_doc_id": "32a9f40d-0fe2-4e78-90a6-12c4c4bd5c22", "chroma:document": "# Image captioning\n\n[[open-in-colab]]\n\nImage captioning is the task of predicting a caption for a given image. Common real world applications of it include\naiding visually impaired people that can help them navigate through different situations. Therefore, image captioning\nhelps to improve content accessibility for people by describing images to them.\n\nThis guide will show you how to:\n\n* Fine-tune an image captioning model.\n* Use the fine-tuned model for inference. \n\nBefore you begin, make sure you have all the necessary libraries installed:\n\n```bash\npip install transformers datasets evaluate -q\npip install jiwer -q\n```\n\nWe encourage you to log in to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to log in:\n\n\n```python\nfrom huggingface_hub import notebook_login\n\nnotebook_login()\n```\n\n## Load the Pok\u00e9mon BLIP captions dataset\n\nUse the \ud83e\udd17 Dataset library to load a dataset that consists of {image-caption} pairs. To create your own image captioning dataset\nin PyTorch, you can follow [this notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/GIT/Fine_tune_GIT_on_an_image_captioning_dataset.ipynb). \n\n\n```python\nfrom datasets import load_dataset\n\nds = load_dataset(\"lambdalabs/pokemon-blip-captions\")\nds\n```\n```bash\nDatasetDict({\n    train: Dataset({\n        features: ['image', 'text'],\n        num_rows: 833\n    })\n})\n```\n\nThe dataset has two features, `image` and `text`.\n\n<Tip>\n\nMany image captioning datasets contain multiple captions per image. In those cases, a common strategy is to randomly sample a caption amongst the available ones during training. \n\n</Tip>\n\nSplit the dataset\u2019s train split into a train and test set with the [`~datasets.Dataset.train_test_split`] method:\n\n\n```python\nds = ds[\"train\"].train_test_split(test_size=0.1)\ntrain_ds = ds[\"train\"]\ntest_ds = ds[\"test\"]\n```\n\nLet's visualize a couple of samples from the training set. \n\n\n```python\nfrom textwrap import wrap\nimport matplotlib.pyplot as plt\nimport numpy as np\n\n\ndef plot_images(images, captions):\n    plt.figure(figsize=(20, 20))\n    for i in range(len(images)):\n        ax = plt.subplot(1, len(images), i + 1)\n        caption = captions[i]\n        caption = \"\\n\".join(wrap(caption, 12))\n        plt.title(caption)\n        plt.imshow(images[i])\n        plt.axis(\"off\")"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f77dd0eb-f1f0-4d37-94fd-de6ddf4f841e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/image_captioning", "title": "Image captioning", "tokens": 1900, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f77dd0eb-f1f0-4d37-94fd-de6ddf4f841e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_captioning\", \"title\": \"Image captioning\", \"tokens\": 1900, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"32a9f40d-0fe2-4e78-90a6-12c4c4bd5c22\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_captioning\", \"title\": \"Image captioning\", \"tokens\": 1900, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"91c7f9bca67878ef01d86e644d523e7e994743923414f046881da32189729c0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ac30ffe2-b3d5-4792-901f-ba1f72bb80cb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_captioning\", \"title\": \"Image captioning\", \"tokens\": 1900, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"63084dee75d6f3ef67a51b598853a291a78897a734d023accc3622aab45067a0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5359143f-0a61-4bcc-9158-354e7eead63c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f49ccd7582a09ead0a203446ff9618ecaa9be430bd6807e6068076d0b6a013ae\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 789, \"end_char_idx\": 3953, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "32a9f40d-0fe2-4e78-90a6-12c4c4bd5c22", "doc_id": "32a9f40d-0fe2-4e78-90a6-12c4c4bd5c22", "ref_doc_id": "32a9f40d-0fe2-4e78-90a6-12c4c4bd5c22", "chroma:document": "```python\nfrom huggingface_hub import notebook_login\n\nnotebook_login()\n```\n\n## Load the Pok\u00e9mon BLIP captions dataset\n\nUse the \ud83e\udd17 Dataset library to load a dataset that consists of {image-caption} pairs. To create your own image captioning dataset\nin PyTorch, you can follow [this notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/GIT/Fine_tune_GIT_on_an_image_captioning_dataset.ipynb). \n\n\n```python\nfrom datasets import load_dataset\n\nds = load_dataset(\"lambdalabs/pokemon-blip-captions\")\nds\n```\n```bash\nDatasetDict({\n    train: Dataset({\n        features: ['image', 'text'],\n        num_rows: 833\n    })\n})\n```\n\nThe dataset has two features, `image` and `text`.\n\n<Tip>\n\nMany image captioning datasets contain multiple captions per image. In those cases, a common strategy is to randomly sample a caption amongst the available ones during training. \n\n</Tip>\n\nSplit the dataset\u2019s train split into a train and test set with the [`~datasets.Dataset.train_test_split`] method:\n\n\n```python\nds = ds[\"train\"].train_test_split(test_size=0.1)\ntrain_ds = ds[\"train\"]\ntest_ds = ds[\"test\"]\n```\n\nLet's visualize a couple of samples from the training set. \n\n\n```python\nfrom textwrap import wrap\nimport matplotlib.pyplot as plt\nimport numpy as np\n\n\ndef plot_images(images, captions):\n    plt.figure(figsize=(20, 20))\n    for i in range(len(images)):\n        ax = plt.subplot(1, len(images), i + 1)\n        caption = captions[i]\n        caption = \"\\n\".join(wrap(caption, 12))\n        plt.title(caption)\n        plt.imshow(images[i])\n        plt.axis(\"off\")\n\n\nsample_images_to_visualize = [np.array(train_ds[i][\"image\"]) for i in range(5)]\nsample_captions = [train_ds[i][\"text\"] for i in range(5)]\nplot_images(sample_images_to_visualize, sample_captions)\n```\n    \n<div class=\"flex justify-center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/sample_training_images_image_cap.png\" alt=\"Sample training images\"/>\n</div>\n\n## Preprocess the dataset\n\nSince the dataset has two modalities (image and text), the pre-processing pipeline will preprocess images and the captions.\n\nTo do so, load the processor class associated with the model you are about to fine-tune. \n\n```python\nfrom transformers import AutoProcessor\n\ncheckpoint = \"microsoft/git-base\"\nprocessor = AutoProcessor.from_pretrained(checkpoint)\n```\n\nThe processor will internally pre-process the image (which includes resizing, and pixel scaling) and tokenize the caption. \n\n```python\ndef transforms(example_batch):\n    images = [x for x in example_batch[\"image\"]]\n    captions = [x for x in example_batch[\"text\"]]\n    inputs = processor(images=images, text=captions, padding=\"max_length\")\n    inputs.update({\"labels\": inputs[\"input_ids\"]})\n    return inputs\n\n\ntrain_ds.set_transform(transforms)\ntest_ds.set_transform(transforms)\n```\n\nWith the dataset ready, you can now set up the model for fine-tuning. \n\n## Load a base model\n\nLoad the [\"microsoft/git-base\"](https://huggingface.co/microsoft/git-base) into a [`AutoModelForCausalLM`](https://huggingface.co/docs/transformers/model_doc/auto#transformers.AutoModelForCausalLM) object."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 5359143f-0a61-4bcc-9158-354e7eead63c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/image_captioning", "title": "Image captioning", "tokens": 1900, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5359143f-0a61-4bcc-9158-354e7eead63c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_captioning\", \"title\": \"Image captioning\", \"tokens\": 1900, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"32a9f40d-0fe2-4e78-90a6-12c4c4bd5c22\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_captioning\", \"title\": \"Image captioning\", \"tokens\": 1900, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"91c7f9bca67878ef01d86e644d523e7e994743923414f046881da32189729c0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f77dd0eb-f1f0-4d37-94fd-de6ddf4f841e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_captioning\", \"title\": \"Image captioning\", \"tokens\": 1900, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a449ca6da5e097c1f201cd14a1cc2d886e1eed3bfd8267933b816b49bfae5acc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"659b88a0-009c-45c8-b919-2405bf890fe1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1f1680a6ba28c4f5e499e7141a37120323fd0a735223608b73bd21b35475c047\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2357, \"end_char_idx\": 5146, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "32a9f40d-0fe2-4e78-90a6-12c4c4bd5c22", "doc_id": "32a9f40d-0fe2-4e78-90a6-12c4c4bd5c22", "ref_doc_id": "32a9f40d-0fe2-4e78-90a6-12c4c4bd5c22", "chroma:document": "sample_images_to_visualize = [np.array(train_ds[i][\"image\"]) for i in range(5)]\nsample_captions = [train_ds[i][\"text\"] for i in range(5)]\nplot_images(sample_images_to_visualize, sample_captions)\n```\n    \n<div class=\"flex justify-center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/sample_training_images_image_cap.png\" alt=\"Sample training images\"/>\n</div>\n\n## Preprocess the dataset\n\nSince the dataset has two modalities (image and text), the pre-processing pipeline will preprocess images and the captions.\n\nTo do so, load the processor class associated with the model you are about to fine-tune. \n\n```python\nfrom transformers import AutoProcessor\n\ncheckpoint = \"microsoft/git-base\"\nprocessor = AutoProcessor.from_pretrained(checkpoint)\n```\n\nThe processor will internally pre-process the image (which includes resizing, and pixel scaling) and tokenize the caption. \n\n```python\ndef transforms(example_batch):\n    images = [x for x in example_batch[\"image\"]]\n    captions = [x for x in example_batch[\"text\"]]\n    inputs = processor(images=images, text=captions, padding=\"max_length\")\n    inputs.update({\"labels\": inputs[\"input_ids\"]})\n    return inputs\n\n\ntrain_ds.set_transform(transforms)\ntest_ds.set_transform(transforms)\n```\n\nWith the dataset ready, you can now set up the model for fine-tuning. \n\n## Load a base model\n\nLoad the [\"microsoft/git-base\"](https://huggingface.co/microsoft/git-base) into a [`AutoModelForCausalLM`](https://huggingface.co/docs/transformers/model_doc/auto#transformers.AutoModelForCausalLM) object.\n\n\n```python\nfrom transformers import AutoModelForCausalLM\n\nmodel = AutoModelForCausalLM.from_pretrained(checkpoint)\n```\n\n## Evaluate\n\nImage captioning models are typically evaluated with the [Rouge Score](https://huggingface.co/spaces/evaluate-metric/rouge) or [Word Error Rate](https://huggingface.co/spaces/evaluate-metric/wer). For this guide, you will use the Word Error Rate (WER). \n\nWe use the \ud83e\udd17 Evaluate library to do so. For potential limitations and other gotchas of the WER, refer to [this guide](https://huggingface.co/spaces/evaluate-metric/wer). \n\n\n```python\nfrom evaluate import load\nimport torch\n\nwer = load(\"wer\")\n\n\ndef compute_metrics(eval_pred):\n    logits, labels = eval_pred\n    predicted = logits.argmax(-1)\n    decoded_labels = processor.batch_decode(labels, skip_special_tokens=True)\n    decoded_predictions = processor.batch_decode(predicted, skip_special_tokens=True)\n    wer_score = wer.compute(predictions=decoded_predictions, references=decoded_labels)\n    return {\"wer_score\": wer_score}\n```\n\n## Train!\n\nNow, you are ready to start fine-tuning the model. You will use the \ud83e\udd17 [`Trainer`] for this. \n\nFirst, define the training arguments using [`TrainingArguments`]."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 659b88a0-009c-45c8-b919-2405bf890fe1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/image_captioning", "title": "Image captioning", "tokens": 1900, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"659b88a0-009c-45c8-b919-2405bf890fe1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_captioning\", \"title\": \"Image captioning\", \"tokens\": 1900, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"32a9f40d-0fe2-4e78-90a6-12c4c4bd5c22\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_captioning\", \"title\": \"Image captioning\", \"tokens\": 1900, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"91c7f9bca67878ef01d86e644d523e7e994743923414f046881da32189729c0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5359143f-0a61-4bcc-9158-354e7eead63c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_captioning\", \"title\": \"Image captioning\", \"tokens\": 1900, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1da929197ddfb4dce336d1b008e4ce305de8d79028943830ef5ea9c780dd479f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ec3a9f8c-a654-42ae-8393-0d402a1cb619\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d47d08787f0017b2e6b4f1b639a84997b6effc52688873af0686e1c9e9071aed\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3580, \"end_char_idx\": 6411, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "32a9f40d-0fe2-4e78-90a6-12c4c4bd5c22", "doc_id": "32a9f40d-0fe2-4e78-90a6-12c4c4bd5c22", "ref_doc_id": "32a9f40d-0fe2-4e78-90a6-12c4c4bd5c22", "chroma:document": "train_ds.set_transform(transforms)\ntest_ds.set_transform(transforms)\n```\n\nWith the dataset ready, you can now set up the model for fine-tuning. \n\n## Load a base model\n\nLoad the [\"microsoft/git-base\"](https://huggingface.co/microsoft/git-base) into a [`AutoModelForCausalLM`](https://huggingface.co/docs/transformers/model_doc/auto#transformers.AutoModelForCausalLM) object.\n\n\n```python\nfrom transformers import AutoModelForCausalLM\n\nmodel = AutoModelForCausalLM.from_pretrained(checkpoint)\n```\n\n## Evaluate\n\nImage captioning models are typically evaluated with the [Rouge Score](https://huggingface.co/spaces/evaluate-metric/rouge) or [Word Error Rate](https://huggingface.co/spaces/evaluate-metric/wer). For this guide, you will use the Word Error Rate (WER). \n\nWe use the \ud83e\udd17 Evaluate library to do so. For potential limitations and other gotchas of the WER, refer to [this guide](https://huggingface.co/spaces/evaluate-metric/wer). \n\n\n```python\nfrom evaluate import load\nimport torch\n\nwer = load(\"wer\")\n\n\ndef compute_metrics(eval_pred):\n    logits, labels = eval_pred\n    predicted = logits.argmax(-1)\n    decoded_labels = processor.batch_decode(labels, skip_special_tokens=True)\n    decoded_predictions = processor.batch_decode(predicted, skip_special_tokens=True)\n    wer_score = wer.compute(predictions=decoded_predictions, references=decoded_labels)\n    return {\"wer_score\": wer_score}\n```\n\n## Train!\n\nNow, you are ready to start fine-tuning the model. You will use the \ud83e\udd17 [`Trainer`] for this. \n\nFirst, define the training arguments using [`TrainingArguments`].\n\n\n```python\nfrom transformers import TrainingArguments, Trainer\n\nmodel_name = checkpoint.split(\"/\")[1]\n\ntraining_args = TrainingArguments(\n    output_dir=f\"{model_name}-pokemon\",\n    learning_rate=5e-5,\n    num_train_epochs=50,\n    fp16=True,\n    per_device_train_batch_size=32,\n    per_device_eval_batch_size=32,\n    gradient_accumulation_steps=2,\n    save_total_limit=3,\n    eval_strategy=\"steps\",\n    eval_steps=50,\n    save_strategy=\"steps\",\n    save_steps=50,\n    logging_steps=50,\n    remove_unused_columns=False,\n    push_to_hub=True,\n    label_names=[\"labels\"],\n    load_best_model_at_end=True,\n)\n```\n\nThen pass them along with the datasets and the model to \ud83e\udd17 Trainer. \n\n```python\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=train_ds,\n    eval_dataset=test_ds,\n    compute_metrics=compute_metrics,\n)\n```\n\nTo start training, simply call [`~Trainer.train`] on the [`Trainer`] object.\n\n```python \ntrainer.train()\n```\n\nYou should see the training loss drop smoothly as training progresses.\n\nOnce training is completed, share your model to the Hub with the [`~Trainer.push_to_hub`] method so everyone can use your model:\n\n\n```python\ntrainer.push_to_hub()\n```\n\n## Inference\n\nTake a sample image from `test_ds` to test the model."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: ec3a9f8c-a654-42ae-8393-0d402a1cb619
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/image_captioning", "title": "Image captioning", "tokens": 1900, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ec3a9f8c-a654-42ae-8393-0d402a1cb619\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_captioning\", \"title\": \"Image captioning\", \"tokens\": 1900, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"32a9f40d-0fe2-4e78-90a6-12c4c4bd5c22\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_captioning\", \"title\": \"Image captioning\", \"tokens\": 1900, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"91c7f9bca67878ef01d86e644d523e7e994743923414f046881da32189729c0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"659b88a0-009c-45c8-b919-2405bf890fe1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_captioning\", \"title\": \"Image captioning\", \"tokens\": 1900, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6d836cea26a90adaeb76467296216abbe5e6145c18c4aba064226f0d6000dbfb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5149, \"end_char_idx\": 7385, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "32a9f40d-0fe2-4e78-90a6-12c4c4bd5c22", "doc_id": "32a9f40d-0fe2-4e78-90a6-12c4c4bd5c22", "ref_doc_id": "32a9f40d-0fe2-4e78-90a6-12c4c4bd5c22", "chroma:document": "```python\nfrom transformers import TrainingArguments, Trainer\n\nmodel_name = checkpoint.split(\"/\")[1]\n\ntraining_args = TrainingArguments(\n    output_dir=f\"{model_name}-pokemon\",\n    learning_rate=5e-5,\n    num_train_epochs=50,\n    fp16=True,\n    per_device_train_batch_size=32,\n    per_device_eval_batch_size=32,\n    gradient_accumulation_steps=2,\n    save_total_limit=3,\n    eval_strategy=\"steps\",\n    eval_steps=50,\n    save_strategy=\"steps\",\n    save_steps=50,\n    logging_steps=50,\n    remove_unused_columns=False,\n    push_to_hub=True,\n    label_names=[\"labels\"],\n    load_best_model_at_end=True,\n)\n```\n\nThen pass them along with the datasets and the model to \ud83e\udd17 Trainer. \n\n```python\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=train_ds,\n    eval_dataset=test_ds,\n    compute_metrics=compute_metrics,\n)\n```\n\nTo start training, simply call [`~Trainer.train`] on the [`Trainer`] object.\n\n```python \ntrainer.train()\n```\n\nYou should see the training loss drop smoothly as training progresses.\n\nOnce training is completed, share your model to the Hub with the [`~Trainer.push_to_hub`] method so everyone can use your model:\n\n\n```python\ntrainer.push_to_hub()\n```\n\n## Inference\n\nTake a sample image from `test_ds` to test the model.\n\n\n```python\nfrom PIL import Image\nimport requests\n\nurl = \"https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/pokemon.png\"\nimage = Image.open(requests.get(url, stream=True).raw)\nimage\n```\n\n<div class=\"flex justify-center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/test_image_image_cap.png\" alt=\"Test image\"/>\n</div>\n    \nPrepare image for the model.\n\n```python\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n\ninputs = processor(images=image, return_tensors=\"pt\").to(device)\npixel_values = inputs.pixel_values\n```\n\nCall [`generate`] and decode the predictions. \n\n```python\ngenerated_ids = model.generate(pixel_values=pixel_values, max_length=50)\ngenerated_caption = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]\nprint(generated_caption)\n```\n```bash\na drawing of a pink and blue pokemon\n```\n\nLooks like the fine-tuned model generated a pretty good caption!"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f5b93891-1eb4-4782-b415-c8e9ea917cca
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/image_text_to_text", "title": "Image-text-to-text", "tokens": 1936, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f5b93891-1eb4-4782-b415-c8e9ea917cca\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_text_to_text\", \"title\": \"Image-text-to-text\", \"tokens\": 1936, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5200b0cb-8fe5-4e61-97c4-ebc887c150aa\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_text_to_text\", \"title\": \"Image-text-to-text\", \"tokens\": 1936, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e9c2f2f6611fc285f25ce1b46f84a11985fff7798a572e3349601e8720a6d5de\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f87c4f15-e8c6-467a-9b82-bd4e1678299d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"dfff0139cabab5dd294e0f21af1aacc0a2872d281258f9ba9804f68f50435191\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2561, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5200b0cb-8fe5-4e61-97c4-ebc887c150aa", "doc_id": "5200b0cb-8fe5-4e61-97c4-ebc887c150aa", "ref_doc_id": "5200b0cb-8fe5-4e61-97c4-ebc887c150aa", "chroma:document": "# Image-text-to-text\n\n[[open-in-colab]]\n\nImage-text-to-text models, also known as vision language models (VLMs), are language models that take an image input. These models can tackle various tasks, from visual question answering to image segmentation. This task shares many similarities with image-to-text, but\u00a0with some overlapping use cases like image captioning. Image-to-text models only take image inputs and often accomplish a specific task, whereas VLMs take open-ended text and image inputs and are more generalist models.\n\nIn this guide, we provide a brief overview of VLMs and show how to use them with Transformers for inference.\n\nTo begin with, there are multiple types of VLMs:\n- base models used for fine-tuning\n- chat fine-tuned models for conversation\n- instruction fine-tuned models\n\nThis guide focuses on inference with an instruction-tuned model. \n\nLet's begin installing the dependencies.\n\n```bash\npip install -q transformers accelerate flash_attn \n```\n\nLet's initialize the model and the processor. \n\n```python\nfrom transformers import AutoProcessor, Idefics2ForConditionalGeneration\nimport torch\n\ndevice = torch.device(\"cuda\")\nmodel = Idefics2ForConditionalGeneration.from_pretrained(\n    \"HuggingFaceM4/idefics2-8b\",\n    torch_dtype=torch.bfloat16,\n    attn_implementation=\"flash_attention_2\",\n).to(device)\n\nprocessor = AutoProcessor.from_pretrained(\"HuggingFaceM4/idefics2-8b\")\n```\n\nThis model has a [chat template](./chat_templating) that helps user parse chat outputs. Moreover, the model can also accept multiple images as input in a single conversation or message. We will now prepare the inputs. \n\nThe image inputs look like the following.\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/cats.png\" alt=\"Two cats sitting on a net\"/>\n</div>\n\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg\" alt=\"A bee on a pink flower\"/>\n</div>\n\n\n```python\nfrom PIL import Image\nimport requests\n\nimg_urls =[\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/cats.png\",\n           \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg\"]\nimages = [Image.open(requests.get(img_urls[0], stream=True).raw),\n          Image.open(requests.get(img_urls[1], stream=True).raw)]\n```\n\nBelow is an example of the chat template. We can feed conversation turns and the last message as an input by appending it at the end of the template."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f87c4f15-e8c6-467a-9b82-bd4e1678299d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/image_text_to_text", "title": "Image-text-to-text", "tokens": 1936, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f87c4f15-e8c6-467a-9b82-bd4e1678299d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_text_to_text\", \"title\": \"Image-text-to-text\", \"tokens\": 1936, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5200b0cb-8fe5-4e61-97c4-ebc887c150aa\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_text_to_text\", \"title\": \"Image-text-to-text\", \"tokens\": 1936, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e9c2f2f6611fc285f25ce1b46f84a11985fff7798a572e3349601e8720a6d5de\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f5b93891-1eb4-4782-b415-c8e9ea917cca\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_text_to_text\", \"title\": \"Image-text-to-text\", \"tokens\": 1936, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"afc3fac2f189afead04497a5d766e332c068412ac730efba1e48513a73a56145\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"48bc4f69-dee7-495b-bcd5-565659126c20\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fbd0a5c8b592fadba8b90674eb436326c6b9d909a205e7c764876ad2e3e6cc31\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2025, \"end_char_idx\": 4500, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5200b0cb-8fe5-4e61-97c4-ebc887c150aa", "doc_id": "5200b0cb-8fe5-4e61-97c4-ebc887c150aa", "ref_doc_id": "5200b0cb-8fe5-4e61-97c4-ebc887c150aa", "chroma:document": "```python\nfrom PIL import Image\nimport requests\n\nimg_urls =[\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/cats.png\",\n           \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg\"]\nimages = [Image.open(requests.get(img_urls[0], stream=True).raw),\n          Image.open(requests.get(img_urls[1], stream=True).raw)]\n```\n\nBelow is an example of the chat template. We can feed conversation turns and the last message as an input by appending it at the end of the template. \n\n\n```python\nmessages = [\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image\"},\n            {\"type\": \"text\", \"text\": \"What do we see in this image?\"},\n        ]\n    },\n    {\n        \"role\": \"assistant\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"In this image we can see two cats on the nets.\"},\n        ]\n    },\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image\"},\n            {\"type\": \"text\", \"text\": \"And how about this image?\"},\n        ]\n    },       \n]\n```\n\nWe will now call the processors' [`~ProcessorMixin.apply_chat_template`] method to preprocess its output along with the image inputs.\n\n```python\nprompt = processor.apply_chat_template(messages, add_generation_prompt=True)\ninputs = processor(text=prompt, images=[images[0], images[1]], return_tensors=\"pt\").to(device)\n```\n\nWe can now pass the preprocessed inputs to the model.\n\n```python\nwith torch.no_grad():\n    generated_ids = model.generate(**inputs, max_new_tokens=500)\ngenerated_texts = processor.batch_decode(generated_ids, skip_special_tokens=True)\n\nprint(generated_texts)\n## ['User: What do we see in this image? \\nAssistant: In this image we can see two cats on the nets. \\nUser: And how about this image? \\nAssistant: In this image we can see flowers, plants and insect.']\n```\n\n## Streaming\n\nWe can use [text streaming](./generation_strategies#streaming) for a better generation experience. Transformers supports streaming with the [`TextStreamer`] or [`TextIteratorStreamer`] classes. We will use the [`TextIteratorStreamer`] with IDEFICS-8B.\n\nAssume we have an application that keeps chat history and takes in the new user input. We will preprocess the inputs as usual and initialize [`TextIteratorStreamer`] to handle the generation in a separate thread. This allows you to stream the generated text tokens in real-time. Any generation arguments can be passed to [`TextIteratorStreamer`]."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 48bc4f69-dee7-495b-bcd5-565659126c20
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/image_text_to_text", "title": "Image-text-to-text", "tokens": 1936, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"48bc4f69-dee7-495b-bcd5-565659126c20\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_text_to_text\", \"title\": \"Image-text-to-text\", \"tokens\": 1936, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5200b0cb-8fe5-4e61-97c4-ebc887c150aa\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_text_to_text\", \"title\": \"Image-text-to-text\", \"tokens\": 1936, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e9c2f2f6611fc285f25ce1b46f84a11985fff7798a572e3349601e8720a6d5de\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f87c4f15-e8c6-467a-9b82-bd4e1678299d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_text_to_text\", \"title\": \"Image-text-to-text\", \"tokens\": 1936, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fc27437c836012644ef191fb31ea21a13ee5e167f68bb367f5654f8d76088f1b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4503, \"end_char_idx\": 7704, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5200b0cb-8fe5-4e61-97c4-ebc887c150aa", "doc_id": "5200b0cb-8fe5-4e61-97c4-ebc887c150aa", "ref_doc_id": "5200b0cb-8fe5-4e61-97c4-ebc887c150aa", "chroma:document": "```python\nimport time\nfrom transformers import TextIteratorStreamer\nfrom threading import Thread\n\ndef model_inference(\n    user_prompt,\n    chat_history,\n    max_new_tokens,\n    images\n):\n    user_prompt = {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image\"},\n            {\"type\": \"text\", \"text\": user_prompt},\n        ]\n    }\n    chat_history.append(user_prompt)\n    streamer = TextIteratorStreamer(\n        processor.tokenizer,\n        skip_prompt=True,\n        timeout=5.0,\n    )\n\n    generation_args = {\n        \"max_new_tokens\": max_new_tokens,\n        \"streamer\": streamer,\n        \"do_sample\": False\n    }\n\n    # add_generation_prompt=True makes model generate bot response\n    prompt = processor.apply_chat_template(chat_history, add_generation_prompt=True)\n    inputs = processor(\n        text=prompt,\n        images=images,\n        return_tensors=\"pt\",\n    ).to(device)\n    generation_args.update(inputs)\n\n    thread = Thread(\n        target=model.generate,\n        kwargs=generation_args,\n    )\n    thread.start()\n\n    acc_text = \"\"\n    for text_token in streamer:\n        time.sleep(0.04)\n        acc_text += text_token\n        if acc_text.endswith(\"<end_of_utterance>\"):\n            acc_text = acc_text[:-18]\n        yield acc_text\n    \n    thread.join()\n```\n\nNow let's call the `model_inference` function we created and stream the values. \n\n```python\ngenerator = model_inference(\n    user_prompt=\"And what is in this image?\",\n    chat_history=messages,\n    max_new_tokens=100,\n    images=images\n)\n\nfor value in generator:\n  print(value)\n\n# In\n# In this\n# In this image ...\n```\n\n## Fit models in smaller hardware\n\nVLMs are often large and need to be optimized to fit in smaller hardware. Transformers supports many model quantization libraries, and here we will only show int8 quantization with [Quanto](./quantization/quanto#quanto). int8 quantization offers memory improvements up to 75 percent (if all weights are quantized). However it is no free lunch, since 8-bit is not a CUDA-native precision, the weights are quantized back and forth on the fly, which adds up to latency. \n\nFirst, install dependencies.\n\n```bash\npip install -U quanto bitsandbytes\n```\n\nTo quantize a model during loading, we need to first create [`QuantoConfig`]. Then load the model as usual, but pass `quantization_config`\u00a0during model initialization.\n\n```python\nfrom transformers import Idefics2ForConditionalGeneration, AutoTokenizer, QuantoConfig\n\nmodel_id = \"HuggingFaceM4/idefics2-8b\"\nquantization_config = QuantoConfig(weights=\"int8\")\nquantized_model = Idefics2ForConditionalGeneration.from_pretrained(model_id, device_map=\"cuda\", quantization_config=quantization_config)\n```\n\nAnd that's it, we can use the model the same way with no changes. \n\n## Further Reading\n\nHere are some more resources for the image-text-to-text task.\n\n- [Image-text-to-text\u00a0task page](https://huggingface.co/tasks/image-text-to-text) covers model types, use cases, datasets, and more. \n- [Vision Language Models Explained](https://huggingface.co/blog/vlms) is a blog post that covers everything about vision language models and supervised fine-tuning using [TRL](https://huggingface.co/docs/trl/en/index)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 1f9fe395-f72e-49fc-adfd-e9b1958bd3d5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"1f9fe395-f72e-49fc-adfd-e9b1958bd3d5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c67b77e2-0310-4df5-8659-cf6d0618167d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"98e2e437-2985-4250-afca-54da0a4b10e4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6a5369e3d9198abbec82957c2ef70bb255363843ea67c8b218315278721eb334\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2401, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "doc_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "ref_doc_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "chroma:document": "# Object detection\n\n[[open-in-colab]]\n\nObject detection is the computer vision task of detecting instances (such as humans, buildings, or cars) in an image. Object detection models receive an image as input and output\ncoordinates of the bounding boxes and associated labels of the detected objects. An image can contain multiple objects,\neach with its own bounding box and a label (e.g. it can have a car and a building), and each object can\nbe present in different parts of an image (e.g. the image can have several cars).\nThis task is commonly used in autonomous driving for detecting things like pedestrians, road signs, and traffic lights.\nOther applications include counting objects in images, image search, and more.\n\nIn this guide, you will learn how to:\n\n 1. Finetune [DETR](https://huggingface.co/docs/transformers/model_doc/detr), a model that combines a convolutional\n backbone with an encoder-decoder Transformer, on the [CPPE-5](https://huggingface.co/datasets/cppe-5)\n dataset.\n 2. Use your finetuned model for inference.\n\n<Tip>\n\nTo see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/object-detection)\n\n</Tip>\n\nBefore you begin, make sure you have all the necessary libraries installed:\n\n```bash\npip install -q datasets transformers accelerate timm\npip install -q -U albumentations>=1.4.5 torchmetrics pycocotools\n```\n\nYou'll use \ud83e\udd17 Datasets to load a dataset from the Hugging Face Hub, \ud83e\udd17 Transformers to train your model,\nand `albumentations` to augment the data.\n\nWe encourage you to share your model with the community. Log in to your Hugging Face account to upload it to the Hub.\nWhen prompted, enter your token to log in:\n\n```py\n>>> from huggingface_hub import notebook_login\n\n>>> notebook_login()\n```\n\nTo get started, we'll define global constants, namely the model name and image size. For this tutorial, we'll use the conditional DETR model due to its faster convergence. Feel free to select any object detection model available in the `transformers` library.\n\n```py\n>>> MODEL_NAME = \"microsoft/conditional-detr-resnet-50\"  # or \"facebook/detr-resnet-50\"\n>>> IMAGE_SIZE = 480\n```\n\n## Load the CPPE-5 dataset\n\nThe [CPPE-5 dataset](https://huggingface.co/datasets/cppe-5) contains images with\nannotations identifying medical personal protective equipment (PPE) in the context of the COVID-19 pandemic."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 98e2e437-2985-4250-afca-54da0a4b10e4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"98e2e437-2985-4250-afca-54da0a4b10e4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c67b77e2-0310-4df5-8659-cf6d0618167d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1f9fe395-f72e-49fc-adfd-e9b1958bd3d5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"d6f38ba8fbdfced4ca39ad1b5c30477f6e4672eeadb518e179c25fa6edc3d9b1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c5b93b6a-3494-4477-a6c8-ae282a8fe7b8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e42d4eb927780ad0d43cef68cecf1a16d035757554871ad9b7283b7aef6b443e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 993, \"end_char_idx\": 3318, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "doc_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "ref_doc_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "chroma:document": "2. Use your finetuned model for inference.\n\n<Tip>\n\nTo see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/object-detection)\n\n</Tip>\n\nBefore you begin, make sure you have all the necessary libraries installed:\n\n```bash\npip install -q datasets transformers accelerate timm\npip install -q -U albumentations>=1.4.5 torchmetrics pycocotools\n```\n\nYou'll use \ud83e\udd17 Datasets to load a dataset from the Hugging Face Hub, \ud83e\udd17 Transformers to train your model,\nand `albumentations` to augment the data.\n\nWe encourage you to share your model with the community. Log in to your Hugging Face account to upload it to the Hub.\nWhen prompted, enter your token to log in:\n\n```py\n>>> from huggingface_hub import notebook_login\n\n>>> notebook_login()\n```\n\nTo get started, we'll define global constants, namely the model name and image size. For this tutorial, we'll use the conditional DETR model due to its faster convergence. Feel free to select any object detection model available in the `transformers` library.\n\n```py\n>>> MODEL_NAME = \"microsoft/conditional-detr-resnet-50\"  # or \"facebook/detr-resnet-50\"\n>>> IMAGE_SIZE = 480\n```\n\n## Load the CPPE-5 dataset\n\nThe [CPPE-5 dataset](https://huggingface.co/datasets/cppe-5) contains images with\nannotations identifying medical personal protective equipment (PPE) in the context of the COVID-19 pandemic.\n\nStart by loading the dataset and creating a `validation` split from `train`:\n\n```py\n>>> from datasets import load_dataset\n\n>>> cppe5 = load_dataset(\"cppe-5\")\n\n>>> if \"validation\" not in cppe5:\n...     split = cppe5[\"train\"].train_test_split(0.15, seed=1337)\n...     cppe5[\"train\"] = split[\"train\"]\n...     cppe5[\"validation\"] = split[\"test\"]\n\n>>> cppe5\nDatasetDict({\n    train: Dataset({\n        features: ['image_id', 'image', 'width', 'height', 'objects'],\n        num_rows: 850\n    })\n    test: Dataset({\n        features: ['image_id', 'image', 'width', 'height', 'objects'],\n        num_rows: 29\n    })\n    validation: Dataset({\n        features: ['image_id', 'image', 'width', 'height', 'objects'],\n        num_rows: 150\n    })\n})\n```\n\nYou'll see that this dataset has 1000 images for train and validation sets and a test set with 29 images.\n\nTo get familiar with the data, explore what the examples look like."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: c5b93b6a-3494-4477-a6c8-ae282a8fe7b8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"c5b93b6a-3494-4477-a6c8-ae282a8fe7b8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c67b77e2-0310-4df5-8659-cf6d0618167d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"98e2e437-2985-4250-afca-54da0a4b10e4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"fb5e992bfefa7a687936e33d77a88565fb9b8a01c9ba28f2c330dc418ec6161b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b02f4f10-f265-42fc-b777-d90872293f52\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d0a64fc7743877eee49195caa32541c90469e2b61e9763f75539e033815242cf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1887, \"end_char_idx\": 4475, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "doc_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "ref_doc_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "chroma:document": "For this tutorial, we'll use the conditional DETR model due to its faster convergence. Feel free to select any object detection model available in the `transformers` library.\n\n```py\n>>> MODEL_NAME = \"microsoft/conditional-detr-resnet-50\"  # or \"facebook/detr-resnet-50\"\n>>> IMAGE_SIZE = 480\n```\n\n## Load the CPPE-5 dataset\n\nThe [CPPE-5 dataset](https://huggingface.co/datasets/cppe-5) contains images with\nannotations identifying medical personal protective equipment (PPE) in the context of the COVID-19 pandemic.\n\nStart by loading the dataset and creating a `validation` split from `train`:\n\n```py\n>>> from datasets import load_dataset\n\n>>> cppe5 = load_dataset(\"cppe-5\")\n\n>>> if \"validation\" not in cppe5:\n...     split = cppe5[\"train\"].train_test_split(0.15, seed=1337)\n...     cppe5[\"train\"] = split[\"train\"]\n...     cppe5[\"validation\"] = split[\"test\"]\n\n>>> cppe5\nDatasetDict({\n    train: Dataset({\n        features: ['image_id', 'image', 'width', 'height', 'objects'],\n        num_rows: 850\n    })\n    test: Dataset({\n        features: ['image_id', 'image', 'width', 'height', 'objects'],\n        num_rows: 29\n    })\n    validation: Dataset({\n        features: ['image_id', 'image', 'width', 'height', 'objects'],\n        num_rows: 150\n    })\n})\n```\n\nYou'll see that this dataset has 1000 images for train and validation sets and a test set with 29 images.\n\nTo get familiar with the data, explore what the examples look like.\n\n```py\n>>> cppe5[\"train\"][0]\n{\n  'image_id': 366,\n  'image': <PIL.PngImagePlugin.PngImageFile image mode=RGBA size=500x290>,\n  'width': 500,\n  'height': 500,\n  'objects': {\n    'id': [1932, 1933, 1934],\n    'area': [27063, 34200, 32431],\n    'bbox': [[29.0, 11.0, 97.0, 279.0],\n      [201.0, 1.0, 120.0, 285.0],\n      [382.0, 0.0, 113.0, 287.0]],\n    'category': [0, 0, 0]\n  }\n}\n```\n\nThe examples in the dataset have the following fields:\n- `image_id`: the example image id\n- `image`: a `PIL.Image.Image` object containing the image\n- `width`: width of the image\n- `height`: height of the image\n- `objects`: a dictionary containing bounding box metadata for the objects in the image:\n  - `id`: the annotation id\n  - `area`: the area of the bounding box\n  - `bbox`: the object's bounding box (in the [COCO format](https://albumentations.ai/docs/getting_started/bounding_boxes_augmentation/#coco) )\n  - `category`: the object's category, with possible values including `Coverall (0)`, `Face_Shield (1)`, `Gloves (2)`, `Goggles (3)` and `Mask (4)`\n\nYou may notice that the `bbox` field follows the COCO format, which is the format that the DETR model expects."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: b02f4f10-f265-42fc-b777-d90872293f52
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"b02f4f10-f265-42fc-b777-d90872293f52\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c67b77e2-0310-4df5-8659-cf6d0618167d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c5b93b6a-3494-4477-a6c8-ae282a8fe7b8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"4d4a5b58470ff6062871fbb8018f2a047431d7f0c606b47279d78be270b5ee10\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"21cb8211-b140-4bb4-a2d8-62b84dfb0cd0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"76dcc5b53250fae83d0ef8ef2392f35fbdc19b1934b867946bb7c4b697025fe5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3320, \"end_char_idx\": 4760, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "doc_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "ref_doc_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "chroma:document": "```py\n>>> cppe5[\"train\"][0]\n{\n  'image_id': 366,\n  'image': <PIL.PngImagePlugin.PngImageFile image mode=RGBA size=500x290>,\n  'width': 500,\n  'height': 500,\n  'objects': {\n    'id': [1932, 1933, 1934],\n    'area': [27063, 34200, 32431],\n    'bbox': [[29.0, 11.0, 97.0, 279.0],\n      [201.0, 1.0, 120.0, 285.0],\n      [382.0, 0.0, 113.0, 287.0]],\n    'category': [0, 0, 0]\n  }\n}\n```\n\nThe examples in the dataset have the following fields:\n- `image_id`: the example image id\n- `image`: a `PIL.Image.Image` object containing the image\n- `width`: width of the image\n- `height`: height of the image\n- `objects`: a dictionary containing bounding box metadata for the objects in the image:\n  - `id`: the annotation id\n  - `area`: the area of the bounding box\n  - `bbox`: the object's bounding box (in the [COCO format](https://albumentations.ai/docs/getting_started/bounding_boxes_augmentation/#coco) )\n  - `category`: the object's category, with possible values including `Coverall (0)`, `Face_Shield (1)`, `Gloves (2)`, `Goggles (3)` and `Mask (4)`\n\nYou may notice that the `bbox` field follows the COCO format, which is the format that the DETR model expects.\nHowever, the grouping of the fields inside `objects` differs from the annotation format DETR requires. You will\nneed to apply some preprocessing transformations before using this data for training.\n\nTo get an even better understanding of the data, visualize an example in the dataset."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 21cb8211-b140-4bb4-a2d8-62b84dfb0cd0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"21cb8211-b140-4bb4-a2d8-62b84dfb0cd0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c67b77e2-0310-4df5-8659-cf6d0618167d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b02f4f10-f265-42fc-b777-d90872293f52\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"f1143e19fc5aefe896eeb089aa35651e4823c2f7beabacf48633fa7509579910\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"badedd7f-d690-4f73-92a6-829eef17391f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7c29284067bf392a3b88dae6f2f99a9958154e4b108ac43616af81f2d9b5b4ac\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4476, \"end_char_idx\": 7433, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "doc_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "ref_doc_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "chroma:document": "However, the grouping of the fields inside `objects` differs from the annotation format DETR requires. You will\nneed to apply some preprocessing transformations before using this data for training.\n\nTo get an even better understanding of the data, visualize an example in the dataset.\n\n```py\n>>> import numpy as np\n>>> import os\n>>> from PIL import Image, ImageDraw\n\n>>> image = cppe5[\"train\"][2][\"image\"]\n>>> annotations = cppe5[\"train\"][2][\"objects\"]\n>>> draw = ImageDraw.Draw(image)\n\n>>> categories = cppe5[\"train\"].features[\"objects\"].feature[\"category\"].names\n\n>>> id2label = {index: x for index, x in enumerate(categories, start=0)}\n>>> label2id = {v: k for k, v in id2label.items()}\n\n>>> for i in range(len(annotations[\"id\"])):\n...     box = annotations[\"bbox\"][i]\n...     class_idx = annotations[\"category\"][i]\n...     x, y, w, h = tuple(box)\n...     # Check if coordinates are normalized or not\n...     if max(box) > 1.0:\n...         # Coordinates are un-normalized, no need to re-scale them\n...         x1, y1 = int(x), int(y)\n...         x2, y2 = int(x + w), int(y + h)\n...     else:\n...         # Coordinates are normalized, re-scale them\n...         x1 = int(x * width)\n...         y1 = int(y * height)\n...         x2 = int((x + w) * width)\n...         y2 = int((y + h) * height)\n...     draw.rectangle((x, y, x + w, y + h), outline=\"red\", width=1)\n...     draw.text((x, y), id2label[class_idx], fill=\"white\")\n\n>>> image\n```\n<div class=\"flex justify-center\">\n    <img src=\"https://i.imgur.com/oVQb9SF.png\" alt=\"CPPE-5 Image Example\"/>\n</div>\n\n\nTo visualize the bounding boxes with associated labels, you can get the labels from the dataset's metadata, specifically\nthe `category` field.\nYou'll also want to create dictionaries that map a label id to a label class (`id2label`) and the other way around (`label2id`).\nYou can use them later when setting up the model. Including these maps will make your model reusable by others if you share\nit on the Hugging Face Hub. Please note that, the part of above code that draws the bounding boxes assume that it is in `COCO` format `(x_min, y_min, width, height)`. It has to be adjusted to work for other formats like `(x_min, y_min, x_max, y_max)`.\n\nAs a final step of getting familiar with the data, explore it for potential issues. One common problem with datasets for\nobject detection is bounding boxes that \"stretch\" beyond the edge of the image. Such \"runaway\" bounding boxes can raise\nerrors during training and should be addressed. There are a few examples with this issue in this dataset.\nTo keep things simple in this guide, we will set `clip=True` for `BboxParams` in transformations below.\n\n## Preprocess the data\n\nTo finetune a model, you must preprocess the data you plan to use to match precisely the approach used for the pre-trained model.\n[`AutoImageProcessor`] takes care of processing image data to create `pixel_values`, `pixel_mask`, and\n`labels` that a DETR model can train with."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: badedd7f-d690-4f73-92a6-829eef17391f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"badedd7f-d690-4f73-92a6-829eef17391f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c67b77e2-0310-4df5-8659-cf6d0618167d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"21cb8211-b140-4bb4-a2d8-62b84dfb0cd0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"081c3c7d56afcca8c583c9c0d912b4f336db28f1e7091f448695bf4369200b45\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ad89c284-4d87-444a-89f4-bcea83f1546d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c35e953b9512e753411c4004b1f2b62e16c042dfb529d7f77cc8dea243add8fc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6033, \"end_char_idx\": 9114, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "doc_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "ref_doc_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "chroma:document": "To visualize the bounding boxes with associated labels, you can get the labels from the dataset's metadata, specifically\nthe `category` field.\nYou'll also want to create dictionaries that map a label id to a label class (`id2label`) and the other way around (`label2id`).\nYou can use them later when setting up the model. Including these maps will make your model reusable by others if you share\nit on the Hugging Face Hub. Please note that, the part of above code that draws the bounding boxes assume that it is in `COCO` format `(x_min, y_min, width, height)`. It has to be adjusted to work for other formats like `(x_min, y_min, x_max, y_max)`.\n\nAs a final step of getting familiar with the data, explore it for potential issues. One common problem with datasets for\nobject detection is bounding boxes that \"stretch\" beyond the edge of the image. Such \"runaway\" bounding boxes can raise\nerrors during training and should be addressed. There are a few examples with this issue in this dataset.\nTo keep things simple in this guide, we will set `clip=True` for `BboxParams` in transformations below.\n\n## Preprocess the data\n\nTo finetune a model, you must preprocess the data you plan to use to match precisely the approach used for the pre-trained model.\n[`AutoImageProcessor`] takes care of processing image data to create `pixel_values`, `pixel_mask`, and\n`labels` that a DETR model can train with. The image processor has some attributes that you won't have to worry about:\n\n- `image_mean = [0.485, 0.456, 0.406 ]`\n- `image_std = [0.229, 0.224, 0.225]`\n\nThese are the mean and standard deviation used to normalize images during the model pre-training. These values are crucial\nto replicate when doing inference or finetuning a pre-trained image model.\n\nInstantiate the image processor from the same checkpoint as the model you want to finetune.\n\n```py\n>>> from transformers import AutoImageProcessor\n\n>>> MAX_SIZE = IMAGE_SIZE\n\n>>> image_processor = AutoImageProcessor.from_pretrained(\n...     MODEL_NAME,\n...     do_resize=True,\n...     size={\"max_height\": MAX_SIZE, \"max_width\": MAX_SIZE},\n...     do_pad=True,\n...     pad_size={\"height\": MAX_SIZE, \"width\": MAX_SIZE},\n... )\n```\n\nBefore passing the images to the `image_processor`, apply two preprocessing transformations to the dataset:\n- Augmenting images\n- Reformatting annotations to meet DETR expectations\n\nFirst, to make sure the model does not overfit on the training data, you can apply image augmentation with any data augmentation library. Here we use [Albumentations](https://albumentations.ai/docs/).\nThis library ensures that transformations affect the image and update the bounding boxes accordingly.\nThe \ud83e\udd17 Datasets library documentation has a detailed [guide on how to augment images for object detection](https://huggingface.co/docs/datasets/object_detection),\nand it uses the exact same dataset as an example. Apply some geometric and color transformations to the image. For additional augmentation options, explore the [Albumentations Demo Space](https://huggingface.co/spaces/qubvel-hf/albumentations-demo)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: ad89c284-4d87-444a-89f4-bcea83f1546d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"ad89c284-4d87-444a-89f4-bcea83f1546d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c67b77e2-0310-4df5-8659-cf6d0618167d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"badedd7f-d690-4f73-92a6-829eef17391f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"549ff75ee709b05712c0812ff314a076a1619e8dc17cd01a0e13ed5ac7146eae\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"374ec709-a61c-4e41-ac96-0582daa5a52b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e3d3408c28ef45c5a0f556e3d1aa06cacde680bf44f90dd18b57b2524787728a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7434, \"end_char_idx\": 10095, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "doc_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "ref_doc_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "chroma:document": "The image processor has some attributes that you won't have to worry about:\n\n- `image_mean = [0.485, 0.456, 0.406 ]`\n- `image_std = [0.229, 0.224, 0.225]`\n\nThese are the mean and standard deviation used to normalize images during the model pre-training. These values are crucial\nto replicate when doing inference or finetuning a pre-trained image model.\n\nInstantiate the image processor from the same checkpoint as the model you want to finetune.\n\n```py\n>>> from transformers import AutoImageProcessor\n\n>>> MAX_SIZE = IMAGE_SIZE\n\n>>> image_processor = AutoImageProcessor.from_pretrained(\n...     MODEL_NAME,\n...     do_resize=True,\n...     size={\"max_height\": MAX_SIZE, \"max_width\": MAX_SIZE},\n...     do_pad=True,\n...     pad_size={\"height\": MAX_SIZE, \"width\": MAX_SIZE},\n... )\n```\n\nBefore passing the images to the `image_processor`, apply two preprocessing transformations to the dataset:\n- Augmenting images\n- Reformatting annotations to meet DETR expectations\n\nFirst, to make sure the model does not overfit on the training data, you can apply image augmentation with any data augmentation library. Here we use [Albumentations](https://albumentations.ai/docs/).\nThis library ensures that transformations affect the image and update the bounding boxes accordingly.\nThe \ud83e\udd17 Datasets library documentation has a detailed [guide on how to augment images for object detection](https://huggingface.co/docs/datasets/object_detection),\nand it uses the exact same dataset as an example. Apply some geometric and color transformations to the image. For additional augmentation options, explore the [Albumentations Demo Space](https://huggingface.co/spaces/qubvel-hf/albumentations-demo).\n\n```py\n>>> import albumentations as A\n\n>>> train_augment_and_transform = A.Compose(\n...     [\n...         A.Perspective(p=0.1),\n...         A.HorizontalFlip(p=0.5),\n...         A.RandomBrightnessContrast(p=0.5),\n...         A.HueSaturationValue(p=0.1),\n...     ],\n...     bbox_params=A.BboxParams(format=\"coco\", label_fields=[\"category\"], clip=True, min_area=25),\n... )\n\n>>> validation_transform = A.Compose(\n...     [A.NoOp()],\n...     bbox_params=A.BboxParams(format=\"coco\", label_fields=[\"category\"], clip=True),\n... )\n```\n\nThe `image_processor` expects the annotations to be in the following format: `{'image_id': int, 'annotations': List[Dict]}`,\n where each dictionary is a COCO object annotation. Let's add a function to reformat annotations for a single example:\n\n```py\n>>> def format_image_annotations_as_coco(image_id, categories, areas, bboxes):\n...     \"\"\"Format one set of image annotations to the COCO format\n\n...     Args:\n...         image_id (str): image id. e.g."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 374ec709-a61c-4e41-ac96-0582daa5a52b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"374ec709-a61c-4e41-ac96-0582daa5a52b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c67b77e2-0310-4df5-8659-cf6d0618167d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ad89c284-4d87-444a-89f4-bcea83f1546d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"eb1f571d66366d801c6728c9cc06ff1d48326836ad03c9c2b6b16b9ad1c94739\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"64520fcd-832d-41d8-a937-2caa109015df\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b84914bda320b3f08e5e432afb8242660428b333d68de5ea81aa07cef3ea7ced\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8538, \"end_char_idx\": 12459, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "doc_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "ref_doc_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "chroma:document": "Here we use [Albumentations](https://albumentations.ai/docs/).\nThis library ensures that transformations affect the image and update the bounding boxes accordingly.\nThe \ud83e\udd17 Datasets library documentation has a detailed [guide on how to augment images for object detection](https://huggingface.co/docs/datasets/object_detection),\nand it uses the exact same dataset as an example. Apply some geometric and color transformations to the image. For additional augmentation options, explore the [Albumentations Demo Space](https://huggingface.co/spaces/qubvel-hf/albumentations-demo).\n\n```py\n>>> import albumentations as A\n\n>>> train_augment_and_transform = A.Compose(\n...     [\n...         A.Perspective(p=0.1),\n...         A.HorizontalFlip(p=0.5),\n...         A.RandomBrightnessContrast(p=0.5),\n...         A.HueSaturationValue(p=0.1),\n...     ],\n...     bbox_params=A.BboxParams(format=\"coco\", label_fields=[\"category\"], clip=True, min_area=25),\n... )\n\n>>> validation_transform = A.Compose(\n...     [A.NoOp()],\n...     bbox_params=A.BboxParams(format=\"coco\", label_fields=[\"category\"], clip=True),\n... )\n```\n\nThe `image_processor` expects the annotations to be in the following format: `{'image_id': int, 'annotations': List[Dict]}`,\n where each dictionary is a COCO object annotation. Let's add a function to reformat annotations for a single example:\n\n```py\n>>> def format_image_annotations_as_coco(image_id, categories, areas, bboxes):\n...     \"\"\"Format one set of image annotations to the COCO format\n\n...     Args:\n...         image_id (str): image id. e.g. \"0001\"\n...         categories (List[int]): list of categories/class labels corresponding to provided bounding boxes\n...         areas (List[float]): list of corresponding areas to provided bounding boxes\n...         bboxes (List[Tuple[float]]): list of bounding boxes provided in COCO format\n...             ([center_x, center_y, width, height] in absolute coordinates)\n\n...     Returns:\n...         dict: {\n...             \"image_id\": image id,\n...             \"annotations\": list of formatted annotations\n...         }\n...     \"\"\"\n...     annotations = []\n...     for category, area, bbox in zip(categories, areas, bboxes):\n...         formatted_annotation = {\n...             \"image_id\": image_id,\n...             \"category_id\": category,\n...             \"iscrowd\": 0,\n...             \"area\": area,\n...             \"bbox\": list(bbox),\n...         }\n...         annotations.append(formatted_annotation)\n\n...     return {\n...         \"image_id\": image_id,\n...         \"annotations\": annotations,\n...     }\n\n```\n\nNow you can combine the image and annotation transformations to use on a batch of examples:\n\n```py\n>>> def augment_and_transform_batch(examples, transform, image_processor, return_pixel_mask=False):\n...     \"\"\"Apply augmentations and format annotations in COCO format for object detection task\"\"\"\n\n...     images = []\n...     annotations = []\n...     for image_id, image, objects in zip(examples[\"image_id\"], examples[\"image\"], examples[\"objects\"]):\n...         image = np.array(image.convert(\"RGB\"))\n\n...         # apply augmentations\n...         output = transform(image=image, bboxes=objects[\"bbox\"], category=objects[\"category\"])\n...         images.append(output[\"image\"])\n\n...         # format annotations in COCO format\n...         formatted_annotations = format_image_annotations_as_coco(\n...             image_id, output[\"category\"], objects[\"area\"], output[\"bboxes\"]\n...         )\n...         annotations.append(formatted_annotations)\n\n...     # Apply the image processor transformations: resizing, rescaling, normalization\n...     result = image_processor(images=images, annotations=annotations, return_tensors=\"pt\")\n\n...     if not return_pixel_mask:\n...         result.pop(\"pixel_mask\", None)\n\n...     return result\n```\n\nApply this preprocessing function to the entire dataset using \ud83e\udd17 Datasets [`~datasets.Dataset.with_transform`] method."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 64520fcd-832d-41d8-a937-2caa109015df
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"64520fcd-832d-41d8-a937-2caa109015df\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c67b77e2-0310-4df5-8659-cf6d0618167d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"374ec709-a61c-4e41-ac96-0582daa5a52b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"fe71bd0699a05148e04169c9045199cf2337e343f4d0aa148ecb27078aa3dc53\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f7249101-a2ad-4eb9-ba8d-d7d34282ec7a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"16362efe4a5f5d6d55d670328bfffcfbc9133716ef95184389c0dc299ad84d30\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "doc_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "ref_doc_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "chroma:document": "This method applies\ntransformations on the fly when you load an element of the dataset.\n\nAt this point, you can check what an example from the dataset looks like after the transformations. You should see a tensor\nwith `pixel_values`, a tensor with `pixel_mask`, and `labels`.\n\n```py\n>>> from functools import partial\n\n>>> # Make transform functions for batch and apply for dataset splits\n>>> train_transform_batch = partial(\n.     augment_and_transform_batch, transform=train_augment_and_transform, image_processor=image_processor\n. )\n>>> validation_transform_batch = partial(\n.     augment_and_transform_batch, transform=validation_transform, image_processor=image_processor\n. )\n\n>>> cppe5[\"train\"] = cppe5[\"train\"].with_transform(train_transform_batch)\n>>> cppe5[\"validation\"] = cppe5[\"validation\"].with_transform(validation_transform_batch)\n>>> cppe5[\"test\"] = cppe5[\"test\"].with_transform(validation_transform_batch)\n\n>>> cppe5[\"train\"][15]\n{'pixel_values': tensor([[[ 1.9235,  1.9407,  1.9749,  . -0.7822, -0.7479, -0.6965],\n          [ 1.9578,  1.9749,  1.9920,  . -0.7993, -0.7650, -0.7308],\n          [ 2.0092,  2.0092,  2.0263,  . -0.8507, -0.8164, -0.7822],\n          .\n          [ 0.0741,  0.0741,  0.0741,  .  0.0741,  0.0741,  0.0741],\n          [ 0.0741,  0.0741,  0.0741,  .  0.0741,  0.0741,  0.0741],\n          [ 0.0741,  0.0741,  0.0741,  .  0.0741,  0.0741,  0.0741]],\n  \n          [[ 1.6232,  1.6408,  1.6583,  .  0.8704,  1.0105,  1.1331],\n          [ 1.6408,  1.6583,  1.6758,  .  0.8529,  0.9930,  1.0980],\n          [ 1.6933,  1.6933,  1.7108,  .  0.8179,  0.9580,  1.0630],\n          .\n          [ 0.2052,  0.2052,  0.2052,  .  0.2052,  0.2052,  0.2052],\n          [ 0.2052,  0.2052,  0.2052,  .  0.2052,  0.2052,  0.2052],\n          [ 0.2052,  0.2052,  0.2052,  .  0.2052,  0.2052,  0.2052]],\n  \n          [[ 1.8905,  1.9080,  1.9428,  ."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f7249101-a2ad-4eb9-ba8d-d7d34282ec7a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"f7249101-a2ad-4eb9-ba8d-d7d34282ec7a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c67b77e2-0310-4df5-8659-cf6d0618167d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"64520fcd-832d-41d8-a937-2caa109015df\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8ecec3def1f295328ef58b900262969b60b75961d859f60bfba51f7f7367b3ae\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e178de5b-0b04-4669-bf51-079c387f2396\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"18d08ee815e7860e2de3464b851dacbeb4a07d76cc200bc0f4f0ffa7f265a47e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "doc_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "ref_doc_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "chroma:document": "0741],\n          [ 0.0741,  0.0741,  0.0741,  .  0.0741,  0.0741,  0.0741],\n          [ 0.0741,  0.0741,  0.0741,  .  0.0741,  0.0741,  0.0741]],\n  \n          [[ 1.6232,  1.6408,  1.6583,  .  0.8704,  1.0105,  1.1331],\n          [ 1.6408,  1.6583,  1.6758,  .  0.8529,  0.9930,  1.0980],\n          [ 1.6933,  1.6933,  1.7108,  .  0.8179,  0.9580,  1.0630],\n          .\n          [ 0.2052,  0.2052,  0.2052,  .  0.2052,  0.2052,  0.2052],\n          [ 0.2052,  0.2052,  0.2052,  .  0.2052,  0.2052,  0.2052],\n          [ 0.2052,  0.2052,  0.2052,  .  0.2052,  0.2052,  0.2052]],\n  \n          [[ 1.8905,  1.9080,  1.9428,  . -0.1487, -0.0964, -0.0615],\n          [ 1.9254,  1.9428,  1.9603,  . -0.1661, -0.1138, -0.0790],\n          [ 1.9777,  1.9777,  1.9951,  . -0.2010, -0.1138, -0.0790],\n          .\n          [ 0.4265,  0.4265,  0.4265,  .  0.4265,  0.4265,  0.4265],\n          [ 0.4265,  0.4265,  0.4265,  .  0.4265,  0.4265,  0.4265],\n          [ 0.4265,  0.4265,  0.4265,  .  0.4265,  0.4265,  0.4265]]]),\n  'labels': {'image_id': tensor([688]), 'class_labels': tensor([3, 4, 2, 0, 0]), 'boxes': tensor([[0.4700, 0.1933, 0.1467, 0.0767],\n          [0.4858, 0.2600, 0.1150, 0.1000],\n          [0.4042, 0.4517, 0.1217, 0.1300],\n          [0.4242, 0.3217, 0.3617, 0.5567],"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e178de5b-0b04-4669-bf51-079c387f2396
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"e178de5b-0b04-4669-bf51-079c387f2396\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c67b77e2-0310-4df5-8659-cf6d0618167d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f7249101-a2ad-4eb9-ba8d-d7d34282ec7a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"465d8093f52c6ddf2c020f65f2241f59bc65da76bf92d35b2dded3765073efc9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f20127c4-cd50-4b98-8051-6d38ffe045d6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e58d179bc248edbe2e04351c151c238e22920aa8707ea729758619aa449148a8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "doc_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "ref_doc_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "chroma:document": "9080,  1.9428,  . -0.1487, -0.0964, -0.0615],\n          [ 1.9254,  1.9428,  1.9603,  . -0.1661, -0.1138, -0.0790],\n          [ 1.9777,  1.9777,  1.9951,  . -0.2010, -0.1138, -0.0790],\n          .\n          [ 0.4265,  0.4265,  0.4265,  .  0.4265,  0.4265,  0.4265],\n          [ 0.4265,  0.4265,  0.4265,  .  0.4265,  0.4265,  0.4265],\n          [ 0.4265,  0.4265,  0.4265,  .  0.4265,  0.4265,  0.4265]]]),\n  'labels': {'image_id': tensor([688]), 'class_labels': tensor([3, 4, 2, 0, 0]), 'boxes': tensor([[0.4700, 0.1933, 0.1467, 0.0767],\n          [0.4858, 0.2600, 0.1150, 0.1000],\n          [0.4042, 0.4517, 0.1217, 0.1300],\n          [0.4242, 0.3217, 0.3617, 0.5567],\n          [0.6617, 0.4033, 0.5400, 0.4533]]), 'area': tensor([ 4048.  4140.  5694. 72478. 88128.]), 'iscrowd': tensor([0, 0, 0, 0, 0]), 'orig_size': tensor([480, 480])}}\n```\n\nYou have successfully augmented the individual images and prepared their annotations. However, preprocessing isn't\ncomplete yet. In the final step, create a custom `collate_fn` to batch images together.\nPad images (which are now `pixel_values`) to the largest image in a batch, and create a corresponding `pixel_mask`\nto indicate which pixels are real (1) and which are padding (0).\n\n```py\n>>> import torch\n\n>>> def collate_fn(batch):\n...     data = {}\n...     data[\"pixel_values\"] = torch.stack([x[\"pixel_values\"] for x in batch])\n...     data[\"labels\"] = [x[\"labels\"] for x in batch]\n...     if \"pixel_mask\" in batch[0]:\n...         data[\"pixel_mask\"] = torch.stack([x[\"pixel_mask\"] for x in batch])\n...     return data\n\n```\n\n## Preparing function to compute mAP\n\nObject detection models are commonly evaluated with a set of <a href=\"https://cocodataset.org/#detection-eval\">COCO-style metrics</a>. We are going to use `torchmetrics` to compute `mAP` (mean average precision) and `mAR` (mean average recall) metrics and will wrap it to `compute_metrics` function in order to use in [`Trainer`] for evaluation."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f20127c4-cd50-4b98-8051-6d38ffe045d6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"f20127c4-cd50-4b98-8051-6d38ffe045d6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c67b77e2-0310-4df5-8659-cf6d0618167d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e178de5b-0b04-4669-bf51-079c387f2396\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"78542935591fa9e23c7707e33976bab8c19775aae3daa42733bd8b5537a37d08\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c612183d-5c60-4bff-802f-d4bbd1bb1243\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e8654c2503d832dfc38d850656525f4459f74a861f477d5a338907750ae2e01f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "doc_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "ref_doc_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "chroma:document": "0.1300],\n          [0.4242, 0.3217, 0.3617, 0.5567],\n          [0.6617, 0.4033, 0.5400, 0.4533]]), 'area': tensor([ 4048.  4140.  5694. 72478. 88128.]), 'iscrowd': tensor([0, 0, 0, 0, 0]), 'orig_size': tensor([480, 480])}}\n```\n\nYou have successfully augmented the individual images and prepared their annotations. However, preprocessing isn't\ncomplete yet. In the final step, create a custom `collate_fn` to batch images together.\nPad images (which are now `pixel_values`) to the largest image in a batch, and create a corresponding `pixel_mask`\nto indicate which pixels are real (1) and which are padding (0).\n\n```py\n>>> import torch\n\n>>> def collate_fn(batch):\n...     data = {}\n...     data[\"pixel_values\"] = torch.stack([x[\"pixel_values\"] for x in batch])\n...     data[\"labels\"] = [x[\"labels\"] for x in batch]\n...     if \"pixel_mask\" in batch[0]:\n...         data[\"pixel_mask\"] = torch.stack([x[\"pixel_mask\"] for x in batch])\n...     return data\n\n```\n\n## Preparing function to compute mAP\n\nObject detection models are commonly evaluated with a set of <a href=\"https://cocodataset.org/#detection-eval\">COCO-style metrics</a>. We are going to use `torchmetrics` to compute `mAP` (mean average precision) and `mAR` (mean average recall) metrics and will wrap it to `compute_metrics` function in order to use in [`Trainer`] for evaluation.\n\nIntermediate format of boxes used for training is `YOLO` (normalized) but we will compute metrics for boxes in `Pascal VOC` (absolute) format in order to correctly handle box areas. Let's define a function that converts bounding boxes to `Pascal VOC` format:\n\n```py\n>>> from transformers.image_transforms import center_to_corners_format\n\n>>> def convert_bbox_yolo_to_pascal(boxes, image_size):\n...     \"\"\"\n...     Convert bounding boxes from YOLO format (x_center, y_center, width, height) in range [0, 1]\n...     to Pascal VOC format (x_min, y_min, x_max, y_max) in absolute coordinates.\n\n...     Args:\n...         boxes (torch.Tensor): Bounding boxes in YOLO format\n...         image_size (Tuple[int, int]): Image size in format (height, width)\n\n...     Returns:\n...         torch.Tensor: Bounding boxes in Pascal VOC format (x_min, y_min, x_max, y_max)\n...     \"\"\"\n...     # convert center to corners format\n...     boxes = center_to_corners_format(boxes)\n\n...     # convert to absolute coordinates\n...     height, width = image_size\n...     boxes = boxes * torch.tensor([[width, height, width, height]])\n\n...     return boxes\n```\n\nThen, in `compute_metrics` function we collect `predicted` and `target` bounding boxes, scores and labels from evaluation loop results and pass it to the scoring function.\n\n```py\n>>> import numpy as np\n>>> from dataclasses import dataclass\n>>> from torchmetrics.detection.mean_ap import MeanAveragePrecision\n\n\n>>> @dataclass\n>>> class ModelOutput:\n...     logits: torch.Tensor\n...     pred_boxes: torch.Tensor"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: c612183d-5c60-4bff-802f-d4bbd1bb1243
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"c612183d-5c60-4bff-802f-d4bbd1bb1243\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c67b77e2-0310-4df5-8659-cf6d0618167d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f20127c4-cd50-4b98-8051-6d38ffe045d6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"a9df26324f1551da173a70883d3e2bb812122accfe25835fb4317135a8f3fd97\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a4c5847e-d7b4-4332-a37c-bfbf6abb4c0c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1ffb0299a3d99bff0a66034bb4407641898738e0867c0aa7315f300ca0ea0d4b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 16339, \"end_char_idx\": 21366, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "doc_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "ref_doc_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "chroma:document": "Intermediate format of boxes used for training is `YOLO` (normalized) but we will compute metrics for boxes in `Pascal VOC` (absolute) format in order to correctly handle box areas. Let's define a function that converts bounding boxes to `Pascal VOC` format:\n\n```py\n>>> from transformers.image_transforms import center_to_corners_format\n\n>>> def convert_bbox_yolo_to_pascal(boxes, image_size):\n...     \"\"\"\n...     Convert bounding boxes from YOLO format (x_center, y_center, width, height) in range [0, 1]\n...     to Pascal VOC format (x_min, y_min, x_max, y_max) in absolute coordinates.\n\n...     Args:\n...         boxes (torch.Tensor): Bounding boxes in YOLO format\n...         image_size (Tuple[int, int]): Image size in format (height, width)\n\n...     Returns:\n...         torch.Tensor: Bounding boxes in Pascal VOC format (x_min, y_min, x_max, y_max)\n...     \"\"\"\n...     # convert center to corners format\n...     boxes = center_to_corners_format(boxes)\n\n...     # convert to absolute coordinates\n...     height, width = image_size\n...     boxes = boxes * torch.tensor([[width, height, width, height]])\n\n...     return boxes\n```\n\nThen, in `compute_metrics` function we collect `predicted` and `target` bounding boxes, scores and labels from evaluation loop results and pass it to the scoring function.\n\n```py\n>>> import numpy as np\n>>> from dataclasses import dataclass\n>>> from torchmetrics.detection.mean_ap import MeanAveragePrecision\n\n\n>>> @dataclass\n>>> class ModelOutput:\n...     logits: torch.Tensor\n...     pred_boxes: torch.Tensor\n\n\n>>> @torch.no_grad()\n>>> def compute_metrics(evaluation_results, image_processor, threshold=0.0, id2label=None):\n...     \"\"\"\n...     Compute mean average mAP, mAR and their variants for the object detection task.\n\n...     Args:\n...         evaluation_results (EvalPrediction): Predictions and targets from evaluation.\n...         threshold (float, optional): Threshold to filter predicted boxes by confidence. Defaults to 0.0.\n...         id2label (Optional[dict], optional): Mapping from class id to class name. Defaults to None.\n\n...     Returns:\n...         Mapping[str, float]: Metrics in a form of dictionary {<metric_name>: <metric_value>}\n...     \"\"\"\n\n...     predictions, targets = evaluation_results.predictions, evaluation_results.label_ids\n\n...     # For metric computation we need to provide:\n...     #  - targets in a form of list of dictionaries with keys \"boxes\", \"labels\"\n...     #  - predictions in a form of list of dictionaries with keys \"boxes\", \"scores\", \"labels\"\n\n...     image_sizes = []\n...     post_processed_targets = []\n...     post_processed_predictions = []\n\n...     # Collect targets in the required format for metric computation\n...     for batch in targets:\n...         # collect image sizes, we will need them for predictions post processing\n...         batch_image_sizes = torch.tensor(np.array([x[\"orig_size\"] for x in batch]))\n...         image_sizes.append(batch_image_sizes)\n...         # collect targets in the required format for metric computation\n...         # boxes were converted to YOLO format needed for model training\n...         # here we will convert them to Pascal VOC format (x_min, y_min, x_max, y_max)\n...         for image_target in batch:\n...             boxes = torch.tensor(image_target[\"boxes\"])\n...             boxes = convert_bbox_yolo_to_pascal(boxes, image_target[\"orig_size\"])\n...             labels = torch.tensor(image_target[\"class_labels\"])\n...             post_processed_targets.append({\"boxes\": boxes, \"labels\": labels})\n\n...     # Collect predictions in the required format for metric computation,\n...     # model produce boxes in YOLO format, then image_processor convert them to Pascal VOC format\n...     for batch, target_sizes in zip(predictions, image_sizes):\n...         batch_logits, batch_boxes = batch[1], batch[2]\n...         output = ModelOutput(logits=torch.tensor(batch_logits), pred_boxes=torch.tensor(batch_boxes))\n...         post_processed_output = image_processor.post_process_object_detection(\n...             output, threshold=threshold, target_sizes=target_sizes\n...         )\n...         post_processed_predictions.extend(post_processed_output)\n\n...     # Compute metrics\n...     metric = MeanAveragePrecision(box_format=\"xyxy\", class_metrics=True)\n...     metric.update(post_processed_predictions, post_processed_targets)\n...     metrics = metric.compute()\n\n...     # Replace list of per class metrics with separate metric for each class\n...     classes = metrics.pop(\"classes\")\n...     map_per_class = metrics.pop(\"map_per_class\")\n...     mar_100_per_class = metrics.pop(\"mar_100_per_class\")\n...     for class_id, class_map, class_mar in zip(classes, map_per_class, mar_100_per_class):\n...         class_name = id2label[class_id.item()] if id2label is not None else class_id.item()\n...         metrics[f\"map_{class_name}\"] = class_map\n...         metrics[f\"mar_100_{class_name}\"] = class_mar\n\n...     metrics = {k: round(v.item(), 4) for k, v in metrics.items()}\n\n...     return metrics"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a4c5847e-d7b4-4332-a37c-bfbf6abb4c0c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"a4c5847e-d7b4-4332-a37c-bfbf6abb4c0c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c67b77e2-0310-4df5-8659-cf6d0618167d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c612183d-5c60-4bff-802f-d4bbd1bb1243\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"55c79b23536ee0c2c5e518c276850ba6c3369decb88493b9a435605d883a70f3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fbc72b81-8b34-4029-baec-087f7323e1db\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"203b9167d068489e551c4d0a452a1bf3986c1b59b1342a2e5024b962b79e769e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "doc_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "ref_doc_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "chroma:document": ">>> eval_compute_metrics_fn = partial(\n...     compute_metrics, image_processor=image_processor, id2label=id2label, threshold=0.0\n... )\n```\n\n## Training the detection model\n\nYou have done most of the heavy lifting in the previous sections, so now you are ready to train your model!\nThe images in this dataset are still quite large, even after resizing. This means that finetuning this model will\nrequire at least one GPU.\n\nTraining involves the following steps:\n1. Load the model with [`AutoModelForObjectDetection`] using the same checkpoint as in the preprocessing.\n2. Define your training hyperparameters in [`TrainingArguments`].\n3. Pass the training arguments to [`Trainer`] along with the model, dataset, image processor, and data collator.\n4. Call [`~Trainer.train`] to finetune your model.\n\nWhen loading the model from the same checkpoint that you used for the preprocessing, remember to pass the `label2id`\nand `id2label` maps that you created earlier from the dataset's metadata. Additionally, we specify `ignore_mismatched_sizes=True` to replace the existing classification head with a new one.\n\n```py\n>>> from transformers import AutoModelForObjectDetection\n\n>>> model = AutoModelForObjectDetection.from_pretrained(\n...     MODEL_NAME,\n...     id2label=id2label,\n...     label2id=label2id,\n...     ignore_mismatched_sizes=True,\n... )\n```\n\nIn the [`TrainingArguments`] use `output_dir` to specify where to save your model, then configure hyperparameters as you see fit. For `num_train_epochs=30` training will take about 35 minutes in Google Colab T4 GPU, increase the number of epoch to get better results.\n\nImportant notes:\n - Do not remove unused columns because this will drop the image column. Without the image column, you\ncan't create `pixel_values`. For this reason, set `remove_unused_columns` to `False`.\n - Set `eval_do_concat_batches=False` to get proper evaluation results. Images have different number of target boxes, if batches are concatenated we will not be able to determine which boxes belongs to particular image.\n\nIf you wish to share your model by pushing to the Hub, set `push_to_hub` to `True` (you must be signed in to Hugging\nFace to upload your model).\n\n```py\n>>> from transformers import TrainingArguments\n\n>>> training_args = TrainingArguments(\n.     output_dir=\"detr_finetuned_cppe5\",\n.     num_train_epochs=30,\n.     fp16=False,\n.     per_device_train_batch_size=8,\n.     dataloader_num_workers=4,\n.     learning_rate=5e-5,\n.     lr_scheduler_type=\"cosine\",\n.     weight_decay=1e-4,\n.     max_grad_norm=0.01,\n.     metric_for_best_model=\"eval_map\",\n.     greater_is_better=True,\n.     load_best_model_at_end=True,\n.     eval_strategy=\"epoch\",\n.     save_strategy=\"epoch\",\n.     save_total_limit=2,\n.     remove_unused_columns=False,\n.     eval_do_concat_batches=False,\n.     push_to_hub=True,\n. )\n```\n\nFinally, bring everything together, and call [`~transformers.Trainer.train`]:\n\n```py\n>>> from transformers import Trainer\n\n>>> trainer = Trainer(\n.     model=model,\n.     args=training_args,\n.     train_dataset=cppe5[\"train\"],\n.     eval_dataset=cppe5[\"validation\"],\n.     tokenizer=image_processor,\n.     data_collator=collate_fn,\n."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: fbc72b81-8b34-4029-baec-087f7323e1db
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"fbc72b81-8b34-4029-baec-087f7323e1db\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c67b77e2-0310-4df5-8659-cf6d0618167d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a4c5847e-d7b4-4332-a37c-bfbf6abb4c0c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"81831528b8a21a2723008c21d15b4d039a0100f6a1874bd9ac01d5fbc40ff9c5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c48c38dc-1130-469e-b50b-eaefb02af16a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"373bc6419be3b6ecc65f9036aaa36cd93684e1f2bfa34c2b0a662440b84d8ffc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "doc_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "ref_doc_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "chroma:document": "Without the image column, you\ncan't create `pixel_values`. For this reason, set `remove_unused_columns` to `False`.\n - Set `eval_do_concat_batches=False` to get proper evaluation results. Images have different number of target boxes, if batches are concatenated we will not be able to determine which boxes belongs to particular image.\n\nIf you wish to share your model by pushing to the Hub, set `push_to_hub` to `True` (you must be signed in to Hugging\nFace to upload your model).\n\n```py\n>>> from transformers import TrainingArguments\n\n>>> training_args = TrainingArguments(\n.     output_dir=\"detr_finetuned_cppe5\",\n.     num_train_epochs=30,\n.     fp16=False,\n.     per_device_train_batch_size=8,\n.     dataloader_num_workers=4,\n.     learning_rate=5e-5,\n.     lr_scheduler_type=\"cosine\",\n.     weight_decay=1e-4,\n.     max_grad_norm=0.01,\n.     metric_for_best_model=\"eval_map\",\n.     greater_is_better=True,\n.     load_best_model_at_end=True,\n.     eval_strategy=\"epoch\",\n.     save_strategy=\"epoch\",\n.     save_total_limit=2,\n.     remove_unused_columns=False,\n.     eval_do_concat_batches=False,\n.     push_to_hub=True,\n. )\n```\n\nFinally, bring everything together, and call [`~transformers.Trainer.train`]:\n\n```py\n>>> from transformers import Trainer\n\n>>> trainer = Trainer(\n.     model=model,\n.     args=training_args,\n.     train_dataset=cppe5[\"train\"],\n.     eval_dataset=cppe5[\"validation\"],\n.     tokenizer=image_processor,\n.     data_collator=collate_fn,\n.     compute_metrics=eval_compute_metrics_fn,\n. )\n\n>>> trainer.train()\n```\n<div>\n\n  <progress value='3210' max='3210' style='width:300px; height:20px; vertical-align: middle;'></progress>\n  [3210/3210 26:07, Epoch 30/30]\n</div>\n\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: left;\">\n      <th>Epoch</th>\n      <th>Training Loss</th>\n      <th>Validation Loss</th>\n      <th>Map</th>\n      <th>Map 50</th>\n      <th>Map 75</th>\n      <th>Map Small</th>\n      <th>Map Medium</th>\n      <th>Map Large</th>\n      <th>Mar 1</th>\n      <th>Mar 10</th>\n      <th>Mar 100</th>\n      <th>Mar Small</th>\n      <th>Mar Medium</th>\n      <th>Mar Large</th>\n      <th>Map Coverall</th>\n      <th>Mar 100 Coverall</th>\n      <th>Map Face Shield</th>\n      <th>Mar 100 Face Shield</th>\n      <th>Map Gloves</th>\n      <th>Mar 100 Gloves</th>\n      <th>Map Goggles</th>\n      <th>Mar 100 Goggles</th>\n      <th>Map Mask</th>\n      <th>Mar 100 Mask</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <td>1</td>\n      <td>No log</td>\n      <td>2."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: c48c38dc-1130-469e-b50b-eaefb02af16a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"c48c38dc-1130-469e-b50b-eaefb02af16a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c67b77e2-0310-4df5-8659-cf6d0618167d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"fbc72b81-8b34-4029-baec-087f7323e1db\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"0d629bb88d97aa32173202b6aa162f0a49e0f47e4983abd4a1d4cd368a0a08ae\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b7cf020a-5bac-4c3d-b345-b5c09465413e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e3b8d70541fd887d0c347858c3265f25f555d6403e49d8081b16a8f7cb15ef44\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "doc_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "ref_doc_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "chroma:document": "tokenizer=image_processor,\n.     data_collator=collate_fn,\n.     compute_metrics=eval_compute_metrics_fn,\n. )\n\n>>> trainer.train()\n```\n<div>\n\n  <progress value='3210' max='3210' style='width:300px; height:20px; vertical-align: middle;'></progress>\n  [3210/3210 26:07, Epoch 30/30]\n</div>\n\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: left;\">\n      <th>Epoch</th>\n      <th>Training Loss</th>\n      <th>Validation Loss</th>\n      <th>Map</th>\n      <th>Map 50</th>\n      <th>Map 75</th>\n      <th>Map Small</th>\n      <th>Map Medium</th>\n      <th>Map Large</th>\n      <th>Mar 1</th>\n      <th>Mar 10</th>\n      <th>Mar 100</th>\n      <th>Mar Small</th>\n      <th>Mar Medium</th>\n      <th>Mar Large</th>\n      <th>Map Coverall</th>\n      <th>Mar 100 Coverall</th>\n      <th>Map Face Shield</th>\n      <th>Mar 100 Face Shield</th>\n      <th>Map Gloves</th>\n      <th>Mar 100 Gloves</th>\n      <th>Map Goggles</th>\n      <th>Mar 100 Goggles</th>\n      <th>Map Mask</th>\n      <th>Mar 100 Mask</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <td>1</td>\n      <td>No log</td>\n      <td>2.629903</td>\n      <td>0.008900</td>\n      <td>0.023200</td>\n      <td>0.006500</td>\n      <td>0.001300</td>\n      <td>0.002800</td>\n      <td>0.020500</td>\n      <td>0.021500</td>\n      <td>0.070400</td>\n      <td>0.101400</td>\n      <td>0.007600</td>\n      <td>0.106200</td>\n      <td>0.096100</td>\n      <td>0.036700</td>\n      <td>0.232000</td>\n      <td>0.000300</td>\n      <td>0.019000</td>\n      <td>0.003900</td>\n      <td>0.125400</td>\n      <td>0.000100</td>\n      <td>0.003100</td>\n      <td>0.003500</td>\n      <td>0.127600</td>\n    </tr>\n    <tr>\n      <td>2</td>\n      <td>No log</td>\n      <td>3.479864</td>\n      <td>0.014800</td>\n      <td>0.034600</td>\n      <td>0.010800</td>\n      <td>0.008600</td>\n      <td>0.011700</td>\n      <td>0.012500</td>\n      <td>0.041100</td>\n      <td>0.098700</td>\n      <td>0."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: b7cf020a-5bac-4c3d-b345-b5c09465413e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"b7cf020a-5bac-4c3d-b345-b5c09465413e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c67b77e2-0310-4df5-8659-cf6d0618167d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c48c38dc-1130-469e-b50b-eaefb02af16a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/object_detection\", \"title\": \"Object detection\", \"tokens\": 16536, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"29bdec96642f6985411acfd94c6564a67995ae4a9ef1b4e6a735d0a0b9c15a1c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0f2da6d2-6a43-4ccf-ac27-21359ae64502\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4ec71c4473b581d4c45912aa002df0f6d85e3e42ed94493ebe3878eb5a74afbe\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 25657, \"end_char_idx\": 27381, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "doc_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "ref_doc_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "chroma:document": "629903</td>\n      <td>0.008900</td>\n      <td>0.023200</td>\n      <td>0.006500</td>\n      <td>0.001300</td>\n      <td>0.002800</td>\n      <td>0.020500</td>\n      <td>0.021500</td>\n      <td>0.070400</td>\n      <td>0.101400</td>\n      <td>0.007600</td>\n      <td>0.106200</td>\n      <td>0.096100</td>\n      <td>0.036700</td>\n      <td>0.232000</td>\n      <td>0.000300</td>\n      <td>0.019000</td>\n      <td>0.003900</td>\n      <td>0.125400</td>\n      <td>0.000100</td>\n      <td>0.003100</td>\n      <td>0.003500</td>\n      <td>0.127600</td>\n    </tr>\n    <tr>\n      <td>2</td>\n      <td>No log</td>\n      <td>3.479864</td>\n      <td>0.014800</td>\n      <td>0.034600</td>\n      <td>0.010800</td>\n      <td>0.008600</td>\n      <td>0.011700</td>\n      <td>0.012500</td>\n      <td>0.041100</td>\n      <td>0.098700</td>\n      <td>0.130000</td>\n      <td>0.056000</td>\n      <td>0.062200</td>\n      <td>0.111900</td>\n      <td>0.053500</td>\n      <td>0.447300</td>\n      <td>0.010600</td>\n      <td>0.100000</td>\n      <td>0.000200</td>\n      <td>0.022800</td>\n      <td>0.000100</td>\n      <td>0.015400</td>\n      <td>0.009700</td>\n      <td>0.064400</td>\n    </tr>\n    <tr>\n      <td>3</td>\n      <td>No log</td>\n      <td>2.107622</td>\n      <td>0.041700</td>\n      <td>0.094000</td>\n      <td>0.034300</td>\n      <td>0.024100</td>\n      <td>0.026400</td>\n      <td>0.047400</td>\n      <td>0.091500</td>\n      <td>0.182800</td>\n      <td>0.225800</td>\n      <td>0.087200</td>\n      <td>0.199400</td>\n      <td>0.210600</td>\n      <td>0.150900</td>\n      <td>0.571200</td>\n      <td>0.017300</td>\n      <td>0.101300</td>\n      <td>0.007300</td>\n      <td>0.180400</td>\n      <td>0.002100</td>\n      <td>0.026200</td>\n      <td>0."}


