# embeddings_queue

created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: e9a3b04b-d8a5-45cc-ab72-7a8b60a4a632
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2", "title": "SeamlessM4T-v2", "tokens": 2580, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e9a3b04b-d8a5-45cc-ab72-7a8b60a4a632\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2\", \"title\": \"SeamlessM4T-v2\", \"tokens\": 2580, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2d2f92f2-3109-4298-9775-8aa63651dcdb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2\", \"title\": \"SeamlessM4T-v2\", \"tokens\": 2580, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"37fca48a2508dd873d46a86c895976af883f282a000c596c1db84d196a8ed78e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"805db382-3599-403e-9150-e7f4a675258f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2\", \"title\": \"SeamlessM4T-v2\", \"tokens\": 2580, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"eb8f8e04ca46f92288b9e4bc0570d45b3535967ee4543485022d8b06d1b602b9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"45e08bce-8028-4758-bcb6-5441d94441e5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4fa031276ab73fc812a1556270b865cb0bca0c8aa5377fb2dc0ae98808c78b04\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3437, \"end_char_idx\": 6469, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2d2f92f2-3109-4298-9775-8aa63651dcdb", "doc_id": "2d2f92f2-3109-4298-9775-8aa63651dcdb", "ref_doc_id": "2d2f92f2-3109-4298-9775-8aa63651dcdb", "chroma:document": "Consequently, we bring major components from SeamlessExpressive and SeamlessStreaming together to form Seamless, the first publicly available system that unlocks expressive cross-lingual communication in real-time. In sum, Seamless gives us a pivotal look at the technical foundation needed to turn the Universal Speech Translator from a science fiction concept into a real-world technology. Finally, contributions in this work\u2014including models, code, and a watermark detector\u2014are publicly released and accessible at the link below.*\n\n## Usage\n\nIn the following example, we'll load an Arabic audio sample and an English text sample and convert them into Russian speech and French text.\n\nFirst, load the processor and a checkpoint of the model:\n\n```python\n>>> from transformers import AutoProcessor, SeamlessM4Tv2Model\n\n>>> processor = AutoProcessor.from_pretrained(\"facebook/seamless-m4t-v2-large\")\n>>> model = SeamlessM4Tv2Model.from_pretrained(\"facebook/seamless-m4t-v2-large\")\n```\n\nYou can seamlessly use this model on text or on audio, to generated either translated text or translated audio.\n\nHere is how to use the processor to process text and audio:\n\n```python\n>>> # let's load an audio sample from an Arabic speech corpus\n>>> from datasets import load_dataset\n>>> dataset = load_dataset(\"arabic_speech_corpus\", split=\"test\", streaming=True)\n>>> audio_sample = next(iter(dataset))[\"audio\"]\n\n>>> # now, process it\n>>> audio_inputs = processor(audios=audio_sample[\"array\"], return_tensors=\"pt\")\n\n>>> # now, process some English text as well\n>>> text_inputs = processor(text = \"Hello, my dog is cute\", src_lang=\"eng\", return_tensors=\"pt\")\n```\n\n\n### Speech\n\n[`SeamlessM4Tv2Model`] can *seamlessly* generate text or speech with few or no changes. Let's target Russian voice translation:\n\n```python\n>>> audio_array_from_text = model.generate(**text_inputs, tgt_lang=\"rus\")[0].cpu().numpy().squeeze()\n>>> audio_array_from_audio = model.generate(**audio_inputs, tgt_lang=\"rus\")[0].cpu().numpy().squeeze()\n```\n\nWith basically the same code, I've translated English text and Arabic speech to Russian speech samples.\n\n### Text\n\nSimilarly, you can generate translated text from audio files or from text with the same model. You only have to pass `generate_speech=False` to [`SeamlessM4Tv2Model.generate`].\nThis time, let's translate to French.\n\n```python \n>>> # from audio\n>>> output_tokens = model.generate(**audio_inputs, tgt_lang=\"fra\", generate_speech=False)\n>>> translated_text_from_audio = processor.decode(output_tokens[0].tolist()[0], skip_special_tokens=True)\n\n>>> # from text\n>>> output_tokens = model.generate(**text_inputs, tgt_lang=\"fra\", generate_speech=False)\n>>> translated_text_from_text = processor.decode(output_tokens[0].tolist()[0], skip_special_tokens=True)\n```\n\n### Tips\n\n\n#### 1. Use dedicated models\n\n[`SeamlessM4Tv2Model`] is transformers top level model to generate speech and text, but you can also use dedicated models that perform the task without additional components, thus reducing the memory footprint."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 45e08bce-8028-4758-bcb6-5441d94441e5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2", "title": "SeamlessM4T-v2", "tokens": 2580, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"45e08bce-8028-4758-bcb6-5441d94441e5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2\", \"title\": \"SeamlessM4T-v2\", \"tokens\": 2580, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2d2f92f2-3109-4298-9775-8aa63651dcdb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2\", \"title\": \"SeamlessM4T-v2\", \"tokens\": 2580, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"37fca48a2508dd873d46a86c895976af883f282a000c596c1db84d196a8ed78e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e9a3b04b-d8a5-45cc-ab72-7a8b60a4a632\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2\", \"title\": \"SeamlessM4T-v2\", \"tokens\": 2580, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ea4419864f8cf8cd666d8dd7c911f1dff2d7762bd26045c18e49ecebeccc14b3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"87e95055-de21-445a-b6bb-a986e5e72033\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7a5fa43e44f764b39e5975ea825b2b3123485478864f4081c62a8953a4c2a6cd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5087, \"end_char_idx\": 8385, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2d2f92f2-3109-4298-9775-8aa63651dcdb", "doc_id": "2d2f92f2-3109-4298-9775-8aa63651dcdb", "ref_doc_id": "2d2f92f2-3109-4298-9775-8aa63651dcdb", "chroma:document": "### Speech\n\n[`SeamlessM4Tv2Model`] can *seamlessly* generate text or speech with few or no changes. Let's target Russian voice translation:\n\n```python\n>>> audio_array_from_text = model.generate(**text_inputs, tgt_lang=\"rus\")[0].cpu().numpy().squeeze()\n>>> audio_array_from_audio = model.generate(**audio_inputs, tgt_lang=\"rus\")[0].cpu().numpy().squeeze()\n```\n\nWith basically the same code, I've translated English text and Arabic speech to Russian speech samples.\n\n### Text\n\nSimilarly, you can generate translated text from audio files or from text with the same model. You only have to pass `generate_speech=False` to [`SeamlessM4Tv2Model.generate`].\nThis time, let's translate to French.\n\n```python \n>>> # from audio\n>>> output_tokens = model.generate(**audio_inputs, tgt_lang=\"fra\", generate_speech=False)\n>>> translated_text_from_audio = processor.decode(output_tokens[0].tolist()[0], skip_special_tokens=True)\n\n>>> # from text\n>>> output_tokens = model.generate(**text_inputs, tgt_lang=\"fra\", generate_speech=False)\n>>> translated_text_from_text = processor.decode(output_tokens[0].tolist()[0], skip_special_tokens=True)\n```\n\n### Tips\n\n\n#### 1. Use dedicated models\n\n[`SeamlessM4Tv2Model`] is transformers top level model to generate speech and text, but you can also use dedicated models that perform the task without additional components, thus reducing the memory footprint.\nFor example, you can replace the audio-to-audio generation snippet with the model dedicated to the S2ST task, the rest is exactly the same code: \n\n```python\n>>> from transformers import SeamlessM4Tv2ForSpeechToSpeech\n>>> model = SeamlessM4Tv2ForSpeechToSpeech.from_pretrained(\"facebook/seamless-m4t-v2-large\")\n```\n\nOr you can replace the text-to-text generation snippet with the model dedicated to the T2TT task, you only have to remove `generate_speech=False`.\n\n```python\n>>> from transformers import SeamlessM4Tv2ForTextToText\n>>> model = SeamlessM4Tv2ForTextToText.from_pretrained(\"facebook/seamless-m4t-v2-large\")\n```\n\nFeel free to try out [`SeamlessM4Tv2ForSpeechToText`] and [`SeamlessM4Tv2ForTextToSpeech`] as well.\n\n#### 2. Change the speaker identity\n\nYou have the possibility to change the speaker used for speech synthesis with the `speaker_id` argument. Some `speaker_id` works better than other for some languages!\n\n#### 3. Change the generation strategy\n\nYou can use different [generation strategies](../generation_strategies) for text generation, e.g `.generate(input_ids=input_ids, text_num_beams=4, text_do_sample=True)` which will perform multinomial beam-search decoding on the text model. Note that speech generation only supports greedy - by default - or multinomial sampling, which can be used with e.g. `.generate(..., speech_do_sample=True, speech_temperature=0.6)`.\n\n#### 4. Generate speech and text at the same time\n\nUse `return_intermediate_token_ids=True` with [`SeamlessM4Tv2Model`] to return both speech and text !\n\n## Model architecture\n\nSeamlessM4T-v2 features a versatile architecture that smoothly handles the sequential generation of text and speech. This setup comprises two sequence-to-sequence (seq2seq) models. The first model translates the input modality into translated text, while the second model generates speech tokens, known as \"unit tokens,\" from the translated text."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 87e95055-de21-445a-b6bb-a986e5e72033
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2", "title": "SeamlessM4T-v2", "tokens": 2580, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"87e95055-de21-445a-b6bb-a986e5e72033\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2\", \"title\": \"SeamlessM4T-v2\", \"tokens\": 2580, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2d2f92f2-3109-4298-9775-8aa63651dcdb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2\", \"title\": \"SeamlessM4T-v2\", \"tokens\": 2580, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"37fca48a2508dd873d46a86c895976af883f282a000c596c1db84d196a8ed78e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"45e08bce-8028-4758-bcb6-5441d94441e5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2\", \"title\": \"SeamlessM4T-v2\", \"tokens\": 2580, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b761eb8710fc563499cd5b4d68bf4798f4838e9f6e0a192189cd4814c38e2245\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"854178f0-36db-45c7-b2eb-30e661d18179\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1930bd56ca3593f025900111e8145bad1e17ab45ace2f0dcf739e1bddd907cab\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6933, \"end_char_idx\": 10436, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2d2f92f2-3109-4298-9775-8aa63651dcdb", "doc_id": "2d2f92f2-3109-4298-9775-8aa63651dcdb", "ref_doc_id": "2d2f92f2-3109-4298-9775-8aa63651dcdb", "chroma:document": "```python\n>>> from transformers import SeamlessM4Tv2ForTextToText\n>>> model = SeamlessM4Tv2ForTextToText.from_pretrained(\"facebook/seamless-m4t-v2-large\")\n```\n\nFeel free to try out [`SeamlessM4Tv2ForSpeechToText`] and [`SeamlessM4Tv2ForTextToSpeech`] as well.\n\n#### 2. Change the speaker identity\n\nYou have the possibility to change the speaker used for speech synthesis with the `speaker_id` argument. Some `speaker_id` works better than other for some languages!\n\n#### 3. Change the generation strategy\n\nYou can use different [generation strategies](../generation_strategies) for text generation, e.g `.generate(input_ids=input_ids, text_num_beams=4, text_do_sample=True)` which will perform multinomial beam-search decoding on the text model. Note that speech generation only supports greedy - by default - or multinomial sampling, which can be used with e.g. `.generate(..., speech_do_sample=True, speech_temperature=0.6)`.\n\n#### 4. Generate speech and text at the same time\n\nUse `return_intermediate_token_ids=True` with [`SeamlessM4Tv2Model`] to return both speech and text !\n\n## Model architecture\n\nSeamlessM4T-v2 features a versatile architecture that smoothly handles the sequential generation of text and speech. This setup comprises two sequence-to-sequence (seq2seq) models. The first model translates the input modality into translated text, while the second model generates speech tokens, known as \"unit tokens,\" from the translated text.\n\nEach modality has its own dedicated encoder with a unique architecture. Additionally, for speech output, a vocoder inspired by the [HiFi-GAN](https://arxiv.org/abs/2010.05646) architecture is placed on top of the second seq2seq model.\n\n### Difference with SeamlessM4T-v1\n\nThe architecture of this new version differs from the first in a few aspects:\n\n#### Improvements on the second-pass model\n\nThe second seq2seq model, named text-to-unit model, is now non-auto regressive, meaning that it computes units in a **single forward pass**. This achievement is made possible by:\n- the use of **character-level embeddings**, meaning that each character of the predicted translated text has its own embeddings, which are then used to predict the unit tokens.\n- the use of an intermediate duration predictor, that predicts speech duration at the **character-level** on the predicted translated text.\n- the use of a new text-to-unit decoder mixing convolutions and self-attention to handle longer context.\n\n#### Difference in the speech encoder\n\nThe speech encoder, which is used during the first-pass generation process to predict the translated text, differs mainly from the previous speech encoder through these mechanisms:\n- the use of chunked attention mask to prevent attention across chunks, ensuring that each position attends only to positions within its own chunk and a fixed number of previous chunks.\n- the use of relative position embeddings which only considers distance between sequence elements rather than absolute positions. Please refer to [Self-Attentionwith Relative Position Representations (Shaw et al.)](https://arxiv.org/abs/1803.02155) for more details.\n- the use of a causal depth-wise convolution instead of a non-causal one.\n\n### Generation process\n\nHere's how the generation process works:\n\n- Input text or speech is processed through its specific encoder.\n- A decoder creates text tokens in the desired language.\n- If speech generation is required, the second seq2seq model, generates unit tokens in an non auto-regressive way."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 854178f0-36db-45c7-b2eb-30e661d18179
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2", "title": "SeamlessM4T-v2", "tokens": 2580, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"854178f0-36db-45c7-b2eb-30e661d18179\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2\", \"title\": \"SeamlessM4T-v2\", \"tokens\": 2580, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2d2f92f2-3109-4298-9775-8aa63651dcdb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2\", \"title\": \"SeamlessM4T-v2\", \"tokens\": 2580, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"37fca48a2508dd873d46a86c895976af883f282a000c596c1db84d196a8ed78e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"87e95055-de21-445a-b6bb-a986e5e72033\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2\", \"title\": \"SeamlessM4T-v2\", \"tokens\": 2580, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b0930f04b1126154ea8dd5221b6e36f87fd077fbd5917bdc161359a1c559bb44\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8623, \"end_char_idx\": 11248, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2d2f92f2-3109-4298-9775-8aa63651dcdb", "doc_id": "2d2f92f2-3109-4298-9775-8aa63651dcdb", "ref_doc_id": "2d2f92f2-3109-4298-9775-8aa63651dcdb", "chroma:document": "### Difference with SeamlessM4T-v1\n\nThe architecture of this new version differs from the first in a few aspects:\n\n#### Improvements on the second-pass model\n\nThe second seq2seq model, named text-to-unit model, is now non-auto regressive, meaning that it computes units in a **single forward pass**. This achievement is made possible by:\n- the use of **character-level embeddings**, meaning that each character of the predicted translated text has its own embeddings, which are then used to predict the unit tokens.\n- the use of an intermediate duration predictor, that predicts speech duration at the **character-level** on the predicted translated text.\n- the use of a new text-to-unit decoder mixing convolutions and self-attention to handle longer context.\n\n#### Difference in the speech encoder\n\nThe speech encoder, which is used during the first-pass generation process to predict the translated text, differs mainly from the previous speech encoder through these mechanisms:\n- the use of chunked attention mask to prevent attention across chunks, ensuring that each position attends only to positions within its own chunk and a fixed number of previous chunks.\n- the use of relative position embeddings which only considers distance between sequence elements rather than absolute positions. Please refer to [Self-Attentionwith Relative Position Representations (Shaw et al.)](https://arxiv.org/abs/1803.02155) for more details.\n- the use of a causal depth-wise convolution instead of a non-causal one.\n\n### Generation process\n\nHere's how the generation process works:\n\n- Input text or speech is processed through its specific encoder.\n- A decoder creates text tokens in the desired language.\n- If speech generation is required, the second seq2seq model, generates unit tokens in an non auto-regressive way.\n- These unit tokens are then passed through the final vocoder to produce the actual speech.\n\n\nThis model was contributed by [ylacombe](https://huggingface.co/ylacombe). The original code can be found [here](https://github.com/facebookresearch/seamless_communication).\n\n## SeamlessM4Tv2Model\n\n[[autodoc]] SeamlessM4Tv2Model\n    - generate\n\n\n## SeamlessM4Tv2ForTextToSpeech\n\n[[autodoc]] SeamlessM4Tv2ForTextToSpeech\n    - generate\n\n\n## SeamlessM4Tv2ForSpeechToSpeech\n\n[[autodoc]] SeamlessM4Tv2ForSpeechToSpeech\n    - generate\n\n\n## SeamlessM4Tv2ForTextToText\n\n[[autodoc]] transformers.SeamlessM4Tv2ForTextToText\n    - forward\n    - generate\n\n## SeamlessM4Tv2ForSpeechToText\n\n[[autodoc]] transformers.SeamlessM4Tv2ForSpeechToText\n    - forward\n    - generate\n\n## SeamlessM4Tv2Config\n\n[[autodoc]] SeamlessM4Tv2Config"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 6ff4e023-13bd-44b9-8cd6-920f8e655ff7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vit_msn", "title": "ViTMSN", "tokens": 1465, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6ff4e023-13bd-44b9-8cd6-920f8e655ff7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_msn\", \"title\": \"ViTMSN\", \"tokens\": 1465, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"82b79054-5fb8-46d6-9874-a6dc53699aee\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_msn\", \"title\": \"ViTMSN\", \"tokens\": 1465, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3190299d5fce31c7d896ea5b4190690ffc9b3c8c3d4d5769212bf67361cd974b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f7f1252c-d76f-49cc-93b9-9a974972604e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"aa255320c91e0db463d18ac19d0834b0ae60749832dfa14119603bb8193913da\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2941, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "82b79054-5fb8-46d6-9874-a6dc53699aee", "doc_id": "82b79054-5fb8-46d6-9874-a6dc53699aee", "ref_doc_id": "82b79054-5fb8-46d6-9874-a6dc53699aee", "chroma:document": "# ViTMSN\n\n## Overview\n\nThe ViTMSN model was proposed in [Masked Siamese Networks for Label-Efficient Learning](https://arxiv.org/abs/2204.07141) by Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes,\nPascal Vincent, Armand Joulin, Michael Rabbat, Nicolas Ballas. The paper presents a joint-embedding architecture to match the prototypes\nof masked patches with that of the unmasked patches. With this setup, their method yields excellent performance in the low-shot and extreme low-shot\nregimes.\n\nThe abstract from the paper is the following:\n\n*We propose Masked Siamese Networks (MSN), a self-supervised learning framework for learning image representations. Our\napproach matches the representation of an image view containing randomly masked patches to the representation of the original\nunmasked image. This self-supervised pre-training strategy is particularly scalable when applied to Vision Transformers since only the\nunmasked patches are processed by the network. As a result, MSNs improve the scalability of joint-embedding architectures,\nwhile producing representations of a high semantic level that perform competitively on low-shot image classification. For instance,\non ImageNet-1K, with only 5,000 annotated images, our base MSN model achieves 72.4% top-1 accuracy,\nand with 1% of ImageNet-1K labels, we achieve 75.7% top-1 accuracy, setting a new state-of-the-art for self-supervised learning on this benchmark.*\n\n<img src=\"https://i.ibb.co/W6PQMdC/Screenshot-2022-09-13-at-9-08-40-AM.png\" alt=\"drawing\" width=\"600\"/> \n\n<small> MSN architecture. Taken from the <a href=\"https://arxiv.org/abs/2204.07141\">original paper.</a> </small>\n\nThis model was contributed by [sayakpaul](https://huggingface.co/sayakpaul). The original code can be found [here](https://github.com/facebookresearch/msn). \n\n## Usage tips\n\n- MSN (masked siamese networks) is a method for self-supervised pre-training of Vision Transformers (ViTs). The pre-training\nobjective is to match the prototypes assigned to the unmasked views of the images to that of the masked views of the same images.\n- The authors have only released pre-trained weights of the backbone (ImageNet-1k pre-training). So, to use that on your own image classification dataset,\nuse the [`ViTMSNForImageClassification`] class which is initialized from [`ViTMSNModel`]. Follow\n[this notebook](https://github.com/huggingface/notebooks/blob/main/examples/image_classification.ipynb) for a detailed tutorial on fine-tuning.\n- MSN is particularly useful in the low-shot and extreme low-shot regimes. Notably, it achieves 75.7% top-1 accuracy with only 1% of ImageNet-1K\nlabels when fine-tuned.\n\n### Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: f7f1252c-d76f-49cc-93b9-9a974972604e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vit_msn", "title": "ViTMSN", "tokens": 1465, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f7f1252c-d76f-49cc-93b9-9a974972604e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_msn\", \"title\": \"ViTMSN\", \"tokens\": 1465, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"82b79054-5fb8-46d6-9874-a6dc53699aee\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_msn\", \"title\": \"ViTMSN\", \"tokens\": 1465, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3190299d5fce31c7d896ea5b4190690ffc9b3c8c3d4d5769212bf67361cd974b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6ff4e023-13bd-44b9-8cd6-920f8e655ff7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_msn\", \"title\": \"ViTMSN\", \"tokens\": 1465, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f5940b0f1fd37c3b961c35d8f0643586bc1087079573dbe966f8c9254e97d34b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"51534b42-40ec-4d45-99f1-053053d86d1b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f18e90a610b2c2bf2562ac4e6bb1b4e04c4470ecc99d5f7d7d1d92a7709b5604\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1452, \"end_char_idx\": 3904, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "82b79054-5fb8-46d6-9874-a6dc53699aee", "doc_id": "82b79054-5fb8-46d6-9874-a6dc53699aee", "ref_doc_id": "82b79054-5fb8-46d6-9874-a6dc53699aee", "chroma:document": "*\n\n<img src=\"https://i.ibb.co/W6PQMdC/Screenshot-2022-09-13-at-9-08-40-AM.png\" alt=\"drawing\" width=\"600\"/> \n\n<small> MSN architecture. Taken from the <a href=\"https://arxiv.org/abs/2204.07141\">original paper.</a> </small>\n\nThis model was contributed by [sayakpaul](https://huggingface.co/sayakpaul). The original code can be found [here](https://github.com/facebookresearch/msn). \n\n## Usage tips\n\n- MSN (masked siamese networks) is a method for self-supervised pre-training of Vision Transformers (ViTs). The pre-training\nobjective is to match the prototypes assigned to the unmasked views of the images to that of the masked views of the same images.\n- The authors have only released pre-trained weights of the backbone (ImageNet-1k pre-training). So, to use that on your own image classification dataset,\nuse the [`ViTMSNForImageClassification`] class which is initialized from [`ViTMSNModel`]. Follow\n[this notebook](https://github.com/huggingface/notebooks/blob/main/examples/image_classification.ipynb) for a detailed tutorial on fine-tuning.\n- MSN is particularly useful in the low-shot and extreme low-shot regimes. Notably, it achieves 75.7% top-1 accuracy with only 1% of ImageNet-1K\nlabels when fine-tuned.\n\n### Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```\nfrom transformers import ViTMSNForImageClassification\nmodel = ViTMSNForImageClassification.from_pretrained(\"facebook/vit-msn-base\", attn_implementation=\"sdpa\", torch_dtype=torch.float16)\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `facebook/vit-msn-base` model, we saw the following speedups during inference."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 51534b42-40ec-4d45-99f1-053053d86d1b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vit_msn", "title": "ViTMSN", "tokens": 1465, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"51534b42-40ec-4d45-99f1-053053d86d1b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_msn\", \"title\": \"ViTMSN\", \"tokens\": 1465, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"82b79054-5fb8-46d6-9874-a6dc53699aee\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_msn\", \"title\": \"ViTMSN\", \"tokens\": 1465, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3190299d5fce31c7d896ea5b4190690ffc9b3c8c3d4d5769212bf67361cd974b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f7f1252c-d76f-49cc-93b9-9a974972604e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_msn\", \"title\": \"ViTMSN\", \"tokens\": 1465, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a42f440f1be22aa93224a5a436e55275441ce2799536995499343d1737158d14\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2349, \"end_char_idx\": 5668, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "82b79054-5fb8-46d6-9874-a6dc53699aee", "doc_id": "82b79054-5fb8-46d6-9874-a6dc53699aee", "ref_doc_id": "82b79054-5fb8-46d6-9874-a6dc53699aee", "chroma:document": "Follow\n[this notebook](https://github.com/huggingface/notebooks/blob/main/examples/image_classification.ipynb) for a detailed tutorial on fine-tuning.\n- MSN is particularly useful in the low-shot and extreme low-shot regimes. Notably, it achieves 75.7% top-1 accuracy with only 1% of ImageNet-1K\nlabels when fine-tuned.\n\n### Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```\nfrom transformers import ViTMSNForImageClassification\nmodel = ViTMSNForImageClassification.from_pretrained(\"facebook/vit-msn-base\", attn_implementation=\"sdpa\", torch_dtype=torch.float16)\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `facebook/vit-msn-base` model, we saw the following speedups during inference.\n\n|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |\n|--------------|-------------------------------------------|-------------------------------------------|------------------------------|\n|            1 |                                         7 |                                         6 |                      1.17 |\n|            2 |                                         8 |                                         6 |                      1.33 |\n|            4 |                                         8 |                                         6 |                      1.33 |\n|            8 |                                         8 |                                         6 |                      1.33 |\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with ViT MSN.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`ViTMSNForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## ViTMSNConfig\n\n[[autodoc]] ViTMSNConfig\n\n## ViTMSNModel\n\n[[autodoc]] ViTMSNModel\n    - forward\n\n## ViTMSNForImageClassification\n\n[[autodoc]] ViTMSNForImageClassification\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 42cb88e4-43b1-4684-ac1b-4ce5908d2255
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/oneformer", "title": "OneFormer", "tokens": 1357, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"42cb88e4-43b1-4684-ac1b-4ce5908d2255\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/oneformer\", \"title\": \"OneFormer\", \"tokens\": 1357, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"46b8cdf7-5cf7-432e-8918-02159991cdb6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/oneformer\", \"title\": \"OneFormer\", \"tokens\": 1357, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e25644105ec2643710c168684894bbc3a9e2bf1f03240f4133eb2e748ec8217b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7fd1a260-1236-4412-b630-dabd1a852eee\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2f6bedb1c220372b107306358a0681d3b21299a7c7f5a177d8900ea26200c0a6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3482, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "46b8cdf7-5cf7-432e-8918-02159991cdb6", "doc_id": "46b8cdf7-5cf7-432e-8918-02159991cdb6", "ref_doc_id": "46b8cdf7-5cf7-432e-8918-02159991cdb6", "chroma:document": "# OneFormer\n\n## Overview\n\nThe OneFormer model was proposed in [OneFormer: One Transformer to Rule Universal Image Segmentation](https://arxiv.org/abs/2211.06220) by Jitesh Jain, Jiachen Li, MangTik Chiu, Ali Hassani, Nikita Orlov, Humphrey Shi. OneFormer is a universal image segmentation framework that can be trained on a single panoptic dataset to perform semantic, instance, and panoptic segmentation tasks. OneFormer uses a task token to condition the model on the task in focus, making the architecture task-guided for training, and task-dynamic for inference.\n\n<img width=\"600\" src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/oneformer_teaser.png\"/>\n\nThe abstract from the paper is the following:\n\n*Universal Image Segmentation is not a new concept. Past attempts to unify image segmentation in the last decades include scene parsing, panoptic segmentation, and, more recently, new panoptic architectures. However, such panoptic architectures do not truly unify image segmentation because they need to be trained individually on the semantic, instance, or panoptic segmentation to achieve the best performance. Ideally, a truly universal framework should be trained only once and achieve SOTA performance across all three image segmentation tasks. To that end, we propose OneFormer, a universal image segmentation framework that unifies segmentation with a multi-task train-once design. We first propose a task-conditioned joint training strategy that enables training on ground truths of each domain (semantic, instance, and panoptic segmentation) within a single multi-task training process. Secondly, we introduce a task token to condition our model on the task at hand, making our model task-dynamic to support multi-task training and inference. Thirdly, we propose using a query-text contrastive loss during training to establish better inter-task and inter-class distinctions. Notably, our single OneFormer model outperforms specialized Mask2Former models across all three segmentation tasks on ADE20k, CityScapes, and COCO, despite the latter being trained on each of the three tasks individually with three times the resources. With new ConvNeXt and DiNAT backbones, we observe even more performance improvement. We believe OneFormer is a significant step towards making image segmentation more universal and accessible.*\n\nThe figure below illustrates the architecture of OneFormer. Taken from the [original paper](https://arxiv.org/abs/2211.06220).\n\n<img width=\"600\" src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/oneformer_architecture.png\"/>\n\nThis model was contributed by [Jitesh Jain](https://huggingface.co/praeclarumjj3). The original code can be found [here](https://github.com/SHI-Labs/OneFormer).\n\n## Usage tips\n\n-  OneFormer requires two inputs during inference: *image* and *task token*. \n- During training, OneFormer only uses panoptic annotations.\n- If you want to train the model in a distributed environment across multiple nodes, then one should update the\n  `get_num_masks` function inside in the `OneFormerLoss` class of `modeling_oneformer.py`. When training on multiple nodes, this should be\n  set to the average number of target masks across all nodes, as can be seen in the original implementation [here](https://github.com/SHI-Labs/OneFormer/blob/33ebb56ed34f970a30ae103e786c0cb64c653d9a/oneformer/modeling/criterion.py#L287)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 7fd1a260-1236-4412-b630-dabd1a852eee
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/oneformer", "title": "OneFormer", "tokens": 1357, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7fd1a260-1236-4412-b630-dabd1a852eee\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/oneformer\", \"title\": \"OneFormer\", \"tokens\": 1357, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"46b8cdf7-5cf7-432e-8918-02159991cdb6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/oneformer\", \"title\": \"OneFormer\", \"tokens\": 1357, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e25644105ec2643710c168684894bbc3a9e2bf1f03240f4133eb2e748ec8217b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"42cb88e4-43b1-4684-ac1b-4ce5908d2255\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/oneformer\", \"title\": \"OneFormer\", \"tokens\": 1357, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fb99f3051e6bb3bcf462cdb3bf4b6626efaf567cf080eb68977e4836a4b8d5ed\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"18afe308-1a00-44d3-b456-1830d3b0115d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f1b33ed8f1a7972823247100188125b82de29ee031ecf2af4efc41f862f06e5a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1815, \"end_char_idx\": 4766, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "46b8cdf7-5cf7-432e-8918-02159991cdb6", "doc_id": "46b8cdf7-5cf7-432e-8918-02159991cdb6", "ref_doc_id": "46b8cdf7-5cf7-432e-8918-02159991cdb6", "chroma:document": "Thirdly, we propose using a query-text contrastive loss during training to establish better inter-task and inter-class distinctions. Notably, our single OneFormer model outperforms specialized Mask2Former models across all three segmentation tasks on ADE20k, CityScapes, and COCO, despite the latter being trained on each of the three tasks individually with three times the resources. With new ConvNeXt and DiNAT backbones, we observe even more performance improvement. We believe OneFormer is a significant step towards making image segmentation more universal and accessible.*\n\nThe figure below illustrates the architecture of OneFormer. Taken from the [original paper](https://arxiv.org/abs/2211.06220).\n\n<img width=\"600\" src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/oneformer_architecture.png\"/>\n\nThis model was contributed by [Jitesh Jain](https://huggingface.co/praeclarumjj3). The original code can be found [here](https://github.com/SHI-Labs/OneFormer).\n\n## Usage tips\n\n-  OneFormer requires two inputs during inference: *image* and *task token*. \n- During training, OneFormer only uses panoptic annotations.\n- If you want to train the model in a distributed environment across multiple nodes, then one should update the\n  `get_num_masks` function inside in the `OneFormerLoss` class of `modeling_oneformer.py`. When training on multiple nodes, this should be\n  set to the average number of target masks across all nodes, as can be seen in the original implementation [here](https://github.com/SHI-Labs/OneFormer/blob/33ebb56ed34f970a30ae103e786c0cb64c653d9a/oneformer/modeling/criterion.py#L287).\n- One can use [`OneFormerProcessor`] to prepare input images and task inputs for the model and optional targets for the model. [`OneformerProcessor`] wraps [`OneFormerImageProcessor`] and [`CLIPTokenizer`] into a single instance to both prepare the images and encode the task inputs.\n- To get the final segmentation, depending on the task, you can call [`~OneFormerProcessor.post_process_semantic_segmentation`] or [`~OneFormerImageProcessor.post_process_instance_segmentation`] or [`~OneFormerImageProcessor.post_process_panoptic_segmentation`]. All three tasks can be solved using [`OneFormerForUniversalSegmentation`] output, panoptic segmentation accepts an optional `label_ids_to_fuse` argument to fuse instances of the target object/s (e.g. sky) together.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with OneFormer.\n\n- Demo notebooks regarding inference + fine-tuning on custom data can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/OneFormer).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it.\nThe resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 18afe308-1a00-44d3-b456-1830d3b0115d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/oneformer", "title": "OneFormer", "tokens": 1357, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"18afe308-1a00-44d3-b456-1830d3b0115d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/oneformer\", \"title\": \"OneFormer\", \"tokens\": 1357, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"46b8cdf7-5cf7-432e-8918-02159991cdb6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/oneformer\", \"title\": \"OneFormer\", \"tokens\": 1357, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e25644105ec2643710c168684894bbc3a9e2bf1f03240f4133eb2e748ec8217b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7fd1a260-1236-4412-b630-dabd1a852eee\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/oneformer\", \"title\": \"OneFormer\", \"tokens\": 1357, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"939a1c7e988168925c49b6d2d1d15560056dc37d89e4db4a9aef4c76cf16a72d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3197, \"end_char_idx\": 5445, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "46b8cdf7-5cf7-432e-8918-02159991cdb6", "doc_id": "46b8cdf7-5cf7-432e-8918-02159991cdb6", "ref_doc_id": "46b8cdf7-5cf7-432e-8918-02159991cdb6", "chroma:document": "When training on multiple nodes, this should be\n  set to the average number of target masks across all nodes, as can be seen in the original implementation [here](https://github.com/SHI-Labs/OneFormer/blob/33ebb56ed34f970a30ae103e786c0cb64c653d9a/oneformer/modeling/criterion.py#L287).\n- One can use [`OneFormerProcessor`] to prepare input images and task inputs for the model and optional targets for the model. [`OneformerProcessor`] wraps [`OneFormerImageProcessor`] and [`CLIPTokenizer`] into a single instance to both prepare the images and encode the task inputs.\n- To get the final segmentation, depending on the task, you can call [`~OneFormerProcessor.post_process_semantic_segmentation`] or [`~OneFormerImageProcessor.post_process_instance_segmentation`] or [`~OneFormerImageProcessor.post_process_panoptic_segmentation`]. All three tasks can be solved using [`OneFormerForUniversalSegmentation`] output, panoptic segmentation accepts an optional `label_ids_to_fuse` argument to fuse instances of the target object/s (e.g. sky) together.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with OneFormer.\n\n- Demo notebooks regarding inference + fine-tuning on custom data can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/OneFormer).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it.\nThe resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## OneFormer specific outputs\n\n[[autodoc]] models.oneformer.modeling_oneformer.OneFormerModelOutput\n\n[[autodoc]] models.oneformer.modeling_oneformer.OneFormerForUniversalSegmentationOutput\n\n## OneFormerConfig\n\n[[autodoc]] OneFormerConfig\n\n## OneFormerImageProcessor\n\n[[autodoc]] OneFormerImageProcessor\n    - preprocess\n    - encode_inputs\n    - post_process_semantic_segmentation\n    - post_process_instance_segmentation\n    - post_process_panoptic_segmentation\n\n## OneFormerProcessor\n\n[[autodoc]] OneFormerProcessor\n\n## OneFormerModel\n\n[[autodoc]] OneFormerModel\n    - forward\n\n## OneFormerForUniversalSegmentation\n\n[[autodoc]] OneFormerForUniversalSegmentation\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 5d53c6f2-8746-42fe-9ac3-6e4a4926e9f4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/sew", "title": "SEW", "tokens": 629, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5d53c6f2-8746-42fe-9ac3-6e4a4926e9f4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/sew\", \"title\": \"SEW\", \"tokens\": 629, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"91072039-36a8-49c3-a58b-f15559a3bb91\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/sew\", \"title\": \"SEW\", \"tokens\": 629, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"48c93e7aacb4789c8c0d458461eafea0471b5599c273e6d4b3cedc401536e322\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1811, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "91072039-36a8-49c3-a58b-f15559a3bb91", "doc_id": "91072039-36a8-49c3-a58b-f15559a3bb91", "ref_doc_id": "91072039-36a8-49c3-a58b-f15559a3bb91", "chroma:document": "# SEW\n\n## Overview\n\nSEW (Squeezed and Efficient Wav2Vec) was proposed in [Performance-Efficiency Trade-offs in Unsupervised Pre-training\nfor Speech Recognition](https://arxiv.org/abs/2109.06870) by Felix Wu, Kwangyoun Kim, Jing Pan, Kyu Han, Kilian Q.\nWeinberger, Yoav Artzi.\n\nThe abstract from the paper is the following:\n\n*This paper is a study of performance-efficiency trade-offs in pre-trained models for automatic speech recognition\n(ASR). We focus on wav2vec 2.0, and formalize several architecture designs that influence both the model performance\nand its efficiency. Putting together all our observations, we introduce SEW (Squeezed and Efficient Wav2vec), a\npre-trained model architecture with significant improvements along both performance and efficiency dimensions across a\nvariety of training setups. For example, under the 100h-960h semi-supervised setup on LibriSpeech, SEW achieves a 1.9x\ninference speedup compared to wav2vec 2.0, with a 13.5% relative reduction in word error rate. With a similar inference\ntime, SEW reduces word error rate by 25-50% across different model sizes.*\n\nThis model was contributed by [anton-l](https://huggingface.co/anton-l).\n\n## Usage tips\n\n- SEW is a speech model that accepts a float array corresponding to the raw waveform of the speech signal.\n- SEWForCTC is fine-tuned using connectionist temporal classification (CTC) so the model output has to be decoded using\n  [`Wav2Vec2CTCTokenizer`].\n\n## Resources\n\n- [Audio classification task guide](../tasks/audio_classification)\n- [Automatic speech recognition task guide](../tasks/asr)\n\n## SEWConfig\n\n[[autodoc]] SEWConfig\n\n## SEWModel\n\n[[autodoc]] SEWModel\n    - forward\n\n## SEWForCTC\n\n[[autodoc]] SEWForCTC\n    - forward\n\n## SEWForSequenceClassification\n\n[[autodoc]] SEWForSequenceClassification\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 0eb67e93-77e5-4ca6-a262-81986421b5b6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/altclip", "title": "AltCLIP", "tokens": 1185, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0eb67e93-77e5-4ca6-a262-81986421b5b6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/altclip\", \"title\": \"AltCLIP\", \"tokens\": 1185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"09298fbb-62d2-4c25-a0da-d7dafea1067e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/altclip\", \"title\": \"AltCLIP\", \"tokens\": 1185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7f5f33c1273d0ef1283fb7b8cb8974be2aa8c1405422beaed3e59413eb0eafd8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"96814ca3-4aef-4fa0-8550-24516a7c814a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"592ccbb7cfd3ec4ff63797fb6a80e4e7dd13e4036ff74e872b8fd85bb8c09661\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2996, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "09298fbb-62d2-4c25-a0da-d7dafea1067e", "doc_id": "09298fbb-62d2-4c25-a0da-d7dafea1067e", "ref_doc_id": "09298fbb-62d2-4c25-a0da-d7dafea1067e", "chroma:document": "# AltCLIP\n\n## Overview\n\nThe AltCLIP model was proposed in [AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities](https://arxiv.org/abs/2211.06679v2) by Zhongzhi Chen, Guang Liu, Bo-Wen Zhang, Fulong Ye, Qinghong Yang, Ledell Wu. AltCLIP\n(Altering the Language Encoder in CLIP) is a neural network trained on a variety of image-text and text-text pairs. By switching CLIP's\ntext encoder with a pretrained multilingual text encoder XLM-R, we could obtain very close performances with CLIP on almost all tasks, and extended original CLIP's capabilities such as multilingual understanding.\n\nThe abstract from the paper is the following:\n\n*In this work, we present a conceptually simple and effective method to train a strong bilingual multimodal representation model. \nStarting from the pretrained multimodal representation model CLIP released by OpenAI, we switched its text encoder with a pretrained \nmultilingual text encoder XLM-R, and aligned both languages and image representations by a two-stage training schema consisting of \nteacher learning and contrastive learning. We validate our method through evaluations of a wide range of tasks. We set new state-of-the-art \nperformances on a bunch of tasks including ImageNet-CN, Flicker30k- CN, and COCO-CN. Further, we obtain very close performances with \nCLIP on almost all tasks, suggesting that one can simply alter the text encoder in CLIP for extended capabilities such as multilingual understanding.*\n\nThis model was contributed by [jongjyh](https://huggingface.co/jongjyh).\n\n## Usage tips and example\n\nThe usage of AltCLIP is very similar to the CLIP. the difference between CLIP is the text encoder. Note that we use bidirectional attention instead of casual attention\nand we take the [CLS] token in XLM-R to represent text embedding.\n\nAltCLIP is a multi-modal vision and language model. It can be used for image-text similarity and for zero-shot image\nclassification. AltCLIP uses a ViT like transformer to get visual features and a bidirectional language model to get the text\nfeatures. Both the text and visual features are then projected to a latent space with identical dimension. The dot\nproduct between the projected image and text features is then used as a similar score.\n\nTo feed images to the Transformer encoder, each image is split into a sequence of fixed-size non-overlapping patches,\nwhich are then linearly embedded. A [CLS] token is added to serve as representation of an entire image. The authors\nalso add absolute position embeddings, and feed the resulting sequence of vectors to a standard Transformer encoder.\nThe [`CLIPImageProcessor`] can be used to resize (or rescale) and normalize images for the model.\n\nThe [`AltCLIPProcessor`] wraps a [`CLIPImageProcessor`] and a [`XLMRobertaTokenizer`] into a single instance to both\nencode the text and prepare the images. The following example shows how to get the image-text similarity scores using\n[`AltCLIPProcessor`] and [`AltCLIPModel`]."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 96814ca3-4aef-4fa0-8550-24516a7c814a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/altclip", "title": "AltCLIP", "tokens": 1185, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"96814ca3-4aef-4fa0-8550-24516a7c814a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/altclip\", \"title\": \"AltCLIP\", \"tokens\": 1185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"09298fbb-62d2-4c25-a0da-d7dafea1067e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/altclip\", \"title\": \"AltCLIP\", \"tokens\": 1185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7f5f33c1273d0ef1283fb7b8cb8974be2aa8c1405422beaed3e59413eb0eafd8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0eb67e93-77e5-4ca6-a262-81986421b5b6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/altclip\", \"title\": \"AltCLIP\", \"tokens\": 1185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b1ea86842c272b8e69db36ecbb603c80d80ba84983bf0300d3c4aaa1d0bbce47\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1286, \"end_char_idx\": 4294, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "09298fbb-62d2-4c25-a0da-d7dafea1067e", "doc_id": "09298fbb-62d2-4c25-a0da-d7dafea1067e", "ref_doc_id": "09298fbb-62d2-4c25-a0da-d7dafea1067e", "chroma:document": "Further, we obtain very close performances with \nCLIP on almost all tasks, suggesting that one can simply alter the text encoder in CLIP for extended capabilities such as multilingual understanding.*\n\nThis model was contributed by [jongjyh](https://huggingface.co/jongjyh).\n\n## Usage tips and example\n\nThe usage of AltCLIP is very similar to the CLIP. the difference between CLIP is the text encoder. Note that we use bidirectional attention instead of casual attention\nand we take the [CLS] token in XLM-R to represent text embedding.\n\nAltCLIP is a multi-modal vision and language model. It can be used for image-text similarity and for zero-shot image\nclassification. AltCLIP uses a ViT like transformer to get visual features and a bidirectional language model to get the text\nfeatures. Both the text and visual features are then projected to a latent space with identical dimension. The dot\nproduct between the projected image and text features is then used as a similar score.\n\nTo feed images to the Transformer encoder, each image is split into a sequence of fixed-size non-overlapping patches,\nwhich are then linearly embedded. A [CLS] token is added to serve as representation of an entire image. The authors\nalso add absolute position embeddings, and feed the resulting sequence of vectors to a standard Transformer encoder.\nThe [`CLIPImageProcessor`] can be used to resize (or rescale) and normalize images for the model.\n\nThe [`AltCLIPProcessor`] wraps a [`CLIPImageProcessor`] and a [`XLMRobertaTokenizer`] into a single instance to both\nencode the text and prepare the images. The following example shows how to get the image-text similarity scores using\n[`AltCLIPProcessor`] and [`AltCLIPModel`].\n\n```python\n>>> from PIL import Image\n>>> import requests\n\n>>> from transformers import AltCLIPModel, AltCLIPProcessor\n\n>>> model = AltCLIPModel.from_pretrained(\"BAAI/AltCLIP\")\n>>> processor = AltCLIPProcessor.from_pretrained(\"BAAI/AltCLIP\")\n\n>>> url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n\n>>> inputs = processor(text=[\"a photo of a cat\", \"a photo of a dog\"], images=image, return_tensors=\"pt\", padding=True)\n\n>>> outputs = model(**inputs)\n>>> logits_per_image = outputs.logits_per_image  # this is the image-text similarity score\n>>> probs = logits_per_image.softmax(dim=1)  # we can take the softmax to get the label probabilities\n```\n\n<Tip>\n\nThis model is based on `CLIPModel`, use it like you would use the original [CLIP](clip).\n\n</Tip>\n\n## AltCLIPConfig\n\n[[autodoc]] AltCLIPConfig\n    - from_text_vision_configs\n\n## AltCLIPTextConfig\n\n[[autodoc]] AltCLIPTextConfig\n\n## AltCLIPVisionConfig\n\n[[autodoc]] AltCLIPVisionConfig\n\n## AltCLIPProcessor\n\n[[autodoc]] AltCLIPProcessor\n\n## AltCLIPModel\n\n[[autodoc]] AltCLIPModel\n    - forward\n    - get_text_features\n    - get_image_features\n\n## AltCLIPTextModel\n\n[[autodoc]] AltCLIPTextModel\n    - forward\n\n## AltCLIPVisionModel\n\n[[autodoc]] AltCLIPVisionModel\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 21382cd0-8ffd-4eea-98c2-68f4c2a22c39
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/paligemma", "title": "PaliGemma", "tokens": 1039, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"21382cd0-8ffd-4eea-98c2-68f4c2a22c39\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/paligemma\", \"title\": \"PaliGemma\", \"tokens\": 1039, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8bedb4b7-0b40-4c9a-90b2-7ff6fa1330f9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/paligemma\", \"title\": \"PaliGemma\", \"tokens\": 1039, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bedc083f2570c825df7e8f8b4f61b04ab2a997667c2ed1cbe90abd7b6c28d376\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9f61fe8e-8fd9-49fa-ad51-ae8f163c43b7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1c0a9a223d9d212987dc2f2a45726c2147d1b52cf1f64272755adcf7d43a0848\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3094, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8bedb4b7-0b40-4c9a-90b2-7ff6fa1330f9", "doc_id": "8bedb4b7-0b40-4c9a-90b2-7ff6fa1330f9", "ref_doc_id": "8bedb4b7-0b40-4c9a-90b2-7ff6fa1330f9", "chroma:document": "# PaliGemma\n\n## Overview\n\nThe PaliGemma model was proposed in [PaliGemma \u2013 Google's Cutting-Edge Open Vision Language Model](https://huggingface.co/blog/paligemma) by Google. It is a 3B vision-language model composed by a [SigLIP](siglip) vision encoder and a [Gemma](gemma) language decoder linked by a multimodal linear projection. It cuts an image into a fixed number of VIT tokens and prepends it to an optional prompt. One particularity is that the model uses full block attention on all the image tokens plus the input text tokens. It comes in 3 resolutions, 224x224, 448x448 and 896x896 with 3 base models, with 55 fine-tuned versions for different tasks, and 2 mix models.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/paligemma/paligemma_arch.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> PaliGemma architecture. Taken from the <a href=\"https://huggingface.co/blog/paligemma\">blog post.</a> </small>\n\nThis model was contributed by [Molbap](https://huggingface.co/Molbap).\n\n## Usage tips\n\nInference with PaliGemma can be performed as follows:\n\n```python\nfrom transformers import AutoProcessor, PaliGemmaForConditionalGeneration\n\nmodel_id = \"google/paligemma-3b-mix-224\"\nmodel = PaliGemmaForConditionalGeneration.from_pretrained(model_id)\nprocessor = AutoProcessor.from_pretrained(model_id)\n\nprompt = \"What is on the flower?\"\nimage_file = \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg?download=true\"\nraw_image = Image.open(requests.get(image_file, stream=True).raw)\ninputs = processor(prompt, raw_image, return_tensors=\"pt\")\noutput = model.generate(**inputs, max_new_tokens=20)\n\nprint(processor.decode(output[0], skip_special_tokens=True)[len(prompt):])\n```\n\n- PaliGemma is not meant for conversational use, and it works best when fine-tuning to a specific use case. Some downstream tasks on which PaliGemma can be fine-tuned include image captioning, visual question answering (VQA), object detection, referring expression segmentation and document understanding.\n- One can use `PaliGemmaProcessor` to prepare images, text and optional labels for the model. When fine-tuning a PaliGemma model, the `suffix` argument can be passed to the processor which creates the `labels` for the model:\n\n```python\nprompt = \"What is on the flower?\"\nanswer = \"a bee\"\ninputs = processor(text=prompt, images=raw_image, suffix=answer, return_tensors=\"pt\")\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with PaliGemma. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n- A blog post introducing all the features of PaliGemma can be found [here](https://huggingface.co/blog/paligemma).\n- Demo notebooks on how to fine-tune PaliGemma for VQA with the Trainer API along with inference can be found [here](https://github.com/huggingface/notebooks/tree/main/examples/paligemma)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 9f61fe8e-8fd9-49fa-ad51-ae8f163c43b7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/paligemma", "title": "PaliGemma", "tokens": 1039, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9f61fe8e-8fd9-49fa-ad51-ae8f163c43b7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/paligemma\", \"title\": \"PaliGemma\", \"tokens\": 1039, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8bedb4b7-0b40-4c9a-90b2-7ff6fa1330f9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/paligemma\", \"title\": \"PaliGemma\", \"tokens\": 1039, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bedc083f2570c825df7e8f8b4f61b04ab2a997667c2ed1cbe90abd7b6c28d376\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"21382cd0-8ffd-4eea-98c2-68f4c2a22c39\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/paligemma\", \"title\": \"PaliGemma\", \"tokens\": 1039, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"29b13b939a85a7a340c15623c2a63500b3448408487d054540e59efd59a851fb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1859, \"end_char_idx\": 3508, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8bedb4b7-0b40-4c9a-90b2-7ff6fa1330f9", "doc_id": "8bedb4b7-0b40-4c9a-90b2-7ff6fa1330f9", "ref_doc_id": "8bedb4b7-0b40-4c9a-90b2-7ff6fa1330f9", "chroma:document": "Some downstream tasks on which PaliGemma can be fine-tuned include image captioning, visual question answering (VQA), object detection, referring expression segmentation and document understanding.\n- One can use `PaliGemmaProcessor` to prepare images, text and optional labels for the model. When fine-tuning a PaliGemma model, the `suffix` argument can be passed to the processor which creates the `labels` for the model:\n\n```python\nprompt = \"What is on the flower?\"\nanswer = \"a bee\"\ninputs = processor(text=prompt, images=raw_image, suffix=answer, return_tensors=\"pt\")\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with PaliGemma. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n- A blog post introducing all the features of PaliGemma can be found [here](https://huggingface.co/blog/paligemma).\n- Demo notebooks on how to fine-tune PaliGemma for VQA with the Trainer API along with inference can be found [here](https://github.com/huggingface/notebooks/tree/main/examples/paligemma).\n- Demo notebooks on how to fine-tune PaliGemma on a custom dataset (receipt image -> JSON) along with inference can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/PaliGemma). \ud83c\udf0e\n\n## PaliGemmaConfig\n\n[[autodoc]] PaliGemmaConfig\n\n## PaliGemmaProcessor\n\n[[autodoc]] PaliGemmaProcessor\n\n## PaliGemmaForConditionalGeneration\n\n[[autodoc]] PaliGemmaForConditionalGeneration\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 217170e3-34e2-4bca-8b0c-95536c113b69
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/encoder-decoder", "title": "Encoder Decoder Models", "tokens": 2117, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"217170e3-34e2-4bca-8b0c-95536c113b69\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/encoder-decoder\", \"title\": \"Encoder Decoder Models\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b4ae63ab-a48f-4597-b98b-c9a88839c867\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/encoder-decoder\", \"title\": \"Encoder Decoder Models\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e2efdaa1bfb1a39545221ba209d082233a8c938ece0da1cbc4b100f3bb8b46f5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"de27a28d-c2f6-4075-9896-fb45240a17e8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b95a7e96913cb57cce5d38f405fab9a95088afff62ccc00a0ec909cc4d1f2278\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3392, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b4ae63ab-a48f-4597-b98b-c9a88839c867", "doc_id": "b4ae63ab-a48f-4597-b98b-c9a88839c867", "ref_doc_id": "b4ae63ab-a48f-4597-b98b-c9a88839c867", "chroma:document": "# Encoder Decoder Models\n\n## Overview\n\nThe [`EncoderDecoderModel`] can be used to initialize a sequence-to-sequence model with any\npretrained autoencoding model as the encoder and any pretrained autoregressive model as the decoder.\n\nThe effectiveness of initializing sequence-to-sequence models with pretrained checkpoints for sequence generation tasks\nwas shown in [Leveraging Pre-trained Checkpoints for Sequence Generation Tasks](https://arxiv.org/abs/1907.12461) by\nSascha Rothe, Shashi Narayan, Aliaksei Severyn.\n\nAfter such an [`EncoderDecoderModel`] has been trained/fine-tuned, it can be saved/loaded just like\nany other models (see the examples for more information).\n\nAn application of this architecture could be to leverage two pretrained [`BertModel`] as the encoder\nand decoder for a summarization model as was shown in: [Text Summarization with Pretrained Encoders](https://arxiv.org/abs/1908.08345) by Yang Liu and Mirella Lapata.\n\n## Randomly initializing `EncoderDecoderModel` from model configurations.\n\n[`EncoderDecoderModel`] can be randomly initialized from an encoder and a decoder config. In the following example, we show how to do this using the default [`BertModel`] configuration for the encoder and the default [`BertForCausalLM`] configuration for the decoder.\n\n```python\n>>> from transformers import BertConfig, EncoderDecoderConfig, EncoderDecoderModel\n\n>>> config_encoder = BertConfig()\n>>> config_decoder = BertConfig()\n\n>>> config = EncoderDecoderConfig.from_encoder_decoder_configs(config_encoder, config_decoder)\n>>> model = EncoderDecoderModel(config=config)\n```\n\n## Initialising `EncoderDecoderModel` from a pretrained encoder and a pretrained decoder.\n\n[`EncoderDecoderModel`] can be initialized from a pretrained encoder checkpoint and a pretrained decoder checkpoint. Note that any pretrained auto-encoding model, *e.g.* BERT, can serve as the encoder and both pretrained auto-encoding models, *e.g.* BERT, pretrained causal language models, *e.g.* GPT2, as well as the pretrained decoder part of sequence-to-sequence models, *e.g.* decoder of BART, can be used as the decoder.\nDepending on which architecture you choose as the decoder, the cross-attention layers might be randomly initialized.\nInitializing [`EncoderDecoderModel`] from a pretrained encoder and decoder checkpoint requires the model to be fine-tuned on a downstream task, as has been shown in [the *Warm-starting-encoder-decoder blog post*](https://huggingface.co/blog/warm-starting-encoder-decoder).\nTo do so, the `EncoderDecoderModel` class provides a [`EncoderDecoderModel.from_encoder_decoder_pretrained`] method.\n\n```python\n>>> from transformers import EncoderDecoderModel, BertTokenizer\n\n>>> tokenizer = BertTokenizer.from_pretrained(\"google-bert/bert-base-uncased\")\n>>> model = EncoderDecoderModel.from_encoder_decoder_pretrained(\"google-bert/bert-base-uncased\", \"google-bert/bert-base-uncased\")\n```\n\n## Loading an existing `EncoderDecoderModel` checkpoint and perform inference.\n\nTo load fine-tuned checkpoints of the `EncoderDecoderModel` class, [`EncoderDecoderModel`] provides the `from_pretrained(...)` method just like any other model architecture in Transformers.\n\nTo perform inference, one uses the [`generate`] method, which allows to autoregressively generate text. This method supports various forms of decoding, such as greedy, beam search and multinomial sampling."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: de27a28d-c2f6-4075-9896-fb45240a17e8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/encoder-decoder", "title": "Encoder Decoder Models", "tokens": 2117, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"de27a28d-c2f6-4075-9896-fb45240a17e8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/encoder-decoder\", \"title\": \"Encoder Decoder Models\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b4ae63ab-a48f-4597-b98b-c9a88839c867\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/encoder-decoder\", \"title\": \"Encoder Decoder Models\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e2efdaa1bfb1a39545221ba209d082233a8c938ece0da1cbc4b100f3bb8b46f5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"217170e3-34e2-4bca-8b0c-95536c113b69\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/encoder-decoder\", \"title\": \"Encoder Decoder Models\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e88bb2fdb3af0bb37f9240ce62105bb947bfe5a76ba187dedfec987f2496b910\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5dca5bc7-9145-44b0-988a-2b6e52c89bcf\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b852675c1986f54dcd15c5d45b997ca5592027b501a1171722a88a4ea9ceabea\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1692, \"end_char_idx\": 4961, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b4ae63ab-a48f-4597-b98b-c9a88839c867", "doc_id": "b4ae63ab-a48f-4597-b98b-c9a88839c867", "ref_doc_id": "b4ae63ab-a48f-4597-b98b-c9a88839c867", "chroma:document": "[`EncoderDecoderModel`] can be initialized from a pretrained encoder checkpoint and a pretrained decoder checkpoint. Note that any pretrained auto-encoding model, *e.g.* BERT, can serve as the encoder and both pretrained auto-encoding models, *e.g.* BERT, pretrained causal language models, *e.g.* GPT2, as well as the pretrained decoder part of sequence-to-sequence models, *e.g.* decoder of BART, can be used as the decoder.\nDepending on which architecture you choose as the decoder, the cross-attention layers might be randomly initialized.\nInitializing [`EncoderDecoderModel`] from a pretrained encoder and decoder checkpoint requires the model to be fine-tuned on a downstream task, as has been shown in [the *Warm-starting-encoder-decoder blog post*](https://huggingface.co/blog/warm-starting-encoder-decoder).\nTo do so, the `EncoderDecoderModel` class provides a [`EncoderDecoderModel.from_encoder_decoder_pretrained`] method.\n\n```python\n>>> from transformers import EncoderDecoderModel, BertTokenizer\n\n>>> tokenizer = BertTokenizer.from_pretrained(\"google-bert/bert-base-uncased\")\n>>> model = EncoderDecoderModel.from_encoder_decoder_pretrained(\"google-bert/bert-base-uncased\", \"google-bert/bert-base-uncased\")\n```\n\n## Loading an existing `EncoderDecoderModel` checkpoint and perform inference.\n\nTo load fine-tuned checkpoints of the `EncoderDecoderModel` class, [`EncoderDecoderModel`] provides the `from_pretrained(...)` method just like any other model architecture in Transformers.\n\nTo perform inference, one uses the [`generate`] method, which allows to autoregressively generate text. This method supports various forms of decoding, such as greedy, beam search and multinomial sampling.\n\n```python\n>>> from transformers import AutoTokenizer, EncoderDecoderModel\n\n>>> # load a fine-tuned seq2seq model and corresponding tokenizer\n>>> model = EncoderDecoderModel.from_pretrained(\"patrickvonplaten/bert2bert_cnn_daily_mail\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"patrickvonplaten/bert2bert_cnn_daily_mail\")\n\n>>> # let's perform inference on a long piece of text\n>>> ARTICLE_TO_SUMMARIZE = (\n...     \"PG&E stated it scheduled the blackouts in response to forecasts for high winds \"\n...     \"amid dry conditions. The aim is to reduce the risk of wildfires. Nearly 800 thousand customers were \"\n...     \"scheduled to be affected by the shutoffs which were expected to last through at least midday tomorrow.\"\n... )\n>>> input_ids = tokenizer(ARTICLE_TO_SUMMARIZE, return_tensors=\"pt\").input_ids\n\n>>> # autoregressively generate summary (uses greedy decoding by default)\n>>> generated_ids = model.generate(input_ids)\n>>> generated_text = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]\n>>> print(generated_text)\nnearly 800 thousand customers were affected by the shutoffs. the aim is to reduce the risk of wildfires. nearly 800, 000 customers were expected to be affected by high winds amid dry conditions. pg & e said it scheduled the blackouts to last through at least midday tomorrow.\n```\n\n## Loading a PyTorch checkpoint into `TFEncoderDecoderModel`.\n\n[`TFEncoderDecoderModel.from_pretrained`] currently doesn't support initializing the model from a\npytorch checkpoint. Passing `from_pt=True` to this method will throw an exception."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 5dca5bc7-9145-44b0-988a-2b6e52c89bcf
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/encoder-decoder", "title": "Encoder Decoder Models", "tokens": 2117, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5dca5bc7-9145-44b0-988a-2b6e52c89bcf\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/encoder-decoder\", \"title\": \"Encoder Decoder Models\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b4ae63ab-a48f-4597-b98b-c9a88839c867\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/encoder-decoder\", \"title\": \"Encoder Decoder Models\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e2efdaa1bfb1a39545221ba209d082233a8c938ece0da1cbc4b100f3bb8b46f5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"de27a28d-c2f6-4075-9896-fb45240a17e8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/encoder-decoder\", \"title\": \"Encoder Decoder Models\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6e1fc1bdfb6bf9a1d70d26b1b45639cb40fe00216303fe7ea5b7b3d641afb613\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c7446985-eee0-44e1-a858-810dcd0d4bbc\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e5c66299667eee86a06fd473c110aeb101ff6e5a7e8525ef41ce2ec6f8d6674d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3187, \"end_char_idx\": 6570, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b4ae63ab-a48f-4597-b98b-c9a88839c867", "doc_id": "b4ae63ab-a48f-4597-b98b-c9a88839c867", "ref_doc_id": "b4ae63ab-a48f-4597-b98b-c9a88839c867", "chroma:document": "To perform inference, one uses the [`generate`] method, which allows to autoregressively generate text. This method supports various forms of decoding, such as greedy, beam search and multinomial sampling.\n\n```python\n>>> from transformers import AutoTokenizer, EncoderDecoderModel\n\n>>> # load a fine-tuned seq2seq model and corresponding tokenizer\n>>> model = EncoderDecoderModel.from_pretrained(\"patrickvonplaten/bert2bert_cnn_daily_mail\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"patrickvonplaten/bert2bert_cnn_daily_mail\")\n\n>>> # let's perform inference on a long piece of text\n>>> ARTICLE_TO_SUMMARIZE = (\n...     \"PG&E stated it scheduled the blackouts in response to forecasts for high winds \"\n...     \"amid dry conditions. The aim is to reduce the risk of wildfires. Nearly 800 thousand customers were \"\n...     \"scheduled to be affected by the shutoffs which were expected to last through at least midday tomorrow.\"\n... )\n>>> input_ids = tokenizer(ARTICLE_TO_SUMMARIZE, return_tensors=\"pt\").input_ids\n\n>>> # autoregressively generate summary (uses greedy decoding by default)\n>>> generated_ids = model.generate(input_ids)\n>>> generated_text = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]\n>>> print(generated_text)\nnearly 800 thousand customers were affected by the shutoffs. the aim is to reduce the risk of wildfires. nearly 800, 000 customers were expected to be affected by high winds amid dry conditions. pg & e said it scheduled the blackouts to last through at least midday tomorrow.\n```\n\n## Loading a PyTorch checkpoint into `TFEncoderDecoderModel`.\n\n[`TFEncoderDecoderModel.from_pretrained`] currently doesn't support initializing the model from a\npytorch checkpoint. Passing `from_pt=True` to this method will throw an exception. If there are only pytorch\ncheckpoints for a particular encoder-decoder model, a workaround is:\n\n```python\n>>> # a workaround to load from pytorch checkpoint\n>>> from transformers import EncoderDecoderModel, TFEncoderDecoderModel\n\n>>> _model = EncoderDecoderModel.from_pretrained(\"patrickvonplaten/bert2bert-cnn_dailymail-fp16\")\n\n>>> _model.encoder.save_pretrained(\"./encoder\")\n>>> _model.decoder.save_pretrained(\"./decoder\")\n\n>>> model = TFEncoderDecoderModel.from_encoder_decoder_pretrained(\n...     \"./encoder\", \"./decoder\", encoder_from_pt=True, decoder_from_pt=True\n... )\n>>> # This is only for copying some specific attributes of this particular model.\n>>> model.config = _model.config\n```\n\n## Training\n\nOnce the model is created, it can be fine-tuned similar to BART, T5 or any other encoder-decoder model.\nAs you can see, only 2 inputs are required for the model in order to compute a loss: `input_ids` (which are the\n`input_ids` of the encoded input sequence) and `labels` (which are the `input_ids` of the encoded\ntarget sequence).\n\n```python\n>>> from transformers import BertTokenizer, EncoderDecoderModel\n\n>>> tokenizer = BertTokenizer.from_pretrained(\"google-bert/bert-base-uncased\")\n>>> model = EncoderDecoderModel.from_encoder_decoder_pretrained(\"google-bert/bert-base-uncased\", \"google-bert/bert-base-uncased\")\n\n>>> model.config.decoder_start_token_id = tokenizer.cls_token_id\n>>> model.config.pad_token_id = tokenizer.pad_token_id\n\n>>> input_ids = tokenizer(\n...     \"The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building, and the tallest structure in Paris."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: c7446985-eee0-44e1-a858-810dcd0d4bbc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/encoder-decoder", "title": "Encoder Decoder Models", "tokens": 2117, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c7446985-eee0-44e1-a858-810dcd0d4bbc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/encoder-decoder\", \"title\": \"Encoder Decoder Models\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b4ae63ab-a48f-4597-b98b-c9a88839c867\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/encoder-decoder\", \"title\": \"Encoder Decoder Models\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e2efdaa1bfb1a39545221ba209d082233a8c938ece0da1cbc4b100f3bb8b46f5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5dca5bc7-9145-44b0-988a-2b6e52c89bcf\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/encoder-decoder\", \"title\": \"Encoder Decoder Models\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2cdaca6044c3864aa4509ca7a978528f02a7dcfebff13f4cabb3a971f28a9f97\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4cd22145-cb05-409b-8b6b-bc71da986769\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c0f072ed16a5400a026af7494c78925cda08ed3c4904b3455cfc26033194c93d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4899, \"end_char_idx\": 8077, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b4ae63ab-a48f-4597-b98b-c9a88839c867", "doc_id": "b4ae63ab-a48f-4597-b98b-c9a88839c867", "ref_doc_id": "b4ae63ab-a48f-4597-b98b-c9a88839c867", "chroma:document": "Passing `from_pt=True` to this method will throw an exception. If there are only pytorch\ncheckpoints for a particular encoder-decoder model, a workaround is:\n\n```python\n>>> # a workaround to load from pytorch checkpoint\n>>> from transformers import EncoderDecoderModel, TFEncoderDecoderModel\n\n>>> _model = EncoderDecoderModel.from_pretrained(\"patrickvonplaten/bert2bert-cnn_dailymail-fp16\")\n\n>>> _model.encoder.save_pretrained(\"./encoder\")\n>>> _model.decoder.save_pretrained(\"./decoder\")\n\n>>> model = TFEncoderDecoderModel.from_encoder_decoder_pretrained(\n...     \"./encoder\", \"./decoder\", encoder_from_pt=True, decoder_from_pt=True\n... )\n>>> # This is only for copying some specific attributes of this particular model.\n>>> model.config = _model.config\n```\n\n## Training\n\nOnce the model is created, it can be fine-tuned similar to BART, T5 or any other encoder-decoder model.\nAs you can see, only 2 inputs are required for the model in order to compute a loss: `input_ids` (which are the\n`input_ids` of the encoded input sequence) and `labels` (which are the `input_ids` of the encoded\ntarget sequence).\n\n```python\n>>> from transformers import BertTokenizer, EncoderDecoderModel\n\n>>> tokenizer = BertTokenizer.from_pretrained(\"google-bert/bert-base-uncased\")\n>>> model = EncoderDecoderModel.from_encoder_decoder_pretrained(\"google-bert/bert-base-uncased\", \"google-bert/bert-base-uncased\")\n\n>>> model.config.decoder_start_token_id = tokenizer.cls_token_id\n>>> model.config.pad_token_id = tokenizer.pad_token_id\n\n>>> input_ids = tokenizer(\n...     \"The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building, and the tallest structure in Paris. Its base is square, measuring 125 metres (410 ft) on each side.During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest man-made structure in the world, a title it held for 41 years until the Chrysler Building in New York City was  finished in 1930. It was the first structure to reach a height of 300 metres. Due to the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the Chrysler Building by 5.2 metres (17 ft).Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France after the Millau Viaduct.\",\n...     return_tensors=\"pt\",\n... ).input_ids\n\n>>> labels = tokenizer(\n...     \"the eiffel tower surpassed the washington monument to become the tallest structure in the world. it was the first structure to reach a height of 300 metres in paris in 1930. it is now taller than the chrysler building by 5. 2 metres ( 17 ft ) and is the second tallest free - standing structure in paris.\",\n...     return_tensors=\"pt\",\n... ).input_ids\n\n>>> # the forward function automatically creates the correct decoder_input_ids\n>>> loss = model(input_ids=input_ids, labels=labels).loss\n```\n\nDetailed [colab](https://colab.research.google.com/drive/1WIk2bxglElfZewOHboPFNj8H44_VAyKE?usp=sharing#scrollTo=ZwQIEhKOrJpl) for training.\n\nThis model was contributed by [thomwolf](https://github.com/thomwolf). This model's TensorFlow and Flax versions\nwere contributed by [ydshieh](https://github.com/ydshieh)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 4cd22145-cb05-409b-8b6b-bc71da986769
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/encoder-decoder", "title": "Encoder Decoder Models", "tokens": 2117, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4cd22145-cb05-409b-8b6b-bc71da986769\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/encoder-decoder\", \"title\": \"Encoder Decoder Models\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b4ae63ab-a48f-4597-b98b-c9a88839c867\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/encoder-decoder\", \"title\": \"Encoder Decoder Models\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e2efdaa1bfb1a39545221ba209d082233a8c938ece0da1cbc4b100f3bb8b46f5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c7446985-eee0-44e1-a858-810dcd0d4bbc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/encoder-decoder\", \"title\": \"Encoder Decoder Models\", \"tokens\": 2117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b9a801f70aefeed36f908428af1ea393cf905cd2115111554ae05922f37a5509\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6571, \"end_char_idx\": 8552, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b4ae63ab-a48f-4597-b98b-c9a88839c867", "doc_id": "b4ae63ab-a48f-4597-b98b-c9a88839c867", "ref_doc_id": "b4ae63ab-a48f-4597-b98b-c9a88839c867", "chroma:document": "Its base is square, measuring 125 metres (410 ft) on each side.During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest man-made structure in the world, a title it held for 41 years until the Chrysler Building in New York City was  finished in 1930. It was the first structure to reach a height of 300 metres. Due to the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the Chrysler Building by 5.2 metres (17 ft).Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France after the Millau Viaduct.\",\n...     return_tensors=\"pt\",\n... ).input_ids\n\n>>> labels = tokenizer(\n...     \"the eiffel tower surpassed the washington monument to become the tallest structure in the world. it was the first structure to reach a height of 300 metres in paris in 1930. it is now taller than the chrysler building by 5. 2 metres ( 17 ft ) and is the second tallest free - standing structure in paris.\",\n...     return_tensors=\"pt\",\n... ).input_ids\n\n>>> # the forward function automatically creates the correct decoder_input_ids\n>>> loss = model(input_ids=input_ids, labels=labels).loss\n```\n\nDetailed [colab](https://colab.research.google.com/drive/1WIk2bxglElfZewOHboPFNj8H44_VAyKE?usp=sharing#scrollTo=ZwQIEhKOrJpl) for training.\n\nThis model was contributed by [thomwolf](https://github.com/thomwolf). This model's TensorFlow and Flax versions\nwere contributed by [ydshieh](https://github.com/ydshieh).\n\n\n## EncoderDecoderConfig\n\n[[autodoc]] EncoderDecoderConfig\n\n<frameworkcontent>\n<pt>\n\n## EncoderDecoderModel\n\n[[autodoc]] EncoderDecoderModel\n    - forward\n    - from_encoder_decoder_pretrained\n\n</pt>\n<tf>\n\n## TFEncoderDecoderModel\n\n[[autodoc]] TFEncoderDecoderModel\n    - call\n    - from_encoder_decoder_pretrained\n\n</tf>\n<jax>\n\n## FlaxEncoderDecoderModel\n\n[[autodoc]] FlaxEncoderDecoderModel\n    - __call__\n    - from_encoder_decoder_pretrained\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 0002762b-e504-4916-8ab6-e24e3facc959
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/siglip", "title": "SigLIP", "tokens": 2478, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0002762b-e504-4916-8ab6-e24e3facc959\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/siglip\", \"title\": \"SigLIP\", \"tokens\": 2478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7ddb0116-2896-4623-98a3-a45401f30619\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/siglip\", \"title\": \"SigLIP\", \"tokens\": 2478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c5e211219231a89121d9f8f1930d41d4def76c363b0aba344eb20d2975703e02\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c167a44a-6594-4427-95e9-0de313ce33b3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"25d2a0215c0df4b5d8ed28cb3447787b102f66870704c54698f3aba2359e5b68\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2731, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7ddb0116-2896-4623-98a3-a45401f30619", "doc_id": "7ddb0116-2896-4623-98a3-a45401f30619", "ref_doc_id": "7ddb0116-2896-4623-98a3-a45401f30619", "chroma:document": "# SigLIP\n\n## Overview\n\nThe SigLIP model was proposed in [Sigmoid Loss for Language Image Pre-Training](https://arxiv.org/abs/2303.15343) by Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, Lucas Beyer. SigLIP proposes to replace the loss function used in [CLIP](clip) by a simple pairwise sigmoid loss. This results in better performance in terms of zero-shot classification accuracy on ImageNet.\n\nThe abstract from the paper is the following:\n\n*We propose a simple pairwise Sigmoid loss for Language-Image Pre-training (SigLIP). Unlike standard contrastive learning with softmax normalization, the sigmoid loss operates solely on image-text pairs and does not require a global view of the pairwise similarities for normalization. The sigmoid loss simultaneously allows further scaling up the batch size, while also performing better at smaller batch sizes. Combined with Locked-image Tuning, with only four TPUv4 chips, we train a SigLiT model that achieves 84.5% ImageNet zero-shot accuracy in two days. The disentanglement of the batch size from the loss further allows us to study the impact of examples vs pairs and negative to positive ratio. Finally, we push the batch size to the extreme, up to one million, and find that the benefits of growing batch size quickly diminish, with a more reasonable batch size of 32k being sufficient.*\n\n## Usage tips\n\n- Usage of SigLIP is similar to [CLIP](clip). The main difference is the training loss, which does not require a global view of all the pairwise similarities of images and texts within a batch. One needs to apply the sigmoid activation function to the logits, rather than the softmax.\n- Training is supported but does not use `torch.distributed` utilities which may limit the scalability of batch size. However, DDP and FDSP works on single-node multi-gpu setup.\n- When using the standalone [`SiglipTokenizer`] or [`SiglipProcessor`], make sure to pass `padding=\"max_length\"` as that's how the model was trained.\n- To get the same results as the pipeline, a prompt template of \"This is a photo of {label}.\" should be used.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/siglip_table.jpeg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> SigLIP evaluation results compared to CLIP. Taken from the <a href=\"https://arxiv.org/abs/2303.15343\">original paper</a>.</small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/google-research/big_vision/tree/main).\n\n## Usage example\n\nThere are 2 main ways to use SigLIP: either using the pipeline API, which abstracts away all the complexity for you, or by using the `SiglipModel` class yourself."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: c167a44a-6594-4427-95e9-0de313ce33b3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/siglip", "title": "SigLIP", "tokens": 2478, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c167a44a-6594-4427-95e9-0de313ce33b3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/siglip\", \"title\": \"SigLIP\", \"tokens\": 2478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7ddb0116-2896-4623-98a3-a45401f30619\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/siglip\", \"title\": \"SigLIP\", \"tokens\": 2478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c5e211219231a89121d9f8f1930d41d4def76c363b0aba344eb20d2975703e02\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0002762b-e504-4916-8ab6-e24e3facc959\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/siglip\", \"title\": \"SigLIP\", \"tokens\": 2478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b4ea411b6c4134b543ce30c25ad92ce9c44dc20963413fba7159a3b8c03e5513\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7dbea716-cd30-4586-bc83-6130a95adddb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c1dda6edc01caeedb43185c9921a9034699c21f9fc8a60bdcf49a7dbb5bd1311\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1151, \"end_char_idx\": 4239, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7ddb0116-2896-4623-98a3-a45401f30619", "doc_id": "7ddb0116-2896-4623-98a3-a45401f30619", "ref_doc_id": "7ddb0116-2896-4623-98a3-a45401f30619", "chroma:document": "Finally, we push the batch size to the extreme, up to one million, and find that the benefits of growing batch size quickly diminish, with a more reasonable batch size of 32k being sufficient.*\n\n## Usage tips\n\n- Usage of SigLIP is similar to [CLIP](clip). The main difference is the training loss, which does not require a global view of all the pairwise similarities of images and texts within a batch. One needs to apply the sigmoid activation function to the logits, rather than the softmax.\n- Training is supported but does not use `torch.distributed` utilities which may limit the scalability of batch size. However, DDP and FDSP works on single-node multi-gpu setup.\n- When using the standalone [`SiglipTokenizer`] or [`SiglipProcessor`], make sure to pass `padding=\"max_length\"` as that's how the model was trained.\n- To get the same results as the pipeline, a prompt template of \"This is a photo of {label}.\" should be used.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/siglip_table.jpeg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> SigLIP evaluation results compared to CLIP. Taken from the <a href=\"https://arxiv.org/abs/2303.15343\">original paper</a>.</small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/google-research/big_vision/tree/main).\n\n## Usage example\n\nThere are 2 main ways to use SigLIP: either using the pipeline API, which abstracts away all the complexity for you, or by using the `SiglipModel` class yourself.\n\n### Pipeline API\n\nThe pipeline allows to use the model in a few lines of code:\n\n```python\n>>> from transformers import pipeline\n>>> from PIL import Image\n>>> import requests\n\n>>> # load pipe\n>>> image_classifier = pipeline(task=\"zero-shot-image-classification\", model=\"google/siglip-base-patch16-224\")\n\n>>> # load image\n>>> url = 'http://images.cocodataset.org/val2017/000000039769.jpg'\n>>> image = Image.open(requests.get(url, stream=True).raw)\n\n>>> # inference\n>>> candidate_labels = [\"2 cats\", \"a plane\", \"a remote\"]\n>>> outputs = image_classifier(image, candidate_labels=candidate_labels)\n>>> outputs = [{\"score\": round(output[\"score\"], 4), \"label\": output[\"label\"] } for output in outputs]\n>>> print(outputs)\n[{'score': 0.1979, 'label': '2 cats'}, {'score': 0.0, 'label': 'a remote'}, {'score': 0.0, 'label': 'a plane'}]\n```\n\n### Using the model yourself\n\nIf you want to do the pre- and postprocessing yourself, here's how to do that:\n\n```python\n>>> from PIL import Image\n>>> import requests\n>>> from transformers import AutoProcessor, AutoModel\n>>> import torch\n\n>>> model = AutoModel.from_pretrained(\"google/siglip-base-patch16-224\")\n>>> processor = AutoProcessor.from_pretrained(\"google/siglip-base-patch16-224\")\n\n>>> url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n\n>>> candidate_labels = [\"2 cats\", \"2 dogs\"]\n# follows the pipeline prompt template to get same results\n>>> candidate_labels = [f'This is a photo of {label}.'"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 7dbea716-cd30-4586-bc83-6130a95adddb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/siglip", "title": "SigLIP", "tokens": 2478, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7dbea716-cd30-4586-bc83-6130a95adddb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/siglip\", \"title\": \"SigLIP\", \"tokens\": 2478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7ddb0116-2896-4623-98a3-a45401f30619\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/siglip\", \"title\": \"SigLIP\", \"tokens\": 2478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c5e211219231a89121d9f8f1930d41d4def76c363b0aba344eb20d2975703e02\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c167a44a-6594-4427-95e9-0de313ce33b3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/siglip\", \"title\": \"SigLIP\", \"tokens\": 2478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a479b257c24371ddf1d442cedaddf6ee6bed2acf1cf9a3a86e11992be5c6d4a5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1bfc0689-83ba-483e-93dd-f63010697709\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f0ff1f7a0d2c2b1df80b7422161ad67e3a91f0a1ed985764b0a01aed5b780737\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2733, \"end_char_idx\": 5305, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7ddb0116-2896-4623-98a3-a45401f30619", "doc_id": "7ddb0116-2896-4623-98a3-a45401f30619", "ref_doc_id": "7ddb0116-2896-4623-98a3-a45401f30619", "chroma:document": "### Pipeline API\n\nThe pipeline allows to use the model in a few lines of code:\n\n```python\n>>> from transformers import pipeline\n>>> from PIL import Image\n>>> import requests\n\n>>> # load pipe\n>>> image_classifier = pipeline(task=\"zero-shot-image-classification\", model=\"google/siglip-base-patch16-224\")\n\n>>> # load image\n>>> url = 'http://images.cocodataset.org/val2017/000000039769.jpg'\n>>> image = Image.open(requests.get(url, stream=True).raw)\n\n>>> # inference\n>>> candidate_labels = [\"2 cats\", \"a plane\", \"a remote\"]\n>>> outputs = image_classifier(image, candidate_labels=candidate_labels)\n>>> outputs = [{\"score\": round(output[\"score\"], 4), \"label\": output[\"label\"] } for output in outputs]\n>>> print(outputs)\n[{'score': 0.1979, 'label': '2 cats'}, {'score': 0.0, 'label': 'a remote'}, {'score': 0.0, 'label': 'a plane'}]\n```\n\n### Using the model yourself\n\nIf you want to do the pre- and postprocessing yourself, here's how to do that:\n\n```python\n>>> from PIL import Image\n>>> import requests\n>>> from transformers import AutoProcessor, AutoModel\n>>> import torch\n\n>>> model = AutoModel.from_pretrained(\"google/siglip-base-patch16-224\")\n>>> processor = AutoProcessor.from_pretrained(\"google/siglip-base-patch16-224\")\n\n>>> url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n\n>>> candidate_labels = [\"2 cats\", \"2 dogs\"]\n# follows the pipeline prompt template to get same results\n>>> candidate_labels = [f'This is a photo of {label}.' for label in candidate_labels]\n>>> # important: we pass `padding=max_length` since the model was trained with this\n>>> inputs = processor(text=texts, images=image, padding=\"max_length\", return_tensors=\"pt\")\n\n>>> with torch.no_grad():\n...     outputs = model(**inputs)\n\n>>> logits_per_image = outputs.logits_per_image\n>>> probs = torch.sigmoid(logits_per_image) # these are the probabilities\n>>> print(f\"{probs[0][0]:.1%} that image 0 is '{texts[0]}'\")\n31.9% that image 0 is 'a photo of 2 cats'\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with SigLIP.\n\n- [Zero-shot image classification task guide](../tasks/zero_shot_image_classification_md)\n- Demo notebooks for SigLIP can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/SigLIP). \ud83c\udf0e\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 1bfc0689-83ba-483e-93dd-f63010697709
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/siglip", "title": "SigLIP", "tokens": 2478, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1bfc0689-83ba-483e-93dd-f63010697709\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/siglip\", \"title\": \"SigLIP\", \"tokens\": 2478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7ddb0116-2896-4623-98a3-a45401f30619\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/siglip\", \"title\": \"SigLIP\", \"tokens\": 2478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c5e211219231a89121d9f8f1930d41d4def76c363b0aba344eb20d2975703e02\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7dbea716-cd30-4586-bc83-6130a95adddb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/siglip\", \"title\": \"SigLIP\", \"tokens\": 2478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0bb46e7113b19679fbda4e938414fb37e1f2deb469efd0e708e277cc962f6e80\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7ba80776-8184-4774-81c4-991f59aa2cfd\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3edbd5c5dd12dd084626cea7668416fc55fb582760d2106086ede213cd3aede1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4240, \"end_char_idx\": 7141, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7ddb0116-2896-4623-98a3-a45401f30619", "doc_id": "7ddb0116-2896-4623-98a3-a45401f30619", "ref_doc_id": "7ddb0116-2896-4623-98a3-a45401f30619", "chroma:document": "for label in candidate_labels]\n>>> # important: we pass `padding=max_length` since the model was trained with this\n>>> inputs = processor(text=texts, images=image, padding=\"max_length\", return_tensors=\"pt\")\n\n>>> with torch.no_grad():\n...     outputs = model(**inputs)\n\n>>> logits_per_image = outputs.logits_per_image\n>>> probs = torch.sigmoid(logits_per_image) # these are the probabilities\n>>> print(f\"{probs[0][0]:.1%} that image 0 is '{texts[0]}'\")\n31.9% that image 0 is 'a photo of 2 cats'\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with SigLIP.\n\n- [Zero-shot image classification task guide](../tasks/zero_shot_image_classification_md)\n- Demo notebooks for SigLIP can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/SigLIP). \ud83c\udf0e\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n\n## Combining SigLIP and Flash Attention 2\n\nFirst, make sure to install the latest version of Flash Attention 2.\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nMake also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of flash-attn repository. Make also sure to load your model in half-precision (e.g. `torch.float16``)\n\nTo load and run a model using Flash Attention 2, refer to the snippet below:\n\n```python\n>>> import torch\n>>> import requests\n>>> from PIL import Image\n>>> from transformers import SiglipProcessor, SiglipModel\n>>> device = \"cuda\" # the device to load the model onto\n\n>>> model = SiglipModel.from_pretrained(\n...     \"google/siglip-so400m-patch14-384\",\n...     attn_implementation=\"flash_attention_2\",\n...     torch_dtype=torch.float16,\n...     device_map=device,\n... )\n>>> processor = SiglipProcessor.from_pretrained(\"google/siglip-so400m-patch14-384\")\n\n>>> url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n\n>>> candidate_labels = [\"2 cats\", \"2 dogs\"]\n# follows the pipeline prompt template to get same results\n>>> candidate_labels = [f'This is a photo of {label}.' for label in candidate_labels]\n# important: we pass `padding=max_length` since the model was trained with this\n>>> inputs = processor(text=candidate_labels, images=image, padding=\"max_length\", return_tensors=\"pt\")\n>>> inputs.to(device)\n\n>>> with torch.no_grad():\n...     with torch.autocast(device):\n...         outputs = model(**inputs)\n\n>>> logits_per_image = outputs.logits_per_image\n>>> probs = torch.sigmoid(logits_per_image) # these are the probabilities\n>>> print(f\"{probs[0][0]:.1%} that image 0 is '{candidate_labels[0]}'\")\n51.3% that image 0 is 'This is a photo of 2 cats.'\n```"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 7ba80776-8184-4774-81c4-991f59aa2cfd
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/siglip", "title": "SigLIP", "tokens": 2478, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7ba80776-8184-4774-81c4-991f59aa2cfd\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/siglip\", \"title\": \"SigLIP\", \"tokens\": 2478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7ddb0116-2896-4623-98a3-a45401f30619\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/siglip\", \"title\": \"SigLIP\", \"tokens\": 2478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c5e211219231a89121d9f8f1930d41d4def76c363b0aba344eb20d2975703e02\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1bfc0689-83ba-483e-93dd-f63010697709\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/siglip\", \"title\": \"SigLIP\", \"tokens\": 2478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0d6885281b80b90be9fabf8b5bbac91e30556baa2e844a7710d01d669c18c5b8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7144, \"end_char_idx\": 9400, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7ddb0116-2896-4623-98a3-a45401f30619", "doc_id": "7ddb0116-2896-4623-98a3-a45401f30619", "ref_doc_id": "7ddb0116-2896-4623-98a3-a45401f30619", "chroma:document": "## Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nYou may set `attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used. Make sure you have `torch>=2.1.1`.\n\n```python\n>>> from transformers import SiglipModel\n\n>>> model = SiglipModel.from_pretrained(\n...     \"google/siglip-so400m-patch14-384\",\n...     attn_implementation=\"sdpa\",\n...     torch_dtype=torch.float16,\n...     device_map=device,\n... )\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\n\n## Expected speedups\n\nBelow is an expected speedup diagram that compares inference time between the native implementation in transformers using `google/siglip-so400m-patch14-384` checkpoint in `float16` precision and the Flash Attention 2 / SDPA version of the model using different batch sizes.\n\n<div style=\"text-align: center\">\n<img src=\"https://i.imgur.com/cWm4rsn.png\">\n</div>\n\n\n## SiglipConfig\n\n[[autodoc]] SiglipConfig\n    - from_text_vision_configs\n\n## SiglipTextConfig\n\n[[autodoc]] SiglipTextConfig\n\n## SiglipVisionConfig\n\n[[autodoc]] SiglipVisionConfig\n\n## SiglipTokenizer\n\n[[autodoc]] SiglipTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## SiglipImageProcessor\n\n[[autodoc]] SiglipImageProcessor\n    - preprocess\n\n## SiglipProcessor\n\n[[autodoc]] SiglipProcessor\n\n## SiglipModel\n\n[[autodoc]] SiglipModel\n    - forward\n    - get_text_features\n    - get_image_features\n\n## SiglipTextModel\n\n[[autodoc]] SiglipTextModel\n    - forward\n\n## SiglipVisionModel\n\n[[autodoc]] SiglipVisionModel\n    - forward\n\n\n## SiglipForImageClassification\n\n[[autodoc]] SiglipForImageClassification\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: ada98b73-62d3-4590-9a6b-b98391a6fd35
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/plbart", "title": "PLBart", "tokens": 1313, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ada98b73-62d3-4590-9a6b-b98391a6fd35\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/plbart\", \"title\": \"PLBart\", \"tokens\": 1313, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1dd24fc0-806d-4d7d-819b-8125d7b00c76\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/plbart\", \"title\": \"PLBart\", \"tokens\": 1313, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b2017e88fe0136f82e097e15171a99d7f9987bc6c4222552a59a4da36795544d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"486ec1cd-51dc-4ad2-b041-7457b393d95c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b248ac5d29fed31524e38895ab8ecff7c23812b40791f4125c134ad413f249cc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3357, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1dd24fc0-806d-4d7d-819b-8125d7b00c76", "doc_id": "1dd24fc0-806d-4d7d-819b-8125d7b00c76", "ref_doc_id": "1dd24fc0-806d-4d7d-819b-8125d7b00c76", "chroma:document": "# PLBart\n\n## Overview\n\nThe PLBART model was proposed in [Unified Pre-training for Program Understanding and Generation](https://arxiv.org/abs/2103.06333) by Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, Kai-Wei Chang.\nThis is a BART-like model which can be used to perform code-summarization, code-generation, and code-translation tasks. The pre-trained model `plbart-base` has been trained using multilingual denoising task\non Java, Python and English.\n\nAccording to the abstract\n\n*Code summarization and generation empower conversion between programming language (PL) and natural language (NL),\nwhile code translation avails the migration of legacy code from one PL to another. This paper introduces PLBART, \na sequence-to-sequence model capable of performing a broad spectrum of program and language understanding and generation tasks.\nPLBART is pre-trained on an extensive collection of Java and Python functions and associated NL text via denoising autoencoding.\nExperiments on code summarization in the English language, code generation, and code translation in seven programming languages\nshow that PLBART outperforms or rivals state-of-the-art models. Moreover, experiments on discriminative tasks, e.g., program\nrepair, clone detection, and vulnerable code detection, demonstrate PLBART's effectiveness in program understanding.\nFurthermore, analysis reveals that PLBART learns program syntax, style (e.g., identifier naming convention), logical flow\n(e.g., if block inside an else block is equivalent to else if block) that are crucial to program semantics and thus excels\neven with limited annotations.*\n\nThis model was contributed by [gchhablani](https://huggingface.co/gchhablani). The Authors' code can be found [here](https://github.com/wasiahmad/PLBART).\n\n## Usage examples\n\nPLBart is a multilingual encoder-decoder (sequence-to-sequence) model primarily intended for code-to-text, text-to-code, code-to-code tasks. As the\nmodel is multilingual it expects the sequences in a different format. A special language id token is added in both the\nsource and target text. The source text format is `X [eos, src_lang_code]` where `X` is the source text. The\ntarget text format is `[tgt_lang_code] X [eos]`. `bos` is never used.\n\nHowever, for fine-tuning, in some cases no language token is provided in cases where a single language is used. Please refer to [the paper](https://arxiv.org/abs/2103.06333) to learn more about this.\n\nIn cases where the language code is needed, the regular [`~PLBartTokenizer.__call__`] will encode source text format \nwhen you pass texts as the first argument or with the keyword argument `text`, and will encode target text format if\nit's passed with the `text_target` keyword argument.\n\n### Supervised training\n\n```python\n>>> from transformers import PLBartForConditionalGeneration, PLBartTokenizer\n\n>>> tokenizer = PLBartTokenizer.from_pretrained(\"uclanlp/plbart-base\", src_lang=\"en_XX\", tgt_lang=\"python\")\n>>> example_python_phrase = \"def maximum(a,b,c):NEW_LINE_INDENTreturn max([a,b,c])\"\n>>> expected_translation_english = \"Returns the maximum value of a b c.\"\n>>> inputs = tokenizer(example_python_phrase, text_target=expected_translation_english, return_tensors=\"pt\")\n>>> model(**inputs)\n```\n\n### Generation\n\n  While generating the target text set the `decoder_start_token_id` to the target language id."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 486ec1cd-51dc-4ad2-b041-7457b393d95c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/plbart", "title": "PLBart", "tokens": 1313, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"486ec1cd-51dc-4ad2-b041-7457b393d95c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/plbart\", \"title\": \"PLBart\", \"tokens\": 1313, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1dd24fc0-806d-4d7d-819b-8125d7b00c76\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/plbart\", \"title\": \"PLBart\", \"tokens\": 1313, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b2017e88fe0136f82e097e15171a99d7f9987bc6c4222552a59a4da36795544d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ada98b73-62d3-4590-9a6b-b98391a6fd35\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/plbart\", \"title\": \"PLBart\", \"tokens\": 1313, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"60c88eaaf2404eb53eb93210f3cc90abc5851d84628bb88d7fbe71116090800c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1779, \"end_char_idx\": 4835, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1dd24fc0-806d-4d7d-819b-8125d7b00c76", "doc_id": "1dd24fc0-806d-4d7d-819b-8125d7b00c76", "ref_doc_id": "1dd24fc0-806d-4d7d-819b-8125d7b00c76", "chroma:document": "## Usage examples\n\nPLBart is a multilingual encoder-decoder (sequence-to-sequence) model primarily intended for code-to-text, text-to-code, code-to-code tasks. As the\nmodel is multilingual it expects the sequences in a different format. A special language id token is added in both the\nsource and target text. The source text format is `X [eos, src_lang_code]` where `X` is the source text. The\ntarget text format is `[tgt_lang_code] X [eos]`. `bos` is never used.\n\nHowever, for fine-tuning, in some cases no language token is provided in cases where a single language is used. Please refer to [the paper](https://arxiv.org/abs/2103.06333) to learn more about this.\n\nIn cases where the language code is needed, the regular [`~PLBartTokenizer.__call__`] will encode source text format \nwhen you pass texts as the first argument or with the keyword argument `text`, and will encode target text format if\nit's passed with the `text_target` keyword argument.\n\n### Supervised training\n\n```python\n>>> from transformers import PLBartForConditionalGeneration, PLBartTokenizer\n\n>>> tokenizer = PLBartTokenizer.from_pretrained(\"uclanlp/plbart-base\", src_lang=\"en_XX\", tgt_lang=\"python\")\n>>> example_python_phrase = \"def maximum(a,b,c):NEW_LINE_INDENTreturn max([a,b,c])\"\n>>> expected_translation_english = \"Returns the maximum value of a b c.\"\n>>> inputs = tokenizer(example_python_phrase, text_target=expected_translation_english, return_tensors=\"pt\")\n>>> model(**inputs)\n```\n\n### Generation\n\n  While generating the target text set the `decoder_start_token_id` to the target language id. The following\n  example shows how to translate Python to English using the `uclanlp/plbart-python-en_XX` model.\n\n```python\n>>> from transformers import PLBartForConditionalGeneration, PLBartTokenizer\n\n>>> tokenizer = PLBartTokenizer.from_pretrained(\"uclanlp/plbart-python-en_XX\", src_lang=\"python\", tgt_lang=\"en_XX\")\n>>> example_python_phrase = \"def maximum(a,b,c):NEW_LINE_INDENTreturn max([a,b,c])\"\n>>> inputs = tokenizer(example_python_phrase, return_tensors=\"pt\")\n>>> model = PLBartForConditionalGeneration.from_pretrained(\"uclanlp/plbart-python-en_XX\")\n>>> translated_tokens = model.generate(**inputs, decoder_start_token_id=tokenizer.lang_code_to_id[\"en_XX\"])\n>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]\n\"Returns the maximum value of a b c.\"\n```\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n## PLBartConfig\n\n[[autodoc]] PLBartConfig\n\n## PLBartTokenizer\n\n[[autodoc]] PLBartTokenizer\n    - build_inputs_with_special_tokens\n\n## PLBartModel\n\n[[autodoc]] PLBartModel\n    - forward\n\n## PLBartForConditionalGeneration\n\n[[autodoc]] PLBartForConditionalGeneration\n    - forward\n\n## PLBartForSequenceClassification\n\n[[autodoc]] PLBartForSequenceClassification\n    - forward\n\n## PLBartForCausalLM\n\n[[autodoc]] PLBartForCausalLM\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 95723ae9-6ffc-4c34-94f3-7f4973516e05
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"95723ae9-6ffc-4c34-94f3-7f4973516e05\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"37f4469d-5ad4-4e1b-aad3-32a73a3e59f1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f68922bd49394cfb3f1e7866e006558355b27bedf0eb11e95c4a9d3a9e21ab15\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3290, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "doc_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "ref_doc_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "chroma:document": "# T5\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=t5\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-t5-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/t5-base\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n<a href=\"https://huggingface.co/papers/1910.10683\">\n<img alt=\"Paper page\" src=\"https://img.shields.io/badge/Paper%20page-1910.10683-green\">\n</a>\n</div>\n\n## Overview\n\nThe T5 model was presented in [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/pdf/1910.10683.pdf) by [Colin Raffel](https://huggingface.co/craffel), Noam Shazeer, [Adam Roberts](https://huggingface.co/adarob), Katherine Lee, Sharan Narang,\nMichael Matena, Yanqi Zhou, Wei Li, [Peter J. Liu](https://huggingface.co/peterjliu).\n\nThe abstract from the paper is the following:\n\n*Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream\ntask, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning\nhas given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of\ntransfer learning techniques for NLP by introducing a unified framework that converts every language problem into a\ntext-to-text format. Our systematic study compares pretraining objectives, architectures, unlabeled datasets, transfer\napproaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration\nwith scale and our new \"Colossal Clean Crawled Corpus\", we achieve state-of-the-art results on many benchmarks covering\nsummarization, question answering, text classification, and more. To facilitate future work on transfer learning for\nNLP, we release our dataset, pre-trained models, and code.*\n\nAll checkpoints can be found on the [hub](https://huggingface.co/models?search=t5).\n\nThis model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/google-research/text-to-text-transfer-transformer).\n\n## Usage tips\n\n- T5 is an encoder-decoder model pre-trained on a multi-task mixture of unsupervised and supervised tasks and for which\neach task is converted into a text-to-text format. T5 works well on a variety of tasks out-of-the-box by prepending a\ndifferent prefix to the input corresponding to each task, e.g., for translation: *translate English to German: ...*,\nfor summarization: *summarize: ...*.\n- The pretraining includes both supervised and self-supervised training. Supervised training is conducted on downstream tasks provided by the GLUE and SuperGLUE benchmarks (converting them into text-to-text tasks as explained above).\n- Self-supervised training uses corrupted tokens, by randomly removing 15% of the tokens and replacing them with individual sentinel tokens (if several consecutive tokens are marked for removal, the whole group is replaced with a single sentinel token). The input of the encoder is the corrupted sentence, the input of the decoder is the original sentence and the target is then the dropped out tokens delimited by their sentinel tokens."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 37f4469d-5ad4-4e1b-aad3-32a73a3e59f1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"37f4469d-5ad4-4e1b-aad3-32a73a3e59f1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"95723ae9-6ffc-4c34-94f3-7f4973516e05\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9fbdcb72acc43d1bec68a88d3dda44b8cc2548e81b864558560ac3ba23399ab0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7feaa001-116a-4a79-8823-8076370fef08\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"52522f4920ba72cef3a57155961d36f56a76c52177fd6d68f05ba5a22a66927d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1426, \"end_char_idx\": 4545, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "doc_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "ref_doc_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "chroma:document": "Our systematic study compares pretraining objectives, architectures, unlabeled datasets, transfer\napproaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration\nwith scale and our new \"Colossal Clean Crawled Corpus\", we achieve state-of-the-art results on many benchmarks covering\nsummarization, question answering, text classification, and more. To facilitate future work on transfer learning for\nNLP, we release our dataset, pre-trained models, and code.*\n\nAll checkpoints can be found on the [hub](https://huggingface.co/models?search=t5).\n\nThis model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/google-research/text-to-text-transfer-transformer).\n\n## Usage tips\n\n- T5 is an encoder-decoder model pre-trained on a multi-task mixture of unsupervised and supervised tasks and for which\neach task is converted into a text-to-text format. T5 works well on a variety of tasks out-of-the-box by prepending a\ndifferent prefix to the input corresponding to each task, e.g., for translation: *translate English to German: ...*,\nfor summarization: *summarize: ...*.\n- The pretraining includes both supervised and self-supervised training. Supervised training is conducted on downstream tasks provided by the GLUE and SuperGLUE benchmarks (converting them into text-to-text tasks as explained above).\n- Self-supervised training uses corrupted tokens, by randomly removing 15% of the tokens and replacing them with individual sentinel tokens (if several consecutive tokens are marked for removal, the whole group is replaced with a single sentinel token). The input of the encoder is the corrupted sentence, the input of the decoder is the original sentence and the target is then the dropped out tokens delimited by their sentinel tokens.\n\n- T5 uses relative scalar embeddings. Encoder input padding can be done on the left and on the right.\n\n- See the [training](#training), [inference](#inference) and [resources](#resources) sections below for all details regarding usage.\n\nT5 comes in different sizes:\n\n- [google-t5/t5-small](https://huggingface.co/google-t5/t5-small)\n\n- [google-t5/t5-base](https://huggingface.co/google-t5/t5-base)\n\n- [google-t5/t5-large](https://huggingface.co/google-t5/t5-large)\n\n- [google-t5/t5-3b](https://huggingface.co/google-t5/t5-3b)\n\n- [google-t5/t5-11b](https://huggingface.co/google-t5/t5-11b).\n\nBased on the original T5 model, Google has released some follow-up works:\n\n- **T5v1.1**: T5v1.1 is an improved version of T5 with some architectural tweaks, and is pre-trained on C4 only without\n  mixing in the supervised tasks. Refer to the documentation of T5v1.1 which can be found [here](t5v1.1).\n\n- **mT5**: mT5 is a multilingual T5 model. It is pre-trained on the mC4 corpus, which includes 101 languages. Refer to\n  the documentation of mT5 which can be found [here](mt5).\n\n- **byT5**: byT5 is a T5 model pre-trained on byte sequences rather than SentencePiece subword token sequences. Refer\n  to the documentation of byT5 which can be found [here](byt5)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 7feaa001-116a-4a79-8823-8076370fef08
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7feaa001-116a-4a79-8823-8076370fef08\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"37f4469d-5ad4-4e1b-aad3-32a73a3e59f1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3846021b12366a62c285cee3e21ea0d6b6fdcab1c820793363b6dc450b2df969\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e7537e5c-8266-4eb8-9635-8a2756a82ebd\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"54c4a5bfc4c3ccebb41f3ad9d8bb7771056809f593ba47b7258e46411c9bff66\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3292, \"end_char_idx\": 6094, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "doc_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "ref_doc_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "chroma:document": "- T5 uses relative scalar embeddings. Encoder input padding can be done on the left and on the right.\n\n- See the [training](#training), [inference](#inference) and [resources](#resources) sections below for all details regarding usage.\n\nT5 comes in different sizes:\n\n- [google-t5/t5-small](https://huggingface.co/google-t5/t5-small)\n\n- [google-t5/t5-base](https://huggingface.co/google-t5/t5-base)\n\n- [google-t5/t5-large](https://huggingface.co/google-t5/t5-large)\n\n- [google-t5/t5-3b](https://huggingface.co/google-t5/t5-3b)\n\n- [google-t5/t5-11b](https://huggingface.co/google-t5/t5-11b).\n\nBased on the original T5 model, Google has released some follow-up works:\n\n- **T5v1.1**: T5v1.1 is an improved version of T5 with some architectural tweaks, and is pre-trained on C4 only without\n  mixing in the supervised tasks. Refer to the documentation of T5v1.1 which can be found [here](t5v1.1).\n\n- **mT5**: mT5 is a multilingual T5 model. It is pre-trained on the mC4 corpus, which includes 101 languages. Refer to\n  the documentation of mT5 which can be found [here](mt5).\n\n- **byT5**: byT5 is a T5 model pre-trained on byte sequences rather than SentencePiece subword token sequences. Refer\n  to the documentation of byT5 which can be found [here](byt5).\n\n- **UL2**: UL2 is a T5 like model pretrained on various denoising objectives\n\n- **Flan-T5**: Flan is a pretraining methods that is based on prompting. The Flan-T5 are T5 models trained on the Flan collection of \n    datasets which include: `taskmaster2`, `djaym7/wiki_dialog`, `deepmind/code_contests`, `lambada`, `gsm8k`, `aqua_rat`, `esnli`, `quasc` and `qed`.\n\n- **FLan-UL2** : the UL2 model finetuned using the \"Flan\" prompt tuning and dataset collection.\n\n- **UMT5**: UmT5 is a multilingual T5 model trained on an improved and refreshed mC4 multilingual corpus,  29 trillion characters across 107 language, using a new sampling method, UniMax. Refer to\n the documentation of mT5 which can be found [here](umt5).\n\n## Training\n\nT5 is an encoder-decoder model and converts all NLP problems into a text-to-text format. It is trained using teacher\nforcing. This means that for training, we always need an input sequence and a corresponding target sequence. The input\nsequence is fed to the model using `input_ids`. The target sequence is shifted to the right, i.e., prepended by a\nstart-sequence token and fed to the decoder using the `decoder_input_ids`. In teacher-forcing style, the target\nsequence is then appended by the EOS token and corresponds to the `labels`. The PAD token is hereby used as the\nstart-sequence token. T5 can be trained / fine-tuned both in a supervised and unsupervised fashion.\n\nOne can use [`T5ForConditionalGeneration`] (or the Tensorflow/Flax variant), which includes the\nlanguage modeling head on top of the decoder."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: e7537e5c-8266-4eb8-9635-8a2756a82ebd
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e7537e5c-8266-4eb8-9635-8a2756a82ebd\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7feaa001-116a-4a79-8823-8076370fef08\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"21381c074555690cf11e8a1785dbc8de89fdcc2b4abe0820584b64db48a607d7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"333d4695-e967-4396-9e67-a2d2666d8324\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bb633e13c4f3b9c1b17eecb5489dcd6ff77b3c3b9b0d9403e8a375645edbd533\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4698, \"end_char_idx\": 7660, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "doc_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "ref_doc_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "chroma:document": "The Flan-T5 are T5 models trained on the Flan collection of \n    datasets which include: `taskmaster2`, `djaym7/wiki_dialog`, `deepmind/code_contests`, `lambada`, `gsm8k`, `aqua_rat`, `esnli`, `quasc` and `qed`.\n\n- **FLan-UL2** : the UL2 model finetuned using the \"Flan\" prompt tuning and dataset collection.\n\n- **UMT5**: UmT5 is a multilingual T5 model trained on an improved and refreshed mC4 multilingual corpus,  29 trillion characters across 107 language, using a new sampling method, UniMax. Refer to\n the documentation of mT5 which can be found [here](umt5).\n\n## Training\n\nT5 is an encoder-decoder model and converts all NLP problems into a text-to-text format. It is trained using teacher\nforcing. This means that for training, we always need an input sequence and a corresponding target sequence. The input\nsequence is fed to the model using `input_ids`. The target sequence is shifted to the right, i.e., prepended by a\nstart-sequence token and fed to the decoder using the `decoder_input_ids`. In teacher-forcing style, the target\nsequence is then appended by the EOS token and corresponds to the `labels`. The PAD token is hereby used as the\nstart-sequence token. T5 can be trained / fine-tuned both in a supervised and unsupervised fashion.\n\nOne can use [`T5ForConditionalGeneration`] (or the Tensorflow/Flax variant), which includes the\nlanguage modeling head on top of the decoder.\n\n- Unsupervised denoising training\n\nIn this setup, spans of the input sequence are masked by so-called sentinel tokens (*a.k.a* unique mask tokens) and\nthe output sequence is formed as a concatenation of the same sentinel tokens and the *real* masked tokens. Each\nsentinel token represents a unique mask token for this sentence and should start with `<extra_id_0>`,\n`<extra_id_1>`, ... up to `<extra_id_99>`. As a default, 100 sentinel tokens are available in\n[`T5Tokenizer`].\n\nFor instance, the sentence \"The cute dog walks in the park\" with the masks put on \"cute dog\" and \"the\" should be\nprocessed as follows:\n\n```python\n>>> from transformers import T5Tokenizer, T5ForConditionalGeneration\n\n>>> tokenizer = T5Tokenizer.from_pretrained(\"google-t5/t5-small\")\n>>> model = T5ForConditionalGeneration.from_pretrained(\"google-t5/t5-small\")\n\n>>> input_ids = tokenizer(\"The <extra_id_0> walks in <extra_id_1> park\", return_tensors=\"pt\").input_ids\n>>> labels = tokenizer(\"<extra_id_0> cute dog <extra_id_1> the <extra_id_2>\", return_tensors=\"pt\").input_ids\n\n>>> # the forward function automatically creates the correct decoder_input_ids\n>>> loss = model(input_ids=input_ids, labels=labels).loss\n>>> loss.item()\n3.7837\n```\n\nIf you're interested in pre-training T5 on a new corpus, check out the [run_t5_mlm_flax.py](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling) script in the Examples\ndirectory.\n\n- Supervised training\n\nIn this setup, the input sequence and output sequence are a standard sequence-to-sequence input-output mapping."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 333d4695-e967-4396-9e67-a2d2666d8324
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"333d4695-e967-4396-9e67-a2d2666d8324\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e7537e5c-8266-4eb8-9635-8a2756a82ebd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ce295dc9dfd08ef08f85f5cdfa8b992c81c4f730ca9cf33d55625b28cf3fa0ad\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"928ac05f-9ac4-4323-a3cb-848b8c527d54\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a8a16971c0d7f35c23d0d23f4ec6274421fe3c2d777b62ef0e1559313e9a40b5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6096, \"end_char_idx\": 9299, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "doc_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "ref_doc_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "chroma:document": "- Unsupervised denoising training\n\nIn this setup, spans of the input sequence are masked by so-called sentinel tokens (*a.k.a* unique mask tokens) and\nthe output sequence is formed as a concatenation of the same sentinel tokens and the *real* masked tokens. Each\nsentinel token represents a unique mask token for this sentence and should start with `<extra_id_0>`,\n`<extra_id_1>`, ... up to `<extra_id_99>`. As a default, 100 sentinel tokens are available in\n[`T5Tokenizer`].\n\nFor instance, the sentence \"The cute dog walks in the park\" with the masks put on \"cute dog\" and \"the\" should be\nprocessed as follows:\n\n```python\n>>> from transformers import T5Tokenizer, T5ForConditionalGeneration\n\n>>> tokenizer = T5Tokenizer.from_pretrained(\"google-t5/t5-small\")\n>>> model = T5ForConditionalGeneration.from_pretrained(\"google-t5/t5-small\")\n\n>>> input_ids = tokenizer(\"The <extra_id_0> walks in <extra_id_1> park\", return_tensors=\"pt\").input_ids\n>>> labels = tokenizer(\"<extra_id_0> cute dog <extra_id_1> the <extra_id_2>\", return_tensors=\"pt\").input_ids\n\n>>> # the forward function automatically creates the correct decoder_input_ids\n>>> loss = model(input_ids=input_ids, labels=labels).loss\n>>> loss.item()\n3.7837\n```\n\nIf you're interested in pre-training T5 on a new corpus, check out the [run_t5_mlm_flax.py](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling) script in the Examples\ndirectory.\n\n- Supervised training\n\nIn this setup, the input sequence and output sequence are a standard sequence-to-sequence input-output mapping.\nSuppose that we want to fine-tune the model for translation for example, and we have a training example: the input\nsequence \"The house is wonderful.\" and output sequence \"Das Haus ist wunderbar.\", then they should be prepared for\nthe model as follows:\n\n```python\n>>> from transformers import T5Tokenizer, T5ForConditionalGeneration\n\n>>> tokenizer = T5Tokenizer.from_pretrained(\"google-t5/t5-small\")\n>>> model = T5ForConditionalGeneration.from_pretrained(\"google-t5/t5-small\")\n\n>>> input_ids = tokenizer(\"translate English to German: The house is wonderful.\", return_tensors=\"pt\").input_ids\n>>> labels = tokenizer(\"Das Haus ist wunderbar.\", return_tensors=\"pt\").input_ids\n\n>>> # the forward function automatically creates the correct decoder_input_ids\n>>> loss = model(input_ids=input_ids, labels=labels).loss\n>>> loss.item()\n0.2542\n```\n\nAs you can see, only 2 inputs are required for the model in order to compute a loss: `input_ids` (which are the\n`input_ids` of the encoded input sequence) and `labels` (which are the `input_ids` of the encoded\ntarget sequence). The model will automatically create the `decoder_input_ids` based on the `labels`, by\nshifting them one position to the right and prepending the `config.decoder_start_token_id`, which for T5 is\nequal to 0 (i.e. the id of the pad token). Also note the task prefix: we prepend the input sequence with 'translate\nEnglish to German: ' before encoding it. This will help in improving the performance, as this task prefix was used\nduring T5's pre-training.\n\nHowever, the example above only shows a single training example. In practice, one trains deep learning models in\nbatches."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 928ac05f-9ac4-4323-a3cb-848b8c527d54
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"928ac05f-9ac4-4323-a3cb-848b8c527d54\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"333d4695-e967-4396-9e67-a2d2666d8324\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"15fab9815d1ab9f894595f1ba42cc0f038638e3bba37c025c34942a1c9c60f89\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"40ecfbe9-4048-4ca8-9cd9-56717449db13\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e6b69ae207503c9dc0da54c984ced6591351883a4949717d29a4acfeb09bea86\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7661, \"end_char_idx\": 10278, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "doc_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "ref_doc_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "chroma:document": "Suppose that we want to fine-tune the model for translation for example, and we have a training example: the input\nsequence \"The house is wonderful.\" and output sequence \"Das Haus ist wunderbar.\", then they should be prepared for\nthe model as follows:\n\n```python\n>>> from transformers import T5Tokenizer, T5ForConditionalGeneration\n\n>>> tokenizer = T5Tokenizer.from_pretrained(\"google-t5/t5-small\")\n>>> model = T5ForConditionalGeneration.from_pretrained(\"google-t5/t5-small\")\n\n>>> input_ids = tokenizer(\"translate English to German: The house is wonderful.\", return_tensors=\"pt\").input_ids\n>>> labels = tokenizer(\"Das Haus ist wunderbar.\", return_tensors=\"pt\").input_ids\n\n>>> # the forward function automatically creates the correct decoder_input_ids\n>>> loss = model(input_ids=input_ids, labels=labels).loss\n>>> loss.item()\n0.2542\n```\n\nAs you can see, only 2 inputs are required for the model in order to compute a loss: `input_ids` (which are the\n`input_ids` of the encoded input sequence) and `labels` (which are the `input_ids` of the encoded\ntarget sequence). The model will automatically create the `decoder_input_ids` based on the `labels`, by\nshifting them one position to the right and prepending the `config.decoder_start_token_id`, which for T5 is\nequal to 0 (i.e. the id of the pad token). Also note the task prefix: we prepend the input sequence with 'translate\nEnglish to German: ' before encoding it. This will help in improving the performance, as this task prefix was used\nduring T5's pre-training.\n\nHowever, the example above only shows a single training example. In practice, one trains deep learning models in\nbatches. This entails that we must pad/truncate examples to the same length. For encoder-decoder models, one\ntypically defines a `max_source_length` and `max_target_length`, which determine the maximum length of the\ninput and output sequences respectively (otherwise they are truncated). These should be carefully set depending on\nthe task.\n\nIn addition, we must make sure that padding token id's of the `labels` are not taken into account by the loss\nfunction. In PyTorch and Tensorflow, this can be done by replacing them with -100, which is the `ignore_index`\nof the `CrossEntropyLoss`. In Flax, one can use the `decoder_attention_mask` to ignore padded tokens from\nthe loss (see the [Flax summarization script](https://github.com/huggingface/transformers/tree/main/examples/flax/summarization) for details). We also pass\n`attention_mask` as additional input to the model, which makes sure that padding tokens of the inputs are\nignored. The code example below illustrates all of this."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 40ecfbe9-4048-4ca8-9cd9-56717449db13
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"40ecfbe9-4048-4ca8-9cd9-56717449db13\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"928ac05f-9ac4-4323-a3cb-848b8c527d54\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a9c0fcf18b2caac0ea1892e5c9870c6e88e5d912ca4b0f51e3211149bc171e71\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e0d502f9-df83-4e8e-af06-b98d3280df19\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bd35e6532e00f9a90d7557771087ff9ccfef3215765ffad0a31f4965c238966d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8726, \"end_char_idx\": 12002, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "doc_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "ref_doc_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "chroma:document": "The model will automatically create the `decoder_input_ids` based on the `labels`, by\nshifting them one position to the right and prepending the `config.decoder_start_token_id`, which for T5 is\nequal to 0 (i.e. the id of the pad token). Also note the task prefix: we prepend the input sequence with 'translate\nEnglish to German: ' before encoding it. This will help in improving the performance, as this task prefix was used\nduring T5's pre-training.\n\nHowever, the example above only shows a single training example. In practice, one trains deep learning models in\nbatches. This entails that we must pad/truncate examples to the same length. For encoder-decoder models, one\ntypically defines a `max_source_length` and `max_target_length`, which determine the maximum length of the\ninput and output sequences respectively (otherwise they are truncated). These should be carefully set depending on\nthe task.\n\nIn addition, we must make sure that padding token id's of the `labels` are not taken into account by the loss\nfunction. In PyTorch and Tensorflow, this can be done by replacing them with -100, which is the `ignore_index`\nof the `CrossEntropyLoss`. In Flax, one can use the `decoder_attention_mask` to ignore padded tokens from\nthe loss (see the [Flax summarization script](https://github.com/huggingface/transformers/tree/main/examples/flax/summarization) for details). We also pass\n`attention_mask` as additional input to the model, which makes sure that padding tokens of the inputs are\nignored. The code example below illustrates all of this.\n\n```python\n>>> from transformers import T5Tokenizer, T5ForConditionalGeneration\n>>> import torch\n\n>>> tokenizer = T5Tokenizer.from_pretrained(\"google-t5/t5-small\")\n>>> model = T5ForConditionalGeneration.from_pretrained(\"google-t5/t5-small\")\n\n>>> # the following 2 hyperparameters are task-specific\n>>> max_source_length = 512\n>>> max_target_length = 128\n\n>>> # Suppose we have the following 2 training examples:\n>>> input_sequence_1 = \"Welcome to NYC\"\n>>> output_sequence_1 = \"Bienvenue \u00e0 NYC\"\n\n>>> input_sequence_2 = \"HuggingFace is a company\"\n>>> output_sequence_2 = \"HuggingFace est une entreprise\"\n\n>>> # encode the inputs\n>>> task_prefix = \"translate English to French: \"\n>>> input_sequences = [input_sequence_1, input_sequence_2]\n\n>>> encoding = tokenizer(\n...     [task_prefix + sequence for sequence in input_sequences],\n...     padding=\"longest\",\n...     max_length=max_source_length,\n...     truncation=True,\n...     return_tensors=\"pt\",\n... )\n\n>>> input_ids, attention_mask = encoding.input_ids, encoding.attention_mask\n\n>>> # encode the targets\n>>> target_encoding = tokenizer(\n...     [output_sequence_1, output_sequence_2],\n...     padding=\"longest\",\n...     max_length=max_target_length,\n...     truncation=True,\n...     return_tensors=\"pt\",\n... )\n>>> labels = target_encoding.input_ids\n\n>>> # replace padding token id's of the labels by -100 so it's ignored by the loss\n>>> labels[labels == tokenizer.pad_token_id] = -100\n\n>>> # forward pass\n>>> loss = model(input_ids=input_ids, attention_mask=attention_mask, labels=labels).loss\n>>> loss.item()\n0.188\n```\n\nAdditional training tips:\n\n- T5 models need a slightly higher learning rate than the default one set in the `Trainer` when using the AdamW\noptimizer."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: e0d502f9-df83-4e8e-af06-b98d3280df19
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e0d502f9-df83-4e8e-af06-b98d3280df19\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"40ecfbe9-4048-4ca8-9cd9-56717449db13\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"318027b0eacb926bf790535c00ae848d0aacc9927eda43669d7d69f6b35244b6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9a1980fa-129c-4247-b2ef-8527aeb2e1c4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f8b8aca254aa0e29bab97502b7aa35174c9b86ccb4aff1c7038476cf1c460d4c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12003, \"end_char_idx\": 15403, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "doc_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "ref_doc_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "chroma:document": "Typically, 1e-4 and 3e-4 work well for most problems (classification, summarization, translation, question\nanswering, question generation). Note that T5 was pre-trained using the AdaFactor optimizer.\n\nAccording to [this forum post](https://discuss.huggingface.co/t/t5-finetuning-tips/684), task prefixes matter when\n(1) doing multi-task training (2) your task is similar or related to one of the supervised tasks used in T5's\npre-training mixture (see Appendix D of the [paper](https://arxiv.org/pdf/1910.10683.pdf) for the task prefixes\nused).\n\nIf training on TPU, it is recommended to pad all examples of the dataset to the same length or make use of\n*pad_to_multiple_of* to have a small number of predefined bucket sizes to fit all examples in. Dynamically padding\nbatches to the longest example is not recommended on TPU as it triggers a recompilation for every batch shape that is\nencountered during training thus significantly slowing down the training. only padding up to the longest example in a\nbatch) leads to very slow training on TPU.\n\n## Inference\n\nAt inference time, it is recommended to use [`~generation.GenerationMixin.generate`]. This\nmethod takes care of encoding the input and feeding the encoded hidden states via cross-attention layers to the decoder\nand auto-regressively generates the decoder output. Check out [this blog post](https://huggingface.co/blog/how-to-generate) to know all the details about generating text with Transformers.\nThere's also [this blog post](https://huggingface.co/blog/encoder-decoder#encoder-decoder) which explains how\ngeneration works in general in encoder-decoder models.\n\n```python\n>>> from transformers import T5Tokenizer, T5ForConditionalGeneration\n\n>>> tokenizer = T5Tokenizer.from_pretrained(\"google-t5/t5-small\")\n>>> model = T5ForConditionalGeneration.from_pretrained(\"google-t5/t5-small\")\n\n>>> input_ids = tokenizer(\"translate English to German: The house is wonderful.\", return_tensors=\"pt\").input_ids\n>>> outputs = model.generate(input_ids)\n>>> print(tokenizer.decode(outputs[0], skip_special_tokens=True))\nDas Haus ist wunderbar.\n```\n\nNote that T5 uses the `pad_token_id` as the `decoder_start_token_id`, so when doing generation without using\n[`~generation.GenerationMixin.generate`], make sure you start it with the `pad_token_id`.\n\nThe example above only shows a single example. You can also do batched inference, like so:\n\n```python\n>>> from transformers import T5Tokenizer, T5ForConditionalGeneration\n\n>>> tokenizer = T5Tokenizer.from_pretrained(\"google-t5/t5-small\")\n>>> model = T5ForConditionalGeneration.from_pretrained(\"google-t5/t5-small\")\n\n>>> task_prefix = \"translate English to German: \"\n>>> # use different length sentences to test batching\n>>> sentences = [\"The house is wonderful.\", \"I like to work in NYC.\"]\n\n>>> inputs = tokenizer([task_prefix + sentence for sentence in sentences], return_tensors=\"pt\", padding=True)\n\n>>> output_sequences = model.generate(\n...     input_ids=inputs[\"input_ids\"],\n...     attention_mask=inputs[\"attention_mask\"],\n...     do_sample=False,  # disable sampling to test if batching affects output\n... )\n\n>>> print(tokenizer.batch_decode(output_sequences, skip_special_tokens=True))\n['Das Haus ist wunderbar.', 'Ich arbeite gerne in NYC.']\n```\n\nBecause T5 has been trained with the span-mask denoising objective,\nit can be used to predict the sentinel (masked-out) tokens during inference."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 9a1980fa-129c-4247-b2ef-8527aeb2e1c4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9a1980fa-129c-4247-b2ef-8527aeb2e1c4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e0d502f9-df83-4e8e-af06-b98d3280df19\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a9bcb5e14a8a2f5ccbe9696883824a5e157e3a9f3eb16260c36b5d91bb9a51fb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1466b2f8-99cb-4e31-bb04-430990ccf71b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"aaadb9ca0ec2d5b4fa1e4c683e430ca143106d255ebdf553d31d09019a2b6a53\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13934, \"end_char_idx\": 17180, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "doc_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "ref_doc_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "chroma:document": "\", return_tensors=\"pt\").input_ids\n>>> outputs = model.generate(input_ids)\n>>> print(tokenizer.decode(outputs[0], skip_special_tokens=True))\nDas Haus ist wunderbar.\n```\n\nNote that T5 uses the `pad_token_id` as the `decoder_start_token_id`, so when doing generation without using\n[`~generation.GenerationMixin.generate`], make sure you start it with the `pad_token_id`.\n\nThe example above only shows a single example. You can also do batched inference, like so:\n\n```python\n>>> from transformers import T5Tokenizer, T5ForConditionalGeneration\n\n>>> tokenizer = T5Tokenizer.from_pretrained(\"google-t5/t5-small\")\n>>> model = T5ForConditionalGeneration.from_pretrained(\"google-t5/t5-small\")\n\n>>> task_prefix = \"translate English to German: \"\n>>> # use different length sentences to test batching\n>>> sentences = [\"The house is wonderful.\", \"I like to work in NYC.\"]\n\n>>> inputs = tokenizer([task_prefix + sentence for sentence in sentences], return_tensors=\"pt\", padding=True)\n\n>>> output_sequences = model.generate(\n...     input_ids=inputs[\"input_ids\"],\n...     attention_mask=inputs[\"attention_mask\"],\n...     do_sample=False,  # disable sampling to test if batching affects output\n... )\n\n>>> print(tokenizer.batch_decode(output_sequences, skip_special_tokens=True))\n['Das Haus ist wunderbar.', 'Ich arbeite gerne in NYC.']\n```\n\nBecause T5 has been trained with the span-mask denoising objective,\nit can be used to predict the sentinel (masked-out) tokens during inference.\nThe predicted tokens will then be placed between the sentinel tokens.\n\n```python\n>>> from transformers import T5Tokenizer, T5ForConditionalGeneration\n\n>>> tokenizer = T5Tokenizer.from_pretrained(\"google-t5/t5-small\")\n>>> model = T5ForConditionalGeneration.from_pretrained(\"google-t5/t5-small\")\n\n>>> input_ids = tokenizer(\"The <extra_id_0> walks in <extra_id_1> park\", return_tensors=\"pt\").input_ids\n\n>>> sequence_ids = model.generate(input_ids)\n>>> sequences = tokenizer.batch_decode(sequence_ids)\n>>> sequences\n['<pad> <extra_id_0> park offers <extra_id_1> the <extra_id_2> park.</s>']\n```\n\n## Performance\n\nIf you'd like a faster training and inference performance, install [NVIDIA APEX](https://github.com/NVIDIA/apex#quick-start) for NVIDIA GPUs, or [ROCm APEX](https://github.com/ROCmSoftwarePlatform/apex) for AMD GPUs and then the model will automatically use `apex.normalization.FusedRMSNorm` instead of `T5LayerNorm`. The former uses an optimized fused kernel which is several times faster than the latter.\n\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with T5. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- A notebook for how to [finetune T5 for classification and multiple choice](https://colab.research.google.com/github/patil-suraj/exploring-T5/blob/master/t5_fine_tuning.ipynb).\n- A notebook for how to [finetune T5 for sentiment span extraction](https://colab.research.google.com/github/enzoampil/t5-intro/blob/master/t5_qa_training_pytorch_span_extraction.ipynb)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 1466b2f8-99cb-4e31-bb04-430990ccf71b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1466b2f8-99cb-4e31-bb04-430990ccf71b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9a1980fa-129c-4247-b2ef-8527aeb2e1c4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e611367013c76f92267a4b6d9c51677bcf242c2fcd9f20bc7a549ecde5437e6f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d5af6e79-1dee-4cb8-b561-40e082524adb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e0846fb07b468242044b588834cf5575d8e8886a8a335e2b73ce5b974ccfd3cd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 16330, \"end_char_idx\": 19274, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "doc_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "ref_doc_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "chroma:document": "The former uses an optimized fused kernel which is several times faster than the latter.\n\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with T5. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- A notebook for how to [finetune T5 for classification and multiple choice](https://colab.research.google.com/github/patil-suraj/exploring-T5/blob/master/t5_fine_tuning.ipynb).\n- A notebook for how to [finetune T5 for sentiment span extraction](https://colab.research.google.com/github/enzoampil/t5-intro/blob/master/t5_qa_training_pytorch_span_extraction.ipynb). \ud83c\udf0e\n\n<PipelineTag pipeline=\"token-classification\"/>\n\n- A notebook for how to [finetune T5 for named entity recognition](https://colab.research.google.com/drive/1obr78FY_cBmWY5ODViCmzdY6O1KB65Vc?usp=sharing). \ud83c\udf0e\n\n<PipelineTag pipeline=\"text-generation\"/>\n\n- A notebook for [Finetuning CodeT5 for generating docstrings from Ruby code](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/T5/Fine_tune_CodeT5_for_generating_docstrings_from_Ruby_code.ipynb).\n\n<PipelineTag pipeline=\"summarization\"/>\n\n- A notebook to [Finetune T5-base-dutch to perform Dutch abstractive summarization on a TPU](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/T5/Fine_tuning_Dutch_T5_base_on_CNN_Daily_Mail_for_summarization_(on_TPU_using_HuggingFace_Accelerate).ipynb).\n- A notebook for how to [finetune T5 for summarization in PyTorch and track experiments with WandB](https://colab.research.google.com/github/abhimishra91/transformers-tutorials/blob/master/transformers_summarization_wandb.ipynb#scrollTo=OKRpFvYhBauC). \ud83c\udf0e\n- A blog post on [Distributed Training: Train BART/T5 for Summarization using \ud83e\udd17 Transformers and Amazon SageMaker](https://huggingface.co/blog/sagemaker-distributed-training-seq2seq).\n- [`T5ForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization.ipynb).\n- [`TFT5ForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/summarization) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization-tf.ipynb).\n- [`FlaxT5ForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/summarization).\n- [Summarization](https://huggingface.co/course/chapter7/5?fw=pt#summarization) chapter of the \ud83e\udd17 Hugging Face course."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: d5af6e79-1dee-4cb8-b561-40e082524adb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d5af6e79-1dee-4cb8-b561-40e082524adb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1466b2f8-99cb-4e31-bb04-430990ccf71b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b7af59cb6076a6d4fad1d83f0bd6b43e14266390bba22b174d67d68287a5408e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"99c73119-902b-4654-9e10-daf0d23491f4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7aae1798735c66dba75a40823f3a1a053805c1176ae65f33a962872f835a67b5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 18000, \"end_char_idx\": 20960, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "doc_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "ref_doc_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "chroma:document": "- A notebook for how to [finetune T5 for summarization in PyTorch and track experiments with WandB](https://colab.research.google.com/github/abhimishra91/transformers-tutorials/blob/master/transformers_summarization_wandb.ipynb#scrollTo=OKRpFvYhBauC). \ud83c\udf0e\n- A blog post on [Distributed Training: Train BART/T5 for Summarization using \ud83e\udd17 Transformers and Amazon SageMaker](https://huggingface.co/blog/sagemaker-distributed-training-seq2seq).\n- [`T5ForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization.ipynb).\n- [`TFT5ForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/summarization) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization-tf.ipynb).\n- [`FlaxT5ForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/summarization).\n- [Summarization](https://huggingface.co/course/chapter7/5?fw=pt#summarization) chapter of the \ud83e\udd17 Hugging Face course.\n- [Summarization task guide](../tasks/summarization)\n\n<PipelineTag pipeline=\"fill-mask\"/>\n\n- [`FlaxT5ForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#t5-like-span-masked-language-modeling) for training T5 with a span-masked language model objective. The script also shows how to train a T5 tokenizer. [`FlaxT5ForConditionalGeneration`] is also supported by this [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).\n\n<PipelineTag pipeline=\"translation\"/>\n\n- [`T5ForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/translation) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation.ipynb).\n- [`TFT5ForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/translation) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation-tf.ipynb).\n- [Translation task guide](../tasks/translation)\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- A notebook on how to [finetune T5 for question answering with TensorFlow 2](https://colab.research.google.com/github/snapthat/TF-T5-text-to-text/blob/master/snapthatT5/notebooks/TF-T5-Datasets%20Training.ipynb). \ud83c\udf0e\n- A notebook on how to [finetune T5 for question answering on a TPU](https://colab.research.google.com/github/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb#scrollTo=QLGiFCDqvuil)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 99c73119-902b-4654-9e10-daf0d23491f4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"99c73119-902b-4654-9e10-daf0d23491f4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d5af6e79-1dee-4cb8-b561-40e082524adb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/t5\", \"title\": \"T5\", \"tokens\": 5728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"41e6f7d959dc03f25232c1b699e700a631b1c90760839cc88043e83f6d8d7740\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 19627, \"end_char_idx\": 22332, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "doc_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "ref_doc_id": "a6fc8f90-d37e-4c28-a204-cb6c4db0ee8a", "chroma:document": "The script also shows how to train a T5 tokenizer. [`FlaxT5ForConditionalGeneration`] is also supported by this [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).\n\n<PipelineTag pipeline=\"translation\"/>\n\n- [`T5ForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/translation) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation.ipynb).\n- [`TFT5ForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/translation) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation-tf.ipynb).\n- [Translation task guide](../tasks/translation)\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- A notebook on how to [finetune T5 for question answering with TensorFlow 2](https://colab.research.google.com/github/snapthat/TF-T5-text-to-text/blob/master/snapthatT5/notebooks/TF-T5-Datasets%20Training.ipynb). \ud83c\udf0e\n- A notebook on how to [finetune T5 for question answering on a TPU](https://colab.research.google.com/github/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb#scrollTo=QLGiFCDqvuil).\n\n\ud83d\ude80 **Deploy**\n- A blog post on how to deploy [T5 11B for inference for less than $500](https://www.philschmid.de/deploy-t5-11b).\n\n## T5Config\n\n[[autodoc]] T5Config\n\n## T5Tokenizer\n\n[[autodoc]] T5Tokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## T5TokenizerFast\n\n[[autodoc]] T5TokenizerFast\n\n<frameworkcontent>\n<pt>\n\n## T5Model\n\n[[autodoc]] T5Model\n    - forward\n\n## T5ForConditionalGeneration\n\n[[autodoc]] T5ForConditionalGeneration\n    - forward\n\n## T5EncoderModel\n\n[[autodoc]] T5EncoderModel\n    - forward\n\n## T5ForSequenceClassification\n\n[[autodoc]] T5ForSequenceClassification\n    - forward\n\n## T5ForTokenClassification\n\n[[autodoc]] T5ForTokenClassification\n    - forward\n\n## T5ForQuestionAnswering\n\n[[autodoc]] T5ForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFT5Model\n\n[[autodoc]] TFT5Model\n    - call\n\n## TFT5ForConditionalGeneration\n\n[[autodoc]] TFT5ForConditionalGeneration\n    - call\n\n## TFT5EncoderModel\n\n[[autodoc]] TFT5EncoderModel\n    - call\n\n</tf>\n<jax>\n\n## FlaxT5Model\n\n[[autodoc]] FlaxT5Model\n    - __call__\n    - encode\n    - decode\n\n## FlaxT5ForConditionalGeneration\n\n[[autodoc]] FlaxT5ForConditionalGeneration\n    - __call__\n    - encode\n    - decode\n\n## FlaxT5EncoderModel\n\n[[autodoc]] FlaxT5EncoderModel\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 0c621815-2461-42d9-b4bd-d977469a6ffc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0c621815-2461-42d9-b4bd-d977469a6ffc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d0dd02a0-dbdc-48a8-a986-7cde833c91ba\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8b008555e541080dd81bad44cd27b474127daf556111cf7282721a29acfc2f31\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"377fbf2f-f066-4420-99bc-3ddc3449a4e7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5c546cc2f33c219d055b079dd32ff47547c5ee34f803231abcd7426b847fcc0e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2776, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d0dd02a0-dbdc-48a8-a986-7cde833c91ba", "doc_id": "d0dd02a0-dbdc-48a8-a986-7cde833c91ba", "ref_doc_id": "d0dd02a0-dbdc-48a8-a986-7cde833c91ba", "chroma:document": "# OpenAI GPT2\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=gpt2\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-gpt2-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/gpt2\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nOpenAI GPT-2 model was proposed in [Language Models are Unsupervised Multitask Learners](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) by Alec\nRadford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei and Ilya Sutskever from [OpenAI](https://huggingface.co/openai). It's a causal (unidirectional)\ntransformer pretrained using language modeling on a very large corpus of ~40 GB of text data.\n\nThe abstract from the paper is the following:\n\n*GPT-2 is a large transformer-based language model with 1.5 billion parameters, trained on a dataset[1] of 8 million\nweb pages. GPT-2 is trained with a simple objective: predict the next word, given all of the previous words within some\ntext. The diversity of the dataset causes this simple goal to contain naturally occurring demonstrations of many tasks\nacross diverse domains. GPT-2 is a direct scale-up of GPT, with more than 10X the parameters and trained on more than\n10X the amount of data.*\n\n[Write With Transformer](https://transformer.huggingface.co/doc/gpt2-large) is a webapp created and hosted by\nHugging Face showcasing the generative capabilities of several models. GPT-2 is one of them and is available in five\ndifferent sizes: small, medium, large, xl and a distilled version of the small checkpoint: *distilgpt-2*.\n\nThis model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://openai.com/blog/better-language-models/).\n\n## Usage tips\n\n- GPT-2 is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than\n  the left.\n- GPT-2 was trained with a causal language modeling (CLM) objective and is therefore powerful at predicting the next\n  token in a sequence. Leveraging this feature allows GPT-2 to generate syntactically coherent text as it can be\n  observed in the *run_generation.py* example script.\n- The model can take the *past_key_values* (for PyTorch) or *past* (for TF) as input, which is the previously computed\n  key/value attention pairs. Using this (*past_key_values* or *past*) value prevents the model from re-computing\n  pre-computed values in the context of text generation. For PyTorch, see *past_key_values* argument of the\n  [`GPT2Model.forward`] method, or for TF the *past* argument of the\n  [`TFGPT2Model.call`] method for more information on its usage."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 377fbf2f-f066-4420-99bc-3ddc3449a4e7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"377fbf2f-f066-4420-99bc-3ddc3449a4e7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d0dd02a0-dbdc-48a8-a986-7cde833c91ba\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8b008555e541080dd81bad44cd27b474127daf556111cf7282721a29acfc2f31\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0c621815-2461-42d9-b4bd-d977469a6ffc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2afcc1e6695848523a447a8fd4aae9c493a3019851468330c043eb0fe0ff51a0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d5b0410f-baa4-4739-a9ad-00609a787738\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9e5fc09560edb7effbd9ab67a333c03eda5b98ed4acbc8cdf0de13133277290d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1254, \"end_char_idx\": 4229, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d0dd02a0-dbdc-48a8-a986-7cde833c91ba", "doc_id": "d0dd02a0-dbdc-48a8-a986-7cde833c91ba", "ref_doc_id": "d0dd02a0-dbdc-48a8-a986-7cde833c91ba", "chroma:document": "GPT-2 is a direct scale-up of GPT, with more than 10X the parameters and trained on more than\n10X the amount of data.*\n\n[Write With Transformer](https://transformer.huggingface.co/doc/gpt2-large) is a webapp created and hosted by\nHugging Face showcasing the generative capabilities of several models. GPT-2 is one of them and is available in five\ndifferent sizes: small, medium, large, xl and a distilled version of the small checkpoint: *distilgpt-2*.\n\nThis model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://openai.com/blog/better-language-models/).\n\n## Usage tips\n\n- GPT-2 is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than\n  the left.\n- GPT-2 was trained with a causal language modeling (CLM) objective and is therefore powerful at predicting the next\n  token in a sequence. Leveraging this feature allows GPT-2 to generate syntactically coherent text as it can be\n  observed in the *run_generation.py* example script.\n- The model can take the *past_key_values* (for PyTorch) or *past* (for TF) as input, which is the previously computed\n  key/value attention pairs. Using this (*past_key_values* or *past*) value prevents the model from re-computing\n  pre-computed values in the context of text generation. For PyTorch, see *past_key_values* argument of the\n  [`GPT2Model.forward`] method, or for TF the *past* argument of the\n  [`TFGPT2Model.call`] method for more information on its usage.\n- Enabling the *scale_attn_by_inverse_layer_idx* and *reorder_and_upcast_attn* flags will apply the training stability\n  improvements from [Mistral](https://github.com/stanford-crfm/mistral/) (for PyTorch only).\n\n## Usage example\n\nThe `generate()` method can be used to generate text using GPT2 model.\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"gpt2\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"gpt2\")\n\n>>> prompt = \"GPT2 is a model developed by OpenAI.\"\n\n>>> input_ids = tokenizer(prompt, return_tensors=\"pt\").input_ids\n\n>>> gen_tokens = model.generate(\n...     input_ids,\n...     do_sample=True,\n...     temperature=0.9,\n...     max_length=100,\n... )\n>>> gen_text = tokenizer.batch_decode(gen_tokens)[0]\n```\n\n## Using Flash Attention 2\n\nFlash Attention 2 is a faster, optimized version of the attention scores computation which relies on `cuda` kernels.\n\n### Installation \n\nFirst, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features). If your hardware is not compatible with Flash Attention 2, you can still benefit from attention kernel optimisations through Better Transformer support covered [above](https://huggingface.co/docs/transformers/main/en/model_doc/bark#using-better-transformer)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: d5b0410f-baa4-4739-a9ad-00609a787738
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d5b0410f-baa4-4739-a9ad-00609a787738\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d0dd02a0-dbdc-48a8-a986-7cde833c91ba\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8b008555e541080dd81bad44cd27b474127daf556111cf7282721a29acfc2f31\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"377fbf2f-f066-4420-99bc-3ddc3449a4e7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5d8f4f5ab7558d55de998f8a0c48985380ae7606db0f52835daea5d2e00d3ce4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5986dcbb-961e-4d8a-ae28-0b3158cef244\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"813428da12c70079b3f413631e3a9d54dd1d9f554818584433187e584e95ae71\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2592, \"end_char_idx\": 5876, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d0dd02a0-dbdc-48a8-a986-7cde833c91ba", "doc_id": "d0dd02a0-dbdc-48a8-a986-7cde833c91ba", "ref_doc_id": "d0dd02a0-dbdc-48a8-a986-7cde833c91ba", "chroma:document": "For PyTorch, see *past_key_values* argument of the\n  [`GPT2Model.forward`] method, or for TF the *past* argument of the\n  [`TFGPT2Model.call`] method for more information on its usage.\n- Enabling the *scale_attn_by_inverse_layer_idx* and *reorder_and_upcast_attn* flags will apply the training stability\n  improvements from [Mistral](https://github.com/stanford-crfm/mistral/) (for PyTorch only).\n\n## Usage example\n\nThe `generate()` method can be used to generate text using GPT2 model.\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"gpt2\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"gpt2\")\n\n>>> prompt = \"GPT2 is a model developed by OpenAI.\"\n\n>>> input_ids = tokenizer(prompt, return_tensors=\"pt\").input_ids\n\n>>> gen_tokens = model.generate(\n...     input_ids,\n...     do_sample=True,\n...     temperature=0.9,\n...     max_length=100,\n... )\n>>> gen_text = tokenizer.batch_decode(gen_tokens)[0]\n```\n\n## Using Flash Attention 2\n\nFlash Attention 2 is a faster, optimized version of the attention scores computation which relies on `cuda` kernels.\n\n### Installation \n\nFirst, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features). If your hardware is not compatible with Flash Attention 2, you can still benefit from attention kernel optimisations through Better Transformer support covered [above](https://huggingface.co/docs/transformers/main/en/model_doc/bark#using-better-transformer).\n\nNext, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\n### Usage\n\nTo load a model using Flash Attention 2, we can pass the argument `attn_implementation=\"flash_attention_2\"` to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). We'll also load the model in half-precision (e.g. `torch.float16`), since it results in almost no degradation to audio quality but significantly lower memory usage and faster inference:\n\n```python\n>>> import torch\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n>>> device = \"cuda\" # the device to load the model onto\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"gpt2\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"gpt2\")\n\n>>> prompt = \"def hello_world():\"\n\n>>> model_inputs = tokenizer([prompt], return_tensors=\"pt\").to(device)\n>>> model.to(device)\n\n>>> generated_ids = model.generate(**model_inputs, max_new_tokens=100, do_sample=True)\n>>> tokenizer.batch_decode(generated_ids)[0]\n```\n\n\n### Expected speedups\n\nBelow is an expected speedup diagram that compares pure inference time between the native implementation in transformers using `gpt2` checkpoint and the Flash Attention 2 version of the model using a sequence length of 512.\n\n<div style=\"text-align: center\">\n<img src=\"https://huggingface.co/datasets/EduardoPacheco/documentation-images/resolve/main/gpt2_flash_attention_2_speedup.jpg\">\n</div>"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 5986dcbb-961e-4d8a-ae28-0b3158cef244
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5986dcbb-961e-4d8a-ae28-0b3158cef244\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d0dd02a0-dbdc-48a8-a986-7cde833c91ba\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8b008555e541080dd81bad44cd27b474127daf556111cf7282721a29acfc2f31\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d5b0410f-baa4-4739-a9ad-00609a787738\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cc49e1c2393ac5bf979cce9ffc9f3e5c253769660a5fd577935b3e962bd495f9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"10a06938-12ac-4abe-b8a0-45e9d72580cb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3990f5f95ce583b47a4df6958c2ccdd47cac1ea2db295384b46798e9ea3109bb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4231, \"end_char_idx\": 7135, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d0dd02a0-dbdc-48a8-a986-7cde833c91ba", "doc_id": "d0dd02a0-dbdc-48a8-a986-7cde833c91ba", "ref_doc_id": "d0dd02a0-dbdc-48a8-a986-7cde833c91ba", "chroma:document": "Next, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\n### Usage\n\nTo load a model using Flash Attention 2, we can pass the argument `attn_implementation=\"flash_attention_2\"` to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). We'll also load the model in half-precision (e.g. `torch.float16`), since it results in almost no degradation to audio quality but significantly lower memory usage and faster inference:\n\n```python\n>>> import torch\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n>>> device = \"cuda\" # the device to load the model onto\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"gpt2\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"gpt2\")\n\n>>> prompt = \"def hello_world():\"\n\n>>> model_inputs = tokenizer([prompt], return_tensors=\"pt\").to(device)\n>>> model.to(device)\n\n>>> generated_ids = model.generate(**model_inputs, max_new_tokens=100, do_sample=True)\n>>> tokenizer.batch_decode(generated_ids)[0]\n```\n\n\n### Expected speedups\n\nBelow is an expected speedup diagram that compares pure inference time between the native implementation in transformers using `gpt2` checkpoint and the Flash Attention 2 version of the model using a sequence length of 512.\n\n<div style=\"text-align: center\">\n<img src=\"https://huggingface.co/datasets/EduardoPacheco/documentation-images/resolve/main/gpt2_flash_attention_2_speedup.jpg\">\n</div>\n\n\n## Using Scaled Dot Product Attention (SDPA)\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function\nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the\n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html)\nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set\n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```python\nfrom transformers import AutoModelForCausalLM\nmodel = AutoModelForCausalLM.from_pretrained(\"gpt2\", torch_dtype=torch.float16, attn_implementation=\"sdpa\")\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (rtx3080ti-16GB, PyTorch 2.2.1, OS Ubuntu 22.04) using `float16` with\n[gpt2-large](https://huggingface.co/openai-community/gpt2-large), we saw the\nfollowing speedups during training and inference."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 10a06938-12ac-4abe-b8a0-45e9d72580cb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"10a06938-12ac-4abe-b8a0-45e9d72580cb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d0dd02a0-dbdc-48a8-a986-7cde833c91ba\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8b008555e541080dd81bad44cd27b474127daf556111cf7282721a29acfc2f31\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5986dcbb-961e-4d8a-ae28-0b3158cef244\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1f696c2265f8edebd6ec03d5daa62290059acc0f3538eacefe51b3e4e8e84196\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"77265173-593a-4b7e-8d08-ba70ffe489a8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"aa500fff1de9a4a5a23454332b8ea8924cfa71a69d2a4a1f8e659de5fbdd6c49\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5879, \"end_char_idx\": 8725, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d0dd02a0-dbdc-48a8-a986-7cde833c91ba", "doc_id": "d0dd02a0-dbdc-48a8-a986-7cde833c91ba", "ref_doc_id": "d0dd02a0-dbdc-48a8-a986-7cde833c91ba", "chroma:document": "## Using Scaled Dot Product Attention (SDPA)\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function\nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the\n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html)\nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set\n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```python\nfrom transformers import AutoModelForCausalLM\nmodel = AutoModelForCausalLM.from_pretrained(\"gpt2\", torch_dtype=torch.float16, attn_implementation=\"sdpa\")\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (rtx3080ti-16GB, PyTorch 2.2.1, OS Ubuntu 22.04) using `float16` with\n[gpt2-large](https://huggingface.co/openai-community/gpt2-large), we saw the\nfollowing speedups during training and inference.\n\n### Training\n| Batch size | Seq len |  Time per batch (Eager - s) | Time per batch (SDPA - s) | Speedup (%) | Eager peak mem (MB) | SDPA peak mem (MB) |    Mem saving (%) |\n|-----------:|--------:|----------------------------:|--------------------------:|------------:|--------------------:|-------------------:|------------------:|\n|          1 |     128 |                       0.039 |                     0.032 |      23.042 |             3482.32 |            3494.62 |            -0.352 |\n|          1 |     256 |                       0.073 |                     0.059 |       25.15 |             3546.66 |             3552.6 |            -0.167 |\n|          1 |     512 |                       0.155 |                     0.118 |       30.96 |              4230.1 |            3665.59 |              15.4 |\n|          1 |    1024 |                       0.316 |                     0.209 |      50.839 |             8682.26 |            4881.09 |            77.875 |\n|          2 |     128 |                        0.07 |                      0.06 |      15.324 |              3557.8 |            3545.91 |             0.335 |\n|          2 |     256 |                       0.143 |                     0.122 |       16.53 |              3901.5 |            3657.68 |             6.666 |\n|          2 |     512 |                       0.267 |                     0.213 |      25.626 |             7062.21 |            4876.47 |            44.822 |\n|          2 |    1024 |                         OOM |                     0.404 |           / |                 OOM |            8096."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 77265173-593a-4b7e-8d08-ba70ffe489a8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"77265173-593a-4b7e-8d08-ba70ffe489a8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d0dd02a0-dbdc-48a8-a986-7cde833c91ba\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8b008555e541080dd81bad44cd27b474127daf556111cf7282721a29acfc2f31\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"10a06938-12ac-4abe-b8a0-45e9d72580cb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6633f199f3d590e898e23eb0248907889d797e4279053318f60cf03b21725492\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"080ded59-8495-4ac4-a96b-f1ca0b786069\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"47019343afe82b5dc7d745c78f3acfc0a99e588645e3a0cb6c11a8bebaa7275c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7519, \"end_char_idx\": 10244, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d0dd02a0-dbdc-48a8-a986-7cde833c91ba", "doc_id": "d0dd02a0-dbdc-48a8-a986-7cde833c91ba", "ref_doc_id": "d0dd02a0-dbdc-48a8-a986-7cde833c91ba", "chroma:document": "039 |                     0.032 |      23.042 |             3482.32 |            3494.62 |            -0.352 |\n|          1 |     256 |                       0.073 |                     0.059 |       25.15 |             3546.66 |             3552.6 |            -0.167 |\n|          1 |     512 |                       0.155 |                     0.118 |       30.96 |              4230.1 |            3665.59 |              15.4 |\n|          1 |    1024 |                       0.316 |                     0.209 |      50.839 |             8682.26 |            4881.09 |            77.875 |\n|          2 |     128 |                        0.07 |                      0.06 |      15.324 |              3557.8 |            3545.91 |             0.335 |\n|          2 |     256 |                       0.143 |                     0.122 |       16.53 |              3901.5 |            3657.68 |             6.666 |\n|          2 |     512 |                       0.267 |                     0.213 |      25.626 |             7062.21 |            4876.47 |            44.822 |\n|          2 |    1024 |                         OOM |                     0.404 |           / |                 OOM |            8096.35 | SDPA does not OOM |\n|          4 |     128 |                       0.134 |                     0.128 |       4.412 |             3675.79 |            3648.72 |             0.742 |\n|          4 |     256 |                       0.243 |                     0.217 |      12.292 |             6129.76 |            4871.12 |            25.839 |\n|          4 |     512 |                       0.494 |                     0.406 |      21.687 |             12466.6 |            8102.64 |            53.858 |\n|          4 |    1024 |                         OOM |                     0.795 |           / |                 OOM |            14568.2 | SDPA does not OOM |\n\n### Inference\n| Batch size | Seq len | Per token latency Eager (ms) | Per token latency SDPA (ms) | Speedup (%) | Mem Eager (MB) | Mem SDPA (MB) | Mem saved (%) |\n|-----------:|--------:|-----------------------------:|----------------------------:|------------:|---------------:|--------------:|--------------:|\n|          1 |     128 |                        7.991 |                       6.968 |      14.681 |         1685.2 |       1701.32 |        -0.947 |\n|          1 |     256 |                        8.462 |                       7.199 |      17.536 |        1745.49 |       1770.78 |        -1.428 |\n|          1 |     512 |                         8.68 |                       7.853 |      10.529 |        1907.69 |       1921.29 |        -0.708 |\n|          1 |     768 |                        9.101 |                       8.365 |       8."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 080ded59-8495-4ac4-a96b-f1ca0b786069
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"080ded59-8495-4ac4-a96b-f1ca0b786069\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d0dd02a0-dbdc-48a8-a986-7cde833c91ba\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8b008555e541080dd81bad44cd27b474127daf556111cf7282721a29acfc2f31\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"77265173-593a-4b7e-8d08-ba70ffe489a8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"05ecf144cd50956cde7813fec81a113c1b6c2677e1c9e7736b3e9989bea146f8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a6ab99de-fc6b-4343-9201-7525bfe2dbda\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"12acf598121ee8eadf85735bfbbfe4d38748e5fda0f93f832a75e7492c42ed04\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8841, \"end_char_idx\": 11454, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d0dd02a0-dbdc-48a8-a986-7cde833c91ba", "doc_id": "d0dd02a0-dbdc-48a8-a986-7cde833c91ba", "ref_doc_id": "d0dd02a0-dbdc-48a8-a986-7cde833c91ba", "chroma:document": "412 |             3675.79 |            3648.72 |             0.742 |\n|          4 |     256 |                       0.243 |                     0.217 |      12.292 |             6129.76 |            4871.12 |            25.839 |\n|          4 |     512 |                       0.494 |                     0.406 |      21.687 |             12466.6 |            8102.64 |            53.858 |\n|          4 |    1024 |                         OOM |                     0.795 |           / |                 OOM |            14568.2 | SDPA does not OOM |\n\n### Inference\n| Batch size | Seq len | Per token latency Eager (ms) | Per token latency SDPA (ms) | Speedup (%) | Mem Eager (MB) | Mem SDPA (MB) | Mem saved (%) |\n|-----------:|--------:|-----------------------------:|----------------------------:|------------:|---------------:|--------------:|--------------:|\n|          1 |     128 |                        7.991 |                       6.968 |      14.681 |         1685.2 |       1701.32 |        -0.947 |\n|          1 |     256 |                        8.462 |                       7.199 |      17.536 |        1745.49 |       1770.78 |        -1.428 |\n|          1 |     512 |                         8.68 |                       7.853 |      10.529 |        1907.69 |       1921.29 |        -0.708 |\n|          1 |     768 |                        9.101 |                       8.365 |       8.791 |        2032.93 |       2068.12 |        -1.701 |\n|          2 |     128 |                        9.169 |                       9.001 |       1.861 |        1803.84 |        1811.4 |        -0.418 |\n|          2 |     256 |                        9.907 |                        9.78 |       1.294 |        1907.72 |       1921.44 |        -0.714 |\n|          2 |     512 |                       11.519 |                      11.644 |      -1.071 |        2176.86 |       2197.75 |        -0.951 |\n|          2 |     768 |                       13.022 |                      13.407 |      -2.873 |         2464.3 |       2491.06 |        -1.074 |\n|          4 |     128 |                       10.097 |                       9.831 |       2.709 |        1942.25 |       1985.13 |         -2.16 |\n|          4 |     256 |                       11.599 |                      11.398 |       1.764 |        2177.28 |       2197.86 |        -0.937 |\n|          4 |     512 |                       14.653 |                       14.45 |       1.411 |        2753.16 |       2772.57 |          -0.7 |\n|          4 |     768 |                       17.846 |                      17.617 |       1.299 |        3327."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: a6ab99de-fc6b-4343-9201-7525bfe2dbda
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a6ab99de-fc6b-4343-9201-7525bfe2dbda\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d0dd02a0-dbdc-48a8-a986-7cde833c91ba\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8b008555e541080dd81bad44cd27b474127daf556111cf7282721a29acfc2f31\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"080ded59-8495-4ac4-a96b-f1ca0b786069\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d5e8ad10cfc9b13e2e2dc6809d5d19edf1a60889269c5d6055c95c421aa8983f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7b2e5258-421e-4b48-96b6-9dab4aad39be\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c1e143b8950968e7651beafcfef36155e5dc8a158284190cf6846d1c2bb73270\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10200, \"end_char_idx\": 12773, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d0dd02a0-dbdc-48a8-a986-7cde833c91ba", "doc_id": "d0dd02a0-dbdc-48a8-a986-7cde833c91ba", "ref_doc_id": "d0dd02a0-dbdc-48a8-a986-7cde833c91ba", "chroma:document": "101 |                       8.365 |       8.791 |        2032.93 |       2068.12 |        -1.701 |\n|          2 |     128 |                        9.169 |                       9.001 |       1.861 |        1803.84 |        1811.4 |        -0.418 |\n|          2 |     256 |                        9.907 |                        9.78 |       1.294 |        1907.72 |       1921.44 |        -0.714 |\n|          2 |     512 |                       11.519 |                      11.644 |      -1.071 |        2176.86 |       2197.75 |        -0.951 |\n|          2 |     768 |                       13.022 |                      13.407 |      -2.873 |         2464.3 |       2491.06 |        -1.074 |\n|          4 |     128 |                       10.097 |                       9.831 |       2.709 |        1942.25 |       1985.13 |         -2.16 |\n|          4 |     256 |                       11.599 |                      11.398 |       1.764 |        2177.28 |       2197.86 |        -0.937 |\n|          4 |     512 |                       14.653 |                       14.45 |       1.411 |        2753.16 |       2772.57 |          -0.7 |\n|          4 |     768 |                       17.846 |                      17.617 |       1.299 |        3327.04 |       3343.97 |        -0.506 |\n\n\n\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with GPT2. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-generation\"/>\n\n- A blog on how to [Finetune a non-English GPT-2 Model with Hugging Face](https://www.philschmid.de/fine-tune-a-non-english-gpt-2-model-with-huggingface).\n- A blog on [How to generate text: using different decoding methods for language generation with Transformers](https://huggingface.co/blog/how-to-generate) with GPT-2.\n- A blog on [Training CodeParrot \ud83e\udd9c from Scratch](https://huggingface.co/blog/codeparrot), a large GPT-2 model.\n- A blog on [Faster Text Generation with TensorFlow and XLA](https://huggingface.co/blog/tf-xla-generate) with GPT-2.\n- A blog on [How to train a Language Model with Megatron-LM](https://huggingface.co/blog/megatron-training) with a GPT-2 model.\n- A notebook on how to [finetune GPT2 to generate lyrics in the style of your favorite artist](https://colab.research.google.com/github/AlekseyKorshuk/huggingartists/blob/master/huggingartists-demo.ipynb)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 7b2e5258-421e-4b48-96b6-9dab4aad39be
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7b2e5258-421e-4b48-96b6-9dab4aad39be\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d0dd02a0-dbdc-48a8-a986-7cde833c91ba\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8b008555e541080dd81bad44cd27b474127daf556111cf7282721a29acfc2f31\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a6ab99de-fc6b-4343-9201-7525bfe2dbda\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c8fa34a85957facdd49f9769cf37a4a519b7f54a5dbb7b4651f3415d60e64a58\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7d119e95-82ad-4401-98fa-a55cc50b1020\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ec4de0dc4e01cdfe73437432d4bbe67267687c739604a5f35904b7f8ec1ee7fe\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11305, \"end_char_idx\": 14258, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d0dd02a0-dbdc-48a8-a986-7cde833c91ba", "doc_id": "d0dd02a0-dbdc-48a8-a986-7cde833c91ba", "ref_doc_id": "d0dd02a0-dbdc-48a8-a986-7cde833c91ba", "chroma:document": "16 |       2772.57 |          -0.7 |\n|          4 |     768 |                       17.846 |                      17.617 |       1.299 |        3327.04 |       3343.97 |        -0.506 |\n\n\n\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with GPT2. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-generation\"/>\n\n- A blog on how to [Finetune a non-English GPT-2 Model with Hugging Face](https://www.philschmid.de/fine-tune-a-non-english-gpt-2-model-with-huggingface).\n- A blog on [How to generate text: using different decoding methods for language generation with Transformers](https://huggingface.co/blog/how-to-generate) with GPT-2.\n- A blog on [Training CodeParrot \ud83e\udd9c from Scratch](https://huggingface.co/blog/codeparrot), a large GPT-2 model.\n- A blog on [Faster Text Generation with TensorFlow and XLA](https://huggingface.co/blog/tf-xla-generate) with GPT-2.\n- A blog on [How to train a Language Model with Megatron-LM](https://huggingface.co/blog/megatron-training) with a GPT-2 model.\n- A notebook on how to [finetune GPT2 to generate lyrics in the style of your favorite artist](https://colab.research.google.com/github/AlekseyKorshuk/huggingartists/blob/master/huggingartists-demo.ipynb). \ud83c\udf0e\n- A notebook on how to [finetune GPT2 to generate tweets in the style of your favorite Twitter user](https://colab.research.google.com/github/borisdayma/huggingtweets/blob/master/huggingtweets-demo.ipynb). \ud83c\udf0e\n- [Causal language modeling](https://huggingface.co/course/en/chapter7/6?fw=pt#training-a-causal-language-model-from-scratch) chapter of the \ud83e\udd17 Hugging Face Course.\n- [`GPT2LMHeadModel`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#gpt-2gpt-and-causal-language-modeling), [text generation example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-generation), and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n- [`TFGPT2LMHeadModel`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_clmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- [`FlaxGPT2LMHeadModel`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#causal-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/causal_language_modeling_flax.ipynb)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 7d119e95-82ad-4401-98fa-a55cc50b1020
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7d119e95-82ad-4401-98fa-a55cc50b1020\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d0dd02a0-dbdc-48a8-a986-7cde833c91ba\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8b008555e541080dd81bad44cd27b474127daf556111cf7282721a29acfc2f31\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7b2e5258-421e-4b48-96b6-9dab4aad39be\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt2\", \"title\": \"OpenAI GPT2\", \"tokens\": 4429, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"31a8c05e1b50dbab2ffd6f7bf13fd5daf460ba68bab2286030b39bf750933653\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12774, \"end_char_idx\": 15876, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d0dd02a0-dbdc-48a8-a986-7cde833c91ba", "doc_id": "d0dd02a0-dbdc-48a8-a986-7cde833c91ba", "ref_doc_id": "d0dd02a0-dbdc-48a8-a986-7cde833c91ba", "chroma:document": "\ud83c\udf0e\n- A notebook on how to [finetune GPT2 to generate tweets in the style of your favorite Twitter user](https://colab.research.google.com/github/borisdayma/huggingtweets/blob/master/huggingtweets-demo.ipynb). \ud83c\udf0e\n- [Causal language modeling](https://huggingface.co/course/en/chapter7/6?fw=pt#training-a-causal-language-model-from-scratch) chapter of the \ud83e\udd17 Hugging Face Course.\n- [`GPT2LMHeadModel`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#gpt-2gpt-and-causal-language-modeling), [text generation example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-generation), and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n- [`TFGPT2LMHeadModel`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_clmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- [`FlaxGPT2LMHeadModel`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#causal-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/causal_language_modeling_flax.ipynb).\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Causal language modeling task guide](../tasks/language_modeling)\n\n## GPT2Config\n\n[[autodoc]] GPT2Config\n\n## GPT2Tokenizer\n\n[[autodoc]] GPT2Tokenizer\n    - save_vocabulary\n\n## GPT2TokenizerFast\n\n[[autodoc]] GPT2TokenizerFast\n\n## GPT2 specific outputs\n\n[[autodoc]] models.gpt2.modeling_gpt2.GPT2DoubleHeadsModelOutput\n\n[[autodoc]] models.gpt2.modeling_tf_gpt2.TFGPT2DoubleHeadsModelOutput\n\n<frameworkcontent>\n<pt>\n\n## GPT2Model\n\n[[autodoc]] GPT2Model\n    - forward\n\n## GPT2LMHeadModel\n\n[[autodoc]] GPT2LMHeadModel\n    - forward\n\n## GPT2DoubleHeadsModel\n\n[[autodoc]] GPT2DoubleHeadsModel\n    - forward\n\n## GPT2ForQuestionAnswering\n\n[[autodoc]] GPT2ForQuestionAnswering\n    - forward\n\n## GPT2ForSequenceClassification\n\n[[autodoc]] GPT2ForSequenceClassification\n    - forward\n\n## GPT2ForTokenClassification\n\n[[autodoc]] GPT2ForTokenClassification\n    - forward\n\n</pt>\n<tf>\n\n## TFGPT2Model\n\n[[autodoc]] TFGPT2Model\n    - call\n\n## TFGPT2LMHeadModel\n\n[[autodoc]] TFGPT2LMHeadModel\n    - call\n\n## TFGPT2DoubleHeadsModel\n\n[[autodoc]] TFGPT2DoubleHeadsModel\n    - call\n\n## TFGPT2ForSequenceClassification\n\n[[autodoc]] TFGPT2ForSequenceClassification\n    - call\n\n## TFSequenceClassifierOutputWithPast\n\n[[autodoc]] modeling_tf_outputs.TFSequenceClassifierOutputWithPast\n\n## TFGPT2Tokenizer\n\n[[autodoc]] TFGPT2Tokenizer\n\n</tf>\n<jax>\n\n## FlaxGPT2Model\n\n[[autodoc]] FlaxGPT2Model\n    - __call__\n\n## FlaxGPT2LMHeadModel\n\n[[autodoc]] FlaxGPT2LMHeadModel\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 78165be4-0f26-4fd1-b7ea-dd69ab44d1a8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bros", "title": "BROS", "tokens": 1607, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"78165be4-0f26-4fd1-b7ea-dd69ab44d1a8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bros\", \"title\": \"BROS\", \"tokens\": 1607, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b10a5617-f0fa-4f52-9f3b-0fc1dff0b98c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bros\", \"title\": \"BROS\", \"tokens\": 1607, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bd7fb45aaab4c49a95f49dd399e04059e923c316980c3363bdbb76bcc3326d65\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"83c12a25-2781-4576-88cd-9e64bf041e7d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"97d15610b46e52c22c5aef59a729fd3425ebfbce03423e9ce4f3efa9ef32b83f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3536, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b10a5617-f0fa-4f52-9f3b-0fc1dff0b98c", "doc_id": "b10a5617-f0fa-4f52-9f3b-0fc1dff0b98c", "ref_doc_id": "b10a5617-f0fa-4f52-9f3b-0fc1dff0b98c", "chroma:document": "# BROS\n\n## Overview\n\nThe BROS model was proposed in [BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from Documents](https://arxiv.org/abs/2108.04539) by Teakgyu Hong, Donghyun Kim, Mingi Ji, Wonseok Hwang, Daehyun Nam, Sungrae Park.\n\nBROS stands for *BERT Relying On Spatiality*. It is an encoder-only Transformer model that takes a sequence of tokens and their bounding boxes as inputs and outputs a sequence of hidden states. BROS encode relative spatial information instead of using absolute spatial information.\n\nIt is pre-trained with two objectives: a token-masked language modeling objective (TMLM) used in BERT, and a novel area-masked language modeling objective (AMLM)\nIn TMLM, tokens are randomly masked, and the model predicts the masked tokens using spatial information and other unmasked tokens.\nAMLM is a 2D version of TMLM. It randomly masks text tokens and predicts with the same information as TMLM, but it masks text blocks (areas).\n\n`BrosForTokenClassification` has a simple linear layer on top of BrosModel. It predicts the label of each token.\n`BrosSpadeEEForTokenClassification` has an `initial_token_classifier` and `subsequent_token_classifier` on top of BrosModel. `initial_token_classifier` is used to predict the first token of each entity, and `subsequent_token_classifier` is used to predict the next token of within entity. `BrosSpadeELForTokenClassification` has an `entity_linker` on top of BrosModel. `entity_linker` is used to predict the relation between two entities.\n\n`BrosForTokenClassification` and `BrosSpadeEEForTokenClassification` essentially perform the same job. However, `BrosForTokenClassification` assumes input tokens are perfectly serialized (which is very challenging task since they exist in a 2D space), while `BrosSpadeEEForTokenClassification` allows for more flexibility in handling serialization errors as it predicts next connection tokens from one token.\n\n`BrosSpadeELForTokenClassification` perform the intra-entity linking task. It predicts relation from one token (of one entity) to another token (of another entity) if these two entities share some relation.\n\nBROS achieves comparable or better result on Key Information Extraction (KIE) benchmarks such as FUNSD, SROIE, CORD and SciTSR, without relying on explicit visual features.\n\nThe abstract from the paper is the following:\n\n*Key information extraction (KIE) from document images requires understanding the contextual and spatial semantics of texts in two-dimensional (2D) space. Many recent studies try to solve the task by developing pre-trained language models focusing on combining visual features from document images with texts and their layout. On the other hand, this paper tackles the problem by going back to the basic: effective combination of text and layout. Specifically, we propose a pre-trained language model, named BROS (BERT Relying On Spatiality), that encodes relative positions of texts in 2D space and learns from unlabeled documents with area-masking strategy. With this optimized training scheme for understanding texts in 2D space, BROS shows comparable or better performance compared to previous methods on four KIE benchmarks (FUNSD, SROIE*, CORD, and SciTSR) without relying on visual features. This paper also reveals two real-world challenges in KIE tasks-(1) minimizing the error from incorrect text ordering and (2) efficient learning from fewer downstream examples-and demonstrates the superiority of BROS over previous methods."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 83c12a25-2781-4576-88cd-9e64bf041e7d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bros", "title": "BROS", "tokens": 1607, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"83c12a25-2781-4576-88cd-9e64bf041e7d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bros\", \"title\": \"BROS\", \"tokens\": 1607, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b10a5617-f0fa-4f52-9f3b-0fc1dff0b98c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bros\", \"title\": \"BROS\", \"tokens\": 1607, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bd7fb45aaab4c49a95f49dd399e04059e923c316980c3363bdbb76bcc3326d65\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"78165be4-0f26-4fd1-b7ea-dd69ab44d1a8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bros\", \"title\": \"BROS\", \"tokens\": 1607, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0b5d6965c6205045e767e448d585a8eb2337ebc11d893b94e68ced61dfcca0ba\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cdd83f90-9407-440f-8267-bfc883e77d93\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a51d28b889aa12b15039a238fa0681ba8d697c6c02ef2efef96e03139551ee02\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1663, \"end_char_idx\": 4829, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b10a5617-f0fa-4f52-9f3b-0fc1dff0b98c", "doc_id": "b10a5617-f0fa-4f52-9f3b-0fc1dff0b98c", "ref_doc_id": "b10a5617-f0fa-4f52-9f3b-0fc1dff0b98c", "chroma:document": "However, `BrosForTokenClassification` assumes input tokens are perfectly serialized (which is very challenging task since they exist in a 2D space), while `BrosSpadeEEForTokenClassification` allows for more flexibility in handling serialization errors as it predicts next connection tokens from one token.\n\n`BrosSpadeELForTokenClassification` perform the intra-entity linking task. It predicts relation from one token (of one entity) to another token (of another entity) if these two entities share some relation.\n\nBROS achieves comparable or better result on Key Information Extraction (KIE) benchmarks such as FUNSD, SROIE, CORD and SciTSR, without relying on explicit visual features.\n\nThe abstract from the paper is the following:\n\n*Key information extraction (KIE) from document images requires understanding the contextual and spatial semantics of texts in two-dimensional (2D) space. Many recent studies try to solve the task by developing pre-trained language models focusing on combining visual features from document images with texts and their layout. On the other hand, this paper tackles the problem by going back to the basic: effective combination of text and layout. Specifically, we propose a pre-trained language model, named BROS (BERT Relying On Spatiality), that encodes relative positions of texts in 2D space and learns from unlabeled documents with area-masking strategy. With this optimized training scheme for understanding texts in 2D space, BROS shows comparable or better performance compared to previous methods on four KIE benchmarks (FUNSD, SROIE*, CORD, and SciTSR) without relying on visual features. This paper also reveals two real-world challenges in KIE tasks-(1) minimizing the error from incorrect text ordering and (2) efficient learning from fewer downstream examples-and demonstrates the superiority of BROS over previous methods.*\n\nThis model was contributed by [jinho8345](https://huggingface.co/jinho8345). The original code can be found [here](https://github.com/clovaai/bros).\n\n## Usage tips and examples\n\n- [`~transformers.BrosModel.forward`] requires `input_ids` and `bbox` (bounding box). Each bounding box should be in (x0, y0, x1, y1) format (top-left corner, bottom-right corner). Obtaining of Bounding boxes depends on external OCR system. The `x` coordinate should be normalized by document image width, and the `y` coordinate should be normalized by document image height.\n\n```python\ndef expand_and_normalize_bbox(bboxes, doc_width, doc_height):\n    # here, bboxes are numpy array\n\n    # Normalize bbox -> 0 ~ 1\n    bboxes[:, [0, 2]] = bboxes[:, [0, 2]] / width\n    bboxes[:, [1, 3]] = bboxes[:, [1, 3]] / height\n```\n\n- [`~transformers.BrosForTokenClassification.forward`, `~transformers.BrosSpadeEEForTokenClassification.forward`, `~transformers.BrosSpadeEEForTokenClassification.forward`] require not only `input_ids` and `bbox` but also `box_first_token_mask` for loss calculation. It is a mask to filter out non-first tokens of each box. You can obtain this mask by saving start token indices of bounding boxes when creating `input_ids` from words. You can make `box_first_token_mask` with following code,"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: cdd83f90-9407-440f-8267-bfc883e77d93
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bros", "title": "BROS", "tokens": 1607, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"cdd83f90-9407-440f-8267-bfc883e77d93\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bros\", \"title\": \"BROS\", \"tokens\": 1607, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b10a5617-f0fa-4f52-9f3b-0fc1dff0b98c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bros\", \"title\": \"BROS\", \"tokens\": 1607, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bd7fb45aaab4c49a95f49dd399e04059e923c316980c3363bdbb76bcc3326d65\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"83c12a25-2781-4576-88cd-9e64bf041e7d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bros\", \"title\": \"BROS\", \"tokens\": 1607, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0a666a59f58a765d86aa516e27753bd154e4422ea8b740ffe0422bd69eea40be\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3298, \"end_char_idx\": 6382, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b10a5617-f0fa-4f52-9f3b-0fc1dff0b98c", "doc_id": "b10a5617-f0fa-4f52-9f3b-0fc1dff0b98c", "ref_doc_id": "b10a5617-f0fa-4f52-9f3b-0fc1dff0b98c", "chroma:document": "This paper also reveals two real-world challenges in KIE tasks-(1) minimizing the error from incorrect text ordering and (2) efficient learning from fewer downstream examples-and demonstrates the superiority of BROS over previous methods.*\n\nThis model was contributed by [jinho8345](https://huggingface.co/jinho8345). The original code can be found [here](https://github.com/clovaai/bros).\n\n## Usage tips and examples\n\n- [`~transformers.BrosModel.forward`] requires `input_ids` and `bbox` (bounding box). Each bounding box should be in (x0, y0, x1, y1) format (top-left corner, bottom-right corner). Obtaining of Bounding boxes depends on external OCR system. The `x` coordinate should be normalized by document image width, and the `y` coordinate should be normalized by document image height.\n\n```python\ndef expand_and_normalize_bbox(bboxes, doc_width, doc_height):\n    # here, bboxes are numpy array\n\n    # Normalize bbox -> 0 ~ 1\n    bboxes[:, [0, 2]] = bboxes[:, [0, 2]] / width\n    bboxes[:, [1, 3]] = bboxes[:, [1, 3]] / height\n```\n\n- [`~transformers.BrosForTokenClassification.forward`, `~transformers.BrosSpadeEEForTokenClassification.forward`, `~transformers.BrosSpadeEEForTokenClassification.forward`] require not only `input_ids` and `bbox` but also `box_first_token_mask` for loss calculation. It is a mask to filter out non-first tokens of each box. You can obtain this mask by saving start token indices of bounding boxes when creating `input_ids` from words. You can make `box_first_token_mask` with following code,\n\n\n```python\ndef make_box_first_token_mask(bboxes, words, tokenizer, max_seq_length=512):\n\n    box_first_token_mask = np.zeros(max_seq_length, dtype=np.bool_)\n\n    # encode(tokenize) each word from words (List[str])\n    input_ids_list: List[List[int]] = [tokenizer.encode(e, add_special_tokens=False) for e in words]\n\n    # get the length of each box\n    tokens_length_list: List[int] = [len(l) for l in input_ids_list]\n\n    box_end_token_indices = np.array(list(itertools.accumulate(tokens_length_list)))\n    box_start_token_indices = box_end_token_indices - np.array(tokens_length_list)\n\n    # filter out the indices that are out of max_seq_length\n    box_end_token_indices = box_end_token_indices[box_end_token_indices < max_seq_length - 1]\n    if len(box_start_token_indices) > len(box_end_token_indices):\n        box_start_token_indices = box_start_token_indices[: len(box_end_token_indices)]\n\n    # set box_start_token_indices to True\n    box_first_token_mask[box_start_token_indices] = True\n\n    return box_first_token_mask\n\n```\n\n## Resources\n\n- Demo scripts can be found [here](https://github.com/clovaai/bros).\n\n## BrosConfig\n\n[[autodoc]] BrosConfig\n\n## BrosProcessor\n\n[[autodoc]] BrosProcessor\n    - __call__\n\n## BrosModel\n\n[[autodoc]] BrosModel\n    - forward\n\n\n## BrosForTokenClassification\n\n[[autodoc]] BrosForTokenClassification\n    - forward\n\n## BrosSpadeEEForTokenClassification\n\n[[autodoc]] BrosSpadeEEForTokenClassification\n    - forward\n\n## BrosSpadeELForTokenClassification\n\n[[autodoc]] BrosSpadeELForTokenClassification\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 926fa27b-f07b-4b7e-b12f-e110c50685d6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"926fa27b-f07b-4b7e-b12f-e110c50685d6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta\", \"title\": \"RoBERTa\", \"tokens\": 3026, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f19435bd-da59-4db2-a96e-d489189cb136\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta\", \"title\": \"RoBERTa\", \"tokens\": 3026, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5600d5c6dca955efe139ace218bf61e3b2034c83533b5923f69810afa2660ed1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"365a6391-73e6-4a1c-bc53-618b94093636\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7e9a9bea7f2e43028c15abf364e8ff9680e7e382d6c64fa6eb7e1b10a403b0dc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3006, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f19435bd-da59-4db2-a96e-d489189cb136", "doc_id": "f19435bd-da59-4db2-a96e-d489189cb136", "ref_doc_id": "f19435bd-da59-4db2-a96e-d489189cb136", "chroma:document": "# RoBERTa\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=roberta\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-roberta-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/roberta-base\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n<a href=\"https://huggingface.co/papers/1907.11692\">\n<img alt=\"Paper page\" src=\"https://img.shields.io/badge/Paper%20page-1907.11692-green\">\n</a>\n</div>\n\n## Overview\n\nThe RoBERTa model was proposed in [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://arxiv.org/abs/1907.11692) by Yinhan Liu, [Myle Ott](https://huggingface.co/myleott), Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer\nLevy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov. It is based on Google's BERT model released in 2018.\n\nIt builds on BERT and modifies key hyperparameters, removing the next-sentence pretraining objective and training with\nmuch larger mini-batches and learning rates.\n\nThe abstract from the paper is the following:\n\n*Language model pretraining has led to significant performance gains but careful comparison between different\napproaches is challenging. Training is computationally expensive, often done on private datasets of different sizes,\nand, as we will show, hyperparameter choices have significant impact on the final results. We present a replication\nstudy of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and\ntraining data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every\nmodel published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results\nhighlight the importance of previously overlooked design choices, and raise questions about the source of recently\nreported improvements. We release our models and code.*\n\nThis model was contributed by [julien-c](https://huggingface.co/julien-c). The original code can be found [here](https://github.com/pytorch/fairseq/tree/master/examples/roberta).\n\n## Usage tips\n\n- This implementation is the same as [`BertModel`] with a minor tweak to the embeddings, as well as a setup\n  for RoBERTa pretrained models.\n- RoBERTa has the same architecture as BERT but uses a byte-level BPE as a tokenizer (same as GPT-2) and uses a\n  different pretraining scheme.\n- RoBERTa doesn't have `token_type_ids`, so you don't need to indicate which token belongs to which segment. Just\n  separate your segments with the separation token `tokenizer.sep_token` (or `</s>`).\n- RoBERTa is similar to BERT but with better pretraining techniques:\n\n    * Dynamic masking: tokens are masked differently at each epoch, whereas BERT does it once and for all.\n    * Sentence packing: Sentences are packed together to reach 512 tokens (so the sentences are in an order that may span several documents).\n    * Larger batches: Training uses larger batches."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 365a6391-73e6-4a1c-bc53-618b94093636
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"365a6391-73e6-4a1c-bc53-618b94093636\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta\", \"title\": \"RoBERTa\", \"tokens\": 3026, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f19435bd-da59-4db2-a96e-d489189cb136\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta\", \"title\": \"RoBERTa\", \"tokens\": 3026, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5600d5c6dca955efe139ace218bf61e3b2034c83533b5923f69810afa2660ed1\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"926fa27b-f07b-4b7e-b12f-e110c50685d6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta\", \"title\": \"RoBERTa\", \"tokens\": 3026, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f919e9d84035c7557106fe58a50fae2bb5596a128f5914b320741e3d14102c87\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"50b1bcaa-256f-4bcd-83d3-a9f67289daf2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"795a632c853f71615069acd4102ba841a2189e49b056d2a7ce1b461bb4899719\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1411, \"end_char_idx\": 4760, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f19435bd-da59-4db2-a96e-d489189cb136", "doc_id": "f19435bd-da59-4db2-a96e-d489189cb136", "ref_doc_id": "f19435bd-da59-4db2-a96e-d489189cb136", "chroma:document": "We present a replication\nstudy of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and\ntraining data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every\nmodel published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results\nhighlight the importance of previously overlooked design choices, and raise questions about the source of recently\nreported improvements. We release our models and code.*\n\nThis model was contributed by [julien-c](https://huggingface.co/julien-c). The original code can be found [here](https://github.com/pytorch/fairseq/tree/master/examples/roberta).\n\n## Usage tips\n\n- This implementation is the same as [`BertModel`] with a minor tweak to the embeddings, as well as a setup\n  for RoBERTa pretrained models.\n- RoBERTa has the same architecture as BERT but uses a byte-level BPE as a tokenizer (same as GPT-2) and uses a\n  different pretraining scheme.\n- RoBERTa doesn't have `token_type_ids`, so you don't need to indicate which token belongs to which segment. Just\n  separate your segments with the separation token `tokenizer.sep_token` (or `</s>`).\n- RoBERTa is similar to BERT but with better pretraining techniques:\n\n    * Dynamic masking: tokens are masked differently at each epoch, whereas BERT does it once and for all.\n    * Sentence packing: Sentences are packed together to reach 512 tokens (so the sentences are in an order that may span several documents).\n    * Larger batches: Training uses larger batches.\n    * Byte-level BPE vocabulary: Uses BPE with bytes as a subunit instead of characters, accommodating Unicode characters.\n- [CamemBERT](camembert) is a wrapper around RoBERTa. Refer to its model page for usage examples.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with RoBERTa. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- A blog on [Getting Started with Sentiment Analysis on Twitter](https://huggingface.co/blog/sentiment-analysis-twitter) using RoBERTa and the [Inference API](https://huggingface.co/inference-api).\n- A blog on [Opinion Classification with Kili and Hugging Face AutoTrain](https://huggingface.co/blog/opinion-classification-with-kili) using RoBERTa.\n- A notebook on how to [finetune RoBERTa for sentiment analysis](https://colab.research.google.com/github/DhavalTaunk08/NLP_scripts/blob/master/sentiment_analysis_using_roberta.ipynb). \ud83c\udf0e\n- [`RobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification.ipynb).\n- [`TFRobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification-tf.ipynb)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 50b1bcaa-256f-4bcd-83d3-a9f67289daf2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"50b1bcaa-256f-4bcd-83d3-a9f67289daf2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta\", \"title\": \"RoBERTa\", \"tokens\": 3026, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f19435bd-da59-4db2-a96e-d489189cb136\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta\", \"title\": \"RoBERTa\", \"tokens\": 3026, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5600d5c6dca955efe139ace218bf61e3b2034c83533b5923f69810afa2660ed1\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"365a6391-73e6-4a1c-bc53-618b94093636\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta\", \"title\": \"RoBERTa\", \"tokens\": 3026, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5fcf0966c0c2522aa49518d5dec924edefff4264ba581f7f50e961b2fe5c6faa\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"538f3a57-0878-4ff9-a7b8-4eec740b866c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"dff54f4b19bb58959515f781b6e03fca32540dc10c7c42313b3d29bfd90cf894\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3130, \"end_char_idx\": 6308, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f19435bd-da59-4db2-a96e-d489189cb136", "doc_id": "f19435bd-da59-4db2-a96e-d489189cb136", "ref_doc_id": "f19435bd-da59-4db2-a96e-d489189cb136", "chroma:document": "- [CamemBERT](camembert) is a wrapper around RoBERTa. Refer to its model page for usage examples.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with RoBERTa. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- A blog on [Getting Started with Sentiment Analysis on Twitter](https://huggingface.co/blog/sentiment-analysis-twitter) using RoBERTa and the [Inference API](https://huggingface.co/inference-api).\n- A blog on [Opinion Classification with Kili and Hugging Face AutoTrain](https://huggingface.co/blog/opinion-classification-with-kili) using RoBERTa.\n- A notebook on how to [finetune RoBERTa for sentiment analysis](https://colab.research.google.com/github/DhavalTaunk08/NLP_scripts/blob/master/sentiment_analysis_using_roberta.ipynb). \ud83c\udf0e\n- [`RobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification.ipynb).\n- [`TFRobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification-tf.ipynb).\n- [`FlaxRobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification_flax.ipynb).\n- [Text classification task guide](../tasks/sequence_classification)\n\n<PipelineTag pipeline=\"token-classification\"/>\n\n- [`RobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification.ipynb).\n- [`TFRobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification-tf.ipynb).\n- [`FlaxRobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/token-classification).\n- [Token classification](https://huggingface.co/course/chapter7/2?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Token classification task guide](../tasks/token_classification)\n\n<PipelineTag pipeline=\"fill-mask\"/>\n\n- A blog on [How to train a new language model from scratch using Transformers and Tokenizers](https://huggingface.co/blog/how-to-train) with RoBERTa."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 538f3a57-0878-4ff9-a7b8-4eec740b866c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"538f3a57-0878-4ff9-a7b8-4eec740b866c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta\", \"title\": \"RoBERTa\", \"tokens\": 3026, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f19435bd-da59-4db2-a96e-d489189cb136\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta\", \"title\": \"RoBERTa\", \"tokens\": 3026, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5600d5c6dca955efe139ace218bf61e3b2034c83533b5923f69810afa2660ed1\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"50b1bcaa-256f-4bcd-83d3-a9f67289daf2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta\", \"title\": \"RoBERTa\", \"tokens\": 3026, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1c8edd7cd0038c9ff98768bf24ce02978a963097e08e9033a1973f66eea6b5ca\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0bebc9b3-9b7e-4769-8345-ead090e702b8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"778595b00b96a9a20dc44777b666f8f07b8ce664384bef41d5348caabfbd1321\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4761, \"end_char_idx\": 7927, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f19435bd-da59-4db2-a96e-d489189cb136", "doc_id": "f19435bd-da59-4db2-a96e-d489189cb136", "ref_doc_id": "f19435bd-da59-4db2-a96e-d489189cb136", "chroma:document": "- [`FlaxRobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification_flax.ipynb).\n- [Text classification task guide](../tasks/sequence_classification)\n\n<PipelineTag pipeline=\"token-classification\"/>\n\n- [`RobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification.ipynb).\n- [`TFRobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification-tf.ipynb).\n- [`FlaxRobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/token-classification).\n- [Token classification](https://huggingface.co/course/chapter7/2?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Token classification task guide](../tasks/token_classification)\n\n<PipelineTag pipeline=\"fill-mask\"/>\n\n- A blog on [How to train a new language model from scratch using Transformers and Tokenizers](https://huggingface.co/blog/how-to-train) with RoBERTa.\n- [`RobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#robertabertdistilbert-and-masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n- [`TFRobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_mlmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- [`FlaxRobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).\n- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- A blog on [Accelerated Inference with Optimum and Transformers Pipelines](https://huggingface.co/blog/optimum-inference) with RoBERTa for question answering.\n- [`RobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 0bebc9b3-9b7e-4769-8345-ead090e702b8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0bebc9b3-9b7e-4769-8345-ead090e702b8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta\", \"title\": \"RoBERTa\", \"tokens\": 3026, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f19435bd-da59-4db2-a96e-d489189cb136\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta\", \"title\": \"RoBERTa\", \"tokens\": 3026, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5600d5c6dca955efe139ace218bf61e3b2034c83533b5923f69810afa2660ed1\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"538f3a57-0878-4ff9-a7b8-4eec740b866c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta\", \"title\": \"RoBERTa\", \"tokens\": 3026, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e4ee4ab04e6d3ac9e114bfae173a82207e06a76022c96955f213bfba316f3d50\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e5089f9d-e331-49f2-a178-0f7ea5baa6f2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d3f3a2e5f89a5f00130aaf662421b136792d37673c33ead1919633573853632d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6636, \"end_char_idx\": 9144, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f19435bd-da59-4db2-a96e-d489189cb136", "doc_id": "f19435bd-da59-4db2-a96e-d489189cb136", "ref_doc_id": "f19435bd-da59-4db2-a96e-d489189cb136", "chroma:document": "- [`TFRobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_mlmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- [`FlaxRobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).\n- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- A blog on [Accelerated Inference with Optimum and Transformers Pipelines](https://huggingface.co/blog/optimum-inference) with RoBERTa for question answering.\n- [`RobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).\n- [`TFRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).\n- [`FlaxRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering).\n- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Question answering task guide](../tasks/question_answering)\n\n**Multiple choice**\n- [`RobertaForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb).\n- [`TFRobertaForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice-tf.ipynb)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: e5089f9d-e331-49f2-a178-0f7ea5baa6f2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e5089f9d-e331-49f2-a178-0f7ea5baa6f2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta\", \"title\": \"RoBERTa\", \"tokens\": 3026, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f19435bd-da59-4db2-a96e-d489189cb136\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta\", \"title\": \"RoBERTa\", \"tokens\": 3026, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5600d5c6dca955efe139ace218bf61e3b2034c83533b5923f69810afa2660ed1\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0bebc9b3-9b7e-4769-8345-ead090e702b8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta\", \"title\": \"RoBERTa\", \"tokens\": 3026, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d170bbc4843d16505b2b2d462c8b143ddb0e41eff1604afe9b61b88f199bf288\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7641, \"end_char_idx\": 11330, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f19435bd-da59-4db2-a96e-d489189cb136", "doc_id": "f19435bd-da59-4db2-a96e-d489189cb136", "ref_doc_id": "f19435bd-da59-4db2-a96e-d489189cb136", "chroma:document": "- [`RobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).\n- [`TFRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).\n- [`FlaxRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering).\n- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Question answering task guide](../tasks/question_answering)\n\n**Multiple choice**\n- [`RobertaForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb).\n- [`TFRobertaForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice-tf.ipynb).\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## RobertaConfig\n\n[[autodoc]] RobertaConfig\n\n## RobertaTokenizer\n\n[[autodoc]] RobertaTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## RobertaTokenizerFast\n\n[[autodoc]] RobertaTokenizerFast\n    - build_inputs_with_special_tokens\n\n<frameworkcontent>\n<pt>\n\n## RobertaModel\n\n[[autodoc]] RobertaModel\n    - forward\n\n## RobertaForCausalLM\n\n[[autodoc]] RobertaForCausalLM\n    - forward\n\n## RobertaForMaskedLM\n\n[[autodoc]] RobertaForMaskedLM\n    - forward\n\n## RobertaForSequenceClassification\n\n[[autodoc]] RobertaForSequenceClassification\n    - forward\n\n## RobertaForMultipleChoice\n\n[[autodoc]] RobertaForMultipleChoice\n    - forward\n\n## RobertaForTokenClassification\n\n[[autodoc]] RobertaForTokenClassification\n    - forward\n\n## RobertaForQuestionAnswering\n\n[[autodoc]] RobertaForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFRobertaModel\n\n[[autodoc]] TFRobertaModel\n    - call\n\n## TFRobertaForCausalLM\n\n[[autodoc]] TFRobertaForCausalLM\n    - call\n\n## TFRobertaForMaskedLM\n\n[[autodoc]] TFRobertaForMaskedLM\n    - call\n\n## TFRobertaForSequenceClassification\n\n[[autodoc]] TFRobertaForSequenceClassification\n    - call\n\n## TFRobertaForMultipleChoice\n\n[[autodoc]] TFRobertaForMultipleChoice\n    - call\n\n## TFRobertaForTokenClassification\n\n[[autodoc]] TFRobertaForTokenClassification\n    - call\n\n## TFRobertaForQuestionAnswering\n\n[[autodoc]] TFRobertaForQuestionAnswering\n    - call\n\n</tf>\n<jax>\n\n## FlaxRobertaModel\n\n[[autodoc]] FlaxRobertaModel\n    - __call__\n\n## FlaxRobertaForCausalLM\n\n[[autodoc]] FlaxRobertaForCausalLM\n    - __call__\n\n## FlaxRobertaForMaskedLM\n\n[[autodoc]] FlaxRobertaForMaskedLM\n    - __call__\n\n## FlaxRobertaForSequenceClassification\n\n[[autodoc]] FlaxRobertaForSequenceClassification\n    - __call__\n\n## FlaxRobertaForMultipleChoice\n\n[[autodoc]] FlaxRobertaForMultipleChoice\n    - __call__\n\n## FlaxRobertaForTokenClassification\n\n[[autodoc]] FlaxRobertaForTokenClassification\n    - __call__\n\n## FlaxRobertaForQuestionAnswering\n\n[[autodoc]] FlaxRobertaForQuestionAnswering\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 071dcae3-e255-4f85-b187-050686447244
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/swinv2", "title": "Swin Transformer V2", "tokens": 894, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"071dcae3-e255-4f85-b187-050686447244\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/swinv2\", \"title\": \"Swin Transformer V2\", \"tokens\": 894, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4f6ed90e-b24c-4fa1-ba43-bf7a432832f9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/swinv2\", \"title\": \"Swin Transformer V2\", \"tokens\": 894, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6cdd98d6b1a403b4f727ab32240655dd8b9094f2947e7a6d4566035fd54ffb19\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3253, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4f6ed90e-b24c-4fa1-ba43-bf7a432832f9", "doc_id": "4f6ed90e-b24c-4fa1-ba43-bf7a432832f9", "ref_doc_id": "4f6ed90e-b24c-4fa1-ba43-bf7a432832f9", "chroma:document": "# Swin Transformer V2\n\n## Overview\n\nThe Swin Transformer V2 model was proposed in [Swin Transformer V2: Scaling Up Capacity and Resolution](https://arxiv.org/abs/2111.09883) by Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, Baining Guo.\n\nThe abstract from the paper is the following:\n\n*Large-scale NLP models have been shown to significantly improve the performance on language tasks with no signs of saturation. They also demonstrate amazing few-shot capabilities like that of human beings. This paper aims to explore large-scale models in computer vision. We tackle three major issues in training and application of large vision models, including training instability, resolution gaps between pre-training and fine-tuning, and hunger on labelled data. Three main techniques are proposed: 1) a residual-post-norm method combined with cosine attention to improve training stability; 2) A log-spaced continuous position bias method to effectively transfer models pre-trained using low-resolution images to downstream tasks with high-resolution inputs; 3) A self-supervised pre-training method, SimMIM, to reduce the needs of vast labeled images. Through these techniques, this paper successfully trained a 3 billion-parameter Swin Transformer V2 model, which is the largest dense vision model to date, and makes it capable of training with images of up to 1,536\u00d71,536 resolution. It set new performance records on 4 representative vision tasks, including ImageNet-V2 image classification, COCO object detection, ADE20K semantic segmentation, and Kinetics-400 video action classification. Also note our training is much more efficient than that in Google's billion-level visual models, which consumes 40 times less labelled data and 40 times less training time.*\n\nThis model was contributed by [nandwalritik](https://huggingface.co/nandwalritik).\nThe original code can be found [here](https://github.com/microsoft/Swin-Transformer).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Swin Transformer v2.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`Swinv2ForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nBesides that:\n\n- [`Swinv2ForMaskedImageModeling`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## Swinv2Config\n\n[[autodoc]] Swinv2Config\n\n## Swinv2Model\n\n[[autodoc]] Swinv2Model\n    - forward\n\n## Swinv2ForMaskedImageModeling\n\n[[autodoc]] Swinv2ForMaskedImageModeling\n    - forward\n\n## Swinv2ForImageClassification\n\n[[autodoc]] transformers.Swinv2ForImageClassification\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 23fc9a2a-8621-43eb-92a3-745d8cfbf8bb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/led", "title": "LED", "tokens": 1332, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"23fc9a2a-8621-43eb-92a3-745d8cfbf8bb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/led\", \"title\": \"LED\", \"tokens\": 1332, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d7309d6d-bb78-4fbe-a1e7-6804847f4125\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/led\", \"title\": \"LED\", \"tokens\": 1332, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6f16b660fabbfc08e155795602a8ab7f5609b196ea8bf201a6a4026553c0d7c3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9de0aaa6-430d-47c9-8300-0e7fa1d5f6d2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4f2a5129568f2763e46b6b15dad6e397bc22d863f8988e1eacbfaf6ce921818b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3070, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d7309d6d-bb78-4fbe-a1e7-6804847f4125", "doc_id": "d7309d6d-bb78-4fbe-a1e7-6804847f4125", "ref_doc_id": "d7309d6d-bb78-4fbe-a1e7-6804847f4125", "chroma:document": "# LED\n\n## Overview\n\nThe LED model was proposed in [Longformer: The Long-Document Transformer](https://arxiv.org/abs/2004.05150) by Iz\nBeltagy, Matthew E. Peters, Arman Cohan.\n\nThe abstract from the paper is the following:\n\n*Transformer-based models are unable to process long sequences due to their self-attention operation, which scales\nquadratically with the sequence length. To address this limitation, we introduce the Longformer with an attention\nmechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or\nlonger. Longformer's attention mechanism is a drop-in replacement for the standard self-attention and combines a local\nwindowed attention with a task motivated global attention. Following prior work on long-sequence transformers, we\nevaluate Longformer on character-level language modeling and achieve state-of-the-art results on text8 and enwik8. In\ncontrast to most prior work, we also pretrain Longformer and finetune it on a variety of downstream tasks. Our\npretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on\nWikiHop and TriviaQA. We finally introduce the Longformer-Encoder-Decoder (LED), a Longformer variant for supporting\nlong document generative sequence-to-sequence tasks, and demonstrate its effectiveness on the arXiv summarization\ndataset.*\n\n## Usage tips\n\n- [`LEDForConditionalGeneration`] is an extension of\n  [`BartForConditionalGeneration`] exchanging the traditional *self-attention* layer with\n  *Longformer*'s *chunked self-attention* layer. [`LEDTokenizer`] is an alias of\n  [`BartTokenizer`].\n- LED works very well on long-range *sequence-to-sequence* tasks where the `input_ids` largely exceed a length of\n  1024 tokens.\n- LED pads the `input_ids` to be a multiple of `config.attention_window` if required. Therefore a small speed-up is\n  gained, when [`LEDTokenizer`] is used with the `pad_to_multiple_of` argument.\n- LED makes use of *global attention* by means of the `global_attention_mask` (see\n  [`LongformerModel`]). For summarization, it is advised to put *global attention* only on the first\n  `<s>` token. For question answering, it is advised to put *global attention* on all tokens of the question.\n- To fine-tune LED on all 16384, *gradient checkpointing* can be enabled in case training leads to out-of-memory (OOM)\n  errors. This can be done by executing `model.gradient_checkpointing_enable()`. \n Moreover, the `use_cache=False`\n  flag can be used to disable the caching mechanism to save memory.\n- LED is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than\n  the left.\n\nThis model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten).\n\n## Resources\n\n- [A notebook showing how to evaluate LED](https://colab.research.google.com/drive/12INTTR6n64TzS4RrXZxMSXfrOd9Xzamo?usp=sharing).\n- [A notebook showing how to fine-tune LED](https://colab.research.google.com/drive/12LjJazBl7Gam0XBPy_y0CTOJZeZ34c2v?usp=sharing)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 9de0aaa6-430d-47c9-8300-0e7fa1d5f6d2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/led", "title": "LED", "tokens": 1332, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9de0aaa6-430d-47c9-8300-0e7fa1d5f6d2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/led\", \"title\": \"LED\", \"tokens\": 1332, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d7309d6d-bb78-4fbe-a1e7-6804847f4125\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/led\", \"title\": \"LED\", \"tokens\": 1332, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6f16b660fabbfc08e155795602a8ab7f5609b196ea8bf201a6a4026553c0d7c3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"23fc9a2a-8621-43eb-92a3-745d8cfbf8bb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/led\", \"title\": \"LED\", \"tokens\": 1332, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d8f4db522a042a98ba9f0700d1e9f7c98553a052f1a26f5be81c0ec6fa890f22\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1597, \"end_char_idx\": 4619, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d7309d6d-bb78-4fbe-a1e7-6804847f4125", "doc_id": "d7309d6d-bb78-4fbe-a1e7-6804847f4125", "ref_doc_id": "d7309d6d-bb78-4fbe-a1e7-6804847f4125", "chroma:document": "[`LEDTokenizer`] is an alias of\n  [`BartTokenizer`].\n- LED works very well on long-range *sequence-to-sequence* tasks where the `input_ids` largely exceed a length of\n  1024 tokens.\n- LED pads the `input_ids` to be a multiple of `config.attention_window` if required. Therefore a small speed-up is\n  gained, when [`LEDTokenizer`] is used with the `pad_to_multiple_of` argument.\n- LED makes use of *global attention* by means of the `global_attention_mask` (see\n  [`LongformerModel`]). For summarization, it is advised to put *global attention* only on the first\n  `<s>` token. For question answering, it is advised to put *global attention* on all tokens of the question.\n- To fine-tune LED on all 16384, *gradient checkpointing* can be enabled in case training leads to out-of-memory (OOM)\n  errors. This can be done by executing `model.gradient_checkpointing_enable()`. \n Moreover, the `use_cache=False`\n  flag can be used to disable the caching mechanism to save memory.\n- LED is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than\n  the left.\n\nThis model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten).\n\n## Resources\n\n- [A notebook showing how to evaluate LED](https://colab.research.google.com/drive/12INTTR6n64TzS4RrXZxMSXfrOd9Xzamo?usp=sharing).\n- [A notebook showing how to fine-tune LED](https://colab.research.google.com/drive/12LjJazBl7Gam0XBPy_y0CTOJZeZ34c2v?usp=sharing).\n- [Text classification task guide](../tasks/sequence_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n## LEDConfig\n\n[[autodoc]] LEDConfig\n\n## LEDTokenizer\n\n[[autodoc]] LEDTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## LEDTokenizerFast\n\n[[autodoc]] LEDTokenizerFast\n\n## LED specific outputs\n\n[[autodoc]] models.led.modeling_led.LEDEncoderBaseModelOutput\n\n[[autodoc]] models.led.modeling_led.LEDSeq2SeqModelOutput\n\n[[autodoc]] models.led.modeling_led.LEDSeq2SeqLMOutput\n\n[[autodoc]] models.led.modeling_led.LEDSeq2SeqSequenceClassifierOutput\n\n[[autodoc]] models.led.modeling_led.LEDSeq2SeqQuestionAnsweringModelOutput\n\n[[autodoc]] models.led.modeling_tf_led.TFLEDEncoderBaseModelOutput\n\n[[autodoc]] models.led.modeling_tf_led.TFLEDSeq2SeqModelOutput\n\n[[autodoc]] models.led.modeling_tf_led.TFLEDSeq2SeqLMOutput\n\n<frameworkcontent>\n<pt>\n\n## LEDModel\n\n[[autodoc]] LEDModel\n    - forward\n\n## LEDForConditionalGeneration\n\n[[autodoc]] LEDForConditionalGeneration\n    - forward\n\n## LEDForSequenceClassification\n\n[[autodoc]] LEDForSequenceClassification\n    - forward\n\n## LEDForQuestionAnswering\n\n[[autodoc]] LEDForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFLEDModel\n\n[[autodoc]] TFLEDModel\n    - call\n\n## TFLEDForConditionalGeneration\n\n[[autodoc]] TFLEDForConditionalGeneration\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: b0e9b55f-fcf4-4eda-9b08-5d5f90c47b7a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/phi3", "title": "Phi-3", "tokens": 1045, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b0e9b55f-fcf4-4eda-9b08-5d5f90c47b7a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi3\", \"title\": \"Phi-3\", \"tokens\": 1045, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b08a2d86-fccb-4ae5-ba8c-a2b4bcb0613a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi3\", \"title\": \"Phi-3\", \"tokens\": 1045, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bd10ad10af0e8b050bfea8d3953598adc79825f145963f32a162c4946178a2bc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e2575cd6-1e63-4fcb-bd0e-0741d807d612\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3f0fb0bf75e05b5df4469bf46b388c65e2e7dd434605d83756abf6a6b9c48d61\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3023, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b08a2d86-fccb-4ae5-ba8c-a2b4bcb0613a", "doc_id": "b08a2d86-fccb-4ae5-ba8c-a2b4bcb0613a", "ref_doc_id": "b08a2d86-fccb-4ae5-ba8c-a2b4bcb0613a", "chroma:document": "# Phi-3\n\n## Overview\n\nThe Phi-3 model was proposed in [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://arxiv.org/abs/2404.14219) by Microsoft.\n\n### Summary\n\nThe abstract from the Phi-3 paper is the following:\n\nWe introduce phi-3-mini, a 3.8 billion parameter language model trained on 3.3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and GPT-3.5 (e.g., phi-3-mini achieves 69% on MMLU and 8.38 on MT-bench), despite being small enough to be deployed on a phone. The innovation lies entirely in our dataset for training, a scaled-up version of the one used for phi-2, composed of heavily filtered web data and synthetic data. The model is also further aligned for robustness, safety, and chat format. We also provide some initial parameter-scaling results with a 7B and 14B models trained for 4.8T tokens, called phi-3-small and phi-3-medium, both significantly more capable than phi-3-mini (e.g., respectively 75% and 78% on MMLU, and 8.7 and 8.9 on MT-bench).\n\nThe original code for Phi-3 can be found [here](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct).\n\n## Usage tips\n\n- This model is very similar to `Llama` with the main difference of [`Phi3SuScaledRotaryEmbedding`] and [`Phi3YarnScaledRotaryEmbedding`], where they are used to extend the context of the rotary embeddings. The query, key and values are fused, and the MLP's up and gate projection layers are also fused.\n- The tokenizer used for this model is identical to the [`LlamaTokenizer`], with the exception of additional tokens.\n\n## How to use Phi-3\n\n<Tip warning={true}>\n\nPhi-3 has been integrated in the development version (4.40.0.dev) of `transformers`. Until the official version is released through `pip`, ensure that you are doing one of the following:\n\n* When loading the model, ensure that `trust_remote_code=True` is passed as an argument of the `from_pretrained()` function.\n\n* Update your local `transformers` to the development version: `pip uninstall -y transformers && pip install git+https://github.com/huggingface/transformers`. The previous command is an alternative to cloning and installing from the source.\n\n</Tip>\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"microsoft/Phi-3-mini-4k-instruct\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"microsoft/Phi-3-mini-4k-instruct\")\n\n>>> messages = [{\"role\": \"user\", \"content\": \"Can you provide ways to eat combinations of bananas and dragonfruits?\"}]\n>>> inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors=\"pt\")\n\n>>> outputs = model.generate(inputs, max_new_tokens=32)\n>>> text = tokenizer.batch_decode(outputs)[0]\n>>> print(text)\n<s><|user|> \nCan you provide ways to eat combinations of bananas and dragonfruits?<|end|> \n<|assistant|> \nCertainly! Bananas and dragonfruits can be combined in various delicious ways."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: e2575cd6-1e63-4fcb-bd0e-0741d807d612
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/phi3", "title": "Phi-3", "tokens": 1045, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e2575cd6-1e63-4fcb-bd0e-0741d807d612\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi3\", \"title\": \"Phi-3\", \"tokens\": 1045, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b08a2d86-fccb-4ae5-ba8c-a2b4bcb0613a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi3\", \"title\": \"Phi-3\", \"tokens\": 1045, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bd10ad10af0e8b050bfea8d3953598adc79825f145963f32a162c4946178a2bc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b0e9b55f-fcf4-4eda-9b08-5d5f90c47b7a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi3\", \"title\": \"Phi-3\", \"tokens\": 1045, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"23f8da943fcf1bf639bae95c6dd74a7f324beac8f61e3b19049bb70c83c8913f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1434, \"end_char_idx\": 3482, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b08a2d86-fccb-4ae5-ba8c-a2b4bcb0613a", "doc_id": "b08a2d86-fccb-4ae5-ba8c-a2b4bcb0613a", "ref_doc_id": "b08a2d86-fccb-4ae5-ba8c-a2b4bcb0613a", "chroma:document": "The query, key and values are fused, and the MLP's up and gate projection layers are also fused.\n- The tokenizer used for this model is identical to the [`LlamaTokenizer`], with the exception of additional tokens.\n\n## How to use Phi-3\n\n<Tip warning={true}>\n\nPhi-3 has been integrated in the development version (4.40.0.dev) of `transformers`. Until the official version is released through `pip`, ensure that you are doing one of the following:\n\n* When loading the model, ensure that `trust_remote_code=True` is passed as an argument of the `from_pretrained()` function.\n\n* Update your local `transformers` to the development version: `pip uninstall -y transformers && pip install git+https://github.com/huggingface/transformers`. The previous command is an alternative to cloning and installing from the source.\n\n</Tip>\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"microsoft/Phi-3-mini-4k-instruct\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"microsoft/Phi-3-mini-4k-instruct\")\n\n>>> messages = [{\"role\": \"user\", \"content\": \"Can you provide ways to eat combinations of bananas and dragonfruits?\"}]\n>>> inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors=\"pt\")\n\n>>> outputs = model.generate(inputs, max_new_tokens=32)\n>>> text = tokenizer.batch_decode(outputs)[0]\n>>> print(text)\n<s><|user|> \nCan you provide ways to eat combinations of bananas and dragonfruits?<|end|> \n<|assistant|> \nCertainly! Bananas and dragonfruits can be combined in various delicious ways. Here are some ideas for eating combinations of bananas and\n```\n\n## Phi3Config\n\n[[autodoc]] Phi3Config\n\n<frameworkcontent>\n<pt>\n\n## Phi3Model\n\n[[autodoc]] Phi3Model\n    - forward\n\n## Phi3ForCausalLM\n\n[[autodoc]] Phi3ForCausalLM\n    - forward\n    - generate\n\n## Phi3ForSequenceClassification\n\n[[autodoc]] Phi3ForSequenceClassification\n    - forward\n\n## Phi3ForTokenClassification\n\n[[autodoc]] Phi3ForTokenClassification\n    - forward\n\n</pt>\n</frameworkcontent>"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: cd357d90-8776-463e-aceb-140aaf0cce2b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/upernet", "title": "UPerNet", "tokens": 993, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"cd357d90-8776-463e-aceb-140aaf0cce2b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/upernet\", \"title\": \"UPerNet\", \"tokens\": 993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"26da89fb-5541-451c-9a26-1416c49d435b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/upernet\", \"title\": \"UPerNet\", \"tokens\": 993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cd643830d234287828240de9e7b16ccfb20df55bcc067de7290ce5dac9ba6b05\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6d2356d8-b2bd-47f0-aa3f-5699452b16c6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3468b4d14c4f66f7d2e223f78298ebb162aeca0c46267819de0f7399ae3dcf4d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3143, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "26da89fb-5541-451c-9a26-1416c49d435b", "doc_id": "26da89fb-5541-451c-9a26-1416c49d435b", "ref_doc_id": "26da89fb-5541-451c-9a26-1416c49d435b", "chroma:document": "# UPerNet\n\n## Overview\n\nThe UPerNet model was proposed in [Unified Perceptual Parsing for Scene Understanding](https://arxiv.org/abs/1807.10221)\nby Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, Jian Sun. UPerNet is a general framework to effectively segment\na wide range of concepts from images, leveraging any vision backbone like [ConvNeXt](convnext) or [Swin](swin).\n\nThe abstract from the paper is the following:\n\n*Humans recognize the visual world at multiple levels: we effortlessly categorize scenes and detect objects inside, while also identifying the textures and surfaces of the objects along with their different compositional parts. In this paper, we study a new task called Unified Perceptual Parsing, which requires the machine vision systems to recognize as many visual concepts as possible from a given image. A multi-task framework called UPerNet and a training strategy are developed to learn from heterogeneous image annotations. We benchmark our framework on Unified Perceptual Parsing and show that it is able to effectively segment a wide range of concepts from images. The trained networks are further applied to discover visual knowledge in natural scenes.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/upernet_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> UPerNet framework. Taken from the <a href=\"https://arxiv.org/abs/1807.10221\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code is based on OpenMMLab's mmsegmentation [here](https://github.com/open-mmlab/mmsegmentation/blob/master/mmseg/models/decode_heads/uper_head.py).\n\n## Usage examples\n\nUPerNet is a general framework for semantic segmentation. It can be used with any vision backbone, like so:\n\n```py\nfrom transformers import SwinConfig, UperNetConfig, UperNetForSemanticSegmentation\n\nbackbone_config = SwinConfig(out_features=[\"stage1\", \"stage2\", \"stage3\", \"stage4\"])\n\nconfig = UperNetConfig(backbone_config=backbone_config)\nmodel = UperNetForSemanticSegmentation(config)\n```\n\nTo use another vision backbone, like [ConvNeXt](convnext), simply instantiate the model with the appropriate backbone:\n\n```py\nfrom transformers import ConvNextConfig, UperNetConfig, UperNetForSemanticSegmentation\n\nbackbone_config = ConvNextConfig(out_features=[\"stage1\", \"stage2\", \"stage3\", \"stage4\"])\n\nconfig = UperNetConfig(backbone_config=backbone_config)\nmodel = UperNetForSemanticSegmentation(config)\n```\n\nNote that this will randomly initialize all the weights of the model.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with UPerNet.\n\n- Demo notebooks for UPerNet can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/UPerNet).\n- [`UperNetForSemanticSegmentation`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/semantic-segmentation) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/semantic_segmentation.ipynb)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 6d2356d8-b2bd-47f0-aa3f-5699452b16c6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/upernet", "title": "UPerNet", "tokens": 993, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6d2356d8-b2bd-47f0-aa3f-5699452b16c6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/upernet\", \"title\": \"UPerNet\", \"tokens\": 993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"26da89fb-5541-451c-9a26-1416c49d435b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/upernet\", \"title\": \"UPerNet\", \"tokens\": 993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cd643830d234287828240de9e7b16ccfb20df55bcc067de7290ce5dac9ba6b05\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cd357d90-8776-463e-aceb-140aaf0cce2b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/upernet\", \"title\": \"UPerNet\", \"tokens\": 993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"38138b6f220a5f7ad2bf742f55a19813bb1c080fdc9850e3b7342f260765aa71\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1542, \"end_char_idx\": 3588, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "26da89fb-5541-451c-9a26-1416c49d435b", "doc_id": "26da89fb-5541-451c-9a26-1416c49d435b", "ref_doc_id": "26da89fb-5541-451c-9a26-1416c49d435b", "chroma:document": "The original code is based on OpenMMLab's mmsegmentation [here](https://github.com/open-mmlab/mmsegmentation/blob/master/mmseg/models/decode_heads/uper_head.py).\n\n## Usage examples\n\nUPerNet is a general framework for semantic segmentation. It can be used with any vision backbone, like so:\n\n```py\nfrom transformers import SwinConfig, UperNetConfig, UperNetForSemanticSegmentation\n\nbackbone_config = SwinConfig(out_features=[\"stage1\", \"stage2\", \"stage3\", \"stage4\"])\n\nconfig = UperNetConfig(backbone_config=backbone_config)\nmodel = UperNetForSemanticSegmentation(config)\n```\n\nTo use another vision backbone, like [ConvNeXt](convnext), simply instantiate the model with the appropriate backbone:\n\n```py\nfrom transformers import ConvNextConfig, UperNetConfig, UperNetForSemanticSegmentation\n\nbackbone_config = ConvNextConfig(out_features=[\"stage1\", \"stage2\", \"stage3\", \"stage4\"])\n\nconfig = UperNetConfig(backbone_config=backbone_config)\nmodel = UperNetForSemanticSegmentation(config)\n```\n\nNote that this will randomly initialize all the weights of the model.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with UPerNet.\n\n- Demo notebooks for UPerNet can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/UPerNet).\n- [`UperNetForSemanticSegmentation`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/semantic-segmentation) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/semantic_segmentation.ipynb).\n- See also: [Semantic segmentation task guide](../tasks/semantic_segmentation)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## UperNetConfig\n\n[[autodoc]] UperNetConfig\n\n## UperNetForSemanticSegmentation\n\n[[autodoc]] UperNetForSemanticSegmentation\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 12db5492-1028-4127-a735-a139a063c096
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/blenderbot", "title": "Blenderbot", "tokens": 1155, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"12db5492-1028-4127-a735-a139a063c096\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blenderbot\", \"title\": \"Blenderbot\", \"tokens\": 1155, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3ca49c4f-2548-485b-a3b0-1b1c33ede8ec\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blenderbot\", \"title\": \"Blenderbot\", \"tokens\": 1155, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"36b1057c151a6bc23415ea0a42657e3f6026eb69a61e5c726366c6a125055b90\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c0f1e826-8bcf-47b4-9b4f-cf5ef9f00a4b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1671e10b84b6b2ba565fc0072b3d01791f6449488c59a86f81a1d80eef670c47\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3294, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3ca49c4f-2548-485b-a3b0-1b1c33ede8ec", "doc_id": "3ca49c4f-2548-485b-a3b0-1b1c33ede8ec", "ref_doc_id": "3ca49c4f-2548-485b-a3b0-1b1c33ede8ec", "chroma:document": "# Blenderbot\n\n## Overview\n\nThe Blender chatbot model was proposed in [Recipes for building an open-domain chatbot](https://arxiv.org/pdf/2004.13637.pdf) Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu,\nJing Xu, Myle Ott, Kurt Shuster, Eric M. Smith, Y-Lan Boureau, Jason Weston on 30 Apr 2020.\n\nThe abstract of the paper is the following:\n\n*Building open-domain chatbots is a challenging area for machine learning research. While prior work has shown that\nscaling neural models in the number of parameters and the size of the data they are trained on gives improved results,\nwe show that other ingredients are important for a high-performing chatbot. Good conversation requires a number of\nskills that an expert conversationalist blends in a seamless way: providing engaging talking points and listening to\ntheir partners, and displaying knowledge, empathy and personality appropriately, while maintaining a consistent\npersona. We show that large scale models can learn these skills when given appropriate training data and choice of\ngeneration strategy. We build variants of these recipes with 90M, 2.7B and 9.4B parameter models, and make our models\nand code publicly available. Human evaluations show our best models are superior to existing approaches in multi-turn\ndialogue in terms of engagingness and humanness measurements. We then discuss the limitations of this work by analyzing\nfailure cases of our models.*\n\nThis model was contributed by [sshleifer](https://huggingface.co/sshleifer). The authors' code can be found [here](https://github.com/facebookresearch/ParlAI) .\n\n## Usage tips and example\n\nBlenderbot is a model with absolute position embeddings so it's usually advised to pad the inputs on the right \nrather than the left.\n\nAn example:\n\n```python\n>>> from transformers import BlenderbotTokenizer, BlenderbotForConditionalGeneration\n\n>>> mname = \"facebook/blenderbot-400M-distill\"\n>>> model = BlenderbotForConditionalGeneration.from_pretrained(mname)\n>>> tokenizer = BlenderbotTokenizer.from_pretrained(mname)\n>>> UTTERANCE = \"My friends are cool but they eat too many carbs.\"\n>>> inputs = tokenizer([UTTERANCE], return_tensors=\"pt\")\n>>> reply_ids = model.generate(**inputs)\n>>> print(tokenizer.batch_decode(reply_ids))\n[\"<s> That's unfortunate. Are they trying to lose weight or are they just trying to be healthier?</s>\"]\n```\n\n## Implementation Notes\n\n- Blenderbot uses a standard [seq2seq model transformer](https://arxiv.org/pdf/1706.03762.pdf) based architecture.\n- Available checkpoints can be found in the [model hub](https://huggingface.co/models?search=blenderbot).\n- This is the *default* Blenderbot model class. However, some smaller checkpoints, such as\n  `facebook/blenderbot_small_90M`, have a different architecture and consequently should be used with\n  [BlenderbotSmall](blenderbot-small).\n\n  \n## Resources\n\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n## BlenderbotConfig\n\n[[autodoc]] BlenderbotConfig\n\n## BlenderbotTokenizer\n\n[[autodoc]] BlenderbotTokenizer\n    - build_inputs_with_special_tokens\n\n## BlenderbotTokenizerFast\n\n[[autodoc]] BlenderbotTokenizerFast\n    - build_inputs_with_special_tokens"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: c0f1e826-8bcf-47b4-9b4f-cf5ef9f00a4b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/blenderbot", "title": "Blenderbot", "tokens": 1155, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c0f1e826-8bcf-47b4-9b4f-cf5ef9f00a4b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blenderbot\", \"title\": \"Blenderbot\", \"tokens\": 1155, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3ca49c4f-2548-485b-a3b0-1b1c33ede8ec\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blenderbot\", \"title\": \"Blenderbot\", \"tokens\": 1155, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"36b1057c151a6bc23415ea0a42657e3f6026eb69a61e5c726366c6a125055b90\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"12db5492-1028-4127-a735-a139a063c096\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blenderbot\", \"title\": \"Blenderbot\", \"tokens\": 1155, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"086afa922d1464c5c9f1c1a4180fe3020cf6b8ead67cd6a88a0969880d5d6cff\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3297, \"end_char_idx\": 4181, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3ca49c4f-2548-485b-a3b0-1b1c33ede8ec", "doc_id": "3ca49c4f-2548-485b-a3b0-1b1c33ede8ec", "ref_doc_id": "3ca49c4f-2548-485b-a3b0-1b1c33ede8ec", "chroma:document": "<frameworkcontent>\n<pt>\n\n## BlenderbotModel\n\nSee [`~transformers.BartModel`] for arguments to *forward* and *generate*\n\n[[autodoc]] BlenderbotModel\n    - forward\n\n## BlenderbotForConditionalGeneration\n\nSee [`~transformers.BartForConditionalGeneration`] for arguments to *forward* and *generate*\n\n[[autodoc]] BlenderbotForConditionalGeneration\n    - forward\n\n## BlenderbotForCausalLM\n\n[[autodoc]] BlenderbotForCausalLM\n    - forward\n\n</pt>\n<tf>\n\n## TFBlenderbotModel\n\n[[autodoc]] TFBlenderbotModel\n    - call\n\n## TFBlenderbotForConditionalGeneration\n\n[[autodoc]] TFBlenderbotForConditionalGeneration\n    - call\n\n</tf>\n<jax>\n\n## FlaxBlenderbotModel\n\n[[autodoc]] FlaxBlenderbotModel\n    - __call__\n    - encode\n    - decode\n\n## FlaxBlenderbotForConditionalGeneration\n\n[[autodoc]] FlaxBlenderbotForConditionalGeneration\n    - __call__\n    - encode\n    - decode\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: c9283ffe-deae-493d-a0f0-5a2ad15ab62f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/splinter", "title": "Splinter", "tokens": 961, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c9283ffe-deae-493d-a0f0-5a2ad15ab62f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/splinter\", \"title\": \"Splinter\", \"tokens\": 961, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"46ba8514-f602-4b2f-b0cd-0830de8369e2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/splinter\", \"title\": \"Splinter\", \"tokens\": 961, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c62477bd3a8359d7de5736bcc1c7e4bc94f775cd16f999207061da2a55290933\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"eb65f290-46d1-4c2b-afb2-fc92da7f27f1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7c86b7e34c683615d4aa424eaea5958ee97a6b3d0998581f868d7d53f1585a1b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2871, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "46ba8514-f602-4b2f-b0cd-0830de8369e2", "doc_id": "46ba8514-f602-4b2f-b0cd-0830de8369e2", "ref_doc_id": "46ba8514-f602-4b2f-b0cd-0830de8369e2", "chroma:document": "# Splinter\n\n## Overview\n\nThe Splinter model was proposed in [Few-Shot Question Answering by Pretraining Span Selection](https://arxiv.org/abs/2101.00438) by Ori Ram, Yuval Kirstain, Jonathan Berant, Amir Globerson, Omer Levy. Splinter\nis an encoder-only transformer (similar to BERT) pretrained using the recurring span selection task on a large corpus\ncomprising Wikipedia and the Toronto Book Corpus.\n\nThe abstract from the paper is the following:\n\nIn several question answering benchmarks, pretrained models have reached human parity through fine-tuning on an order\nof 100,000 annotated questions and answers. We explore the more realistic few-shot setting, where only a few hundred\ntraining examples are available, and observe that standard models perform poorly, highlighting the discrepancy between\ncurrent pretraining objectives and question answering. We propose a new pretraining scheme tailored for question\nanswering: recurring span selection. Given a passage with multiple sets of recurring spans, we mask in each set all\nrecurring spans but one, and ask the model to select the correct span in the passage for each masked span. Masked spans\nare replaced with a special token, viewed as a question representation, that is later used during fine-tuning to select\nthe answer span. The resulting model obtains surprisingly good results on multiple benchmarks (e.g., 72.7 F1 on SQuAD\nwith only 128 training examples), while maintaining competitive performance in the high-resource setting.\n\nThis model was contributed by [yuvalkirstain](https://huggingface.co/yuvalkirstain) and [oriram](https://huggingface.co/oriram). The original code can be found [here](https://github.com/oriram/splinter).\n\n## Usage tips\n\n- Splinter was trained to predict answers spans conditioned on a special [QUESTION] token. These tokens contextualize\n  to question representations which are used to predict the answers. This layer is called QASS, and is the default\n  behaviour in the [`SplinterForQuestionAnswering`] class. Therefore:\n- Use [`SplinterTokenizer`] (rather than [`BertTokenizer`]), as it already\n  contains this special token. Also, its default behavior is to use this token when two sequences are given (for\n  example, in the *run_qa.py* script).\n- If you plan on using Splinter outside *run_qa.py*, please keep in mind the question token - it might be important for\n  the success of your model, especially in a few-shot setting.\n- Please note there are two different checkpoints for each size of Splinter. Both are basically the same, except that\n  one also has the pretrained weights of the QASS layer (*tau/splinter-base-qass* and *tau/splinter-large-qass*) and one\n  doesn't (*tau/splinter-base* and *tau/splinter-large*). This is done to support randomly initializing this layer at\n  fine-tuning, as it is shown to yield better results for some cases in the paper."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: eb65f290-46d1-4c2b-afb2-fc92da7f27f1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/splinter", "title": "Splinter", "tokens": 961, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"eb65f290-46d1-4c2b-afb2-fc92da7f27f1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/splinter\", \"title\": \"Splinter\", \"tokens\": 961, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"46ba8514-f602-4b2f-b0cd-0830de8369e2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/splinter\", \"title\": \"Splinter\", \"tokens\": 961, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c62477bd3a8359d7de5736bcc1c7e4bc94f775cd16f999207061da2a55290933\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c9283ffe-deae-493d-a0f0-5a2ad15ab62f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/splinter\", \"title\": \"Splinter\", \"tokens\": 961, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a51b64a63b5822aa0dd685583af29628804befb10d84db6ab5442ab5a36d7068\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1291, \"end_char_idx\": 3469, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "46ba8514-f602-4b2f-b0cd-0830de8369e2", "doc_id": "46ba8514-f602-4b2f-b0cd-0830de8369e2", "ref_doc_id": "46ba8514-f602-4b2f-b0cd-0830de8369e2", "chroma:document": "The resulting model obtains surprisingly good results on multiple benchmarks (e.g., 72.7 F1 on SQuAD\nwith only 128 training examples), while maintaining competitive performance in the high-resource setting.\n\nThis model was contributed by [yuvalkirstain](https://huggingface.co/yuvalkirstain) and [oriram](https://huggingface.co/oriram). The original code can be found [here](https://github.com/oriram/splinter).\n\n## Usage tips\n\n- Splinter was trained to predict answers spans conditioned on a special [QUESTION] token. These tokens contextualize\n  to question representations which are used to predict the answers. This layer is called QASS, and is the default\n  behaviour in the [`SplinterForQuestionAnswering`] class. Therefore:\n- Use [`SplinterTokenizer`] (rather than [`BertTokenizer`]), as it already\n  contains this special token. Also, its default behavior is to use this token when two sequences are given (for\n  example, in the *run_qa.py* script).\n- If you plan on using Splinter outside *run_qa.py*, please keep in mind the question token - it might be important for\n  the success of your model, especially in a few-shot setting.\n- Please note there are two different checkpoints for each size of Splinter. Both are basically the same, except that\n  one also has the pretrained weights of the QASS layer (*tau/splinter-base-qass* and *tau/splinter-large-qass*) and one\n  doesn't (*tau/splinter-base* and *tau/splinter-large*). This is done to support randomly initializing this layer at\n  fine-tuning, as it is shown to yield better results for some cases in the paper.\n\n## Resources\n\n- [Question answering task guide](../tasks/question-answering)\n\n## SplinterConfig\n\n[[autodoc]] SplinterConfig\n\n## SplinterTokenizer\n\n[[autodoc]] SplinterTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## SplinterTokenizerFast\n\n[[autodoc]] SplinterTokenizerFast\n\n## SplinterModel\n\n[[autodoc]] SplinterModel\n    - forward\n\n## SplinterForQuestionAnswering\n\n[[autodoc]] SplinterForQuestionAnswering\n    - forward\n\n## SplinterForPreTraining\n\n[[autodoc]] SplinterForPreTraining\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 537189bf-44a5-452b-8e84-dbfe66c5da74
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/sew-d", "title": "SEW-D", "tokens": 637, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"537189bf-44a5-452b-8e84-dbfe66c5da74\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/sew-d\", \"title\": \"SEW-D\", \"tokens\": 637, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b680134e-3f48-4dac-9dda-1d25c20ecf6e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/sew-d\", \"title\": \"SEW-D\", \"tokens\": 637, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7932e84258bad487dcc35ad86f38719dc0750fa3f4768f89de9adc2e5acbcf9f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1854, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b680134e-3f48-4dac-9dda-1d25c20ecf6e", "doc_id": "b680134e-3f48-4dac-9dda-1d25c20ecf6e", "ref_doc_id": "b680134e-3f48-4dac-9dda-1d25c20ecf6e", "chroma:document": "# SEW-D\n\n## Overview\n\nSEW-D (Squeezed and Efficient Wav2Vec with Disentangled attention) was proposed in [Performance-Efficiency Trade-offs\nin Unsupervised Pre-training for Speech Recognition](https://arxiv.org/abs/2109.06870) by Felix Wu, Kwangyoun Kim,\nJing Pan, Kyu Han, Kilian Q. Weinberger, Yoav Artzi.\n\nThe abstract from the paper is the following:\n\n*This paper is a study of performance-efficiency trade-offs in pre-trained models for automatic speech recognition\n(ASR). We focus on wav2vec 2.0, and formalize several architecture designs that influence both the model performance\nand its efficiency. Putting together all our observations, we introduce SEW (Squeezed and Efficient Wav2vec), a\npre-trained model architecture with significant improvements along both performance and efficiency dimensions across a\nvariety of training setups. For example, under the 100h-960h semi-supervised setup on LibriSpeech, SEW achieves a 1.9x\ninference speedup compared to wav2vec 2.0, with a 13.5% relative reduction in word error rate. With a similar inference\ntime, SEW reduces word error rate by 25-50% across different model sizes.*\n\nThis model was contributed by [anton-l](https://huggingface.co/anton-l).\n\n## Usage tips\n\n- SEW-D is a speech model that accepts a float array corresponding to the raw waveform of the speech signal.\n- SEWDForCTC is fine-tuned using connectionist temporal classification (CTC) so the model output has to be decoded\n  using [`Wav2Vec2CTCTokenizer`].\n\n## Resources\n\n- [Audio classification task guide](../tasks/audio_classification)\n- [Automatic speech recognition task guide](../tasks/asr)\n\n## SEWDConfig\n\n[[autodoc]] SEWDConfig\n\n## SEWDModel\n\n[[autodoc]] SEWDModel\n    - forward\n\n## SEWDForCTC\n\n[[autodoc]] SEWDForCTC\n    - forward\n\n## SEWDForSequenceClassification\n\n[[autodoc]] SEWDForSequenceClassification\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 6f7cc510-098e-4c53-8aa3-835452211709
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"6f7cc510-098e-4c53-8aa3-835452211709\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3dcd76eb-e958-488d-a887-8d0ff2acc337\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4b938a64-4c32-45fb-bc52-8bb8716f8d1c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b8e74e9d13d72f4bf23603a5fc8ef2560b7c5797615084d281d7ce1ee71f0eb3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3457, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "doc_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "ref_doc_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "chroma:document": "# TAPAS\n\n## Overview\n\nThe TAPAS model was proposed in [TAPAS: Weakly Supervised Table Parsing via Pre-training](https://www.aclweb.org/anthology/2020.acl-main.398)\nby Jonathan Herzig, Pawe\u0142 Krzysztof Nowak, Thomas M\u00fcller, Francesco Piccinno and Julian Martin Eisenschlos. It's a BERT-based model specifically \ndesigned (and pre-trained) for answering questions about tabular data. Compared to BERT, TAPAS uses relative position embeddings and has 7 \ntoken types that encode tabular structure. TAPAS is pre-trained on the masked language modeling (MLM) objective on a large dataset comprising \nmillions of tables from English Wikipedia and corresponding texts. \n\nFor question answering, TAPAS has 2 heads on top: a cell selection head and an aggregation head, for (optionally) performing aggregations (such as counting or summing) among selected cells. TAPAS has been fine-tuned on several datasets: \n- [SQA](https://www.microsoft.com/en-us/download/details.aspx?id=54253) (Sequential Question Answering by Microsoft)\n- [WTQ](https://github.com/ppasupat/WikiTableQuestions) (Wiki Table Questions by Stanford University)\n- [WikiSQL](https://github.com/salesforce/WikiSQL) (by Salesforce). \n\nIt achieves state-of-the-art on both SQA and WTQ, while having comparable performance to SOTA on WikiSQL, with a much simpler architecture.\n\nThe abstract from the paper is the following:\n\n*Answering natural language questions over tables is usually seen as a semantic parsing task. To alleviate the collection cost of full logical forms, one popular approach focuses on weak supervision consisting of denotations instead of logical forms. However, training semantic parsers from weak supervision poses difficulties, and in addition, the generated logical forms are only used as an intermediate step prior to retrieving the denotation. In this paper, we present TAPAS, an approach to question answering over tables without generating logical forms. TAPAS trains from weak supervision, and predicts the denotation by selecting table cells and optionally applying a corresponding aggregation operator to such selection. TAPAS extends BERT's architecture to encode tables as input, initializes from an effective joint pre-training of text segments and tables crawled from Wikipedia, and is trained end-to-end. We experiment with three different semantic parsing datasets, and find that TAPAS outperforms or rivals semantic parsing models by improving state-of-the-art accuracy on SQA from 55.1 to 67.2 and performing on par with the state-of-the-art on WIKISQL and WIKITQ, but with a simpler model architecture. We additionally find that transfer learning, which is trivial in our setting, from WIKISQL to WIKITQ, yields 48.7 accuracy, 4.2 points above the state-of-the-art.*\n\nIn addition, the authors have further pre-trained TAPAS to recognize **table entailment**, by creating a balanced dataset of millions of automatically created training examples which are learned in an intermediate step prior to fine-tuning. The authors of TAPAS call this further pre-training intermediate pre-training (since TAPAS is first pre-trained on MLM, and then on another dataset). They found that intermediate pre-training further improves performance on SQA, achieving a new state-of-the-art as well as state-of-the-art on [TabFact](https://github.com/wenhuchen/Table-Fact-Checking), a large-scale dataset with 16k Wikipedia tables for table entailment (a binary classification task)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 4b938a64-4c32-45fb-bc52-8bb8716f8d1c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"4b938a64-4c32-45fb-bc52-8bb8716f8d1c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3dcd76eb-e958-488d-a887-8d0ff2acc337\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6f7cc510-098e-4c53-8aa3-835452211709\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"d6b3cdf8449415b8bbe646328e23bedbdef5b9b067de10e76e87a9d3965719f1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"30324346-5201-4150-b427-9e6b43cbad6e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"54f75347f167662c9e9b12b575e19602357874423cd0452b5e98e5f7693efd94\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1628, \"end_char_idx\": 5153, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "doc_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "ref_doc_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "chroma:document": "However, training semantic parsers from weak supervision poses difficulties, and in addition, the generated logical forms are only used as an intermediate step prior to retrieving the denotation. In this paper, we present TAPAS, an approach to question answering over tables without generating logical forms. TAPAS trains from weak supervision, and predicts the denotation by selecting table cells and optionally applying a corresponding aggregation operator to such selection. TAPAS extends BERT's architecture to encode tables as input, initializes from an effective joint pre-training of text segments and tables crawled from Wikipedia, and is trained end-to-end. We experiment with three different semantic parsing datasets, and find that TAPAS outperforms or rivals semantic parsing models by improving state-of-the-art accuracy on SQA from 55.1 to 67.2 and performing on par with the state-of-the-art on WIKISQL and WIKITQ, but with a simpler model architecture. We additionally find that transfer learning, which is trivial in our setting, from WIKISQL to WIKITQ, yields 48.7 accuracy, 4.2 points above the state-of-the-art.*\n\nIn addition, the authors have further pre-trained TAPAS to recognize **table entailment**, by creating a balanced dataset of millions of automatically created training examples which are learned in an intermediate step prior to fine-tuning. The authors of TAPAS call this further pre-training intermediate pre-training (since TAPAS is first pre-trained on MLM, and then on another dataset). They found that intermediate pre-training further improves performance on SQA, achieving a new state-of-the-art as well as state-of-the-art on [TabFact](https://github.com/wenhuchen/Table-Fact-Checking), a large-scale dataset with 16k Wikipedia tables for table entailment (a binary classification task). For more details, see their follow-up paper: [Understanding tables with intermediate pre-training](https://www.aclweb.org/anthology/2020.findings-emnlp.27/) by Julian Martin Eisenschlos, Syrine Krichene and Thomas M\u00fcller.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/tapas_architecture.png\"\nalt=\"drawing\" width=\"600\"/> \n\n<small> TAPAS architecture. Taken from the <a href=\"https://ai.googleblog.com/2020/04/using-neural-networks-to-find-answers.html\">original blog post</a>.</small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The Tensorflow version of this model was contributed by [kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/google-research/tapas).\n\n## Usage tips\n\n- TAPAS is a model that uses relative position embeddings by default (restarting the position embeddings at every cell of the table). Note that this is something that was added after the publication of the original TAPAS paper. According to the authors, this usually results in a slightly better performance, and allows you to encode longer sequences without running out of embeddings. This is reflected in the `reset_position_index_per_cell` parameter of [`TapasConfig`], which is set to `True` by default. The default versions of the models available on the [hub](https://huggingface.co/models?search=tapas) all use relative position embeddings. You can still use the ones with absolute position embeddings by passing in an additional argument `revision=\"no_reset\"` when calling the `from_pretrained()` method. Note that it's usually advised to pad the inputs on the right rather than the left."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 30324346-5201-4150-b427-9e6b43cbad6e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"30324346-5201-4150-b427-9e6b43cbad6e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3dcd76eb-e958-488d-a887-8d0ff2acc337\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4b938a64-4c32-45fb-bc52-8bb8716f8d1c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"c5c67b023e5d77d4ef4825f5cda00f03a40a730ffba7398b1d44095406639359\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ba8dc3d0-8e79-4ad9-8100-6577a1c6a30f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9e7e7fff44098dce7ab89839730c10a6d7350eb0dbf2808b4123757f6337765e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3458, \"end_char_idx\": 6633, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "doc_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "ref_doc_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "chroma:document": "For more details, see their follow-up paper: [Understanding tables with intermediate pre-training](https://www.aclweb.org/anthology/2020.findings-emnlp.27/) by Julian Martin Eisenschlos, Syrine Krichene and Thomas M\u00fcller.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/tapas_architecture.png\"\nalt=\"drawing\" width=\"600\"/> \n\n<small> TAPAS architecture. Taken from the <a href=\"https://ai.googleblog.com/2020/04/using-neural-networks-to-find-answers.html\">original blog post</a>.</small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The Tensorflow version of this model was contributed by [kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/google-research/tapas).\n\n## Usage tips\n\n- TAPAS is a model that uses relative position embeddings by default (restarting the position embeddings at every cell of the table). Note that this is something that was added after the publication of the original TAPAS paper. According to the authors, this usually results in a slightly better performance, and allows you to encode longer sequences without running out of embeddings. This is reflected in the `reset_position_index_per_cell` parameter of [`TapasConfig`], which is set to `True` by default. The default versions of the models available on the [hub](https://huggingface.co/models?search=tapas) all use relative position embeddings. You can still use the ones with absolute position embeddings by passing in an additional argument `revision=\"no_reset\"` when calling the `from_pretrained()` method. Note that it's usually advised to pad the inputs on the right rather than the left.\n- TAPAS is based on BERT, so `TAPAS-base` for example corresponds to a `BERT-base` architecture. Of course, `TAPAS-large` will result in the best performance (the results reported in the paper are from `TAPAS-large`). Results of the various sized models are shown on the [original GitHub repository](https://github.com/google-research/tapas).\n- TAPAS has checkpoints fine-tuned on SQA, which are capable of answering questions related to a table in a conversational set-up. This means that you can ask follow-up questions such as \"what is his age?\" related to the previous question. Note that the forward pass of TAPAS is a bit different in case of a conversational set-up: in that case, you have to feed every table-question pair one by one to the model, such that the `prev_labels` token type ids can be overwritten by the predicted `labels` of the model to the previous question. See \"Usage\" section for more info.\n- TAPAS is similar to BERT and therefore relies on the masked language modeling (MLM) objective. It is therefore efficient at predicting masked tokens and at NLU in general, but is not optimal for text generation. Models trained with a causal language modeling (CLM) objective are better in that regard. Note that TAPAS can be used as an encoder in the EncoderDecoderModel framework, to combine it with an autoregressive text decoder such as GPT-2.\n\n## Usage: fine-tuning\n\nHere we explain how you can fine-tune [`TapasForQuestionAnswering`] on your own dataset."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: ba8dc3d0-8e79-4ad9-8100-6577a1c6a30f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"ba8dc3d0-8e79-4ad9-8100-6577a1c6a30f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3dcd76eb-e958-488d-a887-8d0ff2acc337\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"30324346-5201-4150-b427-9e6b43cbad6e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"53b4fd07ba325d6af22ef1bdf229cf88a811f6466c00b0f19f5e20000ad0dc14\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2bc5b3c4-bb1b-48b6-be34-6c4f0de404af\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5853700e59e5aaa95a01af1e6d806a23720f14a9e488a4c321261854b4657eb3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5070, \"end_char_idx\": 8015, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "doc_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "ref_doc_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "chroma:document": "Note that it's usually advised to pad the inputs on the right rather than the left.\n- TAPAS is based on BERT, so `TAPAS-base` for example corresponds to a `BERT-base` architecture. Of course, `TAPAS-large` will result in the best performance (the results reported in the paper are from `TAPAS-large`). Results of the various sized models are shown on the [original GitHub repository](https://github.com/google-research/tapas).\n- TAPAS has checkpoints fine-tuned on SQA, which are capable of answering questions related to a table in a conversational set-up. This means that you can ask follow-up questions such as \"what is his age?\" related to the previous question. Note that the forward pass of TAPAS is a bit different in case of a conversational set-up: in that case, you have to feed every table-question pair one by one to the model, such that the `prev_labels` token type ids can be overwritten by the predicted `labels` of the model to the previous question. See \"Usage\" section for more info.\n- TAPAS is similar to BERT and therefore relies on the masked language modeling (MLM) objective. It is therefore efficient at predicting masked tokens and at NLU in general, but is not optimal for text generation. Models trained with a causal language modeling (CLM) objective are better in that regard. Note that TAPAS can be used as an encoder in the EncoderDecoderModel framework, to combine it with an autoregressive text decoder such as GPT-2.\n\n## Usage: fine-tuning\n\nHere we explain how you can fine-tune [`TapasForQuestionAnswering`] on your own dataset.\n\n**STEP 1: Choose one of the 3 ways in which you can use TAPAS - or experiment**\n\nBasically, there are 3 different ways in which one can fine-tune [`TapasForQuestionAnswering`], corresponding to the different datasets on which Tapas was fine-tuned:\n\n1. SQA: if you're interested in asking follow-up questions related to a table, in a conversational set-up. For example if you first ask \"what's the name of the first actor?\" then you can ask a follow-up question such as \"how old is he?\". Here, questions do not involve any aggregation (all questions are cell selection questions).\n2. WTQ: if you're not interested in asking questions in a conversational set-up, but rather just asking questions related to a table, which might involve aggregation, such as counting a number of rows, summing up cell values or averaging cell values. You can then for example ask \"what's the total number of goals Cristiano Ronaldo made in his career?\". This case is also called **weak supervision**, since the model itself must learn the appropriate aggregation operator (SUM/COUNT/AVERAGE/NONE) given only the answer to the question as supervision.\n3. WikiSQL-supervised: this dataset is based on WikiSQL with the model being given the ground truth aggregation operator during training. This is also called **strong supervision**. Here, learning the appropriate aggregation operator is much easier."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 2bc5b3c4-bb1b-48b6-be34-6c4f0de404af
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"2bc5b3c4-bb1b-48b6-be34-6c4f0de404af\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3dcd76eb-e958-488d-a887-8d0ff2acc337\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ba8dc3d0-8e79-4ad9-8100-6577a1c6a30f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"815cca00555d4e49fb99d6f06cd5627dfd7d9a9b09a739e7cefab84e6b0e10d7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b41bf89b-8e27-48bc-8ad6-f8b5e11b1e6b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"07c2ee055ce6f362a7410769e0cd6fd2cd1852b46c1f2867f2f2e1141acb5fcf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6286, \"end_char_idx\": 10123, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "doc_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "ref_doc_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "chroma:document": "Models trained with a causal language modeling (CLM) objective are better in that regard. Note that TAPAS can be used as an encoder in the EncoderDecoderModel framework, to combine it with an autoregressive text decoder such as GPT-2.\n\n## Usage: fine-tuning\n\nHere we explain how you can fine-tune [`TapasForQuestionAnswering`] on your own dataset.\n\n**STEP 1: Choose one of the 3 ways in which you can use TAPAS - or experiment**\n\nBasically, there are 3 different ways in which one can fine-tune [`TapasForQuestionAnswering`], corresponding to the different datasets on which Tapas was fine-tuned:\n\n1. SQA: if you're interested in asking follow-up questions related to a table, in a conversational set-up. For example if you first ask \"what's the name of the first actor?\" then you can ask a follow-up question such as \"how old is he?\". Here, questions do not involve any aggregation (all questions are cell selection questions).\n2. WTQ: if you're not interested in asking questions in a conversational set-up, but rather just asking questions related to a table, which might involve aggregation, such as counting a number of rows, summing up cell values or averaging cell values. You can then for example ask \"what's the total number of goals Cristiano Ronaldo made in his career?\". This case is also called **weak supervision**, since the model itself must learn the appropriate aggregation operator (SUM/COUNT/AVERAGE/NONE) given only the answer to the question as supervision.\n3. WikiSQL-supervised: this dataset is based on WikiSQL with the model being given the ground truth aggregation operator during training. This is also called **strong supervision**. Here, learning the appropriate aggregation operator is much easier.\n\nTo summarize:\n\n| **Task**                            | **Example dataset** | **Description**                                                                                         |\n|-------------------------------------|---------------------|---------------------------------------------------------------------------------------------------------|\n| Conversational                      | SQA                 | Conversational, only cell selection questions                                                           |\n| Weak supervision for aggregation    | WTQ                 | Questions might involve aggregation, and the model must learn this given only the answer as supervision |\n| Strong supervision for aggregation  | WikiSQL-supervised  | Questions might involve aggregation, and the model must learn this given the gold aggregation operator  |\n\n<frameworkcontent>\n<pt>\nInitializing a model with a pre-trained base and randomly initialized classification heads from the hub can be done as shown below.\n\n```py\n>>> from transformers import TapasConfig, TapasForQuestionAnswering\n\n>>> # for example, the base sized model with default SQA configuration\n>>> model = TapasForQuestionAnswering.from_pretrained(\"google/tapas-base\")\n\n>>> # or, the base sized model with WTQ configuration\n>>> config = TapasConfig.from_pretrained(\"google/tapas-base-finetuned-wtq\")\n>>> model = TapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n\n>>> # or, the base sized model with WikiSQL configuration\n>>> config = TapasConfig(\"google-base-finetuned-wikisql-supervised\")\n>>> model = TapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n```\n\nOf course, you don't necessarily have to follow one of these three ways in which TAPAS was fine-tuned. You can also experiment by defining any hyperparameters you want when initializing [`TapasConfig`], and then create a [`TapasForQuestionAnswering`] based on that configuration. For example, if you have a dataset that has both conversational questions and questions that might involve aggregation, then you can do it this way."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: b41bf89b-8e27-48bc-8ad6-f8b5e11b1e6b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"b41bf89b-8e27-48bc-8ad6-f8b5e11b1e6b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3dcd76eb-e958-488d-a887-8d0ff2acc337\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2bc5b3c4-bb1b-48b6-be34-6c4f0de404af\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"f4f4f0d59938eaf215ec69c506e83cc666ba1a7ab549265cf9d18100f1754588\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4f32b111-f057-4d11-8086-c89cd0efbe1b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6bae031756ce043224ba32faee981850fd67269f509027fa69a07b2471954c4b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8017, \"end_char_idx\": 11784, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "doc_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "ref_doc_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "chroma:document": "To summarize:\n\n| **Task**                            | **Example dataset** | **Description**                                                                                         |\n|-------------------------------------|---------------------|---------------------------------------------------------------------------------------------------------|\n| Conversational                      | SQA                 | Conversational, only cell selection questions                                                           |\n| Weak supervision for aggregation    | WTQ                 | Questions might involve aggregation, and the model must learn this given only the answer as supervision |\n| Strong supervision for aggregation  | WikiSQL-supervised  | Questions might involve aggregation, and the model must learn this given the gold aggregation operator  |\n\n<frameworkcontent>\n<pt>\nInitializing a model with a pre-trained base and randomly initialized classification heads from the hub can be done as shown below.\n\n```py\n>>> from transformers import TapasConfig, TapasForQuestionAnswering\n\n>>> # for example, the base sized model with default SQA configuration\n>>> model = TapasForQuestionAnswering.from_pretrained(\"google/tapas-base\")\n\n>>> # or, the base sized model with WTQ configuration\n>>> config = TapasConfig.from_pretrained(\"google/tapas-base-finetuned-wtq\")\n>>> model = TapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n\n>>> # or, the base sized model with WikiSQL configuration\n>>> config = TapasConfig(\"google-base-finetuned-wikisql-supervised\")\n>>> model = TapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n```\n\nOf course, you don't necessarily have to follow one of these three ways in which TAPAS was fine-tuned. You can also experiment by defining any hyperparameters you want when initializing [`TapasConfig`], and then create a [`TapasForQuestionAnswering`] based on that configuration. For example, if you have a dataset that has both conversational questions and questions that might involve aggregation, then you can do it this way. Here's an example:\n\n```py\n>>> from transformers import TapasConfig, TapasForQuestionAnswering\n\n>>> # you can initialize the classification heads any way you want (see docs of TapasConfig)\n>>> config = TapasConfig(num_aggregation_labels=3, average_logits_per_cell=True)\n>>> # initializing the pre-trained base sized model with our custom classification heads\n>>> model = TapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n```\n</pt>\n<tf>\nInitializing a model with a pre-trained base and randomly initialized classification heads from the hub can be done as shown below. Be sure to have installed the [tensorflow_probability](https://github.com/tensorflow/probability) dependency:\n\n```py\n>>> from transformers import TapasConfig, TFTapasForQuestionAnswering\n\n>>> # for example, the base sized model with default SQA configuration\n>>> model = TFTapasForQuestionAnswering.from_pretrained(\"google/tapas-base\")\n\n>>> # or, the base sized model with WTQ configuration\n>>> config = TapasConfig.from_pretrained(\"google/tapas-base-finetuned-wtq\")\n>>> model = TFTapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n\n>>> # or, the base sized model with WikiSQL configuration\n>>> config = TapasConfig(\"google-base-finetuned-wikisql-supervised\")\n>>> model = TFTapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n```\n\nOf course, you don't necessarily have to follow one of these three ways in which TAPAS was fine-tuned. You can also experiment by defining any hyperparameters you want when initializing [`TapasConfig`], and then create a [`TFTapasForQuestionAnswering`] based on that configuration."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 4f32b111-f057-4d11-8086-c89cd0efbe1b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"4f32b111-f057-4d11-8086-c89cd0efbe1b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3dcd76eb-e958-488d-a887-8d0ff2acc337\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b41bf89b-8e27-48bc-8ad6-f8b5e11b1e6b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"a2f98c5841119cccaec3faf68f8769ed6e19592332ca63edf45023e853fcdcd0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"82c2d1ef-66f7-42c5-937e-f3f8cedd4f52\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"729ddea90a9e793940fcc8f071cd5c8cec97e6b6c3d3641c1020104c276daeeb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10124, \"end_char_idx\": 13467, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "doc_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "ref_doc_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "chroma:document": "Here's an example:\n\n```py\n>>> from transformers import TapasConfig, TapasForQuestionAnswering\n\n>>> # you can initialize the classification heads any way you want (see docs of TapasConfig)\n>>> config = TapasConfig(num_aggregation_labels=3, average_logits_per_cell=True)\n>>> # initializing the pre-trained base sized model with our custom classification heads\n>>> model = TapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n```\n</pt>\n<tf>\nInitializing a model with a pre-trained base and randomly initialized classification heads from the hub can be done as shown below. Be sure to have installed the [tensorflow_probability](https://github.com/tensorflow/probability) dependency:\n\n```py\n>>> from transformers import TapasConfig, TFTapasForQuestionAnswering\n\n>>> # for example, the base sized model with default SQA configuration\n>>> model = TFTapasForQuestionAnswering.from_pretrained(\"google/tapas-base\")\n\n>>> # or, the base sized model with WTQ configuration\n>>> config = TapasConfig.from_pretrained(\"google/tapas-base-finetuned-wtq\")\n>>> model = TFTapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n\n>>> # or, the base sized model with WikiSQL configuration\n>>> config = TapasConfig(\"google-base-finetuned-wikisql-supervised\")\n>>> model = TFTapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n```\n\nOf course, you don't necessarily have to follow one of these three ways in which TAPAS was fine-tuned. You can also experiment by defining any hyperparameters you want when initializing [`TapasConfig`], and then create a [`TFTapasForQuestionAnswering`] based on that configuration. For example, if you have a dataset that has both conversational questions and questions that might involve aggregation, then you can do it this way. Here's an example:\n\n```py\n>>> from transformers import TapasConfig, TFTapasForQuestionAnswering\n\n>>> # you can initialize the classification heads any way you want (see docs of TapasConfig)\n>>> config = TapasConfig(num_aggregation_labels=3, average_logits_per_cell=True)\n>>> # initializing the pre-trained base sized model with our custom classification heads\n>>> model = TFTapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n```\n</tf>\n</frameworkcontent>\n\nWhat you can also do is start from an already fine-tuned checkpoint. A note here is that the already fine-tuned checkpoint on WTQ has some issues due to the L2-loss which is somewhat brittle. See [here](https://github.com/google-research/tapas/issues/91#issuecomment-735719340) for more info.\n\nFor a list of all pre-trained and fine-tuned TAPAS checkpoints available on HuggingFace's  hub, see [here](https://huggingface.co/models?search=tapas).\n\n**STEP 2: Prepare your data in the SQA format**\n\nSecond, no matter what you picked above, you should prepare your dataset in the [SQA](https://www.microsoft.com/en-us/download/details.aspx?id=54253) format. This format is a TSV/CSV file with the following columns:\n\n- `id`: optional, id of the table-question pair, for bookkeeping purposes.\n- `annotator`: optional, id of the person who annotated the table-question pair, for bookkeeping purposes.\n- `position`: integer indicating if the question is the first, second, third,... related to the table. Only required in case of conversational setup (SQA)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 82c2d1ef-66f7-42c5-937e-f3f8cedd4f52
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"82c2d1ef-66f7-42c5-937e-f3f8cedd4f52\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3dcd76eb-e958-488d-a887-8d0ff2acc337\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4f32b111-f057-4d11-8086-c89cd0efbe1b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"27ff86a8e9e17bf77ea19de0b8d828369a12cf2267cb19533f4965482377d8c2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3f82c876-0908-4433-bc7c-cf61bf31a99e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"05338a496a08ff547ae8a53b5860be98e6963b991e8c251e144094cd293cd2fd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11785, \"end_char_idx\": 14917, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "doc_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "ref_doc_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "chroma:document": "For example, if you have a dataset that has both conversational questions and questions that might involve aggregation, then you can do it this way. Here's an example:\n\n```py\n>>> from transformers import TapasConfig, TFTapasForQuestionAnswering\n\n>>> # you can initialize the classification heads any way you want (see docs of TapasConfig)\n>>> config = TapasConfig(num_aggregation_labels=3, average_logits_per_cell=True)\n>>> # initializing the pre-trained base sized model with our custom classification heads\n>>> model = TFTapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n```\n</tf>\n</frameworkcontent>\n\nWhat you can also do is start from an already fine-tuned checkpoint. A note here is that the already fine-tuned checkpoint on WTQ has some issues due to the L2-loss which is somewhat brittle. See [here](https://github.com/google-research/tapas/issues/91#issuecomment-735719340) for more info.\n\nFor a list of all pre-trained and fine-tuned TAPAS checkpoints available on HuggingFace's  hub, see [here](https://huggingface.co/models?search=tapas).\n\n**STEP 2: Prepare your data in the SQA format**\n\nSecond, no matter what you picked above, you should prepare your dataset in the [SQA](https://www.microsoft.com/en-us/download/details.aspx?id=54253) format. This format is a TSV/CSV file with the following columns:\n\n- `id`: optional, id of the table-question pair, for bookkeeping purposes.\n- `annotator`: optional, id of the person who annotated the table-question pair, for bookkeeping purposes.\n- `position`: integer indicating if the question is the first, second, third,... related to the table. Only required in case of conversational setup (SQA). You don't need this column in case you're going for WTQ/WikiSQL-supervised.\n- `question`: string\n- `table_file`: string, name of a csv file containing the tabular data\n- `answer_coordinates`: list of one or more tuples (each tuple being a cell coordinate, i.e. row, column pair that is part of the answer)\n- `answer_text`: list of one or more strings (each string being a cell value that is part of the answer)\n- `aggregation_label`: index of the aggregation operator. Only required in case of strong supervision for aggregation (the WikiSQL-supervised case)\n- `float_answer`: the float answer to the question, if there is one (np.nan if there isn't). Only required in case of weak supervision for aggregation (such as WTQ and WikiSQL)\n\nThe tables themselves should be present in a folder, each table being a separate csv file. Note that the authors of the TAPAS algorithm used conversion scripts with some automated logic to convert the other datasets (WTQ, WikiSQL) into the SQA format. The author explains this [here](https://github.com/google-research/tapas/issues/50#issuecomment-705465960). A conversion of this script that works with HuggingFace's implementation can be found [here](https://github.com/NielsRogge/tapas_utils). Interestingly, these conversion scripts are not perfect (the `answer_coordinates` and `float_answer` fields are populated based on the `answer_text`), meaning that WTQ and WikiSQL results could actually be improved."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 3f82c876-0908-4433-bc7c-cf61bf31a99e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"3f82c876-0908-4433-bc7c-cf61bf31a99e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3dcd76eb-e958-488d-a887-8d0ff2acc337\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"82c2d1ef-66f7-42c5-937e-f3f8cedd4f52\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"ef8ce7b6fc8cd8e36015416868943c87a105d5d92367cff769a952cc9e61d992\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"789f1f8f-f587-4c44-a7d0-4d885059bdc6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2ad94ff4bcb398e41591ae5d15388101156198403251862b34b3b4bf1f7e1630\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13205, \"end_char_idx\": 16801, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "doc_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "ref_doc_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "chroma:document": "- `annotator`: optional, id of the person who annotated the table-question pair, for bookkeeping purposes.\n- `position`: integer indicating if the question is the first, second, third,... related to the table. Only required in case of conversational setup (SQA). You don't need this column in case you're going for WTQ/WikiSQL-supervised.\n- `question`: string\n- `table_file`: string, name of a csv file containing the tabular data\n- `answer_coordinates`: list of one or more tuples (each tuple being a cell coordinate, i.e. row, column pair that is part of the answer)\n- `answer_text`: list of one or more strings (each string being a cell value that is part of the answer)\n- `aggregation_label`: index of the aggregation operator. Only required in case of strong supervision for aggregation (the WikiSQL-supervised case)\n- `float_answer`: the float answer to the question, if there is one (np.nan if there isn't). Only required in case of weak supervision for aggregation (such as WTQ and WikiSQL)\n\nThe tables themselves should be present in a folder, each table being a separate csv file. Note that the authors of the TAPAS algorithm used conversion scripts with some automated logic to convert the other datasets (WTQ, WikiSQL) into the SQA format. The author explains this [here](https://github.com/google-research/tapas/issues/50#issuecomment-705465960). A conversion of this script that works with HuggingFace's implementation can be found [here](https://github.com/NielsRogge/tapas_utils). Interestingly, these conversion scripts are not perfect (the `answer_coordinates` and `float_answer` fields are populated based on the `answer_text`), meaning that WTQ and WikiSQL results could actually be improved.\n\n**STEP 3: Convert your data into tensors using TapasTokenizer**\n\n<frameworkcontent>\n<pt>\nThird, given that you've prepared your data in this TSV/CSV format (and corresponding CSV files containing the tabular data), you can then use [`TapasTokenizer`] to convert table-question pairs into `input_ids`, `attention_mask`, `token_type_ids` and so on. Again, based on which of the three cases you picked above, [`TapasForQuestionAnswering`] requires different\ninputs to be fine-tuned:\n\n| **Task**                           | **Required inputs**                                                                                                 |\n|------------------------------------|---------------------------------------------------------------------------------------------------------------------|\n| Conversational                     | `input_ids`, `attention_mask`, `token_type_ids`, `labels`                                                           |\n|  Weak supervision for aggregation  | `input_ids`, `attention_mask`, `token_type_ids`, `labels`, `numeric_values`, `numeric_values_scale`, `float_answer` |\n| Strong supervision for aggregation | `input ids`, `attention mask`, `token type ids`, `labels`, `aggregation_labels`                                     |\n\n[`TapasTokenizer`] creates the `labels`, `numeric_values` and `numeric_values_scale` based on the `answer_coordinates` and `answer_text` columns of the TSV file. The `float_answer` and `aggregation_labels` are already in the TSV file of step 2. Here's an example:\n\n```py\n>>> from transformers import TapasTokenizer\n>>> import pandas as pd\n\n>>> model_name = \"google/tapas-base\"\n>>> tokenizer = TapasTokenizer.from_pretrained(model_name)\n\n>>> data = {\"Actors\": [\"Brad Pitt\", \"Leonardo Di Caprio\", \"George Clooney\"], \"Number of movies\": [\"87\", \"53\", \"69\"]}\n>>> queries = [\n...     \"What is the name of the first actor?"}


