# embeddings_queue

created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f54533f1-d071-4fa3-b34c-af2c405606cc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f54533f1-d071-4fa3-b34c-af2c405606cc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta\", \"title\": \"RoBERTa\", \"tokens\": 3026, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5c6eb115-c5bd-4197-94a5-49da19a503aa\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta\", \"title\": \"RoBERTa\", \"tokens\": 3026, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5600d5c6dca955efe139ace218bf61e3b2034c83533b5923f69810afa2660ed1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e571e305-2449-41c7-863c-d8325947cee2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7e9a9bea7f2e43028c15abf364e8ff9680e7e382d6c64fa6eb7e1b10a403b0dc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3006, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5c6eb115-c5bd-4197-94a5-49da19a503aa", "doc_id": "5c6eb115-c5bd-4197-94a5-49da19a503aa", "ref_doc_id": "5c6eb115-c5bd-4197-94a5-49da19a503aa", "chroma:document": "# RoBERTa\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=roberta\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-roberta-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/roberta-base\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n<a href=\"https://huggingface.co/papers/1907.11692\">\n<img alt=\"Paper page\" src=\"https://img.shields.io/badge/Paper%20page-1907.11692-green\">\n</a>\n</div>\n\n## Overview\n\nThe RoBERTa model was proposed in [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://arxiv.org/abs/1907.11692) by Yinhan Liu, [Myle Ott](https://huggingface.co/myleott), Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer\nLevy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov. It is based on Google's BERT model released in 2018.\n\nIt builds on BERT and modifies key hyperparameters, removing the next-sentence pretraining objective and training with\nmuch larger mini-batches and learning rates.\n\nThe abstract from the paper is the following:\n\n*Language model pretraining has led to significant performance gains but careful comparison between different\napproaches is challenging. Training is computationally expensive, often done on private datasets of different sizes,\nand, as we will show, hyperparameter choices have significant impact on the final results. We present a replication\nstudy of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and\ntraining data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every\nmodel published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results\nhighlight the importance of previously overlooked design choices, and raise questions about the source of recently\nreported improvements. We release our models and code.*\n\nThis model was contributed by [julien-c](https://huggingface.co/julien-c). The original code can be found [here](https://github.com/pytorch/fairseq/tree/master/examples/roberta).\n\n## Usage tips\n\n- This implementation is the same as [`BertModel`] with a minor tweak to the embeddings, as well as a setup\n  for RoBERTa pretrained models.\n- RoBERTa has the same architecture as BERT but uses a byte-level BPE as a tokenizer (same as GPT-2) and uses a\n  different pretraining scheme.\n- RoBERTa doesn't have `token_type_ids`, so you don't need to indicate which token belongs to which segment. Just\n  separate your segments with the separation token `tokenizer.sep_token` (or `</s>`).\n- RoBERTa is similar to BERT but with better pretraining techniques:\n\n    * Dynamic masking: tokens are masked differently at each epoch, whereas BERT does it once and for all.\n    * Sentence packing: Sentences are packed together to reach 512 tokens (so the sentences are in an order that may span several documents).\n    * Larger batches: Training uses larger batches."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e571e305-2449-41c7-863c-d8325947cee2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e571e305-2449-41c7-863c-d8325947cee2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta\", \"title\": \"RoBERTa\", \"tokens\": 3026, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5c6eb115-c5bd-4197-94a5-49da19a503aa\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta\", \"title\": \"RoBERTa\", \"tokens\": 3026, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5600d5c6dca955efe139ace218bf61e3b2034c83533b5923f69810afa2660ed1\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f54533f1-d071-4fa3-b34c-af2c405606cc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta\", \"title\": \"RoBERTa\", \"tokens\": 3026, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f919e9d84035c7557106fe58a50fae2bb5596a128f5914b320741e3d14102c87\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a3b7e37f-0904-4b67-b943-fae9afb3b3ca\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"795a632c853f71615069acd4102ba841a2189e49b056d2a7ce1b461bb4899719\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1411, \"end_char_idx\": 4760, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5c6eb115-c5bd-4197-94a5-49da19a503aa", "doc_id": "5c6eb115-c5bd-4197-94a5-49da19a503aa", "ref_doc_id": "5c6eb115-c5bd-4197-94a5-49da19a503aa", "chroma:document": "We present a replication\nstudy of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and\ntraining data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every\nmodel published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results\nhighlight the importance of previously overlooked design choices, and raise questions about the source of recently\nreported improvements. We release our models and code.*\n\nThis model was contributed by [julien-c](https://huggingface.co/julien-c). The original code can be found [here](https://github.com/pytorch/fairseq/tree/master/examples/roberta).\n\n## Usage tips\n\n- This implementation is the same as [`BertModel`] with a minor tweak to the embeddings, as well as a setup\n  for RoBERTa pretrained models.\n- RoBERTa has the same architecture as BERT but uses a byte-level BPE as a tokenizer (same as GPT-2) and uses a\n  different pretraining scheme.\n- RoBERTa doesn't have `token_type_ids`, so you don't need to indicate which token belongs to which segment. Just\n  separate your segments with the separation token `tokenizer.sep_token` (or `</s>`).\n- RoBERTa is similar to BERT but with better pretraining techniques:\n\n    * Dynamic masking: tokens are masked differently at each epoch, whereas BERT does it once and for all.\n    * Sentence packing: Sentences are packed together to reach 512 tokens (so the sentences are in an order that may span several documents).\n    * Larger batches: Training uses larger batches.\n    * Byte-level BPE vocabulary: Uses BPE with bytes as a subunit instead of characters, accommodating Unicode characters.\n- [CamemBERT](camembert) is a wrapper around RoBERTa. Refer to its model page for usage examples.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with RoBERTa. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- A blog on [Getting Started with Sentiment Analysis on Twitter](https://huggingface.co/blog/sentiment-analysis-twitter) using RoBERTa and the [Inference API](https://huggingface.co/inference-api).\n- A blog on [Opinion Classification with Kili and Hugging Face AutoTrain](https://huggingface.co/blog/opinion-classification-with-kili) using RoBERTa.\n- A notebook on how to [finetune RoBERTa for sentiment analysis](https://colab.research.google.com/github/DhavalTaunk08/NLP_scripts/blob/master/sentiment_analysis_using_roberta.ipynb). \ud83c\udf0e\n- [`RobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification.ipynb).\n- [`TFRobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification-tf.ipynb)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a3b7e37f-0904-4b67-b943-fae9afb3b3ca
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a3b7e37f-0904-4b67-b943-fae9afb3b3ca\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta\", \"title\": \"RoBERTa\", \"tokens\": 3026, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5c6eb115-c5bd-4197-94a5-49da19a503aa\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta\", \"title\": \"RoBERTa\", \"tokens\": 3026, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5600d5c6dca955efe139ace218bf61e3b2034c83533b5923f69810afa2660ed1\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e571e305-2449-41c7-863c-d8325947cee2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta\", \"title\": \"RoBERTa\", \"tokens\": 3026, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5fcf0966c0c2522aa49518d5dec924edefff4264ba581f7f50e961b2fe5c6faa\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c2525c8b-df68-49de-bcce-24c3096c4677\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"dff54f4b19bb58959515f781b6e03fca32540dc10c7c42313b3d29bfd90cf894\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3130, \"end_char_idx\": 6308, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5c6eb115-c5bd-4197-94a5-49da19a503aa", "doc_id": "5c6eb115-c5bd-4197-94a5-49da19a503aa", "ref_doc_id": "5c6eb115-c5bd-4197-94a5-49da19a503aa", "chroma:document": "- [CamemBERT](camembert) is a wrapper around RoBERTa. Refer to its model page for usage examples.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with RoBERTa. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- A blog on [Getting Started with Sentiment Analysis on Twitter](https://huggingface.co/blog/sentiment-analysis-twitter) using RoBERTa and the [Inference API](https://huggingface.co/inference-api).\n- A blog on [Opinion Classification with Kili and Hugging Face AutoTrain](https://huggingface.co/blog/opinion-classification-with-kili) using RoBERTa.\n- A notebook on how to [finetune RoBERTa for sentiment analysis](https://colab.research.google.com/github/DhavalTaunk08/NLP_scripts/blob/master/sentiment_analysis_using_roberta.ipynb). \ud83c\udf0e\n- [`RobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification.ipynb).\n- [`TFRobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification-tf.ipynb).\n- [`FlaxRobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification_flax.ipynb).\n- [Text classification task guide](../tasks/sequence_classification)\n\n<PipelineTag pipeline=\"token-classification\"/>\n\n- [`RobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification.ipynb).\n- [`TFRobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification-tf.ipynb).\n- [`FlaxRobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/token-classification).\n- [Token classification](https://huggingface.co/course/chapter7/2?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Token classification task guide](../tasks/token_classification)\n\n<PipelineTag pipeline=\"fill-mask\"/>\n\n- A blog on [How to train a new language model from scratch using Transformers and Tokenizers](https://huggingface.co/blog/how-to-train) with RoBERTa."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: c2525c8b-df68-49de-bcce-24c3096c4677
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c2525c8b-df68-49de-bcce-24c3096c4677\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta\", \"title\": \"RoBERTa\", \"tokens\": 3026, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5c6eb115-c5bd-4197-94a5-49da19a503aa\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta\", \"title\": \"RoBERTa\", \"tokens\": 3026, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5600d5c6dca955efe139ace218bf61e3b2034c83533b5923f69810afa2660ed1\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a3b7e37f-0904-4b67-b943-fae9afb3b3ca\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta\", \"title\": \"RoBERTa\", \"tokens\": 3026, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1c8edd7cd0038c9ff98768bf24ce02978a963097e08e9033a1973f66eea6b5ca\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"725825e8-fd6a-420c-9a4c-ed326c52422a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"778595b00b96a9a20dc44777b666f8f07b8ce664384bef41d5348caabfbd1321\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4761, \"end_char_idx\": 7927, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5c6eb115-c5bd-4197-94a5-49da19a503aa", "doc_id": "5c6eb115-c5bd-4197-94a5-49da19a503aa", "ref_doc_id": "5c6eb115-c5bd-4197-94a5-49da19a503aa", "chroma:document": "- [`FlaxRobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification_flax.ipynb).\n- [Text classification task guide](../tasks/sequence_classification)\n\n<PipelineTag pipeline=\"token-classification\"/>\n\n- [`RobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification.ipynb).\n- [`TFRobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification-tf.ipynb).\n- [`FlaxRobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/token-classification).\n- [Token classification](https://huggingface.co/course/chapter7/2?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Token classification task guide](../tasks/token_classification)\n\n<PipelineTag pipeline=\"fill-mask\"/>\n\n- A blog on [How to train a new language model from scratch using Transformers and Tokenizers](https://huggingface.co/blog/how-to-train) with RoBERTa.\n- [`RobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#robertabertdistilbert-and-masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n- [`TFRobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_mlmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- [`FlaxRobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).\n- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- A blog on [Accelerated Inference with Optimum and Transformers Pipelines](https://huggingface.co/blog/optimum-inference) with RoBERTa for question answering.\n- [`RobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 725825e8-fd6a-420c-9a4c-ed326c52422a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"725825e8-fd6a-420c-9a4c-ed326c52422a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta\", \"title\": \"RoBERTa\", \"tokens\": 3026, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5c6eb115-c5bd-4197-94a5-49da19a503aa\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta\", \"title\": \"RoBERTa\", \"tokens\": 3026, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5600d5c6dca955efe139ace218bf61e3b2034c83533b5923f69810afa2660ed1\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c2525c8b-df68-49de-bcce-24c3096c4677\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta\", \"title\": \"RoBERTa\", \"tokens\": 3026, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e4ee4ab04e6d3ac9e114bfae173a82207e06a76022c96955f213bfba316f3d50\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2feb17f2-7bfa-40d8-81b9-0fcfc7ff77cb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d3f3a2e5f89a5f00130aaf662421b136792d37673c33ead1919633573853632d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6636, \"end_char_idx\": 9144, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5c6eb115-c5bd-4197-94a5-49da19a503aa", "doc_id": "5c6eb115-c5bd-4197-94a5-49da19a503aa", "ref_doc_id": "5c6eb115-c5bd-4197-94a5-49da19a503aa", "chroma:document": "- [`TFRobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_mlmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- [`FlaxRobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).\n- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- A blog on [Accelerated Inference with Optimum and Transformers Pipelines](https://huggingface.co/blog/optimum-inference) with RoBERTa for question answering.\n- [`RobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).\n- [`TFRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).\n- [`FlaxRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering).\n- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Question answering task guide](../tasks/question_answering)\n\n**Multiple choice**\n- [`RobertaForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb).\n- [`TFRobertaForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice-tf.ipynb)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 2feb17f2-7bfa-40d8-81b9-0fcfc7ff77cb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2feb17f2-7bfa-40d8-81b9-0fcfc7ff77cb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta\", \"title\": \"RoBERTa\", \"tokens\": 3026, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5c6eb115-c5bd-4197-94a5-49da19a503aa\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta\", \"title\": \"RoBERTa\", \"tokens\": 3026, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5600d5c6dca955efe139ace218bf61e3b2034c83533b5923f69810afa2660ed1\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"725825e8-fd6a-420c-9a4c-ed326c52422a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roberta\", \"title\": \"RoBERTa\", \"tokens\": 3026, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d170bbc4843d16505b2b2d462c8b143ddb0e41eff1604afe9b61b88f199bf288\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7641, \"end_char_idx\": 11330, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5c6eb115-c5bd-4197-94a5-49da19a503aa", "doc_id": "5c6eb115-c5bd-4197-94a5-49da19a503aa", "ref_doc_id": "5c6eb115-c5bd-4197-94a5-49da19a503aa", "chroma:document": "- [`RobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).\n- [`TFRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).\n- [`FlaxRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering).\n- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Question answering task guide](../tasks/question_answering)\n\n**Multiple choice**\n- [`RobertaForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb).\n- [`TFRobertaForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice-tf.ipynb).\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## RobertaConfig\n\n[[autodoc]] RobertaConfig\n\n## RobertaTokenizer\n\n[[autodoc]] RobertaTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## RobertaTokenizerFast\n\n[[autodoc]] RobertaTokenizerFast\n    - build_inputs_with_special_tokens\n\n<frameworkcontent>\n<pt>\n\n## RobertaModel\n\n[[autodoc]] RobertaModel\n    - forward\n\n## RobertaForCausalLM\n\n[[autodoc]] RobertaForCausalLM\n    - forward\n\n## RobertaForMaskedLM\n\n[[autodoc]] RobertaForMaskedLM\n    - forward\n\n## RobertaForSequenceClassification\n\n[[autodoc]] RobertaForSequenceClassification\n    - forward\n\n## RobertaForMultipleChoice\n\n[[autodoc]] RobertaForMultipleChoice\n    - forward\n\n## RobertaForTokenClassification\n\n[[autodoc]] RobertaForTokenClassification\n    - forward\n\n## RobertaForQuestionAnswering\n\n[[autodoc]] RobertaForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFRobertaModel\n\n[[autodoc]] TFRobertaModel\n    - call\n\n## TFRobertaForCausalLM\n\n[[autodoc]] TFRobertaForCausalLM\n    - call\n\n## TFRobertaForMaskedLM\n\n[[autodoc]] TFRobertaForMaskedLM\n    - call\n\n## TFRobertaForSequenceClassification\n\n[[autodoc]] TFRobertaForSequenceClassification\n    - call\n\n## TFRobertaForMultipleChoice\n\n[[autodoc]] TFRobertaForMultipleChoice\n    - call\n\n## TFRobertaForTokenClassification\n\n[[autodoc]] TFRobertaForTokenClassification\n    - call\n\n## TFRobertaForQuestionAnswering\n\n[[autodoc]] TFRobertaForQuestionAnswering\n    - call\n\n</tf>\n<jax>\n\n## FlaxRobertaModel\n\n[[autodoc]] FlaxRobertaModel\n    - __call__\n\n## FlaxRobertaForCausalLM\n\n[[autodoc]] FlaxRobertaForCausalLM\n    - __call__\n\n## FlaxRobertaForMaskedLM\n\n[[autodoc]] FlaxRobertaForMaskedLM\n    - __call__\n\n## FlaxRobertaForSequenceClassification\n\n[[autodoc]] FlaxRobertaForSequenceClassification\n    - __call__\n\n## FlaxRobertaForMultipleChoice\n\n[[autodoc]] FlaxRobertaForMultipleChoice\n    - __call__\n\n## FlaxRobertaForTokenClassification\n\n[[autodoc]] FlaxRobertaForTokenClassification\n    - __call__\n\n## FlaxRobertaForQuestionAnswering\n\n[[autodoc]] FlaxRobertaForQuestionAnswering\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 5a8142fc-ee19-4a71-ba56-4084fa5ff6dc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/swinv2", "title": "Swin Transformer V2", "tokens": 894, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5a8142fc-ee19-4a71-ba56-4084fa5ff6dc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/swinv2\", \"title\": \"Swin Transformer V2\", \"tokens\": 894, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4e36004e-21c0-44be-8f8b-4d5d0f2ffd53\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/swinv2\", \"title\": \"Swin Transformer V2\", \"tokens\": 894, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6cdd98d6b1a403b4f727ab32240655dd8b9094f2947e7a6d4566035fd54ffb19\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3253, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4e36004e-21c0-44be-8f8b-4d5d0f2ffd53", "doc_id": "4e36004e-21c0-44be-8f8b-4d5d0f2ffd53", "ref_doc_id": "4e36004e-21c0-44be-8f8b-4d5d0f2ffd53", "chroma:document": "# Swin Transformer V2\n\n## Overview\n\nThe Swin Transformer V2 model was proposed in [Swin Transformer V2: Scaling Up Capacity and Resolution](https://arxiv.org/abs/2111.09883) by Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, Baining Guo.\n\nThe abstract from the paper is the following:\n\n*Large-scale NLP models have been shown to significantly improve the performance on language tasks with no signs of saturation. They also demonstrate amazing few-shot capabilities like that of human beings. This paper aims to explore large-scale models in computer vision. We tackle three major issues in training and application of large vision models, including training instability, resolution gaps between pre-training and fine-tuning, and hunger on labelled data. Three main techniques are proposed: 1) a residual-post-norm method combined with cosine attention to improve training stability; 2) A log-spaced continuous position bias method to effectively transfer models pre-trained using low-resolution images to downstream tasks with high-resolution inputs; 3) A self-supervised pre-training method, SimMIM, to reduce the needs of vast labeled images. Through these techniques, this paper successfully trained a 3 billion-parameter Swin Transformer V2 model, which is the largest dense vision model to date, and makes it capable of training with images of up to 1,536\u00d71,536 resolution. It set new performance records on 4 representative vision tasks, including ImageNet-V2 image classification, COCO object detection, ADE20K semantic segmentation, and Kinetics-400 video action classification. Also note our training is much more efficient than that in Google's billion-level visual models, which consumes 40 times less labelled data and 40 times less training time.*\n\nThis model was contributed by [nandwalritik](https://huggingface.co/nandwalritik).\nThe original code can be found [here](https://github.com/microsoft/Swin-Transformer).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Swin Transformer v2.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`Swinv2ForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nBesides that:\n\n- [`Swinv2ForMaskedImageModeling`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## Swinv2Config\n\n[[autodoc]] Swinv2Config\n\n## Swinv2Model\n\n[[autodoc]] Swinv2Model\n    - forward\n\n## Swinv2ForMaskedImageModeling\n\n[[autodoc]] Swinv2ForMaskedImageModeling\n    - forward\n\n## Swinv2ForImageClassification\n\n[[autodoc]] transformers.Swinv2ForImageClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 358f7768-eca1-4ad3-b9d6-2fe9c921b8bd
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/led", "title": "LED", "tokens": 1332, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"358f7768-eca1-4ad3-b9d6-2fe9c921b8bd\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/led\", \"title\": \"LED\", \"tokens\": 1332, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f9573053-9509-4583-b226-9f957fc37593\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/led\", \"title\": \"LED\", \"tokens\": 1332, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6f16b660fabbfc08e155795602a8ab7f5609b196ea8bf201a6a4026553c0d7c3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d02f2d15-418e-45cc-b6b6-47e11b26dbd6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4f2a5129568f2763e46b6b15dad6e397bc22d863f8988e1eacbfaf6ce921818b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3070, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f9573053-9509-4583-b226-9f957fc37593", "doc_id": "f9573053-9509-4583-b226-9f957fc37593", "ref_doc_id": "f9573053-9509-4583-b226-9f957fc37593", "chroma:document": "# LED\n\n## Overview\n\nThe LED model was proposed in [Longformer: The Long-Document Transformer](https://arxiv.org/abs/2004.05150) by Iz\nBeltagy, Matthew E. Peters, Arman Cohan.\n\nThe abstract from the paper is the following:\n\n*Transformer-based models are unable to process long sequences due to their self-attention operation, which scales\nquadratically with the sequence length. To address this limitation, we introduce the Longformer with an attention\nmechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or\nlonger. Longformer's attention mechanism is a drop-in replacement for the standard self-attention and combines a local\nwindowed attention with a task motivated global attention. Following prior work on long-sequence transformers, we\nevaluate Longformer on character-level language modeling and achieve state-of-the-art results on text8 and enwik8. In\ncontrast to most prior work, we also pretrain Longformer and finetune it on a variety of downstream tasks. Our\npretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on\nWikiHop and TriviaQA. We finally introduce the Longformer-Encoder-Decoder (LED), a Longformer variant for supporting\nlong document generative sequence-to-sequence tasks, and demonstrate its effectiveness on the arXiv summarization\ndataset.*\n\n## Usage tips\n\n- [`LEDForConditionalGeneration`] is an extension of\n  [`BartForConditionalGeneration`] exchanging the traditional *self-attention* layer with\n  *Longformer*'s *chunked self-attention* layer. [`LEDTokenizer`] is an alias of\n  [`BartTokenizer`].\n- LED works very well on long-range *sequence-to-sequence* tasks where the `input_ids` largely exceed a length of\n  1024 tokens.\n- LED pads the `input_ids` to be a multiple of `config.attention_window` if required. Therefore a small speed-up is\n  gained, when [`LEDTokenizer`] is used with the `pad_to_multiple_of` argument.\n- LED makes use of *global attention* by means of the `global_attention_mask` (see\n  [`LongformerModel`]). For summarization, it is advised to put *global attention* only on the first\n  `<s>` token. For question answering, it is advised to put *global attention* on all tokens of the question.\n- To fine-tune LED on all 16384, *gradient checkpointing* can be enabled in case training leads to out-of-memory (OOM)\n  errors. This can be done by executing `model.gradient_checkpointing_enable()`. \n Moreover, the `use_cache=False`\n  flag can be used to disable the caching mechanism to save memory.\n- LED is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than\n  the left.\n\nThis model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten).\n\n## Resources\n\n- [A notebook showing how to evaluate LED](https://colab.research.google.com/drive/12INTTR6n64TzS4RrXZxMSXfrOd9Xzamo?usp=sharing).\n- [A notebook showing how to fine-tune LED](https://colab.research.google.com/drive/12LjJazBl7Gam0XBPy_y0CTOJZeZ34c2v?usp=sharing)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: d02f2d15-418e-45cc-b6b6-47e11b26dbd6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/led", "title": "LED", "tokens": 1332, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d02f2d15-418e-45cc-b6b6-47e11b26dbd6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/led\", \"title\": \"LED\", \"tokens\": 1332, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f9573053-9509-4583-b226-9f957fc37593\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/led\", \"title\": \"LED\", \"tokens\": 1332, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6f16b660fabbfc08e155795602a8ab7f5609b196ea8bf201a6a4026553c0d7c3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"358f7768-eca1-4ad3-b9d6-2fe9c921b8bd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/led\", \"title\": \"LED\", \"tokens\": 1332, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d8f4db522a042a98ba9f0700d1e9f7c98553a052f1a26f5be81c0ec6fa890f22\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1597, \"end_char_idx\": 4619, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f9573053-9509-4583-b226-9f957fc37593", "doc_id": "f9573053-9509-4583-b226-9f957fc37593", "ref_doc_id": "f9573053-9509-4583-b226-9f957fc37593", "chroma:document": "[`LEDTokenizer`] is an alias of\n  [`BartTokenizer`].\n- LED works very well on long-range *sequence-to-sequence* tasks where the `input_ids` largely exceed a length of\n  1024 tokens.\n- LED pads the `input_ids` to be a multiple of `config.attention_window` if required. Therefore a small speed-up is\n  gained, when [`LEDTokenizer`] is used with the `pad_to_multiple_of` argument.\n- LED makes use of *global attention* by means of the `global_attention_mask` (see\n  [`LongformerModel`]). For summarization, it is advised to put *global attention* only on the first\n  `<s>` token. For question answering, it is advised to put *global attention* on all tokens of the question.\n- To fine-tune LED on all 16384, *gradient checkpointing* can be enabled in case training leads to out-of-memory (OOM)\n  errors. This can be done by executing `model.gradient_checkpointing_enable()`. \n Moreover, the `use_cache=False`\n  flag can be used to disable the caching mechanism to save memory.\n- LED is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than\n  the left.\n\nThis model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten).\n\n## Resources\n\n- [A notebook showing how to evaluate LED](https://colab.research.google.com/drive/12INTTR6n64TzS4RrXZxMSXfrOd9Xzamo?usp=sharing).\n- [A notebook showing how to fine-tune LED](https://colab.research.google.com/drive/12LjJazBl7Gam0XBPy_y0CTOJZeZ34c2v?usp=sharing).\n- [Text classification task guide](../tasks/sequence_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n## LEDConfig\n\n[[autodoc]] LEDConfig\n\n## LEDTokenizer\n\n[[autodoc]] LEDTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## LEDTokenizerFast\n\n[[autodoc]] LEDTokenizerFast\n\n## LED specific outputs\n\n[[autodoc]] models.led.modeling_led.LEDEncoderBaseModelOutput\n\n[[autodoc]] models.led.modeling_led.LEDSeq2SeqModelOutput\n\n[[autodoc]] models.led.modeling_led.LEDSeq2SeqLMOutput\n\n[[autodoc]] models.led.modeling_led.LEDSeq2SeqSequenceClassifierOutput\n\n[[autodoc]] models.led.modeling_led.LEDSeq2SeqQuestionAnsweringModelOutput\n\n[[autodoc]] models.led.modeling_tf_led.TFLEDEncoderBaseModelOutput\n\n[[autodoc]] models.led.modeling_tf_led.TFLEDSeq2SeqModelOutput\n\n[[autodoc]] models.led.modeling_tf_led.TFLEDSeq2SeqLMOutput\n\n<frameworkcontent>\n<pt>\n\n## LEDModel\n\n[[autodoc]] LEDModel\n    - forward\n\n## LEDForConditionalGeneration\n\n[[autodoc]] LEDForConditionalGeneration\n    - forward\n\n## LEDForSequenceClassification\n\n[[autodoc]] LEDForSequenceClassification\n    - forward\n\n## LEDForQuestionAnswering\n\n[[autodoc]] LEDForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFLEDModel\n\n[[autodoc]] TFLEDModel\n    - call\n\n## TFLEDForConditionalGeneration\n\n[[autodoc]] TFLEDForConditionalGeneration\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: ea4ab563-b980-4fa4-bbdf-35809b1c3130
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/phi3", "title": "Phi-3", "tokens": 1045, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ea4ab563-b980-4fa4-bbdf-35809b1c3130\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi3\", \"title\": \"Phi-3\", \"tokens\": 1045, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"45471918-bdde-4ca9-9e7b-5b003a8f8525\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi3\", \"title\": \"Phi-3\", \"tokens\": 1045, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bd10ad10af0e8b050bfea8d3953598adc79825f145963f32a162c4946178a2bc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e2d588e9-690b-4294-8917-998650565143\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3f0fb0bf75e05b5df4469bf46b388c65e2e7dd434605d83756abf6a6b9c48d61\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3023, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "45471918-bdde-4ca9-9e7b-5b003a8f8525", "doc_id": "45471918-bdde-4ca9-9e7b-5b003a8f8525", "ref_doc_id": "45471918-bdde-4ca9-9e7b-5b003a8f8525", "chroma:document": "# Phi-3\n\n## Overview\n\nThe Phi-3 model was proposed in [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://arxiv.org/abs/2404.14219) by Microsoft.\n\n### Summary\n\nThe abstract from the Phi-3 paper is the following:\n\nWe introduce phi-3-mini, a 3.8 billion parameter language model trained on 3.3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and GPT-3.5 (e.g., phi-3-mini achieves 69% on MMLU and 8.38 on MT-bench), despite being small enough to be deployed on a phone. The innovation lies entirely in our dataset for training, a scaled-up version of the one used for phi-2, composed of heavily filtered web data and synthetic data. The model is also further aligned for robustness, safety, and chat format. We also provide some initial parameter-scaling results with a 7B and 14B models trained for 4.8T tokens, called phi-3-small and phi-3-medium, both significantly more capable than phi-3-mini (e.g., respectively 75% and 78% on MMLU, and 8.7 and 8.9 on MT-bench).\n\nThe original code for Phi-3 can be found [here](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct).\n\n## Usage tips\n\n- This model is very similar to `Llama` with the main difference of [`Phi3SuScaledRotaryEmbedding`] and [`Phi3YarnScaledRotaryEmbedding`], where they are used to extend the context of the rotary embeddings. The query, key and values are fused, and the MLP's up and gate projection layers are also fused.\n- The tokenizer used for this model is identical to the [`LlamaTokenizer`], with the exception of additional tokens.\n\n## How to use Phi-3\n\n<Tip warning={true}>\n\nPhi-3 has been integrated in the development version (4.40.0.dev) of `transformers`. Until the official version is released through `pip`, ensure that you are doing one of the following:\n\n* When loading the model, ensure that `trust_remote_code=True` is passed as an argument of the `from_pretrained()` function.\n\n* Update your local `transformers` to the development version: `pip uninstall -y transformers && pip install git+https://github.com/huggingface/transformers`. The previous command is an alternative to cloning and installing from the source.\n\n</Tip>\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"microsoft/Phi-3-mini-4k-instruct\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"microsoft/Phi-3-mini-4k-instruct\")\n\n>>> messages = [{\"role\": \"user\", \"content\": \"Can you provide ways to eat combinations of bananas and dragonfruits?\"}]\n>>> inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors=\"pt\")\n\n>>> outputs = model.generate(inputs, max_new_tokens=32)\n>>> text = tokenizer.batch_decode(outputs)[0]\n>>> print(text)\n<s><|user|> \nCan you provide ways to eat combinations of bananas and dragonfruits?<|end|> \n<|assistant|> \nCertainly! Bananas and dragonfruits can be combined in various delicious ways."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e2d588e9-690b-4294-8917-998650565143
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/phi3", "title": "Phi-3", "tokens": 1045, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e2d588e9-690b-4294-8917-998650565143\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi3\", \"title\": \"Phi-3\", \"tokens\": 1045, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"45471918-bdde-4ca9-9e7b-5b003a8f8525\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi3\", \"title\": \"Phi-3\", \"tokens\": 1045, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bd10ad10af0e8b050bfea8d3953598adc79825f145963f32a162c4946178a2bc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ea4ab563-b980-4fa4-bbdf-35809b1c3130\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi3\", \"title\": \"Phi-3\", \"tokens\": 1045, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"23f8da943fcf1bf639bae95c6dd74a7f324beac8f61e3b19049bb70c83c8913f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1434, \"end_char_idx\": 3482, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "45471918-bdde-4ca9-9e7b-5b003a8f8525", "doc_id": "45471918-bdde-4ca9-9e7b-5b003a8f8525", "ref_doc_id": "45471918-bdde-4ca9-9e7b-5b003a8f8525", "chroma:document": "The query, key and values are fused, and the MLP's up and gate projection layers are also fused.\n- The tokenizer used for this model is identical to the [`LlamaTokenizer`], with the exception of additional tokens.\n\n## How to use Phi-3\n\n<Tip warning={true}>\n\nPhi-3 has been integrated in the development version (4.40.0.dev) of `transformers`. Until the official version is released through `pip`, ensure that you are doing one of the following:\n\n* When loading the model, ensure that `trust_remote_code=True` is passed as an argument of the `from_pretrained()` function.\n\n* Update your local `transformers` to the development version: `pip uninstall -y transformers && pip install git+https://github.com/huggingface/transformers`. The previous command is an alternative to cloning and installing from the source.\n\n</Tip>\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"microsoft/Phi-3-mini-4k-instruct\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"microsoft/Phi-3-mini-4k-instruct\")\n\n>>> messages = [{\"role\": \"user\", \"content\": \"Can you provide ways to eat combinations of bananas and dragonfruits?\"}]\n>>> inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors=\"pt\")\n\n>>> outputs = model.generate(inputs, max_new_tokens=32)\n>>> text = tokenizer.batch_decode(outputs)[0]\n>>> print(text)\n<s><|user|> \nCan you provide ways to eat combinations of bananas and dragonfruits?<|end|> \n<|assistant|> \nCertainly! Bananas and dragonfruits can be combined in various delicious ways. Here are some ideas for eating combinations of bananas and\n```\n\n## Phi3Config\n\n[[autodoc]] Phi3Config\n\n<frameworkcontent>\n<pt>\n\n## Phi3Model\n\n[[autodoc]] Phi3Model\n    - forward\n\n## Phi3ForCausalLM\n\n[[autodoc]] Phi3ForCausalLM\n    - forward\n    - generate\n\n## Phi3ForSequenceClassification\n\n[[autodoc]] Phi3ForSequenceClassification\n    - forward\n\n## Phi3ForTokenClassification\n\n[[autodoc]] Phi3ForTokenClassification\n    - forward\n\n</pt>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6a0006f6-1752-4aef-826b-2ef38e61320f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/upernet", "title": "UPerNet", "tokens": 993, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6a0006f6-1752-4aef-826b-2ef38e61320f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/upernet\", \"title\": \"UPerNet\", \"tokens\": 993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"721f362b-24da-47bb-90a5-c118c9addf2a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/upernet\", \"title\": \"UPerNet\", \"tokens\": 993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cd643830d234287828240de9e7b16ccfb20df55bcc067de7290ce5dac9ba6b05\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"246a192f-4dec-4e70-a40a-a12ff7dace69\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3468b4d14c4f66f7d2e223f78298ebb162aeca0c46267819de0f7399ae3dcf4d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3143, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "721f362b-24da-47bb-90a5-c118c9addf2a", "doc_id": "721f362b-24da-47bb-90a5-c118c9addf2a", "ref_doc_id": "721f362b-24da-47bb-90a5-c118c9addf2a", "chroma:document": "# UPerNet\n\n## Overview\n\nThe UPerNet model was proposed in [Unified Perceptual Parsing for Scene Understanding](https://arxiv.org/abs/1807.10221)\nby Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, Jian Sun. UPerNet is a general framework to effectively segment\na wide range of concepts from images, leveraging any vision backbone like [ConvNeXt](convnext) or [Swin](swin).\n\nThe abstract from the paper is the following:\n\n*Humans recognize the visual world at multiple levels: we effortlessly categorize scenes and detect objects inside, while also identifying the textures and surfaces of the objects along with their different compositional parts. In this paper, we study a new task called Unified Perceptual Parsing, which requires the machine vision systems to recognize as many visual concepts as possible from a given image. A multi-task framework called UPerNet and a training strategy are developed to learn from heterogeneous image annotations. We benchmark our framework on Unified Perceptual Parsing and show that it is able to effectively segment a wide range of concepts from images. The trained networks are further applied to discover visual knowledge in natural scenes.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/upernet_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> UPerNet framework. Taken from the <a href=\"https://arxiv.org/abs/1807.10221\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code is based on OpenMMLab's mmsegmentation [here](https://github.com/open-mmlab/mmsegmentation/blob/master/mmseg/models/decode_heads/uper_head.py).\n\n## Usage examples\n\nUPerNet is a general framework for semantic segmentation. It can be used with any vision backbone, like so:\n\n```py\nfrom transformers import SwinConfig, UperNetConfig, UperNetForSemanticSegmentation\n\nbackbone_config = SwinConfig(out_features=[\"stage1\", \"stage2\", \"stage3\", \"stage4\"])\n\nconfig = UperNetConfig(backbone_config=backbone_config)\nmodel = UperNetForSemanticSegmentation(config)\n```\n\nTo use another vision backbone, like [ConvNeXt](convnext), simply instantiate the model with the appropriate backbone:\n\n```py\nfrom transformers import ConvNextConfig, UperNetConfig, UperNetForSemanticSegmentation\n\nbackbone_config = ConvNextConfig(out_features=[\"stage1\", \"stage2\", \"stage3\", \"stage4\"])\n\nconfig = UperNetConfig(backbone_config=backbone_config)\nmodel = UperNetForSemanticSegmentation(config)\n```\n\nNote that this will randomly initialize all the weights of the model.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with UPerNet.\n\n- Demo notebooks for UPerNet can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/UPerNet).\n- [`UperNetForSemanticSegmentation`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/semantic-segmentation) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/semantic_segmentation.ipynb)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 246a192f-4dec-4e70-a40a-a12ff7dace69
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/upernet", "title": "UPerNet", "tokens": 993, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"246a192f-4dec-4e70-a40a-a12ff7dace69\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/upernet\", \"title\": \"UPerNet\", \"tokens\": 993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"721f362b-24da-47bb-90a5-c118c9addf2a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/upernet\", \"title\": \"UPerNet\", \"tokens\": 993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cd643830d234287828240de9e7b16ccfb20df55bcc067de7290ce5dac9ba6b05\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6a0006f6-1752-4aef-826b-2ef38e61320f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/upernet\", \"title\": \"UPerNet\", \"tokens\": 993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"38138b6f220a5f7ad2bf742f55a19813bb1c080fdc9850e3b7342f260765aa71\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1542, \"end_char_idx\": 3588, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "721f362b-24da-47bb-90a5-c118c9addf2a", "doc_id": "721f362b-24da-47bb-90a5-c118c9addf2a", "ref_doc_id": "721f362b-24da-47bb-90a5-c118c9addf2a", "chroma:document": "The original code is based on OpenMMLab's mmsegmentation [here](https://github.com/open-mmlab/mmsegmentation/blob/master/mmseg/models/decode_heads/uper_head.py).\n\n## Usage examples\n\nUPerNet is a general framework for semantic segmentation. It can be used with any vision backbone, like so:\n\n```py\nfrom transformers import SwinConfig, UperNetConfig, UperNetForSemanticSegmentation\n\nbackbone_config = SwinConfig(out_features=[\"stage1\", \"stage2\", \"stage3\", \"stage4\"])\n\nconfig = UperNetConfig(backbone_config=backbone_config)\nmodel = UperNetForSemanticSegmentation(config)\n```\n\nTo use another vision backbone, like [ConvNeXt](convnext), simply instantiate the model with the appropriate backbone:\n\n```py\nfrom transformers import ConvNextConfig, UperNetConfig, UperNetForSemanticSegmentation\n\nbackbone_config = ConvNextConfig(out_features=[\"stage1\", \"stage2\", \"stage3\", \"stage4\"])\n\nconfig = UperNetConfig(backbone_config=backbone_config)\nmodel = UperNetForSemanticSegmentation(config)\n```\n\nNote that this will randomly initialize all the weights of the model.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with UPerNet.\n\n- Demo notebooks for UPerNet can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/UPerNet).\n- [`UperNetForSemanticSegmentation`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/semantic-segmentation) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/semantic_segmentation.ipynb).\n- See also: [Semantic segmentation task guide](../tasks/semantic_segmentation)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## UperNetConfig\n\n[[autodoc]] UperNetConfig\n\n## UperNetForSemanticSegmentation\n\n[[autodoc]] UperNetForSemanticSegmentation\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6f080e47-8b11-4a33-ad52-8f0fde3ada87
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/blenderbot", "title": "Blenderbot", "tokens": 1155, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6f080e47-8b11-4a33-ad52-8f0fde3ada87\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blenderbot\", \"title\": \"Blenderbot\", \"tokens\": 1155, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"17f37e06-75ea-443e-994d-1126855f60b0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blenderbot\", \"title\": \"Blenderbot\", \"tokens\": 1155, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"36b1057c151a6bc23415ea0a42657e3f6026eb69a61e5c726366c6a125055b90\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e475039c-88e3-4d47-b206-d63db89d2963\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1671e10b84b6b2ba565fc0072b3d01791f6449488c59a86f81a1d80eef670c47\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3294, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "17f37e06-75ea-443e-994d-1126855f60b0", "doc_id": "17f37e06-75ea-443e-994d-1126855f60b0", "ref_doc_id": "17f37e06-75ea-443e-994d-1126855f60b0", "chroma:document": "# Blenderbot\n\n## Overview\n\nThe Blender chatbot model was proposed in [Recipes for building an open-domain chatbot](https://arxiv.org/pdf/2004.13637.pdf) Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu,\nJing Xu, Myle Ott, Kurt Shuster, Eric M. Smith, Y-Lan Boureau, Jason Weston on 30 Apr 2020.\n\nThe abstract of the paper is the following:\n\n*Building open-domain chatbots is a challenging area for machine learning research. While prior work has shown that\nscaling neural models in the number of parameters and the size of the data they are trained on gives improved results,\nwe show that other ingredients are important for a high-performing chatbot. Good conversation requires a number of\nskills that an expert conversationalist blends in a seamless way: providing engaging talking points and listening to\ntheir partners, and displaying knowledge, empathy and personality appropriately, while maintaining a consistent\npersona. We show that large scale models can learn these skills when given appropriate training data and choice of\ngeneration strategy. We build variants of these recipes with 90M, 2.7B and 9.4B parameter models, and make our models\nand code publicly available. Human evaluations show our best models are superior to existing approaches in multi-turn\ndialogue in terms of engagingness and humanness measurements. We then discuss the limitations of this work by analyzing\nfailure cases of our models.*\n\nThis model was contributed by [sshleifer](https://huggingface.co/sshleifer). The authors' code can be found [here](https://github.com/facebookresearch/ParlAI) .\n\n## Usage tips and example\n\nBlenderbot is a model with absolute position embeddings so it's usually advised to pad the inputs on the right \nrather than the left.\n\nAn example:\n\n```python\n>>> from transformers import BlenderbotTokenizer, BlenderbotForConditionalGeneration\n\n>>> mname = \"facebook/blenderbot-400M-distill\"\n>>> model = BlenderbotForConditionalGeneration.from_pretrained(mname)\n>>> tokenizer = BlenderbotTokenizer.from_pretrained(mname)\n>>> UTTERANCE = \"My friends are cool but they eat too many carbs.\"\n>>> inputs = tokenizer([UTTERANCE], return_tensors=\"pt\")\n>>> reply_ids = model.generate(**inputs)\n>>> print(tokenizer.batch_decode(reply_ids))\n[\"<s> That's unfortunate. Are they trying to lose weight or are they just trying to be healthier?</s>\"]\n```\n\n## Implementation Notes\n\n- Blenderbot uses a standard [seq2seq model transformer](https://arxiv.org/pdf/1706.03762.pdf) based architecture.\n- Available checkpoints can be found in the [model hub](https://huggingface.co/models?search=blenderbot).\n- This is the *default* Blenderbot model class. However, some smaller checkpoints, such as\n  `facebook/blenderbot_small_90M`, have a different architecture and consequently should be used with\n  [BlenderbotSmall](blenderbot-small).\n\n  \n## Resources\n\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n## BlenderbotConfig\n\n[[autodoc]] BlenderbotConfig\n\n## BlenderbotTokenizer\n\n[[autodoc]] BlenderbotTokenizer\n    - build_inputs_with_special_tokens\n\n## BlenderbotTokenizerFast\n\n[[autodoc]] BlenderbotTokenizerFast\n    - build_inputs_with_special_tokens"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e475039c-88e3-4d47-b206-d63db89d2963
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/blenderbot", "title": "Blenderbot", "tokens": 1155, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e475039c-88e3-4d47-b206-d63db89d2963\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blenderbot\", \"title\": \"Blenderbot\", \"tokens\": 1155, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"17f37e06-75ea-443e-994d-1126855f60b0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blenderbot\", \"title\": \"Blenderbot\", \"tokens\": 1155, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"36b1057c151a6bc23415ea0a42657e3f6026eb69a61e5c726366c6a125055b90\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6f080e47-8b11-4a33-ad52-8f0fde3ada87\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blenderbot\", \"title\": \"Blenderbot\", \"tokens\": 1155, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"086afa922d1464c5c9f1c1a4180fe3020cf6b8ead67cd6a88a0969880d5d6cff\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3297, \"end_char_idx\": 4181, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "17f37e06-75ea-443e-994d-1126855f60b0", "doc_id": "17f37e06-75ea-443e-994d-1126855f60b0", "ref_doc_id": "17f37e06-75ea-443e-994d-1126855f60b0", "chroma:document": "<frameworkcontent>\n<pt>\n\n## BlenderbotModel\n\nSee [`~transformers.BartModel`] for arguments to *forward* and *generate*\n\n[[autodoc]] BlenderbotModel\n    - forward\n\n## BlenderbotForConditionalGeneration\n\nSee [`~transformers.BartForConditionalGeneration`] for arguments to *forward* and *generate*\n\n[[autodoc]] BlenderbotForConditionalGeneration\n    - forward\n\n## BlenderbotForCausalLM\n\n[[autodoc]] BlenderbotForCausalLM\n    - forward\n\n</pt>\n<tf>\n\n## TFBlenderbotModel\n\n[[autodoc]] TFBlenderbotModel\n    - call\n\n## TFBlenderbotForConditionalGeneration\n\n[[autodoc]] TFBlenderbotForConditionalGeneration\n    - call\n\n</tf>\n<jax>\n\n## FlaxBlenderbotModel\n\n[[autodoc]] FlaxBlenderbotModel\n    - __call__\n    - encode\n    - decode\n\n## FlaxBlenderbotForConditionalGeneration\n\n[[autodoc]] FlaxBlenderbotForConditionalGeneration\n    - __call__\n    - encode\n    - decode\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 08abde3f-0b2f-4636-b5e6-85bd6fb8fb2b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/splinter", "title": "Splinter", "tokens": 961, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"08abde3f-0b2f-4636-b5e6-85bd6fb8fb2b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/splinter\", \"title\": \"Splinter\", \"tokens\": 961, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"655aebe8-899a-4122-bcf6-e14bf96a21d0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/splinter\", \"title\": \"Splinter\", \"tokens\": 961, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c62477bd3a8359d7de5736bcc1c7e4bc94f775cd16f999207061da2a55290933\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4f80487b-75a6-45fc-af8c-217bccfa48c8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7c86b7e34c683615d4aa424eaea5958ee97a6b3d0998581f868d7d53f1585a1b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2871, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "655aebe8-899a-4122-bcf6-e14bf96a21d0", "doc_id": "655aebe8-899a-4122-bcf6-e14bf96a21d0", "ref_doc_id": "655aebe8-899a-4122-bcf6-e14bf96a21d0", "chroma:document": "# Splinter\n\n## Overview\n\nThe Splinter model was proposed in [Few-Shot Question Answering by Pretraining Span Selection](https://arxiv.org/abs/2101.00438) by Ori Ram, Yuval Kirstain, Jonathan Berant, Amir Globerson, Omer Levy. Splinter\nis an encoder-only transformer (similar to BERT) pretrained using the recurring span selection task on a large corpus\ncomprising Wikipedia and the Toronto Book Corpus.\n\nThe abstract from the paper is the following:\n\nIn several question answering benchmarks, pretrained models have reached human parity through fine-tuning on an order\nof 100,000 annotated questions and answers. We explore the more realistic few-shot setting, where only a few hundred\ntraining examples are available, and observe that standard models perform poorly, highlighting the discrepancy between\ncurrent pretraining objectives and question answering. We propose a new pretraining scheme tailored for question\nanswering: recurring span selection. Given a passage with multiple sets of recurring spans, we mask in each set all\nrecurring spans but one, and ask the model to select the correct span in the passage for each masked span. Masked spans\nare replaced with a special token, viewed as a question representation, that is later used during fine-tuning to select\nthe answer span. The resulting model obtains surprisingly good results on multiple benchmarks (e.g., 72.7 F1 on SQuAD\nwith only 128 training examples), while maintaining competitive performance in the high-resource setting.\n\nThis model was contributed by [yuvalkirstain](https://huggingface.co/yuvalkirstain) and [oriram](https://huggingface.co/oriram). The original code can be found [here](https://github.com/oriram/splinter).\n\n## Usage tips\n\n- Splinter was trained to predict answers spans conditioned on a special [QUESTION] token. These tokens contextualize\n  to question representations which are used to predict the answers. This layer is called QASS, and is the default\n  behaviour in the [`SplinterForQuestionAnswering`] class. Therefore:\n- Use [`SplinterTokenizer`] (rather than [`BertTokenizer`]), as it already\n  contains this special token. Also, its default behavior is to use this token when two sequences are given (for\n  example, in the *run_qa.py* script).\n- If you plan on using Splinter outside *run_qa.py*, please keep in mind the question token - it might be important for\n  the success of your model, especially in a few-shot setting.\n- Please note there are two different checkpoints for each size of Splinter. Both are basically the same, except that\n  one also has the pretrained weights of the QASS layer (*tau/splinter-base-qass* and *tau/splinter-large-qass*) and one\n  doesn't (*tau/splinter-base* and *tau/splinter-large*). This is done to support randomly initializing this layer at\n  fine-tuning, as it is shown to yield better results for some cases in the paper."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 4f80487b-75a6-45fc-af8c-217bccfa48c8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/splinter", "title": "Splinter", "tokens": 961, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4f80487b-75a6-45fc-af8c-217bccfa48c8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/splinter\", \"title\": \"Splinter\", \"tokens\": 961, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"655aebe8-899a-4122-bcf6-e14bf96a21d0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/splinter\", \"title\": \"Splinter\", \"tokens\": 961, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c62477bd3a8359d7de5736bcc1c7e4bc94f775cd16f999207061da2a55290933\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"08abde3f-0b2f-4636-b5e6-85bd6fb8fb2b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/splinter\", \"title\": \"Splinter\", \"tokens\": 961, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a51b64a63b5822aa0dd685583af29628804befb10d84db6ab5442ab5a36d7068\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1291, \"end_char_idx\": 3469, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "655aebe8-899a-4122-bcf6-e14bf96a21d0", "doc_id": "655aebe8-899a-4122-bcf6-e14bf96a21d0", "ref_doc_id": "655aebe8-899a-4122-bcf6-e14bf96a21d0", "chroma:document": "The resulting model obtains surprisingly good results on multiple benchmarks (e.g., 72.7 F1 on SQuAD\nwith only 128 training examples), while maintaining competitive performance in the high-resource setting.\n\nThis model was contributed by [yuvalkirstain](https://huggingface.co/yuvalkirstain) and [oriram](https://huggingface.co/oriram). The original code can be found [here](https://github.com/oriram/splinter).\n\n## Usage tips\n\n- Splinter was trained to predict answers spans conditioned on a special [QUESTION] token. These tokens contextualize\n  to question representations which are used to predict the answers. This layer is called QASS, and is the default\n  behaviour in the [`SplinterForQuestionAnswering`] class. Therefore:\n- Use [`SplinterTokenizer`] (rather than [`BertTokenizer`]), as it already\n  contains this special token. Also, its default behavior is to use this token when two sequences are given (for\n  example, in the *run_qa.py* script).\n- If you plan on using Splinter outside *run_qa.py*, please keep in mind the question token - it might be important for\n  the success of your model, especially in a few-shot setting.\n- Please note there are two different checkpoints for each size of Splinter. Both are basically the same, except that\n  one also has the pretrained weights of the QASS layer (*tau/splinter-base-qass* and *tau/splinter-large-qass*) and one\n  doesn't (*tau/splinter-base* and *tau/splinter-large*). This is done to support randomly initializing this layer at\n  fine-tuning, as it is shown to yield better results for some cases in the paper.\n\n## Resources\n\n- [Question answering task guide](../tasks/question-answering)\n\n## SplinterConfig\n\n[[autodoc]] SplinterConfig\n\n## SplinterTokenizer\n\n[[autodoc]] SplinterTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## SplinterTokenizerFast\n\n[[autodoc]] SplinterTokenizerFast\n\n## SplinterModel\n\n[[autodoc]] SplinterModel\n    - forward\n\n## SplinterForQuestionAnswering\n\n[[autodoc]] SplinterForQuestionAnswering\n    - forward\n\n## SplinterForPreTraining\n\n[[autodoc]] SplinterForPreTraining\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 1901e493-74ed-45e6-b842-8236bd253ac3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/sew-d", "title": "SEW-D", "tokens": 637, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1901e493-74ed-45e6-b842-8236bd253ac3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/sew-d\", \"title\": \"SEW-D\", \"tokens\": 637, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0f3d6593-9057-4868-958f-c4cfc451f1bd\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/sew-d\", \"title\": \"SEW-D\", \"tokens\": 637, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7932e84258bad487dcc35ad86f38719dc0750fa3f4768f89de9adc2e5acbcf9f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1854, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0f3d6593-9057-4868-958f-c4cfc451f1bd", "doc_id": "0f3d6593-9057-4868-958f-c4cfc451f1bd", "ref_doc_id": "0f3d6593-9057-4868-958f-c4cfc451f1bd", "chroma:document": "# SEW-D\n\n## Overview\n\nSEW-D (Squeezed and Efficient Wav2Vec with Disentangled attention) was proposed in [Performance-Efficiency Trade-offs\nin Unsupervised Pre-training for Speech Recognition](https://arxiv.org/abs/2109.06870) by Felix Wu, Kwangyoun Kim,\nJing Pan, Kyu Han, Kilian Q. Weinberger, Yoav Artzi.\n\nThe abstract from the paper is the following:\n\n*This paper is a study of performance-efficiency trade-offs in pre-trained models for automatic speech recognition\n(ASR). We focus on wav2vec 2.0, and formalize several architecture designs that influence both the model performance\nand its efficiency. Putting together all our observations, we introduce SEW (Squeezed and Efficient Wav2vec), a\npre-trained model architecture with significant improvements along both performance and efficiency dimensions across a\nvariety of training setups. For example, under the 100h-960h semi-supervised setup on LibriSpeech, SEW achieves a 1.9x\ninference speedup compared to wav2vec 2.0, with a 13.5% relative reduction in word error rate. With a similar inference\ntime, SEW reduces word error rate by 25-50% across different model sizes.*\n\nThis model was contributed by [anton-l](https://huggingface.co/anton-l).\n\n## Usage tips\n\n- SEW-D is a speech model that accepts a float array corresponding to the raw waveform of the speech signal.\n- SEWDForCTC is fine-tuned using connectionist temporal classification (CTC) so the model output has to be decoded\n  using [`Wav2Vec2CTCTokenizer`].\n\n## Resources\n\n- [Audio classification task guide](../tasks/audio_classification)\n- [Automatic speech recognition task guide](../tasks/asr)\n\n## SEWDConfig\n\n[[autodoc]] SEWDConfig\n\n## SEWDModel\n\n[[autodoc]] SEWDModel\n    - forward\n\n## SEWDForCTC\n\n[[autodoc]] SEWDForCTC\n    - forward\n\n## SEWDForSequenceClassification\n\n[[autodoc]] SEWDForSequenceClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 1eaae299-918c-4640-afce-9cccb64b285c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"1eaae299-918c-4640-afce-9cccb64b285c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f9af8e86-dc56-45f2-b03e-4c0dec92cf4a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0231f515-d82e-41cf-a8cc-958d9c3387d4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b8e74e9d13d72f4bf23603a5fc8ef2560b7c5797615084d281d7ce1ee71f0eb3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3457, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "ref_doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "chroma:document": "# TAPAS\n\n## Overview\n\nThe TAPAS model was proposed in [TAPAS: Weakly Supervised Table Parsing via Pre-training](https://www.aclweb.org/anthology/2020.acl-main.398)\nby Jonathan Herzig, Pawe\u0142 Krzysztof Nowak, Thomas M\u00fcller, Francesco Piccinno and Julian Martin Eisenschlos. It's a BERT-based model specifically \ndesigned (and pre-trained) for answering questions about tabular data. Compared to BERT, TAPAS uses relative position embeddings and has 7 \ntoken types that encode tabular structure. TAPAS is pre-trained on the masked language modeling (MLM) objective on a large dataset comprising \nmillions of tables from English Wikipedia and corresponding texts. \n\nFor question answering, TAPAS has 2 heads on top: a cell selection head and an aggregation head, for (optionally) performing aggregations (such as counting or summing) among selected cells. TAPAS has been fine-tuned on several datasets: \n- [SQA](https://www.microsoft.com/en-us/download/details.aspx?id=54253) (Sequential Question Answering by Microsoft)\n- [WTQ](https://github.com/ppasupat/WikiTableQuestions) (Wiki Table Questions by Stanford University)\n- [WikiSQL](https://github.com/salesforce/WikiSQL) (by Salesforce). \n\nIt achieves state-of-the-art on both SQA and WTQ, while having comparable performance to SOTA on WikiSQL, with a much simpler architecture.\n\nThe abstract from the paper is the following:\n\n*Answering natural language questions over tables is usually seen as a semantic parsing task. To alleviate the collection cost of full logical forms, one popular approach focuses on weak supervision consisting of denotations instead of logical forms. However, training semantic parsers from weak supervision poses difficulties, and in addition, the generated logical forms are only used as an intermediate step prior to retrieving the denotation. In this paper, we present TAPAS, an approach to question answering over tables without generating logical forms. TAPAS trains from weak supervision, and predicts the denotation by selecting table cells and optionally applying a corresponding aggregation operator to such selection. TAPAS extends BERT's architecture to encode tables as input, initializes from an effective joint pre-training of text segments and tables crawled from Wikipedia, and is trained end-to-end. We experiment with three different semantic parsing datasets, and find that TAPAS outperforms or rivals semantic parsing models by improving state-of-the-art accuracy on SQA from 55.1 to 67.2 and performing on par with the state-of-the-art on WIKISQL and WIKITQ, but with a simpler model architecture. We additionally find that transfer learning, which is trivial in our setting, from WIKISQL to WIKITQ, yields 48.7 accuracy, 4.2 points above the state-of-the-art.*\n\nIn addition, the authors have further pre-trained TAPAS to recognize **table entailment**, by creating a balanced dataset of millions of automatically created training examples which are learned in an intermediate step prior to fine-tuning. The authors of TAPAS call this further pre-training intermediate pre-training (since TAPAS is first pre-trained on MLM, and then on another dataset). They found that intermediate pre-training further improves performance on SQA, achieving a new state-of-the-art as well as state-of-the-art on [TabFact](https://github.com/wenhuchen/Table-Fact-Checking), a large-scale dataset with 16k Wikipedia tables for table entailment (a binary classification task)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0231f515-d82e-41cf-a8cc-958d9c3387d4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"0231f515-d82e-41cf-a8cc-958d9c3387d4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f9af8e86-dc56-45f2-b03e-4c0dec92cf4a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1eaae299-918c-4640-afce-9cccb64b285c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"d6b3cdf8449415b8bbe646328e23bedbdef5b9b067de10e76e87a9d3965719f1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0b5d30ca-f7e4-478d-9f6a-079a866be146\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"54f75347f167662c9e9b12b575e19602357874423cd0452b5e98e5f7693efd94\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1628, \"end_char_idx\": 5153, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "ref_doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "chroma:document": "However, training semantic parsers from weak supervision poses difficulties, and in addition, the generated logical forms are only used as an intermediate step prior to retrieving the denotation. In this paper, we present TAPAS, an approach to question answering over tables without generating logical forms. TAPAS trains from weak supervision, and predicts the denotation by selecting table cells and optionally applying a corresponding aggregation operator to such selection. TAPAS extends BERT's architecture to encode tables as input, initializes from an effective joint pre-training of text segments and tables crawled from Wikipedia, and is trained end-to-end. We experiment with three different semantic parsing datasets, and find that TAPAS outperforms or rivals semantic parsing models by improving state-of-the-art accuracy on SQA from 55.1 to 67.2 and performing on par with the state-of-the-art on WIKISQL and WIKITQ, but with a simpler model architecture. We additionally find that transfer learning, which is trivial in our setting, from WIKISQL to WIKITQ, yields 48.7 accuracy, 4.2 points above the state-of-the-art.*\n\nIn addition, the authors have further pre-trained TAPAS to recognize **table entailment**, by creating a balanced dataset of millions of automatically created training examples which are learned in an intermediate step prior to fine-tuning. The authors of TAPAS call this further pre-training intermediate pre-training (since TAPAS is first pre-trained on MLM, and then on another dataset). They found that intermediate pre-training further improves performance on SQA, achieving a new state-of-the-art as well as state-of-the-art on [TabFact](https://github.com/wenhuchen/Table-Fact-Checking), a large-scale dataset with 16k Wikipedia tables for table entailment (a binary classification task). For more details, see their follow-up paper: [Understanding tables with intermediate pre-training](https://www.aclweb.org/anthology/2020.findings-emnlp.27/) by Julian Martin Eisenschlos, Syrine Krichene and Thomas M\u00fcller.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/tapas_architecture.png\"\nalt=\"drawing\" width=\"600\"/> \n\n<small> TAPAS architecture. Taken from the <a href=\"https://ai.googleblog.com/2020/04/using-neural-networks-to-find-answers.html\">original blog post</a>.</small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The Tensorflow version of this model was contributed by [kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/google-research/tapas).\n\n## Usage tips\n\n- TAPAS is a model that uses relative position embeddings by default (restarting the position embeddings at every cell of the table). Note that this is something that was added after the publication of the original TAPAS paper. According to the authors, this usually results in a slightly better performance, and allows you to encode longer sequences without running out of embeddings. This is reflected in the `reset_position_index_per_cell` parameter of [`TapasConfig`], which is set to `True` by default. The default versions of the models available on the [hub](https://huggingface.co/models?search=tapas) all use relative position embeddings. You can still use the ones with absolute position embeddings by passing in an additional argument `revision=\"no_reset\"` when calling the `from_pretrained()` method. Note that it's usually advised to pad the inputs on the right rather than the left."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0b5d30ca-f7e4-478d-9f6a-079a866be146
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"0b5d30ca-f7e4-478d-9f6a-079a866be146\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f9af8e86-dc56-45f2-b03e-4c0dec92cf4a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0231f515-d82e-41cf-a8cc-958d9c3387d4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"c5c67b023e5d77d4ef4825f5cda00f03a40a730ffba7398b1d44095406639359\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8137dc2f-8b41-4c8e-908f-fdd9710df52f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9e7e7fff44098dce7ab89839730c10a6d7350eb0dbf2808b4123757f6337765e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3458, \"end_char_idx\": 6633, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "ref_doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "chroma:document": "For more details, see their follow-up paper: [Understanding tables with intermediate pre-training](https://www.aclweb.org/anthology/2020.findings-emnlp.27/) by Julian Martin Eisenschlos, Syrine Krichene and Thomas M\u00fcller.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/tapas_architecture.png\"\nalt=\"drawing\" width=\"600\"/> \n\n<small> TAPAS architecture. Taken from the <a href=\"https://ai.googleblog.com/2020/04/using-neural-networks-to-find-answers.html\">original blog post</a>.</small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The Tensorflow version of this model was contributed by [kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/google-research/tapas).\n\n## Usage tips\n\n- TAPAS is a model that uses relative position embeddings by default (restarting the position embeddings at every cell of the table). Note that this is something that was added after the publication of the original TAPAS paper. According to the authors, this usually results in a slightly better performance, and allows you to encode longer sequences without running out of embeddings. This is reflected in the `reset_position_index_per_cell` parameter of [`TapasConfig`], which is set to `True` by default. The default versions of the models available on the [hub](https://huggingface.co/models?search=tapas) all use relative position embeddings. You can still use the ones with absolute position embeddings by passing in an additional argument `revision=\"no_reset\"` when calling the `from_pretrained()` method. Note that it's usually advised to pad the inputs on the right rather than the left.\n- TAPAS is based on BERT, so `TAPAS-base` for example corresponds to a `BERT-base` architecture. Of course, `TAPAS-large` will result in the best performance (the results reported in the paper are from `TAPAS-large`). Results of the various sized models are shown on the [original GitHub repository](https://github.com/google-research/tapas).\n- TAPAS has checkpoints fine-tuned on SQA, which are capable of answering questions related to a table in a conversational set-up. This means that you can ask follow-up questions such as \"what is his age?\" related to the previous question. Note that the forward pass of TAPAS is a bit different in case of a conversational set-up: in that case, you have to feed every table-question pair one by one to the model, such that the `prev_labels` token type ids can be overwritten by the predicted `labels` of the model to the previous question. See \"Usage\" section for more info.\n- TAPAS is similar to BERT and therefore relies on the masked language modeling (MLM) objective. It is therefore efficient at predicting masked tokens and at NLU in general, but is not optimal for text generation. Models trained with a causal language modeling (CLM) objective are better in that regard. Note that TAPAS can be used as an encoder in the EncoderDecoderModel framework, to combine it with an autoregressive text decoder such as GPT-2.\n\n## Usage: fine-tuning\n\nHere we explain how you can fine-tune [`TapasForQuestionAnswering`] on your own dataset."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 8137dc2f-8b41-4c8e-908f-fdd9710df52f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"8137dc2f-8b41-4c8e-908f-fdd9710df52f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f9af8e86-dc56-45f2-b03e-4c0dec92cf4a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0b5d30ca-f7e4-478d-9f6a-079a866be146\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"53b4fd07ba325d6af22ef1bdf229cf88a811f6466c00b0f19f5e20000ad0dc14\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"919ff337-caff-41aa-a204-9796c9436568\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5853700e59e5aaa95a01af1e6d806a23720f14a9e488a4c321261854b4657eb3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5070, \"end_char_idx\": 8015, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "ref_doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "chroma:document": "Note that it's usually advised to pad the inputs on the right rather than the left.\n- TAPAS is based on BERT, so `TAPAS-base` for example corresponds to a `BERT-base` architecture. Of course, `TAPAS-large` will result in the best performance (the results reported in the paper are from `TAPAS-large`). Results of the various sized models are shown on the [original GitHub repository](https://github.com/google-research/tapas).\n- TAPAS has checkpoints fine-tuned on SQA, which are capable of answering questions related to a table in a conversational set-up. This means that you can ask follow-up questions such as \"what is his age?\" related to the previous question. Note that the forward pass of TAPAS is a bit different in case of a conversational set-up: in that case, you have to feed every table-question pair one by one to the model, such that the `prev_labels` token type ids can be overwritten by the predicted `labels` of the model to the previous question. See \"Usage\" section for more info.\n- TAPAS is similar to BERT and therefore relies on the masked language modeling (MLM) objective. It is therefore efficient at predicting masked tokens and at NLU in general, but is not optimal for text generation. Models trained with a causal language modeling (CLM) objective are better in that regard. Note that TAPAS can be used as an encoder in the EncoderDecoderModel framework, to combine it with an autoregressive text decoder such as GPT-2.\n\n## Usage: fine-tuning\n\nHere we explain how you can fine-tune [`TapasForQuestionAnswering`] on your own dataset.\n\n**STEP 1: Choose one of the 3 ways in which you can use TAPAS - or experiment**\n\nBasically, there are 3 different ways in which one can fine-tune [`TapasForQuestionAnswering`], corresponding to the different datasets on which Tapas was fine-tuned:\n\n1. SQA: if you're interested in asking follow-up questions related to a table, in a conversational set-up. For example if you first ask \"what's the name of the first actor?\" then you can ask a follow-up question such as \"how old is he?\". Here, questions do not involve any aggregation (all questions are cell selection questions).\n2. WTQ: if you're not interested in asking questions in a conversational set-up, but rather just asking questions related to a table, which might involve aggregation, such as counting a number of rows, summing up cell values or averaging cell values. You can then for example ask \"what's the total number of goals Cristiano Ronaldo made in his career?\". This case is also called **weak supervision**, since the model itself must learn the appropriate aggregation operator (SUM/COUNT/AVERAGE/NONE) given only the answer to the question as supervision.\n3. WikiSQL-supervised: this dataset is based on WikiSQL with the model being given the ground truth aggregation operator during training. This is also called **strong supervision**. Here, learning the appropriate aggregation operator is much easier."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 919ff337-caff-41aa-a204-9796c9436568
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"919ff337-caff-41aa-a204-9796c9436568\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f9af8e86-dc56-45f2-b03e-4c0dec92cf4a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8137dc2f-8b41-4c8e-908f-fdd9710df52f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"815cca00555d4e49fb99d6f06cd5627dfd7d9a9b09a739e7cefab84e6b0e10d7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9097cdfa-94dd-4fb3-baae-5b117d51031d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"07c2ee055ce6f362a7410769e0cd6fd2cd1852b46c1f2867f2f2e1141acb5fcf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6286, \"end_char_idx\": 10123, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "ref_doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "chroma:document": "Models trained with a causal language modeling (CLM) objective are better in that regard. Note that TAPAS can be used as an encoder in the EncoderDecoderModel framework, to combine it with an autoregressive text decoder such as GPT-2.\n\n## Usage: fine-tuning\n\nHere we explain how you can fine-tune [`TapasForQuestionAnswering`] on your own dataset.\n\n**STEP 1: Choose one of the 3 ways in which you can use TAPAS - or experiment**\n\nBasically, there are 3 different ways in which one can fine-tune [`TapasForQuestionAnswering`], corresponding to the different datasets on which Tapas was fine-tuned:\n\n1. SQA: if you're interested in asking follow-up questions related to a table, in a conversational set-up. For example if you first ask \"what's the name of the first actor?\" then you can ask a follow-up question such as \"how old is he?\". Here, questions do not involve any aggregation (all questions are cell selection questions).\n2. WTQ: if you're not interested in asking questions in a conversational set-up, but rather just asking questions related to a table, which might involve aggregation, such as counting a number of rows, summing up cell values or averaging cell values. You can then for example ask \"what's the total number of goals Cristiano Ronaldo made in his career?\". This case is also called **weak supervision**, since the model itself must learn the appropriate aggregation operator (SUM/COUNT/AVERAGE/NONE) given only the answer to the question as supervision.\n3. WikiSQL-supervised: this dataset is based on WikiSQL with the model being given the ground truth aggregation operator during training. This is also called **strong supervision**. Here, learning the appropriate aggregation operator is much easier.\n\nTo summarize:\n\n| **Task**                            | **Example dataset** | **Description**                                                                                         |\n|-------------------------------------|---------------------|---------------------------------------------------------------------------------------------------------|\n| Conversational                      | SQA                 | Conversational, only cell selection questions                                                           |\n| Weak supervision for aggregation    | WTQ                 | Questions might involve aggregation, and the model must learn this given only the answer as supervision |\n| Strong supervision for aggregation  | WikiSQL-supervised  | Questions might involve aggregation, and the model must learn this given the gold aggregation operator  |\n\n<frameworkcontent>\n<pt>\nInitializing a model with a pre-trained base and randomly initialized classification heads from the hub can be done as shown below.\n\n```py\n>>> from transformers import TapasConfig, TapasForQuestionAnswering\n\n>>> # for example, the base sized model with default SQA configuration\n>>> model = TapasForQuestionAnswering.from_pretrained(\"google/tapas-base\")\n\n>>> # or, the base sized model with WTQ configuration\n>>> config = TapasConfig.from_pretrained(\"google/tapas-base-finetuned-wtq\")\n>>> model = TapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n\n>>> # or, the base sized model with WikiSQL configuration\n>>> config = TapasConfig(\"google-base-finetuned-wikisql-supervised\")\n>>> model = TapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n```\n\nOf course, you don't necessarily have to follow one of these three ways in which TAPAS was fine-tuned. You can also experiment by defining any hyperparameters you want when initializing [`TapasConfig`], and then create a [`TapasForQuestionAnswering`] based on that configuration. For example, if you have a dataset that has both conversational questions and questions that might involve aggregation, then you can do it this way."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 9097cdfa-94dd-4fb3-baae-5b117d51031d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"9097cdfa-94dd-4fb3-baae-5b117d51031d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f9af8e86-dc56-45f2-b03e-4c0dec92cf4a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"919ff337-caff-41aa-a204-9796c9436568\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"f4f4f0d59938eaf215ec69c506e83cc666ba1a7ab549265cf9d18100f1754588\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6e79ec25-5c74-41bb-969b-1b9390be12fa\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6bae031756ce043224ba32faee981850fd67269f509027fa69a07b2471954c4b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8017, \"end_char_idx\": 11784, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "ref_doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "chroma:document": "To summarize:\n\n| **Task**                            | **Example dataset** | **Description**                                                                                         |\n|-------------------------------------|---------------------|---------------------------------------------------------------------------------------------------------|\n| Conversational                      | SQA                 | Conversational, only cell selection questions                                                           |\n| Weak supervision for aggregation    | WTQ                 | Questions might involve aggregation, and the model must learn this given only the answer as supervision |\n| Strong supervision for aggregation  | WikiSQL-supervised  | Questions might involve aggregation, and the model must learn this given the gold aggregation operator  |\n\n<frameworkcontent>\n<pt>\nInitializing a model with a pre-trained base and randomly initialized classification heads from the hub can be done as shown below.\n\n```py\n>>> from transformers import TapasConfig, TapasForQuestionAnswering\n\n>>> # for example, the base sized model with default SQA configuration\n>>> model = TapasForQuestionAnswering.from_pretrained(\"google/tapas-base\")\n\n>>> # or, the base sized model with WTQ configuration\n>>> config = TapasConfig.from_pretrained(\"google/tapas-base-finetuned-wtq\")\n>>> model = TapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n\n>>> # or, the base sized model with WikiSQL configuration\n>>> config = TapasConfig(\"google-base-finetuned-wikisql-supervised\")\n>>> model = TapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n```\n\nOf course, you don't necessarily have to follow one of these three ways in which TAPAS was fine-tuned. You can also experiment by defining any hyperparameters you want when initializing [`TapasConfig`], and then create a [`TapasForQuestionAnswering`] based on that configuration. For example, if you have a dataset that has both conversational questions and questions that might involve aggregation, then you can do it this way. Here's an example:\n\n```py\n>>> from transformers import TapasConfig, TapasForQuestionAnswering\n\n>>> # you can initialize the classification heads any way you want (see docs of TapasConfig)\n>>> config = TapasConfig(num_aggregation_labels=3, average_logits_per_cell=True)\n>>> # initializing the pre-trained base sized model with our custom classification heads\n>>> model = TapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n```\n</pt>\n<tf>\nInitializing a model with a pre-trained base and randomly initialized classification heads from the hub can be done as shown below. Be sure to have installed the [tensorflow_probability](https://github.com/tensorflow/probability) dependency:\n\n```py\n>>> from transformers import TapasConfig, TFTapasForQuestionAnswering\n\n>>> # for example, the base sized model with default SQA configuration\n>>> model = TFTapasForQuestionAnswering.from_pretrained(\"google/tapas-base\")\n\n>>> # or, the base sized model with WTQ configuration\n>>> config = TapasConfig.from_pretrained(\"google/tapas-base-finetuned-wtq\")\n>>> model = TFTapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n\n>>> # or, the base sized model with WikiSQL configuration\n>>> config = TapasConfig(\"google-base-finetuned-wikisql-supervised\")\n>>> model = TFTapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n```\n\nOf course, you don't necessarily have to follow one of these three ways in which TAPAS was fine-tuned. You can also experiment by defining any hyperparameters you want when initializing [`TapasConfig`], and then create a [`TFTapasForQuestionAnswering`] based on that configuration."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6e79ec25-5c74-41bb-969b-1b9390be12fa
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"6e79ec25-5c74-41bb-969b-1b9390be12fa\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f9af8e86-dc56-45f2-b03e-4c0dec92cf4a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9097cdfa-94dd-4fb3-baae-5b117d51031d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"a2f98c5841119cccaec3faf68f8769ed6e19592332ca63edf45023e853fcdcd0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"197de4b7-4c2f-4fb6-9bec-a94ac738425e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"729ddea90a9e793940fcc8f071cd5c8cec97e6b6c3d3641c1020104c276daeeb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10124, \"end_char_idx\": 13467, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "ref_doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "chroma:document": "Here's an example:\n\n```py\n>>> from transformers import TapasConfig, TapasForQuestionAnswering\n\n>>> # you can initialize the classification heads any way you want (see docs of TapasConfig)\n>>> config = TapasConfig(num_aggregation_labels=3, average_logits_per_cell=True)\n>>> # initializing the pre-trained base sized model with our custom classification heads\n>>> model = TapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n```\n</pt>\n<tf>\nInitializing a model with a pre-trained base and randomly initialized classification heads from the hub can be done as shown below. Be sure to have installed the [tensorflow_probability](https://github.com/tensorflow/probability) dependency:\n\n```py\n>>> from transformers import TapasConfig, TFTapasForQuestionAnswering\n\n>>> # for example, the base sized model with default SQA configuration\n>>> model = TFTapasForQuestionAnswering.from_pretrained(\"google/tapas-base\")\n\n>>> # or, the base sized model with WTQ configuration\n>>> config = TapasConfig.from_pretrained(\"google/tapas-base-finetuned-wtq\")\n>>> model = TFTapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n\n>>> # or, the base sized model with WikiSQL configuration\n>>> config = TapasConfig(\"google-base-finetuned-wikisql-supervised\")\n>>> model = TFTapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n```\n\nOf course, you don't necessarily have to follow one of these three ways in which TAPAS was fine-tuned. You can also experiment by defining any hyperparameters you want when initializing [`TapasConfig`], and then create a [`TFTapasForQuestionAnswering`] based on that configuration. For example, if you have a dataset that has both conversational questions and questions that might involve aggregation, then you can do it this way. Here's an example:\n\n```py\n>>> from transformers import TapasConfig, TFTapasForQuestionAnswering\n\n>>> # you can initialize the classification heads any way you want (see docs of TapasConfig)\n>>> config = TapasConfig(num_aggregation_labels=3, average_logits_per_cell=True)\n>>> # initializing the pre-trained base sized model with our custom classification heads\n>>> model = TFTapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n```\n</tf>\n</frameworkcontent>\n\nWhat you can also do is start from an already fine-tuned checkpoint. A note here is that the already fine-tuned checkpoint on WTQ has some issues due to the L2-loss which is somewhat brittle. See [here](https://github.com/google-research/tapas/issues/91#issuecomment-735719340) for more info.\n\nFor a list of all pre-trained and fine-tuned TAPAS checkpoints available on HuggingFace's  hub, see [here](https://huggingface.co/models?search=tapas).\n\n**STEP 2: Prepare your data in the SQA format**\n\nSecond, no matter what you picked above, you should prepare your dataset in the [SQA](https://www.microsoft.com/en-us/download/details.aspx?id=54253) format. This format is a TSV/CSV file with the following columns:\n\n- `id`: optional, id of the table-question pair, for bookkeeping purposes.\n- `annotator`: optional, id of the person who annotated the table-question pair, for bookkeeping purposes.\n- `position`: integer indicating if the question is the first, second, third,... related to the table. Only required in case of conversational setup (SQA)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 197de4b7-4c2f-4fb6-9bec-a94ac738425e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"197de4b7-4c2f-4fb6-9bec-a94ac738425e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f9af8e86-dc56-45f2-b03e-4c0dec92cf4a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6e79ec25-5c74-41bb-969b-1b9390be12fa\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"27ff86a8e9e17bf77ea19de0b8d828369a12cf2267cb19533f4965482377d8c2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"35b5cb87-a786-4c60-82af-335a6bdf07a8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"05338a496a08ff547ae8a53b5860be98e6963b991e8c251e144094cd293cd2fd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11785, \"end_char_idx\": 14917, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "ref_doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "chroma:document": "For example, if you have a dataset that has both conversational questions and questions that might involve aggregation, then you can do it this way. Here's an example:\n\n```py\n>>> from transformers import TapasConfig, TFTapasForQuestionAnswering\n\n>>> # you can initialize the classification heads any way you want (see docs of TapasConfig)\n>>> config = TapasConfig(num_aggregation_labels=3, average_logits_per_cell=True)\n>>> # initializing the pre-trained base sized model with our custom classification heads\n>>> model = TFTapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n```\n</tf>\n</frameworkcontent>\n\nWhat you can also do is start from an already fine-tuned checkpoint. A note here is that the already fine-tuned checkpoint on WTQ has some issues due to the L2-loss which is somewhat brittle. See [here](https://github.com/google-research/tapas/issues/91#issuecomment-735719340) for more info.\n\nFor a list of all pre-trained and fine-tuned TAPAS checkpoints available on HuggingFace's  hub, see [here](https://huggingface.co/models?search=tapas).\n\n**STEP 2: Prepare your data in the SQA format**\n\nSecond, no matter what you picked above, you should prepare your dataset in the [SQA](https://www.microsoft.com/en-us/download/details.aspx?id=54253) format. This format is a TSV/CSV file with the following columns:\n\n- `id`: optional, id of the table-question pair, for bookkeeping purposes.\n- `annotator`: optional, id of the person who annotated the table-question pair, for bookkeeping purposes.\n- `position`: integer indicating if the question is the first, second, third,... related to the table. Only required in case of conversational setup (SQA). You don't need this column in case you're going for WTQ/WikiSQL-supervised.\n- `question`: string\n- `table_file`: string, name of a csv file containing the tabular data\n- `answer_coordinates`: list of one or more tuples (each tuple being a cell coordinate, i.e. row, column pair that is part of the answer)\n- `answer_text`: list of one or more strings (each string being a cell value that is part of the answer)\n- `aggregation_label`: index of the aggregation operator. Only required in case of strong supervision for aggregation (the WikiSQL-supervised case)\n- `float_answer`: the float answer to the question, if there is one (np.nan if there isn't). Only required in case of weak supervision for aggregation (such as WTQ and WikiSQL)\n\nThe tables themselves should be present in a folder, each table being a separate csv file. Note that the authors of the TAPAS algorithm used conversion scripts with some automated logic to convert the other datasets (WTQ, WikiSQL) into the SQA format. The author explains this [here](https://github.com/google-research/tapas/issues/50#issuecomment-705465960). A conversion of this script that works with HuggingFace's implementation can be found [here](https://github.com/NielsRogge/tapas_utils). Interestingly, these conversion scripts are not perfect (the `answer_coordinates` and `float_answer` fields are populated based on the `answer_text`), meaning that WTQ and WikiSQL results could actually be improved."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 35b5cb87-a786-4c60-82af-335a6bdf07a8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"35b5cb87-a786-4c60-82af-335a6bdf07a8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f9af8e86-dc56-45f2-b03e-4c0dec92cf4a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"197de4b7-4c2f-4fb6-9bec-a94ac738425e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"ef8ce7b6fc8cd8e36015416868943c87a105d5d92367cff769a952cc9e61d992\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3feb354d-a3fa-48d8-91b5-f76e117d1fe1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2ad94ff4bcb398e41591ae5d15388101156198403251862b34b3b4bf1f7e1630\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13205, \"end_char_idx\": 16801, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "ref_doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "chroma:document": "- `annotator`: optional, id of the person who annotated the table-question pair, for bookkeeping purposes.\n- `position`: integer indicating if the question is the first, second, third,... related to the table. Only required in case of conversational setup (SQA). You don't need this column in case you're going for WTQ/WikiSQL-supervised.\n- `question`: string\n- `table_file`: string, name of a csv file containing the tabular data\n- `answer_coordinates`: list of one or more tuples (each tuple being a cell coordinate, i.e. row, column pair that is part of the answer)\n- `answer_text`: list of one or more strings (each string being a cell value that is part of the answer)\n- `aggregation_label`: index of the aggregation operator. Only required in case of strong supervision for aggregation (the WikiSQL-supervised case)\n- `float_answer`: the float answer to the question, if there is one (np.nan if there isn't). Only required in case of weak supervision for aggregation (such as WTQ and WikiSQL)\n\nThe tables themselves should be present in a folder, each table being a separate csv file. Note that the authors of the TAPAS algorithm used conversion scripts with some automated logic to convert the other datasets (WTQ, WikiSQL) into the SQA format. The author explains this [here](https://github.com/google-research/tapas/issues/50#issuecomment-705465960). A conversion of this script that works with HuggingFace's implementation can be found [here](https://github.com/NielsRogge/tapas_utils). Interestingly, these conversion scripts are not perfect (the `answer_coordinates` and `float_answer` fields are populated based on the `answer_text`), meaning that WTQ and WikiSQL results could actually be improved.\n\n**STEP 3: Convert your data into tensors using TapasTokenizer**\n\n<frameworkcontent>\n<pt>\nThird, given that you've prepared your data in this TSV/CSV format (and corresponding CSV files containing the tabular data), you can then use [`TapasTokenizer`] to convert table-question pairs into `input_ids`, `attention_mask`, `token_type_ids` and so on. Again, based on which of the three cases you picked above, [`TapasForQuestionAnswering`] requires different\ninputs to be fine-tuned:\n\n| **Task**                           | **Required inputs**                                                                                                 |\n|------------------------------------|---------------------------------------------------------------------------------------------------------------------|\n| Conversational                     | `input_ids`, `attention_mask`, `token_type_ids`, `labels`                                                           |\n|  Weak supervision for aggregation  | `input_ids`, `attention_mask`, `token_type_ids`, `labels`, `numeric_values`, `numeric_values_scale`, `float_answer` |\n| Strong supervision for aggregation | `input ids`, `attention mask`, `token type ids`, `labels`, `aggregation_labels`                                     |\n\n[`TapasTokenizer`] creates the `labels`, `numeric_values` and `numeric_values_scale` based on the `answer_coordinates` and `answer_text` columns of the TSV file. The `float_answer` and `aggregation_labels` are already in the TSV file of step 2. Here's an example:\n\n```py\n>>> from transformers import TapasTokenizer\n>>> import pandas as pd\n\n>>> model_name = \"google/tapas-base\"\n>>> tokenizer = TapasTokenizer.from_pretrained(model_name)\n\n>>> data = {\"Actors\": [\"Brad Pitt\", \"Leonardo Di Caprio\", \"George Clooney\"], \"Number of movies\": [\"87\", \"53\", \"69\"]}\n>>> queries = [\n...     \"What is the name of the first actor?"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 3feb354d-a3fa-48d8-91b5-f76e117d1fe1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"3feb354d-a3fa-48d8-91b5-f76e117d1fe1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f9af8e86-dc56-45f2-b03e-4c0dec92cf4a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"35b5cb87-a786-4c60-82af-335a6bdf07a8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"197055e2896a9033362415f589ac2647bd715cce8e27b70f6190a9fc6a4da435\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cfc9bbf4-cb35-49bd-8102-70b6896cd8a3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1aca70a288eb127f649e47744011c0fccf867966ce500164d4cab620c34e7516\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14919, \"end_char_idx\": 17968, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "ref_doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "chroma:document": "**STEP 3: Convert your data into tensors using TapasTokenizer**\n\n<frameworkcontent>\n<pt>\nThird, given that you've prepared your data in this TSV/CSV format (and corresponding CSV files containing the tabular data), you can then use [`TapasTokenizer`] to convert table-question pairs into `input_ids`, `attention_mask`, `token_type_ids` and so on. Again, based on which of the three cases you picked above, [`TapasForQuestionAnswering`] requires different\ninputs to be fine-tuned:\n\n| **Task**                           | **Required inputs**                                                                                                 |\n|------------------------------------|---------------------------------------------------------------------------------------------------------------------|\n| Conversational                     | `input_ids`, `attention_mask`, `token_type_ids`, `labels`                                                           |\n|  Weak supervision for aggregation  | `input_ids`, `attention_mask`, `token_type_ids`, `labels`, `numeric_values`, `numeric_values_scale`, `float_answer` |\n| Strong supervision for aggregation | `input ids`, `attention mask`, `token type ids`, `labels`, `aggregation_labels`                                     |\n\n[`TapasTokenizer`] creates the `labels`, `numeric_values` and `numeric_values_scale` based on the `answer_coordinates` and `answer_text` columns of the TSV file. The `float_answer` and `aggregation_labels` are already in the TSV file of step 2. Here's an example:\n\n```py\n>>> from transformers import TapasTokenizer\n>>> import pandas as pd\n\n>>> model_name = \"google/tapas-base\"\n>>> tokenizer = TapasTokenizer.from_pretrained(model_name)\n\n>>> data = {\"Actors\": [\"Brad Pitt\", \"Leonardo Di Caprio\", \"George Clooney\"], \"Number of movies\": [\"87\", \"53\", \"69\"]}\n>>> queries = [\n...     \"What is the name of the first actor?\",\n...     \"How many movies has George Clooney played in?\",\n...     \"What is the total number of movies?\",\n... ]\n>>> answer_coordinates = [[(0, 0)], [(2, 1)], [(0, 1), (1, 1), (2, 1)]]\n>>> answer_text = [[\"Brad Pitt\"], [\"69\"], [\"209\"]]\n>>> table = pd.DataFrame.from_dict(data)\n>>> inputs = tokenizer(\n...     table=table,\n...     queries=queries,\n...     answer_coordinates=answer_coordinates,\n...     answer_text=answer_text,\n...     padding=\"max_length\",\n...     return_tensors=\"pt\",\n... )\n>>> inputs\n{'input_ids': tensor([[ ... ]]), 'attention_mask': tensor([[...]]), 'token_type_ids': tensor([[[...]]]),\n'numeric_values': tensor([[ ... ]]), 'numeric_values_scale: tensor([[ ... ]]), labels: tensor([[ ... ]])}\n```\n\nNote that [`TapasTokenizer`] expects the data of the table to be **text-only**. You can use `.astype(str)` on a dataframe to turn it into text-only data.\nOf course, this only shows how to encode a single training example. It is advised to create a dataloader to iterate over batches:\n\n```py\n>>> import torch\n>>> import pandas as pd\n\n>>> tsv_path = \"your_path_to_the_tsv_file\"\n>>> table_csv_path = \"your_path_to_a_directory_containing_all_csv_files\""}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: cfc9bbf4-cb35-49bd-8102-70b6896cd8a3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"cfc9bbf4-cb35-49bd-8102-70b6896cd8a3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f9af8e86-dc56-45f2-b03e-4c0dec92cf4a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3feb354d-a3fa-48d8-91b5-f76e117d1fe1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"644bc8e515abf499a9c8ca27f59b15cd173f241d13ec900295ad21382409ea96\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"05d6bbe7-3387-4c08-be1c-ed57625792d4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"10ab27f0dffc801c0708d0f5be38c1d5b4d944fce26212363e9b01a37c3921c3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 16801, \"end_char_idx\": 19106, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "ref_doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "chroma:document": "\",\n...     \"How many movies has George Clooney played in?\",\n...     \"What is the total number of movies?\",\n... ]\n>>> answer_coordinates = [[(0, 0)], [(2, 1)], [(0, 1), (1, 1), (2, 1)]]\n>>> answer_text = [[\"Brad Pitt\"], [\"69\"], [\"209\"]]\n>>> table = pd.DataFrame.from_dict(data)\n>>> inputs = tokenizer(\n...     table=table,\n...     queries=queries,\n...     answer_coordinates=answer_coordinates,\n...     answer_text=answer_text,\n...     padding=\"max_length\",\n...     return_tensors=\"pt\",\n... )\n>>> inputs\n{'input_ids': tensor([[ ... ]]), 'attention_mask': tensor([[...]]), 'token_type_ids': tensor([[[...]]]),\n'numeric_values': tensor([[ ... ]]), 'numeric_values_scale: tensor([[ ... ]]), labels: tensor([[ ... ]])}\n```\n\nNote that [`TapasTokenizer`] expects the data of the table to be **text-only**. You can use `.astype(str)` on a dataframe to turn it into text-only data.\nOf course, this only shows how to encode a single training example. It is advised to create a dataloader to iterate over batches:\n\n```py\n>>> import torch\n>>> import pandas as pd\n\n>>> tsv_path = \"your_path_to_the_tsv_file\"\n>>> table_csv_path = \"your_path_to_a_directory_containing_all_csv_files\"\n\n\n>>> class TableDataset(torch.utils.data.Dataset):\n...     def __init__(self, data, tokenizer):\n...         self.data = data\n...         self.tokenizer = tokenizer\n\n...     def __getitem__(self, idx):\n...         item = data.iloc[idx]\n...         table = pd.read_csv(table_csv_path + item.table_file).astype(\n...             str\n...         )  # be sure to make your table data text only\n...         encoding = self.tokenizer(\n...             table=table,\n...             queries=item.question,\n...             answer_coordinates=item.answer_coordinates,\n...             answer_text=item.answer_text,\n...             truncation=True,\n...             padding=\"max_length\",\n...             return_tensors=\"pt\",\n...         )\n...         # remove the batch dimension which the tokenizer adds by default\n...         encoding = {key: val.squeeze(0) for key, val in encoding.items()}\n...         # add the float_answer which is also required (weak supervision for aggregation case)\n...         encoding[\"float_answer\"] = torch.tensor(item.float_answer)\n...         return encoding\n\n...     def __len__(self):\n...         return len(self.data)"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 05d6bbe7-3387-4c08-be1c-ed57625792d4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"05d6bbe7-3387-4c08-be1c-ed57625792d4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f9af8e86-dc56-45f2-b03e-4c0dec92cf4a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cfc9bbf4-cb35-49bd-8102-70b6896cd8a3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"22befb726717d656ae180ac0772117d7bf38d69df35118d15f74fd8f5aea19eb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"af66a608-9b0e-49b6-a33c-53304ba83a6c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5490a13e6d08534c3f45d0017a5a95613556f89727d1492687d4ad4cac0f95ab\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 17600, \"end_char_idx\": 22271, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "ref_doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "chroma:document": "You can use `.astype(str)` on a dataframe to turn it into text-only data.\nOf course, this only shows how to encode a single training example. It is advised to create a dataloader to iterate over batches:\n\n```py\n>>> import torch\n>>> import pandas as pd\n\n>>> tsv_path = \"your_path_to_the_tsv_file\"\n>>> table_csv_path = \"your_path_to_a_directory_containing_all_csv_files\"\n\n\n>>> class TableDataset(torch.utils.data.Dataset):\n...     def __init__(self, data, tokenizer):\n...         self.data = data\n...         self.tokenizer = tokenizer\n\n...     def __getitem__(self, idx):\n...         item = data.iloc[idx]\n...         table = pd.read_csv(table_csv_path + item.table_file).astype(\n...             str\n...         )  # be sure to make your table data text only\n...         encoding = self.tokenizer(\n...             table=table,\n...             queries=item.question,\n...             answer_coordinates=item.answer_coordinates,\n...             answer_text=item.answer_text,\n...             truncation=True,\n...             padding=\"max_length\",\n...             return_tensors=\"pt\",\n...         )\n...         # remove the batch dimension which the tokenizer adds by default\n...         encoding = {key: val.squeeze(0) for key, val in encoding.items()}\n...         # add the float_answer which is also required (weak supervision for aggregation case)\n...         encoding[\"float_answer\"] = torch.tensor(item.float_answer)\n...         return encoding\n\n...     def __len__(self):\n...         return len(self.data)\n\n\n>>> data = pd.read_csv(tsv_path, sep=\"\\t\")\n>>> train_dataset = TableDataset(data, tokenizer)\n>>> train_dataloader = torch.utils.data.DataLoader(train_dataset, batch_size=32)\n```\n</pt>\n<tf>\nThird, given that you've prepared your data in this TSV/CSV format (and corresponding CSV files containing the tabular data), you can then use [`TapasTokenizer`] to convert table-question pairs into `input_ids`, `attention_mask`, `token_type_ids` and so on. Again, based on which of the three cases you picked above, [`TFTapasForQuestionAnswering`] requires different\ninputs to be fine-tuned:\n\n| **Task**                           | **Required inputs**                                                                                                 |\n|------------------------------------|---------------------------------------------------------------------------------------------------------------------|\n| Conversational                     | `input_ids`, `attention_mask`, `token_type_ids`, `labels`                                                           |\n|  Weak supervision for aggregation  | `input_ids`, `attention_mask`, `token_type_ids`, `labels`, `numeric_values`, `numeric_values_scale`, `float_answer` |\n| Strong supervision for aggregation | `input ids`, `attention mask`, `token type ids`, `labels`, `aggregation_labels`                                     |\n\n[`TapasTokenizer`] creates the `labels`, `numeric_values` and `numeric_values_scale` based on the `answer_coordinates` and `answer_text` columns of the TSV file. The `float_answer` and `aggregation_labels` are already in the TSV file of step 2. Here's an example:\n\n```py\n>>> from transformers import TapasTokenizer\n>>> import pandas as pd\n\n>>> model_name = \"google/tapas-base\"\n>>> tokenizer = TapasTokenizer.from_pretrained(model_name)\n\n>>> data = {\"Actors\": [\"Brad Pitt\", \"Leonardo Di Caprio\", \"George Clooney\"], \"Number of movies\": [\"87\", \"53\", \"69\"]}\n>>> queries = [\n...     \"What is the name of the first actor?\",\n...     \"How many movies has George Clooney played in?\",\n...     \"What is the total number of movies?\",\n... ]\n>>> answer_coordinates = [[(0, 0)], [(2, 1)], [(0, 1), (1, 1), (2, 1)]]\n>>> answer_text = [[\"Brad Pitt\"], [\"69\"], [\"209\"]]\n>>> table = pd.DataFrame.from_dict(data)\n>>> inputs = tokenizer(\n...     table=table,\n...     queries=queries,\n...     answer_coordinates=answer_coordinates,\n...     answer_text=answer_text,\n...     padding=\"max_length\",\n...     return_tensors=\"tf\",\n... )\n>>> inputs\n{'input_ids': tensor([[ ... ]]), 'attention_mask': tensor([[...]]), 'token_type_ids': tensor([[[...]]]),\n'numeric_values': tensor([[ ... ]]), 'numeric_values_scale: tensor([[ ... ]]), labels: tensor([[ ... ]])}\n```\n\nNote that [`TapasTokenizer`] expects the data of the table to be **text-only**. You can use `.astype(str)` on a dataframe to turn it into text-only data.\nOf course, this only shows how to encode a single training example. It is advised to create a dataloader to iterate over batches:\n\n```py\n>>> import tensorflow as tf\n>>> import pandas as pd\n\n>>> tsv_path = \"your_path_to_the_tsv_file\"\n>>> table_csv_path = \"your_path_to_a_directory_containing_all_csv_files\""}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: af66a608-9b0e-49b6-a33c-53304ba83a6c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"af66a608-9b0e-49b6-a33c-53304ba83a6c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f9af8e86-dc56-45f2-b03e-4c0dec92cf4a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"05d6bbe7-3387-4c08-be1c-ed57625792d4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"d3bef3c03a0bb62fdfd22809525e3b40d9573bf3c045bc3fcc165f1da3ad634f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4b1a7dbb-4ac7-4b23-b0fc-95fa6269719a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bf3e72011ba82d8e9164c074330aba58095415c84d7dd5ec0f3f6aa5c8b619c7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "ref_doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "chroma:document": ">>> class TableDataset:\n...     def __init__(self, data, tokenizer):\n...         self.data = data\n...         self.tokenizer = tokenizer\n\n...     def __iter__(self):\n...         for idx in range(self.__len__()):\n...             item = self.data.iloc[idx]\n...             table = pd.read_csv(table_csv_path + item.table_file).astype(\n...                 str\n...             )  # be sure to make your table data text only\n...             encoding = self.tokenizer(\n...                 table=table,\n...                 queries=item.question,\n...                 answer_coordinates=item.answer_coordinates,\n...                 answer_text=item.answer_text,\n...                 truncation=True,\n...                 padding=\"max_length\",\n...                 return_tensors=\"tf\",\n...             )\n...             # remove the batch dimension which the tokenizer adds by default\n...             encoding = {key: tf.squeeze(val, 0) for key, val in encoding.items()}\n...             # add the float_answer which is also required (weak supervision for aggregation case)\n...             encoding[\"float_answer\"] = tf.convert_to_tensor(item.float_answer, dtype=tf.float32)\n...             yield encoding[\"input_ids\"], encoding[\"attention_mask\"], encoding[\"numeric_values\"], encoding[\n...                 \"numeric_values_scale\"\n...             ], encoding[\"token_type_ids\"], encoding[\"labels\"], encoding[\"float_answer\"]\n\n...     def __len__(self):\n...         return len(self.data)\n\n\n>>> data = pd.read_csv(tsv_path, sep=\"\\t\")\n>>> train_dataset = TableDataset(data, tokenizer)\n>>> output_signature = (\n...     tf.TensorSpec(shape=(512,), dtype=tf.int32),\n...     tf.TensorSpec(shape=(512,), dtype=tf.int32),\n...     tf.TensorSpec(shape=(512,), dtype=tf.float32),\n...     tf.TensorSpec(shape=(512,), dtype=tf.float32),\n...     tf.TensorSpec(shape=(512, 7), dtype=tf.int32),\n...     tf.TensorSpec(shape=(512,), dtype=tf.int32),\n...     tf.TensorSpec(shape=(512,), dtype=tf.float32),\n... )\n>>> train_dataloader = tf.data.Dataset.from_generator(train_dataset, output_signature=output_signature).batch(32)\n```\n</tf>\n</frameworkcontent>\n\nNote that here, we encode each table-question pair independently. This is fine as long as your dataset is **not conversational**. In case your dataset involves conversational questions (such as in SQA), then you should first group together the `queries`, `answer_coordinates` and `answer_text` per table (in the order of their `position`\nindex) and batch encode each table with its questions. This will make sure that the `prev_labels` token types (see docs of [`TapasTokenizer`]) are set correctly. See [this notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/TAPAS/Fine_tuning_TapasForQuestionAnswering_on_SQA.ipynb) for more info. See [this notebook](https://github.com/kamalkraj/Tapas-Tutorial/blob/master/TAPAS/Fine_tuning_TapasForQuestionAnswering_on_SQA.ipynb) for more info regarding using the TensorFlow model.\n\n**STEP 4: Train (fine-tune) the model\n\n<frameworkcontent>\n<pt>\nYou can then fine-tune [`TapasForQuestionAnswering`] as follows (shown here for the weak supervision for aggregation case):\n\n```py\n>>> from transformers import TapasConfig, TapasForQuestionAnswering, AdamW\n\n>>> # this is the default WTQ configuration\n>>> config = TapasConfig(\n.     num_aggregation_labels=4,\n.     use_answer_as_supervision=True,\n.     answer_loss_cutoff=0.664694,\n."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 4b1a7dbb-4ac7-4b23-b0fc-95fa6269719a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"4b1a7dbb-4ac7-4b23-b0fc-95fa6269719a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f9af8e86-dc56-45f2-b03e-4c0dec92cf4a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"af66a608-9b0e-49b6-a33c-53304ba83a6c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"9287cc4bb0f5f09d33e75fe4210174a0942bac5a08a0ea1aae8e3dbcd6613642\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4b1188f1-3371-46d7-9fe5-c31ce529bad4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"473e2b92ae58897c38b18d4a0d30fd273ca0a21b89f9be891ebcc87d865bcc4e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "ref_doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "chroma:document": "This is fine as long as your dataset is **not conversational**. In case your dataset involves conversational questions (such as in SQA), then you should first group together the `queries`, `answer_coordinates` and `answer_text` per table (in the order of their `position`\nindex) and batch encode each table with its questions. This will make sure that the `prev_labels` token types (see docs of [`TapasTokenizer`]) are set correctly. See [this notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/TAPAS/Fine_tuning_TapasForQuestionAnswering_on_SQA.ipynb) for more info. See [this notebook](https://github.com/kamalkraj/Tapas-Tutorial/blob/master/TAPAS/Fine_tuning_TapasForQuestionAnswering_on_SQA.ipynb) for more info regarding using the TensorFlow model.\n\n**STEP 4: Train (fine-tune) the model\n\n<frameworkcontent>\n<pt>\nYou can then fine-tune [`TapasForQuestionAnswering`] as follows (shown here for the weak supervision for aggregation case):\n\n```py\n>>> from transformers import TapasConfig, TapasForQuestionAnswering, AdamW\n\n>>> # this is the default WTQ configuration\n>>> config = TapasConfig(\n.     num_aggregation_labels=4,\n.     use_answer_as_supervision=True,\n.     answer_loss_cutoff=0.664694,\n.     cell_selection_preference=0.207951,\n.     huber_loss_delta=0.121194,\n.     init_cell_selection_weights_to_zero=True,\n.     select_one_column=True,\n.     allow_empty_column_selection=False,\n.     temperature=0.0352513,\n. )\n>>> model = TapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n\n>>> optimizer = AdamW(model.parameters(), lr=5e-5)\n\n>>> model.train()\n>>> for epoch in range(2):  # loop over the dataset multiple times\n.     for batch in train_dataloader:\n.         # get the inputs;\n.         input_ids = batch[\"input_ids\"]\n.         attention_mask = batch[\"attention_mask\"]\n.         token_type_ids = batch[\"token_type_ids\"]\n.         labels = batch[\"labels\"]\n.         numeric_values = batch[\"numeric_values\"]\n.         numeric_values_scale = batch[\"numeric_values_scale\"]\n.         float_answer = batch[\"float_answer\"]\n\n.         # zero the parameter gradients\n.         optimizer.zero_grad()\n\n.         # forward + backward + optimize\n.         outputs = model(\n.             input_ids=input_ids,\n.             attention_mask=attention_mask,\n.             token_type_ids=token_type_ids,\n.             labels=labels,\n.             numeric_values=numeric_values,\n.             numeric_values_scale=numeric_values_scale,\n.             float_answer=float_answer,\n.         )\n.         loss = outputs.loss\n.         loss.backward()\n.         optimizer.step()\n```\n</pt>\n<tf>\nYou can then fine-tune [`TFTapasForQuestionAnswering`] as follows (shown here for the weak supervision for aggregation case):\n\n```py\n>>> import tensorflow as tf\n>>> from transformers import TapasConfig, TFTapasForQuestionAnswering\n\n>>> # this is the default WTQ configuration\n>>> config = TapasConfig(\n.     num_aggregation_labels=4,\n.     use_answer_as_supervision=True,\n.     answer_loss_cutoff=0.664694,\n.     cell_selection_preference=0.207951,\n.     huber_loss_delta=0.121194,\n."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 4b1188f1-3371-46d7-9fe5-c31ce529bad4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"4b1188f1-3371-46d7-9fe5-c31ce529bad4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f9af8e86-dc56-45f2-b03e-4c0dec92cf4a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4b1a7dbb-4ac7-4b23-b0fc-95fa6269719a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"79d4a4602f87a5f2395579edb72aaf4fc3b8c112d896e5a6c9d27da941f0874c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c4a47dd7-b667-401d-a3ea-259b68e75347\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"13daf7eb36d0d077b78ee3959811fe83697d4b661e94c8eca429d1b6b7bb72f2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "ref_doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "chroma:document": "from_pretrained(\"google/tapas-base\", config=config)\n\n>>> optimizer = AdamW(model.parameters(), lr=5e-5)\n\n>>> model.train()\n>>> for epoch in range(2):  # loop over the dataset multiple times\n.     for batch in train_dataloader:\n.         # get the inputs;\n.         input_ids = batch[\"input_ids\"]\n.         attention_mask = batch[\"attention_mask\"]\n.         token_type_ids = batch[\"token_type_ids\"]\n.         labels = batch[\"labels\"]\n.         numeric_values = batch[\"numeric_values\"]\n.         numeric_values_scale = batch[\"numeric_values_scale\"]\n.         float_answer = batch[\"float_answer\"]\n\n.         # zero the parameter gradients\n.         optimizer.zero_grad()\n\n.         # forward + backward + optimize\n.         outputs = model(\n.             input_ids=input_ids,\n.             attention_mask=attention_mask,\n.             token_type_ids=token_type_ids,\n.             labels=labels,\n.             numeric_values=numeric_values,\n.             numeric_values_scale=numeric_values_scale,\n.             float_answer=float_answer,\n.         )\n.         loss = outputs.loss\n.         loss.backward()\n.         optimizer.step()\n```\n</pt>\n<tf>\nYou can then fine-tune [`TFTapasForQuestionAnswering`] as follows (shown here for the weak supervision for aggregation case):\n\n```py\n>>> import tensorflow as tf\n>>> from transformers import TapasConfig, TFTapasForQuestionAnswering\n\n>>> # this is the default WTQ configuration\n>>> config = TapasConfig(\n.     num_aggregation_labels=4,\n.     use_answer_as_supervision=True,\n.     answer_loss_cutoff=0.664694,\n.     cell_selection_preference=0.207951,\n.     huber_loss_delta=0.121194,\n.     init_cell_selection_weights_to_zero=True,\n.     select_one_column=True,\n.     allow_empty_column_selection=False,\n.     temperature=0.0352513,\n. )\n>>> model = TFTapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n\n>>> optimizer = tf.keras.optimizers.Adam(learning_rate=5e-5)\n\n>>> for epoch in range(2):  # loop over the dataset multiple times\n.     for batch in train_dataloader:\n.         # get the inputs;\n.         input_ids = batch[0]\n.         attention_mask = batch[1]\n.         token_type_ids = batch[4]\n.         labels = batch[-1]\n.         numeric_values = batch[2]\n.         numeric_values_scale = batch[3]\n.         float_answer = batch[6]\n\n.         # forward + backward + optimize\n.         with tf.GradientTape() as tape:\n.             outputs = model(\n.                 input_ids=input_ids,\n.                 attention_mask=attention_mask,\n.                 token_type_ids=token_type_ids,\n.                 labels=labels,\n.                 numeric_values=numeric_values,\n.                 numeric_values_scale=numeric_values_scale,\n.                 float_answer=float_answer,\n.             )\n.         grads = tape.gradient(outputs.loss, model.trainable_weights)\n.         optimizer.apply_gradients(zip(grads, model.trainable_weights))\n```\n</tf>\n</frameworkcontent>\n\n## Usage: inference\n\n<frameworkcontent>\n<pt>\nHere we explain how you can use [`TapasForQuestionAnswering`] or [`TFTapasForQuestionAnswering`] for inference (i.e. making predictions on new data)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: c4a47dd7-b667-401d-a3ea-259b68e75347
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"c4a47dd7-b667-401d-a3ea-259b68e75347\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f9af8e86-dc56-45f2-b03e-4c0dec92cf4a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4b1188f1-3371-46d7-9fe5-c31ce529bad4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"2c3c642019f4dea310311c0de08ffd38fc99049d28518bfc687bd6adfeb6e420\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1017f0df-6bca-497d-a4aa-808c218a0dc5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"89dd0ff4dff0907c673ed19305715fa40b1bf3be6c9bccde79f40b78654ec9ac\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "ref_doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "chroma:document": "664694,\n.     cell_selection_preference=0.207951,\n.     huber_loss_delta=0.121194,\n.     init_cell_selection_weights_to_zero=True,\n.     select_one_column=True,\n.     allow_empty_column_selection=False,\n.     temperature=0.0352513,\n. )\n>>> model = TFTapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n\n>>> optimizer = tf.keras.optimizers.Adam(learning_rate=5e-5)\n\n>>> for epoch in range(2):  # loop over the dataset multiple times\n.     for batch in train_dataloader:\n.         # get the inputs;\n.         input_ids = batch[0]\n.         attention_mask = batch[1]\n.         token_type_ids = batch[4]\n.         labels = batch[-1]\n.         numeric_values = batch[2]\n.         numeric_values_scale = batch[3]\n.         float_answer = batch[6]\n\n.         # forward + backward + optimize\n.         with tf.GradientTape() as tape:\n.             outputs = model(\n.                 input_ids=input_ids,\n.                 attention_mask=attention_mask,\n.                 token_type_ids=token_type_ids,\n.                 labels=labels,\n.                 numeric_values=numeric_values,\n.                 numeric_values_scale=numeric_values_scale,\n.                 float_answer=float_answer,\n.             )\n.         grads = tape.gradient(outputs.loss, model.trainable_weights)\n.         optimizer.apply_gradients(zip(grads, model.trainable_weights))\n```\n</tf>\n</frameworkcontent>\n\n## Usage: inference\n\n<frameworkcontent>\n<pt>\nHere we explain how you can use [`TapasForQuestionAnswering`] or [`TFTapasForQuestionAnswering`] for inference (i.e. making predictions on new data). For inference, only `input_ids`, `attention_mask` and `token_type_ids` (which you can obtain using [`TapasTokenizer`]) have to be provided to the model to obtain the logits. Next, you can use the handy [`~models.tapas.tokenization_tapas.convert_logits_to_predictions`] method to convert these into predicted coordinates and optional aggregation indices.\n\nHowever, note that inference is **different** depending on whether or not the setup is conversational. In a non-conversational set-up, inference can be done in parallel on all table-question pairs of a batch. Here's an example of that:\n\n```py\n>>> from transformers import TapasTokenizer, TapasForQuestionAnswering\n>>> import pandas as pd\n\n>>> model_name = \"google/tapas-base-finetuned-wtq\"\n>>> model = TapasForQuestionAnswering.from_pretrained(model_name)\n>>> tokenizer = TapasTokenizer.from_pretrained(model_name)\n\n>>> data = {\"Actors\": [\"Brad Pitt\", \"Leonardo Di Caprio\", \"George Clooney\"], \"Number of movies\": [\"87\", \"53\", \"69\"]}\n>>> queries = [\n...     \"What is the name of the first actor?\",\n...     \"How many movies has George Clooney played in?\",\n...     \"What is the total number of movies?"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 1017f0df-6bca-497d-a4aa-808c218a0dc5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"1017f0df-6bca-497d-a4aa-808c218a0dc5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f9af8e86-dc56-45f2-b03e-4c0dec92cf4a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c4a47dd7-b667-401d-a3ea-259b68e75347\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"70269f57b62152b6fcb733717729e470bf8f4bfd2b3a0fd735a9eb19ad7b7b96\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fa70dce4-b195-42c7-a324-102ff2903344\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9717b0fabfb02fcb3c2ead98d633de276ed56bd1eab76e6e02ce5d15ce051801\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "ref_doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "chroma:document": "numeric_values=numeric_values,\n.                 numeric_values_scale=numeric_values_scale,\n.                 float_answer=float_answer,\n.             )\n.         grads = tape.gradient(outputs.loss, model.trainable_weights)\n.         optimizer.apply_gradients(zip(grads, model.trainable_weights))\n```\n</tf>\n</frameworkcontent>\n\n## Usage: inference\n\n<frameworkcontent>\n<pt>\nHere we explain how you can use [`TapasForQuestionAnswering`] or [`TFTapasForQuestionAnswering`] for inference (i.e. making predictions on new data). For inference, only `input_ids`, `attention_mask` and `token_type_ids` (which you can obtain using [`TapasTokenizer`]) have to be provided to the model to obtain the logits. Next, you can use the handy [`~models.tapas.tokenization_tapas.convert_logits_to_predictions`] method to convert these into predicted coordinates and optional aggregation indices.\n\nHowever, note that inference is **different** depending on whether or not the setup is conversational. In a non-conversational set-up, inference can be done in parallel on all table-question pairs of a batch. Here's an example of that:\n\n```py\n>>> from transformers import TapasTokenizer, TapasForQuestionAnswering\n>>> import pandas as pd\n\n>>> model_name = \"google/tapas-base-finetuned-wtq\"\n>>> model = TapasForQuestionAnswering.from_pretrained(model_name)\n>>> tokenizer = TapasTokenizer.from_pretrained(model_name)\n\n>>> data = {\"Actors\": [\"Brad Pitt\", \"Leonardo Di Caprio\", \"George Clooney\"], \"Number of movies\": [\"87\", \"53\", \"69\"]}\n>>> queries = [\n...     \"What is the name of the first actor?\",\n...     \"How many movies has George Clooney played in?\",\n...     \"What is the total number of movies?\",\n... ]\n>>> table = pd.DataFrame.from_dict(data)\n>>> inputs = tokenizer(table=table, queries=queries, padding=\"max_length\", return_tensors=\"pt\")\n>>> outputs = model(**inputs)\n>>> predicted_answer_coordinates, predicted_aggregation_indices = tokenizer.convert_logits_to_predictions(\n...     inputs, outputs.logits.detach(), outputs.logits_aggregation.detach()\n... )\n\n>>> # let's print out the results:\n>>> id2aggregation = {0: \"NONE\", 1: \"SUM\", 2: \"AVERAGE\", 3: \"COUNT\"}\n>>> aggregation_predictions_string = [id2aggregation[x] for x in predicted_aggregation_indices]\n\n>>> answers = []\n>>> for coordinates in predicted_answer_coordinates:\n...     if len(coordinates) == 1:\n...         # only a single cell:\n...         answers.append(table.iat[coordinates[0]])\n...     else:\n...         # multiple cells\n...         cell_values = []\n...         for coordinate in coordinates:\n...             cell_values.append(table.iat[coordinate])\n...         answers.append(\", \".join(cell_values))\n\n>>> display(table)\n>>> print(\"\")\n>>> for query, answer, predicted_agg in zip(queries, answers, aggregation_predictions_string):\n...     print(query)\n...     if predicted_agg == \"NONE\":\n...         print(\"Predicted answer: \" + answer)\n...     else:\n...         print(\"Predicted answer: \" + predicted_agg + \" > \" + answer)\nWhat is the name of the first actor?\nPredicted answer: Brad Pitt\nHow many movies has George Clooney played in?\nPredicted answer: COUNT > 69\nWhat is the total number of movies?\nPredicted answer: SUM > 87, 53, 69\n```\n</pt>\n<tf>\nHere we explain how you can use [`TFTapasForQuestionAnswering`] for inference (i.e."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: fa70dce4-b195-42c7-a324-102ff2903344
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"fa70dce4-b195-42c7-a324-102ff2903344\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f9af8e86-dc56-45f2-b03e-4c0dec92cf4a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1017f0df-6bca-497d-a4aa-808c218a0dc5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"20c2c399cd2ed793df9058b5015b196735f266b6d18c9b7ce1152483cc6d226e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7c30f659-7919-41e1-b5ee-62c659a19dd3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"16148826af60d7d6f5f0365c8155bc86b92cb18b7174ca5fb72da00db3919f50\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 30325, \"end_char_idx\": 33178, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "ref_doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "chroma:document": "\",\n...     \"What is the total number of movies?\",\n... ]\n>>> table = pd.DataFrame.from_dict(data)\n>>> inputs = tokenizer(table=table, queries=queries, padding=\"max_length\", return_tensors=\"pt\")\n>>> outputs = model(**inputs)\n>>> predicted_answer_coordinates, predicted_aggregation_indices = tokenizer.convert_logits_to_predictions(\n...     inputs, outputs.logits.detach(), outputs.logits_aggregation.detach()\n... )\n\n>>> # let's print out the results:\n>>> id2aggregation = {0: \"NONE\", 1: \"SUM\", 2: \"AVERAGE\", 3: \"COUNT\"}\n>>> aggregation_predictions_string = [id2aggregation[x] for x in predicted_aggregation_indices]\n\n>>> answers = []\n>>> for coordinates in predicted_answer_coordinates:\n...     if len(coordinates) == 1:\n...         # only a single cell:\n...         answers.append(table.iat[coordinates[0]])\n...     else:\n...         # multiple cells\n...         cell_values = []\n...         for coordinate in coordinates:\n...             cell_values.append(table.iat[coordinate])\n...         answers.append(\", \".join(cell_values))\n\n>>> display(table)\n>>> print(\"\")\n>>> for query, answer, predicted_agg in zip(queries, answers, aggregation_predictions_string):\n...     print(query)\n...     if predicted_agg == \"NONE\":\n...         print(\"Predicted answer: \" + answer)\n...     else:\n...         print(\"Predicted answer: \" + predicted_agg + \" > \" + answer)\nWhat is the name of the first actor?\nPredicted answer: Brad Pitt\nHow many movies has George Clooney played in?\nPredicted answer: COUNT > 69\nWhat is the total number of movies?\nPredicted answer: SUM > 87, 53, 69\n```\n</pt>\n<tf>\nHere we explain how you can use [`TFTapasForQuestionAnswering`] for inference (i.e. making predictions on new data). For inference, only `input_ids`, `attention_mask` and `token_type_ids` (which you can obtain using [`TapasTokenizer`]) have to be provided to the model to obtain the logits. Next, you can use the handy [`~models.tapas.tokenization_tapas.convert_logits_to_predictions`] method to convert these into predicted coordinates and optional aggregation indices.\n\nHowever, note that inference is **different** depending on whether or not the setup is conversational. In a non-conversational set-up, inference can be done in parallel on all table-question pairs of a batch. Here's an example of that:\n\n```py\n>>> from transformers import TapasTokenizer, TFTapasForQuestionAnswering\n>>> import pandas as pd\n\n>>> model_name = \"google/tapas-base-finetuned-wtq\"\n>>> model = TFTapasForQuestionAnswering.from_pretrained(model_name)\n>>> tokenizer = TapasTokenizer.from_pretrained(model_name)\n\n>>> data = {\"Actors\": [\"Brad Pitt\", \"Leonardo Di Caprio\", \"George Clooney\"], \"Number of movies\": [\"87\", \"53\", \"69\"]}\n>>> queries = [\n...     \"What is the name of the first actor?\",\n...     \"How many movies has George Clooney played in?\",\n...     \"What is the total number of movies?"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 7c30f659-7919-41e1-b5ee-62c659a19dd3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"7c30f659-7919-41e1-b5ee-62c659a19dd3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f9af8e86-dc56-45f2-b03e-4c0dec92cf4a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"fa70dce4-b195-42c7-a324-102ff2903344\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"42905cb43de7c5c6c7d27340b15d198b991a17a4145339957edfc35cbec2dc2b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fb9bd799-defe-4865-80b0-f8f9a508b01c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"22475e8c44e5f9d42ac182183fa5e68edfe42edfaed79b206d56d8a84cf538bc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 31715, \"end_char_idx\": 34943, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "ref_doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "chroma:document": "Predicted answer: Brad Pitt\nHow many movies has George Clooney played in?\nPredicted answer: COUNT > 69\nWhat is the total number of movies?\nPredicted answer: SUM > 87, 53, 69\n```\n</pt>\n<tf>\nHere we explain how you can use [`TFTapasForQuestionAnswering`] for inference (i.e. making predictions on new data). For inference, only `input_ids`, `attention_mask` and `token_type_ids` (which you can obtain using [`TapasTokenizer`]) have to be provided to the model to obtain the logits. Next, you can use the handy [`~models.tapas.tokenization_tapas.convert_logits_to_predictions`] method to convert these into predicted coordinates and optional aggregation indices.\n\nHowever, note that inference is **different** depending on whether or not the setup is conversational. In a non-conversational set-up, inference can be done in parallel on all table-question pairs of a batch. Here's an example of that:\n\n```py\n>>> from transformers import TapasTokenizer, TFTapasForQuestionAnswering\n>>> import pandas as pd\n\n>>> model_name = \"google/tapas-base-finetuned-wtq\"\n>>> model = TFTapasForQuestionAnswering.from_pretrained(model_name)\n>>> tokenizer = TapasTokenizer.from_pretrained(model_name)\n\n>>> data = {\"Actors\": [\"Brad Pitt\", \"Leonardo Di Caprio\", \"George Clooney\"], \"Number of movies\": [\"87\", \"53\", \"69\"]}\n>>> queries = [\n...     \"What is the name of the first actor?\",\n...     \"How many movies has George Clooney played in?\",\n...     \"What is the total number of movies?\",\n... ]\n>>> table = pd.DataFrame.from_dict(data)\n>>> inputs = tokenizer(table=table, queries=queries, padding=\"max_length\", return_tensors=\"tf\")\n>>> outputs = model(**inputs)\n>>> predicted_answer_coordinates, predicted_aggregation_indices = tokenizer.convert_logits_to_predictions(\n...     inputs, outputs.logits, outputs.logits_aggregation\n... )\n\n>>> # let's print out the results:\n>>> id2aggregation = {0: \"NONE\", 1: \"SUM\", 2: \"AVERAGE\", 3: \"COUNT\"}\n>>> aggregation_predictions_string = [id2aggregation[x] for x in predicted_aggregation_indices]\n\n>>> answers = []\n>>> for coordinates in predicted_answer_coordinates:\n...     if len(coordinates) == 1:\n...         # only a single cell:\n...         answers.append(table.iat[coordinates[0]])\n...     else:\n...         # multiple cells\n...         cell_values = []\n...         for coordinate in coordinates:\n...             cell_values.append(table.iat[coordinate])\n...         answers.append(\", \".join(cell_values))\n\n>>> display(table)\n>>> print(\"\")\n>>> for query, answer, predicted_agg in zip(queries, answers, aggregation_predictions_string):\n...     print(query)\n...     if predicted_agg == \"NONE\":\n...         print(\"Predicted answer: \" + answer)\n...     else:\n...         print(\"Predicted answer: \" + predicted_agg + \" > \" + answer)\nWhat is the name of the first actor?\nPredicted answer: Brad Pitt\nHow many movies has George Clooney played in?\nPredicted answer: COUNT > 69\nWhat is the total number of movies?\nPredicted answer: SUM > 87, 53, 69\n```\n</tf>\n</frameworkcontent>\n\nIn case of a conversational set-up, then each table-question pair must be provided **sequentially** to the model, such that the `prev_labels` token types can be overwritten by the predicted `labels` of the previous table-question pair."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: fb9bd799-defe-4865-80b0-f8f9a508b01c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers", "_node_content": "{\"id_\": \"fb9bd799-defe-4865-80b0-f8f9a508b01c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f9af8e86-dc56-45f2-b03e-4c0dec92cf4a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7c30f659-7919-41e1-b5ee-62c659a19dd3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"transformers\"}, \"hash\": \"e58991972fc6735dfa3b608f3b432ebba2d704a9573b469d180f1eebec0481d8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 34503, \"end_char_idx\": 36359, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "ref_doc_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "chroma:document": "Predicted answer: Brad Pitt\nHow many movies has George Clooney played in?\nPredicted answer: COUNT > 69\nWhat is the total number of movies?\nPredicted answer: SUM > 87, 53, 69\n```\n</tf>\n</frameworkcontent>\n\nIn case of a conversational set-up, then each table-question pair must be provided **sequentially** to the model, such that the `prev_labels` token types can be overwritten by the predicted `labels` of the previous table-question pair. Again, more info can be found in [this notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/TAPAS/Fine_tuning_TapasForQuestionAnswering_on_SQA.ipynb) (for PyTorch) and [this notebook](https://github.com/kamalkraj/Tapas-Tutorial/blob/master/TAPAS/Fine_tuning_TapasForQuestionAnswering_on_SQA.ipynb) (for TensorFlow).\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n\n## TAPAS specific outputs\n[[autodoc]] models.tapas.modeling_tapas.TableQuestionAnsweringOutput\n\n## TapasConfig\n[[autodoc]] TapasConfig\n\n## TapasTokenizer\n[[autodoc]] TapasTokenizer\n    - __call__\n    - convert_logits_to_predictions\n    - save_vocabulary\n\n<frameworkcontent>\n<pt>\n\n## TapasModel\n[[autodoc]] TapasModel\n    - forward\n    \n## TapasForMaskedLM\n[[autodoc]] TapasForMaskedLM\n    - forward\n\n## TapasForSequenceClassification\n[[autodoc]] TapasForSequenceClassification\n    - forward\n    \n## TapasForQuestionAnswering\n[[autodoc]] TapasForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFTapasModel\n[[autodoc]] TFTapasModel\n    - call\n    \n## TFTapasForMaskedLM\n[[autodoc]] TFTapasForMaskedLM\n    - call\n\n## TFTapasForSequenceClassification\n[[autodoc]] TFTapasForSequenceClassification\n    - call\n    \n## TFTapasForQuestionAnswering\n[[autodoc]] TFTapasForQuestionAnswering\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 34b34841-00a8-42e8-8672-9124f64e3b5a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/lxmert", "title": "LXMERT", "tokens": 1183, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"34b34841-00a8-42e8-8672-9124f64e3b5a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/lxmert\", \"title\": \"LXMERT\", \"tokens\": 1183, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"893e5b19-96fc-49d7-8d12-1ab97b8bf67c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/lxmert\", \"title\": \"LXMERT\", \"tokens\": 1183, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"824354140f99d988dc7a66f6a4824d0281e10ec39c4c3654187d84104ce96d51\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"314e5d4e-53fe-43e9-b0f1-d4c30ca5752d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e564a0391f04e2e1af5b3fcec19d1145d103f78db3cebc46031ee9100fda3cb6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3312, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "893e5b19-96fc-49d7-8d12-1ab97b8bf67c", "doc_id": "893e5b19-96fc-49d7-8d12-1ab97b8bf67c", "ref_doc_id": "893e5b19-96fc-49d7-8d12-1ab97b8bf67c", "chroma:document": "# LXMERT\n\n## Overview\n\nThe LXMERT model was proposed in [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://arxiv.org/abs/1908.07490) by Hao Tan & Mohit Bansal. It is a series of bidirectional transformer encoders\n(one for the vision modality, one for the language modality, and then one to fuse both modalities) pretrained using a\ncombination of masked language modeling, visual-language text alignment, ROI-feature regression, masked\nvisual-attribute modeling, masked visual-object modeling, and visual-question answering objectives. The pretraining\nconsists of multiple multi-modal datasets: MSCOCO, Visual-Genome + Visual-Genome Question Answering, VQA 2.0, and GQA.\n\nThe abstract from the paper is the following:\n\n*Vision-and-language reasoning requires an understanding of visual concepts, language semantics, and, most importantly,\nthe alignment and relationships between these two modalities. We thus propose the LXMERT (Learning Cross-Modality\nEncoder Representations from Transformers) framework to learn these vision-and-language connections. In LXMERT, we\nbuild a large-scale Transformer model that consists of three encoders: an object relationship encoder, a language\nencoder, and a cross-modality encoder. Next, to endow our model with the capability of connecting vision and language\nsemantics, we pre-train the model with large amounts of image-and-sentence pairs, via five diverse representative\npretraining tasks: masked language modeling, masked object prediction (feature regression and label classification),\ncross-modality matching, and image question answering. These tasks help in learning both intra-modality and\ncross-modality relationships. After fine-tuning from our pretrained parameters, our model achieves the state-of-the-art\nresults on two visual question answering datasets (i.e., VQA and GQA). We also show the generalizability of our\npretrained cross-modality model by adapting it to a challenging visual-reasoning task, NLVR, and improve the previous\nbest result by 22% absolute (54% to 76%). Lastly, we demonstrate detailed ablation studies to prove that both our novel\nmodel components and pretraining strategies significantly contribute to our strong results; and also present several\nattention visualizations for the different encoders*\n\nThis model was contributed by [eltoto1219](https://huggingface.co/eltoto1219). The original code can be found [here](https://github.com/airsplay/lxmert).\n\n## Usage tips\n\n- Bounding boxes are not necessary to be used in the visual feature embeddings, any kind of visual-spacial features\n  will work.\n- Both the language hidden states and the visual hidden states that LXMERT outputs are passed through the\n  cross-modality layer, so they contain information from both modalities. To access a modality that only attends to\n  itself, select the vision/language hidden states from the first input in the tuple.\n- The bidirectional cross-modality encoder attention only returns attention values when the language modality is used\n  as the input and the vision modality is used as the context vector. Further, while the cross-modality encoder\n  contains self-attention for each respective modality and cross-attention, only the cross attention is returned and\n  both self attention outputs are disregarded."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 314e5d4e-53fe-43e9-b0f1-d4c30ca5752d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/lxmert", "title": "LXMERT", "tokens": 1183, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"314e5d4e-53fe-43e9-b0f1-d4c30ca5752d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/lxmert\", \"title\": \"LXMERT\", \"tokens\": 1183, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"893e5b19-96fc-49d7-8d12-1ab97b8bf67c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/lxmert\", \"title\": \"LXMERT\", \"tokens\": 1183, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"824354140f99d988dc7a66f6a4824d0281e10ec39c4c3654187d84104ce96d51\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"34b34841-00a8-42e8-8672-9124f64e3b5a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/lxmert\", \"title\": \"LXMERT\", \"tokens\": 1183, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a80a91485425c84cebe9bbc35b31447b0cf52cdaaad664256041bce0637452ed\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1618, \"end_char_idx\": 4322, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "893e5b19-96fc-49d7-8d12-1ab97b8bf67c", "doc_id": "893e5b19-96fc-49d7-8d12-1ab97b8bf67c", "ref_doc_id": "893e5b19-96fc-49d7-8d12-1ab97b8bf67c", "chroma:document": "These tasks help in learning both intra-modality and\ncross-modality relationships. After fine-tuning from our pretrained parameters, our model achieves the state-of-the-art\nresults on two visual question answering datasets (i.e., VQA and GQA). We also show the generalizability of our\npretrained cross-modality model by adapting it to a challenging visual-reasoning task, NLVR, and improve the previous\nbest result by 22% absolute (54% to 76%). Lastly, we demonstrate detailed ablation studies to prove that both our novel\nmodel components and pretraining strategies significantly contribute to our strong results; and also present several\nattention visualizations for the different encoders*\n\nThis model was contributed by [eltoto1219](https://huggingface.co/eltoto1219). The original code can be found [here](https://github.com/airsplay/lxmert).\n\n## Usage tips\n\n- Bounding boxes are not necessary to be used in the visual feature embeddings, any kind of visual-spacial features\n  will work.\n- Both the language hidden states and the visual hidden states that LXMERT outputs are passed through the\n  cross-modality layer, so they contain information from both modalities. To access a modality that only attends to\n  itself, select the vision/language hidden states from the first input in the tuple.\n- The bidirectional cross-modality encoder attention only returns attention values when the language modality is used\n  as the input and the vision modality is used as the context vector. Further, while the cross-modality encoder\n  contains self-attention for each respective modality and cross-attention, only the cross attention is returned and\n  both self attention outputs are disregarded.\n\n## Resources\n\n- [Question answering task guide](../tasks/question_answering)\n\n## LxmertConfig\n\n[[autodoc]] LxmertConfig\n\n## LxmertTokenizer\n\n[[autodoc]] LxmertTokenizer\n\n## LxmertTokenizerFast\n\n[[autodoc]] LxmertTokenizerFast\n\n## Lxmert specific outputs\n\n[[autodoc]] models.lxmert.modeling_lxmert.LxmertModelOutput\n\n[[autodoc]] models.lxmert.modeling_lxmert.LxmertForPreTrainingOutput\n\n[[autodoc]] models.lxmert.modeling_lxmert.LxmertForQuestionAnsweringOutput\n\n[[autodoc]] models.lxmert.modeling_tf_lxmert.TFLxmertModelOutput\n\n[[autodoc]] models.lxmert.modeling_tf_lxmert.TFLxmertForPreTrainingOutput\n\n<frameworkcontent>\n<pt>\n\n## LxmertModel\n\n[[autodoc]] LxmertModel\n    - forward\n\n## LxmertForPreTraining\n\n[[autodoc]] LxmertForPreTraining\n    - forward\n\n## LxmertForQuestionAnswering\n\n[[autodoc]] LxmertForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFLxmertModel\n\n[[autodoc]] TFLxmertModel\n    - call\n\n## TFLxmertForPreTraining\n\n[[autodoc]] TFLxmertForPreTraining\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 9b0457b7-6ebe-4dfc-9bb0-d0719b65a039
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/dpt", "title": "DPT", "tokens": 1085, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9b0457b7-6ebe-4dfc-9bb0-d0719b65a039\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dpt\", \"title\": \"DPT\", \"tokens\": 1085, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3515131e-ba54-4f46-8fd5-484448e3e393\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dpt\", \"title\": \"DPT\", \"tokens\": 1085, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"17fd551fd712ca727ea3d1a97dc6e44926a8ce1ac6daf07d80f77850878fa9f9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2db6aab4-fd2c-4146-8fcc-d0dff2738b24\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ef621a0e76d8e3fd4223cefff8da8a75cce207dd7f7212953fc6e8e55d23a5c0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3416, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3515131e-ba54-4f46-8fd5-484448e3e393", "doc_id": "3515131e-ba54-4f46-8fd5-484448e3e393", "ref_doc_id": "3515131e-ba54-4f46-8fd5-484448e3e393", "chroma:document": "# DPT\n\n## Overview\n\nThe DPT model was proposed in [Vision Transformers for Dense Prediction](https://arxiv.org/abs/2103.13413) by Ren\u00e9 Ranftl, Alexey Bochkovskiy, Vladlen Koltun.\nDPT is a model that leverages the [Vision Transformer (ViT)](vit) as backbone for dense prediction tasks like semantic segmentation and depth estimation.\n\nThe abstract from the paper is the following:\n\n*We introduce dense vision transformers, an architecture that leverages vision transformers in place of convolutional networks as a backbone for dense prediction tasks. We assemble tokens from various stages of the vision transformer into image-like representations at various resolutions and progressively combine them into full-resolution predictions using a convolutional decoder. The transformer backbone processes representations at a constant and relatively high resolution and has a global receptive field at every stage. These properties allow the dense vision transformer to provide finer-grained and more globally coherent predictions when compared to fully-convolutional networks. Our experiments show that this architecture yields substantial improvements on dense prediction tasks, especially when a large amount of training data is available. For monocular depth estimation, we observe an improvement of up to 28% in relative performance when compared to a state-of-the-art fully-convolutional network. When applied to semantic segmentation, dense vision transformers set a new state of the art on ADE20K with 49.02% mIoU. We further show that the architecture can be fine-tuned on smaller datasets such as NYUv2, KITTI, and Pascal Context where it also sets the new state of the art.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/dpt_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> DPT architecture. Taken from the <a href=\"https://arxiv.org/abs/2103.13413\" target=\"_blank\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/isl-org/DPT).\n\n## Usage tips\n\nDPT is compatible with the [`AutoBackbone`] class. This allows to use the DPT framework with various computer vision backbones available in the library, such as [`VitDetBackbone`] or [`Dinov2Backbone`]. One can create it as follows:\n\n```python\nfrom transformers import Dinov2Config, DPTConfig, DPTForDepthEstimation\n\n# initialize with a Transformer-based backbone such as DINOv2\n# in that case, we also specify `reshape_hidden_states=False` to get feature maps of shape (batch_size, num_channels, height, width)\nbackbone_config = Dinov2Config.from_pretrained(\"facebook/dinov2-base\", out_features=[\"stage1\", \"stage2\", \"stage3\", \"stage4\"], reshape_hidden_states=False)\n\nconfig = DPTConfig(backbone_config=backbone_config)\nmodel = DPTForDepthEstimation(config=config)\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with DPT.\n\n- Demo notebooks for [`DPTForDepthEstimation`] can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DPT).\n\n- [Semantic segmentation task guide](../tasks/semantic_segmentation)\n- [Monocular depth estimation task guide](../tasks/monocular_depth_estimation)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it!"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 2db6aab4-fd2c-4146-8fcc-d0dff2738b24
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/dpt", "title": "DPT", "tokens": 1085, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2db6aab4-fd2c-4146-8fcc-d0dff2738b24\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dpt\", \"title\": \"DPT\", \"tokens\": 1085, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3515131e-ba54-4f46-8fd5-484448e3e393\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dpt\", \"title\": \"DPT\", \"tokens\": 1085, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"17fd551fd712ca727ea3d1a97dc6e44926a8ce1ac6daf07d80f77850878fa9f9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9b0457b7-6ebe-4dfc-9bb0-d0719b65a039\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dpt\", \"title\": \"DPT\", \"tokens\": 1085, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"70a20fe5895b116c67d4d36ee3bfe27be499f9c990e4b9bef75cc8c8f33fc4b9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1941, \"end_char_idx\": 3985, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3515131e-ba54-4f46-8fd5-484448e3e393", "doc_id": "3515131e-ba54-4f46-8fd5-484448e3e393", "ref_doc_id": "3515131e-ba54-4f46-8fd5-484448e3e393", "chroma:document": "</small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/isl-org/DPT).\n\n## Usage tips\n\nDPT is compatible with the [`AutoBackbone`] class. This allows to use the DPT framework with various computer vision backbones available in the library, such as [`VitDetBackbone`] or [`Dinov2Backbone`]. One can create it as follows:\n\n```python\nfrom transformers import Dinov2Config, DPTConfig, DPTForDepthEstimation\n\n# initialize with a Transformer-based backbone such as DINOv2\n# in that case, we also specify `reshape_hidden_states=False` to get feature maps of shape (batch_size, num_channels, height, width)\nbackbone_config = Dinov2Config.from_pretrained(\"facebook/dinov2-base\", out_features=[\"stage1\", \"stage2\", \"stage3\", \"stage4\"], reshape_hidden_states=False)\n\nconfig = DPTConfig(backbone_config=backbone_config)\nmodel = DPTForDepthEstimation(config=config)\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with DPT.\n\n- Demo notebooks for [`DPTForDepthEstimation`] can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DPT).\n\n- [Semantic segmentation task guide](../tasks/semantic_segmentation)\n- [Monocular depth estimation task guide](../tasks/monocular_depth_estimation)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## DPTConfig\n\n[[autodoc]] DPTConfig\n\n## DPTFeatureExtractor\n\n[[autodoc]] DPTFeatureExtractor\n    - __call__\n    - post_process_semantic_segmentation\n\n## DPTImageProcessor\n\n[[autodoc]] DPTImageProcessor\n    - preprocess\n    - post_process_semantic_segmentation\n\n## DPTModel\n\n[[autodoc]] DPTModel\n    - forward\n\n## DPTForDepthEstimation\n\n[[autodoc]] DPTForDepthEstimation\n    - forward\n\n## DPTForSemanticSegmentation\n\n[[autodoc]] DPTForSemanticSegmentation\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 8b64bcdc-f76c-4c8b-8c23-cda43fadc390
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/canine", "title": "CANINE", "tokens": 1446, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8b64bcdc-f76c-4c8b-8c23-cda43fadc390\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/canine\", \"title\": \"CANINE\", \"tokens\": 1446, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"57fc9403-20ca-479d-a89e-7644b11f66bf\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/canine\", \"title\": \"CANINE\", \"tokens\": 1446, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f59dcff97f3e1d243275ecb5e64ce123396e3adbfb60ef6e63c92dba3da20e78\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f91954f6-1c59-463d-950d-56ac2644a817\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b1d5a869de85b830a05286ec1e0ae4567c892fefbab81eece9fb43d3f749df7c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3319, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "57fc9403-20ca-479d-a89e-7644b11f66bf", "doc_id": "57fc9403-20ca-479d-a89e-7644b11f66bf", "ref_doc_id": "57fc9403-20ca-479d-a89e-7644b11f66bf", "chroma:document": "# CANINE\n\n## Overview\n\nThe CANINE model was proposed in [CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language\nRepresentation](https://arxiv.org/abs/2103.06874) by Jonathan H. Clark, Dan Garrette, Iulia Turc, John Wieting. It's\namong the first papers that trains a Transformer without using an explicit tokenization step (such as Byte Pair\nEncoding (BPE), WordPiece or SentencePiece). Instead, the model is trained directly at a Unicode character-level.\nTraining at a character-level inevitably comes with a longer sequence length, which CANINE solves with an efficient\ndownsampling strategy, before applying a deep Transformer encoder.\n\nThe abstract from the paper is the following:\n\n*Pipelined NLP systems have largely been superseded by end-to-end neural modeling, yet nearly all commonly-used models\nstill require an explicit tokenization step. While recent tokenization approaches based on data-derived subword\nlexicons are less brittle than manually engineered tokenizers, these techniques are not equally suited to all\nlanguages, and the use of any fixed vocabulary may limit a model's ability to adapt. In this paper, we present CANINE,\na neural encoder that operates directly on character sequences, without explicit tokenization or vocabulary, and a\npre-training strategy that operates either directly on characters or optionally uses subwords as a soft inductive bias.\nTo use its finer-grained input effectively and efficiently, CANINE combines downsampling, which reduces the input\nsequence length, with a deep transformer stack, which encodes context. CANINE outperforms a comparable mBERT model by\n2.8 F1 on TyDi QA, a challenging multilingual benchmark, despite having 28% fewer model parameters.*\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/google-research/language/tree/master/language/canine).\n\n## Usage tips\n\n- CANINE uses no less than 3 Transformer encoders internally: 2 \"shallow\" encoders (which only consist of a single\n  layer) and 1 \"deep\" encoder (which is a regular BERT encoder). First, a \"shallow\" encoder is used to contextualize\n  the character embeddings, using local attention. Next, after downsampling, a \"deep\" encoder is applied. Finally,\n  after upsampling, a \"shallow\" encoder is used to create the final character embeddings. Details regarding up- and\n  downsampling can be found in the paper.\n- CANINE uses a max sequence length of 2048 characters by default. One can use [`CanineTokenizer`]\n  to prepare text for the model.\n- Classification can be done by placing a linear layer on top of the final hidden state of the special [CLS] token\n  (which has a predefined Unicode code point). For token classification tasks however, the downsampled sequence of\n  tokens needs to be upsampled again to match the length of the original character sequence (which is 2048). The\n  details for this can be found in the paper.\n\nModel checkpoints:\n\n  - [google/canine-c](https://huggingface.co/google/canine-c): Pre-trained with autoregressive character loss,\n    12-layer, 768-hidden, 12-heads, 121M parameters (size ~500 MB).\n  - [google/canine-s](https://huggingface.co/google/canine-s): Pre-trained with subword loss, 12-layer,\n    768-hidden, 12-heads, 121M parameters (size ~500 MB)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f91954f6-1c59-463d-950d-56ac2644a817
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/canine", "title": "CANINE", "tokens": 1446, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f91954f6-1c59-463d-950d-56ac2644a817\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/canine\", \"title\": \"CANINE\", \"tokens\": 1446, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"57fc9403-20ca-479d-a89e-7644b11f66bf\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/canine\", \"title\": \"CANINE\", \"tokens\": 1446, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f59dcff97f3e1d243275ecb5e64ce123396e3adbfb60ef6e63c92dba3da20e78\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8b64bcdc-f76c-4c8b-8c23-cda43fadc390\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/canine\", \"title\": \"CANINE\", \"tokens\": 1446, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4c4d6c9980cba714dac6e501bec0e2b2a66ba7b9107ec4d47a96e2ec585f0ca7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3322, \"end_char_idx\": 5566, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "57fc9403-20ca-479d-a89e-7644b11f66bf", "doc_id": "57fc9403-20ca-479d-a89e-7644b11f66bf", "ref_doc_id": "57fc9403-20ca-479d-a89e-7644b11f66bf", "chroma:document": "## Usage example\n\nCANINE works on raw characters, so it can be used **without a tokenizer**:\n\n```python\n>>> from transformers import CanineModel\n>>> import torch\n\n>>> model = CanineModel.from_pretrained(\"google/canine-c\")  # model pre-trained with autoregressive character loss\n\n>>> text = \"hello world\"\n>>> # use Python's built-in ord() function to turn each character into its unicode code point id\n>>> input_ids = torch.tensor([[ord(char) for char in text]])\n\n>>> outputs = model(input_ids)  # forward pass\n>>> pooled_output = outputs.pooler_output\n>>> sequence_output = outputs.last_hidden_state\n```\n\nFor batched inference and training, it is however recommended to make use of the tokenizer (to pad/truncate all\nsequences to the same length):\n\n```python\n>>> from transformers import CanineTokenizer, CanineModel\n\n>>> model = CanineModel.from_pretrained(\"google/canine-c\")\n>>> tokenizer = CanineTokenizer.from_pretrained(\"google/canine-c\")\n\n>>> inputs = [\"Life is like a box of chocolates.\", \"You never know what you gonna get.\"]\n>>> encoding = tokenizer(inputs, padding=\"longest\", truncation=True, return_tensors=\"pt\")\n\n>>> outputs = model(**encoding)  # forward pass\n>>> pooled_output = outputs.pooler_output\n>>> sequence_output = outputs.last_hidden_state\n```\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## CanineConfig\n\n[[autodoc]] CanineConfig\n\n## CanineTokenizer\n\n[[autodoc]] CanineTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n\n## CANINE specific outputs\n\n[[autodoc]] models.canine.modeling_canine.CanineModelOutputWithPooling\n\n## CanineModel\n\n[[autodoc]] CanineModel\n    - forward\n\n## CanineForSequenceClassification\n\n[[autodoc]] CanineForSequenceClassification\n    - forward\n\n## CanineForMultipleChoice\n\n[[autodoc]] CanineForMultipleChoice\n    - forward\n\n## CanineForTokenClassification\n\n[[autodoc]] CanineForTokenClassification\n    - forward\n\n## CanineForQuestionAnswering\n\n[[autodoc]] CanineForQuestionAnswering\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 30d65d37-3d7a-418f-8b44-846d41291653
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vipllava", "title": "VipLlava", "tokens": 1159, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"30d65d37-3d7a-418f-8b44-846d41291653\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vipllava\", \"title\": \"VipLlava\", \"tokens\": 1159, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d7b5ffa8-6dd8-46a3-b7dd-851625361d77\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vipllava\", \"title\": \"VipLlava\", \"tokens\": 1159, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"07f9f1dfc1472c71e6a676c5000f98818aa134a11fb06b9a1978798cb75221d8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bde4f876-d5b9-4f3a-b071-c9860dfb96ba\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"93be05c5d0cf367ff6ed445162b83a659557faa32e35e1cf703f9b6d6bf38f38\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1708, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d7b5ffa8-6dd8-46a3-b7dd-851625361d77", "doc_id": "d7b5ffa8-6dd8-46a3-b7dd-851625361d77", "ref_doc_id": "d7b5ffa8-6dd8-46a3-b7dd-851625361d77", "chroma:document": "# VipLlava\n\n## Overview\n\nThe VipLlava model was proposed in [Making Large Multimodal Models Understand Arbitrary Visual Prompts](https://arxiv.org/abs/2312.00784) by Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P. Meyer, Yuning Chai, Dennis Park, Yong Jae Lee.\n\nVipLlava enhances the training protocol of Llava by marking images and interact with the model using natural cues like a \"red bounding box\" or \"pointed arrow\" during training.\n\nThe abstract from the paper is the following:\n\n*While existing large vision-language multimodal models focus on whole image understanding, there is a prominent gap in achieving region-specific comprehension. Current approaches that use textual coordinates or spatial encodings often fail to provide a user-friendly interface for visual prompting. To address this challenge, we introduce a novel multimodal model capable of decoding arbitrary visual prompts. This allows users to intuitively mark images and interact with the model using natural cues like a \"red bounding box\" or \"pointed arrow\". Our simple design directly overlays visual markers onto the RGB image, eliminating the need for complex region encodings, yet achieves state-of-the-art performance on region-understanding tasks like Visual7W, PointQA, and Visual Commonsense Reasoning benchmark. Furthermore, we present ViP-Bench, a comprehensive benchmark to assess the capability of models in understanding visual prompts across multiple dimensions, enabling future research in this domain. Code, data, and model are publicly available.*\n\nThe original code can be found [here](https://github.com/mu-cai/ViP-LLaVA).\n\nThis model was contributed by [Younes Belkada](https://huggingface.co/ybelkada)"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: bde4f876-d5b9-4f3a-b071-c9860dfb96ba
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vipllava", "title": "VipLlava", "tokens": 1159, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"bde4f876-d5b9-4f3a-b071-c9860dfb96ba\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vipllava\", \"title\": \"VipLlava\", \"tokens\": 1159, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d7b5ffa8-6dd8-46a3-b7dd-851625361d77\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vipllava\", \"title\": \"VipLlava\", \"tokens\": 1159, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"07f9f1dfc1472c71e6a676c5000f98818aa134a11fb06b9a1978798cb75221d8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"30d65d37-3d7a-418f-8b44-846d41291653\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vipllava\", \"title\": \"VipLlava\", \"tokens\": 1159, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"25d652330a465a6dfb45458464a53c5e431e16d85d8a076d195846de4f02915f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"03d60dea-5d0d-4b62-80e3-76c31f782ebf\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"98ee1174bd6aac28765f57e81130931612ca885aaa5c686caabeb6a339670a2f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 4336, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d7b5ffa8-6dd8-46a3-b7dd-851625361d77", "doc_id": "d7b5ffa8-6dd8-46a3-b7dd-851625361d77", "ref_doc_id": "d7b5ffa8-6dd8-46a3-b7dd-851625361d77", "chroma:document": "# VipLlava\n\n## Overview\n\nThe VipLlava model was proposed in [Making Large Multimodal Models Understand Arbitrary Visual Prompts](https://arxiv.org/abs/2312.00784) by Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P. Meyer, Yuning Chai, Dennis Park, Yong Jae Lee.\n\nVipLlava enhances the training protocol of Llava by marking images and interact with the model using natural cues like a \"red bounding box\" or \"pointed arrow\" during training.\n\nThe abstract from the paper is the following:\n\n*While existing large vision-language multimodal models focus on whole image understanding, there is a prominent gap in achieving region-specific comprehension. Current approaches that use textual coordinates or spatial encodings often fail to provide a user-friendly interface for visual prompting. To address this challenge, we introduce a novel multimodal model capable of decoding arbitrary visual prompts. This allows users to intuitively mark images and interact with the model using natural cues like a \"red bounding box\" or \"pointed arrow\". Our simple design directly overlays visual markers onto the RGB image, eliminating the need for complex region encodings, yet achieves state-of-the-art performance on region-understanding tasks like Visual7W, PointQA, and Visual Commonsense Reasoning benchmark. Furthermore, we present ViP-Bench, a comprehensive benchmark to assess the capability of models in understanding visual prompts across multiple dimensions, enabling future research in this domain. Code, data, and model are publicly available.*\n\nThe original code can be found [here](https://github.com/mu-cai/ViP-LLaVA).\n\nThis model was contributed by [Younes Belkada](https://huggingface.co/ybelkada)\n\n\n## Usage tips:\n\n- The architecture is similar than llava architecture except that the multi-modal projector takes a set of concatenated vision hidden states and has an additional layernorm layer on that module.\n\n- We advise users to use `padding_side=\"left\"` when computing batched generation as it leads to more accurate results. Simply make sure to call `processor.tokenizer.padding_side = \"left\"` before generating.\n\n- Note the model has not been explicitly trained to process multiple images in the same prompt, although this is technically possible, you may experience inaccurate results.\n\n- For better results, we recommend users to use the processor's `apply_chat_template()` method to format your prompt correctly. For that you need to construct a conversation history, passing in a plain string will not format your prompt. Each message in the conversation history for chat templates is a dictionary with keys \"role\" and \"content\". The \"content\" should be a list of dictionaries, for \"text\" and \"image\" modalities, as follows:\n\n```python\nfrom transformers import AutoProcessor\n\nprocessor = AutoProcessor.from_pretrained(\"llava-hf/vip-llava-7b-hf\")\n\nconversation = [\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image\"},\n            {\"type\": \"text\", \"text\": \"What\u2019s shown in this image?\"},\n         ,\n    },\n    {\n        \"role\": \"assistant\",\n        \"content\": [{\"type\": \"text\", \"text\": \"This image shows a red stop sign.\"},]\n    },\n    {\n\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"Describe the image in more details.\"},\n        ],\n    },\n]\n\ntext_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)\n\n# Note that the template simply formats your prompt, you still have to tokenize it and obtain pixel values for your images\nprint(text_prompt)\n>>> \"###Human: <image>\\nWhat\u2019s shown in this image?###Assistant: This image shows a red stop sign.###Human: Describe the image in more details.###Assistant:\"\n```\n\n- If you want to construct a chat prompt yourself, below is a list of prompt formats accepted by VipLLaVa checkpoints:\n```bash\nA chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.###Human: <image>\\n<prompt>###Assistant:\n```\n\nFor multiple turns conversation:\n```bash\nA chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.###Human: <image>\\n<prompt1>###Assistant: <answer1>###Human: <prompt2>###Assistant:\n```"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 03d60dea-5d0d-4b62-80e3-76c31f782ebf
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vipllava", "title": "VipLlava", "tokens": 1159, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"03d60dea-5d0d-4b62-80e3-76c31f782ebf\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vipllava\", \"title\": \"VipLlava\", \"tokens\": 1159, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d7b5ffa8-6dd8-46a3-b7dd-851625361d77\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vipllava\", \"title\": \"VipLlava\", \"tokens\": 1159, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"07f9f1dfc1472c71e6a676c5000f98818aa134a11fb06b9a1978798cb75221d8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bde4f876-d5b9-4f3a-b071-c9860dfb96ba\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vipllava\", \"title\": \"VipLlava\", \"tokens\": 1159, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ed15255c288e104d76811a249adbd344afd2c5b4157010b6205ea37e86b444f8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4339, \"end_char_idx\": 4481, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d7b5ffa8-6dd8-46a3-b7dd-851625361d77", "doc_id": "d7b5ffa8-6dd8-46a3-b7dd-851625361d77", "ref_doc_id": "d7b5ffa8-6dd8-46a3-b7dd-851625361d77", "chroma:document": "## VipLlavaConfig\n\n[[autodoc]] VipLlavaConfig\n\n## VipLlavaForConditionalGeneration\n\n[[autodoc]] VipLlavaForConditionalGeneration\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a2712ccc-1b6b-4438-8e06-f0b833d738f8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a2712ccc-1b6b-4438-8e06-f0b833d738f8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb\", \"title\": \"NLLB\", \"tokens\": 2326, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5a548662-ff11-4b4f-8b7e-0d36729dea0a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb\", \"title\": \"NLLB\", \"tokens\": 2326, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2bfe8515d58245cd9b09950479b511e8896843df8a89f9ff8d60f2e237078321\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a0c33592-59ae-43b2-bf45-a34a17e0c0b5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4999b2128a4caac8a2de2ceb2add6f3c9773df08a0b09e6befa9d18d535a379a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2901, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5a548662-ff11-4b4f-8b7e-0d36729dea0a", "doc_id": "5a548662-ff11-4b4f-8b7e-0d36729dea0a", "ref_doc_id": "5a548662-ff11-4b4f-8b7e-0d36729dea0a", "chroma:document": "# NLLB\n\n## Updated tokenizer behavior \n\n**DISCLAIMER:** The default behaviour for the tokenizer was fixed and thus changed in April 2023.\nThe previous version adds `[self.eos_token_id, self.cur_lang_code]` at the end of the token sequence for both target and source tokenization. This is wrong as the NLLB paper mentions (page 48, 6.1.1. Model Architecture) :\n\n*Note that we prefix the source sequence with the source language, as opposed to the target\nlanguage as previously done in several works (Arivazhagan et al., 2019; Johnson et al.,\n2017). This is primarily because we prioritize optimizing zero-shot performance of our\nmodel on any pair of 200 languages at a minor cost to supervised performance.*\n\nPrevious behaviour:\n\n```python\n>>> from transformers import NllbTokenizer\n\n>>> tokenizer = NllbTokenizer.from_pretrained(\"facebook/nllb-200-distilled-600M\")\n>>> tokenizer(\"How was your day?\").input_ids\n[13374, 1398, 4260, 4039, 248130, 2, 256047]\n\n>>> # 2: '</s>'\n>>> # 256047 : 'eng_Latn'\n```\nNew behaviour\n\n```python\n>>> from transformers import NllbTokenizer\n\n>>> tokenizer = NllbTokenizer.from_pretrained(\"facebook/nllb-200-distilled-600M\")\n>>> tokenizer(\"How was your day?\").input_ids\n[256047, 13374, 1398, 4260, 4039, 248130, 2]\n ```\n\nEnabling the old behaviour can be done as follows:\n```python\n>>> from transformers import NllbTokenizer\n\n>>> tokenizer = NllbTokenizer.from_pretrained(\"facebook/nllb-200-distilled-600M\", legacy_behaviour=True)\n```\n\nFor more details, feel free to check the linked [PR](https://github.com/huggingface/transformers/pull/22313) and [Issue](https://github.com/huggingface/transformers/issues/19943).\n\n## Overview\n\nThe NLLB model was presented in [No Language Left Behind: Scaling Human-Centered Machine Translation](https://arxiv.org/abs/2207.04672) by Marta R. Costa-juss\u00e0, James Cross, Onur \u00c7elebi,\nMaha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula,\nLoic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews,\nNecip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzm\u00e1n, Philipp Koehn, Alexandre Mourachko, Christophe Ropers,\nSafiyyah Saleem, Holger Schwenk, and Jeff Wang.\n\nThe abstract of the paper is the following:\n\n*Driven by the goal of eradicating language barriers on a global scale, machine translation has solidified itself as a key focus of artificial intelligence research today.\nHowever, such efforts have coalesced around a small subset of languages, leaving behind the vast majority of mostly low-resource languages. What does it take to break the\n200 language barrier while ensuring safe, high quality results, all while keeping ethical considerations in mind?"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a0c33592-59ae-43b2-bf45-a34a17e0c0b5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a0c33592-59ae-43b2-bf45-a34a17e0c0b5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb\", \"title\": \"NLLB\", \"tokens\": 2326, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5a548662-ff11-4b4f-8b7e-0d36729dea0a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb\", \"title\": \"NLLB\", \"tokens\": 2326, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2bfe8515d58245cd9b09950479b511e8896843df8a89f9ff8d60f2e237078321\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a2712ccc-1b6b-4438-8e06-f0b833d738f8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb\", \"title\": \"NLLB\", \"tokens\": 2326, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"069e487edb67c2ba49361b3cc923ef8ddf78f05883e7a66e0a11ad69fd3c1d15\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"606bf67e-f9ae-4d71-ab1b-4835cdc400bc\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c3bc72fbff32006d6c0e81560f7760df9b1d97674b43d8eef427a52dc297efce\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1645, \"end_char_idx\": 4778, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5a548662-ff11-4b4f-8b7e-0d36729dea0a", "doc_id": "5a548662-ff11-4b4f-8b7e-0d36729dea0a", "ref_doc_id": "5a548662-ff11-4b4f-8b7e-0d36729dea0a", "chroma:document": "## Overview\n\nThe NLLB model was presented in [No Language Left Behind: Scaling Human-Centered Machine Translation](https://arxiv.org/abs/2207.04672) by Marta R. Costa-juss\u00e0, James Cross, Onur \u00c7elebi,\nMaha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula,\nLoic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews,\nNecip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzm\u00e1n, Philipp Koehn, Alexandre Mourachko, Christophe Ropers,\nSafiyyah Saleem, Holger Schwenk, and Jeff Wang.\n\nThe abstract of the paper is the following:\n\n*Driven by the goal of eradicating language barriers on a global scale, machine translation has solidified itself as a key focus of artificial intelligence research today.\nHowever, such efforts have coalesced around a small subset of languages, leaving behind the vast majority of mostly low-resource languages. What does it take to break the\n200 language barrier while ensuring safe, high quality results, all while keeping ethical considerations in mind? In No Language Left Behind, we took on this challenge by\nfirst contextualizing the need for low-resource language translation support through exploratory interviews with native speakers. Then, we created datasets and models aimed\nat narrowing the performance gap between low and high-resource languages. More specifically, we developed a conditional compute model based on Sparsely Gated Mixture of\nExperts that is trained on data obtained with novel and effective data mining techniques tailored for low-resource languages. We propose multiple architectural and training\nimprovements to counteract overfitting while training on thousands of tasks. Critically, we evaluated the performance of over 40,000 different translation directions using\na human-translated benchmark, Flores-200, and combined human evaluation with a novel toxicity benchmark covering all languages in Flores-200 to assess translation safety.\nOur model achieves an improvement of 44% BLEU relative to the previous state-of-the-art, laying important groundwork towards realizing a universal translation system.*\n\nThis implementation contains the dense models available on release.\n\n**The sparse model NLLB-MoE (Mixture of Expert) is now available! More details [here](nllb-moe)**\n\nThis model was contributed by [Lysandre](https://huggingface.co/lysandre). The authors' code can be found [here](https://github.com/facebookresearch/fairseq/tree/nllb).\n\n## Generating with NLLB\n\nWhile generating the target text set the `forced_bos_token_id` to the target language id. The following\nexample shows how to translate English to French using the *facebook/nllb-200-distilled-600M* model.\n\nNote that we're using the BCP-47 code for French `fra_Latn`. See [here](https://github.com/facebookresearch/flores/blob/main/flores200/README.md#languages-in-flores-200)\nfor the list of all BCP-47 in the Flores 200 dataset."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 606bf67e-f9ae-4d71-ab1b-4835cdc400bc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"606bf67e-f9ae-4d71-ab1b-4835cdc400bc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb\", \"title\": \"NLLB\", \"tokens\": 2326, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5a548662-ff11-4b4f-8b7e-0d36729dea0a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb\", \"title\": \"NLLB\", \"tokens\": 2326, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2bfe8515d58245cd9b09950479b511e8896843df8a89f9ff8d60f2e237078321\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a0c33592-59ae-43b2-bf45-a34a17e0c0b5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb\", \"title\": \"NLLB\", \"tokens\": 2326, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"599e06f839ed76580ce020084de6d114d2ad4e9de092ce5c268f22d467cbb9a4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"78b3d3f4-141c-4c08-99b8-60682823d9e3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"33de18180082bae679fafe25b290fa989ff8c5b661bb0089c2b9fd392e34eb3d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3089, \"end_char_idx\": 5748, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5a548662-ff11-4b4f-8b7e-0d36729dea0a", "doc_id": "5a548662-ff11-4b4f-8b7e-0d36729dea0a", "ref_doc_id": "5a548662-ff11-4b4f-8b7e-0d36729dea0a", "chroma:document": "Then, we created datasets and models aimed\nat narrowing the performance gap between low and high-resource languages. More specifically, we developed a conditional compute model based on Sparsely Gated Mixture of\nExperts that is trained on data obtained with novel and effective data mining techniques tailored for low-resource languages. We propose multiple architectural and training\nimprovements to counteract overfitting while training on thousands of tasks. Critically, we evaluated the performance of over 40,000 different translation directions using\na human-translated benchmark, Flores-200, and combined human evaluation with a novel toxicity benchmark covering all languages in Flores-200 to assess translation safety.\nOur model achieves an improvement of 44% BLEU relative to the previous state-of-the-art, laying important groundwork towards realizing a universal translation system.*\n\nThis implementation contains the dense models available on release.\n\n**The sparse model NLLB-MoE (Mixture of Expert) is now available! More details [here](nllb-moe)**\n\nThis model was contributed by [Lysandre](https://huggingface.co/lysandre). The authors' code can be found [here](https://github.com/facebookresearch/fairseq/tree/nllb).\n\n## Generating with NLLB\n\nWhile generating the target text set the `forced_bos_token_id` to the target language id. The following\nexample shows how to translate English to French using the *facebook/nllb-200-distilled-600M* model.\n\nNote that we're using the BCP-47 code for French `fra_Latn`. See [here](https://github.com/facebookresearch/flores/blob/main/flores200/README.md#languages-in-flores-200)\nfor the list of all BCP-47 in the Flores 200 dataset.\n\n```python\n>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"facebook/nllb-200-distilled-600M\")\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"facebook/nllb-200-distilled-600M\")\n\n>>> article = \"UN Chief says there is no military solution in Syria\"\n>>> inputs = tokenizer(article, return_tensors=\"pt\")\n\n>>> translated_tokens = model.generate(\n...     **inputs, forced_bos_token_id=tokenizer.convert_tokens_to_ids(\"fra_Latn\"), max_length=30\n... )\n>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]\nLe chef de l'ONU dit qu'il n'y a pas de solution militaire en Syrie\n```\n\n### Generating from any other language than English\n\nEnglish (`eng_Latn`) is set as the default language from which to translate. In order to specify that you'd like to translate from a different language,\nyou should specify the BCP-47 code in the `src_lang` keyword argument of the tokenizer initialization."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 78b3d3f4-141c-4c08-99b8-60682823d9e3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"78b3d3f4-141c-4c08-99b8-60682823d9e3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb\", \"title\": \"NLLB\", \"tokens\": 2326, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5a548662-ff11-4b4f-8b7e-0d36729dea0a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb\", \"title\": \"NLLB\", \"tokens\": 2326, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2bfe8515d58245cd9b09950479b511e8896843df8a89f9ff8d60f2e237078321\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"606bf67e-f9ae-4d71-ab1b-4835cdc400bc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb\", \"title\": \"NLLB\", \"tokens\": 2326, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"28878c13e282282d82526530c84df08f43e58f578c6b89d9fce2103f238ba53a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1549202b-6530-4475-9211-b85bae98cd69\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2e396380330057478b5db6f533a4ef117313ba5bd5065a58dc42286aae40b797\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4229, \"end_char_idx\": 7169, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5a548662-ff11-4b4f-8b7e-0d36729dea0a", "doc_id": "5a548662-ff11-4b4f-8b7e-0d36729dea0a", "ref_doc_id": "5a548662-ff11-4b4f-8b7e-0d36729dea0a", "chroma:document": "The authors' code can be found [here](https://github.com/facebookresearch/fairseq/tree/nllb).\n\n## Generating with NLLB\n\nWhile generating the target text set the `forced_bos_token_id` to the target language id. The following\nexample shows how to translate English to French using the *facebook/nllb-200-distilled-600M* model.\n\nNote that we're using the BCP-47 code for French `fra_Latn`. See [here](https://github.com/facebookresearch/flores/blob/main/flores200/README.md#languages-in-flores-200)\nfor the list of all BCP-47 in the Flores 200 dataset.\n\n```python\n>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"facebook/nllb-200-distilled-600M\")\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"facebook/nllb-200-distilled-600M\")\n\n>>> article = \"UN Chief says there is no military solution in Syria\"\n>>> inputs = tokenizer(article, return_tensors=\"pt\")\n\n>>> translated_tokens = model.generate(\n...     **inputs, forced_bos_token_id=tokenizer.convert_tokens_to_ids(\"fra_Latn\"), max_length=30\n... )\n>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]\nLe chef de l'ONU dit qu'il n'y a pas de solution militaire en Syrie\n```\n\n### Generating from any other language than English\n\nEnglish (`eng_Latn`) is set as the default language from which to translate. In order to specify that you'd like to translate from a different language,\nyou should specify the BCP-47 code in the `src_lang` keyword argument of the tokenizer initialization.\n\nSee example below for a translation from romanian to german:\n\n```py\n>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\n...     \"facebook/nllb-200-distilled-600M\", token=True, src_lang=\"ron_Latn\"\n... )\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"facebook/nllb-200-distilled-600M\", token=True)\n\n>>> article = \"\u015eeful ONU spune c\u0103 nu exist\u0103 o solu\u0163ie militar\u0103 \u00een Siria\"\n>>> inputs = tokenizer(article, return_tensors=\"pt\")\n\n>>> translated_tokens = model.generate(\n...     **inputs, forced_bos_token_id=tokenizer.convert_tokens_to_ids(\"deu_Latn\"), max_length=30\n... )\n>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]\nUN-Chef sagt, es gibt keine milit\u00e4rische L\u00f6sung in Syrien\n```\n\n## Resources\n\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n## NllbTokenizer\n\n[[autodoc]] NllbTokenizer\n    - build_inputs_with_special_tokens\n\n## NllbTokenizerFast\n\n[[autodoc]] NllbTokenizerFast\n\n## Using Flash Attention 2\n\nFlash Attention 2 is a faster, optimized version of the attention scores computation which relies on `cuda` kernels.\n\n### Installation \n\nFirst, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 1549202b-6530-4475-9211-b85bae98cd69
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1549202b-6530-4475-9211-b85bae98cd69\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb\", \"title\": \"NLLB\", \"tokens\": 2326, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5a548662-ff11-4b4f-8b7e-0d36729dea0a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb\", \"title\": \"NLLB\", \"tokens\": 2326, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2bfe8515d58245cd9b09950479b511e8896843df8a89f9ff8d60f2e237078321\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"78b3d3f4-141c-4c08-99b8-60682823d9e3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb\", \"title\": \"NLLB\", \"tokens\": 2326, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cc071fa82e71d18ce57049ab00c6ebd9ada0c0a49e423a24e074da5411fa903d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"266cd43f-7f84-48dc-8eb7-ce102d96b07d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"70d3c6af1abb7ba92b9151bb8c5b3520351c73b33dbd88c3b8f70fa3864f3a27\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5570, \"end_char_idx\": 8610, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5a548662-ff11-4b4f-8b7e-0d36729dea0a", "doc_id": "5a548662-ff11-4b4f-8b7e-0d36729dea0a", "ref_doc_id": "5a548662-ff11-4b4f-8b7e-0d36729dea0a", "chroma:document": "In order to specify that you'd like to translate from a different language,\nyou should specify the BCP-47 code in the `src_lang` keyword argument of the tokenizer initialization.\n\nSee example below for a translation from romanian to german:\n\n```py\n>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\n...     \"facebook/nllb-200-distilled-600M\", token=True, src_lang=\"ron_Latn\"\n... )\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"facebook/nllb-200-distilled-600M\", token=True)\n\n>>> article = \"\u015eeful ONU spune c\u0103 nu exist\u0103 o solu\u0163ie militar\u0103 \u00een Siria\"\n>>> inputs = tokenizer(article, return_tensors=\"pt\")\n\n>>> translated_tokens = model.generate(\n...     **inputs, forced_bos_token_id=tokenizer.convert_tokens_to_ids(\"deu_Latn\"), max_length=30\n... )\n>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]\nUN-Chef sagt, es gibt keine milit\u00e4rische L\u00f6sung in Syrien\n```\n\n## Resources\n\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n## NllbTokenizer\n\n[[autodoc]] NllbTokenizer\n    - build_inputs_with_special_tokens\n\n## NllbTokenizerFast\n\n[[autodoc]] NllbTokenizerFast\n\n## Using Flash Attention 2\n\nFlash Attention 2 is a faster, optimized version of the attention scores computation which relies on `cuda` kernels.\n\n### Installation \n\nFirst, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features).\n\nNext, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\n### Usage\n\nTo load a model using Flash Attention 2, we can pass the argument `attn_implementation=\"flash_attention_2\"` to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). You can use either `torch.float16` or `torch.bfloat16` precision.\n\n```python\n>>> import torch\n>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"facebook/nllb-200-distilled-600M\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\").to(\"cuda\").eval()\n>>> tokenizer = AutoTokenizer.from_pretrained(\"facebook/nllb-200-distilled-600M\")\n\n>>> article = \"\u015eeful ONU spune c\u0103 nu exist\u0103 o solu\u0163ie militar\u0103 \u00een Siria\"\n>>> inputs = tokenizer(article, return_tensors=\"pt\").to(\"cuda\")\n\n>>> translated_tokens = model.generate(\n...     **inputs, forced_bos_token_id=tokenizer.convert_tokens_to_ids(\"deu_Latn\"), max_length=30\n... )\n>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]\n\"UN-Chef sagt, es gibt keine milit\u00e4rische L\u00f6sung in Syrien\"\n```\n\n### Expected speedups\n\nBelow is an expected speedup diagram that compares pure inference time between the native implementation and the Flash Attention 2."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 266cd43f-7f84-48dc-8eb7-ce102d96b07d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"266cd43f-7f84-48dc-8eb7-ce102d96b07d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb\", \"title\": \"NLLB\", \"tokens\": 2326, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5a548662-ff11-4b4f-8b7e-0d36729dea0a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb\", \"title\": \"NLLB\", \"tokens\": 2326, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2bfe8515d58245cd9b09950479b511e8896843df8a89f9ff8d60f2e237078321\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1549202b-6530-4475-9211-b85bae98cd69\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb\", \"title\": \"NLLB\", \"tokens\": 2326, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"453531d98e92fcbdf376a917a2584b1439791ab420c159674a55e12568859f39\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7013, \"end_char_idx\": 8758, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5a548662-ff11-4b4f-8b7e-0d36729dea0a", "doc_id": "5a548662-ff11-4b4f-8b7e-0d36729dea0a", "ref_doc_id": "5a548662-ff11-4b4f-8b7e-0d36729dea0a", "chroma:document": "The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features).\n\nNext, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\n### Usage\n\nTo load a model using Flash Attention 2, we can pass the argument `attn_implementation=\"flash_attention_2\"` to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). You can use either `torch.float16` or `torch.bfloat16` precision.\n\n```python\n>>> import torch\n>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"facebook/nllb-200-distilled-600M\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\").to(\"cuda\").eval()\n>>> tokenizer = AutoTokenizer.from_pretrained(\"facebook/nllb-200-distilled-600M\")\n\n>>> article = \"\u015eeful ONU spune c\u0103 nu exist\u0103 o solu\u0163ie militar\u0103 \u00een Siria\"\n>>> inputs = tokenizer(article, return_tensors=\"pt\").to(\"cuda\")\n\n>>> translated_tokens = model.generate(\n...     **inputs, forced_bos_token_id=tokenizer.convert_tokens_to_ids(\"deu_Latn\"), max_length=30\n... )\n>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]\n\"UN-Chef sagt, es gibt keine milit\u00e4rische L\u00f6sung in Syrien\"\n```\n\n### Expected speedups\n\nBelow is an expected speedup diagram that compares pure inference time between the native implementation and the Flash Attention 2.\n\n<div style=\"text-align: center\">\n<img src=\"https://huggingface.co/datasets/visheratin/documentation-images/resolve/main/nllb-speedup.webp\">\n</div>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 1bc266f3-0f89-47a1-918c-aec9f5373169
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/deit", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1bc266f3-0f89-47a1-918c-aec9f5373169\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deit\", \"title\": \"DeiT\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"540f096f-67a0-4958-bbbc-3db91b5ac24d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deit\", \"title\": \"DeiT\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5e5619f19de12a1b3e8f588745eb38e816eb35c4884c8b5951de149ca446e346\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"113a5b02-de4c-4cf9-bff9-4bb6899ee481\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f39a0e5529512db7b56c4c5e69f1d32a58ab45bc145f1178d7a17b01bf978729\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3321, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "540f096f-67a0-4958-bbbc-3db91b5ac24d", "doc_id": "540f096f-67a0-4958-bbbc-3db91b5ac24d", "ref_doc_id": "540f096f-67a0-4958-bbbc-3db91b5ac24d", "chroma:document": "# DeiT\n\n## Overview\n\nThe DeiT model was proposed in [Training data-efficient image transformers & distillation through attention](https://arxiv.org/abs/2012.12877) by Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre\nSablayrolles, Herv\u00e9 J\u00e9gou. The [Vision Transformer (ViT)](vit) introduced in [Dosovitskiy et al., 2020](https://arxiv.org/abs/2010.11929) has shown that one can match or even outperform existing convolutional neural\nnetworks using a Transformer encoder (BERT-like). However, the ViT models introduced in that paper required training on\nexpensive infrastructure for multiple weeks, using external data. DeiT (data-efficient image transformers) are more\nefficiently trained transformers for image classification, requiring far less data and far less computing resources\ncompared to the original ViT models.\n\nThe abstract from the paper is the following:\n\n*Recently, neural networks purely based on attention were shown to address image understanding tasks such as image\nclassification. However, these visual transformers are pre-trained with hundreds of millions of images using an\nexpensive infrastructure, thereby limiting their adoption. In this work, we produce a competitive convolution-free\ntransformer by training on Imagenet only. We train them on a single computer in less than 3 days. Our reference vision\ntransformer (86M parameters) achieves top-1 accuracy of 83.1% (single-crop evaluation) on ImageNet with no external\ndata. More importantly, we introduce a teacher-student strategy specific to transformers. It relies on a distillation\ntoken ensuring that the student learns from the teacher through attention. We show the interest of this token-based\ndistillation, especially when using a convnet as a teacher. This leads us to report results competitive with convnets\nfor both Imagenet (where we obtain up to 85.2% accuracy) and when transferring to other tasks. We share our code and\nmodels.*\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The TensorFlow version of this model was added by [amyeroberts](https://huggingface.co/amyeroberts).\n\n## Usage tips\n\n- Compared to ViT, DeiT models use a so-called distillation token to effectively learn from a teacher (which, in the\n  DeiT paper, is a ResNet like-model). The distillation token is learned through backpropagation, by interacting with\n  the class ([CLS]) and patch tokens through the self-attention layers.\n- There are 2 ways to fine-tune distilled models, either (1) in a classic way, by only placing a prediction head on top\n  of the final hidden state of the class token and not using the distillation signal, or (2) by placing both a\n  prediction head on top of the class token and on top of the distillation token. In that case, the [CLS] prediction\n  head is trained using regular cross-entropy between the prediction of the head and the ground-truth label, while the\n  distillation prediction head is trained using hard distillation (cross-entropy between the prediction of the\n  distillation head and the label predicted by the teacher). At inference time, one takes the average prediction\n  between both heads as final prediction. (2) is also called \"fine-tuning with distillation\", because one relies on a\n  teacher that has already been fine-tuned on the downstream dataset."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 113a5b02-de4c-4cf9-bff9-4bb6899ee481
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/deit", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"113a5b02-de4c-4cf9-bff9-4bb6899ee481\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deit\", \"title\": \"DeiT\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"540f096f-67a0-4958-bbbc-3db91b5ac24d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deit\", \"title\": \"DeiT\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5e5619f19de12a1b3e8f588745eb38e816eb35c4884c8b5951de149ca446e346\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1bc266f3-0f89-47a1-918c-aec9f5373169\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deit\", \"title\": \"DeiT\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"441e3e7de384fe5762793e113bab4cd0119e40f4193b67866e01a082afa87d59\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e4ab58df-a62a-4448-8c35-a89740fd27d2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8900adc0324e373391c6583eb4712e15c41d06e2414db67c22b253bde705edbe\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1662, \"end_char_idx\": 4842, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "540f096f-67a0-4958-bbbc-3db91b5ac24d", "doc_id": "540f096f-67a0-4958-bbbc-3db91b5ac24d", "ref_doc_id": "540f096f-67a0-4958-bbbc-3db91b5ac24d", "chroma:document": "We show the interest of this token-based\ndistillation, especially when using a convnet as a teacher. This leads us to report results competitive with convnets\nfor both Imagenet (where we obtain up to 85.2% accuracy) and when transferring to other tasks. We share our code and\nmodels.*\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The TensorFlow version of this model was added by [amyeroberts](https://huggingface.co/amyeroberts).\n\n## Usage tips\n\n- Compared to ViT, DeiT models use a so-called distillation token to effectively learn from a teacher (which, in the\n  DeiT paper, is a ResNet like-model). The distillation token is learned through backpropagation, by interacting with\n  the class ([CLS]) and patch tokens through the self-attention layers.\n- There are 2 ways to fine-tune distilled models, either (1) in a classic way, by only placing a prediction head on top\n  of the final hidden state of the class token and not using the distillation signal, or (2) by placing both a\n  prediction head on top of the class token and on top of the distillation token. In that case, the [CLS] prediction\n  head is trained using regular cross-entropy between the prediction of the head and the ground-truth label, while the\n  distillation prediction head is trained using hard distillation (cross-entropy between the prediction of the\n  distillation head and the label predicted by the teacher). At inference time, one takes the average prediction\n  between both heads as final prediction. (2) is also called \"fine-tuning with distillation\", because one relies on a\n  teacher that has already been fine-tuned on the downstream dataset. In terms of models, (1) corresponds to\n  [`DeiTForImageClassification`] and (2) corresponds to\n  [`DeiTForImageClassificationWithTeacher`].\n- Note that the authors also did try soft distillation for (2) (in which case the distillation prediction head is\n  trained using KL divergence to match the softmax output of the teacher), but hard distillation gave the best results.\n- All released checkpoints were pre-trained and fine-tuned on ImageNet-1k only. No external data was used. This is in\n  contrast with the original ViT model, which used external data like the JFT-300M dataset/Imagenet-21k for\n  pre-training.\n- The authors of DeiT also released more efficiently trained ViT models, which you can directly plug into\n  [`ViTModel`] or [`ViTForImageClassification`]. Techniques like data\n  augmentation, optimization, and regularization were used in order to simulate training on a much larger dataset\n  (while only using ImageNet-1k for pre-training). There are 4 variants available (in 3 different sizes):\n  *facebook/deit-tiny-patch16-224*, *facebook/deit-small-patch16-224*, *facebook/deit-base-patch16-224* and\n  *facebook/deit-base-patch16-384*. Note that one should use [`DeiTImageProcessor`] in order to\n  prepare images for the model.\n\n### Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e4ab58df-a62a-4448-8c35-a89740fd27d2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/deit", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e4ab58df-a62a-4448-8c35-a89740fd27d2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deit\", \"title\": \"DeiT\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"540f096f-67a0-4958-bbbc-3db91b5ac24d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deit\", \"title\": \"DeiT\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5e5619f19de12a1b3e8f588745eb38e816eb35c4884c8b5951de149ca446e346\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"113a5b02-de4c-4cf9-bff9-4bb6899ee481\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deit\", \"title\": \"DeiT\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a61caf16365eadfd663d915a8e69dfb8fccaf9e99ea1b4c687b852fe55c757e4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ee39c02b-8fd3-4e87-b4f5-3267899750d3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"029ca90e3a085074fd8493a422be77182cdb82522184983bf5b3107400b982dd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3322, \"end_char_idx\": 5839, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "540f096f-67a0-4958-bbbc-3db91b5ac24d", "doc_id": "540f096f-67a0-4958-bbbc-3db91b5ac24d", "ref_doc_id": "540f096f-67a0-4958-bbbc-3db91b5ac24d", "chroma:document": "In terms of models, (1) corresponds to\n  [`DeiTForImageClassification`] and (2) corresponds to\n  [`DeiTForImageClassificationWithTeacher`].\n- Note that the authors also did try soft distillation for (2) (in which case the distillation prediction head is\n  trained using KL divergence to match the softmax output of the teacher), but hard distillation gave the best results.\n- All released checkpoints were pre-trained and fine-tuned on ImageNet-1k only. No external data was used. This is in\n  contrast with the original ViT model, which used external data like the JFT-300M dataset/Imagenet-21k for\n  pre-training.\n- The authors of DeiT also released more efficiently trained ViT models, which you can directly plug into\n  [`ViTModel`] or [`ViTForImageClassification`]. Techniques like data\n  augmentation, optimization, and regularization were used in order to simulate training on a much larger dataset\n  (while only using ImageNet-1k for pre-training). There are 4 variants available (in 3 different sizes):\n  *facebook/deit-tiny-patch16-224*, *facebook/deit-small-patch16-224*, *facebook/deit-base-patch16-224* and\n  *facebook/deit-base-patch16-384*. Note that one should use [`DeiTImageProcessor`] in order to\n  prepare images for the model.\n\n### Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```\nfrom transformers import DeiTForImageClassification\nmodel = DeiTForImageClassification.from_pretrained(\"facebook/deit-base-distilled-patch16-224\", attn_implementation=\"sdpa\", torch_dtype=torch.float16)\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `facebook/deit-base-distilled-patch16-224` model, we saw the following speedups during inference."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: ee39c02b-8fd3-4e87-b4f5-3267899750d3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/deit", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ee39c02b-8fd3-4e87-b4f5-3267899750d3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deit\", \"title\": \"DeiT\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"540f096f-67a0-4958-bbbc-3db91b5ac24d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deit\", \"title\": \"DeiT\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5e5619f19de12a1b3e8f588745eb38e816eb35c4884c8b5951de149ca446e346\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e4ab58df-a62a-4448-8c35-a89740fd27d2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deit\", \"title\": \"DeiT\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"667d94f09a853d5e030af32ed35b1fed8b792b55bbc22602707dafaa533d9d5b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"911713fe-975f-4e3f-a087-18bd0f55b821\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"48d04040b93eee91b93afc81ec5bba08c29cfde39234bf8ac7a8f2bfa36633dc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4478, \"end_char_idx\": 7589, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "540f096f-67a0-4958-bbbc-3db91b5ac24d", "doc_id": "540f096f-67a0-4958-bbbc-3db91b5ac24d", "ref_doc_id": "540f096f-67a0-4958-bbbc-3db91b5ac24d", "chroma:document": "Note that one should use [`DeiTImageProcessor`] in order to\n  prepare images for the model.\n\n### Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```\nfrom transformers import DeiTForImageClassification\nmodel = DeiTForImageClassification.from_pretrained(\"facebook/deit-base-distilled-patch16-224\", attn_implementation=\"sdpa\", torch_dtype=torch.float16)\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `facebook/deit-base-distilled-patch16-224` model, we saw the following speedups during inference.\n\n|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |\n|--------------|-------------------------------------------|-------------------------------------------|------------------------------|\n|            1 |                                         8 |                                         6 |                      1.33 |\n|            2 |                                         9 |                                         6 |                      1.5  |\n|            4 |                                         9 |                                         6 |                      1.5  |\n|            8 |                                         8 |                                         6 |                      1.33 |\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with DeiT.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`DeiTForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nBesides that:\n\n- [`DeiTForMaskedImageModeling`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 911713fe-975f-4e3f-a087-18bd0f55b821
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/deit", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"911713fe-975f-4e3f-a087-18bd0f55b821\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deit\", \"title\": \"DeiT\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"540f096f-67a0-4958-bbbc-3db91b5ac24d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deit\", \"title\": \"DeiT\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5e5619f19de12a1b3e8f588745eb38e816eb35c4884c8b5951de149ca446e346\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ee39c02b-8fd3-4e87-b4f5-3267899750d3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deit\", \"title\": \"DeiT\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cd83813f908408df4533323524bdc6c90912a0f6c096b10602640297998e3b62\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5658, \"end_char_idx\": 8498, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "540f096f-67a0-4958-bbbc-3db91b5ac24d", "doc_id": "540f096f-67a0-4958-bbbc-3db91b5ac24d", "ref_doc_id": "540f096f-67a0-4958-bbbc-3db91b5ac24d", "chroma:document": "On a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `facebook/deit-base-distilled-patch16-224` model, we saw the following speedups during inference.\n\n|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |\n|--------------|-------------------------------------------|-------------------------------------------|------------------------------|\n|            1 |                                         8 |                                         6 |                      1.33 |\n|            2 |                                         9 |                                         6 |                      1.5  |\n|            4 |                                         9 |                                         6 |                      1.5  |\n|            8 |                                         8 |                                         6 |                      1.33 |\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with DeiT.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`DeiTForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nBesides that:\n\n- [`DeiTForMaskedImageModeling`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## DeiTConfig\n\n[[autodoc]] DeiTConfig\n\n## DeiTFeatureExtractor\n\n[[autodoc]] DeiTFeatureExtractor\n    - __call__\n\n## DeiTImageProcessor\n\n[[autodoc]] DeiTImageProcessor\n    - preprocess\n\n<frameworkcontent>\n<pt>\n\n## DeiTModel\n\n[[autodoc]] DeiTModel\n    - forward\n\n## DeiTForMaskedImageModeling\n\n[[autodoc]] DeiTForMaskedImageModeling\n    - forward\n\n## DeiTForImageClassification\n\n[[autodoc]] DeiTForImageClassification\n    - forward\n\n## DeiTForImageClassificationWithTeacher\n\n[[autodoc]] DeiTForImageClassificationWithTeacher\n    - forward\n\n</pt>\n<tf>\n\n## TFDeiTModel\n\n[[autodoc]] TFDeiTModel\n    - call\n\n## TFDeiTForMaskedImageModeling\n\n[[autodoc]] TFDeiTForMaskedImageModeling\n    - call\n\n## TFDeiTForImageClassification\n\n[[autodoc]] TFDeiTForImageClassification\n    - call\n\n## TFDeiTForImageClassificationWithTeacher\n\n[[autodoc]] TFDeiTForImageClassificationWithTeacher\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f4326f60-a7fa-4a35-a074-025a472c2fae
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/pegasus_x", "title": "PEGASUS-X", "tokens": 630, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f4326f60-a7fa-4a35-a074-025a472c2fae\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pegasus_x\", \"title\": \"PEGASUS-X\", \"tokens\": 630, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"38ec819c-1a97-4e8c-ac50-135704a23e0d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pegasus_x\", \"title\": \"PEGASUS-X\", \"tokens\": 630, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2ecb7f0c047d782b936dfcdeac84b858c4d7fa5cd16967f6d4028bd9be2f1789\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2134, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "38ec819c-1a97-4e8c-ac50-135704a23e0d", "doc_id": "38ec819c-1a97-4e8c-ac50-135704a23e0d", "ref_doc_id": "38ec819c-1a97-4e8c-ac50-135704a23e0d", "chroma:document": "# PEGASUS-X\n\n## Overview\n\nThe PEGASUS-X model was proposed in [Investigating Efficiently Extending Transformers for Long Input Summarization](https://arxiv.org/abs/2208.04347)  by Jason Phang, Yao Zhao and Peter J. Liu.\n\nPEGASUS-X (PEGASUS eXtended) extends the PEGASUS models for long input summarization through additional long input pretraining and using staggered block-local attention with global tokens in the encoder.\n\nThe abstract from the paper is the following:\n\n*While large pretrained Transformer models have proven highly capable at tackling natural language tasks, handling long sequence inputs continues to be a significant challenge. One such task is long input summarization, where inputs are longer than the maximum input context of most pretrained models. Through an extensive set of experiments, we investigate what model architectural changes and pretraining paradigms can most efficiently adapt a pretrained Transformer for long input summarization. We find that a staggered, block-local Transformer with global encoder tokens strikes a good balance of performance and efficiency, and that an additional pretraining phase on long sequences meaningfully improves downstream summarization performance. Based on our findings, we introduce PEGASUS-X, an extension of the PEGASUS model with additional long input pretraining to handle inputs of up to 16K tokens. PEGASUS-X achieves strong performance on long input summarization tasks comparable with much larger models while adding few additional parameters and not requiring model parallelism to train.*\n\nThis model was contributed by [zphang](https://huggingface.co/zphang). The original code can be found [here](https://github.com/google-research/pegasus).\n\n## Documentation resources\n\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n<Tip>\n\nPEGASUS-X uses the same tokenizer as [PEGASUS](pegasus).\n\n</Tip>\n\n## PegasusXConfig\n\n[[autodoc]] PegasusXConfig\n\n## PegasusXModel\n\n[[autodoc]] PegasusXModel\n    - forward\n\n## PegasusXForConditionalGeneration\n\n[[autodoc]] PegasusXForConditionalGeneration\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 9b25f1ff-b231-400b-a84b-1d3a52eafdc1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert", "title": "Wav2Vec2-BERT", "tokens": 1431, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9b25f1ff-b231-400b-a84b-1d3a52eafdc1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert\", \"title\": \"Wav2Vec2-BERT\", \"tokens\": 1431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2650753e-bea2-4c35-baff-7795ba332a33\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert\", \"title\": \"Wav2Vec2-BERT\", \"tokens\": 1431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"76199f83168ac684935b57258de857f533f2c2aac536894f6087d91de6052432\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"db87bad8-5088-4b18-bf91-d5b484e9c133\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"65d57d19aec87f1f8f24b08ef79d17237774788559ef0021cc1da7a25f288e53\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3857, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2650753e-bea2-4c35-baff-7795ba332a33", "doc_id": "2650753e-bea2-4c35-baff-7795ba332a33", "ref_doc_id": "2650753e-bea2-4c35-baff-7795ba332a33", "chroma:document": "# Wav2Vec2-BERT\n\n## Overview\n\nThe Wav2Vec2-BERT model was proposed in [Seamless: Multilingual Expressive and Streaming Speech Translation](https://ai.meta.com/research/publications/seamless-multilingual-expressive-and-streaming-speech-translation/) by the Seamless Communication team from Meta AI.\n\nThis model was pre-trained on 4.5M hours of unlabeled audio data covering more than 143 languages. It requires finetuning to be used for downstream tasks such as Automatic Speech Recognition (ASR), or Audio Classification.\n\nThe official results of the model can be found in Section 3.2.1 of the paper.\n\nThe abstract from the paper is the following:\n\n*Recent advancements in automatic speech translation have dramatically expanded language coverage, improved multimodal capabilities, and enabled a wide range of tasks and functionalities. That said, large-scale automatic speech translation systems today lack key features that help machine-mediated communication feel seamless when compared to human-to-human dialogue. In this work, we introduce a family of models that enable end-to-end expressive and multilingual translations in a streaming fashion. First, we contribute an improved version of the massively multilingual and multimodal SeamlessM4T model\u2014SeamlessM4T v2. This newer model, incorporating an updated UnitY2 framework, was trained on more low-resource language data. The expanded version of SeamlessAlign adds 114,800 hours of automatically aligned data for a total of 76 languages. SeamlessM4T v2 provides the foundation on which our two newest models, SeamlessExpressive and SeamlessStreaming, are initiated. SeamlessExpressive enables translation that preserves vocal styles and prosody. Compared to previous efforts in expressive speech research, our work addresses certain underexplored aspects of prosody, such as speech rate and pauses, while also preserving the style of one\u2019s voice. As for SeamlessStreaming, our model leverages the Efficient Monotonic Multihead Attention (EMMA) mechanism to generate low-latency target translations without waiting for complete source utterances. As the first of its kind, SeamlessStreaming enables simultaneous speech-to-speech/text translation for multiple source and target languages. To understand the performance of these models, we combined novel and modified versions of existing automatic metrics to evaluate prosody, latency, and robustness. For human evaluations, we adapted existing protocols tailored for measuring the most relevant attributes in the preservation of meaning, naturalness, and expressivity. To ensure that our models can be used safely and responsibly, we implemented the first known red-teaming effort for multimodal machine translation, a system for the detection and mitigation of added toxicity, a systematic evaluation of gender bias, and an inaudible localized watermarking mechanism designed to dampen the impact of deepfakes. Consequently, we bring major components from SeamlessExpressive and SeamlessStreaming together to form Seamless, the first publicly available system that unlocks expressive cross-lingual communication in real-time. In sum, Seamless gives us a pivotal look at the technical foundation needed to turn the Universal Speech Translator from a science fiction concept into a real-world technology. Finally, contributions in this work\u2014including models, code, and a watermark detector\u2014are publicly released and accessible at the link below.*\n\nThis model was contributed by [ylacombe](https://huggingface.co/ylacombe). The original code can be found [here](https://github.com/facebookresearch/seamless_communication).\n\n## Usage tips\n\n- Wav2Vec2-BERT follows the same architecture as Wav2Vec2-Conformer, but employs a causal depthwise convolutional layer and uses as input a mel-spectrogram representation of the audio instead of the raw waveform."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: db87bad8-5088-4b18-bf91-d5b484e9c133
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert", "title": "Wav2Vec2-BERT", "tokens": 1431, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"db87bad8-5088-4b18-bf91-d5b484e9c133\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert\", \"title\": \"Wav2Vec2-BERT\", \"tokens\": 1431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2650753e-bea2-4c35-baff-7795ba332a33\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert\", \"title\": \"Wav2Vec2-BERT\", \"tokens\": 1431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"76199f83168ac684935b57258de857f533f2c2aac536894f6087d91de6052432\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9b25f1ff-b231-400b-a84b-1d3a52eafdc1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert\", \"title\": \"Wav2Vec2-BERT\", \"tokens\": 1431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2b0d106a5362f899f9b16eebdff7cc010e29d965dcb6f6e925ef321302577821\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"968606fc-31d1-4116-9ddd-ac21c95be431\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2e633c2b946c447014c7daa89f3b7aabd29d91ce849309fba975fbcf4b232c52\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1906, \"end_char_idx\": 5121, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2650753e-bea2-4c35-baff-7795ba332a33", "doc_id": "2650753e-bea2-4c35-baff-7795ba332a33", "ref_doc_id": "2650753e-bea2-4c35-baff-7795ba332a33", "chroma:document": "As for SeamlessStreaming, our model leverages the Efficient Monotonic Multihead Attention (EMMA) mechanism to generate low-latency target translations without waiting for complete source utterances. As the first of its kind, SeamlessStreaming enables simultaneous speech-to-speech/text translation for multiple source and target languages. To understand the performance of these models, we combined novel and modified versions of existing automatic metrics to evaluate prosody, latency, and robustness. For human evaluations, we adapted existing protocols tailored for measuring the most relevant attributes in the preservation of meaning, naturalness, and expressivity. To ensure that our models can be used safely and responsibly, we implemented the first known red-teaming effort for multimodal machine translation, a system for the detection and mitigation of added toxicity, a systematic evaluation of gender bias, and an inaudible localized watermarking mechanism designed to dampen the impact of deepfakes. Consequently, we bring major components from SeamlessExpressive and SeamlessStreaming together to form Seamless, the first publicly available system that unlocks expressive cross-lingual communication in real-time. In sum, Seamless gives us a pivotal look at the technical foundation needed to turn the Universal Speech Translator from a science fiction concept into a real-world technology. Finally, contributions in this work\u2014including models, code, and a watermark detector\u2014are publicly released and accessible at the link below.*\n\nThis model was contributed by [ylacombe](https://huggingface.co/ylacombe). The original code can be found [here](https://github.com/facebookresearch/seamless_communication).\n\n## Usage tips\n\n- Wav2Vec2-BERT follows the same architecture as Wav2Vec2-Conformer, but employs a causal depthwise convolutional layer and uses as input a mel-spectrogram representation of the audio instead of the raw waveform.\n- Wav2Vec2-BERT can use either no relative position embeddings, Shaw-like position embeddings, Transformer-XL-like position embeddings, or\n  rotary position embeddings by setting the correct `config.position_embeddings_type`.\n- Wav2Vec2-BERT also introduces a Conformer-based adapter network instead of a simple convolutional network.\n\n## Resources\n\n<PipelineTag pipeline=\"automatic-speech-recognition\"/>\n\n- [`Wav2Vec2BertForCTC`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/speech-recognition).\n- You can also adapt these notebooks on [how to finetune a speech recognition model in English](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/speech_recognition.ipynb), and [how to finetune a speech recognition model in any language](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multi_lingual_speech_recognition.ipynb).\n\n<PipelineTag pipeline=\"audio-classification\"/>\n\n- [`Wav2Vec2BertForSequenceClassification`] can be used by adapting this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/audio-classification).\n- See also: [Audio classification task guide](../tasks/audio_classification)"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 968606fc-31d1-4116-9ddd-ac21c95be431
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert", "title": "Wav2Vec2-BERT", "tokens": 1431, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"968606fc-31d1-4116-9ddd-ac21c95be431\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert\", \"title\": \"Wav2Vec2-BERT\", \"tokens\": 1431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2650753e-bea2-4c35-baff-7795ba332a33\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert\", \"title\": \"Wav2Vec2-BERT\", \"tokens\": 1431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"76199f83168ac684935b57258de857f533f2c2aac536894f6087d91de6052432\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"db87bad8-5088-4b18-bf91-d5b484e9c133\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert\", \"title\": \"Wav2Vec2-BERT\", \"tokens\": 1431, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7dff3bf64c6c2812e1f5de426679ae9d826cc48c860730fff49a94583e87dab1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3452, \"end_char_idx\": 5770, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2650753e-bea2-4c35-baff-7795ba332a33", "doc_id": "2650753e-bea2-4c35-baff-7795ba332a33", "ref_doc_id": "2650753e-bea2-4c35-baff-7795ba332a33", "chroma:document": "*\n\nThis model was contributed by [ylacombe](https://huggingface.co/ylacombe). The original code can be found [here](https://github.com/facebookresearch/seamless_communication).\n\n## Usage tips\n\n- Wav2Vec2-BERT follows the same architecture as Wav2Vec2-Conformer, but employs a causal depthwise convolutional layer and uses as input a mel-spectrogram representation of the audio instead of the raw waveform.\n- Wav2Vec2-BERT can use either no relative position embeddings, Shaw-like position embeddings, Transformer-XL-like position embeddings, or\n  rotary position embeddings by setting the correct `config.position_embeddings_type`.\n- Wav2Vec2-BERT also introduces a Conformer-based adapter network instead of a simple convolutional network.\n\n## Resources\n\n<PipelineTag pipeline=\"automatic-speech-recognition\"/>\n\n- [`Wav2Vec2BertForCTC`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/speech-recognition).\n- You can also adapt these notebooks on [how to finetune a speech recognition model in English](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/speech_recognition.ipynb), and [how to finetune a speech recognition model in any language](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multi_lingual_speech_recognition.ipynb).\n\n<PipelineTag pipeline=\"audio-classification\"/>\n\n- [`Wav2Vec2BertForSequenceClassification`] can be used by adapting this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/audio-classification).\n- See also: [Audio classification task guide](../tasks/audio_classification)\n\n\n## Wav2Vec2BertConfig\n\n[[autodoc]] Wav2Vec2BertConfig\n\n## Wav2Vec2BertProcessor\n\n[[autodoc]] Wav2Vec2BertProcessor\n    - __call__\n    - pad\n    - from_pretrained\n    - save_pretrained\n    - batch_decode\n    - decode\n\n## Wav2Vec2BertModel\n\n[[autodoc]] Wav2Vec2BertModel\n    - forward\n\n## Wav2Vec2BertForCTC\n\n[[autodoc]] Wav2Vec2BertForCTC\n    - forward\n\n## Wav2Vec2BertForSequenceClassification\n\n[[autodoc]] Wav2Vec2BertForSequenceClassification\n    - forward\n\n## Wav2Vec2BertForAudioFrameClassification\n\n[[autodoc]] Wav2Vec2BertForAudioFrameClassification\n    - forward\n\n## Wav2Vec2BertForXVector\n\n[[autodoc]] Wav2Vec2BertForXVector\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 5cb4c880-3957-49da-a3b0-c4fcfd2c12f0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/roformer", "title": "RoFormer", "tokens": 1190, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5cb4c880-3957-49da-a3b0-c4fcfd2c12f0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roformer\", \"title\": \"RoFormer\", \"tokens\": 1190, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97aba92d-60a2-42d5-bea8-f97e614103fa\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roformer\", \"title\": \"RoFormer\", \"tokens\": 1190, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"17479d3a217cd878a98f8600f35ecc4d0c3d51fd8eb0402d29764483afabb83b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"064fd466-830b-433b-99a1-014ae163ec38\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ffab6bcf96998c282e8a43a358cf36503f958dc381f5c1735009fae9a74a59e8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1700, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97aba92d-60a2-42d5-bea8-f97e614103fa", "doc_id": "97aba92d-60a2-42d5-bea8-f97e614103fa", "ref_doc_id": "97aba92d-60a2-42d5-bea8-f97e614103fa", "chroma:document": "# RoFormer\n\n## Overview\n\nThe RoFormer model was proposed in [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://arxiv.org/pdf/2104.09864v1.pdf) by Jianlin Su and Yu Lu and Shengfeng Pan and Bo Wen and Yunfeng Liu.\n\nThe abstract from the paper is the following:\n\n*Position encoding in transformer architecture provides supervision for dependency modeling between elements at\ndifferent positions in the sequence. We investigate various methods to encode positional information in\ntransformer-based language models and propose a novel implementation named Rotary Position Embedding(RoPE). The\nproposed RoPE encodes absolute positional information with rotation matrix and naturally incorporates explicit relative\nposition dependency in self-attention formulation. Notably, RoPE comes with valuable properties such as flexibility of\nbeing expand to any sequence lengths, decaying inter-token dependency with increasing relative distances, and\ncapability of equipping the linear self-attention with relative position encoding. As a result, the enhanced\ntransformer with rotary position embedding, or RoFormer, achieves superior performance in tasks with long texts. We\nrelease the theoretical analysis along with some preliminary experiment results on Chinese data. The undergoing\nexperiment for English benchmark will soon be updated.*\n\nThis model was contributed by [junnyu](https://huggingface.co/junnyu). The original code can be found [here](https://github.com/ZhuiyiTechnology/roformer).\n\n## Usage tips\nRoFormer is a BERT-like autoencoding model with rotary position embeddings. Rotary position embeddings have shown \nimproved performance on classification tasks with long texts."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 064fd466-830b-433b-99a1-014ae163ec38
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/roformer", "title": "RoFormer", "tokens": 1190, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"064fd466-830b-433b-99a1-014ae163ec38\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roformer\", \"title\": \"RoFormer\", \"tokens\": 1190, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97aba92d-60a2-42d5-bea8-f97e614103fa\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roformer\", \"title\": \"RoFormer\", \"tokens\": 1190, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"17479d3a217cd878a98f8600f35ecc4d0c3d51fd8eb0402d29764483afabb83b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5cb4c880-3957-49da-a3b0-c4fcfd2c12f0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roformer\", \"title\": \"RoFormer\", \"tokens\": 1190, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b7834f109bb1166e071a7b3456f8211ab4d7fa7a96438056e035befca1299ddb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 4211, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97aba92d-60a2-42d5-bea8-f97e614103fa", "doc_id": "97aba92d-60a2-42d5-bea8-f97e614103fa", "ref_doc_id": "97aba92d-60a2-42d5-bea8-f97e614103fa", "chroma:document": "# RoFormer\n\n## Overview\n\nThe RoFormer model was proposed in [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://arxiv.org/pdf/2104.09864v1.pdf) by Jianlin Su and Yu Lu and Shengfeng Pan and Bo Wen and Yunfeng Liu.\n\nThe abstract from the paper is the following:\n\n*Position encoding in transformer architecture provides supervision for dependency modeling between elements at\ndifferent positions in the sequence. We investigate various methods to encode positional information in\ntransformer-based language models and propose a novel implementation named Rotary Position Embedding(RoPE). The\nproposed RoPE encodes absolute positional information with rotation matrix and naturally incorporates explicit relative\nposition dependency in self-attention formulation. Notably, RoPE comes with valuable properties such as flexibility of\nbeing expand to any sequence lengths, decaying inter-token dependency with increasing relative distances, and\ncapability of equipping the linear self-attention with relative position encoding. As a result, the enhanced\ntransformer with rotary position embedding, or RoFormer, achieves superior performance in tasks with long texts. We\nrelease the theoretical analysis along with some preliminary experiment results on Chinese data. The undergoing\nexperiment for English benchmark will soon be updated.*\n\nThis model was contributed by [junnyu](https://huggingface.co/junnyu). The original code can be found [here](https://github.com/ZhuiyiTechnology/roformer).\n\n## Usage tips\nRoFormer is a BERT-like autoencoding model with rotary position embeddings. Rotary position embeddings have shown \nimproved performance on classification tasks with long texts.\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## RoFormerConfig\n\n[[autodoc]] RoFormerConfig\n\n## RoFormerTokenizer\n\n[[autodoc]] RoFormerTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## RoFormerTokenizerFast\n\n[[autodoc]] RoFormerTokenizerFast\n    - build_inputs_with_special_tokens\n\n<frameworkcontent>\n<pt>\n\n## RoFormerModel\n\n[[autodoc]] RoFormerModel\n    - forward\n\n## RoFormerForCausalLM\n\n[[autodoc]] RoFormerForCausalLM\n    - forward\n\n## RoFormerForMaskedLM\n\n[[autodoc]] RoFormerForMaskedLM\n    - forward\n\n## RoFormerForSequenceClassification\n\n[[autodoc]] RoFormerForSequenceClassification\n    - forward\n\n## RoFormerForMultipleChoice\n\n[[autodoc]] RoFormerForMultipleChoice\n    - forward\n\n## RoFormerForTokenClassification\n\n[[autodoc]] RoFormerForTokenClassification\n    - forward\n\n## RoFormerForQuestionAnswering\n\n[[autodoc]] RoFormerForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFRoFormerModel\n\n[[autodoc]] TFRoFormerModel\n    - call\n\n## TFRoFormerForMaskedLM\n\n[[autodoc]] TFRoFormerForMaskedLM\n    - call\n\n## TFRoFormerForCausalLM\n\n[[autodoc]] TFRoFormerForCausalLM\n    - call\n\n## TFRoFormerForSequenceClassification\n\n[[autodoc]] TFRoFormerForSequenceClassification\n    - call\n\n## TFRoFormerForMultipleChoice\n\n[[autodoc]] TFRoFormerForMultipleChoice\n    - call\n\n## TFRoFormerForTokenClassification\n\n[[autodoc]] TFRoFormerForTokenClassification\n    - call\n\n## TFRoFormerForQuestionAnswering\n\n[[autodoc]] TFRoFormerForQuestionAnswering\n    - call\n\n</tf>\n<jax>\n\n## FlaxRoFormerModel\n\n[[autodoc]] FlaxRoFormerModel\n    - __call__\n\n## FlaxRoFormerForMaskedLM\n\n[[autodoc]] FlaxRoFormerForMaskedLM\n    - __call__\n\n## FlaxRoFormerForSequenceClassification\n\n[[autodoc]] FlaxRoFormerForSequenceClassification\n    - __call__\n\n## FlaxRoFormerForMultipleChoice\n\n[[autodoc]] FlaxRoFormerForMultipleChoice\n    - __call__\n\n## FlaxRoFormerForTokenClassification\n\n[[autodoc]] FlaxRoFormerForTokenClassification\n    - __call__\n\n## FlaxRoFormerForQuestionAnswering\n\n[[autodoc]] FlaxRoFormerForQuestionAnswering\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 78724783-2b89-4d67-892c-40b70bc7f3ce
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/owlvit", "title": "OWL-ViT", "tokens": 1628, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"78724783-2b89-4d67-892c-40b70bc7f3ce\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/owlvit\", \"title\": \"OWL-ViT\", \"tokens\": 1628, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7e5e5ee0-fd35-4b10-902b-a535875dfe30\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/owlvit\", \"title\": \"OWL-ViT\", \"tokens\": 1628, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b6e34bf2940493b31335054f63c3e644a3eeb7c581e0f8e60cf899b66ac57329\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b9480d4d-4b21-4c51-b1a2-50394c6c035f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"81e302000b73b7f8ce2ec8be03622b6551b8faebac0a5553ec089d84c0e61ac6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3421, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7e5e5ee0-fd35-4b10-902b-a535875dfe30", "doc_id": "7e5e5ee0-fd35-4b10-902b-a535875dfe30", "ref_doc_id": "7e5e5ee0-fd35-4b10-902b-a535875dfe30", "chroma:document": "# OWL-ViT\n\n## Overview\n\nThe OWL-ViT (short for Vision Transformer for Open-World Localization) was proposed in [Simple Open-Vocabulary Object Detection with Vision Transformers](https://arxiv.org/abs/2205.06230) by Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, Xiao Wang, Xiaohua Zhai, Thomas Kipf, and Neil Houlsby. OWL-ViT is an open-vocabulary object detection network trained on a variety of (image, text) pairs. It can be used to query an image with one or multiple text queries to search for and detect target objects described in text.\n\nThe abstract from the paper is the following:\n\n*Combining simple architectures with large-scale pre-training has led to massive improvements in image classification. For object detection, pre-training and scaling approaches are less well established, especially in the long-tailed and open-vocabulary setting, where training data is relatively scarce. In this paper, we propose a strong recipe for transferring image-text models to open-vocabulary object detection. We use a standard Vision Transformer architecture with minimal modifications, contrastive image-text pre-training, and end-to-end detection fine-tuning. Our analysis of the scaling properties of this setup shows that increasing image-level pre-training and model size yield consistent improvements on the downstream detection task. We provide the adaptation strategies and regularizations needed to attain very strong performance on zero-shot text-conditioned and one-shot image-conditioned object detection. Code and models are available on GitHub.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/owlvit_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> OWL-ViT architecture. Taken from the <a href=\"https://arxiv.org/abs/2205.06230\">original paper</a>. </small>\n\nThis model was contributed by [adirik](https://huggingface.co/adirik). The original code can be found [here](https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit).\n\n## Usage tips\n\nOWL-ViT is a zero-shot text-conditioned object detection model. OWL-ViT uses [CLIP](clip) as its multi-modal backbone, with a ViT-like Transformer to get visual features and a causal language model to get the text features. To use CLIP for detection, OWL-ViT removes the final token pooling layer of the vision model and attaches a lightweight classification and box head to each transformer output token. Open-vocabulary classification is enabled by replacing the fixed classification layer weights with the class-name embeddings obtained from the text model. The authors first train CLIP from scratch and fine-tune it end-to-end with the classification and box heads on standard detection datasets using a bipartite matching loss. One or multiple text queries per image can be used to perform zero-shot text-conditioned object detection.\n\n[`OwlViTImageProcessor`] can be used to resize (or rescale) and normalize images for the model and [`CLIPTokenizer`] is used to encode the text. [`OwlViTProcessor`] wraps [`OwlViTImageProcessor`] and [`CLIPTokenizer`] into a single instance to both encode the text and prepare the images. The following example shows how to perform object detection using [`OwlViTProcessor`] and [`OwlViTForObjectDetection`]."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: b9480d4d-4b21-4c51-b1a2-50394c6c035f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/owlvit", "title": "OWL-ViT", "tokens": 1628, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b9480d4d-4b21-4c51-b1a2-50394c6c035f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/owlvit\", \"title\": \"OWL-ViT\", \"tokens\": 1628, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7e5e5ee0-fd35-4b10-902b-a535875dfe30\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/owlvit\", \"title\": \"OWL-ViT\", \"tokens\": 1628, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b6e34bf2940493b31335054f63c3e644a3eeb7c581e0f8e60cf899b66ac57329\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"78724783-2b89-4d67-892c-40b70bc7f3ce\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/owlvit\", \"title\": \"OWL-ViT\", \"tokens\": 1628, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8c4bcb4ebe89b6a230fbd6f613aaea9a7fb9e195fca451dfc4ded5821d59ef12\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c0f216e4-e89d-484f-8e57-36d5612115f3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0b1eb6de101be3d103adb3b894974e3eec3d64ad5f84861710a948747eb302af\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1881, \"end_char_idx\": 5186, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7e5e5ee0-fd35-4b10-902b-a535875dfe30", "doc_id": "7e5e5ee0-fd35-4b10-902b-a535875dfe30", "ref_doc_id": "7e5e5ee0-fd35-4b10-902b-a535875dfe30", "chroma:document": "Taken from the <a href=\"https://arxiv.org/abs/2205.06230\">original paper</a>. </small>\n\nThis model was contributed by [adirik](https://huggingface.co/adirik). The original code can be found [here](https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit).\n\n## Usage tips\n\nOWL-ViT is a zero-shot text-conditioned object detection model. OWL-ViT uses [CLIP](clip) as its multi-modal backbone, with a ViT-like Transformer to get visual features and a causal language model to get the text features. To use CLIP for detection, OWL-ViT removes the final token pooling layer of the vision model and attaches a lightweight classification and box head to each transformer output token. Open-vocabulary classification is enabled by replacing the fixed classification layer weights with the class-name embeddings obtained from the text model. The authors first train CLIP from scratch and fine-tune it end-to-end with the classification and box heads on standard detection datasets using a bipartite matching loss. One or multiple text queries per image can be used to perform zero-shot text-conditioned object detection.\n\n[`OwlViTImageProcessor`] can be used to resize (or rescale) and normalize images for the model and [`CLIPTokenizer`] is used to encode the text. [`OwlViTProcessor`] wraps [`OwlViTImageProcessor`] and [`CLIPTokenizer`] into a single instance to both encode the text and prepare the images. The following example shows how to perform object detection using [`OwlViTProcessor`] and [`OwlViTForObjectDetection`].\n\n```python\n>>> import requests\n>>> from PIL import Image\n>>> import torch\n\n>>> from transformers import OwlViTProcessor, OwlViTForObjectDetection\n\n>>> processor = OwlViTProcessor.from_pretrained(\"google/owlvit-base-patch32\")\n>>> model = OwlViTForObjectDetection.from_pretrained(\"google/owlvit-base-patch32\")\n\n>>> url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n>>> texts = [[\"a photo of a cat\", \"a photo of a dog\"]]\n>>> inputs = processor(text=texts, images=image, return_tensors=\"pt\")\n>>> outputs = model(**inputs)\n\n>>> # Target image sizes (height, width) to rescale box predictions [batch_size, 2]\n>>> target_sizes = torch.Tensor([image.size[::-1]])\n>>> # Convert outputs (bounding boxes and class logits) to Pascal VOC format (xmin, ymin, xmax, ymax)\n>>> results = processor.post_process_object_detection(outputs=outputs, target_sizes=target_sizes, threshold=0.1)\n>>> i = 0  # Retrieve predictions for the first image for the corresponding text queries\n>>> text = texts[i]\n>>> boxes, scores, labels = results[i][\"boxes\"], results[i][\"scores\"], results[i][\"labels\"]\n>>> for box, score, label in zip(boxes, scores, labels):\n...     box = [round(i, 2) for i in box.tolist()]\n...     print(f\"Detected {text[label]} with confidence {round(score.item(), 3)} at location {box}\")\nDetected a photo of a cat with confidence 0.707 at location [324.97, 20.44, 640.58, 373.29]\nDetected a photo of a cat with confidence 0.717 at location [1.46, 55.26, 315.55, 472.17]\n```\n\n## Resources\n\nA demo notebook on using OWL-ViT for zero- and one-shot (image-guided) object detection can be found [here](https://github.com/huggingface/notebooks/blob/main/examples/zeroshot_object_detection_with_owlvit.ipynb)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: c0f216e4-e89d-484f-8e57-36d5612115f3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/owlvit", "title": "OWL-ViT", "tokens": 1628, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c0f216e4-e89d-484f-8e57-36d5612115f3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/owlvit\", \"title\": \"OWL-ViT\", \"tokens\": 1628, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7e5e5ee0-fd35-4b10-902b-a535875dfe30\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/owlvit\", \"title\": \"OWL-ViT\", \"tokens\": 1628, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b6e34bf2940493b31335054f63c3e644a3eeb7c581e0f8e60cf899b66ac57329\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b9480d4d-4b21-4c51-b1a2-50394c6c035f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/owlvit\", \"title\": \"OWL-ViT\", \"tokens\": 1628, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"069228ac6e7f32ee56fc606c95a0475003183ae4624651071b9ce1e1f790ee8e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5188, \"end_char_idx\": 6052, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7e5e5ee0-fd35-4b10-902b-a535875dfe30", "doc_id": "7e5e5ee0-fd35-4b10-902b-a535875dfe30", "ref_doc_id": "7e5e5ee0-fd35-4b10-902b-a535875dfe30", "chroma:document": "## OwlViTConfig\n\n[[autodoc]] OwlViTConfig\n    - from_text_vision_configs\n\n## OwlViTTextConfig\n\n[[autodoc]] OwlViTTextConfig\n\n## OwlViTVisionConfig\n\n[[autodoc]] OwlViTVisionConfig\n\n## OwlViTImageProcessor\n\n[[autodoc]] OwlViTImageProcessor\n    - preprocess\n    - post_process_object_detection\n    - post_process_image_guided_detection\n\n## OwlViTFeatureExtractor\n\n[[autodoc]] OwlViTFeatureExtractor\n    - __call__\n    - post_process\n    - post_process_image_guided_detection\n\n## OwlViTProcessor\n\n[[autodoc]] OwlViTProcessor\n\n## OwlViTModel\n\n[[autodoc]] OwlViTModel\n    - forward\n    - get_text_features\n    - get_image_features\n\n## OwlViTTextModel\n\n[[autodoc]] OwlViTTextModel\n    - forward\n\n## OwlViTVisionModel\n\n[[autodoc]] OwlViTVisionModel\n    - forward\n\n## OwlViTForObjectDetection\n\n[[autodoc]] OwlViTForObjectDetection\n    - forward\n    - image_guided_detection"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0953080d-064d-4161-afa8-465c0ce4be27
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/rwkv", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0953080d-064d-4161-afa8-465c0ce4be27\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rwkv\", \"title\": \"RWKV\", \"tokens\": 2201, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"14ec845c-6a7d-45c0-88c7-e35436a441c9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rwkv\", \"title\": \"RWKV\", \"tokens\": 2201, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3bacb966fdc0bfa53f5976446af703cbd85b949e2283109b598be93c2d0ae83d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4fc3f7f9-8551-4bb9-b65e-12a623e04555\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f8d1190655f679eb1475dd006586be6f3ff80afc08dcd97ee8696cf225586bda\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3003, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "14ec845c-6a7d-45c0-88c7-e35436a441c9", "doc_id": "14ec845c-6a7d-45c0-88c7-e35436a441c9", "ref_doc_id": "14ec845c-6a7d-45c0-88c7-e35436a441c9", "chroma:document": "# RWKV\n\n## Overview\n\nThe RWKV model was proposed in [this repo](https://github.com/BlinkDL/RWKV-LM)\n\nIt suggests a tweak in the traditional Transformer attention to make it linear. This way, the model can be used as recurrent network: passing inputs for timestamp 0 and timestamp 1 together is the same as passing inputs at timestamp 0, then inputs at timestamp 1 along with the state of timestamp 0 (see example below).\n\nThis can be more efficient than a regular Transformer and can deal with sentence of any length (even if the model uses a fixed context length for training).\n\nThis model was contributed by [sgugger](https://huggingface.co/sgugger).\nThe original code can be found [here](https://github.com/BlinkDL/RWKV-LM).\n\n## Usage example\n\n```py\nimport torch\nfrom transformers import AutoTokenizer, RwkvConfig, RwkvModel\n\nmodel = RwkvModel.from_pretrained(\"sgugger/rwkv-430M-pile\")\ntokenizer = AutoTokenizer.from_pretrained(\"sgugger/rwkv-430M-pile\")\n\ninputs = tokenizer(\"This is an example.\", return_tensors=\"pt\")\n# Feed everything to the model\noutputs = model(inputs[\"input_ids\"])\noutput_whole = outputs.last_hidden_state\n\noutputs = model(inputs[\"input_ids\"][:, :2])\noutput_one = outputs.last_hidden_state\n\n# Using the state computed on the first inputs, we will get the same output\noutputs = model(inputs[\"input_ids\"][:, 2:], state=outputs.state)\noutput_two = outputs.last_hidden_state\n\ntorch.allclose(torch.cat([output_one, output_two], dim=1), output_whole, atol=1e-5)\n```\n\nIf you want to make sure the model stops generating when `'\\n\\n'` is detected, we recommend using the following stopping criteria:\n\n```python \nfrom transformers import StoppingCriteria\n\nclass RwkvStoppingCriteria(StoppingCriteria):\n    def __init__(self, eos_sequence = [187,187], eos_token_id = 537):\n        self.eos_sequence = eos_sequence\n        self.eos_token_id = eos_token_id\n\n    def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool:\n        last_2_ids = input_ids[:,-2:].tolist()\n        return self.eos_sequence in last_2_ids\n\n\noutput = model.generate(inputs[\"input_ids\"], max_new_tokens=64, stopping_criteria = [RwkvStoppingCriteria()])\n```\n\n## RwkvConfig\n\n[[autodoc]] RwkvConfig\n\n## RwkvModel\n\n[[autodoc]] RwkvModel\n    - forward\n\n## RwkvLMHeadModel\n\n[[autodoc]] RwkvForCausalLM\n    - forward\n\n## Rwkv attention and the recurrent formulas\n\nIn a traditional auto-regressive Transformer, attention is written as\n\n$$O = \\hbox{softmax}(QK^{T} / \\sqrt{d}) V$$\n\nwith \\\\(Q\\\\), \\\\(K\\\\) and \\\\(V\\\\) are matrices of shape `seq_len x hidden_size` named query, key and value (they are actually bigger matrices with a batch dimension and an attention head dimension but we're only interested in the last two, which is where the matrix product is taken, so for the sake of simplicity we only consider those two). The product \\\\(QK^{T}\\\\) then has shape `seq_len x seq_len` and we can take the matrix product with \\\\(V\\\\) to get the output \\\\(O\\\\) of the same shape as the others."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 4fc3f7f9-8551-4bb9-b65e-12a623e04555
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/rwkv", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4fc3f7f9-8551-4bb9-b65e-12a623e04555\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rwkv\", \"title\": \"RWKV\", \"tokens\": 2201, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"14ec845c-6a7d-45c0-88c7-e35436a441c9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rwkv\", \"title\": \"RWKV\", \"tokens\": 2201, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3bacb966fdc0bfa53f5976446af703cbd85b949e2283109b598be93c2d0ae83d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0953080d-064d-4161-afa8-465c0ce4be27\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rwkv\", \"title\": \"RWKV\", \"tokens\": 2201, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"735511b3960569bb826cedaa9a89fa62fbcf1af733529378c5ea4b4abc2ef438\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"05dc3fe4-8ead-4c92-aa06-283f585babec\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"45d12045c247d8d9d94d6b0070a8be57312cf7cc67a0497b44465a1bb9e6efb2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2064, \"end_char_idx\": 3986, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "14ec845c-6a7d-45c0-88c7-e35436a441c9", "doc_id": "14ec845c-6a7d-45c0-88c7-e35436a441c9", "ref_doc_id": "14ec845c-6a7d-45c0-88c7-e35436a441c9", "chroma:document": "output = model.generate(inputs[\"input_ids\"], max_new_tokens=64, stopping_criteria = [RwkvStoppingCriteria()])\n```\n\n## RwkvConfig\n\n[[autodoc]] RwkvConfig\n\n## RwkvModel\n\n[[autodoc]] RwkvModel\n    - forward\n\n## RwkvLMHeadModel\n\n[[autodoc]] RwkvForCausalLM\n    - forward\n\n## Rwkv attention and the recurrent formulas\n\nIn a traditional auto-regressive Transformer, attention is written as\n\n$$O = \\hbox{softmax}(QK^{T} / \\sqrt{d}) V$$\n\nwith \\\\(Q\\\\), \\\\(K\\\\) and \\\\(V\\\\) are matrices of shape `seq_len x hidden_size` named query, key and value (they are actually bigger matrices with a batch dimension and an attention head dimension but we're only interested in the last two, which is where the matrix product is taken, so for the sake of simplicity we only consider those two). The product \\\\(QK^{T}\\\\) then has shape `seq_len x seq_len` and we can take the matrix product with \\\\(V\\\\) to get the output \\\\(O\\\\) of the same shape as the others.  \n\nReplacing the softmax by its value gives:\n\n$$O_{i} = \\frac{\\sum_{j=1}^{i} e^{Q_{i} K_{j}^{T} / \\sqrt{d}} V_{j}}{\\sum_{j=1}^{i} e^{Q_{i} K_{j}^{T} / \\sqrt{d}}}$$\n\nNote that the entries in \\\\(QK^{T}\\\\) corresponding to \\\\(j > i\\\\) are masked (the sum stops at j) because the attention is not allowed to look at future tokens (only past ones).\n\nIn comparison, the RWKV attention is given by\n\n$$O_{i} = \\sigma(R_{i}) \\frac{\\sum_{j=1}^{i} e^{W_{i-j} + K_{j}} V_{j}}{\\sum_{j=1}^{i} e^{W_{i-j} + K_{j}}}$$\n\nwhere \\\\(R\\\\) is a new matrix called receptance by the author, \\\\(K\\\\) and \\\\(V\\\\) are still the key and value (\\\\(\\sigma\\\\) here is the sigmoid function). \\\\(W\\\\) is a new vector that represents the position of the token and is given by\n\n$$W_{0} = u \\hbox{  and  } W_{k} = (k-1)w \\hbox{ for } k \\geq 1$$\n\nwith \\\\(u\\\\) and \\\\(w\\\\) learnable parameters called in the code `time_first` and `time_decay` respectively. The numerator and denominator can both be expressed recursively."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 05dc3fe4-8ead-4c92-aa06-283f585babec
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/rwkv", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"05dc3fe4-8ead-4c92-aa06-283f585babec\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rwkv\", \"title\": \"RWKV\", \"tokens\": 2201, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"14ec845c-6a7d-45c0-88c7-e35436a441c9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rwkv\", \"title\": \"RWKV\", \"tokens\": 2201, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3bacb966fdc0bfa53f5976446af703cbd85b949e2283109b598be93c2d0ae83d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4fc3f7f9-8551-4bb9-b65e-12a623e04555\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rwkv\", \"title\": \"RWKV\", \"tokens\": 2201, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8d1b70b95ef83f40dbcb4b0ce55acc38d535591b803fb44be8fb4a8c0e7b474d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"838d9526-2396-4978-8856-b7d108ba1bed\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9a4394f13ca6b85438cddb0baecbdb6c1cf8dcd382f551326a7abc2c410d054a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3007, \"end_char_idx\": 4787, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "14ec845c-6a7d-45c0-88c7-e35436a441c9", "doc_id": "14ec845c-6a7d-45c0-88c7-e35436a441c9", "ref_doc_id": "14ec845c-6a7d-45c0-88c7-e35436a441c9", "chroma:document": "Replacing the softmax by its value gives:\n\n$$O_{i} = \\frac{\\sum_{j=1}^{i} e^{Q_{i} K_{j}^{T} / \\sqrt{d}} V_{j}}{\\sum_{j=1}^{i} e^{Q_{i} K_{j}^{T} / \\sqrt{d}}}$$\n\nNote that the entries in \\\\(QK^{T}\\\\) corresponding to \\\\(j > i\\\\) are masked (the sum stops at j) because the attention is not allowed to look at future tokens (only past ones).\n\nIn comparison, the RWKV attention is given by\n\n$$O_{i} = \\sigma(R_{i}) \\frac{\\sum_{j=1}^{i} e^{W_{i-j} + K_{j}} V_{j}}{\\sum_{j=1}^{i} e^{W_{i-j} + K_{j}}}$$\n\nwhere \\\\(R\\\\) is a new matrix called receptance by the author, \\\\(K\\\\) and \\\\(V\\\\) are still the key and value (\\\\(\\sigma\\\\) here is the sigmoid function). \\\\(W\\\\) is a new vector that represents the position of the token and is given by\n\n$$W_{0} = u \\hbox{  and  } W_{k} = (k-1)w \\hbox{ for } k \\geq 1$$\n\nwith \\\\(u\\\\) and \\\\(w\\\\) learnable parameters called in the code `time_first` and `time_decay` respectively. The numerator and denominator can both be expressed recursively. Naming them \\\\(N_{i}\\\\) and \\\\(D_{i}\\\\) we have:\n\n$$N_{i} = e^{u + K_{i}} V_{i} + \\hat{N}_{i} \\hbox{  where  } \\hat{N}_{i} = e^{K_{i-1}} V_{i-1} + e^{w + K_{i-2}} V_{i-2} \\cdots + e^{(i-2)w + K_{1}} V_{1}$$\n\nso \\\\(\\hat{N}_{i}\\\\) (called `numerator_state` in the code) satisfies\n\n$$\\hat{N}_{0} = 0 \\hbox{  and  } \\hat{N}_{j+1} = e^{K_{j}} V_{j} + e^{w} \\hat{N}_{j}$$\n\nand\n\n$$D_{i} = e^{u + K_{i}} + \\hat{D}_{i} \\hbox{  where  } \\hat{D}_{i} = e^{K_{i-1}} + e^{w + K_{i-2}} \\cdots + e^{(i-2)w + K_{1}}$$\n\nso \\\\(\\hat{D}_{i}\\\\) (called `denominator_state` in the code) satisfies\n\n$$\\hat{D}_{0} = 0 \\hbox{  and  } \\hat{D}_{j+1} = e^{K_{j}} + e^{w} \\hat{D}_{j}$$\n\nThe actual recurrent formula used are a tiny bit more complex, as for numerical stability we don't want to compute exponentials of big numbers."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 838d9526-2396-4978-8856-b7d108ba1bed
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/rwkv", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"838d9526-2396-4978-8856-b7d108ba1bed\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rwkv\", \"title\": \"RWKV\", \"tokens\": 2201, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"14ec845c-6a7d-45c0-88c7-e35436a441c9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rwkv\", \"title\": \"RWKV\", \"tokens\": 2201, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3bacb966fdc0bfa53f5976446af703cbd85b949e2283109b598be93c2d0ae83d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"05dc3fe4-8ead-4c92-aa06-283f585babec\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rwkv\", \"title\": \"RWKV\", \"tokens\": 2201, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b8f242904abfe0f180319d22a9ee88217e017df62cfbf4d87d71b5c8c3f16f9e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"124e3432-6ac4-421f-aabb-4398174006b9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"131f8a39cbec6c5f7e458881eab01c3e4c0255e2444cc44b2aecc9c64db9c021\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3922, \"end_char_idx\": 5245, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "14ec845c-6a7d-45c0-88c7-e35436a441c9", "doc_id": "14ec845c-6a7d-45c0-88c7-e35436a441c9", "ref_doc_id": "14ec845c-6a7d-45c0-88c7-e35436a441c9", "chroma:document": "The numerator and denominator can both be expressed recursively. Naming them \\\\(N_{i}\\\\) and \\\\(D_{i}\\\\) we have:\n\n$$N_{i} = e^{u + K_{i}} V_{i} + \\hat{N}_{i} \\hbox{  where  } \\hat{N}_{i} = e^{K_{i-1}} V_{i-1} + e^{w + K_{i-2}} V_{i-2} \\cdots + e^{(i-2)w + K_{1}} V_{1}$$\n\nso \\\\(\\hat{N}_{i}\\\\) (called `numerator_state` in the code) satisfies\n\n$$\\hat{N}_{0} = 0 \\hbox{  and  } \\hat{N}_{j+1} = e^{K_{j}} V_{j} + e^{w} \\hat{N}_{j}$$\n\nand\n\n$$D_{i} = e^{u + K_{i}} + \\hat{D}_{i} \\hbox{  where  } \\hat{D}_{i} = e^{K_{i-1}} + e^{w + K_{i-2}} \\cdots + e^{(i-2)w + K_{1}}$$\n\nso \\\\(\\hat{D}_{i}\\\\) (called `denominator_state` in the code) satisfies\n\n$$\\hat{D}_{0} = 0 \\hbox{  and  } \\hat{D}_{j+1} = e^{K_{j}} + e^{w} \\hat{D}_{j}$$\n\nThe actual recurrent formula used are a tiny bit more complex, as for numerical stability we don't want to compute exponentials of big numbers. Usually the softmax is not computed as is, but the exponential of the maximum term is divided of the numerator and denominator:\n\n$$\\frac{e^{x_{i}}}{\\sum_{j=1}^{n} e^{x_{j}}} = \\frac{e^{x_{i} - M}}{\\sum_{j=1}^{n} e^{x_{j} - M}}$$\n\nwith \\\\(M\\\\) the maximum of all \\\\(x_{j}\\\\). So here on top of saving the numerator state (\\\\(\\hat{N}\\\\)) and the denominator state (\\\\(\\hat{D}\\\\)) we also keep track of the maximum of all terms encountered in the exponentials."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 124e3432-6ac4-421f-aabb-4398174006b9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/rwkv", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"124e3432-6ac4-421f-aabb-4398174006b9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rwkv\", \"title\": \"RWKV\", \"tokens\": 2201, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"14ec845c-6a7d-45c0-88c7-e35436a441c9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rwkv\", \"title\": \"RWKV\", \"tokens\": 2201, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3bacb966fdc0bfa53f5976446af703cbd85b949e2283109b598be93c2d0ae83d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"838d9526-2396-4978-8856-b7d108ba1bed\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rwkv\", \"title\": \"RWKV\", \"tokens\": 2201, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"26ec72e9d3b6f5221ee6753093cdf86f43ad651ac91c830c60bf3a3e99d08b23\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4788, \"end_char_idx\": 6055, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "14ec845c-6a7d-45c0-88c7-e35436a441c9", "doc_id": "14ec845c-6a7d-45c0-88c7-e35436a441c9", "ref_doc_id": "14ec845c-6a7d-45c0-88c7-e35436a441c9", "chroma:document": "Usually the softmax is not computed as is, but the exponential of the maximum term is divided of the numerator and denominator:\n\n$$\\frac{e^{x_{i}}}{\\sum_{j=1}^{n} e^{x_{j}}} = \\frac{e^{x_{i} - M}}{\\sum_{j=1}^{n} e^{x_{j} - M}}$$\n\nwith \\\\(M\\\\) the maximum of all \\\\(x_{j}\\\\). So here on top of saving the numerator state (\\\\(\\hat{N}\\\\)) and the denominator state (\\\\(\\hat{D}\\\\)) we also keep track of the maximum of all terms encountered in the exponentials. So we actually use\n\n$$\\tilde{N}_{i} = e^{-M_{i}} \\hat{N}_{i} \\hbox{  and  } \\tilde{D}_{i} = e^{-M_{i}} \\hat{D}_{i}$$\n\ndefined by the following recurrent formulas:\n\n$$\\tilde{N}_{0} = 0 \\hbox{  and  } \\tilde{N}_{j+1} = e^{K_{j} - q} V_{j} + e^{w + M_{j} - q} \\tilde{N}_{j} \\hbox{  where  } q = \\max(K_{j}, w + M_{j})$$\n\nand\n\n$$\\tilde{D}_{0} = 0 \\hbox{  and  } \\tilde{D}_{j+1} = e^{K_{j} - q} + e^{w + M_{j} - q} \\tilde{D}_{j} \\hbox{  where  } q = \\max(K_{j}, w + M_{j})$$\n\nand \\\\(M_{j+1} = q\\\\). With those, we can then compute\n\n$$N_{i} = e^{u + K_{i} - q} V_{i} + e^{M_{i}} \\tilde{N}_{i} \\hbox{  where  } q = \\max(u + K_{i}, M_{i})$$\n\nand\n\n$$D_{i} = e^{u + K_{i} - q} + e^{M_{i}} \\tilde{D}_{i} \\hbox{  where  } q = \\max(u + K_{i}, M_{i})$$\n\nwhich finally gives us\n\n$$O_{i} = \\sigma(R_{i}) \\frac{N_{i}}{D_{i}}$$"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f6960842-b042-41f2-8b4a-cad62a1eb8ad
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/dbrx", "title": "DBRX", "tokens": 1388, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f6960842-b042-41f2-8b4a-cad62a1eb8ad\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dbrx\", \"title\": \"DBRX\", \"tokens\": 1388, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"237d368a-51d7-40b7-8218-61fa59853413\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dbrx\", \"title\": \"DBRX\", \"tokens\": 1388, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cbeefe8faa3f53bbc16032eaa766d728570874f11046f75511c923cc57c3c9f2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3eb0f280-ddba-419b-b005-8b09dfb4793d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d51dd502d58096dfb044b6e88d1e14bd808ca8f17971a1d98c9d9f615b8069da\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3218, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "237d368a-51d7-40b7-8218-61fa59853413", "doc_id": "237d368a-51d7-40b7-8218-61fa59853413", "ref_doc_id": "237d368a-51d7-40b7-8218-61fa59853413", "chroma:document": "# DBRX\n\n## Overview\n\nDBRX is a [transformer-based](https://www.isattentionallyouneed.com/) decoder-only large language model (LLM) that was trained using next-token prediction.\nIt uses a *fine-grained* mixture-of-experts (MoE) architecture with 132B total parameters of which 36B parameters are active on any input.\nIt was pre-trained on 12T tokens of text and code data.\nCompared to other open MoE models like Mixtral-8x7B and Grok-1, DBRX is fine-grained, meaning it uses a larger number of smaller experts. DBRX has 16 experts and chooses 4, while Mixtral-8x7B and Grok-1 have 8 experts and choose 2.\nThis provides 65x more possible combinations of experts and we found that this improves model quality.\nDBRX uses rotary position encodings (RoPE), gated linear units (GLU), and grouped query attention (GQA).\nIt is a BPE based model and uses the GPT-4 tokenizer as described in the [tiktoken](https://github.com/openai/tiktoken) repository.\nWe made these choices based on exhaustive evaluation and scaling experiments.\n\nDBRX was pretrained on 12T tokens of carefully curated data and a maximum context length of 32K tokens.\nWe estimate that this data is at least 2x better token-for-token than the data we used to pretrain the MPT family of models.\nThis new dataset was developed using the full suite of Databricks tools, including Apache Spark\u2122 and Databricks notebooks for data processing, and Unity Catalog for data management and governance.\nWe used curriculum learning for pretraining, changing the data mix during training in ways we found to substantially improve model quality.\n\n\nMore detailed information about DBRX Instruct and DBRX Base can be found in our [technical blog post](https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm).\n\nThis model was contributed by [eitan-turok](https://huggingface.co/eitanturok) and [abhi-db](https://huggingface.co/abhi-db). The original code can be found [here](https://github.com/databricks/dbrx-instruct), though this may not be up to date.\n\n## Usage Examples\n\nThe `generate()` method can be used to generate text using DBRX. You can generate using the standard attention implementation, flash-attention, and the PyTorch scaled dot product attention. The last two attention implementations give speed ups.\n\n```python\nfrom transformers import DbrxForCausalLM, AutoTokenizer\nimport torch\n\ntokenizer = AutoTokenizer.from_pretrained(\"databricks/dbrx-instruct\", token=\"YOUR_HF_TOKEN\")\nmodel = DbrxForCausalLM.from_pretrained(\n    \"databricks/dbrx-instruct\",\n    device_map=\"auto\",\n    torch_dtype=torch.bfloat16,\n    token=\"YOUR_HF_TOKEN\",\n    )\n\ninput_text = \"What does it take to build a great LLM?\"\nmessages = [{\"role\": \"user\", \"content\": input_text}]\ninput_ids = tokenizer.apply_chat_template(messages, return_dict=True, tokenize=True, add_generation_prompt=True, return_tensors=\"pt\").to(\"cuda\")\n\noutputs = model.generate(**input_ids, max_new_tokens=200)\nprint(tokenizer.decode(outputs[0]))\n```\n\nIf you have flash-attention installed (`pip install flash-attn`), it is possible to generate faster. (The HuggingFace documentation for flash-attention can be found [here](https://huggingface.co/docs/transformers/perf_infer_gpu_one#flashattention-2).)"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 3eb0f280-ddba-419b-b005-8b09dfb4793d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/dbrx", "title": "DBRX", "tokens": 1388, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3eb0f280-ddba-419b-b005-8b09dfb4793d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dbrx\", \"title\": \"DBRX\", \"tokens\": 1388, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"237d368a-51d7-40b7-8218-61fa59853413\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dbrx\", \"title\": \"DBRX\", \"tokens\": 1388, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cbeefe8faa3f53bbc16032eaa766d728570874f11046f75511c923cc57c3c9f2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f6960842-b042-41f2-8b4a-cad62a1eb8ad\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dbrx\", \"title\": \"DBRX\", \"tokens\": 1388, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b149284a770cf4b201410c88b19b87948d5b7cafa831b00452fb9115c5555418\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c7bcca2b-f562-4bc2-a1b1-ab16d0c973d1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5cb61f56f7d938e69d7b06c63e1ccdd68aac00938c7f9bf61ff3644798933570\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1768, \"end_char_idx\": 4634, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "237d368a-51d7-40b7-8218-61fa59853413", "doc_id": "237d368a-51d7-40b7-8218-61fa59853413", "ref_doc_id": "237d368a-51d7-40b7-8218-61fa59853413", "chroma:document": "This model was contributed by [eitan-turok](https://huggingface.co/eitanturok) and [abhi-db](https://huggingface.co/abhi-db). The original code can be found [here](https://github.com/databricks/dbrx-instruct), though this may not be up to date.\n\n## Usage Examples\n\nThe `generate()` method can be used to generate text using DBRX. You can generate using the standard attention implementation, flash-attention, and the PyTorch scaled dot product attention. The last two attention implementations give speed ups.\n\n```python\nfrom transformers import DbrxForCausalLM, AutoTokenizer\nimport torch\n\ntokenizer = AutoTokenizer.from_pretrained(\"databricks/dbrx-instruct\", token=\"YOUR_HF_TOKEN\")\nmodel = DbrxForCausalLM.from_pretrained(\n    \"databricks/dbrx-instruct\",\n    device_map=\"auto\",\n    torch_dtype=torch.bfloat16,\n    token=\"YOUR_HF_TOKEN\",\n    )\n\ninput_text = \"What does it take to build a great LLM?\"\nmessages = [{\"role\": \"user\", \"content\": input_text}]\ninput_ids = tokenizer.apply_chat_template(messages, return_dict=True, tokenize=True, add_generation_prompt=True, return_tensors=\"pt\").to(\"cuda\")\n\noutputs = model.generate(**input_ids, max_new_tokens=200)\nprint(tokenizer.decode(outputs[0]))\n```\n\nIf you have flash-attention installed (`pip install flash-attn`), it is possible to generate faster. (The HuggingFace documentation for flash-attention can be found [here](https://huggingface.co/docs/transformers/perf_infer_gpu_one#flashattention-2).)\n```python\nfrom transformers import DbrxForCausalLM, AutoTokenizer\nimport torch\n\ntokenizer = AutoTokenizer.from_pretrained(\"databricks/dbrx-instruct\", token=\"YOUR_HF_TOKEN\")\nmodel = DbrxForCausalLM.from_pretrained(\n    \"databricks/dbrx-instruct\",\n    device_map=\"auto\",\n    torch_dtype=torch.bfloat16,\n    token=\"YOUR_HF_TOKEN\",\n    attn_implementation=\"flash_attention_2\",\n    )\n\ninput_text = \"What does it take to build a great LLM?\"\nmessages = [{\"role\": \"user\", \"content\": input_text}]\ninput_ids = tokenizer.apply_chat_template(messages, return_dict=True, tokenize=True, add_generation_prompt=True, return_tensors=\"pt\").to(\"cuda\")\n\noutputs = model.generate(**input_ids, max_new_tokens=200)\nprint(tokenizer.decode(outputs[0]))\n```\n\nYou can also generate faster using the PyTorch scaled dot product attention. (The HuggingFace documentation for scaled dot product attention can be found [here](https://huggingface.co/docs/transformers/perf_infer_gpu_one#pytorch-scaled-dot-product-attention).)\n```python\nfrom transformers import DbrxForCausalLM, AutoTokenizer\nimport torch\n\ntokenizer = AutoTokenizer.from_pretrained(\"databricks/dbrx-instruct\", token=\"YOUR_HF_TOKEN\")\nmodel = DbrxForCausalLM.from_pretrained(\n    \"databricks/dbrx-instruct\",\n    device_map=\"auto\",\n    torch_dtype=torch.bfloat16,\n    token=\"YOUR_HF_TOKEN\",\n    attn_implementation=\"sdpa\",\n    )\n\ninput_text = \"What does it take to build a great LLM?\""}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: c7bcca2b-f562-4bc2-a1b1-ab16d0c973d1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/dbrx", "title": "DBRX", "tokens": 1388, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c7bcca2b-f562-4bc2-a1b1-ab16d0c973d1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dbrx\", \"title\": \"DBRX\", \"tokens\": 1388, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"237d368a-51d7-40b7-8218-61fa59853413\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dbrx\", \"title\": \"DBRX\", \"tokens\": 1388, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cbeefe8faa3f53bbc16032eaa766d728570874f11046f75511c923cc57c3c9f2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3eb0f280-ddba-419b-b005-8b09dfb4793d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dbrx\", \"title\": \"DBRX\", \"tokens\": 1388, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a519ffefa34499c14e1d1305f0da40d6b1e718ce83f6187dd854c212607c5772\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3068, \"end_char_idx\": 5086, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "237d368a-51d7-40b7-8218-61fa59853413", "doc_id": "237d368a-51d7-40b7-8218-61fa59853413", "ref_doc_id": "237d368a-51d7-40b7-8218-61fa59853413", "chroma:document": "(The HuggingFace documentation for flash-attention can be found [here](https://huggingface.co/docs/transformers/perf_infer_gpu_one#flashattention-2).)\n```python\nfrom transformers import DbrxForCausalLM, AutoTokenizer\nimport torch\n\ntokenizer = AutoTokenizer.from_pretrained(\"databricks/dbrx-instruct\", token=\"YOUR_HF_TOKEN\")\nmodel = DbrxForCausalLM.from_pretrained(\n    \"databricks/dbrx-instruct\",\n    device_map=\"auto\",\n    torch_dtype=torch.bfloat16,\n    token=\"YOUR_HF_TOKEN\",\n    attn_implementation=\"flash_attention_2\",\n    )\n\ninput_text = \"What does it take to build a great LLM?\"\nmessages = [{\"role\": \"user\", \"content\": input_text}]\ninput_ids = tokenizer.apply_chat_template(messages, return_dict=True, tokenize=True, add_generation_prompt=True, return_tensors=\"pt\").to(\"cuda\")\n\noutputs = model.generate(**input_ids, max_new_tokens=200)\nprint(tokenizer.decode(outputs[0]))\n```\n\nYou can also generate faster using the PyTorch scaled dot product attention. (The HuggingFace documentation for scaled dot product attention can be found [here](https://huggingface.co/docs/transformers/perf_infer_gpu_one#pytorch-scaled-dot-product-attention).)\n```python\nfrom transformers import DbrxForCausalLM, AutoTokenizer\nimport torch\n\ntokenizer = AutoTokenizer.from_pretrained(\"databricks/dbrx-instruct\", token=\"YOUR_HF_TOKEN\")\nmodel = DbrxForCausalLM.from_pretrained(\n    \"databricks/dbrx-instruct\",\n    device_map=\"auto\",\n    torch_dtype=torch.bfloat16,\n    token=\"YOUR_HF_TOKEN\",\n    attn_implementation=\"sdpa\",\n    )\n\ninput_text = \"What does it take to build a great LLM?\"\nmessages = [{\"role\": \"user\", \"content\": input_text}]\ninput_ids = tokenizer.apply_chat_template(messages, return_dict=True, tokenize=True, add_generation_prompt=True, return_tensors=\"pt\").to(\"cuda\")\n\noutputs = model.generate(**input_ids, max_new_tokens=200)\nprint(tokenizer.decode(outputs[0]))\n```\n\n## DbrxConfig\n\n[[autodoc]] DbrxConfig\n\n\n## DbrxModel\n\n[[autodoc]] DbrxModel\n    - forward\n\n\n## DbrxForCausalLM\n\n[[autodoc]] DbrxForCausalLM\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 2aa0d97c-cb0b-47b9-bb6f-db699e7d4bdc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/byt5", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2aa0d97c-cb0b-47b9-bb6f-db699e7d4bdc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/byt5\", \"title\": \"ByT5\", \"tokens\": 2066, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8472fab1-94d8-4c02-981c-9a672cc56d3b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/byt5\", \"title\": \"ByT5\", \"tokens\": 2066, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5bec4a969d99f261bb697a9e1acac237a8aea813db6eaee65bef496d484d8428\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"750a8169-a7ee-4dae-aa53-3bba533805e8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8e8eddd5f7ab2d7b9b24912405efe684b6db71a6036a80028ecfdfe42a89158f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2941, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8472fab1-94d8-4c02-981c-9a672cc56d3b", "doc_id": "8472fab1-94d8-4c02-981c-9a672cc56d3b", "ref_doc_id": "8472fab1-94d8-4c02-981c-9a672cc56d3b", "chroma:document": "# ByT5\n\n## Overview\n\nThe ByT5 model was presented in [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://arxiv.org/abs/2105.13626) by Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir\nKale, Adam Roberts, Colin Raffel.\n\nThe abstract from the paper is the following:\n\n*Most widely-used pre-trained language models operate on sequences of tokens corresponding to word or subword units.\nEncoding text as a sequence of tokens requires a tokenizer, which is typically created as an independent artifact from\nthe model. Token-free models that instead operate directly on raw text (bytes or characters) have many benefits: they\ncan process text in any language out of the box, they are more robust to noise, and they minimize technical debt by\nremoving complex and error-prone text preprocessing pipelines. Since byte or character sequences are longer than token\nsequences, past work on token-free models has often introduced new model architectures designed to amortize the cost of\noperating directly on raw text. In this paper, we show that a standard Transformer architecture can be used with\nminimal modifications to process byte sequences. We carefully characterize the trade-offs in terms of parameter count,\ntraining FLOPs, and inference speed, and show that byte-level models are competitive with their token-level\ncounterparts. We also demonstrate that byte-level models are significantly more robust to noise and perform better on\ntasks that are sensitive to spelling and pronunciation. As part of our contribution, we release a new set of\npre-trained byte-level Transformer models based on the T5 architecture, as well as all code and data used in our\nexperiments.*\n\nThis model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The original code can be\nfound [here](https://github.com/google-research/byt5).\n\n<Tip>\n\nByT5's architecture is based on the T5v1.1 model, refer to [T5v1.1's documentation page](t5v1.1) for the API reference. They\nonly differ in how inputs should be prepared for the model, see the code examples below.\n\n</Tip>\n\nSince ByT5 was pre-trained unsupervisedly, there's no real advantage to using a task prefix during single-task\nfine-tuning. If you are doing multi-task fine-tuning, you should use a prefix.\n\n\n## Usage example\n\nByT5 works on raw UTF-8 bytes, so it can be used without a tokenizer:\n\n```python\n>>> from transformers import T5ForConditionalGeneration\n>>> import torch\n\n>>> model = T5ForConditionalGeneration.from_pretrained(\"google/byt5-small\")\n\n>>> num_special_tokens = 3\n>>> # Model has 3 special tokens which take up the input ids 0,1,2 of ByT5.\n>>> # => Need to shift utf-8 character encodings by 3 before passing ids to model.\n\n>>> input_ids = torch.tensor([list(\"Life is like a box of chocolates.\".encode(\"utf-8\"))]) + num_special_tokens\n\n>>> labels = torch.tensor([list(\"La vie est comme une bo\u00eete de chocolat."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 750a8169-a7ee-4dae-aa53-3bba533805e8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/byt5", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"750a8169-a7ee-4dae-aa53-3bba533805e8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/byt5\", \"title\": \"ByT5\", \"tokens\": 2066, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8472fab1-94d8-4c02-981c-9a672cc56d3b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/byt5\", \"title\": \"ByT5\", \"tokens\": 2066, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5bec4a969d99f261bb697a9e1acac237a8aea813db6eaee65bef496d484d8428\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2aa0d97c-cb0b-47b9-bb6f-db699e7d4bdc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/byt5\", \"title\": \"ByT5\", \"tokens\": 2066, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7aa1bdd2b7ad8a226274f3e143a90d7a8458be97857aa9e1089e40d3429b0493\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b7458ae8-c284-450e-b0d5-57cb8253ff81\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4937b9471477d1fdadf8a629cdcdff26318a5c7b76842911025759b3e2d6bc13\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2320, \"end_char_idx\": 4950, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8472fab1-94d8-4c02-981c-9a672cc56d3b", "doc_id": "8472fab1-94d8-4c02-981c-9a672cc56d3b", "ref_doc_id": "8472fab1-94d8-4c02-981c-9a672cc56d3b", "chroma:document": "## Usage example\n\nByT5 works on raw UTF-8 bytes, so it can be used without a tokenizer:\n\n```python\n>>> from transformers import T5ForConditionalGeneration\n>>> import torch\n\n>>> model = T5ForConditionalGeneration.from_pretrained(\"google/byt5-small\")\n\n>>> num_special_tokens = 3\n>>> # Model has 3 special tokens which take up the input ids 0,1,2 of ByT5.\n>>> # => Need to shift utf-8 character encodings by 3 before passing ids to model.\n\n>>> input_ids = torch.tensor([list(\"Life is like a box of chocolates.\".encode(\"utf-8\"))]) + num_special_tokens\n\n>>> labels = torch.tensor([list(\"La vie est comme une bo\u00eete de chocolat.\".encode(\"utf-8\"))]) + num_special_tokens\n\n>>> loss = model(input_ids, labels=labels).loss\n>>> loss.item()\n2.66\n```\n\nFor batched inference and training it is however recommended to make use of the tokenizer:\n\n```python\n>>> from transformers import T5ForConditionalGeneration, AutoTokenizer\n\n>>> model = T5ForConditionalGeneration.from_pretrained(\"google/byt5-small\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"google/byt5-small\")\n\n>>> model_inputs = tokenizer(\n...     [\"Life is like a box of chocolates.\", \"Today is Monday.\"], padding=\"longest\", return_tensors=\"pt\"\n... )\n>>> labels_dict = tokenizer(\n...     [\"La vie est comme une bo\u00eete de chocolat.\", \"Aujourd'hui c'est lundi.\"], padding=\"longest\", return_tensors=\"pt\"\n... )\n>>> labels = labels_dict.input_ids\n\n>>> loss = model(**model_inputs, labels=labels).loss\n>>> loss.item()\n17.9\n```\n\nSimilar to [T5](t5), ByT5 was trained on the span-mask denoising task. However, \nsince the model works directly on characters, the pretraining task is a bit \ndifferent. Let's corrupt some characters of the \ninput sentence `\"The dog chases a ball in the park.\"` and ask ByT5 to predict them \nfor us.\n\n```python\n>>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM\n>>> import torch\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"google/byt5-base\")\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"google/byt5-base\")\n\n>>> input_ids_prompt = \"The dog chases a ball in the park.\"\n>>> input_ids = tokenizer(input_ids_prompt).input_ids\n\n>>> # Note that we cannot add \"{extra_id_...}\" to the string directly\n>>> # as the Byte tokenizer would incorrectly merge the tokens\n>>> # For ByT5, we need to work directly on the character level\n>>> # Contrary to T5, ByT5 does not use sentinel tokens for masking, but instead\n>>> # uses final utf character ids.\n>>> # UTF-8 is represented by 8 bits and ByT5 has 3 special tokens.\n>>> # => There are 2**8+2 = 259 input ids and mask tokens count down from index 258.\n>>> # => mask to \"The dog [258]a ball [257]park.\""}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: b7458ae8-c284-450e-b0d5-57cb8253ff81
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/byt5", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b7458ae8-c284-450e-b0d5-57cb8253ff81\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/byt5\", \"title\": \"ByT5\", \"tokens\": 2066, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8472fab1-94d8-4c02-981c-9a672cc56d3b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/byt5\", \"title\": \"ByT5\", \"tokens\": 2066, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5bec4a969d99f261bb697a9e1acac237a8aea813db6eaee65bef496d484d8428\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"750a8169-a7ee-4dae-aa53-3bba533805e8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/byt5\", \"title\": \"ByT5\", \"tokens\": 2066, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"20100c34e76cea0c27c1ed3eef40ea016b61ed4a6e8aba5fc5225f852104060d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0a5c0962-7232-4448-b98c-e9ce6065b0b0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"063d4e0a601f19634d41e30172f0792ddfc1faa4844db2875f1eb8450258025d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3469, \"end_char_idx\": 5319, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8472fab1-94d8-4c02-981c-9a672cc56d3b", "doc_id": "8472fab1-94d8-4c02-981c-9a672cc56d3b", "ref_doc_id": "8472fab1-94d8-4c02-981c-9a672cc56d3b", "chroma:document": "\"], padding=\"longest\", return_tensors=\"pt\"\n... )\n>>> labels_dict = tokenizer(\n...     [\"La vie est comme une bo\u00eete de chocolat.\", \"Aujourd'hui c'est lundi.\"], padding=\"longest\", return_tensors=\"pt\"\n... )\n>>> labels = labels_dict.input_ids\n\n>>> loss = model(**model_inputs, labels=labels).loss\n>>> loss.item()\n17.9\n```\n\nSimilar to [T5](t5), ByT5 was trained on the span-mask denoising task. However, \nsince the model works directly on characters, the pretraining task is a bit \ndifferent. Let's corrupt some characters of the \ninput sentence `\"The dog chases a ball in the park.\"` and ask ByT5 to predict them \nfor us.\n\n```python\n>>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM\n>>> import torch\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"google/byt5-base\")\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"google/byt5-base\")\n\n>>> input_ids_prompt = \"The dog chases a ball in the park.\"\n>>> input_ids = tokenizer(input_ids_prompt).input_ids\n\n>>> # Note that we cannot add \"{extra_id_...}\" to the string directly\n>>> # as the Byte tokenizer would incorrectly merge the tokens\n>>> # For ByT5, we need to work directly on the character level\n>>> # Contrary to T5, ByT5 does not use sentinel tokens for masking, but instead\n>>> # uses final utf character ids.\n>>> # UTF-8 is represented by 8 bits and ByT5 has 3 special tokens.\n>>> # => There are 2**8+2 = 259 input ids and mask tokens count down from index 258.\n>>> # => mask to \"The dog [258]a ball [257]park.\"\n\n>>> input_ids = torch.tensor([input_ids[:8] + [258] + input_ids[14:21] + [257] + input_ids[28:]])\n>>> input_ids\ntensor([[ 87, 107, 104,  35, 103, 114, 106,  35, 258,  35, 100,  35, 101, 100, 111, 111, 257,  35, 115, 100, 117, 110,  49,   1]])\n\n>>> # ByT5 produces only one char at a time so we need to produce many more output characters here -> set `max_length=100`."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0a5c0962-7232-4448-b98c-e9ce6065b0b0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/byt5", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0a5c0962-7232-4448-b98c-e9ce6065b0b0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/byt5\", \"title\": \"ByT5\", \"tokens\": 2066, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8472fab1-94d8-4c02-981c-9a672cc56d3b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/byt5\", \"title\": \"ByT5\", \"tokens\": 2066, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5bec4a969d99f261bb697a9e1acac237a8aea813db6eaee65bef496d484d8428\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b7458ae8-c284-450e-b0d5-57cb8253ff81\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/byt5\", \"title\": \"ByT5\", \"tokens\": 2066, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"89dac8d6a41dacc28ed77b4207dbef97a5b59faf5ed57a834743aca95780c040\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"73054ac0-4c0d-43dd-90f9-152f45b472c2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"16f949deb325dceb8c3100c6f57c1d74538b8b4a93110c00fd716dee66d4c944\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4046, \"end_char_idx\": 6522, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8472fab1-94d8-4c02-981c-9a672cc56d3b", "doc_id": "8472fab1-94d8-4c02-981c-9a672cc56d3b", "ref_doc_id": "8472fab1-94d8-4c02-981c-9a672cc56d3b", "chroma:document": "\"` and ask ByT5 to predict them \nfor us.\n\n```python\n>>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM\n>>> import torch\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"google/byt5-base\")\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"google/byt5-base\")\n\n>>> input_ids_prompt = \"The dog chases a ball in the park.\"\n>>> input_ids = tokenizer(input_ids_prompt).input_ids\n\n>>> # Note that we cannot add \"{extra_id_...}\" to the string directly\n>>> # as the Byte tokenizer would incorrectly merge the tokens\n>>> # For ByT5, we need to work directly on the character level\n>>> # Contrary to T5, ByT5 does not use sentinel tokens for masking, but instead\n>>> # uses final utf character ids.\n>>> # UTF-8 is represented by 8 bits and ByT5 has 3 special tokens.\n>>> # => There are 2**8+2 = 259 input ids and mask tokens count down from index 258.\n>>> # => mask to \"The dog [258]a ball [257]park.\"\n\n>>> input_ids = torch.tensor([input_ids[:8] + [258] + input_ids[14:21] + [257] + input_ids[28:]])\n>>> input_ids\ntensor([[ 87, 107, 104,  35, 103, 114, 106,  35, 258,  35, 100,  35, 101, 100, 111, 111, 257,  35, 115, 100, 117, 110,  49,   1]])\n\n>>> # ByT5 produces only one char at a time so we need to produce many more output characters here -> set `max_length=100`.\n>>> output_ids = model.generate(input_ids, max_length=100)[0].tolist()\n>>> output_ids\n[0, 258, 108, 118,  35, 119, 107, 104,  35, 114, 113, 104,  35, 122, 107, 114,  35, 103, 114, 104, 118, 257,  35, 108, 113,  35, 119, 107, 104,  35, 103, 108, 118, 102, 114, 256, 108, 113,  35, 119, 107, 104, 35, 115, 100, 117, 110,  49,  35,  87, 107, 104,  35, 103, 114, 106, 35, 108, 118,  35, 119, 107, 104,  35, 114, 113, 104,  35, 122, 107, 114,  35, 103, 114, 104, 118,  35, 100,  35, 101, 100, 111, 111,  35, 108, 113, 255,  35, 108, 113,  35, 119, 107, 104,  35, 115, 100, 117, 110,  49]\n\n>>> # ^- Note how 258 descends to 257, 256, 255\n\n>>> # Now we need to split on the sentinel tokens, let's write a short loop for this\n>>> output_ids_list = []\n>>> start_token = 0\n>>> sentinel_token = 258\n>>> while sentinel_token in output_ids:\n...     split_idx = output_ids.index(sentinel_token)\n...     output_ids_list.append(output_ids[start_token:split_idx])\n...     start_token = split_idx\n...     sentinel_token -= 1\n\n>>> output_ids_list.append(output_ids[start_token:])\n>>> output_string = tokenizer.batch_decode(output_ids_list)\n>>> output_string\n['<pad>', 'is the one who does', ' in the disco', 'in the park."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 73054ac0-4c0d-43dd-90f9-152f45b472c2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/byt5", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"73054ac0-4c0d-43dd-90f9-152f45b472c2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/byt5\", \"title\": \"ByT5\", \"tokens\": 2066, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8472fab1-94d8-4c02-981c-9a672cc56d3b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/byt5\", \"title\": \"ByT5\", \"tokens\": 2066, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5bec4a969d99f261bb697a9e1acac237a8aea813db6eaee65bef496d484d8428\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0a5c0962-7232-4448-b98c-e9ce6065b0b0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/byt5\", \"title\": \"ByT5\", \"tokens\": 2066, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"764f4e4d4902264f1622523d98e7dbec19c3e6886060e4dfc68b7dc467ee6095\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6523, \"end_char_idx\": 6669, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8472fab1-94d8-4c02-981c-9a672cc56d3b", "doc_id": "8472fab1-94d8-4c02-981c-9a672cc56d3b", "ref_doc_id": "8472fab1-94d8-4c02-981c-9a672cc56d3b", "chroma:document": "The dog is the one who does a ball in', ' in the park.']\n```\n\n\n## ByT5Tokenizer\n\n[[autodoc]] ByT5Tokenizer\n\nSee [`ByT5Tokenizer`] for all details."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6fdf25bc-a77e-47f7-9b06-b86bf613626b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/qwen2_vl", "title": "Qwen2_VL", "tokens": 3135, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6fdf25bc-a77e-47f7-9b06-b86bf613626b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qwen2_vl\", \"title\": \"Qwen2_VL\", \"tokens\": 3135, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"93cebb7f-4797-48ef-92a3-21f9fb96ebd7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qwen2_vl\", \"title\": \"Qwen2_VL\", \"tokens\": 3135, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2fc8e4b0b8ac9b8d0ca57e6642ffb81888e062f0d3c11a45f4c62dcb738337b3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5d6f1937-bfa4-470e-ae37-a6c4d2c27d32\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0b4d60cc3a1f981cf028354a64d30a7f2dcefa6502239ed6ef88f9c396d2b289\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2863, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "93cebb7f-4797-48ef-92a3-21f9fb96ebd7", "doc_id": "93cebb7f-4797-48ef-92a3-21f9fb96ebd7", "ref_doc_id": "93cebb7f-4797-48ef-92a3-21f9fb96ebd7", "chroma:document": "# Qwen2_VL\n\n\n## Overview\n\nThe [Qwen2_VL](https://qwenlm.github.io/blog/qwen2-vl/) is a major update to our [Qwen-VL](https://arxiv.org/pdf/2308.12966) model from the Qwen team. \n\nThe abstract from the blog is the following:\n\n*This blog introduces Qwen2-VL, an advanced version of the Qwen-VL model that has undergone significant enhancements over the past year. Key improvements include enhanced image comprehension, advanced video understanding, integrated visual agent functionality, and expanded multilingual support. The model architecture has been optimized for handling arbitrary image resolutions through Naive Dynamic Resolution support and utilizes Multimodal Rotary Position Embedding (M-ROPE) to effectively process both 1D textual and multi-dimensional visual data. This updated model demonstrates competitive performance against leading AI systems like GPT-4o and Claude 3.5 Sonnet in vision-related tasks and ranks highly among open-source models in text capabilities. These advancements make Qwen2-VL a versatile tool for various applications requiring robust multimodal processing and reasoning abilities.*\n\n\n## Usage example\n\n### Single Media inference\n\nThe model can accept both images and videos as input. Here's an example code for inference.\n\n```python\n\nfrom PIL import Image\nimport requests\nimport torch\nfrom torchvision import io\nfrom typing import Dict\nfrom transformers import Qwen2VLForConditionalGeneration, AutoTokenizer, AutoProcessor\n\n# Load the model in half-precision on the available device(s)\nmodel = Qwen2VLForConditionalGeneration.from_pretrained(\"Qwen/Qwen2-VL-7B-Instruct\", device_map=\"auto\")\nprocessor = AutoProcessor.from_pretrained(\"Qwen/Qwen2-VL-7B-Instruct\")\n\n# Image\nurl = \"https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg\"\nimage = Image.open(requests.get(url, stream=True).raw)\n\nconversation = [\n    {\n        \"role\":\"user\",\n        \"content\":[\n            {\n                \"type\":\"image\",\n            },\n            {\n                \"type\":\"text\",\n                \"text\":\"Describe this image.\"\n            }\n        ]\n    }\n]\n\n\n# Preprocess the inputs\ntext_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)\n# Excepted output: '<|im_start|>system\\nYou are a helpful assistant.<|im_end|>\\n<|im_start|>user\\n<|vision_start|><|image_pad|><|vision_end|>Describe this image.<|im_end|>\\n<|im_start|>assistant\\n'\n\ninputs = processor(text=[text_prompt], images=[image], padding=True, return_tensors=\"pt\")\ninputs = inputs.to('cuda')\n\n# Inference: Generation of the output\noutput_ids = model.generate(**inputs, max_new_tokens=128)\ngenerated_ids = [output_ids[len(input_ids):] for input_ids, output_ids in zip(inputs.input_ids, output_ids)]\noutput_text = processor.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)\nprint(output_text)"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 5d6f1937-bfa4-470e-ae37-a6c4d2c27d32
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/qwen2_vl", "title": "Qwen2_VL", "tokens": 3135, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5d6f1937-bfa4-470e-ae37-a6c4d2c27d32\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qwen2_vl\", \"title\": \"Qwen2_VL\", \"tokens\": 3135, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"93cebb7f-4797-48ef-92a3-21f9fb96ebd7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qwen2_vl\", \"title\": \"Qwen2_VL\", \"tokens\": 3135, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2fc8e4b0b8ac9b8d0ca57e6642ffb81888e062f0d3c11a45f4c62dcb738337b3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6fdf25bc-a77e-47f7-9b06-b86bf613626b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qwen2_vl\", \"title\": \"Qwen2_VL\", \"tokens\": 3135, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1e1d41d4a3cff7f045dfb9bc470dfadffcab29f12312e33de07bf216a98868bf\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e70d5322-d9ed-4e3a-b245-8c1c82decfc6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"923b082cb31373e304c72081acff7cf8d55adcf364bf49cd84b977648dd607d0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2100, \"end_char_idx\": 4917, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "93cebb7f-4797-48ef-92a3-21f9fb96ebd7", "doc_id": "93cebb7f-4797-48ef-92a3-21f9fb96ebd7", "ref_doc_id": "93cebb7f-4797-48ef-92a3-21f9fb96ebd7", "chroma:document": "# Preprocess the inputs\ntext_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)\n# Excepted output: '<|im_start|>system\\nYou are a helpful assistant.<|im_end|>\\n<|im_start|>user\\n<|vision_start|><|image_pad|><|vision_end|>Describe this image.<|im_end|>\\n<|im_start|>assistant\\n'\n\ninputs = processor(text=[text_prompt], images=[image], padding=True, return_tensors=\"pt\")\ninputs = inputs.to('cuda')\n\n# Inference: Generation of the output\noutput_ids = model.generate(**inputs, max_new_tokens=128)\ngenerated_ids = [output_ids[len(input_ids):] for input_ids, output_ids in zip(inputs.input_ids, output_ids)]\noutput_text = processor.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)\nprint(output_text)\n\n\n\n# Video\ndef fetch_video(ele: Dict, nframe_factor=2):\n    if isinstance(ele['video'], str):\n        def round_by_factor(number: int, factor: int) -> int:\n            return round(number / factor) * factor\n\n        video = ele[\"video\"]\n        if video.startswith(\"file://\"):\n            video = video[7:]\n\n        video, _, info = io.read_video(\n            video,\n            start_pts=ele.get(\"video_start\", 0.0),\n            end_pts=ele.get(\"video_end\", None),\n            pts_unit=\"sec\",\n            output_format=\"TCHW\",\n        )\n        assert not (\"fps\" in ele and \"nframes\" in ele), \"Only accept either `fps` or `nframes`\"\n        if \"nframes\" in ele:\n            nframes = round_by_factor(ele[\"nframes\"], nframe_factor)\n        else:\n            fps = ele.get(\"fps\", 1.0)\n            nframes = round_by_factor(video.size(0) / info[\"video_fps\"] * fps, nframe_factor)\n        idx = torch.linspace(0, video.size(0) - 1, nframes, dtype=torch.int64)\n        return video[idx]\n\nvideo_info = {\"type\": \"video\", \"video\": \"/path/to/video.mp4\", \"fps\": 1.0}\nvideo = fetch_video(video_info)\nconversation = [\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"video\"},\n            {\"type\": \"text\", \"text\": \"What happened in the video?\"},\n        ],\n    }\n]\n\n# Preprocess the inputs\ntext_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)\n# Excepted output: '<|im_start|>system\\nYou are a helpful assistant.<|im_end|>\\n<|im_start|>user\\n<|vision_start|><|video_pad|><|vision_end|>What happened in the video?<|im_end|>\\n<|im_start|>assistant\\n'\n\ninputs = processor(text=[text_prompt], videos=[video], padding=True, return_tensors=\"pt\")\ninputs = inputs.to('cuda')\n\n# Inference: Generation of the output\noutput_ids = model.generate(**inputs, max_new_tokens=128)\ngenerated_ids = [output_ids[len(input_ids):] for input_ids, output_ids in zip(inputs.input_ids, output_ids)]\noutput_text = processor.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)\nprint(output_text)\n\n```"}


