# embeddings_queue

created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 895de9e3-3061-4e56-befa-88a4d9aa5cb6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas/", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "HF_Transformers", "_node_content": "{\"id_\": \"895de9e3-3061-4e56-befa-88a4d9aa5cb6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6f8b7462-3307-4ed8-a23a-6d7f8d36fa39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"02f9d1e611bfdcdddc25ffc55f689dfee098357342da90a329bd3b562cf383cc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8a9bde77-589b-44ba-b186-3fb6e620210a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"d7d25c9d124a177a5909fd6009fd5d7ea2f1146ecffa5e304440016c96ae83f1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e546318c-3edd-42f2-ad96-9f1cf6b7dc78\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9e7e7fff44098dce7ab89839730c10a6d7350eb0dbf2808b4123757f6337765e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3458, \"end_char_idx\": 6633, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "ref_doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "chroma:document": "For more details, see their follow-up paper: [Understanding tables with intermediate pre-training](https://www.aclweb.org/anthology/2020.findings-emnlp.27/) by Julian Martin Eisenschlos, Syrine Krichene and Thomas M\u00fcller.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/tapas_architecture.png\"\nalt=\"drawing\" width=\"600\"/> \n\n<small> TAPAS architecture. Taken from the <a href=\"https://ai.googleblog.com/2020/04/using-neural-networks-to-find-answers.html\">original blog post</a>.</small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The Tensorflow version of this model was contributed by [kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/google-research/tapas).\n\n## Usage tips\n\n- TAPAS is a model that uses relative position embeddings by default (restarting the position embeddings at every cell of the table). Note that this is something that was added after the publication of the original TAPAS paper. According to the authors, this usually results in a slightly better performance, and allows you to encode longer sequences without running out of embeddings. This is reflected in the `reset_position_index_per_cell` parameter of [`TapasConfig`], which is set to `True` by default. The default versions of the models available on the [hub](https://huggingface.co/models?search=tapas) all use relative position embeddings. You can still use the ones with absolute position embeddings by passing in an additional argument `revision=\"no_reset\"` when calling the `from_pretrained()` method. Note that it's usually advised to pad the inputs on the right rather than the left.\n- TAPAS is based on BERT, so `TAPAS-base` for example corresponds to a `BERT-base` architecture. Of course, `TAPAS-large` will result in the best performance (the results reported in the paper are from `TAPAS-large`). Results of the various sized models are shown on the [original GitHub repository](https://github.com/google-research/tapas).\n- TAPAS has checkpoints fine-tuned on SQA, which are capable of answering questions related to a table in a conversational set-up. This means that you can ask follow-up questions such as \"what is his age?\" related to the previous question. Note that the forward pass of TAPAS is a bit different in case of a conversational set-up: in that case, you have to feed every table-question pair one by one to the model, such that the `prev_labels` token type ids can be overwritten by the predicted `labels` of the model to the previous question. See \"Usage\" section for more info.\n- TAPAS is similar to BERT and therefore relies on the masked language modeling (MLM) objective. It is therefore efficient at predicting masked tokens and at NLU in general, but is not optimal for text generation. Models trained with a causal language modeling (CLM) objective are better in that regard. Note that TAPAS can be used as an encoder in the EncoderDecoderModel framework, to combine it with an autoregressive text decoder such as GPT-2.\n\n## Usage: fine-tuning\n\nHere we explain how you can fine-tune [`TapasForQuestionAnswering`] on your own dataset."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: e546318c-3edd-42f2-ad96-9f1cf6b7dc78
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas/", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "HF_Transformers", "_node_content": "{\"id_\": \"e546318c-3edd-42f2-ad96-9f1cf6b7dc78\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6f8b7462-3307-4ed8-a23a-6d7f8d36fa39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"02f9d1e611bfdcdddc25ffc55f689dfee098357342da90a329bd3b562cf383cc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"895de9e3-3061-4e56-befa-88a4d9aa5cb6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"cdf471b0b24641918796a520c629e5ada0be9113df1f4d36034bdf2e84b6ace7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2b597932-77c6-46db-bb95-bcd99b0f91ac\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5853700e59e5aaa95a01af1e6d806a23720f14a9e488a4c321261854b4657eb3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5070, \"end_char_idx\": 8015, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "ref_doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "chroma:document": "Note that it's usually advised to pad the inputs on the right rather than the left.\n- TAPAS is based on BERT, so `TAPAS-base` for example corresponds to a `BERT-base` architecture. Of course, `TAPAS-large` will result in the best performance (the results reported in the paper are from `TAPAS-large`). Results of the various sized models are shown on the [original GitHub repository](https://github.com/google-research/tapas).\n- TAPAS has checkpoints fine-tuned on SQA, which are capable of answering questions related to a table in a conversational set-up. This means that you can ask follow-up questions such as \"what is his age?\" related to the previous question. Note that the forward pass of TAPAS is a bit different in case of a conversational set-up: in that case, you have to feed every table-question pair one by one to the model, such that the `prev_labels` token type ids can be overwritten by the predicted `labels` of the model to the previous question. See \"Usage\" section for more info.\n- TAPAS is similar to BERT and therefore relies on the masked language modeling (MLM) objective. It is therefore efficient at predicting masked tokens and at NLU in general, but is not optimal for text generation. Models trained with a causal language modeling (CLM) objective are better in that regard. Note that TAPAS can be used as an encoder in the EncoderDecoderModel framework, to combine it with an autoregressive text decoder such as GPT-2.\n\n## Usage: fine-tuning\n\nHere we explain how you can fine-tune [`TapasForQuestionAnswering`] on your own dataset.\n\n**STEP 1: Choose one of the 3 ways in which you can use TAPAS - or experiment**\n\nBasically, there are 3 different ways in which one can fine-tune [`TapasForQuestionAnswering`], corresponding to the different datasets on which Tapas was fine-tuned:\n\n1. SQA: if you're interested in asking follow-up questions related to a table, in a conversational set-up. For example if you first ask \"what's the name of the first actor?\" then you can ask a follow-up question such as \"how old is he?\". Here, questions do not involve any aggregation (all questions are cell selection questions).\n2. WTQ: if you're not interested in asking questions in a conversational set-up, but rather just asking questions related to a table, which might involve aggregation, such as counting a number of rows, summing up cell values or averaging cell values. You can then for example ask \"what's the total number of goals Cristiano Ronaldo made in his career?\". This case is also called **weak supervision**, since the model itself must learn the appropriate aggregation operator (SUM/COUNT/AVERAGE/NONE) given only the answer to the question as supervision.\n3. WikiSQL-supervised: this dataset is based on WikiSQL with the model being given the ground truth aggregation operator during training. This is also called **strong supervision**. Here, learning the appropriate aggregation operator is much easier."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 2b597932-77c6-46db-bb95-bcd99b0f91ac
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas/", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "HF_Transformers", "_node_content": "{\"id_\": \"2b597932-77c6-46db-bb95-bcd99b0f91ac\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6f8b7462-3307-4ed8-a23a-6d7f8d36fa39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"02f9d1e611bfdcdddc25ffc55f689dfee098357342da90a329bd3b562cf383cc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e546318c-3edd-42f2-ad96-9f1cf6b7dc78\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"5d96c07218ec5e48a6c85750cc0771b6f39d6c1b34118421f42a147e8188a56e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"71072054-d74f-4d27-9ad5-167722b0e429\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"07c2ee055ce6f362a7410769e0cd6fd2cd1852b46c1f2867f2f2e1141acb5fcf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6286, \"end_char_idx\": 10123, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "ref_doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "chroma:document": "Models trained with a causal language modeling (CLM) objective are better in that regard. Note that TAPAS can be used as an encoder in the EncoderDecoderModel framework, to combine it with an autoregressive text decoder such as GPT-2.\n\n## Usage: fine-tuning\n\nHere we explain how you can fine-tune [`TapasForQuestionAnswering`] on your own dataset.\n\n**STEP 1: Choose one of the 3 ways in which you can use TAPAS - or experiment**\n\nBasically, there are 3 different ways in which one can fine-tune [`TapasForQuestionAnswering`], corresponding to the different datasets on which Tapas was fine-tuned:\n\n1. SQA: if you're interested in asking follow-up questions related to a table, in a conversational set-up. For example if you first ask \"what's the name of the first actor?\" then you can ask a follow-up question such as \"how old is he?\". Here, questions do not involve any aggregation (all questions are cell selection questions).\n2. WTQ: if you're not interested in asking questions in a conversational set-up, but rather just asking questions related to a table, which might involve aggregation, such as counting a number of rows, summing up cell values or averaging cell values. You can then for example ask \"what's the total number of goals Cristiano Ronaldo made in his career?\". This case is also called **weak supervision**, since the model itself must learn the appropriate aggregation operator (SUM/COUNT/AVERAGE/NONE) given only the answer to the question as supervision.\n3. WikiSQL-supervised: this dataset is based on WikiSQL with the model being given the ground truth aggregation operator during training. This is also called **strong supervision**. Here, learning the appropriate aggregation operator is much easier.\n\nTo summarize:\n\n| **Task**                            | **Example dataset** | **Description**                                                                                         |\n|-------------------------------------|---------------------|---------------------------------------------------------------------------------------------------------|\n| Conversational                      | SQA                 | Conversational, only cell selection questions                                                           |\n| Weak supervision for aggregation    | WTQ                 | Questions might involve aggregation, and the model must learn this given only the answer as supervision |\n| Strong supervision for aggregation  | WikiSQL-supervised  | Questions might involve aggregation, and the model must learn this given the gold aggregation operator  |\n\n<frameworkcontent>\n<pt>\nInitializing a model with a pre-trained base and randomly initialized classification heads from the hub can be done as shown below.\n\n```py\n>>> from transformers import TapasConfig, TapasForQuestionAnswering\n\n>>> # for example, the base sized model with default SQA configuration\n>>> model = TapasForQuestionAnswering.from_pretrained(\"google/tapas-base\")\n\n>>> # or, the base sized model with WTQ configuration\n>>> config = TapasConfig.from_pretrained(\"google/tapas-base-finetuned-wtq\")\n>>> model = TapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n\n>>> # or, the base sized model with WikiSQL configuration\n>>> config = TapasConfig(\"google-base-finetuned-wikisql-supervised\")\n>>> model = TapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n```\n\nOf course, you don't necessarily have to follow one of these three ways in which TAPAS was fine-tuned. You can also experiment by defining any hyperparameters you want when initializing [`TapasConfig`], and then create a [`TapasForQuestionAnswering`] based on that configuration. For example, if you have a dataset that has both conversational questions and questions that might involve aggregation, then you can do it this way."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 71072054-d74f-4d27-9ad5-167722b0e429
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas/", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "HF_Transformers", "_node_content": "{\"id_\": \"71072054-d74f-4d27-9ad5-167722b0e429\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6f8b7462-3307-4ed8-a23a-6d7f8d36fa39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"02f9d1e611bfdcdddc25ffc55f689dfee098357342da90a329bd3b562cf383cc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2b597932-77c6-46db-bb95-bcd99b0f91ac\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"b85cca7e0fe54fd124d507e7ef4e075f20ba452d1575b93743ec5cf0301bc4be\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"71935350-5c10-41cb-ac12-5709c9b71c17\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6bae031756ce043224ba32faee981850fd67269f509027fa69a07b2471954c4b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8017, \"end_char_idx\": 11784, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "ref_doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "chroma:document": "To summarize:\n\n| **Task**                            | **Example dataset** | **Description**                                                                                         |\n|-------------------------------------|---------------------|---------------------------------------------------------------------------------------------------------|\n| Conversational                      | SQA                 | Conversational, only cell selection questions                                                           |\n| Weak supervision for aggregation    | WTQ                 | Questions might involve aggregation, and the model must learn this given only the answer as supervision |\n| Strong supervision for aggregation  | WikiSQL-supervised  | Questions might involve aggregation, and the model must learn this given the gold aggregation operator  |\n\n<frameworkcontent>\n<pt>\nInitializing a model with a pre-trained base and randomly initialized classification heads from the hub can be done as shown below.\n\n```py\n>>> from transformers import TapasConfig, TapasForQuestionAnswering\n\n>>> # for example, the base sized model with default SQA configuration\n>>> model = TapasForQuestionAnswering.from_pretrained(\"google/tapas-base\")\n\n>>> # or, the base sized model with WTQ configuration\n>>> config = TapasConfig.from_pretrained(\"google/tapas-base-finetuned-wtq\")\n>>> model = TapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n\n>>> # or, the base sized model with WikiSQL configuration\n>>> config = TapasConfig(\"google-base-finetuned-wikisql-supervised\")\n>>> model = TapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n```\n\nOf course, you don't necessarily have to follow one of these three ways in which TAPAS was fine-tuned. You can also experiment by defining any hyperparameters you want when initializing [`TapasConfig`], and then create a [`TapasForQuestionAnswering`] based on that configuration. For example, if you have a dataset that has both conversational questions and questions that might involve aggregation, then you can do it this way. Here's an example:\n\n```py\n>>> from transformers import TapasConfig, TapasForQuestionAnswering\n\n>>> # you can initialize the classification heads any way you want (see docs of TapasConfig)\n>>> config = TapasConfig(num_aggregation_labels=3, average_logits_per_cell=True)\n>>> # initializing the pre-trained base sized model with our custom classification heads\n>>> model = TapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n```\n</pt>\n<tf>\nInitializing a model with a pre-trained base and randomly initialized classification heads from the hub can be done as shown below. Be sure to have installed the [tensorflow_probability](https://github.com/tensorflow/probability) dependency:\n\n```py\n>>> from transformers import TapasConfig, TFTapasForQuestionAnswering\n\n>>> # for example, the base sized model with default SQA configuration\n>>> model = TFTapasForQuestionAnswering.from_pretrained(\"google/tapas-base\")\n\n>>> # or, the base sized model with WTQ configuration\n>>> config = TapasConfig.from_pretrained(\"google/tapas-base-finetuned-wtq\")\n>>> model = TFTapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n\n>>> # or, the base sized model with WikiSQL configuration\n>>> config = TapasConfig(\"google-base-finetuned-wikisql-supervised\")\n>>> model = TFTapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n```\n\nOf course, you don't necessarily have to follow one of these three ways in which TAPAS was fine-tuned. You can also experiment by defining any hyperparameters you want when initializing [`TapasConfig`], and then create a [`TFTapasForQuestionAnswering`] based on that configuration."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 71935350-5c10-41cb-ac12-5709c9b71c17
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas/", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "HF_Transformers", "_node_content": "{\"id_\": \"71935350-5c10-41cb-ac12-5709c9b71c17\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6f8b7462-3307-4ed8-a23a-6d7f8d36fa39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"02f9d1e611bfdcdddc25ffc55f689dfee098357342da90a329bd3b562cf383cc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"71072054-d74f-4d27-9ad5-167722b0e429\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"6eab3e2a3845b1f28f4cf7b189226732a37cdd47d62f96aeb3eb552cb819f97e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f1ba6367-31bf-496b-95fe-4533c81c6f86\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"729ddea90a9e793940fcc8f071cd5c8cec97e6b6c3d3641c1020104c276daeeb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10124, \"end_char_idx\": 13467, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "ref_doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "chroma:document": "Here's an example:\n\n```py\n>>> from transformers import TapasConfig, TapasForQuestionAnswering\n\n>>> # you can initialize the classification heads any way you want (see docs of TapasConfig)\n>>> config = TapasConfig(num_aggregation_labels=3, average_logits_per_cell=True)\n>>> # initializing the pre-trained base sized model with our custom classification heads\n>>> model = TapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n```\n</pt>\n<tf>\nInitializing a model with a pre-trained base and randomly initialized classification heads from the hub can be done as shown below. Be sure to have installed the [tensorflow_probability](https://github.com/tensorflow/probability) dependency:\n\n```py\n>>> from transformers import TapasConfig, TFTapasForQuestionAnswering\n\n>>> # for example, the base sized model with default SQA configuration\n>>> model = TFTapasForQuestionAnswering.from_pretrained(\"google/tapas-base\")\n\n>>> # or, the base sized model with WTQ configuration\n>>> config = TapasConfig.from_pretrained(\"google/tapas-base-finetuned-wtq\")\n>>> model = TFTapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n\n>>> # or, the base sized model with WikiSQL configuration\n>>> config = TapasConfig(\"google-base-finetuned-wikisql-supervised\")\n>>> model = TFTapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n```\n\nOf course, you don't necessarily have to follow one of these three ways in which TAPAS was fine-tuned. You can also experiment by defining any hyperparameters you want when initializing [`TapasConfig`], and then create a [`TFTapasForQuestionAnswering`] based on that configuration. For example, if you have a dataset that has both conversational questions and questions that might involve aggregation, then you can do it this way. Here's an example:\n\n```py\n>>> from transformers import TapasConfig, TFTapasForQuestionAnswering\n\n>>> # you can initialize the classification heads any way you want (see docs of TapasConfig)\n>>> config = TapasConfig(num_aggregation_labels=3, average_logits_per_cell=True)\n>>> # initializing the pre-trained base sized model with our custom classification heads\n>>> model = TFTapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n```\n</tf>\n</frameworkcontent>\n\nWhat you can also do is start from an already fine-tuned checkpoint. A note here is that the already fine-tuned checkpoint on WTQ has some issues due to the L2-loss which is somewhat brittle. See [here](https://github.com/google-research/tapas/issues/91#issuecomment-735719340) for more info.\n\nFor a list of all pre-trained and fine-tuned TAPAS checkpoints available on HuggingFace's  hub, see [here](https://huggingface.co/models?search=tapas).\n\n**STEP 2: Prepare your data in the SQA format**\n\nSecond, no matter what you picked above, you should prepare your dataset in the [SQA](https://www.microsoft.com/en-us/download/details.aspx?id=54253) format. This format is a TSV/CSV file with the following columns:\n\n- `id`: optional, id of the table-question pair, for bookkeeping purposes.\n- `annotator`: optional, id of the person who annotated the table-question pair, for bookkeeping purposes.\n- `position`: integer indicating if the question is the first, second, third,... related to the table. Only required in case of conversational setup (SQA)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: f1ba6367-31bf-496b-95fe-4533c81c6f86
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas/", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "HF_Transformers", "_node_content": "{\"id_\": \"f1ba6367-31bf-496b-95fe-4533c81c6f86\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6f8b7462-3307-4ed8-a23a-6d7f8d36fa39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"02f9d1e611bfdcdddc25ffc55f689dfee098357342da90a329bd3b562cf383cc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"71935350-5c10-41cb-ac12-5709c9b71c17\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"a9ebc3415c14bda6aec954556141784173e6555f1cab7a479f790f9320d6e3fa\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b6d78865-c41f-4f6c-9dbe-88d442a7b93c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"05338a496a08ff547ae8a53b5860be98e6963b991e8c251e144094cd293cd2fd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11785, \"end_char_idx\": 14917, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "ref_doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "chroma:document": "For example, if you have a dataset that has both conversational questions and questions that might involve aggregation, then you can do it this way. Here's an example:\n\n```py\n>>> from transformers import TapasConfig, TFTapasForQuestionAnswering\n\n>>> # you can initialize the classification heads any way you want (see docs of TapasConfig)\n>>> config = TapasConfig(num_aggregation_labels=3, average_logits_per_cell=True)\n>>> # initializing the pre-trained base sized model with our custom classification heads\n>>> model = TFTapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n```\n</tf>\n</frameworkcontent>\n\nWhat you can also do is start from an already fine-tuned checkpoint. A note here is that the already fine-tuned checkpoint on WTQ has some issues due to the L2-loss which is somewhat brittle. See [here](https://github.com/google-research/tapas/issues/91#issuecomment-735719340) for more info.\n\nFor a list of all pre-trained and fine-tuned TAPAS checkpoints available on HuggingFace's  hub, see [here](https://huggingface.co/models?search=tapas).\n\n**STEP 2: Prepare your data in the SQA format**\n\nSecond, no matter what you picked above, you should prepare your dataset in the [SQA](https://www.microsoft.com/en-us/download/details.aspx?id=54253) format. This format is a TSV/CSV file with the following columns:\n\n- `id`: optional, id of the table-question pair, for bookkeeping purposes.\n- `annotator`: optional, id of the person who annotated the table-question pair, for bookkeeping purposes.\n- `position`: integer indicating if the question is the first, second, third,... related to the table. Only required in case of conversational setup (SQA). You don't need this column in case you're going for WTQ/WikiSQL-supervised.\n- `question`: string\n- `table_file`: string, name of a csv file containing the tabular data\n- `answer_coordinates`: list of one or more tuples (each tuple being a cell coordinate, i.e. row, column pair that is part of the answer)\n- `answer_text`: list of one or more strings (each string being a cell value that is part of the answer)\n- `aggregation_label`: index of the aggregation operator. Only required in case of strong supervision for aggregation (the WikiSQL-supervised case)\n- `float_answer`: the float answer to the question, if there is one (np.nan if there isn't). Only required in case of weak supervision for aggregation (such as WTQ and WikiSQL)\n\nThe tables themselves should be present in a folder, each table being a separate csv file. Note that the authors of the TAPAS algorithm used conversion scripts with some automated logic to convert the other datasets (WTQ, WikiSQL) into the SQA format. The author explains this [here](https://github.com/google-research/tapas/issues/50#issuecomment-705465960). A conversion of this script that works with HuggingFace's implementation can be found [here](https://github.com/NielsRogge/tapas_utils). Interestingly, these conversion scripts are not perfect (the `answer_coordinates` and `float_answer` fields are populated based on the `answer_text`), meaning that WTQ and WikiSQL results could actually be improved."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: b6d78865-c41f-4f6c-9dbe-88d442a7b93c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas/", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "HF_Transformers", "_node_content": "{\"id_\": \"b6d78865-c41f-4f6c-9dbe-88d442a7b93c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6f8b7462-3307-4ed8-a23a-6d7f8d36fa39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"02f9d1e611bfdcdddc25ffc55f689dfee098357342da90a329bd3b562cf383cc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f1ba6367-31bf-496b-95fe-4533c81c6f86\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"6ab5da49fc5618c7af42b8a6cecdb77404ebe16710cf3b16ad142c6a1c0d7bda\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"367ba07f-0557-42af-982b-b9621292215d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2ad94ff4bcb398e41591ae5d15388101156198403251862b34b3b4bf1f7e1630\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13205, \"end_char_idx\": 16801, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "ref_doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "chroma:document": "- `annotator`: optional, id of the person who annotated the table-question pair, for bookkeeping purposes.\n- `position`: integer indicating if the question is the first, second, third,... related to the table. Only required in case of conversational setup (SQA). You don't need this column in case you're going for WTQ/WikiSQL-supervised.\n- `question`: string\n- `table_file`: string, name of a csv file containing the tabular data\n- `answer_coordinates`: list of one or more tuples (each tuple being a cell coordinate, i.e. row, column pair that is part of the answer)\n- `answer_text`: list of one or more strings (each string being a cell value that is part of the answer)\n- `aggregation_label`: index of the aggregation operator. Only required in case of strong supervision for aggregation (the WikiSQL-supervised case)\n- `float_answer`: the float answer to the question, if there is one (np.nan if there isn't). Only required in case of weak supervision for aggregation (such as WTQ and WikiSQL)\n\nThe tables themselves should be present in a folder, each table being a separate csv file. Note that the authors of the TAPAS algorithm used conversion scripts with some automated logic to convert the other datasets (WTQ, WikiSQL) into the SQA format. The author explains this [here](https://github.com/google-research/tapas/issues/50#issuecomment-705465960). A conversion of this script that works with HuggingFace's implementation can be found [here](https://github.com/NielsRogge/tapas_utils). Interestingly, these conversion scripts are not perfect (the `answer_coordinates` and `float_answer` fields are populated based on the `answer_text`), meaning that WTQ and WikiSQL results could actually be improved.\n\n**STEP 3: Convert your data into tensors using TapasTokenizer**\n\n<frameworkcontent>\n<pt>\nThird, given that you've prepared your data in this TSV/CSV format (and corresponding CSV files containing the tabular data), you can then use [`TapasTokenizer`] to convert table-question pairs into `input_ids`, `attention_mask`, `token_type_ids` and so on. Again, based on which of the three cases you picked above, [`TapasForQuestionAnswering`] requires different\ninputs to be fine-tuned:\n\n| **Task**                           | **Required inputs**                                                                                                 |\n|------------------------------------|---------------------------------------------------------------------------------------------------------------------|\n| Conversational                     | `input_ids`, `attention_mask`, `token_type_ids`, `labels`                                                           |\n|  Weak supervision for aggregation  | `input_ids`, `attention_mask`, `token_type_ids`, `labels`, `numeric_values`, `numeric_values_scale`, `float_answer` |\n| Strong supervision for aggregation | `input ids`, `attention mask`, `token type ids`, `labels`, `aggregation_labels`                                     |\n\n[`TapasTokenizer`] creates the `labels`, `numeric_values` and `numeric_values_scale` based on the `answer_coordinates` and `answer_text` columns of the TSV file. The `float_answer` and `aggregation_labels` are already in the TSV file of step 2. Here's an example:\n\n```py\n>>> from transformers import TapasTokenizer\n>>> import pandas as pd\n\n>>> model_name = \"google/tapas-base\"\n>>> tokenizer = TapasTokenizer.from_pretrained(model_name)\n\n>>> data = {\"Actors\": [\"Brad Pitt\", \"Leonardo Di Caprio\", \"George Clooney\"], \"Number of movies\": [\"87\", \"53\", \"69\"]}\n>>> queries = [\n...     \"What is the name of the first actor?"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 367ba07f-0557-42af-982b-b9621292215d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas/", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "HF_Transformers", "_node_content": "{\"id_\": \"367ba07f-0557-42af-982b-b9621292215d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6f8b7462-3307-4ed8-a23a-6d7f8d36fa39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"02f9d1e611bfdcdddc25ffc55f689dfee098357342da90a329bd3b562cf383cc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b6d78865-c41f-4f6c-9dbe-88d442a7b93c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"8ce00e80d0437bb32a6217c9ad6bce12841ad3398823ad801ec48503ca92ca91\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"959c706b-4c0c-4c70-9a7e-acf61c3a0944\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1aca70a288eb127f649e47744011c0fccf867966ce500164d4cab620c34e7516\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14919, \"end_char_idx\": 17968, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "ref_doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "chroma:document": "**STEP 3: Convert your data into tensors using TapasTokenizer**\n\n<frameworkcontent>\n<pt>\nThird, given that you've prepared your data in this TSV/CSV format (and corresponding CSV files containing the tabular data), you can then use [`TapasTokenizer`] to convert table-question pairs into `input_ids`, `attention_mask`, `token_type_ids` and so on. Again, based on which of the three cases you picked above, [`TapasForQuestionAnswering`] requires different\ninputs to be fine-tuned:\n\n| **Task**                           | **Required inputs**                                                                                                 |\n|------------------------------------|---------------------------------------------------------------------------------------------------------------------|\n| Conversational                     | `input_ids`, `attention_mask`, `token_type_ids`, `labels`                                                           |\n|  Weak supervision for aggregation  | `input_ids`, `attention_mask`, `token_type_ids`, `labels`, `numeric_values`, `numeric_values_scale`, `float_answer` |\n| Strong supervision for aggregation | `input ids`, `attention mask`, `token type ids`, `labels`, `aggregation_labels`                                     |\n\n[`TapasTokenizer`] creates the `labels`, `numeric_values` and `numeric_values_scale` based on the `answer_coordinates` and `answer_text` columns of the TSV file. The `float_answer` and `aggregation_labels` are already in the TSV file of step 2. Here's an example:\n\n```py\n>>> from transformers import TapasTokenizer\n>>> import pandas as pd\n\n>>> model_name = \"google/tapas-base\"\n>>> tokenizer = TapasTokenizer.from_pretrained(model_name)\n\n>>> data = {\"Actors\": [\"Brad Pitt\", \"Leonardo Di Caprio\", \"George Clooney\"], \"Number of movies\": [\"87\", \"53\", \"69\"]}\n>>> queries = [\n...     \"What is the name of the first actor?\",\n...     \"How many movies has George Clooney played in?\",\n...     \"What is the total number of movies?\",\n... ]\n>>> answer_coordinates = [[(0, 0)], [(2, 1)], [(0, 1), (1, 1), (2, 1)]]\n>>> answer_text = [[\"Brad Pitt\"], [\"69\"], [\"209\"]]\n>>> table = pd.DataFrame.from_dict(data)\n>>> inputs = tokenizer(\n...     table=table,\n...     queries=queries,\n...     answer_coordinates=answer_coordinates,\n...     answer_text=answer_text,\n...     padding=\"max_length\",\n...     return_tensors=\"pt\",\n... )\n>>> inputs\n{'input_ids': tensor([[ ... ]]), 'attention_mask': tensor([[...]]), 'token_type_ids': tensor([[[...]]]),\n'numeric_values': tensor([[ ... ]]), 'numeric_values_scale: tensor([[ ... ]]), labels: tensor([[ ... ]])}\n```\n\nNote that [`TapasTokenizer`] expects the data of the table to be **text-only**. You can use `.astype(str)` on a dataframe to turn it into text-only data.\nOf course, this only shows how to encode a single training example. It is advised to create a dataloader to iterate over batches:\n\n```py\n>>> import torch\n>>> import pandas as pd\n\n>>> tsv_path = \"your_path_to_the_tsv_file\"\n>>> table_csv_path = \"your_path_to_a_directory_containing_all_csv_files\""}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 959c706b-4c0c-4c70-9a7e-acf61c3a0944
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas/", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "HF_Transformers", "_node_content": "{\"id_\": \"959c706b-4c0c-4c70-9a7e-acf61c3a0944\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6f8b7462-3307-4ed8-a23a-6d7f8d36fa39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"02f9d1e611bfdcdddc25ffc55f689dfee098357342da90a329bd3b562cf383cc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"367ba07f-0557-42af-982b-b9621292215d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"4968748f8963c6037cf48b54098f46d4830feefd0a5d5f91860ed0b921a709e9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0789208d-1942-49f6-b3cc-9d55354676ac\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"10ab27f0dffc801c0708d0f5be38c1d5b4d944fce26212363e9b01a37c3921c3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 16801, \"end_char_idx\": 19106, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "ref_doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "chroma:document": "\",\n...     \"How many movies has George Clooney played in?\",\n...     \"What is the total number of movies?\",\n... ]\n>>> answer_coordinates = [[(0, 0)], [(2, 1)], [(0, 1), (1, 1), (2, 1)]]\n>>> answer_text = [[\"Brad Pitt\"], [\"69\"], [\"209\"]]\n>>> table = pd.DataFrame.from_dict(data)\n>>> inputs = tokenizer(\n...     table=table,\n...     queries=queries,\n...     answer_coordinates=answer_coordinates,\n...     answer_text=answer_text,\n...     padding=\"max_length\",\n...     return_tensors=\"pt\",\n... )\n>>> inputs\n{'input_ids': tensor([[ ... ]]), 'attention_mask': tensor([[...]]), 'token_type_ids': tensor([[[...]]]),\n'numeric_values': tensor([[ ... ]]), 'numeric_values_scale: tensor([[ ... ]]), labels: tensor([[ ... ]])}\n```\n\nNote that [`TapasTokenizer`] expects the data of the table to be **text-only**. You can use `.astype(str)` on a dataframe to turn it into text-only data.\nOf course, this only shows how to encode a single training example. It is advised to create a dataloader to iterate over batches:\n\n```py\n>>> import torch\n>>> import pandas as pd\n\n>>> tsv_path = \"your_path_to_the_tsv_file\"\n>>> table_csv_path = \"your_path_to_a_directory_containing_all_csv_files\"\n\n\n>>> class TableDataset(torch.utils.data.Dataset):\n...     def __init__(self, data, tokenizer):\n...         self.data = data\n...         self.tokenizer = tokenizer\n\n...     def __getitem__(self, idx):\n...         item = data.iloc[idx]\n...         table = pd.read_csv(table_csv_path + item.table_file).astype(\n...             str\n...         )  # be sure to make your table data text only\n...         encoding = self.tokenizer(\n...             table=table,\n...             queries=item.question,\n...             answer_coordinates=item.answer_coordinates,\n...             answer_text=item.answer_text,\n...             truncation=True,\n...             padding=\"max_length\",\n...             return_tensors=\"pt\",\n...         )\n...         # remove the batch dimension which the tokenizer adds by default\n...         encoding = {key: val.squeeze(0) for key, val in encoding.items()}\n...         # add the float_answer which is also required (weak supervision for aggregation case)\n...         encoding[\"float_answer\"] = torch.tensor(item.float_answer)\n...         return encoding\n\n...     def __len__(self):\n...         return len(self.data)"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 0789208d-1942-49f6-b3cc-9d55354676ac
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas/", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "HF_Transformers", "_node_content": "{\"id_\": \"0789208d-1942-49f6-b3cc-9d55354676ac\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6f8b7462-3307-4ed8-a23a-6d7f8d36fa39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"02f9d1e611bfdcdddc25ffc55f689dfee098357342da90a329bd3b562cf383cc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"959c706b-4c0c-4c70-9a7e-acf61c3a0944\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"97213c95cc2dfa8fd97de96ba2af9fb214718ba80041221a279fd7c68a823ec7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e0a371b2-5ee2-443b-aada-91a5aaf9eca7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5490a13e6d08534c3f45d0017a5a95613556f89727d1492687d4ad4cac0f95ab\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 17600, \"end_char_idx\": 22271, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "ref_doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "chroma:document": "You can use `.astype(str)` on a dataframe to turn it into text-only data.\nOf course, this only shows how to encode a single training example. It is advised to create a dataloader to iterate over batches:\n\n```py\n>>> import torch\n>>> import pandas as pd\n\n>>> tsv_path = \"your_path_to_the_tsv_file\"\n>>> table_csv_path = \"your_path_to_a_directory_containing_all_csv_files\"\n\n\n>>> class TableDataset(torch.utils.data.Dataset):\n...     def __init__(self, data, tokenizer):\n...         self.data = data\n...         self.tokenizer = tokenizer\n\n...     def __getitem__(self, idx):\n...         item = data.iloc[idx]\n...         table = pd.read_csv(table_csv_path + item.table_file).astype(\n...             str\n...         )  # be sure to make your table data text only\n...         encoding = self.tokenizer(\n...             table=table,\n...             queries=item.question,\n...             answer_coordinates=item.answer_coordinates,\n...             answer_text=item.answer_text,\n...             truncation=True,\n...             padding=\"max_length\",\n...             return_tensors=\"pt\",\n...         )\n...         # remove the batch dimension which the tokenizer adds by default\n...         encoding = {key: val.squeeze(0) for key, val in encoding.items()}\n...         # add the float_answer which is also required (weak supervision for aggregation case)\n...         encoding[\"float_answer\"] = torch.tensor(item.float_answer)\n...         return encoding\n\n...     def __len__(self):\n...         return len(self.data)\n\n\n>>> data = pd.read_csv(tsv_path, sep=\"\\t\")\n>>> train_dataset = TableDataset(data, tokenizer)\n>>> train_dataloader = torch.utils.data.DataLoader(train_dataset, batch_size=32)\n```\n</pt>\n<tf>\nThird, given that you've prepared your data in this TSV/CSV format (and corresponding CSV files containing the tabular data), you can then use [`TapasTokenizer`] to convert table-question pairs into `input_ids`, `attention_mask`, `token_type_ids` and so on. Again, based on which of the three cases you picked above, [`TFTapasForQuestionAnswering`] requires different\ninputs to be fine-tuned:\n\n| **Task**                           | **Required inputs**                                                                                                 |\n|------------------------------------|---------------------------------------------------------------------------------------------------------------------|\n| Conversational                     | `input_ids`, `attention_mask`, `token_type_ids`, `labels`                                                           |\n|  Weak supervision for aggregation  | `input_ids`, `attention_mask`, `token_type_ids`, `labels`, `numeric_values`, `numeric_values_scale`, `float_answer` |\n| Strong supervision for aggregation | `input ids`, `attention mask`, `token type ids`, `labels`, `aggregation_labels`                                     |\n\n[`TapasTokenizer`] creates the `labels`, `numeric_values` and `numeric_values_scale` based on the `answer_coordinates` and `answer_text` columns of the TSV file. The `float_answer` and `aggregation_labels` are already in the TSV file of step 2. Here's an example:\n\n```py\n>>> from transformers import TapasTokenizer\n>>> import pandas as pd\n\n>>> model_name = \"google/tapas-base\"\n>>> tokenizer = TapasTokenizer.from_pretrained(model_name)\n\n>>> data = {\"Actors\": [\"Brad Pitt\", \"Leonardo Di Caprio\", \"George Clooney\"], \"Number of movies\": [\"87\", \"53\", \"69\"]}\n>>> queries = [\n...     \"What is the name of the first actor?\",\n...     \"How many movies has George Clooney played in?\",\n...     \"What is the total number of movies?\",\n... ]\n>>> answer_coordinates = [[(0, 0)], [(2, 1)], [(0, 1), (1, 1), (2, 1)]]\n>>> answer_text = [[\"Brad Pitt\"], [\"69\"], [\"209\"]]\n>>> table = pd.DataFrame.from_dict(data)\n>>> inputs = tokenizer(\n...     table=table,\n...     queries=queries,\n...     answer_coordinates=answer_coordinates,\n...     answer_text=answer_text,\n...     padding=\"max_length\",\n...     return_tensors=\"tf\",\n... )\n>>> inputs\n{'input_ids': tensor([[ ... ]]), 'attention_mask': tensor([[...]]), 'token_type_ids': tensor([[[...]]]),\n'numeric_values': tensor([[ ... ]]), 'numeric_values_scale: tensor([[ ... ]]), labels: tensor([[ ... ]])}\n```\n\nNote that [`TapasTokenizer`] expects the data of the table to be **text-only**. You can use `.astype(str)` on a dataframe to turn it into text-only data.\nOf course, this only shows how to encode a single training example. It is advised to create a dataloader to iterate over batches:\n\n```py\n>>> import tensorflow as tf\n>>> import pandas as pd\n\n>>> tsv_path = \"your_path_to_the_tsv_file\"\n>>> table_csv_path = \"your_path_to_a_directory_containing_all_csv_files\""}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: e0a371b2-5ee2-443b-aada-91a5aaf9eca7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas/", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "HF_Transformers", "_node_content": "{\"id_\": \"e0a371b2-5ee2-443b-aada-91a5aaf9eca7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6f8b7462-3307-4ed8-a23a-6d7f8d36fa39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"02f9d1e611bfdcdddc25ffc55f689dfee098357342da90a329bd3b562cf383cc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0789208d-1942-49f6-b3cc-9d55354676ac\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"c3ce95def5b287759de26d8f37802cafd92ae23c6cad3b1ca7d4d1f4efcaf711\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"999da0b6-050e-4e74-b3bf-4c83cbeb3ff8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bf3e72011ba82d8e9164c074330aba58095415c84d7dd5ec0f3f6aa5c8b619c7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "ref_doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "chroma:document": ">>> class TableDataset:\n...     def __init__(self, data, tokenizer):\n...         self.data = data\n...         self.tokenizer = tokenizer\n\n...     def __iter__(self):\n...         for idx in range(self.__len__()):\n...             item = self.data.iloc[idx]\n...             table = pd.read_csv(table_csv_path + item.table_file).astype(\n...                 str\n...             )  # be sure to make your table data text only\n...             encoding = self.tokenizer(\n...                 table=table,\n...                 queries=item.question,\n...                 answer_coordinates=item.answer_coordinates,\n...                 answer_text=item.answer_text,\n...                 truncation=True,\n...                 padding=\"max_length\",\n...                 return_tensors=\"tf\",\n...             )\n...             # remove the batch dimension which the tokenizer adds by default\n...             encoding = {key: tf.squeeze(val, 0) for key, val in encoding.items()}\n...             # add the float_answer which is also required (weak supervision for aggregation case)\n...             encoding[\"float_answer\"] = tf.convert_to_tensor(item.float_answer, dtype=tf.float32)\n...             yield encoding[\"input_ids\"], encoding[\"attention_mask\"], encoding[\"numeric_values\"], encoding[\n...                 \"numeric_values_scale\"\n...             ], encoding[\"token_type_ids\"], encoding[\"labels\"], encoding[\"float_answer\"]\n\n...     def __len__(self):\n...         return len(self.data)\n\n\n>>> data = pd.read_csv(tsv_path, sep=\"\\t\")\n>>> train_dataset = TableDataset(data, tokenizer)\n>>> output_signature = (\n...     tf.TensorSpec(shape=(512,), dtype=tf.int32),\n...     tf.TensorSpec(shape=(512,), dtype=tf.int32),\n...     tf.TensorSpec(shape=(512,), dtype=tf.float32),\n...     tf.TensorSpec(shape=(512,), dtype=tf.float32),\n...     tf.TensorSpec(shape=(512, 7), dtype=tf.int32),\n...     tf.TensorSpec(shape=(512,), dtype=tf.int32),\n...     tf.TensorSpec(shape=(512,), dtype=tf.float32),\n... )\n>>> train_dataloader = tf.data.Dataset.from_generator(train_dataset, output_signature=output_signature).batch(32)\n```\n</tf>\n</frameworkcontent>\n\nNote that here, we encode each table-question pair independently. This is fine as long as your dataset is **not conversational**. In case your dataset involves conversational questions (such as in SQA), then you should first group together the `queries`, `answer_coordinates` and `answer_text` per table (in the order of their `position`\nindex) and batch encode each table with its questions. This will make sure that the `prev_labels` token types (see docs of [`TapasTokenizer`]) are set correctly. See [this notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/TAPAS/Fine_tuning_TapasForQuestionAnswering_on_SQA.ipynb) for more info. See [this notebook](https://github.com/kamalkraj/Tapas-Tutorial/blob/master/TAPAS/Fine_tuning_TapasForQuestionAnswering_on_SQA.ipynb) for more info regarding using the TensorFlow model.\n\n**STEP 4: Train (fine-tune) the model\n\n<frameworkcontent>\n<pt>\nYou can then fine-tune [`TapasForQuestionAnswering`] as follows (shown here for the weak supervision for aggregation case):\n\n```py\n>>> from transformers import TapasConfig, TapasForQuestionAnswering, AdamW\n\n>>> # this is the default WTQ configuration\n>>> config = TapasConfig(\n.     num_aggregation_labels=4,\n.     use_answer_as_supervision=True,\n.     answer_loss_cutoff=0.664694,\n."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 999da0b6-050e-4e74-b3bf-4c83cbeb3ff8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas/", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "HF_Transformers", "_node_content": "{\"id_\": \"999da0b6-050e-4e74-b3bf-4c83cbeb3ff8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6f8b7462-3307-4ed8-a23a-6d7f8d36fa39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"02f9d1e611bfdcdddc25ffc55f689dfee098357342da90a329bd3b562cf383cc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e0a371b2-5ee2-443b-aada-91a5aaf9eca7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"36263415ce1e30c8aac0ab373dd3c9256e316429d6f893055ee4788fb277231d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"255e80b2-4ca0-4a6c-a444-c69d52de956d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"473e2b92ae58897c38b18d4a0d30fd273ca0a21b89f9be891ebcc87d865bcc4e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "ref_doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "chroma:document": "This is fine as long as your dataset is **not conversational**. In case your dataset involves conversational questions (such as in SQA), then you should first group together the `queries`, `answer_coordinates` and `answer_text` per table (in the order of their `position`\nindex) and batch encode each table with its questions. This will make sure that the `prev_labels` token types (see docs of [`TapasTokenizer`]) are set correctly. See [this notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/TAPAS/Fine_tuning_TapasForQuestionAnswering_on_SQA.ipynb) for more info. See [this notebook](https://github.com/kamalkraj/Tapas-Tutorial/blob/master/TAPAS/Fine_tuning_TapasForQuestionAnswering_on_SQA.ipynb) for more info regarding using the TensorFlow model.\n\n**STEP 4: Train (fine-tune) the model\n\n<frameworkcontent>\n<pt>\nYou can then fine-tune [`TapasForQuestionAnswering`] as follows (shown here for the weak supervision for aggregation case):\n\n```py\n>>> from transformers import TapasConfig, TapasForQuestionAnswering, AdamW\n\n>>> # this is the default WTQ configuration\n>>> config = TapasConfig(\n.     num_aggregation_labels=4,\n.     use_answer_as_supervision=True,\n.     answer_loss_cutoff=0.664694,\n.     cell_selection_preference=0.207951,\n.     huber_loss_delta=0.121194,\n.     init_cell_selection_weights_to_zero=True,\n.     select_one_column=True,\n.     allow_empty_column_selection=False,\n.     temperature=0.0352513,\n. )\n>>> model = TapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n\n>>> optimizer = AdamW(model.parameters(), lr=5e-5)\n\n>>> model.train()\n>>> for epoch in range(2):  # loop over the dataset multiple times\n.     for batch in train_dataloader:\n.         # get the inputs;\n.         input_ids = batch[\"input_ids\"]\n.         attention_mask = batch[\"attention_mask\"]\n.         token_type_ids = batch[\"token_type_ids\"]\n.         labels = batch[\"labels\"]\n.         numeric_values = batch[\"numeric_values\"]\n.         numeric_values_scale = batch[\"numeric_values_scale\"]\n.         float_answer = batch[\"float_answer\"]\n\n.         # zero the parameter gradients\n.         optimizer.zero_grad()\n\n.         # forward + backward + optimize\n.         outputs = model(\n.             input_ids=input_ids,\n.             attention_mask=attention_mask,\n.             token_type_ids=token_type_ids,\n.             labels=labels,\n.             numeric_values=numeric_values,\n.             numeric_values_scale=numeric_values_scale,\n.             float_answer=float_answer,\n.         )\n.         loss = outputs.loss\n.         loss.backward()\n.         optimizer.step()\n```\n</pt>\n<tf>\nYou can then fine-tune [`TFTapasForQuestionAnswering`] as follows (shown here for the weak supervision for aggregation case):\n\n```py\n>>> import tensorflow as tf\n>>> from transformers import TapasConfig, TFTapasForQuestionAnswering\n\n>>> # this is the default WTQ configuration\n>>> config = TapasConfig(\n.     num_aggregation_labels=4,\n.     use_answer_as_supervision=True,\n.     answer_loss_cutoff=0.664694,\n.     cell_selection_preference=0.207951,\n.     huber_loss_delta=0.121194,\n."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 255e80b2-4ca0-4a6c-a444-c69d52de956d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas/", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "HF_Transformers", "_node_content": "{\"id_\": \"255e80b2-4ca0-4a6c-a444-c69d52de956d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6f8b7462-3307-4ed8-a23a-6d7f8d36fa39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"02f9d1e611bfdcdddc25ffc55f689dfee098357342da90a329bd3b562cf383cc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"999da0b6-050e-4e74-b3bf-4c83cbeb3ff8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"deabfcecc3a5aa69effc2f7482ed4d0d23b34b69c4c2989f77f25aad66c04238\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e7ef9ddc-f411-4421-9014-83666e8a79ad\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"13daf7eb36d0d077b78ee3959811fe83697d4b661e94c8eca429d1b6b7bb72f2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "ref_doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "chroma:document": "from_pretrained(\"google/tapas-base\", config=config)\n\n>>> optimizer = AdamW(model.parameters(), lr=5e-5)\n\n>>> model.train()\n>>> for epoch in range(2):  # loop over the dataset multiple times\n.     for batch in train_dataloader:\n.         # get the inputs;\n.         input_ids = batch[\"input_ids\"]\n.         attention_mask = batch[\"attention_mask\"]\n.         token_type_ids = batch[\"token_type_ids\"]\n.         labels = batch[\"labels\"]\n.         numeric_values = batch[\"numeric_values\"]\n.         numeric_values_scale = batch[\"numeric_values_scale\"]\n.         float_answer = batch[\"float_answer\"]\n\n.         # zero the parameter gradients\n.         optimizer.zero_grad()\n\n.         # forward + backward + optimize\n.         outputs = model(\n.             input_ids=input_ids,\n.             attention_mask=attention_mask,\n.             token_type_ids=token_type_ids,\n.             labels=labels,\n.             numeric_values=numeric_values,\n.             numeric_values_scale=numeric_values_scale,\n.             float_answer=float_answer,\n.         )\n.         loss = outputs.loss\n.         loss.backward()\n.         optimizer.step()\n```\n</pt>\n<tf>\nYou can then fine-tune [`TFTapasForQuestionAnswering`] as follows (shown here for the weak supervision for aggregation case):\n\n```py\n>>> import tensorflow as tf\n>>> from transformers import TapasConfig, TFTapasForQuestionAnswering\n\n>>> # this is the default WTQ configuration\n>>> config = TapasConfig(\n.     num_aggregation_labels=4,\n.     use_answer_as_supervision=True,\n.     answer_loss_cutoff=0.664694,\n.     cell_selection_preference=0.207951,\n.     huber_loss_delta=0.121194,\n.     init_cell_selection_weights_to_zero=True,\n.     select_one_column=True,\n.     allow_empty_column_selection=False,\n.     temperature=0.0352513,\n. )\n>>> model = TFTapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n\n>>> optimizer = tf.keras.optimizers.Adam(learning_rate=5e-5)\n\n>>> for epoch in range(2):  # loop over the dataset multiple times\n.     for batch in train_dataloader:\n.         # get the inputs;\n.         input_ids = batch[0]\n.         attention_mask = batch[1]\n.         token_type_ids = batch[4]\n.         labels = batch[-1]\n.         numeric_values = batch[2]\n.         numeric_values_scale = batch[3]\n.         float_answer = batch[6]\n\n.         # forward + backward + optimize\n.         with tf.GradientTape() as tape:\n.             outputs = model(\n.                 input_ids=input_ids,\n.                 attention_mask=attention_mask,\n.                 token_type_ids=token_type_ids,\n.                 labels=labels,\n.                 numeric_values=numeric_values,\n.                 numeric_values_scale=numeric_values_scale,\n.                 float_answer=float_answer,\n.             )\n.         grads = tape.gradient(outputs.loss, model.trainable_weights)\n.         optimizer.apply_gradients(zip(grads, model.trainable_weights))\n```\n</tf>\n</frameworkcontent>\n\n## Usage: inference\n\n<frameworkcontent>\n<pt>\nHere we explain how you can use [`TapasForQuestionAnswering`] or [`TFTapasForQuestionAnswering`] for inference (i.e. making predictions on new data)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: e7ef9ddc-f411-4421-9014-83666e8a79ad
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas/", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "HF_Transformers", "_node_content": "{\"id_\": \"e7ef9ddc-f411-4421-9014-83666e8a79ad\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6f8b7462-3307-4ed8-a23a-6d7f8d36fa39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"02f9d1e611bfdcdddc25ffc55f689dfee098357342da90a329bd3b562cf383cc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"255e80b2-4ca0-4a6c-a444-c69d52de956d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"b34e60d70b4a3aa530460f9c53c49955a5acab4d38259c5ff725f97459b061d6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a2716656-cc9d-49b9-94d6-993690dddf8c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"73d42871b21a252a4a31887feb1e6c88f58c8556c066d677e27b4719b7c5edd8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "ref_doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "chroma:document": "664694,\n.     cell_selection_preference=0.207951,\n.     huber_loss_delta=0.121194,\n.     init_cell_selection_weights_to_zero=True,\n.     select_one_column=True,\n.     allow_empty_column_selection=False,\n.     temperature=0.0352513,\n. )\n>>> model = TFTapasForQuestionAnswering.from_pretrained(\"google/tapas-base\", config=config)\n\n>>> optimizer = tf.keras.optimizers.Adam(learning_rate=5e-5)\n\n>>> for epoch in range(2):  # loop over the dataset multiple times\n.     for batch in train_dataloader:\n.         # get the inputs;\n.         input_ids = batch[0]\n.         attention_mask = batch[1]\n.         token_type_ids = batch[4]\n.         labels = batch[-1]\n.         numeric_values = batch[2]\n.         numeric_values_scale = batch[3]\n.         float_answer = batch[6]\n\n.         # forward + backward + optimize\n.         with tf.GradientTape() as tape:\n.             outputs = model(\n.                 input_ids=input_ids,\n.                 attention_mask=attention_mask,\n.                 token_type_ids=token_type_ids,\n.                 labels=labels,\n.                 numeric_values=numeric_values,\n.                 numeric_values_scale=numeric_values_scale,\n.                 float_answer=float_answer,\n.             )\n.         grads = tape.gradient(outputs.loss, model.trainable_weights)\n.         optimizer.apply_gradients(zip(grads, model.trainable_weights))\n```\n</tf>\n</frameworkcontent>\n\n## Usage: inference\n\n<frameworkcontent>\n<pt>\nHere we explain how you can use [`TapasForQuestionAnswering`] or [`TFTapasForQuestionAnswering`] for inference (i.e. making predictions on new data). For inference, only `input_ids`, `attention_mask` and `token_type_ids` (which you can obtain using [`TapasTokenizer`]) have to be provided to the model to obtain the logits. Next, you can use the handy [`~models.tapas.tokenization_tapas.convert_logits_to_predictions`] method to convert these into predicted coordinates and optional aggregation indices.\n\nHowever, note that inference is **different** depending on whether or not the setup is conversational. In a non-conversational set-up, inference can be done in parallel on all table-question pairs of a batch. Here's an example of that:\n\n```py\n>>> from transformers import TapasTokenizer, TapasForQuestionAnswering\n>>> import pandas as pd\n\n>>> model_name = \"google/tapas-base-finetuned-wtq\"\n>>> model = TapasForQuestionAnswering.from_pretrained(model_name)\n>>> tokenizer = TapasTokenizer.from_pretrained(model_name)\n\n>>> data = {\"Actors\": [\"Brad Pitt\", \"Leonardo Di Caprio\", \"George Clooney\"], \"Number of movies\": [\"87\", \"53\", \"69\"]}\n>>> queries = [\n...     \"What is the name of the first actor?\",\n...     \"How many movies has George Clooney played in?\",\n...     \"What is the total number of movies?"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: a2716656-cc9d-49b9-94d6-993690dddf8c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas/", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "HF_Transformers", "_node_content": "{\"id_\": \"a2716656-cc9d-49b9-94d6-993690dddf8c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6f8b7462-3307-4ed8-a23a-6d7f8d36fa39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"02f9d1e611bfdcdddc25ffc55f689dfee098357342da90a329bd3b562cf383cc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e7ef9ddc-f411-4421-9014-83666e8a79ad\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"d3e8ec6fa684eafa2f211dbd3733f99611fe7b00f1ef10891e34761a3b87b60f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6b0bd6df-694d-4303-9646-ccea4b4a8d97\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"530707ef451f539d44381658ad85719032521fb0c5ff7acbe79abd2bcc0bf978\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "ref_doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "chroma:document": "numeric_values=numeric_values,\n.                 numeric_values_scale=numeric_values_scale,\n.                 float_answer=float_answer,\n.             )\n.         grads = tape.gradient(outputs.loss, model.trainable_weights)\n.         optimizer.apply_gradients(zip(grads, model.trainable_weights))\n```\n</tf>\n</frameworkcontent>\n\n## Usage: inference\n\n<frameworkcontent>\n<pt>\nHere we explain how you can use [`TapasForQuestionAnswering`] or [`TFTapasForQuestionAnswering`] for inference (i.e. making predictions on new data). For inference, only `input_ids`, `attention_mask` and `token_type_ids` (which you can obtain using [`TapasTokenizer`]) have to be provided to the model to obtain the logits. Next, you can use the handy [`~models.tapas.tokenization_tapas.convert_logits_to_predictions`] method to convert these into predicted coordinates and optional aggregation indices.\n\nHowever, note that inference is **different** depending on whether or not the setup is conversational. In a non-conversational set-up, inference can be done in parallel on all table-question pairs of a batch. Here's an example of that:\n\n```py\n>>> from transformers import TapasTokenizer, TapasForQuestionAnswering\n>>> import pandas as pd\n\n>>> model_name = \"google/tapas-base-finetuned-wtq\"\n>>> model = TapasForQuestionAnswering.from_pretrained(model_name)\n>>> tokenizer = TapasTokenizer.from_pretrained(model_name)\n\n>>> data = {\"Actors\": [\"Brad Pitt\", \"Leonardo Di Caprio\", \"George Clooney\"], \"Number of movies\": [\"87\", \"53\", \"69\"]}\n>>> queries = [\n...     \"What is the name of the first actor?\",\n...     \"How many movies has George Clooney played in?\",\n...     \"What is the total number of movies?\",\n... ]\n>>> table = pd.DataFrame.from_dict(data)\n>>> inputs = tokenizer(table=table, queries=queries, padding=\"max_length\", return_tensors=\"pt\")\n>>> outputs = model(**inputs)\n>>> predicted_answer_coordinates, predicted_aggregation_indices = tokenizer.convert_logits_to_predictions(\n...     inputs, outputs.logits.detach(), outputs.logits_aggregation.detach()\n... )\n\n>>> # let's print out the results:\n>>> id2aggregation = {0: \"NONE\", 1: \"SUM\", 2: \"AVERAGE\", 3: \"COUNT\"}\n>>> aggregation_predictions_string = [id2aggregation[x] for x in predicted_aggregation_indices]\n\n>>> answers = []\n>>> for coordinates in predicted_answer_coordinates:\n...     if len(coordinates) == 1:\n...         # only a single cell:\n...         answers.append(table.iat[coordinates[0]])\n...     else:\n...         # multiple cells\n...         cell_values = []\n...         for coordinate in coordinates:\n...             cell_values.append(table.iat[coordinate])\n...         answers.append(\", \".join(cell_values))\n\n>>> display(table)\n>>> print(\"\")\n>>> for query, answer, predicted_agg in zip(queries, answers, aggregation_predictions_string):\n...     print(query)\n...     if predicted_agg == \"NONE\":\n...         print(\"Predicted answer: \" + answer)\n...     else:\n...         print(\"Predicted answer: \" + predicted_agg + \" > \" + answer)\nWhat is the name of the first actor?\nPredicted answer: Brad Pitt\nHow many movies has George Clooney played in?\nPredicted answer: COUNT > 69\nWhat is the total number of movies?"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 6b0bd6df-694d-4303-9646-ccea4b4a8d97
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas/", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "HF_Transformers", "_node_content": "{\"id_\": \"6b0bd6df-694d-4303-9646-ccea4b4a8d97\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6f8b7462-3307-4ed8-a23a-6d7f8d36fa39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"02f9d1e611bfdcdddc25ffc55f689dfee098357342da90a329bd3b562cf383cc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a2716656-cc9d-49b9-94d6-993690dddf8c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"698f9439788eb4f2c1ed18fa53da286a33733fa9bb721daddd61d00233f4c754\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"166bdabb-6f35-4b8a-9729-6500b0ef9959\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"16148826af60d7d6f5f0365c8155bc86b92cb18b7174ca5fb72da00db3919f50\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 30268, \"end_char_idx\": 33178, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "ref_doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "chroma:document": "\",\n...     \"How many movies has George Clooney played in?\",\n...     \"What is the total number of movies?\",\n... ]\n>>> table = pd.DataFrame.from_dict(data)\n>>> inputs = tokenizer(table=table, queries=queries, padding=\"max_length\", return_tensors=\"pt\")\n>>> outputs = model(**inputs)\n>>> predicted_answer_coordinates, predicted_aggregation_indices = tokenizer.convert_logits_to_predictions(\n...     inputs, outputs.logits.detach(), outputs.logits_aggregation.detach()\n... )\n\n>>> # let's print out the results:\n>>> id2aggregation = {0: \"NONE\", 1: \"SUM\", 2: \"AVERAGE\", 3: \"COUNT\"}\n>>> aggregation_predictions_string = [id2aggregation[x] for x in predicted_aggregation_indices]\n\n>>> answers = []\n>>> for coordinates in predicted_answer_coordinates:\n...     if len(coordinates) == 1:\n...         # only a single cell:\n...         answers.append(table.iat[coordinates[0]])\n...     else:\n...         # multiple cells\n...         cell_values = []\n...         for coordinate in coordinates:\n...             cell_values.append(table.iat[coordinate])\n...         answers.append(\", \".join(cell_values))\n\n>>> display(table)\n>>> print(\"\")\n>>> for query, answer, predicted_agg in zip(queries, answers, aggregation_predictions_string):\n...     print(query)\n...     if predicted_agg == \"NONE\":\n...         print(\"Predicted answer: \" + answer)\n...     else:\n...         print(\"Predicted answer: \" + predicted_agg + \" > \" + answer)\nWhat is the name of the first actor?\nPredicted answer: Brad Pitt\nHow many movies has George Clooney played in?\nPredicted answer: COUNT > 69\nWhat is the total number of movies?\nPredicted answer: SUM > 87, 53, 69\n```\n</pt>\n<tf>\nHere we explain how you can use [`TFTapasForQuestionAnswering`] for inference (i.e. making predictions on new data). For inference, only `input_ids`, `attention_mask` and `token_type_ids` (which you can obtain using [`TapasTokenizer`]) have to be provided to the model to obtain the logits. Next, you can use the handy [`~models.tapas.tokenization_tapas.convert_logits_to_predictions`] method to convert these into predicted coordinates and optional aggregation indices.\n\nHowever, note that inference is **different** depending on whether or not the setup is conversational. In a non-conversational set-up, inference can be done in parallel on all table-question pairs of a batch. Here's an example of that:\n\n```py\n>>> from transformers import TapasTokenizer, TFTapasForQuestionAnswering\n>>> import pandas as pd\n\n>>> model_name = \"google/tapas-base-finetuned-wtq\"\n>>> model = TFTapasForQuestionAnswering.from_pretrained(model_name)\n>>> tokenizer = TapasTokenizer.from_pretrained(model_name)\n\n>>> data = {\"Actors\": [\"Brad Pitt\", \"Leonardo Di Caprio\", \"George Clooney\"], \"Number of movies\": [\"87\", \"53\", \"69\"]}\n>>> queries = [\n...     \"What is the name of the first actor?\",\n...     \"How many movies has George Clooney played in?\",\n...     \"What is the total number of movies?"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 166bdabb-6f35-4b8a-9729-6500b0ef9959
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas/", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "HF_Transformers", "_node_content": "{\"id_\": \"166bdabb-6f35-4b8a-9729-6500b0ef9959\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6f8b7462-3307-4ed8-a23a-6d7f8d36fa39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"02f9d1e611bfdcdddc25ffc55f689dfee098357342da90a329bd3b562cf383cc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6b0bd6df-694d-4303-9646-ccea4b4a8d97\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"fffc952ad5eda056a045af794738a916ffb33931c0112c560c638926ecf5e09d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d03b2662-0b5c-45bb-9a36-a9c6394e905c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"22475e8c44e5f9d42ac182183fa5e68edfe42edfaed79b206d56d8a84cf538bc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 31715, \"end_char_idx\": 34943, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "ref_doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "chroma:document": "Predicted answer: Brad Pitt\nHow many movies has George Clooney played in?\nPredicted answer: COUNT > 69\nWhat is the total number of movies?\nPredicted answer: SUM > 87, 53, 69\n```\n</pt>\n<tf>\nHere we explain how you can use [`TFTapasForQuestionAnswering`] for inference (i.e. making predictions on new data). For inference, only `input_ids`, `attention_mask` and `token_type_ids` (which you can obtain using [`TapasTokenizer`]) have to be provided to the model to obtain the logits. Next, you can use the handy [`~models.tapas.tokenization_tapas.convert_logits_to_predictions`] method to convert these into predicted coordinates and optional aggregation indices.\n\nHowever, note that inference is **different** depending on whether or not the setup is conversational. In a non-conversational set-up, inference can be done in parallel on all table-question pairs of a batch. Here's an example of that:\n\n```py\n>>> from transformers import TapasTokenizer, TFTapasForQuestionAnswering\n>>> import pandas as pd\n\n>>> model_name = \"google/tapas-base-finetuned-wtq\"\n>>> model = TFTapasForQuestionAnswering.from_pretrained(model_name)\n>>> tokenizer = TapasTokenizer.from_pretrained(model_name)\n\n>>> data = {\"Actors\": [\"Brad Pitt\", \"Leonardo Di Caprio\", \"George Clooney\"], \"Number of movies\": [\"87\", \"53\", \"69\"]}\n>>> queries = [\n...     \"What is the name of the first actor?\",\n...     \"How many movies has George Clooney played in?\",\n...     \"What is the total number of movies?\",\n... ]\n>>> table = pd.DataFrame.from_dict(data)\n>>> inputs = tokenizer(table=table, queries=queries, padding=\"max_length\", return_tensors=\"tf\")\n>>> outputs = model(**inputs)\n>>> predicted_answer_coordinates, predicted_aggregation_indices = tokenizer.convert_logits_to_predictions(\n...     inputs, outputs.logits, outputs.logits_aggregation\n... )\n\n>>> # let's print out the results:\n>>> id2aggregation = {0: \"NONE\", 1: \"SUM\", 2: \"AVERAGE\", 3: \"COUNT\"}\n>>> aggregation_predictions_string = [id2aggregation[x] for x in predicted_aggregation_indices]\n\n>>> answers = []\n>>> for coordinates in predicted_answer_coordinates:\n...     if len(coordinates) == 1:\n...         # only a single cell:\n...         answers.append(table.iat[coordinates[0]])\n...     else:\n...         # multiple cells\n...         cell_values = []\n...         for coordinate in coordinates:\n...             cell_values.append(table.iat[coordinate])\n...         answers.append(\", \".join(cell_values))\n\n>>> display(table)\n>>> print(\"\")\n>>> for query, answer, predicted_agg in zip(queries, answers, aggregation_predictions_string):\n...     print(query)\n...     if predicted_agg == \"NONE\":\n...         print(\"Predicted answer: \" + answer)\n...     else:\n...         print(\"Predicted answer: \" + predicted_agg + \" > \" + answer)\nWhat is the name of the first actor?\nPredicted answer: Brad Pitt\nHow many movies has George Clooney played in?\nPredicted answer: COUNT > 69\nWhat is the total number of movies?\nPredicted answer: SUM > 87, 53, 69\n```\n</tf>\n</frameworkcontent>\n\nIn case of a conversational set-up, then each table-question pair must be provided **sequentially** to the model, such that the `prev_labels` token types can be overwritten by the predicted `labels` of the previous table-question pair."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: d03b2662-0b5c-45bb-9a36-a9c6394e905c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tapas/", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "HF_Transformers", "_node_content": "{\"id_\": \"d03b2662-0b5c-45bb-9a36-a9c6394e905c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6f8b7462-3307-4ed8-a23a-6d7f8d36fa39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"02f9d1e611bfdcdddc25ffc55f689dfee098357342da90a329bd3b562cf383cc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"166bdabb-6f35-4b8a-9729-6500b0ef9959\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tapas/\", \"title\": \"TAPAS\", \"tokens\": 8355, \"retrieve_doc\": false, \"source\": \"HF_Transformers\"}, \"hash\": \"cb794490f6be4d87cd5d35f7b96f15aba31a2e5eac89bbca8ba7d3cbedb1e669\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 34503, \"end_char_idx\": 36359, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "ref_doc_id": "6f8b7462-3307-4ed8-a23a-6d7f8d36fa39", "chroma:document": "Predicted answer: Brad Pitt\nHow many movies has George Clooney played in?\nPredicted answer: COUNT > 69\nWhat is the total number of movies?\nPredicted answer: SUM > 87, 53, 69\n```\n</tf>\n</frameworkcontent>\n\nIn case of a conversational set-up, then each table-question pair must be provided **sequentially** to the model, such that the `prev_labels` token types can be overwritten by the predicted `labels` of the previous table-question pair. Again, more info can be found in [this notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/TAPAS/Fine_tuning_TapasForQuestionAnswering_on_SQA.ipynb) (for PyTorch) and [this notebook](https://github.com/kamalkraj/Tapas-Tutorial/blob/master/TAPAS/Fine_tuning_TapasForQuestionAnswering_on_SQA.ipynb) (for TensorFlow).\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n\n## TAPAS specific outputs\n[[autodoc]] models.tapas.modeling_tapas.TableQuestionAnsweringOutput\n\n## TapasConfig\n[[autodoc]] TapasConfig\n\n## TapasTokenizer\n[[autodoc]] TapasTokenizer\n    - __call__\n    - convert_logits_to_predictions\n    - save_vocabulary\n\n<frameworkcontent>\n<pt>\n\n## TapasModel\n[[autodoc]] TapasModel\n    - forward\n    \n## TapasForMaskedLM\n[[autodoc]] TapasForMaskedLM\n    - forward\n\n## TapasForSequenceClassification\n[[autodoc]] TapasForSequenceClassification\n    - forward\n    \n## TapasForQuestionAnswering\n[[autodoc]] TapasForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFTapasModel\n[[autodoc]] TFTapasModel\n    - call\n    \n## TFTapasForMaskedLM\n[[autodoc]] TFTapasForMaskedLM\n    - call\n\n## TFTapasForSequenceClassification\n[[autodoc]] TFTapasForSequenceClassification\n    - call\n    \n## TFTapasForQuestionAnswering\n[[autodoc]] TFTapasForQuestionAnswering\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: b8b8c104-7c03-475d-bb3f-533977358dc3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/lxmert/", "title": "LXMERT", "tokens": 1183, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"b8b8c104-7c03-475d-bb3f-533977358dc3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/lxmert/\", \"title\": \"LXMERT\", \"tokens\": 1183, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"75334940-e0a6-4785-a632-f3fb7af3ea65\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/lxmert/\", \"title\": \"LXMERT\", \"tokens\": 1183, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"d60eff1fe03be5e3f2a4db59c9e2275754572a54c9cf286e9f9e96f143e650cd\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"360e306b-a0cb-4852-8703-53addc9d10f1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e564a0391f04e2e1af5b3fcec19d1145d103f78db3cebc46031ee9100fda3cb6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3312, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "75334940-e0a6-4785-a632-f3fb7af3ea65", "doc_id": "75334940-e0a6-4785-a632-f3fb7af3ea65", "ref_doc_id": "75334940-e0a6-4785-a632-f3fb7af3ea65", "chroma:document": "# LXMERT\n\n## Overview\n\nThe LXMERT model was proposed in [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://arxiv.org/abs/1908.07490) by Hao Tan & Mohit Bansal. It is a series of bidirectional transformer encoders\n(one for the vision modality, one for the language modality, and then one to fuse both modalities) pretrained using a\ncombination of masked language modeling, visual-language text alignment, ROI-feature regression, masked\nvisual-attribute modeling, masked visual-object modeling, and visual-question answering objectives. The pretraining\nconsists of multiple multi-modal datasets: MSCOCO, Visual-Genome + Visual-Genome Question Answering, VQA 2.0, and GQA.\n\nThe abstract from the paper is the following:\n\n*Vision-and-language reasoning requires an understanding of visual concepts, language semantics, and, most importantly,\nthe alignment and relationships between these two modalities. We thus propose the LXMERT (Learning Cross-Modality\nEncoder Representations from Transformers) framework to learn these vision-and-language connections. In LXMERT, we\nbuild a large-scale Transformer model that consists of three encoders: an object relationship encoder, a language\nencoder, and a cross-modality encoder. Next, to endow our model with the capability of connecting vision and language\nsemantics, we pre-train the model with large amounts of image-and-sentence pairs, via five diverse representative\npretraining tasks: masked language modeling, masked object prediction (feature regression and label classification),\ncross-modality matching, and image question answering. These tasks help in learning both intra-modality and\ncross-modality relationships. After fine-tuning from our pretrained parameters, our model achieves the state-of-the-art\nresults on two visual question answering datasets (i.e., VQA and GQA). We also show the generalizability of our\npretrained cross-modality model by adapting it to a challenging visual-reasoning task, NLVR, and improve the previous\nbest result by 22% absolute (54% to 76%). Lastly, we demonstrate detailed ablation studies to prove that both our novel\nmodel components and pretraining strategies significantly contribute to our strong results; and also present several\nattention visualizations for the different encoders*\n\nThis model was contributed by [eltoto1219](https://huggingface.co/eltoto1219). The original code can be found [here](https://github.com/airsplay/lxmert).\n\n## Usage tips\n\n- Bounding boxes are not necessary to be used in the visual feature embeddings, any kind of visual-spacial features\n  will work.\n- Both the language hidden states and the visual hidden states that LXMERT outputs are passed through the\n  cross-modality layer, so they contain information from both modalities. To access a modality that only attends to\n  itself, select the vision/language hidden states from the first input in the tuple.\n- The bidirectional cross-modality encoder attention only returns attention values when the language modality is used\n  as the input and the vision modality is used as the context vector. Further, while the cross-modality encoder\n  contains self-attention for each respective modality and cross-attention, only the cross attention is returned and\n  both self attention outputs are disregarded."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 360e306b-a0cb-4852-8703-53addc9d10f1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/lxmert/", "title": "LXMERT", "tokens": 1183, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"360e306b-a0cb-4852-8703-53addc9d10f1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/lxmert/\", \"title\": \"LXMERT\", \"tokens\": 1183, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"75334940-e0a6-4785-a632-f3fb7af3ea65\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/lxmert/\", \"title\": \"LXMERT\", \"tokens\": 1183, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"d60eff1fe03be5e3f2a4db59c9e2275754572a54c9cf286e9f9e96f143e650cd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b8b8c104-7c03-475d-bb3f-533977358dc3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/lxmert/\", \"title\": \"LXMERT\", \"tokens\": 1183, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"c10e173073d1b82cedf334178912e9b6b33083dd002cc4ac818b7d0d580b17b9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1618, \"end_char_idx\": 4322, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "75334940-e0a6-4785-a632-f3fb7af3ea65", "doc_id": "75334940-e0a6-4785-a632-f3fb7af3ea65", "ref_doc_id": "75334940-e0a6-4785-a632-f3fb7af3ea65", "chroma:document": "These tasks help in learning both intra-modality and\ncross-modality relationships. After fine-tuning from our pretrained parameters, our model achieves the state-of-the-art\nresults on two visual question answering datasets (i.e., VQA and GQA). We also show the generalizability of our\npretrained cross-modality model by adapting it to a challenging visual-reasoning task, NLVR, and improve the previous\nbest result by 22% absolute (54% to 76%). Lastly, we demonstrate detailed ablation studies to prove that both our novel\nmodel components and pretraining strategies significantly contribute to our strong results; and also present several\nattention visualizations for the different encoders*\n\nThis model was contributed by [eltoto1219](https://huggingface.co/eltoto1219). The original code can be found [here](https://github.com/airsplay/lxmert).\n\n## Usage tips\n\n- Bounding boxes are not necessary to be used in the visual feature embeddings, any kind of visual-spacial features\n  will work.\n- Both the language hidden states and the visual hidden states that LXMERT outputs are passed through the\n  cross-modality layer, so they contain information from both modalities. To access a modality that only attends to\n  itself, select the vision/language hidden states from the first input in the tuple.\n- The bidirectional cross-modality encoder attention only returns attention values when the language modality is used\n  as the input and the vision modality is used as the context vector. Further, while the cross-modality encoder\n  contains self-attention for each respective modality and cross-attention, only the cross attention is returned and\n  both self attention outputs are disregarded.\n\n## Resources\n\n- [Question answering task guide](../tasks/question_answering)\n\n## LxmertConfig\n\n[[autodoc]] LxmertConfig\n\n## LxmertTokenizer\n\n[[autodoc]] LxmertTokenizer\n\n## LxmertTokenizerFast\n\n[[autodoc]] LxmertTokenizerFast\n\n## Lxmert specific outputs\n\n[[autodoc]] models.lxmert.modeling_lxmert.LxmertModelOutput\n\n[[autodoc]] models.lxmert.modeling_lxmert.LxmertForPreTrainingOutput\n\n[[autodoc]] models.lxmert.modeling_lxmert.LxmertForQuestionAnsweringOutput\n\n[[autodoc]] models.lxmert.modeling_tf_lxmert.TFLxmertModelOutput\n\n[[autodoc]] models.lxmert.modeling_tf_lxmert.TFLxmertForPreTrainingOutput\n\n<frameworkcontent>\n<pt>\n\n## LxmertModel\n\n[[autodoc]] LxmertModel\n    - forward\n\n## LxmertForPreTraining\n\n[[autodoc]] LxmertForPreTraining\n    - forward\n\n## LxmertForQuestionAnswering\n\n[[autodoc]] LxmertForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFLxmertModel\n\n[[autodoc]] TFLxmertModel\n    - call\n\n## TFLxmertForPreTraining\n\n[[autodoc]] TFLxmertForPreTraining\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 666e7b53-903f-4ee6-80fc-6ab97a193adf
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/dpt/", "title": "DPT", "tokens": 1085, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"666e7b53-903f-4ee6-80fc-6ab97a193adf\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dpt/\", \"title\": \"DPT\", \"tokens\": 1085, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9932d2d7-cf98-497e-910d-fc6aaac5af96\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dpt/\", \"title\": \"DPT\", \"tokens\": 1085, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"15e13a42a10d611db2ba5bb3731e97f78e10780d0b3de6d7cd3e2eee63aff647\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4b104236-6899-4f50-b08a-f343a30ec1a1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ef621a0e76d8e3fd4223cefff8da8a75cce207dd7f7212953fc6e8e55d23a5c0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3416, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9932d2d7-cf98-497e-910d-fc6aaac5af96", "doc_id": "9932d2d7-cf98-497e-910d-fc6aaac5af96", "ref_doc_id": "9932d2d7-cf98-497e-910d-fc6aaac5af96", "chroma:document": "# DPT\n\n## Overview\n\nThe DPT model was proposed in [Vision Transformers for Dense Prediction](https://arxiv.org/abs/2103.13413) by Ren\u00e9 Ranftl, Alexey Bochkovskiy, Vladlen Koltun.\nDPT is a model that leverages the [Vision Transformer (ViT)](vit) as backbone for dense prediction tasks like semantic segmentation and depth estimation.\n\nThe abstract from the paper is the following:\n\n*We introduce dense vision transformers, an architecture that leverages vision transformers in place of convolutional networks as a backbone for dense prediction tasks. We assemble tokens from various stages of the vision transformer into image-like representations at various resolutions and progressively combine them into full-resolution predictions using a convolutional decoder. The transformer backbone processes representations at a constant and relatively high resolution and has a global receptive field at every stage. These properties allow the dense vision transformer to provide finer-grained and more globally coherent predictions when compared to fully-convolutional networks. Our experiments show that this architecture yields substantial improvements on dense prediction tasks, especially when a large amount of training data is available. For monocular depth estimation, we observe an improvement of up to 28% in relative performance when compared to a state-of-the-art fully-convolutional network. When applied to semantic segmentation, dense vision transformers set a new state of the art on ADE20K with 49.02% mIoU. We further show that the architecture can be fine-tuned on smaller datasets such as NYUv2, KITTI, and Pascal Context where it also sets the new state of the art.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/dpt_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> DPT architecture. Taken from the <a href=\"https://arxiv.org/abs/2103.13413\" target=\"_blank\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/isl-org/DPT).\n\n## Usage tips\n\nDPT is compatible with the [`AutoBackbone`] class. This allows to use the DPT framework with various computer vision backbones available in the library, such as [`VitDetBackbone`] or [`Dinov2Backbone`]. One can create it as follows:\n\n```python\nfrom transformers import Dinov2Config, DPTConfig, DPTForDepthEstimation\n\n# initialize with a Transformer-based backbone such as DINOv2\n# in that case, we also specify `reshape_hidden_states=False` to get feature maps of shape (batch_size, num_channels, height, width)\nbackbone_config = Dinov2Config.from_pretrained(\"facebook/dinov2-base\", out_features=[\"stage1\", \"stage2\", \"stage3\", \"stage4\"], reshape_hidden_states=False)\n\nconfig = DPTConfig(backbone_config=backbone_config)\nmodel = DPTForDepthEstimation(config=config)\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with DPT.\n\n- Demo notebooks for [`DPTForDepthEstimation`] can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DPT).\n\n- [Semantic segmentation task guide](../tasks/semantic_segmentation)\n- [Monocular depth estimation task guide](../tasks/monocular_depth_estimation)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it!"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 4b104236-6899-4f50-b08a-f343a30ec1a1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/dpt/", "title": "DPT", "tokens": 1085, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"4b104236-6899-4f50-b08a-f343a30ec1a1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dpt/\", \"title\": \"DPT\", \"tokens\": 1085, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9932d2d7-cf98-497e-910d-fc6aaac5af96\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dpt/\", \"title\": \"DPT\", \"tokens\": 1085, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"15e13a42a10d611db2ba5bb3731e97f78e10780d0b3de6d7cd3e2eee63aff647\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"666e7b53-903f-4ee6-80fc-6ab97a193adf\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dpt/\", \"title\": \"DPT\", \"tokens\": 1085, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"db9da9ec8fb19410145ff98c1ccf01b4a682a8c2e687b46a10964bb19dfc2c7b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1941, \"end_char_idx\": 3985, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9932d2d7-cf98-497e-910d-fc6aaac5af96", "doc_id": "9932d2d7-cf98-497e-910d-fc6aaac5af96", "ref_doc_id": "9932d2d7-cf98-497e-910d-fc6aaac5af96", "chroma:document": "</small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/isl-org/DPT).\n\n## Usage tips\n\nDPT is compatible with the [`AutoBackbone`] class. This allows to use the DPT framework with various computer vision backbones available in the library, such as [`VitDetBackbone`] or [`Dinov2Backbone`]. One can create it as follows:\n\n```python\nfrom transformers import Dinov2Config, DPTConfig, DPTForDepthEstimation\n\n# initialize with a Transformer-based backbone such as DINOv2\n# in that case, we also specify `reshape_hidden_states=False` to get feature maps of shape (batch_size, num_channels, height, width)\nbackbone_config = Dinov2Config.from_pretrained(\"facebook/dinov2-base\", out_features=[\"stage1\", \"stage2\", \"stage3\", \"stage4\"], reshape_hidden_states=False)\n\nconfig = DPTConfig(backbone_config=backbone_config)\nmodel = DPTForDepthEstimation(config=config)\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with DPT.\n\n- Demo notebooks for [`DPTForDepthEstimation`] can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DPT).\n\n- [Semantic segmentation task guide](../tasks/semantic_segmentation)\n- [Monocular depth estimation task guide](../tasks/monocular_depth_estimation)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## DPTConfig\n\n[[autodoc]] DPTConfig\n\n## DPTFeatureExtractor\n\n[[autodoc]] DPTFeatureExtractor\n    - __call__\n    - post_process_semantic_segmentation\n\n## DPTImageProcessor\n\n[[autodoc]] DPTImageProcessor\n    - preprocess\n    - post_process_semantic_segmentation\n\n## DPTModel\n\n[[autodoc]] DPTModel\n    - forward\n\n## DPTForDepthEstimation\n\n[[autodoc]] DPTForDepthEstimation\n    - forward\n\n## DPTForSemanticSegmentation\n\n[[autodoc]] DPTForSemanticSegmentation\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 83e92c16-4ca1-4b8d-92a0-cc93ece3c014
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/canine/", "title": "CANINE", "tokens": 1446, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"83e92c16-4ca1-4b8d-92a0-cc93ece3c014\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/canine/\", \"title\": \"CANINE\", \"tokens\": 1446, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"eea2c1e9-e5b8-4f10-995d-dedeb0003409\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/canine/\", \"title\": \"CANINE\", \"tokens\": 1446, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"38ac959d80ed49c782610673e9a18abb667edef7a520d17cc49a250c83066566\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6fa9a60b-d196-4d0a-a137-f7fcbee779ad\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b1d5a869de85b830a05286ec1e0ae4567c892fefbab81eece9fb43d3f749df7c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3319, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "eea2c1e9-e5b8-4f10-995d-dedeb0003409", "doc_id": "eea2c1e9-e5b8-4f10-995d-dedeb0003409", "ref_doc_id": "eea2c1e9-e5b8-4f10-995d-dedeb0003409", "chroma:document": "# CANINE\n\n## Overview\n\nThe CANINE model was proposed in [CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language\nRepresentation](https://arxiv.org/abs/2103.06874) by Jonathan H. Clark, Dan Garrette, Iulia Turc, John Wieting. It's\namong the first papers that trains a Transformer without using an explicit tokenization step (such as Byte Pair\nEncoding (BPE), WordPiece or SentencePiece). Instead, the model is trained directly at a Unicode character-level.\nTraining at a character-level inevitably comes with a longer sequence length, which CANINE solves with an efficient\ndownsampling strategy, before applying a deep Transformer encoder.\n\nThe abstract from the paper is the following:\n\n*Pipelined NLP systems have largely been superseded by end-to-end neural modeling, yet nearly all commonly-used models\nstill require an explicit tokenization step. While recent tokenization approaches based on data-derived subword\nlexicons are less brittle than manually engineered tokenizers, these techniques are not equally suited to all\nlanguages, and the use of any fixed vocabulary may limit a model's ability to adapt. In this paper, we present CANINE,\na neural encoder that operates directly on character sequences, without explicit tokenization or vocabulary, and a\npre-training strategy that operates either directly on characters or optionally uses subwords as a soft inductive bias.\nTo use its finer-grained input effectively and efficiently, CANINE combines downsampling, which reduces the input\nsequence length, with a deep transformer stack, which encodes context. CANINE outperforms a comparable mBERT model by\n2.8 F1 on TyDi QA, a challenging multilingual benchmark, despite having 28% fewer model parameters.*\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/google-research/language/tree/master/language/canine).\n\n## Usage tips\n\n- CANINE uses no less than 3 Transformer encoders internally: 2 \"shallow\" encoders (which only consist of a single\n  layer) and 1 \"deep\" encoder (which is a regular BERT encoder). First, a \"shallow\" encoder is used to contextualize\n  the character embeddings, using local attention. Next, after downsampling, a \"deep\" encoder is applied. Finally,\n  after upsampling, a \"shallow\" encoder is used to create the final character embeddings. Details regarding up- and\n  downsampling can be found in the paper.\n- CANINE uses a max sequence length of 2048 characters by default. One can use [`CanineTokenizer`]\n  to prepare text for the model.\n- Classification can be done by placing a linear layer on top of the final hidden state of the special [CLS] token\n  (which has a predefined Unicode code point). For token classification tasks however, the downsampled sequence of\n  tokens needs to be upsampled again to match the length of the original character sequence (which is 2048). The\n  details for this can be found in the paper.\n\nModel checkpoints:\n\n  - [google/canine-c](https://huggingface.co/google/canine-c): Pre-trained with autoregressive character loss,\n    12-layer, 768-hidden, 12-heads, 121M parameters (size ~500 MB).\n  - [google/canine-s](https://huggingface.co/google/canine-s): Pre-trained with subword loss, 12-layer,\n    768-hidden, 12-heads, 121M parameters (size ~500 MB)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 6fa9a60b-d196-4d0a-a137-f7fcbee779ad
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/canine/", "title": "CANINE", "tokens": 1446, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"6fa9a60b-d196-4d0a-a137-f7fcbee779ad\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/canine/\", \"title\": \"CANINE\", \"tokens\": 1446, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"eea2c1e9-e5b8-4f10-995d-dedeb0003409\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/canine/\", \"title\": \"CANINE\", \"tokens\": 1446, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"38ac959d80ed49c782610673e9a18abb667edef7a520d17cc49a250c83066566\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"83e92c16-4ca1-4b8d-92a0-cc93ece3c014\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/canine/\", \"title\": \"CANINE\", \"tokens\": 1446, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"1653a323432806ef2c97f26f6eb65380a30b9645a28fce98bfb735b9819e8f89\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3322, \"end_char_idx\": 5566, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "eea2c1e9-e5b8-4f10-995d-dedeb0003409", "doc_id": "eea2c1e9-e5b8-4f10-995d-dedeb0003409", "ref_doc_id": "eea2c1e9-e5b8-4f10-995d-dedeb0003409", "chroma:document": "## Usage example\n\nCANINE works on raw characters, so it can be used **without a tokenizer**:\n\n```python\n>>> from transformers import CanineModel\n>>> import torch\n\n>>> model = CanineModel.from_pretrained(\"google/canine-c\")  # model pre-trained with autoregressive character loss\n\n>>> text = \"hello world\"\n>>> # use Python's built-in ord() function to turn each character into its unicode code point id\n>>> input_ids = torch.tensor([[ord(char) for char in text]])\n\n>>> outputs = model(input_ids)  # forward pass\n>>> pooled_output = outputs.pooler_output\n>>> sequence_output = outputs.last_hidden_state\n```\n\nFor batched inference and training, it is however recommended to make use of the tokenizer (to pad/truncate all\nsequences to the same length):\n\n```python\n>>> from transformers import CanineTokenizer, CanineModel\n\n>>> model = CanineModel.from_pretrained(\"google/canine-c\")\n>>> tokenizer = CanineTokenizer.from_pretrained(\"google/canine-c\")\n\n>>> inputs = [\"Life is like a box of chocolates.\", \"You never know what you gonna get.\"]\n>>> encoding = tokenizer(inputs, padding=\"longest\", truncation=True, return_tensors=\"pt\")\n\n>>> outputs = model(**encoding)  # forward pass\n>>> pooled_output = outputs.pooler_output\n>>> sequence_output = outputs.last_hidden_state\n```\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## CanineConfig\n\n[[autodoc]] CanineConfig\n\n## CanineTokenizer\n\n[[autodoc]] CanineTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n\n## CANINE specific outputs\n\n[[autodoc]] models.canine.modeling_canine.CanineModelOutputWithPooling\n\n## CanineModel\n\n[[autodoc]] CanineModel\n    - forward\n\n## CanineForSequenceClassification\n\n[[autodoc]] CanineForSequenceClassification\n    - forward\n\n## CanineForMultipleChoice\n\n[[autodoc]] CanineForMultipleChoice\n    - forward\n\n## CanineForTokenClassification\n\n[[autodoc]] CanineForTokenClassification\n    - forward\n\n## CanineForQuestionAnswering\n\n[[autodoc]] CanineForQuestionAnswering\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: e56edf55-9767-4a2e-a905-a573adee8d9e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vipllava/", "title": "VipLlava", "tokens": 1159, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"e56edf55-9767-4a2e-a905-a573adee8d9e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vipllava/\", \"title\": \"VipLlava\", \"tokens\": 1159, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"084adc52-26fc-4e45-bef7-4ac9e2d1a1f8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vipllava/\", \"title\": \"VipLlava\", \"tokens\": 1159, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"adb44c45f24cb582a184f0418965cd7b199aa52b61657044e6f08e969e941780\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e0a6035f-c2c0-4df1-806f-7e18d67f75dd\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"93be05c5d0cf367ff6ed445162b83a659557faa32e35e1cf703f9b6d6bf38f38\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1708, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "084adc52-26fc-4e45-bef7-4ac9e2d1a1f8", "doc_id": "084adc52-26fc-4e45-bef7-4ac9e2d1a1f8", "ref_doc_id": "084adc52-26fc-4e45-bef7-4ac9e2d1a1f8", "chroma:document": "# VipLlava\n\n## Overview\n\nThe VipLlava model was proposed in [Making Large Multimodal Models Understand Arbitrary Visual Prompts](https://arxiv.org/abs/2312.00784) by Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P. Meyer, Yuning Chai, Dennis Park, Yong Jae Lee.\n\nVipLlava enhances the training protocol of Llava by marking images and interact with the model using natural cues like a \"red bounding box\" or \"pointed arrow\" during training.\n\nThe abstract from the paper is the following:\n\n*While existing large vision-language multimodal models focus on whole image understanding, there is a prominent gap in achieving region-specific comprehension. Current approaches that use textual coordinates or spatial encodings often fail to provide a user-friendly interface for visual prompting. To address this challenge, we introduce a novel multimodal model capable of decoding arbitrary visual prompts. This allows users to intuitively mark images and interact with the model using natural cues like a \"red bounding box\" or \"pointed arrow\". Our simple design directly overlays visual markers onto the RGB image, eliminating the need for complex region encodings, yet achieves state-of-the-art performance on region-understanding tasks like Visual7W, PointQA, and Visual Commonsense Reasoning benchmark. Furthermore, we present ViP-Bench, a comprehensive benchmark to assess the capability of models in understanding visual prompts across multiple dimensions, enabling future research in this domain. Code, data, and model are publicly available.*\n\nThe original code can be found [here](https://github.com/mu-cai/ViP-LLaVA).\n\nThis model was contributed by [Younes Belkada](https://huggingface.co/ybelkada)"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: e0a6035f-c2c0-4df1-806f-7e18d67f75dd
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vipllava/", "title": "VipLlava", "tokens": 1159, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"e0a6035f-c2c0-4df1-806f-7e18d67f75dd\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vipllava/\", \"title\": \"VipLlava\", \"tokens\": 1159, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"084adc52-26fc-4e45-bef7-4ac9e2d1a1f8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vipllava/\", \"title\": \"VipLlava\", \"tokens\": 1159, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"adb44c45f24cb582a184f0418965cd7b199aa52b61657044e6f08e969e941780\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e56edf55-9767-4a2e-a905-a573adee8d9e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vipllava/\", \"title\": \"VipLlava\", \"tokens\": 1159, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"1e00835c648e67d61770712b151734001fcf25cbe61b3ff3ded6ec2da25ed259\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7daea850-e1e0-4fc4-aafd-28e0cad4d37a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"98ee1174bd6aac28765f57e81130931612ca885aaa5c686caabeb6a339670a2f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 4336, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "084adc52-26fc-4e45-bef7-4ac9e2d1a1f8", "doc_id": "084adc52-26fc-4e45-bef7-4ac9e2d1a1f8", "ref_doc_id": "084adc52-26fc-4e45-bef7-4ac9e2d1a1f8", "chroma:document": "# VipLlava\n\n## Overview\n\nThe VipLlava model was proposed in [Making Large Multimodal Models Understand Arbitrary Visual Prompts](https://arxiv.org/abs/2312.00784) by Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P. Meyer, Yuning Chai, Dennis Park, Yong Jae Lee.\n\nVipLlava enhances the training protocol of Llava by marking images and interact with the model using natural cues like a \"red bounding box\" or \"pointed arrow\" during training.\n\nThe abstract from the paper is the following:\n\n*While existing large vision-language multimodal models focus on whole image understanding, there is a prominent gap in achieving region-specific comprehension. Current approaches that use textual coordinates or spatial encodings often fail to provide a user-friendly interface for visual prompting. To address this challenge, we introduce a novel multimodal model capable of decoding arbitrary visual prompts. This allows users to intuitively mark images and interact with the model using natural cues like a \"red bounding box\" or \"pointed arrow\". Our simple design directly overlays visual markers onto the RGB image, eliminating the need for complex region encodings, yet achieves state-of-the-art performance on region-understanding tasks like Visual7W, PointQA, and Visual Commonsense Reasoning benchmark. Furthermore, we present ViP-Bench, a comprehensive benchmark to assess the capability of models in understanding visual prompts across multiple dimensions, enabling future research in this domain. Code, data, and model are publicly available.*\n\nThe original code can be found [here](https://github.com/mu-cai/ViP-LLaVA).\n\nThis model was contributed by [Younes Belkada](https://huggingface.co/ybelkada)\n\n\n## Usage tips:\n\n- The architecture is similar than llava architecture except that the multi-modal projector takes a set of concatenated vision hidden states and has an additional layernorm layer on that module.\n\n- We advise users to use `padding_side=\"left\"` when computing batched generation as it leads to more accurate results. Simply make sure to call `processor.tokenizer.padding_side = \"left\"` before generating.\n\n- Note the model has not been explicitly trained to process multiple images in the same prompt, although this is technically possible, you may experience inaccurate results.\n\n- For better results, we recommend users to use the processor's `apply_chat_template()` method to format your prompt correctly. For that you need to construct a conversation history, passing in a plain string will not format your prompt. Each message in the conversation history for chat templates is a dictionary with keys \"role\" and \"content\". The \"content\" should be a list of dictionaries, for \"text\" and \"image\" modalities, as follows:\n\n```python\nfrom transformers import AutoProcessor\n\nprocessor = AutoProcessor.from_pretrained(\"llava-hf/vip-llava-7b-hf\")\n\nconversation = [\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image\"},\n            {\"type\": \"text\", \"text\": \"What\u2019s shown in this image?\"},\n         ,\n    },\n    {\n        \"role\": \"assistant\",\n        \"content\": [{\"type\": \"text\", \"text\": \"This image shows a red stop sign.\"},]\n    },\n    {\n\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"Describe the image in more details.\"},\n        ],\n    },\n]\n\ntext_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)\n\n# Note that the template simply formats your prompt, you still have to tokenize it and obtain pixel values for your images\nprint(text_prompt)\n>>> \"###Human: <image>\\nWhat\u2019s shown in this image?###Assistant: This image shows a red stop sign.###Human: Describe the image in more details.###Assistant:\"\n```\n\n- If you want to construct a chat prompt yourself, below is a list of prompt formats accepted by VipLLaVa checkpoints:\n```bash\nA chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.###Human: <image>\\n<prompt>###Assistant:\n```\n\nFor multiple turns conversation:\n```bash\nA chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.###Human: <image>\\n<prompt1>###Assistant: <answer1>###Human: <prompt2>###Assistant:\n```"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 7daea850-e1e0-4fc4-aafd-28e0cad4d37a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vipllava/", "title": "VipLlava", "tokens": 1159, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"7daea850-e1e0-4fc4-aafd-28e0cad4d37a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vipllava/\", \"title\": \"VipLlava\", \"tokens\": 1159, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"084adc52-26fc-4e45-bef7-4ac9e2d1a1f8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vipllava/\", \"title\": \"VipLlava\", \"tokens\": 1159, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"adb44c45f24cb582a184f0418965cd7b199aa52b61657044e6f08e969e941780\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e0a6035f-c2c0-4df1-806f-7e18d67f75dd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vipllava/\", \"title\": \"VipLlava\", \"tokens\": 1159, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"e2d1b11e4fc5a14aac5710e6807eb4da8e6ec170ac5d29764bbbc8d6e37c0b7f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4339, \"end_char_idx\": 4481, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "084adc52-26fc-4e45-bef7-4ac9e2d1a1f8", "doc_id": "084adc52-26fc-4e45-bef7-4ac9e2d1a1f8", "ref_doc_id": "084adc52-26fc-4e45-bef7-4ac9e2d1a1f8", "chroma:document": "## VipLlavaConfig\n\n[[autodoc]] VipLlavaConfig\n\n## VipLlavaForConditionalGeneration\n\n[[autodoc]] VipLlavaForConditionalGeneration\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 3a95aa98-7eba-4ba6-a911-ada505007a21
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nllb/", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"3a95aa98-7eba-4ba6-a911-ada505007a21\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb/\", \"title\": \"NLLB\", \"tokens\": 2326, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"80990b33-0bc1-45ba-88e6-e9c1763924a7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb/\", \"title\": \"NLLB\", \"tokens\": 2326, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f10b4a84e173de033af1bc882aa152319c05e52285ead65ebf60a93411275b90\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c4697b6f-70fe-436c-b635-80e70d1dd539\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4999b2128a4caac8a2de2ceb2add6f3c9773df08a0b09e6befa9d18d535a379a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2901, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "80990b33-0bc1-45ba-88e6-e9c1763924a7", "doc_id": "80990b33-0bc1-45ba-88e6-e9c1763924a7", "ref_doc_id": "80990b33-0bc1-45ba-88e6-e9c1763924a7", "chroma:document": "# NLLB\n\n## Updated tokenizer behavior \n\n**DISCLAIMER:** The default behaviour for the tokenizer was fixed and thus changed in April 2023.\nThe previous version adds `[self.eos_token_id, self.cur_lang_code]` at the end of the token sequence for both target and source tokenization. This is wrong as the NLLB paper mentions (page 48, 6.1.1. Model Architecture) :\n\n*Note that we prefix the source sequence with the source language, as opposed to the target\nlanguage as previously done in several works (Arivazhagan et al., 2019; Johnson et al.,\n2017). This is primarily because we prioritize optimizing zero-shot performance of our\nmodel on any pair of 200 languages at a minor cost to supervised performance.*\n\nPrevious behaviour:\n\n```python\n>>> from transformers import NllbTokenizer\n\n>>> tokenizer = NllbTokenizer.from_pretrained(\"facebook/nllb-200-distilled-600M\")\n>>> tokenizer(\"How was your day?\").input_ids\n[13374, 1398, 4260, 4039, 248130, 2, 256047]\n\n>>> # 2: '</s>'\n>>> # 256047 : 'eng_Latn'\n```\nNew behaviour\n\n```python\n>>> from transformers import NllbTokenizer\n\n>>> tokenizer = NllbTokenizer.from_pretrained(\"facebook/nllb-200-distilled-600M\")\n>>> tokenizer(\"How was your day?\").input_ids\n[256047, 13374, 1398, 4260, 4039, 248130, 2]\n ```\n\nEnabling the old behaviour can be done as follows:\n```python\n>>> from transformers import NllbTokenizer\n\n>>> tokenizer = NllbTokenizer.from_pretrained(\"facebook/nllb-200-distilled-600M\", legacy_behaviour=True)\n```\n\nFor more details, feel free to check the linked [PR](https://github.com/huggingface/transformers/pull/22313) and [Issue](https://github.com/huggingface/transformers/issues/19943).\n\n## Overview\n\nThe NLLB model was presented in [No Language Left Behind: Scaling Human-Centered Machine Translation](https://arxiv.org/abs/2207.04672) by Marta R. Costa-juss\u00e0, James Cross, Onur \u00c7elebi,\nMaha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula,\nLoic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews,\nNecip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzm\u00e1n, Philipp Koehn, Alexandre Mourachko, Christophe Ropers,\nSafiyyah Saleem, Holger Schwenk, and Jeff Wang.\n\nThe abstract of the paper is the following:\n\n*Driven by the goal of eradicating language barriers on a global scale, machine translation has solidified itself as a key focus of artificial intelligence research today.\nHowever, such efforts have coalesced around a small subset of languages, leaving behind the vast majority of mostly low-resource languages. What does it take to break the\n200 language barrier while ensuring safe, high quality results, all while keeping ethical considerations in mind?"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: c4697b6f-70fe-436c-b635-80e70d1dd539
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nllb/", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"c4697b6f-70fe-436c-b635-80e70d1dd539\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb/\", \"title\": \"NLLB\", \"tokens\": 2326, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"80990b33-0bc1-45ba-88e6-e9c1763924a7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb/\", \"title\": \"NLLB\", \"tokens\": 2326, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f10b4a84e173de033af1bc882aa152319c05e52285ead65ebf60a93411275b90\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3a95aa98-7eba-4ba6-a911-ada505007a21\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb/\", \"title\": \"NLLB\", \"tokens\": 2326, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"291bac79bf64b4bf7021099eb5740f59ce50de8299f9a5f6e09f79f36ea3e9fd\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7b1fca17-ed9a-4cdb-8ff3-e81d66a9b864\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f33381fbfb3156308d83d230c7eff209467053e762dfd06347dc418cae4c7aba\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1645, \"end_char_idx\": 4778, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "80990b33-0bc1-45ba-88e6-e9c1763924a7", "doc_id": "80990b33-0bc1-45ba-88e6-e9c1763924a7", "ref_doc_id": "80990b33-0bc1-45ba-88e6-e9c1763924a7", "chroma:document": "## Overview\n\nThe NLLB model was presented in [No Language Left Behind: Scaling Human-Centered Machine Translation](https://arxiv.org/abs/2207.04672) by Marta R. Costa-juss\u00e0, James Cross, Onur \u00c7elebi,\nMaha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula,\nLoic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews,\nNecip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzm\u00e1n, Philipp Koehn, Alexandre Mourachko, Christophe Ropers,\nSafiyyah Saleem, Holger Schwenk, and Jeff Wang.\n\nThe abstract of the paper is the following:\n\n*Driven by the goal of eradicating language barriers on a global scale, machine translation has solidified itself as a key focus of artificial intelligence research today.\nHowever, such efforts have coalesced around a small subset of languages, leaving behind the vast majority of mostly low-resource languages. What does it take to break the\n200 language barrier while ensuring safe, high quality results, all while keeping ethical considerations in mind? In No Language Left Behind, we took on this challenge by\nfirst contextualizing the need for low-resource language translation support through exploratory interviews with native speakers. Then, we created datasets and models aimed\nat narrowing the performance gap between low and high-resource languages. More specifically, we developed a conditional compute model based on Sparsely Gated Mixture of\nExperts that is trained on data obtained with novel and effective data mining techniques tailored for low-resource languages. We propose multiple architectural and training\nimprovements to counteract overfitting while training on thousands of tasks. Critically, we evaluated the performance of over 40,000 different translation directions using\na human-translated benchmark, Flores-200, and combined human evaluation with a novel toxicity benchmark covering all languages in Flores-200 to assess translation safety.\nOur model achieves an improvement of 44% BLEU relative to the previous state-of-the-art, laying important groundwork towards realizing a universal translation system.*\n\nThis implementation contains the dense models available on release.\n\n**The sparse model NLLB-MoE (Mixture of Expert) is now available! More details [here](nllb-moe)**\n\nThis model was contributed by [Lysandre](https://huggingface.co/lysandre). The authors' code can be found [here](https://github.com/facebookresearch/fairseq/tree/nllb).\n\n## Generating with NLLB\n\nWhile generating the target text set the `forced_bos_token_id` to the target language id. The following\nexample shows how to translate English to French using the *facebook/nllb-200-distilled-600M* model.\n\nNote that we're using the BCP-47 code for French `fra_Latn`. See [here](https://github.com/facebookresearch/flores/blob/main/flores200/README.md#languages-in-flores-200)\nfor the list of all BCP-47 in the Flores 200 dataset."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 7b1fca17-ed9a-4cdb-8ff3-e81d66a9b864
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nllb/", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"7b1fca17-ed9a-4cdb-8ff3-e81d66a9b864\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb/\", \"title\": \"NLLB\", \"tokens\": 2326, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"80990b33-0bc1-45ba-88e6-e9c1763924a7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb/\", \"title\": \"NLLB\", \"tokens\": 2326, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f10b4a84e173de033af1bc882aa152319c05e52285ead65ebf60a93411275b90\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c4697b6f-70fe-436c-b635-80e70d1dd539\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb/\", \"title\": \"NLLB\", \"tokens\": 2326, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"d2b8499983fd2042072eee743969bc668f2bbd44115a9ed4ca7c6d39679c988d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ebb9b373-33c1-4059-80a9-deab1b4bf9e8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4df3401664828a832c781cd7faf867ac0158deb91063508ade66a5edd6ca4a08\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3089, \"end_char_idx\": 5742, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "80990b33-0bc1-45ba-88e6-e9c1763924a7", "doc_id": "80990b33-0bc1-45ba-88e6-e9c1763924a7", "ref_doc_id": "80990b33-0bc1-45ba-88e6-e9c1763924a7", "chroma:document": "Then, we created datasets and models aimed\nat narrowing the performance gap between low and high-resource languages. More specifically, we developed a conditional compute model based on Sparsely Gated Mixture of\nExperts that is trained on data obtained with novel and effective data mining techniques tailored for low-resource languages. We propose multiple architectural and training\nimprovements to counteract overfitting while training on thousands of tasks. Critically, we evaluated the performance of over 40,000 different translation directions using\na human-translated benchmark, Flores-200, and combined human evaluation with a novel toxicity benchmark covering all languages in Flores-200 to assess translation safety.\nOur model achieves an improvement of 44% BLEU relative to the previous state-of-the-art, laying important groundwork towards realizing a universal translation system.*\n\nThis implementation contains the dense models available on release.\n\n**The sparse model NLLB-MoE (Mixture of Expert) is now available! More details [here](nllb-moe)**\n\nThis model was contributed by [Lysandre](https://huggingface.co/lysandre). The authors' code can be found [here](https://github.com/facebookresearch/fairseq/tree/nllb).\n\n## Generating with NLLB\n\nWhile generating the target text set the `forced_bos_token_id` to the target language id. The following\nexample shows how to translate English to French using the *facebook/nllb-200-distilled-600M* model.\n\nNote that we're using the BCP-47 code for French `fra_Latn`. See [here](https://github.com/facebookresearch/flores/blob/main/flores200/README.md#languages-in-flores-200)\nfor the list of all BCP-47 in the Flores 200 dataset.\n\n```python\n>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"facebook/nllb-200-distilled-600M\")\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"facebook/nllb-200-distilled-600M\")\n\n>>> article = \"UN Chief says there is no military solution in Syria\"\n>>> inputs = tokenizer(article, return_tensors=\"pt\")\n\n>>> translated_tokens = model.generate(\n...     **inputs, forced_bos_token_id=tokenizer.lang_code_to_id[\"fra_Latn\"], max_length=30\n... )\n>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]\nLe chef de l'ONU dit qu'il n'y a pas de solution militaire en Syrie\n```\n\n### Generating from any other language than English\n\nEnglish (`eng_Latn`) is set as the default language from which to translate. In order to specify that you'd like to translate from a different language,\nyou should specify the BCP-47 code in the `src_lang` keyword argument of the tokenizer initialization."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: ebb9b373-33c1-4059-80a9-deab1b4bf9e8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nllb/", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"ebb9b373-33c1-4059-80a9-deab1b4bf9e8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb/\", \"title\": \"NLLB\", \"tokens\": 2326, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"80990b33-0bc1-45ba-88e6-e9c1763924a7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb/\", \"title\": \"NLLB\", \"tokens\": 2326, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f10b4a84e173de033af1bc882aa152319c05e52285ead65ebf60a93411275b90\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7b1fca17-ed9a-4cdb-8ff3-e81d66a9b864\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb/\", \"title\": \"NLLB\", \"tokens\": 2326, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"2815c4bd6edc5d9f22981e147b7953f2c272d8e26d918d171b353bf1efdfe473\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9a1f22b9-0b60-40c7-8c0b-af43c9310859\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"af21da8b814c9356ad6a44eedb0cf51f33238f8a9b613e88fd911e4ec1a3473f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4229, \"end_char_idx\": 7157, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "80990b33-0bc1-45ba-88e6-e9c1763924a7", "doc_id": "80990b33-0bc1-45ba-88e6-e9c1763924a7", "ref_doc_id": "80990b33-0bc1-45ba-88e6-e9c1763924a7", "chroma:document": "The authors' code can be found [here](https://github.com/facebookresearch/fairseq/tree/nllb).\n\n## Generating with NLLB\n\nWhile generating the target text set the `forced_bos_token_id` to the target language id. The following\nexample shows how to translate English to French using the *facebook/nllb-200-distilled-600M* model.\n\nNote that we're using the BCP-47 code for French `fra_Latn`. See [here](https://github.com/facebookresearch/flores/blob/main/flores200/README.md#languages-in-flores-200)\nfor the list of all BCP-47 in the Flores 200 dataset.\n\n```python\n>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"facebook/nllb-200-distilled-600M\")\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"facebook/nllb-200-distilled-600M\")\n\n>>> article = \"UN Chief says there is no military solution in Syria\"\n>>> inputs = tokenizer(article, return_tensors=\"pt\")\n\n>>> translated_tokens = model.generate(\n...     **inputs, forced_bos_token_id=tokenizer.lang_code_to_id[\"fra_Latn\"], max_length=30\n... )\n>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]\nLe chef de l'ONU dit qu'il n'y a pas de solution militaire en Syrie\n```\n\n### Generating from any other language than English\n\nEnglish (`eng_Latn`) is set as the default language from which to translate. In order to specify that you'd like to translate from a different language,\nyou should specify the BCP-47 code in the `src_lang` keyword argument of the tokenizer initialization.\n\nSee example below for a translation from romanian to german:\n\n```py\n>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\n...     \"facebook/nllb-200-distilled-600M\", token=True, src_lang=\"ron_Latn\"\n... )\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"facebook/nllb-200-distilled-600M\", token=True)\n\n>>> article = \"\u015eeful ONU spune c\u0103 nu exist\u0103 o solu\u0163ie militar\u0103 \u00een Siria\"\n>>> inputs = tokenizer(article, return_tensors=\"pt\")\n\n>>> translated_tokens = model.generate(\n...     **inputs, forced_bos_token_id=tokenizer.lang_code_to_id[\"deu_Latn\"], max_length=30\n... )\n>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]\nUN-Chef sagt, es gibt keine milit\u00e4rische L\u00f6sung in Syrien\n```\n\n## Resources\n\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n## NllbTokenizer\n\n[[autodoc]] NllbTokenizer\n    - build_inputs_with_special_tokens\n\n## NllbTokenizerFast\n\n[[autodoc]] NllbTokenizerFast\n\n## Using Flash Attention 2\n\nFlash Attention 2 is a faster, optimized version of the attention scores computation which relies on `cuda` kernels.\n\n### Installation \n\nFirst, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 9a1f22b9-0b60-40c7-8c0b-af43c9310859
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nllb/", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"9a1f22b9-0b60-40c7-8c0b-af43c9310859\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb/\", \"title\": \"NLLB\", \"tokens\": 2326, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"80990b33-0bc1-45ba-88e6-e9c1763924a7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb/\", \"title\": \"NLLB\", \"tokens\": 2326, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f10b4a84e173de033af1bc882aa152319c05e52285ead65ebf60a93411275b90\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ebb9b373-33c1-4059-80a9-deab1b4bf9e8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb/\", \"title\": \"NLLB\", \"tokens\": 2326, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5ae28f5b0e817dca5dc0e694c45e17bc026a3779f07e4422f1ed48bc9f30f8fb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d9b95f20-72fc-4f8b-bb3b-e4b0ceac036c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0460f70ad625e204b926b12d4e75d67ae849511bbb326a0921b91d7e5b21d869\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5564, \"end_char_idx\": 8592, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "80990b33-0bc1-45ba-88e6-e9c1763924a7", "doc_id": "80990b33-0bc1-45ba-88e6-e9c1763924a7", "ref_doc_id": "80990b33-0bc1-45ba-88e6-e9c1763924a7", "chroma:document": "In order to specify that you'd like to translate from a different language,\nyou should specify the BCP-47 code in the `src_lang` keyword argument of the tokenizer initialization.\n\nSee example below for a translation from romanian to german:\n\n```py\n>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\n...     \"facebook/nllb-200-distilled-600M\", token=True, src_lang=\"ron_Latn\"\n... )\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"facebook/nllb-200-distilled-600M\", token=True)\n\n>>> article = \"\u015eeful ONU spune c\u0103 nu exist\u0103 o solu\u0163ie militar\u0103 \u00een Siria\"\n>>> inputs = tokenizer(article, return_tensors=\"pt\")\n\n>>> translated_tokens = model.generate(\n...     **inputs, forced_bos_token_id=tokenizer.lang_code_to_id[\"deu_Latn\"], max_length=30\n... )\n>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]\nUN-Chef sagt, es gibt keine milit\u00e4rische L\u00f6sung in Syrien\n```\n\n## Resources\n\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n## NllbTokenizer\n\n[[autodoc]] NllbTokenizer\n    - build_inputs_with_special_tokens\n\n## NllbTokenizerFast\n\n[[autodoc]] NllbTokenizerFast\n\n## Using Flash Attention 2\n\nFlash Attention 2 is a faster, optimized version of the attention scores computation which relies on `cuda` kernels.\n\n### Installation \n\nFirst, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features).\n\nNext, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\n### Usage\n\nTo load a model using Flash Attention 2, we can pass the argument `attn_implementation=\"flash_attention_2\"` to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). You can use either `torch.float16` or `torch.bfloat16` precision.\n\n```python\n>>> import torch\n>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"facebook/nllb-200-distilled-600M\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\").to(\"cuda\").eval()\n>>> tokenizer = AutoTokenizer.from_pretrained(\"facebook/nllb-200-distilled-600M\")\n\n>>> article = \"\u015eeful ONU spune c\u0103 nu exist\u0103 o solu\u0163ie militar\u0103 \u00een Siria\"\n>>> inputs = tokenizer(article, return_tensors=\"pt\").to(\"cuda\")\n\n>>> translated_tokens = model.generate(\n...     **inputs, forced_bos_token_id=tokenizer.lang_code_to_id[\"deu_Latn\"], max_length=30\n... )\n>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]\n\"UN-Chef sagt, es gibt keine milit\u00e4rische L\u00f6sung in Syrien\"\n```\n\n### Expected speedups\n\nBelow is an expected speedup diagram that compares pure inference time between the native implementation and the Flash Attention 2."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: d9b95f20-72fc-4f8b-bb3b-e4b0ceac036c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nllb/", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"d9b95f20-72fc-4f8b-bb3b-e4b0ceac036c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb/\", \"title\": \"NLLB\", \"tokens\": 2326, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"80990b33-0bc1-45ba-88e6-e9c1763924a7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb/\", \"title\": \"NLLB\", \"tokens\": 2326, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f10b4a84e173de033af1bc882aa152319c05e52285ead65ebf60a93411275b90\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9a1f22b9-0b60-40c7-8c0b-af43c9310859\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nllb/\", \"title\": \"NLLB\", \"tokens\": 2326, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"cc253598bcfa0a80d7ad9ba170883f15d67bfcc3d89f4a5b3eac4e81e0911480\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7001, \"end_char_idx\": 8740, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "80990b33-0bc1-45ba-88e6-e9c1763924a7", "doc_id": "80990b33-0bc1-45ba-88e6-e9c1763924a7", "ref_doc_id": "80990b33-0bc1-45ba-88e6-e9c1763924a7", "chroma:document": "The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features).\n\nNext, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\n### Usage\n\nTo load a model using Flash Attention 2, we can pass the argument `attn_implementation=\"flash_attention_2\"` to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). You can use either `torch.float16` or `torch.bfloat16` precision.\n\n```python\n>>> import torch\n>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"facebook/nllb-200-distilled-600M\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\").to(\"cuda\").eval()\n>>> tokenizer = AutoTokenizer.from_pretrained(\"facebook/nllb-200-distilled-600M\")\n\n>>> article = \"\u015eeful ONU spune c\u0103 nu exist\u0103 o solu\u0163ie militar\u0103 \u00een Siria\"\n>>> inputs = tokenizer(article, return_tensors=\"pt\").to(\"cuda\")\n\n>>> translated_tokens = model.generate(\n...     **inputs, forced_bos_token_id=tokenizer.lang_code_to_id[\"deu_Latn\"], max_length=30\n... )\n>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]\n\"UN-Chef sagt, es gibt keine milit\u00e4rische L\u00f6sung in Syrien\"\n```\n\n### Expected speedups\n\nBelow is an expected speedup diagram that compares pure inference time between the native implementation and the Flash Attention 2.\n\n<div style=\"text-align: center\">\n<img src=\"https://huggingface.co/datasets/visheratin/documentation-images/resolve/main/nllb-speedup.webp\">\n</div>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 90ce42f9-63bf-443c-af45-034153ac1cc1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/deit/", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"90ce42f9-63bf-443c-af45-034153ac1cc1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deit/\", \"title\": \"DeiT\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3f2e7520-58ef-400b-8066-19d395e0e68a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deit/\", \"title\": \"DeiT\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5d6c0ea10d02e22d10ab426d76ed23fc69a4fdefdf5ae500789435fe946758d2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b3e76fb9-ad89-43eb-8291-ab407ce275ac\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f39a0e5529512db7b56c4c5e69f1d32a58ab45bc145f1178d7a17b01bf978729\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3321, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3f2e7520-58ef-400b-8066-19d395e0e68a", "doc_id": "3f2e7520-58ef-400b-8066-19d395e0e68a", "ref_doc_id": "3f2e7520-58ef-400b-8066-19d395e0e68a", "chroma:document": "# DeiT\n\n## Overview\n\nThe DeiT model was proposed in [Training data-efficient image transformers & distillation through attention](https://arxiv.org/abs/2012.12877) by Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre\nSablayrolles, Herv\u00e9 J\u00e9gou. The [Vision Transformer (ViT)](vit) introduced in [Dosovitskiy et al., 2020](https://arxiv.org/abs/2010.11929) has shown that one can match or even outperform existing convolutional neural\nnetworks using a Transformer encoder (BERT-like). However, the ViT models introduced in that paper required training on\nexpensive infrastructure for multiple weeks, using external data. DeiT (data-efficient image transformers) are more\nefficiently trained transformers for image classification, requiring far less data and far less computing resources\ncompared to the original ViT models.\n\nThe abstract from the paper is the following:\n\n*Recently, neural networks purely based on attention were shown to address image understanding tasks such as image\nclassification. However, these visual transformers are pre-trained with hundreds of millions of images using an\nexpensive infrastructure, thereby limiting their adoption. In this work, we produce a competitive convolution-free\ntransformer by training on Imagenet only. We train them on a single computer in less than 3 days. Our reference vision\ntransformer (86M parameters) achieves top-1 accuracy of 83.1% (single-crop evaluation) on ImageNet with no external\ndata. More importantly, we introduce a teacher-student strategy specific to transformers. It relies on a distillation\ntoken ensuring that the student learns from the teacher through attention. We show the interest of this token-based\ndistillation, especially when using a convnet as a teacher. This leads us to report results competitive with convnets\nfor both Imagenet (where we obtain up to 85.2% accuracy) and when transferring to other tasks. We share our code and\nmodels.*\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The TensorFlow version of this model was added by [amyeroberts](https://huggingface.co/amyeroberts).\n\n## Usage tips\n\n- Compared to ViT, DeiT models use a so-called distillation token to effectively learn from a teacher (which, in the\n  DeiT paper, is a ResNet like-model). The distillation token is learned through backpropagation, by interacting with\n  the class ([CLS]) and patch tokens through the self-attention layers.\n- There are 2 ways to fine-tune distilled models, either (1) in a classic way, by only placing a prediction head on top\n  of the final hidden state of the class token and not using the distillation signal, or (2) by placing both a\n  prediction head on top of the class token and on top of the distillation token. In that case, the [CLS] prediction\n  head is trained using regular cross-entropy between the prediction of the head and the ground-truth label, while the\n  distillation prediction head is trained using hard distillation (cross-entropy between the prediction of the\n  distillation head and the label predicted by the teacher). At inference time, one takes the average prediction\n  between both heads as final prediction. (2) is also called \"fine-tuning with distillation\", because one relies on a\n  teacher that has already been fine-tuned on the downstream dataset."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: b3e76fb9-ad89-43eb-8291-ab407ce275ac
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/deit/", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"b3e76fb9-ad89-43eb-8291-ab407ce275ac\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deit/\", \"title\": \"DeiT\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3f2e7520-58ef-400b-8066-19d395e0e68a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deit/\", \"title\": \"DeiT\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5d6c0ea10d02e22d10ab426d76ed23fc69a4fdefdf5ae500789435fe946758d2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"90ce42f9-63bf-443c-af45-034153ac1cc1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deit/\", \"title\": \"DeiT\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"3718ee596024cd284eb36c874f5d95d3fcc9347ee1aadc14760c7df81a92b505\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8be7bcb9-ac1d-47cc-884c-66980e498442\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8900adc0324e373391c6583eb4712e15c41d06e2414db67c22b253bde705edbe\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1662, \"end_char_idx\": 4842, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3f2e7520-58ef-400b-8066-19d395e0e68a", "doc_id": "3f2e7520-58ef-400b-8066-19d395e0e68a", "ref_doc_id": "3f2e7520-58ef-400b-8066-19d395e0e68a", "chroma:document": "We show the interest of this token-based\ndistillation, especially when using a convnet as a teacher. This leads us to report results competitive with convnets\nfor both Imagenet (where we obtain up to 85.2% accuracy) and when transferring to other tasks. We share our code and\nmodels.*\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The TensorFlow version of this model was added by [amyeroberts](https://huggingface.co/amyeroberts).\n\n## Usage tips\n\n- Compared to ViT, DeiT models use a so-called distillation token to effectively learn from a teacher (which, in the\n  DeiT paper, is a ResNet like-model). The distillation token is learned through backpropagation, by interacting with\n  the class ([CLS]) and patch tokens through the self-attention layers.\n- There are 2 ways to fine-tune distilled models, either (1) in a classic way, by only placing a prediction head on top\n  of the final hidden state of the class token and not using the distillation signal, or (2) by placing both a\n  prediction head on top of the class token and on top of the distillation token. In that case, the [CLS] prediction\n  head is trained using regular cross-entropy between the prediction of the head and the ground-truth label, while the\n  distillation prediction head is trained using hard distillation (cross-entropy between the prediction of the\n  distillation head and the label predicted by the teacher). At inference time, one takes the average prediction\n  between both heads as final prediction. (2) is also called \"fine-tuning with distillation\", because one relies on a\n  teacher that has already been fine-tuned on the downstream dataset. In terms of models, (1) corresponds to\n  [`DeiTForImageClassification`] and (2) corresponds to\n  [`DeiTForImageClassificationWithTeacher`].\n- Note that the authors also did try soft distillation for (2) (in which case the distillation prediction head is\n  trained using KL divergence to match the softmax output of the teacher), but hard distillation gave the best results.\n- All released checkpoints were pre-trained and fine-tuned on ImageNet-1k only. No external data was used. This is in\n  contrast with the original ViT model, which used external data like the JFT-300M dataset/Imagenet-21k for\n  pre-training.\n- The authors of DeiT also released more efficiently trained ViT models, which you can directly plug into\n  [`ViTModel`] or [`ViTForImageClassification`]. Techniques like data\n  augmentation, optimization, and regularization were used in order to simulate training on a much larger dataset\n  (while only using ImageNet-1k for pre-training). There are 4 variants available (in 3 different sizes):\n  *facebook/deit-tiny-patch16-224*, *facebook/deit-small-patch16-224*, *facebook/deit-base-patch16-224* and\n  *facebook/deit-base-patch16-384*. Note that one should use [`DeiTImageProcessor`] in order to\n  prepare images for the model.\n\n### Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 8be7bcb9-ac1d-47cc-884c-66980e498442
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/deit/", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"8be7bcb9-ac1d-47cc-884c-66980e498442\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deit/\", \"title\": \"DeiT\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3f2e7520-58ef-400b-8066-19d395e0e68a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deit/\", \"title\": \"DeiT\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5d6c0ea10d02e22d10ab426d76ed23fc69a4fdefdf5ae500789435fe946758d2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b3e76fb9-ad89-43eb-8291-ab407ce275ac\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deit/\", \"title\": \"DeiT\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"44b0ac4a63dd591a8d10a8170c6be760611b3d7be822e05d1854d0ea79583e07\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d00b5127-d9e8-48f0-ae7a-160658c9b88c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"029ca90e3a085074fd8493a422be77182cdb82522184983bf5b3107400b982dd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3322, \"end_char_idx\": 5839, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3f2e7520-58ef-400b-8066-19d395e0e68a", "doc_id": "3f2e7520-58ef-400b-8066-19d395e0e68a", "ref_doc_id": "3f2e7520-58ef-400b-8066-19d395e0e68a", "chroma:document": "In terms of models, (1) corresponds to\n  [`DeiTForImageClassification`] and (2) corresponds to\n  [`DeiTForImageClassificationWithTeacher`].\n- Note that the authors also did try soft distillation for (2) (in which case the distillation prediction head is\n  trained using KL divergence to match the softmax output of the teacher), but hard distillation gave the best results.\n- All released checkpoints were pre-trained and fine-tuned on ImageNet-1k only. No external data was used. This is in\n  contrast with the original ViT model, which used external data like the JFT-300M dataset/Imagenet-21k for\n  pre-training.\n- The authors of DeiT also released more efficiently trained ViT models, which you can directly plug into\n  [`ViTModel`] or [`ViTForImageClassification`]. Techniques like data\n  augmentation, optimization, and regularization were used in order to simulate training on a much larger dataset\n  (while only using ImageNet-1k for pre-training). There are 4 variants available (in 3 different sizes):\n  *facebook/deit-tiny-patch16-224*, *facebook/deit-small-patch16-224*, *facebook/deit-base-patch16-224* and\n  *facebook/deit-base-patch16-384*. Note that one should use [`DeiTImageProcessor`] in order to\n  prepare images for the model.\n\n### Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```\nfrom transformers import DeiTForImageClassification\nmodel = DeiTForImageClassification.from_pretrained(\"facebook/deit-base-distilled-patch16-224\", attn_implementation=\"sdpa\", torch_dtype=torch.float16)\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `facebook/deit-base-distilled-patch16-224` model, we saw the following speedups during inference."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: d00b5127-d9e8-48f0-ae7a-160658c9b88c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/deit/", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"d00b5127-d9e8-48f0-ae7a-160658c9b88c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deit/\", \"title\": \"DeiT\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3f2e7520-58ef-400b-8066-19d395e0e68a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deit/\", \"title\": \"DeiT\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5d6c0ea10d02e22d10ab426d76ed23fc69a4fdefdf5ae500789435fe946758d2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8be7bcb9-ac1d-47cc-884c-66980e498442\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deit/\", \"title\": \"DeiT\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"ff93d70b4174db0d8e83ff6752fab490abb52ecd32a4e6f7138dc7b6812c0ede\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"de287c35-8ab8-4382-b15a-ca1074651f67\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"48d04040b93eee91b93afc81ec5bba08c29cfde39234bf8ac7a8f2bfa36633dc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4478, \"end_char_idx\": 7589, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3f2e7520-58ef-400b-8066-19d395e0e68a", "doc_id": "3f2e7520-58ef-400b-8066-19d395e0e68a", "ref_doc_id": "3f2e7520-58ef-400b-8066-19d395e0e68a", "chroma:document": "Note that one should use [`DeiTImageProcessor`] in order to\n  prepare images for the model.\n\n### Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```\nfrom transformers import DeiTForImageClassification\nmodel = DeiTForImageClassification.from_pretrained(\"facebook/deit-base-distilled-patch16-224\", attn_implementation=\"sdpa\", torch_dtype=torch.float16)\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `facebook/deit-base-distilled-patch16-224` model, we saw the following speedups during inference.\n\n|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |\n|--------------|-------------------------------------------|-------------------------------------------|------------------------------|\n|            1 |                                         8 |                                         6 |                      1.33 |\n|            2 |                                         9 |                                         6 |                      1.5  |\n|            4 |                                         9 |                                         6 |                      1.5  |\n|            8 |                                         8 |                                         6 |                      1.33 |\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with DeiT.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`DeiTForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nBesides that:\n\n- [`DeiTForMaskedImageModeling`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: de287c35-8ab8-4382-b15a-ca1074651f67
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/deit/", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"de287c35-8ab8-4382-b15a-ca1074651f67\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deit/\", \"title\": \"DeiT\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3f2e7520-58ef-400b-8066-19d395e0e68a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deit/\", \"title\": \"DeiT\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5d6c0ea10d02e22d10ab426d76ed23fc69a4fdefdf5ae500789435fe946758d2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d00b5127-d9e8-48f0-ae7a-160658c9b88c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deit/\", \"title\": \"DeiT\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"9edd512e8a73f659268346ade64884c6451a047cccb1aa9ee5d7a476a4c113b3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5658, \"end_char_idx\": 8498, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3f2e7520-58ef-400b-8066-19d395e0e68a", "doc_id": "3f2e7520-58ef-400b-8066-19d395e0e68a", "ref_doc_id": "3f2e7520-58ef-400b-8066-19d395e0e68a", "chroma:document": "On a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `facebook/deit-base-distilled-patch16-224` model, we saw the following speedups during inference.\n\n|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |\n|--------------|-------------------------------------------|-------------------------------------------|------------------------------|\n|            1 |                                         8 |                                         6 |                      1.33 |\n|            2 |                                         9 |                                         6 |                      1.5  |\n|            4 |                                         9 |                                         6 |                      1.5  |\n|            8 |                                         8 |                                         6 |                      1.33 |\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with DeiT.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`DeiTForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nBesides that:\n\n- [`DeiTForMaskedImageModeling`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## DeiTConfig\n\n[[autodoc]] DeiTConfig\n\n## DeiTFeatureExtractor\n\n[[autodoc]] DeiTFeatureExtractor\n    - __call__\n\n## DeiTImageProcessor\n\n[[autodoc]] DeiTImageProcessor\n    - preprocess\n\n<frameworkcontent>\n<pt>\n\n## DeiTModel\n\n[[autodoc]] DeiTModel\n    - forward\n\n## DeiTForMaskedImageModeling\n\n[[autodoc]] DeiTForMaskedImageModeling\n    - forward\n\n## DeiTForImageClassification\n\n[[autodoc]] DeiTForImageClassification\n    - forward\n\n## DeiTForImageClassificationWithTeacher\n\n[[autodoc]] DeiTForImageClassificationWithTeacher\n    - forward\n\n</pt>\n<tf>\n\n## TFDeiTModel\n\n[[autodoc]] TFDeiTModel\n    - call\n\n## TFDeiTForMaskedImageModeling\n\n[[autodoc]] TFDeiTForMaskedImageModeling\n    - call\n\n## TFDeiTForImageClassification\n\n[[autodoc]] TFDeiTForImageClassification\n    - call\n\n## TFDeiTForImageClassificationWithTeacher\n\n[[autodoc]] TFDeiTForImageClassificationWithTeacher\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 9f7796a4-c398-4626-8afe-583b1f975c5b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/pegasus_x/", "title": "PEGASUS-X", "tokens": 630, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"9f7796a4-c398-4626-8afe-583b1f975c5b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pegasus_x/\", \"title\": \"PEGASUS-X\", \"tokens\": 630, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"340991ac-b6e2-4429-99a1-8cd6f0ec7997\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pegasus_x/\", \"title\": \"PEGASUS-X\", \"tokens\": 630, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"57028638737ecd6ec2bf5de8b146e2041f9664f72f89340ab4f43853432488ef\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2134, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "340991ac-b6e2-4429-99a1-8cd6f0ec7997", "doc_id": "340991ac-b6e2-4429-99a1-8cd6f0ec7997", "ref_doc_id": "340991ac-b6e2-4429-99a1-8cd6f0ec7997", "chroma:document": "# PEGASUS-X\n\n## Overview\n\nThe PEGASUS-X model was proposed in [Investigating Efficiently Extending Transformers for Long Input Summarization](https://arxiv.org/abs/2208.04347)  by Jason Phang, Yao Zhao and Peter J. Liu.\n\nPEGASUS-X (PEGASUS eXtended) extends the PEGASUS models for long input summarization through additional long input pretraining and using staggered block-local attention with global tokens in the encoder.\n\nThe abstract from the paper is the following:\n\n*While large pretrained Transformer models have proven highly capable at tackling natural language tasks, handling long sequence inputs continues to be a significant challenge. One such task is long input summarization, where inputs are longer than the maximum input context of most pretrained models. Through an extensive set of experiments, we investigate what model architectural changes and pretraining paradigms can most efficiently adapt a pretrained Transformer for long input summarization. We find that a staggered, block-local Transformer with global encoder tokens strikes a good balance of performance and efficiency, and that an additional pretraining phase on long sequences meaningfully improves downstream summarization performance. Based on our findings, we introduce PEGASUS-X, an extension of the PEGASUS model with additional long input pretraining to handle inputs of up to 16K tokens. PEGASUS-X achieves strong performance on long input summarization tasks comparable with much larger models while adding few additional parameters and not requiring model parallelism to train.*\n\nThis model was contributed by [zphang](https://huggingface.co/zphang). The original code can be found [here](https://github.com/google-research/pegasus).\n\n## Documentation resources\n\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n<Tip>\n\nPEGASUS-X uses the same tokenizer as [PEGASUS](pegasus).\n\n</Tip>\n\n## PegasusXConfig\n\n[[autodoc]] PegasusXConfig\n\n## PegasusXModel\n\n[[autodoc]] PegasusXModel\n    - forward\n\n## PegasusXForConditionalGeneration\n\n[[autodoc]] PegasusXForConditionalGeneration\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 1f1de17d-c3ff-430e-85d4-045636e783d4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert/", "title": "Wav2Vec2-BERT", "tokens": 1431, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"1f1de17d-c3ff-430e-85d4-045636e783d4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert/\", \"title\": \"Wav2Vec2-BERT\", \"tokens\": 1431, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"181c7b59-e927-457b-b651-2febf166e696\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert/\", \"title\": \"Wav2Vec2-BERT\", \"tokens\": 1431, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"af8aaad6301cc0b71ebc91f30fea24855e559d5763a0c5ffd1c73e57cc2928ef\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ec1540ba-1e49-4778-87f5-6d0116c405a7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"65d57d19aec87f1f8f24b08ef79d17237774788559ef0021cc1da7a25f288e53\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3857, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "181c7b59-e927-457b-b651-2febf166e696", "doc_id": "181c7b59-e927-457b-b651-2febf166e696", "ref_doc_id": "181c7b59-e927-457b-b651-2febf166e696", "chroma:document": "# Wav2Vec2-BERT\n\n## Overview\n\nThe Wav2Vec2-BERT model was proposed in [Seamless: Multilingual Expressive and Streaming Speech Translation](https://ai.meta.com/research/publications/seamless-multilingual-expressive-and-streaming-speech-translation/) by the Seamless Communication team from Meta AI.\n\nThis model was pre-trained on 4.5M hours of unlabeled audio data covering more than 143 languages. It requires finetuning to be used for downstream tasks such as Automatic Speech Recognition (ASR), or Audio Classification.\n\nThe official results of the model can be found in Section 3.2.1 of the paper.\n\nThe abstract from the paper is the following:\n\n*Recent advancements in automatic speech translation have dramatically expanded language coverage, improved multimodal capabilities, and enabled a wide range of tasks and functionalities. That said, large-scale automatic speech translation systems today lack key features that help machine-mediated communication feel seamless when compared to human-to-human dialogue. In this work, we introduce a family of models that enable end-to-end expressive and multilingual translations in a streaming fashion. First, we contribute an improved version of the massively multilingual and multimodal SeamlessM4T model\u2014SeamlessM4T v2. This newer model, incorporating an updated UnitY2 framework, was trained on more low-resource language data. The expanded version of SeamlessAlign adds 114,800 hours of automatically aligned data for a total of 76 languages. SeamlessM4T v2 provides the foundation on which our two newest models, SeamlessExpressive and SeamlessStreaming, are initiated. SeamlessExpressive enables translation that preserves vocal styles and prosody. Compared to previous efforts in expressive speech research, our work addresses certain underexplored aspects of prosody, such as speech rate and pauses, while also preserving the style of one\u2019s voice. As for SeamlessStreaming, our model leverages the Efficient Monotonic Multihead Attention (EMMA) mechanism to generate low-latency target translations without waiting for complete source utterances. As the first of its kind, SeamlessStreaming enables simultaneous speech-to-speech/text translation for multiple source and target languages. To understand the performance of these models, we combined novel and modified versions of existing automatic metrics to evaluate prosody, latency, and robustness. For human evaluations, we adapted existing protocols tailored for measuring the most relevant attributes in the preservation of meaning, naturalness, and expressivity. To ensure that our models can be used safely and responsibly, we implemented the first known red-teaming effort for multimodal machine translation, a system for the detection and mitigation of added toxicity, a systematic evaluation of gender bias, and an inaudible localized watermarking mechanism designed to dampen the impact of deepfakes. Consequently, we bring major components from SeamlessExpressive and SeamlessStreaming together to form Seamless, the first publicly available system that unlocks expressive cross-lingual communication in real-time. In sum, Seamless gives us a pivotal look at the technical foundation needed to turn the Universal Speech Translator from a science fiction concept into a real-world technology. Finally, contributions in this work\u2014including models, code, and a watermark detector\u2014are publicly released and accessible at the link below.*\n\nThis model was contributed by [ylacombe](https://huggingface.co/ylacombe). The original code can be found [here](https://github.com/facebookresearch/seamless_communication).\n\n## Usage tips\n\n- Wav2Vec2-BERT follows the same architecture as Wav2Vec2-Conformer, but employs a causal depthwise convolutional layer and uses as input a mel-spectrogram representation of the audio instead of the raw waveform."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: ec1540ba-1e49-4778-87f5-6d0116c405a7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert/", "title": "Wav2Vec2-BERT", "tokens": 1431, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"ec1540ba-1e49-4778-87f5-6d0116c405a7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert/\", \"title\": \"Wav2Vec2-BERT\", \"tokens\": 1431, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"181c7b59-e927-457b-b651-2febf166e696\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert/\", \"title\": \"Wav2Vec2-BERT\", \"tokens\": 1431, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"af8aaad6301cc0b71ebc91f30fea24855e559d5763a0c5ffd1c73e57cc2928ef\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1f1de17d-c3ff-430e-85d4-045636e783d4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert/\", \"title\": \"Wav2Vec2-BERT\", \"tokens\": 1431, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"47bdcb2026f01e66f7e37dfbcddd6cf2e2277d406950bc1d5c64222cce14fbe9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b681480e-aaad-405a-b776-c56de50d5e3f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2e633c2b946c447014c7daa89f3b7aabd29d91ce849309fba975fbcf4b232c52\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1906, \"end_char_idx\": 5121, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "181c7b59-e927-457b-b651-2febf166e696", "doc_id": "181c7b59-e927-457b-b651-2febf166e696", "ref_doc_id": "181c7b59-e927-457b-b651-2febf166e696", "chroma:document": "As for SeamlessStreaming, our model leverages the Efficient Monotonic Multihead Attention (EMMA) mechanism to generate low-latency target translations without waiting for complete source utterances. As the first of its kind, SeamlessStreaming enables simultaneous speech-to-speech/text translation for multiple source and target languages. To understand the performance of these models, we combined novel and modified versions of existing automatic metrics to evaluate prosody, latency, and robustness. For human evaluations, we adapted existing protocols tailored for measuring the most relevant attributes in the preservation of meaning, naturalness, and expressivity. To ensure that our models can be used safely and responsibly, we implemented the first known red-teaming effort for multimodal machine translation, a system for the detection and mitigation of added toxicity, a systematic evaluation of gender bias, and an inaudible localized watermarking mechanism designed to dampen the impact of deepfakes. Consequently, we bring major components from SeamlessExpressive and SeamlessStreaming together to form Seamless, the first publicly available system that unlocks expressive cross-lingual communication in real-time. In sum, Seamless gives us a pivotal look at the technical foundation needed to turn the Universal Speech Translator from a science fiction concept into a real-world technology. Finally, contributions in this work\u2014including models, code, and a watermark detector\u2014are publicly released and accessible at the link below.*\n\nThis model was contributed by [ylacombe](https://huggingface.co/ylacombe). The original code can be found [here](https://github.com/facebookresearch/seamless_communication).\n\n## Usage tips\n\n- Wav2Vec2-BERT follows the same architecture as Wav2Vec2-Conformer, but employs a causal depthwise convolutional layer and uses as input a mel-spectrogram representation of the audio instead of the raw waveform.\n- Wav2Vec2-BERT can use either no relative position embeddings, Shaw-like position embeddings, Transformer-XL-like position embeddings, or\n  rotary position embeddings by setting the correct `config.position_embeddings_type`.\n- Wav2Vec2-BERT also introduces a Conformer-based adapter network instead of a simple convolutional network.\n\n## Resources\n\n<PipelineTag pipeline=\"automatic-speech-recognition\"/>\n\n- [`Wav2Vec2BertForCTC`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/speech-recognition).\n- You can also adapt these notebooks on [how to finetune a speech recognition model in English](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/speech_recognition.ipynb), and [how to finetune a speech recognition model in any language](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multi_lingual_speech_recognition.ipynb).\n\n<PipelineTag pipeline=\"audio-classification\"/>\n\n- [`Wav2Vec2BertForSequenceClassification`] can be used by adapting this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/audio-classification).\n- See also: [Audio classification task guide](../tasks/audio_classification)"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: b681480e-aaad-405a-b776-c56de50d5e3f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert/", "title": "Wav2Vec2-BERT", "tokens": 1431, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"b681480e-aaad-405a-b776-c56de50d5e3f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert/\", \"title\": \"Wav2Vec2-BERT\", \"tokens\": 1431, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"181c7b59-e927-457b-b651-2febf166e696\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert/\", \"title\": \"Wav2Vec2-BERT\", \"tokens\": 1431, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"af8aaad6301cc0b71ebc91f30fea24855e559d5763a0c5ffd1c73e57cc2928ef\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ec1540ba-1e49-4778-87f5-6d0116c405a7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert/\", \"title\": \"Wav2Vec2-BERT\", \"tokens\": 1431, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"fad48bf7ff07a2045629d866814cd22b6774d1ffda9435537ee6e398f2a8c4d6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3452, \"end_char_idx\": 5770, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "181c7b59-e927-457b-b651-2febf166e696", "doc_id": "181c7b59-e927-457b-b651-2febf166e696", "ref_doc_id": "181c7b59-e927-457b-b651-2febf166e696", "chroma:document": "*\n\nThis model was contributed by [ylacombe](https://huggingface.co/ylacombe). The original code can be found [here](https://github.com/facebookresearch/seamless_communication).\n\n## Usage tips\n\n- Wav2Vec2-BERT follows the same architecture as Wav2Vec2-Conformer, but employs a causal depthwise convolutional layer and uses as input a mel-spectrogram representation of the audio instead of the raw waveform.\n- Wav2Vec2-BERT can use either no relative position embeddings, Shaw-like position embeddings, Transformer-XL-like position embeddings, or\n  rotary position embeddings by setting the correct `config.position_embeddings_type`.\n- Wav2Vec2-BERT also introduces a Conformer-based adapter network instead of a simple convolutional network.\n\n## Resources\n\n<PipelineTag pipeline=\"automatic-speech-recognition\"/>\n\n- [`Wav2Vec2BertForCTC`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/speech-recognition).\n- You can also adapt these notebooks on [how to finetune a speech recognition model in English](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/speech_recognition.ipynb), and [how to finetune a speech recognition model in any language](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multi_lingual_speech_recognition.ipynb).\n\n<PipelineTag pipeline=\"audio-classification\"/>\n\n- [`Wav2Vec2BertForSequenceClassification`] can be used by adapting this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/audio-classification).\n- See also: [Audio classification task guide](../tasks/audio_classification)\n\n\n## Wav2Vec2BertConfig\n\n[[autodoc]] Wav2Vec2BertConfig\n\n## Wav2Vec2BertProcessor\n\n[[autodoc]] Wav2Vec2BertProcessor\n    - __call__\n    - pad\n    - from_pretrained\n    - save_pretrained\n    - batch_decode\n    - decode\n\n## Wav2Vec2BertModel\n\n[[autodoc]] Wav2Vec2BertModel\n    - forward\n\n## Wav2Vec2BertForCTC\n\n[[autodoc]] Wav2Vec2BertForCTC\n    - forward\n\n## Wav2Vec2BertForSequenceClassification\n\n[[autodoc]] Wav2Vec2BertForSequenceClassification\n    - forward\n\n## Wav2Vec2BertForAudioFrameClassification\n\n[[autodoc]] Wav2Vec2BertForAudioFrameClassification\n    - forward\n\n## Wav2Vec2BertForXVector\n\n[[autodoc]] Wav2Vec2BertForXVector\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 3281eb4c-4253-4ecc-864f-a7d3df30d9b8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/roformer/", "title": "RoFormer", "tokens": 1190, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"3281eb4c-4253-4ecc-864f-a7d3df30d9b8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roformer/\", \"title\": \"RoFormer\", \"tokens\": 1190, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8c320867-2f5a-4ccc-8ee0-2fb90b121add\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roformer/\", \"title\": \"RoFormer\", \"tokens\": 1190, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8cd4d42d6dc7aa817381d38e8de21b9d50a7a1f2da10c1e4112c1badb3dae05f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"59c91830-13f3-48f2-b173-97824c1d04e2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ffab6bcf96998c282e8a43a358cf36503f958dc381f5c1735009fae9a74a59e8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1700, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8c320867-2f5a-4ccc-8ee0-2fb90b121add", "doc_id": "8c320867-2f5a-4ccc-8ee0-2fb90b121add", "ref_doc_id": "8c320867-2f5a-4ccc-8ee0-2fb90b121add", "chroma:document": "# RoFormer\n\n## Overview\n\nThe RoFormer model was proposed in [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://arxiv.org/pdf/2104.09864v1.pdf) by Jianlin Su and Yu Lu and Shengfeng Pan and Bo Wen and Yunfeng Liu.\n\nThe abstract from the paper is the following:\n\n*Position encoding in transformer architecture provides supervision for dependency modeling between elements at\ndifferent positions in the sequence. We investigate various methods to encode positional information in\ntransformer-based language models and propose a novel implementation named Rotary Position Embedding(RoPE). The\nproposed RoPE encodes absolute positional information with rotation matrix and naturally incorporates explicit relative\nposition dependency in self-attention formulation. Notably, RoPE comes with valuable properties such as flexibility of\nbeing expand to any sequence lengths, decaying inter-token dependency with increasing relative distances, and\ncapability of equipping the linear self-attention with relative position encoding. As a result, the enhanced\ntransformer with rotary position embedding, or RoFormer, achieves superior performance in tasks with long texts. We\nrelease the theoretical analysis along with some preliminary experiment results on Chinese data. The undergoing\nexperiment for English benchmark will soon be updated.*\n\nThis model was contributed by [junnyu](https://huggingface.co/junnyu). The original code can be found [here](https://github.com/ZhuiyiTechnology/roformer).\n\n## Usage tips\nRoFormer is a BERT-like autoencoding model with rotary position embeddings. Rotary position embeddings have shown \nimproved performance on classification tasks with long texts."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 59c91830-13f3-48f2-b173-97824c1d04e2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/roformer/", "title": "RoFormer", "tokens": 1190, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"59c91830-13f3-48f2-b173-97824c1d04e2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roformer/\", \"title\": \"RoFormer\", \"tokens\": 1190, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8c320867-2f5a-4ccc-8ee0-2fb90b121add\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roformer/\", \"title\": \"RoFormer\", \"tokens\": 1190, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8cd4d42d6dc7aa817381d38e8de21b9d50a7a1f2da10c1e4112c1badb3dae05f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3281eb4c-4253-4ecc-864f-a7d3df30d9b8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/roformer/\", \"title\": \"RoFormer\", \"tokens\": 1190, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"9c4529535f2275259572bd8a3825fc57922baa2bcb202ed4a47bec380c7f17b5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 4211, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8c320867-2f5a-4ccc-8ee0-2fb90b121add", "doc_id": "8c320867-2f5a-4ccc-8ee0-2fb90b121add", "ref_doc_id": "8c320867-2f5a-4ccc-8ee0-2fb90b121add", "chroma:document": "# RoFormer\n\n## Overview\n\nThe RoFormer model was proposed in [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://arxiv.org/pdf/2104.09864v1.pdf) by Jianlin Su and Yu Lu and Shengfeng Pan and Bo Wen and Yunfeng Liu.\n\nThe abstract from the paper is the following:\n\n*Position encoding in transformer architecture provides supervision for dependency modeling between elements at\ndifferent positions in the sequence. We investigate various methods to encode positional information in\ntransformer-based language models and propose a novel implementation named Rotary Position Embedding(RoPE). The\nproposed RoPE encodes absolute positional information with rotation matrix and naturally incorporates explicit relative\nposition dependency in self-attention formulation. Notably, RoPE comes with valuable properties such as flexibility of\nbeing expand to any sequence lengths, decaying inter-token dependency with increasing relative distances, and\ncapability of equipping the linear self-attention with relative position encoding. As a result, the enhanced\ntransformer with rotary position embedding, or RoFormer, achieves superior performance in tasks with long texts. We\nrelease the theoretical analysis along with some preliminary experiment results on Chinese data. The undergoing\nexperiment for English benchmark will soon be updated.*\n\nThis model was contributed by [junnyu](https://huggingface.co/junnyu). The original code can be found [here](https://github.com/ZhuiyiTechnology/roformer).\n\n## Usage tips\nRoFormer is a BERT-like autoencoding model with rotary position embeddings. Rotary position embeddings have shown \nimproved performance on classification tasks with long texts.\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## RoFormerConfig\n\n[[autodoc]] RoFormerConfig\n\n## RoFormerTokenizer\n\n[[autodoc]] RoFormerTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## RoFormerTokenizerFast\n\n[[autodoc]] RoFormerTokenizerFast\n    - build_inputs_with_special_tokens\n\n<frameworkcontent>\n<pt>\n\n## RoFormerModel\n\n[[autodoc]] RoFormerModel\n    - forward\n\n## RoFormerForCausalLM\n\n[[autodoc]] RoFormerForCausalLM\n    - forward\n\n## RoFormerForMaskedLM\n\n[[autodoc]] RoFormerForMaskedLM\n    - forward\n\n## RoFormerForSequenceClassification\n\n[[autodoc]] RoFormerForSequenceClassification\n    - forward\n\n## RoFormerForMultipleChoice\n\n[[autodoc]] RoFormerForMultipleChoice\n    - forward\n\n## RoFormerForTokenClassification\n\n[[autodoc]] RoFormerForTokenClassification\n    - forward\n\n## RoFormerForQuestionAnswering\n\n[[autodoc]] RoFormerForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFRoFormerModel\n\n[[autodoc]] TFRoFormerModel\n    - call\n\n## TFRoFormerForMaskedLM\n\n[[autodoc]] TFRoFormerForMaskedLM\n    - call\n\n## TFRoFormerForCausalLM\n\n[[autodoc]] TFRoFormerForCausalLM\n    - call\n\n## TFRoFormerForSequenceClassification\n\n[[autodoc]] TFRoFormerForSequenceClassification\n    - call\n\n## TFRoFormerForMultipleChoice\n\n[[autodoc]] TFRoFormerForMultipleChoice\n    - call\n\n## TFRoFormerForTokenClassification\n\n[[autodoc]] TFRoFormerForTokenClassification\n    - call\n\n## TFRoFormerForQuestionAnswering\n\n[[autodoc]] TFRoFormerForQuestionAnswering\n    - call\n\n</tf>\n<jax>\n\n## FlaxRoFormerModel\n\n[[autodoc]] FlaxRoFormerModel\n    - __call__\n\n## FlaxRoFormerForMaskedLM\n\n[[autodoc]] FlaxRoFormerForMaskedLM\n    - __call__\n\n## FlaxRoFormerForSequenceClassification\n\n[[autodoc]] FlaxRoFormerForSequenceClassification\n    - __call__\n\n## FlaxRoFormerForMultipleChoice\n\n[[autodoc]] FlaxRoFormerForMultipleChoice\n    - __call__\n\n## FlaxRoFormerForTokenClassification\n\n[[autodoc]] FlaxRoFormerForTokenClassification\n    - __call__\n\n## FlaxRoFormerForQuestionAnswering\n\n[[autodoc]] FlaxRoFormerForQuestionAnswering\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: ec5abb8d-ace2-47ff-a5aa-9e1564467af4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/owlvit/", "title": "OWL-ViT", "tokens": 1628, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"ec5abb8d-ace2-47ff-a5aa-9e1564467af4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/owlvit/\", \"title\": \"OWL-ViT\", \"tokens\": 1628, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d175536b-f502-4184-8b61-756468cf19ed\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/owlvit/\", \"title\": \"OWL-ViT\", \"tokens\": 1628, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"ca9f4c025433cd1a153948f3bcfdfbc4466775fe659f7f4bdafc4940766fc991\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c0aeace6-0b73-45d1-9212-ebc65a2c384d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"81e302000b73b7f8ce2ec8be03622b6551b8faebac0a5553ec089d84c0e61ac6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3421, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d175536b-f502-4184-8b61-756468cf19ed", "doc_id": "d175536b-f502-4184-8b61-756468cf19ed", "ref_doc_id": "d175536b-f502-4184-8b61-756468cf19ed", "chroma:document": "# OWL-ViT\n\n## Overview\n\nThe OWL-ViT (short for Vision Transformer for Open-World Localization) was proposed in [Simple Open-Vocabulary Object Detection with Vision Transformers](https://arxiv.org/abs/2205.06230) by Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, Xiao Wang, Xiaohua Zhai, Thomas Kipf, and Neil Houlsby. OWL-ViT is an open-vocabulary object detection network trained on a variety of (image, text) pairs. It can be used to query an image with one or multiple text queries to search for and detect target objects described in text.\n\nThe abstract from the paper is the following:\n\n*Combining simple architectures with large-scale pre-training has led to massive improvements in image classification. For object detection, pre-training and scaling approaches are less well established, especially in the long-tailed and open-vocabulary setting, where training data is relatively scarce. In this paper, we propose a strong recipe for transferring image-text models to open-vocabulary object detection. We use a standard Vision Transformer architecture with minimal modifications, contrastive image-text pre-training, and end-to-end detection fine-tuning. Our analysis of the scaling properties of this setup shows that increasing image-level pre-training and model size yield consistent improvements on the downstream detection task. We provide the adaptation strategies and regularizations needed to attain very strong performance on zero-shot text-conditioned and one-shot image-conditioned object detection. Code and models are available on GitHub.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/owlvit_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> OWL-ViT architecture. Taken from the <a href=\"https://arxiv.org/abs/2205.06230\">original paper</a>. </small>\n\nThis model was contributed by [adirik](https://huggingface.co/adirik). The original code can be found [here](https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit).\n\n## Usage tips\n\nOWL-ViT is a zero-shot text-conditioned object detection model. OWL-ViT uses [CLIP](clip) as its multi-modal backbone, with a ViT-like Transformer to get visual features and a causal language model to get the text features. To use CLIP for detection, OWL-ViT removes the final token pooling layer of the vision model and attaches a lightweight classification and box head to each transformer output token. Open-vocabulary classification is enabled by replacing the fixed classification layer weights with the class-name embeddings obtained from the text model. The authors first train CLIP from scratch and fine-tune it end-to-end with the classification and box heads on standard detection datasets using a bipartite matching loss. One or multiple text queries per image can be used to perform zero-shot text-conditioned object detection.\n\n[`OwlViTImageProcessor`] can be used to resize (or rescale) and normalize images for the model and [`CLIPTokenizer`] is used to encode the text. [`OwlViTProcessor`] wraps [`OwlViTImageProcessor`] and [`CLIPTokenizer`] into a single instance to both encode the text and prepare the images. The following example shows how to perform object detection using [`OwlViTProcessor`] and [`OwlViTForObjectDetection`]."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: c0aeace6-0b73-45d1-9212-ebc65a2c384d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/owlvit/", "title": "OWL-ViT", "tokens": 1628, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"c0aeace6-0b73-45d1-9212-ebc65a2c384d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/owlvit/\", \"title\": \"OWL-ViT\", \"tokens\": 1628, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d175536b-f502-4184-8b61-756468cf19ed\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/owlvit/\", \"title\": \"OWL-ViT\", \"tokens\": 1628, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"ca9f4c025433cd1a153948f3bcfdfbc4466775fe659f7f4bdafc4940766fc991\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ec5abb8d-ace2-47ff-a5aa-9e1564467af4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/owlvit/\", \"title\": \"OWL-ViT\", \"tokens\": 1628, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"dfcae5e0fbda5babbe6e2dae3c603c85c420570f9a8f49b75d983dd25fbe7492\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4202e93e-ae7c-4fb3-90e2-3cba7e90045f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0b1eb6de101be3d103adb3b894974e3eec3d64ad5f84861710a948747eb302af\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1881, \"end_char_idx\": 5186, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d175536b-f502-4184-8b61-756468cf19ed", "doc_id": "d175536b-f502-4184-8b61-756468cf19ed", "ref_doc_id": "d175536b-f502-4184-8b61-756468cf19ed", "chroma:document": "Taken from the <a href=\"https://arxiv.org/abs/2205.06230\">original paper</a>. </small>\n\nThis model was contributed by [adirik](https://huggingface.co/adirik). The original code can be found [here](https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit).\n\n## Usage tips\n\nOWL-ViT is a zero-shot text-conditioned object detection model. OWL-ViT uses [CLIP](clip) as its multi-modal backbone, with a ViT-like Transformer to get visual features and a causal language model to get the text features. To use CLIP for detection, OWL-ViT removes the final token pooling layer of the vision model and attaches a lightweight classification and box head to each transformer output token. Open-vocabulary classification is enabled by replacing the fixed classification layer weights with the class-name embeddings obtained from the text model. The authors first train CLIP from scratch and fine-tune it end-to-end with the classification and box heads on standard detection datasets using a bipartite matching loss. One or multiple text queries per image can be used to perform zero-shot text-conditioned object detection.\n\n[`OwlViTImageProcessor`] can be used to resize (or rescale) and normalize images for the model and [`CLIPTokenizer`] is used to encode the text. [`OwlViTProcessor`] wraps [`OwlViTImageProcessor`] and [`CLIPTokenizer`] into a single instance to both encode the text and prepare the images. The following example shows how to perform object detection using [`OwlViTProcessor`] and [`OwlViTForObjectDetection`].\n\n```python\n>>> import requests\n>>> from PIL import Image\n>>> import torch\n\n>>> from transformers import OwlViTProcessor, OwlViTForObjectDetection\n\n>>> processor = OwlViTProcessor.from_pretrained(\"google/owlvit-base-patch32\")\n>>> model = OwlViTForObjectDetection.from_pretrained(\"google/owlvit-base-patch32\")\n\n>>> url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n>>> texts = [[\"a photo of a cat\", \"a photo of a dog\"]]\n>>> inputs = processor(text=texts, images=image, return_tensors=\"pt\")\n>>> outputs = model(**inputs)\n\n>>> # Target image sizes (height, width) to rescale box predictions [batch_size, 2]\n>>> target_sizes = torch.Tensor([image.size[::-1]])\n>>> # Convert outputs (bounding boxes and class logits) to Pascal VOC format (xmin, ymin, xmax, ymax)\n>>> results = processor.post_process_object_detection(outputs=outputs, target_sizes=target_sizes, threshold=0.1)\n>>> i = 0  # Retrieve predictions for the first image for the corresponding text queries\n>>> text = texts[i]\n>>> boxes, scores, labels = results[i][\"boxes\"], results[i][\"scores\"], results[i][\"labels\"]\n>>> for box, score, label in zip(boxes, scores, labels):\n...     box = [round(i, 2) for i in box.tolist()]\n...     print(f\"Detected {text[label]} with confidence {round(score.item(), 3)} at location {box}\")\nDetected a photo of a cat with confidence 0.707 at location [324.97, 20.44, 640.58, 373.29]\nDetected a photo of a cat with confidence 0.717 at location [1.46, 55.26, 315.55, 472.17]\n```\n\n## Resources\n\nA demo notebook on using OWL-ViT for zero- and one-shot (image-guided) object detection can be found [here](https://github.com/huggingface/notebooks/blob/main/examples/zeroshot_object_detection_with_owlvit.ipynb)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 4202e93e-ae7c-4fb3-90e2-3cba7e90045f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/owlvit/", "title": "OWL-ViT", "tokens": 1628, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"4202e93e-ae7c-4fb3-90e2-3cba7e90045f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/owlvit/\", \"title\": \"OWL-ViT\", \"tokens\": 1628, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d175536b-f502-4184-8b61-756468cf19ed\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/owlvit/\", \"title\": \"OWL-ViT\", \"tokens\": 1628, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"ca9f4c025433cd1a153948f3bcfdfbc4466775fe659f7f4bdafc4940766fc991\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c0aeace6-0b73-45d1-9212-ebc65a2c384d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/owlvit/\", \"title\": \"OWL-ViT\", \"tokens\": 1628, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"4a73b1e28598af42bc8a98ef50b9073e1cb8e9c1ff2d2d78c0fb6b95131900b6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5188, \"end_char_idx\": 6052, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d175536b-f502-4184-8b61-756468cf19ed", "doc_id": "d175536b-f502-4184-8b61-756468cf19ed", "ref_doc_id": "d175536b-f502-4184-8b61-756468cf19ed", "chroma:document": "## OwlViTConfig\n\n[[autodoc]] OwlViTConfig\n    - from_text_vision_configs\n\n## OwlViTTextConfig\n\n[[autodoc]] OwlViTTextConfig\n\n## OwlViTVisionConfig\n\n[[autodoc]] OwlViTVisionConfig\n\n## OwlViTImageProcessor\n\n[[autodoc]] OwlViTImageProcessor\n    - preprocess\n    - post_process_object_detection\n    - post_process_image_guided_detection\n\n## OwlViTFeatureExtractor\n\n[[autodoc]] OwlViTFeatureExtractor\n    - __call__\n    - post_process\n    - post_process_image_guided_detection\n\n## OwlViTProcessor\n\n[[autodoc]] OwlViTProcessor\n\n## OwlViTModel\n\n[[autodoc]] OwlViTModel\n    - forward\n    - get_text_features\n    - get_image_features\n\n## OwlViTTextModel\n\n[[autodoc]] OwlViTTextModel\n    - forward\n\n## OwlViTVisionModel\n\n[[autodoc]] OwlViTVisionModel\n    - forward\n\n## OwlViTForObjectDetection\n\n[[autodoc]] OwlViTForObjectDetection\n    - forward\n    - image_guided_detection"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 9125a206-aad3-46f1-a20a-169f5eaf08dc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/rwkv/", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"9125a206-aad3-46f1-a20a-169f5eaf08dc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rwkv/\", \"title\": \"RWKV\", \"tokens\": 2201, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4e2b5e34-bd3a-451f-a43f-977a8bb12340\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rwkv/\", \"title\": \"RWKV\", \"tokens\": 2201, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"412c46d1ab59e35adb23835e8e610ff0c963d5055620addaaefd06d65fbad063\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"704df197-1868-48b4-987a-f48330a12e91\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f8d1190655f679eb1475dd006586be6f3ff80afc08dcd97ee8696cf225586bda\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3003, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4e2b5e34-bd3a-451f-a43f-977a8bb12340", "doc_id": "4e2b5e34-bd3a-451f-a43f-977a8bb12340", "ref_doc_id": "4e2b5e34-bd3a-451f-a43f-977a8bb12340", "chroma:document": "# RWKV\n\n## Overview\n\nThe RWKV model was proposed in [this repo](https://github.com/BlinkDL/RWKV-LM)\n\nIt suggests a tweak in the traditional Transformer attention to make it linear. This way, the model can be used as recurrent network: passing inputs for timestamp 0 and timestamp 1 together is the same as passing inputs at timestamp 0, then inputs at timestamp 1 along with the state of timestamp 0 (see example below).\n\nThis can be more efficient than a regular Transformer and can deal with sentence of any length (even if the model uses a fixed context length for training).\n\nThis model was contributed by [sgugger](https://huggingface.co/sgugger).\nThe original code can be found [here](https://github.com/BlinkDL/RWKV-LM).\n\n## Usage example\n\n```py\nimport torch\nfrom transformers import AutoTokenizer, RwkvConfig, RwkvModel\n\nmodel = RwkvModel.from_pretrained(\"sgugger/rwkv-430M-pile\")\ntokenizer = AutoTokenizer.from_pretrained(\"sgugger/rwkv-430M-pile\")\n\ninputs = tokenizer(\"This is an example.\", return_tensors=\"pt\")\n# Feed everything to the model\noutputs = model(inputs[\"input_ids\"])\noutput_whole = outputs.last_hidden_state\n\noutputs = model(inputs[\"input_ids\"][:, :2])\noutput_one = outputs.last_hidden_state\n\n# Using the state computed on the first inputs, we will get the same output\noutputs = model(inputs[\"input_ids\"][:, 2:], state=outputs.state)\noutput_two = outputs.last_hidden_state\n\ntorch.allclose(torch.cat([output_one, output_two], dim=1), output_whole, atol=1e-5)\n```\n\nIf you want to make sure the model stops generating when `'\\n\\n'` is detected, we recommend using the following stopping criteria:\n\n```python \nfrom transformers import StoppingCriteria\n\nclass RwkvStoppingCriteria(StoppingCriteria):\n    def __init__(self, eos_sequence = [187,187], eos_token_id = 537):\n        self.eos_sequence = eos_sequence\n        self.eos_token_id = eos_token_id\n\n    def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool:\n        last_2_ids = input_ids[:,-2:].tolist()\n        return self.eos_sequence in last_2_ids\n\n\noutput = model.generate(inputs[\"input_ids\"], max_new_tokens=64, stopping_criteria = [RwkvStoppingCriteria()])\n```\n\n## RwkvConfig\n\n[[autodoc]] RwkvConfig\n\n## RwkvModel\n\n[[autodoc]] RwkvModel\n    - forward\n\n## RwkvLMHeadModel\n\n[[autodoc]] RwkvForCausalLM\n    - forward\n\n## Rwkv attention and the recurrent formulas\n\nIn a traditional auto-regressive Transformer, attention is written as\n\n$$O = \\hbox{softmax}(QK^{T} / \\sqrt{d}) V$$\n\nwith \\\\(Q\\\\), \\\\(K\\\\) and \\\\(V\\\\) are matrices of shape `seq_len x hidden_size` named query, key and value (they are actually bigger matrices with a batch dimension and an attention head dimension but we're only interested in the last two, which is where the matrix product is taken, so for the sake of simplicity we only consider those two). The product \\\\(QK^{T}\\\\) then has shape `seq_len x seq_len` and we can take the matrix product with \\\\(V\\\\) to get the output \\\\(O\\\\) of the same shape as the others."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 704df197-1868-48b4-987a-f48330a12e91
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/rwkv/", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"704df197-1868-48b4-987a-f48330a12e91\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rwkv/\", \"title\": \"RWKV\", \"tokens\": 2201, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4e2b5e34-bd3a-451f-a43f-977a8bb12340\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rwkv/\", \"title\": \"RWKV\", \"tokens\": 2201, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"412c46d1ab59e35adb23835e8e610ff0c963d5055620addaaefd06d65fbad063\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9125a206-aad3-46f1-a20a-169f5eaf08dc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rwkv/\", \"title\": \"RWKV\", \"tokens\": 2201, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"17662df9251c80a146ab443b078dcc281e55d80e1b847d90e3426984872a306a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7fd4cd4e-fbd6-4d9a-9ba9-d75f65b1abfb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"45d12045c247d8d9d94d6b0070a8be57312cf7cc67a0497b44465a1bb9e6efb2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2064, \"end_char_idx\": 3986, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4e2b5e34-bd3a-451f-a43f-977a8bb12340", "doc_id": "4e2b5e34-bd3a-451f-a43f-977a8bb12340", "ref_doc_id": "4e2b5e34-bd3a-451f-a43f-977a8bb12340", "chroma:document": "output = model.generate(inputs[\"input_ids\"], max_new_tokens=64, stopping_criteria = [RwkvStoppingCriteria()])\n```\n\n## RwkvConfig\n\n[[autodoc]] RwkvConfig\n\n## RwkvModel\n\n[[autodoc]] RwkvModel\n    - forward\n\n## RwkvLMHeadModel\n\n[[autodoc]] RwkvForCausalLM\n    - forward\n\n## Rwkv attention and the recurrent formulas\n\nIn a traditional auto-regressive Transformer, attention is written as\n\n$$O = \\hbox{softmax}(QK^{T} / \\sqrt{d}) V$$\n\nwith \\\\(Q\\\\), \\\\(K\\\\) and \\\\(V\\\\) are matrices of shape `seq_len x hidden_size` named query, key and value (they are actually bigger matrices with a batch dimension and an attention head dimension but we're only interested in the last two, which is where the matrix product is taken, so for the sake of simplicity we only consider those two). The product \\\\(QK^{T}\\\\) then has shape `seq_len x seq_len` and we can take the matrix product with \\\\(V\\\\) to get the output \\\\(O\\\\) of the same shape as the others.  \n\nReplacing the softmax by its value gives:\n\n$$O_{i} = \\frac{\\sum_{j=1}^{i} e^{Q_{i} K_{j}^{T} / \\sqrt{d}} V_{j}}{\\sum_{j=1}^{i} e^{Q_{i} K_{j}^{T} / \\sqrt{d}}}$$\n\nNote that the entries in \\\\(QK^{T}\\\\) corresponding to \\\\(j > i\\\\) are masked (the sum stops at j) because the attention is not allowed to look at future tokens (only past ones).\n\nIn comparison, the RWKV attention is given by\n\n$$O_{i} = \\sigma(R_{i}) \\frac{\\sum_{j=1}^{i} e^{W_{i-j} + K_{j}} V_{j}}{\\sum_{j=1}^{i} e^{W_{i-j} + K_{j}}}$$\n\nwhere \\\\(R\\\\) is a new matrix called receptance by the author, \\\\(K\\\\) and \\\\(V\\\\) are still the key and value (\\\\(\\sigma\\\\) here is the sigmoid function). \\\\(W\\\\) is a new vector that represents the position of the token and is given by\n\n$$W_{0} = u \\hbox{  and  } W_{k} = (k-1)w \\hbox{ for } k \\geq 1$$\n\nwith \\\\(u\\\\) and \\\\(w\\\\) learnable parameters called in the code `time_first` and `time_decay` respectively. The numerator and denominator can both be expressed recursively."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 7fd4cd4e-fbd6-4d9a-9ba9-d75f65b1abfb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/rwkv/", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"7fd4cd4e-fbd6-4d9a-9ba9-d75f65b1abfb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rwkv/\", \"title\": \"RWKV\", \"tokens\": 2201, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4e2b5e34-bd3a-451f-a43f-977a8bb12340\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rwkv/\", \"title\": \"RWKV\", \"tokens\": 2201, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"412c46d1ab59e35adb23835e8e610ff0c963d5055620addaaefd06d65fbad063\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"704df197-1868-48b4-987a-f48330a12e91\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rwkv/\", \"title\": \"RWKV\", \"tokens\": 2201, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a25c1b19d8192b3515f982fb5b3df2e8a888f5f2689c1e6443db291f47ea5a61\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1c49511c-6e48-4ea1-ac18-77959517a960\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9a4394f13ca6b85438cddb0baecbdb6c1cf8dcd382f551326a7abc2c410d054a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3007, \"end_char_idx\": 4787, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4e2b5e34-bd3a-451f-a43f-977a8bb12340", "doc_id": "4e2b5e34-bd3a-451f-a43f-977a8bb12340", "ref_doc_id": "4e2b5e34-bd3a-451f-a43f-977a8bb12340", "chroma:document": "Replacing the softmax by its value gives:\n\n$$O_{i} = \\frac{\\sum_{j=1}^{i} e^{Q_{i} K_{j}^{T} / \\sqrt{d}} V_{j}}{\\sum_{j=1}^{i} e^{Q_{i} K_{j}^{T} / \\sqrt{d}}}$$\n\nNote that the entries in \\\\(QK^{T}\\\\) corresponding to \\\\(j > i\\\\) are masked (the sum stops at j) because the attention is not allowed to look at future tokens (only past ones).\n\nIn comparison, the RWKV attention is given by\n\n$$O_{i} = \\sigma(R_{i}) \\frac{\\sum_{j=1}^{i} e^{W_{i-j} + K_{j}} V_{j}}{\\sum_{j=1}^{i} e^{W_{i-j} + K_{j}}}$$\n\nwhere \\\\(R\\\\) is a new matrix called receptance by the author, \\\\(K\\\\) and \\\\(V\\\\) are still the key and value (\\\\(\\sigma\\\\) here is the sigmoid function). \\\\(W\\\\) is a new vector that represents the position of the token and is given by\n\n$$W_{0} = u \\hbox{  and  } W_{k} = (k-1)w \\hbox{ for } k \\geq 1$$\n\nwith \\\\(u\\\\) and \\\\(w\\\\) learnable parameters called in the code `time_first` and `time_decay` respectively. The numerator and denominator can both be expressed recursively. Naming them \\\\(N_{i}\\\\) and \\\\(D_{i}\\\\) we have:\n\n$$N_{i} = e^{u + K_{i}} V_{i} + \\hat{N}_{i} \\hbox{  where  } \\hat{N}_{i} = e^{K_{i-1}} V_{i-1} + e^{w + K_{i-2}} V_{i-2} \\cdots + e^{(i-2)w + K_{1}} V_{1}$$\n\nso \\\\(\\hat{N}_{i}\\\\) (called `numerator_state` in the code) satisfies\n\n$$\\hat{N}_{0} = 0 \\hbox{  and  } \\hat{N}_{j+1} = e^{K_{j}} V_{j} + e^{w} \\hat{N}_{j}$$\n\nand\n\n$$D_{i} = e^{u + K_{i}} + \\hat{D}_{i} \\hbox{  where  } \\hat{D}_{i} = e^{K_{i-1}} + e^{w + K_{i-2}} \\cdots + e^{(i-2)w + K_{1}}$$\n\nso \\\\(\\hat{D}_{i}\\\\) (called `denominator_state` in the code) satisfies\n\n$$\\hat{D}_{0} = 0 \\hbox{  and  } \\hat{D}_{j+1} = e^{K_{j}} + e^{w} \\hat{D}_{j}$$\n\nThe actual recurrent formula used are a tiny bit more complex, as for numerical stability we don't want to compute exponentials of big numbers."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 1c49511c-6e48-4ea1-ac18-77959517a960
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/rwkv/", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"1c49511c-6e48-4ea1-ac18-77959517a960\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rwkv/\", \"title\": \"RWKV\", \"tokens\": 2201, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4e2b5e34-bd3a-451f-a43f-977a8bb12340\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rwkv/\", \"title\": \"RWKV\", \"tokens\": 2201, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"412c46d1ab59e35adb23835e8e610ff0c963d5055620addaaefd06d65fbad063\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7fd4cd4e-fbd6-4d9a-9ba9-d75f65b1abfb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rwkv/\", \"title\": \"RWKV\", \"tokens\": 2201, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"46dcf9083b1ba3fcb59e5b1f0f49fab82b4c37f64bccbbe930912e99c677ec63\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cba26ce7-1da3-4ebe-ad50-ef18322c210c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"131f8a39cbec6c5f7e458881eab01c3e4c0255e2444cc44b2aecc9c64db9c021\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3922, \"end_char_idx\": 5245, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4e2b5e34-bd3a-451f-a43f-977a8bb12340", "doc_id": "4e2b5e34-bd3a-451f-a43f-977a8bb12340", "ref_doc_id": "4e2b5e34-bd3a-451f-a43f-977a8bb12340", "chroma:document": "The numerator and denominator can both be expressed recursively. Naming them \\\\(N_{i}\\\\) and \\\\(D_{i}\\\\) we have:\n\n$$N_{i} = e^{u + K_{i}} V_{i} + \\hat{N}_{i} \\hbox{  where  } \\hat{N}_{i} = e^{K_{i-1}} V_{i-1} + e^{w + K_{i-2}} V_{i-2} \\cdots + e^{(i-2)w + K_{1}} V_{1}$$\n\nso \\\\(\\hat{N}_{i}\\\\) (called `numerator_state` in the code) satisfies\n\n$$\\hat{N}_{0} = 0 \\hbox{  and  } \\hat{N}_{j+1} = e^{K_{j}} V_{j} + e^{w} \\hat{N}_{j}$$\n\nand\n\n$$D_{i} = e^{u + K_{i}} + \\hat{D}_{i} \\hbox{  where  } \\hat{D}_{i} = e^{K_{i-1}} + e^{w + K_{i-2}} \\cdots + e^{(i-2)w + K_{1}}$$\n\nso \\\\(\\hat{D}_{i}\\\\) (called `denominator_state` in the code) satisfies\n\n$$\\hat{D}_{0} = 0 \\hbox{  and  } \\hat{D}_{j+1} = e^{K_{j}} + e^{w} \\hat{D}_{j}$$\n\nThe actual recurrent formula used are a tiny bit more complex, as for numerical stability we don't want to compute exponentials of big numbers. Usually the softmax is not computed as is, but the exponential of the maximum term is divided of the numerator and denominator:\n\n$$\\frac{e^{x_{i}}}{\\sum_{j=1}^{n} e^{x_{j}}} = \\frac{e^{x_{i} - M}}{\\sum_{j=1}^{n} e^{x_{j} - M}}$$\n\nwith \\\\(M\\\\) the maximum of all \\\\(x_{j}\\\\). So here on top of saving the numerator state (\\\\(\\hat{N}\\\\)) and the denominator state (\\\\(\\hat{D}\\\\)) we also keep track of the maximum of all terms encountered in the exponentials."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: cba26ce7-1da3-4ebe-ad50-ef18322c210c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/rwkv/", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"cba26ce7-1da3-4ebe-ad50-ef18322c210c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rwkv/\", \"title\": \"RWKV\", \"tokens\": 2201, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4e2b5e34-bd3a-451f-a43f-977a8bb12340\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rwkv/\", \"title\": \"RWKV\", \"tokens\": 2201, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"412c46d1ab59e35adb23835e8e610ff0c963d5055620addaaefd06d65fbad063\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1c49511c-6e48-4ea1-ac18-77959517a960\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rwkv/\", \"title\": \"RWKV\", \"tokens\": 2201, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0839f852bab3131bde0ae77d6e8778e3088a9535abbd1b3e589bc1473066f622\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4788, \"end_char_idx\": 6055, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4e2b5e34-bd3a-451f-a43f-977a8bb12340", "doc_id": "4e2b5e34-bd3a-451f-a43f-977a8bb12340", "ref_doc_id": "4e2b5e34-bd3a-451f-a43f-977a8bb12340", "chroma:document": "Usually the softmax is not computed as is, but the exponential of the maximum term is divided of the numerator and denominator:\n\n$$\\frac{e^{x_{i}}}{\\sum_{j=1}^{n} e^{x_{j}}} = \\frac{e^{x_{i} - M}}{\\sum_{j=1}^{n} e^{x_{j} - M}}$$\n\nwith \\\\(M\\\\) the maximum of all \\\\(x_{j}\\\\). So here on top of saving the numerator state (\\\\(\\hat{N}\\\\)) and the denominator state (\\\\(\\hat{D}\\\\)) we also keep track of the maximum of all terms encountered in the exponentials. So we actually use\n\n$$\\tilde{N}_{i} = e^{-M_{i}} \\hat{N}_{i} \\hbox{  and  } \\tilde{D}_{i} = e^{-M_{i}} \\hat{D}_{i}$$\n\ndefined by the following recurrent formulas:\n\n$$\\tilde{N}_{0} = 0 \\hbox{  and  } \\tilde{N}_{j+1} = e^{K_{j} - q} V_{j} + e^{w + M_{j} - q} \\tilde{N}_{j} \\hbox{  where  } q = \\max(K_{j}, w + M_{j})$$\n\nand\n\n$$\\tilde{D}_{0} = 0 \\hbox{  and  } \\tilde{D}_{j+1} = e^{K_{j} - q} + e^{w + M_{j} - q} \\tilde{D}_{j} \\hbox{  where  } q = \\max(K_{j}, w + M_{j})$$\n\nand \\\\(M_{j+1} = q\\\\). With those, we can then compute\n\n$$N_{i} = e^{u + K_{i} - q} V_{i} + e^{M_{i}} \\tilde{N}_{i} \\hbox{  where  } q = \\max(u + K_{i}, M_{i})$$\n\nand\n\n$$D_{i} = e^{u + K_{i} - q} + e^{M_{i}} \\tilde{D}_{i} \\hbox{  where  } q = \\max(u + K_{i}, M_{i})$$\n\nwhich finally gives us\n\n$$O_{i} = \\sigma(R_{i}) \\frac{N_{i}}{D_{i}}$$"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: b1ab5964-421b-4c1e-9496-f6dcdea110d8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/dbrx/", "title": "DBRX", "tokens": 1388, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"b1ab5964-421b-4c1e-9496-f6dcdea110d8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dbrx/\", \"title\": \"DBRX\", \"tokens\": 1388, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dc2e5b32-5bbf-4241-922e-771827bc24ad\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dbrx/\", \"title\": \"DBRX\", \"tokens\": 1388, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8df19d53759fcaf85c6d0a7ef8cf2a7133f9b74afb956182a09161295f03bf73\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c2527983-90d6-423e-8208-b9cf98911217\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d51dd502d58096dfb044b6e88d1e14bd808ca8f17971a1d98c9d9f615b8069da\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3218, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dc2e5b32-5bbf-4241-922e-771827bc24ad", "doc_id": "dc2e5b32-5bbf-4241-922e-771827bc24ad", "ref_doc_id": "dc2e5b32-5bbf-4241-922e-771827bc24ad", "chroma:document": "# DBRX\n\n## Overview\n\nDBRX is a [transformer-based](https://www.isattentionallyouneed.com/) decoder-only large language model (LLM) that was trained using next-token prediction.\nIt uses a *fine-grained* mixture-of-experts (MoE) architecture with 132B total parameters of which 36B parameters are active on any input.\nIt was pre-trained on 12T tokens of text and code data.\nCompared to other open MoE models like Mixtral-8x7B and Grok-1, DBRX is fine-grained, meaning it uses a larger number of smaller experts. DBRX has 16 experts and chooses 4, while Mixtral-8x7B and Grok-1 have 8 experts and choose 2.\nThis provides 65x more possible combinations of experts and we found that this improves model quality.\nDBRX uses rotary position encodings (RoPE), gated linear units (GLU), and grouped query attention (GQA).\nIt is a BPE based model and uses the GPT-4 tokenizer as described in the [tiktoken](https://github.com/openai/tiktoken) repository.\nWe made these choices based on exhaustive evaluation and scaling experiments.\n\nDBRX was pretrained on 12T tokens of carefully curated data and a maximum context length of 32K tokens.\nWe estimate that this data is at least 2x better token-for-token than the data we used to pretrain the MPT family of models.\nThis new dataset was developed using the full suite of Databricks tools, including Apache Spark\u2122 and Databricks notebooks for data processing, and Unity Catalog for data management and governance.\nWe used curriculum learning for pretraining, changing the data mix during training in ways we found to substantially improve model quality.\n\n\nMore detailed information about DBRX Instruct and DBRX Base can be found in our [technical blog post](https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm).\n\nThis model was contributed by [eitan-turok](https://huggingface.co/eitanturok) and [abhi-db](https://huggingface.co/abhi-db). The original code can be found [here](https://github.com/databricks/dbrx-instruct), though this may not be up to date.\n\n## Usage Examples\n\nThe `generate()` method can be used to generate text using DBRX. You can generate using the standard attention implementation, flash-attention, and the PyTorch scaled dot product attention. The last two attention implementations give speed ups.\n\n```python\nfrom transformers import DbrxForCausalLM, AutoTokenizer\nimport torch\n\ntokenizer = AutoTokenizer.from_pretrained(\"databricks/dbrx-instruct\", token=\"YOUR_HF_TOKEN\")\nmodel = DbrxForCausalLM.from_pretrained(\n    \"databricks/dbrx-instruct\",\n    device_map=\"auto\",\n    torch_dtype=torch.bfloat16,\n    token=\"YOUR_HF_TOKEN\",\n    )\n\ninput_text = \"What does it take to build a great LLM?\"\nmessages = [{\"role\": \"user\", \"content\": input_text}]\ninput_ids = tokenizer.apply_chat_template(messages, return_dict=True, tokenize=True, add_generation_prompt=True, return_tensors=\"pt\").to(\"cuda\")\n\noutputs = model.generate(**input_ids, max_new_tokens=200)\nprint(tokenizer.decode(outputs[0]))\n```\n\nIf you have flash-attention installed (`pip install flash-attn`), it is possible to generate faster. (The HuggingFace documentation for flash-attention can be found [here](https://huggingface.co/docs/transformers/perf_infer_gpu_one#flashattention-2).)"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: c2527983-90d6-423e-8208-b9cf98911217
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/dbrx/", "title": "DBRX", "tokens": 1388, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"c2527983-90d6-423e-8208-b9cf98911217\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dbrx/\", \"title\": \"DBRX\", \"tokens\": 1388, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dc2e5b32-5bbf-4241-922e-771827bc24ad\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dbrx/\", \"title\": \"DBRX\", \"tokens\": 1388, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8df19d53759fcaf85c6d0a7ef8cf2a7133f9b74afb956182a09161295f03bf73\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b1ab5964-421b-4c1e-9496-f6dcdea110d8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dbrx/\", \"title\": \"DBRX\", \"tokens\": 1388, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"e9129f54609da91f86903f458505eba533f1baf55d394fa4422035c07da6fc97\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a2d27d2e-5539-4273-86d7-9fc6083293da\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5cb61f56f7d938e69d7b06c63e1ccdd68aac00938c7f9bf61ff3644798933570\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1768, \"end_char_idx\": 4634, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dc2e5b32-5bbf-4241-922e-771827bc24ad", "doc_id": "dc2e5b32-5bbf-4241-922e-771827bc24ad", "ref_doc_id": "dc2e5b32-5bbf-4241-922e-771827bc24ad", "chroma:document": "This model was contributed by [eitan-turok](https://huggingface.co/eitanturok) and [abhi-db](https://huggingface.co/abhi-db). The original code can be found [here](https://github.com/databricks/dbrx-instruct), though this may not be up to date.\n\n## Usage Examples\n\nThe `generate()` method can be used to generate text using DBRX. You can generate using the standard attention implementation, flash-attention, and the PyTorch scaled dot product attention. The last two attention implementations give speed ups.\n\n```python\nfrom transformers import DbrxForCausalLM, AutoTokenizer\nimport torch\n\ntokenizer = AutoTokenizer.from_pretrained(\"databricks/dbrx-instruct\", token=\"YOUR_HF_TOKEN\")\nmodel = DbrxForCausalLM.from_pretrained(\n    \"databricks/dbrx-instruct\",\n    device_map=\"auto\",\n    torch_dtype=torch.bfloat16,\n    token=\"YOUR_HF_TOKEN\",\n    )\n\ninput_text = \"What does it take to build a great LLM?\"\nmessages = [{\"role\": \"user\", \"content\": input_text}]\ninput_ids = tokenizer.apply_chat_template(messages, return_dict=True, tokenize=True, add_generation_prompt=True, return_tensors=\"pt\").to(\"cuda\")\n\noutputs = model.generate(**input_ids, max_new_tokens=200)\nprint(tokenizer.decode(outputs[0]))\n```\n\nIf you have flash-attention installed (`pip install flash-attn`), it is possible to generate faster. (The HuggingFace documentation for flash-attention can be found [here](https://huggingface.co/docs/transformers/perf_infer_gpu_one#flashattention-2).)\n```python\nfrom transformers import DbrxForCausalLM, AutoTokenizer\nimport torch\n\ntokenizer = AutoTokenizer.from_pretrained(\"databricks/dbrx-instruct\", token=\"YOUR_HF_TOKEN\")\nmodel = DbrxForCausalLM.from_pretrained(\n    \"databricks/dbrx-instruct\",\n    device_map=\"auto\",\n    torch_dtype=torch.bfloat16,\n    token=\"YOUR_HF_TOKEN\",\n    attn_implementation=\"flash_attention_2\",\n    )\n\ninput_text = \"What does it take to build a great LLM?\"\nmessages = [{\"role\": \"user\", \"content\": input_text}]\ninput_ids = tokenizer.apply_chat_template(messages, return_dict=True, tokenize=True, add_generation_prompt=True, return_tensors=\"pt\").to(\"cuda\")\n\noutputs = model.generate(**input_ids, max_new_tokens=200)\nprint(tokenizer.decode(outputs[0]))\n```\n\nYou can also generate faster using the PyTorch scaled dot product attention. (The HuggingFace documentation for scaled dot product attention can be found [here](https://huggingface.co/docs/transformers/perf_infer_gpu_one#pytorch-scaled-dot-product-attention).)\n```python\nfrom transformers import DbrxForCausalLM, AutoTokenizer\nimport torch\n\ntokenizer = AutoTokenizer.from_pretrained(\"databricks/dbrx-instruct\", token=\"YOUR_HF_TOKEN\")\nmodel = DbrxForCausalLM.from_pretrained(\n    \"databricks/dbrx-instruct\",\n    device_map=\"auto\",\n    torch_dtype=torch.bfloat16,\n    token=\"YOUR_HF_TOKEN\",\n    attn_implementation=\"sdpa\",\n    )\n\ninput_text = \"What does it take to build a great LLM?\""}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: a2d27d2e-5539-4273-86d7-9fc6083293da
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/dbrx/", "title": "DBRX", "tokens": 1388, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"a2d27d2e-5539-4273-86d7-9fc6083293da\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dbrx/\", \"title\": \"DBRX\", \"tokens\": 1388, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dc2e5b32-5bbf-4241-922e-771827bc24ad\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dbrx/\", \"title\": \"DBRX\", \"tokens\": 1388, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8df19d53759fcaf85c6d0a7ef8cf2a7133f9b74afb956182a09161295f03bf73\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c2527983-90d6-423e-8208-b9cf98911217\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dbrx/\", \"title\": \"DBRX\", \"tokens\": 1388, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"432bb5ab0acb7c3722e14f71ae5e673b68bc107f95db8c92e5434ba5edfd267a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3068, \"end_char_idx\": 5086, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dc2e5b32-5bbf-4241-922e-771827bc24ad", "doc_id": "dc2e5b32-5bbf-4241-922e-771827bc24ad", "ref_doc_id": "dc2e5b32-5bbf-4241-922e-771827bc24ad", "chroma:document": "(The HuggingFace documentation for flash-attention can be found [here](https://huggingface.co/docs/transformers/perf_infer_gpu_one#flashattention-2).)\n```python\nfrom transformers import DbrxForCausalLM, AutoTokenizer\nimport torch\n\ntokenizer = AutoTokenizer.from_pretrained(\"databricks/dbrx-instruct\", token=\"YOUR_HF_TOKEN\")\nmodel = DbrxForCausalLM.from_pretrained(\n    \"databricks/dbrx-instruct\",\n    device_map=\"auto\",\n    torch_dtype=torch.bfloat16,\n    token=\"YOUR_HF_TOKEN\",\n    attn_implementation=\"flash_attention_2\",\n    )\n\ninput_text = \"What does it take to build a great LLM?\"\nmessages = [{\"role\": \"user\", \"content\": input_text}]\ninput_ids = tokenizer.apply_chat_template(messages, return_dict=True, tokenize=True, add_generation_prompt=True, return_tensors=\"pt\").to(\"cuda\")\n\noutputs = model.generate(**input_ids, max_new_tokens=200)\nprint(tokenizer.decode(outputs[0]))\n```\n\nYou can also generate faster using the PyTorch scaled dot product attention. (The HuggingFace documentation for scaled dot product attention can be found [here](https://huggingface.co/docs/transformers/perf_infer_gpu_one#pytorch-scaled-dot-product-attention).)\n```python\nfrom transformers import DbrxForCausalLM, AutoTokenizer\nimport torch\n\ntokenizer = AutoTokenizer.from_pretrained(\"databricks/dbrx-instruct\", token=\"YOUR_HF_TOKEN\")\nmodel = DbrxForCausalLM.from_pretrained(\n    \"databricks/dbrx-instruct\",\n    device_map=\"auto\",\n    torch_dtype=torch.bfloat16,\n    token=\"YOUR_HF_TOKEN\",\n    attn_implementation=\"sdpa\",\n    )\n\ninput_text = \"What does it take to build a great LLM?\"\nmessages = [{\"role\": \"user\", \"content\": input_text}]\ninput_ids = tokenizer.apply_chat_template(messages, return_dict=True, tokenize=True, add_generation_prompt=True, return_tensors=\"pt\").to(\"cuda\")\n\noutputs = model.generate(**input_ids, max_new_tokens=200)\nprint(tokenizer.decode(outputs[0]))\n```\n\n## DbrxConfig\n\n[[autodoc]] DbrxConfig\n\n\n## DbrxModel\n\n[[autodoc]] DbrxModel\n    - forward\n\n\n## DbrxForCausalLM\n\n[[autodoc]] DbrxForCausalLM\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: ddba67b7-41dc-4bfb-b15e-301d3920190d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/byt5/", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"ddba67b7-41dc-4bfb-b15e-301d3920190d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/byt5/\", \"title\": \"ByT5\", \"tokens\": 2066, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2e697c2e-2348-4f45-abff-028c40ffd859\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/byt5/\", \"title\": \"ByT5\", \"tokens\": 2066, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"ee0351fb29b4ec6d83b8db7b79a8883f0da5f1bbd607d1f4315e299fb8e4877a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7109887e-e2f6-4142-8fc8-125b5389d33f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8e8eddd5f7ab2d7b9b24912405efe684b6db71a6036a80028ecfdfe42a89158f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2941, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2e697c2e-2348-4f45-abff-028c40ffd859", "doc_id": "2e697c2e-2348-4f45-abff-028c40ffd859", "ref_doc_id": "2e697c2e-2348-4f45-abff-028c40ffd859", "chroma:document": "# ByT5\n\n## Overview\n\nThe ByT5 model was presented in [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://arxiv.org/abs/2105.13626) by Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir\nKale, Adam Roberts, Colin Raffel.\n\nThe abstract from the paper is the following:\n\n*Most widely-used pre-trained language models operate on sequences of tokens corresponding to word or subword units.\nEncoding text as a sequence of tokens requires a tokenizer, which is typically created as an independent artifact from\nthe model. Token-free models that instead operate directly on raw text (bytes or characters) have many benefits: they\ncan process text in any language out of the box, they are more robust to noise, and they minimize technical debt by\nremoving complex and error-prone text preprocessing pipelines. Since byte or character sequences are longer than token\nsequences, past work on token-free models has often introduced new model architectures designed to amortize the cost of\noperating directly on raw text. In this paper, we show that a standard Transformer architecture can be used with\nminimal modifications to process byte sequences. We carefully characterize the trade-offs in terms of parameter count,\ntraining FLOPs, and inference speed, and show that byte-level models are competitive with their token-level\ncounterparts. We also demonstrate that byte-level models are significantly more robust to noise and perform better on\ntasks that are sensitive to spelling and pronunciation. As part of our contribution, we release a new set of\npre-trained byte-level Transformer models based on the T5 architecture, as well as all code and data used in our\nexperiments.*\n\nThis model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The original code can be\nfound [here](https://github.com/google-research/byt5).\n\n<Tip>\n\nByT5's architecture is based on the T5v1.1 model, refer to [T5v1.1's documentation page](t5v1.1) for the API reference. They\nonly differ in how inputs should be prepared for the model, see the code examples below.\n\n</Tip>\n\nSince ByT5 was pre-trained unsupervisedly, there's no real advantage to using a task prefix during single-task\nfine-tuning. If you are doing multi-task fine-tuning, you should use a prefix.\n\n\n## Usage example\n\nByT5 works on raw UTF-8 bytes, so it can be used without a tokenizer:\n\n```python\n>>> from transformers import T5ForConditionalGeneration\n>>> import torch\n\n>>> model = T5ForConditionalGeneration.from_pretrained(\"google/byt5-small\")\n\n>>> num_special_tokens = 3\n>>> # Model has 3 special tokens which take up the input ids 0,1,2 of ByT5.\n>>> # => Need to shift utf-8 character encodings by 3 before passing ids to model.\n\n>>> input_ids = torch.tensor([list(\"Life is like a box of chocolates.\".encode(\"utf-8\"))]) + num_special_tokens\n\n>>> labels = torch.tensor([list(\"La vie est comme une bo\u00eete de chocolat."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 7109887e-e2f6-4142-8fc8-125b5389d33f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/byt5/", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"7109887e-e2f6-4142-8fc8-125b5389d33f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/byt5/\", \"title\": \"ByT5\", \"tokens\": 2066, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2e697c2e-2348-4f45-abff-028c40ffd859\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/byt5/\", \"title\": \"ByT5\", \"tokens\": 2066, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"ee0351fb29b4ec6d83b8db7b79a8883f0da5f1bbd607d1f4315e299fb8e4877a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ddba67b7-41dc-4bfb-b15e-301d3920190d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/byt5/\", \"title\": \"ByT5\", \"tokens\": 2066, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b71fba565db6cd3e98e60b7e6dc840dedd40b342bc527b58664253488aa00759\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"66e5c3a9-b41e-42a7-b1ba-09e9f5190fdd\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4937b9471477d1fdadf8a629cdcdff26318a5c7b76842911025759b3e2d6bc13\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2320, \"end_char_idx\": 4950, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2e697c2e-2348-4f45-abff-028c40ffd859", "doc_id": "2e697c2e-2348-4f45-abff-028c40ffd859", "ref_doc_id": "2e697c2e-2348-4f45-abff-028c40ffd859", "chroma:document": "## Usage example\n\nByT5 works on raw UTF-8 bytes, so it can be used without a tokenizer:\n\n```python\n>>> from transformers import T5ForConditionalGeneration\n>>> import torch\n\n>>> model = T5ForConditionalGeneration.from_pretrained(\"google/byt5-small\")\n\n>>> num_special_tokens = 3\n>>> # Model has 3 special tokens which take up the input ids 0,1,2 of ByT5.\n>>> # => Need to shift utf-8 character encodings by 3 before passing ids to model.\n\n>>> input_ids = torch.tensor([list(\"Life is like a box of chocolates.\".encode(\"utf-8\"))]) + num_special_tokens\n\n>>> labels = torch.tensor([list(\"La vie est comme une bo\u00eete de chocolat.\".encode(\"utf-8\"))]) + num_special_tokens\n\n>>> loss = model(input_ids, labels=labels).loss\n>>> loss.item()\n2.66\n```\n\nFor batched inference and training it is however recommended to make use of the tokenizer:\n\n```python\n>>> from transformers import T5ForConditionalGeneration, AutoTokenizer\n\n>>> model = T5ForConditionalGeneration.from_pretrained(\"google/byt5-small\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"google/byt5-small\")\n\n>>> model_inputs = tokenizer(\n...     [\"Life is like a box of chocolates.\", \"Today is Monday.\"], padding=\"longest\", return_tensors=\"pt\"\n... )\n>>> labels_dict = tokenizer(\n...     [\"La vie est comme une bo\u00eete de chocolat.\", \"Aujourd'hui c'est lundi.\"], padding=\"longest\", return_tensors=\"pt\"\n... )\n>>> labels = labels_dict.input_ids\n\n>>> loss = model(**model_inputs, labels=labels).loss\n>>> loss.item()\n17.9\n```\n\nSimilar to [T5](t5), ByT5 was trained on the span-mask denoising task. However, \nsince the model works directly on characters, the pretraining task is a bit \ndifferent. Let's corrupt some characters of the \ninput sentence `\"The dog chases a ball in the park.\"` and ask ByT5 to predict them \nfor us.\n\n```python\n>>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM\n>>> import torch\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"google/byt5-base\")\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"google/byt5-base\")\n\n>>> input_ids_prompt = \"The dog chases a ball in the park.\"\n>>> input_ids = tokenizer(input_ids_prompt).input_ids\n\n>>> # Note that we cannot add \"{extra_id_...}\" to the string directly\n>>> # as the Byte tokenizer would incorrectly merge the tokens\n>>> # For ByT5, we need to work directly on the character level\n>>> # Contrary to T5, ByT5 does not use sentinel tokens for masking, but instead\n>>> # uses final utf character ids.\n>>> # UTF-8 is represented by 8 bits and ByT5 has 3 special tokens.\n>>> # => There are 2**8+2 = 259 input ids and mask tokens count down from index 258.\n>>> # => mask to \"The dog [258]a ball [257]park.\""}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 66e5c3a9-b41e-42a7-b1ba-09e9f5190fdd
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/byt5/", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"66e5c3a9-b41e-42a7-b1ba-09e9f5190fdd\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/byt5/\", \"title\": \"ByT5\", \"tokens\": 2066, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2e697c2e-2348-4f45-abff-028c40ffd859\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/byt5/\", \"title\": \"ByT5\", \"tokens\": 2066, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"ee0351fb29b4ec6d83b8db7b79a8883f0da5f1bbd607d1f4315e299fb8e4877a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7109887e-e2f6-4142-8fc8-125b5389d33f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/byt5/\", \"title\": \"ByT5\", \"tokens\": 2066, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"63b851e19f51765a8373cbed7456bbecd1c8e4d3662337121d557d6c6b30773e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dad3e290-ac2a-456f-b4ae-9e8e27b438a6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"063d4e0a601f19634d41e30172f0792ddfc1faa4844db2875f1eb8450258025d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3469, \"end_char_idx\": 5319, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2e697c2e-2348-4f45-abff-028c40ffd859", "doc_id": "2e697c2e-2348-4f45-abff-028c40ffd859", "ref_doc_id": "2e697c2e-2348-4f45-abff-028c40ffd859", "chroma:document": "\"], padding=\"longest\", return_tensors=\"pt\"\n... )\n>>> labels_dict = tokenizer(\n...     [\"La vie est comme une bo\u00eete de chocolat.\", \"Aujourd'hui c'est lundi.\"], padding=\"longest\", return_tensors=\"pt\"\n... )\n>>> labels = labels_dict.input_ids\n\n>>> loss = model(**model_inputs, labels=labels).loss\n>>> loss.item()\n17.9\n```\n\nSimilar to [T5](t5), ByT5 was trained on the span-mask denoising task. However, \nsince the model works directly on characters, the pretraining task is a bit \ndifferent. Let's corrupt some characters of the \ninput sentence `\"The dog chases a ball in the park.\"` and ask ByT5 to predict them \nfor us.\n\n```python\n>>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM\n>>> import torch\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"google/byt5-base\")\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"google/byt5-base\")\n\n>>> input_ids_prompt = \"The dog chases a ball in the park.\"\n>>> input_ids = tokenizer(input_ids_prompt).input_ids\n\n>>> # Note that we cannot add \"{extra_id_...}\" to the string directly\n>>> # as the Byte tokenizer would incorrectly merge the tokens\n>>> # For ByT5, we need to work directly on the character level\n>>> # Contrary to T5, ByT5 does not use sentinel tokens for masking, but instead\n>>> # uses final utf character ids.\n>>> # UTF-8 is represented by 8 bits and ByT5 has 3 special tokens.\n>>> # => There are 2**8+2 = 259 input ids and mask tokens count down from index 258.\n>>> # => mask to \"The dog [258]a ball [257]park.\"\n\n>>> input_ids = torch.tensor([input_ids[:8] + [258] + input_ids[14:21] + [257] + input_ids[28:]])\n>>> input_ids\ntensor([[ 87, 107, 104,  35, 103, 114, 106,  35, 258,  35, 100,  35, 101, 100, 111, 111, 257,  35, 115, 100, 117, 110,  49,   1]])\n\n>>> # ByT5 produces only one char at a time so we need to produce many more output characters here -> set `max_length=100`."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: dad3e290-ac2a-456f-b4ae-9e8e27b438a6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/byt5/", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"dad3e290-ac2a-456f-b4ae-9e8e27b438a6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/byt5/\", \"title\": \"ByT5\", \"tokens\": 2066, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2e697c2e-2348-4f45-abff-028c40ffd859\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/byt5/\", \"title\": \"ByT5\", \"tokens\": 2066, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"ee0351fb29b4ec6d83b8db7b79a8883f0da5f1bbd607d1f4315e299fb8e4877a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"66e5c3a9-b41e-42a7-b1ba-09e9f5190fdd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/byt5/\", \"title\": \"ByT5\", \"tokens\": 2066, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"42cf023281dcf0e35226b58bafe6efbebe1d2180fd42e2e130ab3456c2e22ba1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fa74fa73-374c-42c2-9fc6-21aecbbd31e2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"16f949deb325dceb8c3100c6f57c1d74538b8b4a93110c00fd716dee66d4c944\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4046, \"end_char_idx\": 6522, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2e697c2e-2348-4f45-abff-028c40ffd859", "doc_id": "2e697c2e-2348-4f45-abff-028c40ffd859", "ref_doc_id": "2e697c2e-2348-4f45-abff-028c40ffd859", "chroma:document": "\"` and ask ByT5 to predict them \nfor us.\n\n```python\n>>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM\n>>> import torch\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"google/byt5-base\")\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"google/byt5-base\")\n\n>>> input_ids_prompt = \"The dog chases a ball in the park.\"\n>>> input_ids = tokenizer(input_ids_prompt).input_ids\n\n>>> # Note that we cannot add \"{extra_id_...}\" to the string directly\n>>> # as the Byte tokenizer would incorrectly merge the tokens\n>>> # For ByT5, we need to work directly on the character level\n>>> # Contrary to T5, ByT5 does not use sentinel tokens for masking, but instead\n>>> # uses final utf character ids.\n>>> # UTF-8 is represented by 8 bits and ByT5 has 3 special tokens.\n>>> # => There are 2**8+2 = 259 input ids and mask tokens count down from index 258.\n>>> # => mask to \"The dog [258]a ball [257]park.\"\n\n>>> input_ids = torch.tensor([input_ids[:8] + [258] + input_ids[14:21] + [257] + input_ids[28:]])\n>>> input_ids\ntensor([[ 87, 107, 104,  35, 103, 114, 106,  35, 258,  35, 100,  35, 101, 100, 111, 111, 257,  35, 115, 100, 117, 110,  49,   1]])\n\n>>> # ByT5 produces only one char at a time so we need to produce many more output characters here -> set `max_length=100`.\n>>> output_ids = model.generate(input_ids, max_length=100)[0].tolist()\n>>> output_ids\n[0, 258, 108, 118,  35, 119, 107, 104,  35, 114, 113, 104,  35, 122, 107, 114,  35, 103, 114, 104, 118, 257,  35, 108, 113,  35, 119, 107, 104,  35, 103, 108, 118, 102, 114, 256, 108, 113,  35, 119, 107, 104, 35, 115, 100, 117, 110,  49,  35,  87, 107, 104,  35, 103, 114, 106, 35, 108, 118,  35, 119, 107, 104,  35, 114, 113, 104,  35, 122, 107, 114,  35, 103, 114, 104, 118,  35, 100,  35, 101, 100, 111, 111,  35, 108, 113, 255,  35, 108, 113,  35, 119, 107, 104,  35, 115, 100, 117, 110,  49]\n\n>>> # ^- Note how 258 descends to 257, 256, 255\n\n>>> # Now we need to split on the sentinel tokens, let's write a short loop for this\n>>> output_ids_list = []\n>>> start_token = 0\n>>> sentinel_token = 258\n>>> while sentinel_token in output_ids:\n...     split_idx = output_ids.index(sentinel_token)\n...     output_ids_list.append(output_ids[start_token:split_idx])\n...     start_token = split_idx\n...     sentinel_token -= 1\n\n>>> output_ids_list.append(output_ids[start_token:])\n>>> output_string = tokenizer.batch_decode(output_ids_list)\n>>> output_string\n['<pad>', 'is the one who does', ' in the disco', 'in the park."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: fa74fa73-374c-42c2-9fc6-21aecbbd31e2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/byt5/", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"fa74fa73-374c-42c2-9fc6-21aecbbd31e2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/byt5/\", \"title\": \"ByT5\", \"tokens\": 2066, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2e697c2e-2348-4f45-abff-028c40ffd859\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/byt5/\", \"title\": \"ByT5\", \"tokens\": 2066, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"ee0351fb29b4ec6d83b8db7b79a8883f0da5f1bbd607d1f4315e299fb8e4877a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dad3e290-ac2a-456f-b4ae-9e8e27b438a6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/byt5/\", \"title\": \"ByT5\", \"tokens\": 2066, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"946879dfcd6633297ecae4220eddc6c26da67b72094b9bf22df1f6894e183805\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6523, \"end_char_idx\": 6669, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2e697c2e-2348-4f45-abff-028c40ffd859", "doc_id": "2e697c2e-2348-4f45-abff-028c40ffd859", "ref_doc_id": "2e697c2e-2348-4f45-abff-028c40ffd859", "chroma:document": "The dog is the one who does a ball in', ' in the park.']\n```\n\n\n## ByT5Tokenizer\n\n[[autodoc]] ByT5Tokenizer\n\nSee [`ByT5Tokenizer`] for all details."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 15c735c9-81d4-4ba8-aeae-67ee721c1884
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/superpoint/", "title": "SuperPoint", "tokens": 1355, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"15c735c9-81d4-4ba8-aeae-67ee721c1884\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/superpoint/\", \"title\": \"SuperPoint\", \"tokens\": 1355, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5d40ddbf-282b-4df1-a732-d9f4eab04aba\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/superpoint/\", \"title\": \"SuperPoint\", \"tokens\": 1355, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"039b025cc57a87ab4319c9d565047a9566dcf18b50319745188b575924840fef\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"44c58620-659a-4b4b-9651-680913fd6759\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6a9731a06952e7a5b0573498451d0fdf908e2a71fffc0d342a0e0be5c8ecf324\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2725, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5d40ddbf-282b-4df1-a732-d9f4eab04aba", "doc_id": "5d40ddbf-282b-4df1-a732-d9f4eab04aba", "ref_doc_id": "5d40ddbf-282b-4df1-a732-d9f4eab04aba", "chroma:document": "# SuperPoint\n\n## Overview\n\nThe SuperPoint model was proposed\nin [SuperPoint: Self-Supervised Interest Point Detection and Description](https://arxiv.org/abs/1712.07629) by Daniel\nDeTone, Tomasz Malisiewicz and Andrew Rabinovich.\n\nThis model is the result of a self-supervised training of a fully-convolutional network for interest point detection and\ndescription. The model is able to detect interest points that are repeatable under homographic transformations and\nprovide a descriptor for each point. The use of the model in its own is limited, but it can be used as a feature\nextractor for other tasks such as homography estimation, image matching, etc.\n\nThe abstract from the paper is the following:\n\n*This paper presents a self-supervised framework for training interest point detectors and descriptors suitable for a\nlarge number of multiple-view geometry problems in computer vision. As opposed to patch-based neural networks, our\nfully-convolutional model operates on full-sized images and jointly computes pixel-level interest point locations and\nassociated descriptors in one forward pass. We introduce Homographic Adaptation, a multi-scale, multi-homography\napproach for boosting interest point detection repeatability and performing cross-domain adaptation (e.g.,\nsynthetic-to-real). Our model, when trained on the MS-COCO generic image dataset using Homographic Adaptation, is able\nto repeatedly detect a much richer set of interest points than the initial pre-adapted deep model and any other\ntraditional corner detector. The final system gives rise to state-of-the-art homography estimation results on HPatches\nwhen compared to LIFT, SIFT and ORB.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/superpoint_architecture.png\"\nalt=\"drawing\" width=\"500\"/>\n\n<small> SuperPoint overview. Taken from the <a href=\"https://arxiv.org/abs/1712.07629v4\">original paper.</a> </small>\n\n## Usage tips\n\nHere is a quick example of using the model to detect interest points in an image:\n\n```python\nfrom transformers import AutoImageProcessor, SuperPointForKeypointDetection\nimport torch\nfrom PIL import Image\nimport requests\n\nurl = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nimage = Image.open(requests.get(url, stream=True).raw)\n\nprocessor = AutoImageProcessor.from_pretrained(\"magic-leap-community/superpoint\")\nmodel = SuperPointForKeypointDetection.from_pretrained(\"magic-leap-community/superpoint\")\n\ninputs = processor(image, return_tensors=\"pt\")\noutputs = model(**inputs)\n```\n\nThe outputs contain the list of keypoint coordinates with their respective score and description (a 256-long vector).\n\nYou can also feed multiple images to the model."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 44c58620-659a-4b4b-9651-680913fd6759
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/superpoint/", "title": "SuperPoint", "tokens": 1355, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"44c58620-659a-4b4b-9651-680913fd6759\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/superpoint/\", \"title\": \"SuperPoint\", \"tokens\": 1355, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5d40ddbf-282b-4df1-a732-d9f4eab04aba\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/superpoint/\", \"title\": \"SuperPoint\", \"tokens\": 1355, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"039b025cc57a87ab4319c9d565047a9566dcf18b50319745188b575924840fef\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"15c735c9-81d4-4ba8-aeae-67ee721c1884\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/superpoint/\", \"title\": \"SuperPoint\", \"tokens\": 1355, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"2113618f7ea83d440d2c2c2ea7704fc963e97810c349522d0649ba6c89696564\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1e935740-7796-4ada-9ed7-3f486b524779\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"09da46a3b3df86abdc55d0ec6994a73466c32e30f9c1281aa502b712e3117a07\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1100, \"end_char_idx\": 4342, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5d40ddbf-282b-4df1-a732-d9f4eab04aba", "doc_id": "5d40ddbf-282b-4df1-a732-d9f4eab04aba", "ref_doc_id": "5d40ddbf-282b-4df1-a732-d9f4eab04aba", "chroma:document": "We introduce Homographic Adaptation, a multi-scale, multi-homography\napproach for boosting interest point detection repeatability and performing cross-domain adaptation (e.g.,\nsynthetic-to-real). Our model, when trained on the MS-COCO generic image dataset using Homographic Adaptation, is able\nto repeatedly detect a much richer set of interest points than the initial pre-adapted deep model and any other\ntraditional corner detector. The final system gives rise to state-of-the-art homography estimation results on HPatches\nwhen compared to LIFT, SIFT and ORB.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/superpoint_architecture.png\"\nalt=\"drawing\" width=\"500\"/>\n\n<small> SuperPoint overview. Taken from the <a href=\"https://arxiv.org/abs/1712.07629v4\">original paper.</a> </small>\n\n## Usage tips\n\nHere is a quick example of using the model to detect interest points in an image:\n\n```python\nfrom transformers import AutoImageProcessor, SuperPointForKeypointDetection\nimport torch\nfrom PIL import Image\nimport requests\n\nurl = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nimage = Image.open(requests.get(url, stream=True).raw)\n\nprocessor = AutoImageProcessor.from_pretrained(\"magic-leap-community/superpoint\")\nmodel = SuperPointForKeypointDetection.from_pretrained(\"magic-leap-community/superpoint\")\n\ninputs = processor(image, return_tensors=\"pt\")\noutputs = model(**inputs)\n```\n\nThe outputs contain the list of keypoint coordinates with their respective score and description (a 256-long vector).\n\nYou can also feed multiple images to the model. Due to the nature of SuperPoint, to output a dynamic number of keypoints,\nyou will need to use the mask attribute to retrieve the respective information :\n\n```python\nfrom transformers import AutoImageProcessor, SuperPointForKeypointDetection\nimport torch\nfrom PIL import Image\nimport requests\n\nurl_image_1 = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nimage_1 = Image.open(requests.get(url_image_1, stream=True).raw)\nurl_image_2 = \"http://images.cocodataset.org/test-stuff2017/000000000568.jpg\"\nimage_2 = Image.open(requests.get(url_image_2, stream=True).raw)\n\nimages = [image_1, image_2]\n\nprocessor = AutoImageProcessor.from_pretrained(\"magic-leap-community/superpoint\")\nmodel = SuperPointForKeypointDetection.from_pretrained(\"magic-leap-community/superpoint\")\n\ninputs = processor(images, return_tensors=\"pt\")\noutputs = model(**inputs)\n\nfor i in range(len(images)):\n    image_mask = outputs.mask[i]\n    image_indices = torch.nonzero(image_mask).squeeze()\n    image_keypoints = outputs.keypoints[i][image_indices]\n    image_scores = outputs.scores[i][image_indices]\n    image_descriptors = outputs.descriptors[i][image_indices]\n```\n\nYou can then print the keypoints on the image to visualize the result :\n```python\nimport cv2\nfor keypoint, score in zip(image_keypoints, image_scores):\n    keypoint_x, keypoint_y = int(keypoint[0].item()), int(keypoint[1].item())\n    color = tuple([score.item() * 255] * 3)\n    image = cv2.circle(image, (keypoint_x, keypoint_y), 2, color)\ncv2.imwrite(\"output_image.png\", image)\n```\n\nThis model was contributed by [stevenbucaille](https://huggingface.co/stevenbucaille)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 1e935740-7796-4ada-9ed7-3f486b524779
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/superpoint/", "title": "SuperPoint", "tokens": 1355, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"1e935740-7796-4ada-9ed7-3f486b524779\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/superpoint/\", \"title\": \"SuperPoint\", \"tokens\": 1355, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5d40ddbf-282b-4df1-a732-d9f4eab04aba\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/superpoint/\", \"title\": \"SuperPoint\", \"tokens\": 1355, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"039b025cc57a87ab4319c9d565047a9566dcf18b50319745188b575924840fef\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"44c58620-659a-4b4b-9651-680913fd6759\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/superpoint/\", \"title\": \"SuperPoint\", \"tokens\": 1355, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"6e818acac2aad1ccf43f3c3d9d4be2de9d605a61cc38e6490468dff76bc2be97\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4343, \"end_char_idx\": 5256, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5d40ddbf-282b-4df1-a732-d9f4eab04aba", "doc_id": "5d40ddbf-282b-4df1-a732-d9f4eab04aba", "ref_doc_id": "5d40ddbf-282b-4df1-a732-d9f4eab04aba", "chroma:document": "The original code can be found [here](https://github.com/magicleap/SuperPointPretrainedNetwork).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with SuperPoint. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n- A notebook showcasing inference and visualization with SuperPoint can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/SuperPoint/Inference_with_SuperPoint_to_detect_interest_points_in_an_image.ipynb). \ud83c\udf0e\n\n## SuperPointConfig\n\n[[autodoc]] SuperPointConfig\n\n## SuperPointImageProcessor\n\n[[autodoc]] SuperPointImageProcessor\n\n- preprocess\n\n## SuperPointForKeypointDetection\n\n[[autodoc]] SuperPointForKeypointDetection\n\n- forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 78677fcf-bea6-4465-bf29-9eed0fa09a03
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/flan-t5/", "title": "FLAN-T5", "tokens": 610, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"78677fcf-bea6-4465-bf29-9eed0fa09a03\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/flan-t5/\", \"title\": \"FLAN-T5\", \"tokens\": 610, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b8c4f7e3-032e-45c9-9e7f-91df001953d9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/flan-t5/\", \"title\": \"FLAN-T5\", \"tokens\": 610, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"4f325eb26836e10fc74c745f1666dcd5bbab4c5fd2a76e6931e2b73c51a7d3b3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1685, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b8c4f7e3-032e-45c9-9e7f-91df001953d9", "doc_id": "b8c4f7e3-032e-45c9-9e7f-91df001953d9", "ref_doc_id": "b8c4f7e3-032e-45c9-9e7f-91df001953d9", "chroma:document": "# FLAN-T5\n\n## Overview\n\nFLAN-T5 was released in the paper [Scaling Instruction-Finetuned Language Models](https://arxiv.org/pdf/2210.11416.pdf) - it is an enhanced version of T5 that has been finetuned in a mixture of tasks.\n\nOne can directly use FLAN-T5 weights without finetuning the model:\n\n```python\n>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"google/flan-t5-small\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"google/flan-t5-small\")\n\n>>> inputs = tokenizer(\"A step by step recipe to make bolognese pasta:\", return_tensors=\"pt\")\n>>> outputs = model.generate(**inputs)\n>>> print(tokenizer.batch_decode(outputs, skip_special_tokens=True))\n['Pour a cup of bolognese into a large bowl and add the pasta']\n```\n\nFLAN-T5 includes the same improvements as T5 version 1.1 (see [here](https://huggingface.co/docs/transformers/model_doc/t5v1.1) for the full details of the model's improvements.)\n\nGoogle has released the following variants:\n\n- [google/flan-t5-small](https://huggingface.co/google/flan-t5-small)\n\n- [google/flan-t5-base](https://huggingface.co/google/flan-t5-base)\n\n- [google/flan-t5-large](https://huggingface.co/google/flan-t5-large)\n\n- [google/flan-t5-xl](https://huggingface.co/google/flan-t5-xl)\n\n- [google/flan-t5-xxl](https://huggingface.co/google/flan-t5-xxl).\n\nThe original checkpoints can be found [here](https://github.com/google-research/t5x/blob/main/docs/models.md#flan-t5-checkpoints).\n\n<Tip>\n\nRefer to [T5's documentation page](t5) for all API reference, code examples and notebooks. For more details regarding training and evaluation of the FLAN-T5, refer to the model card.\n\n</Tip>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 69c85a2c-6250-4d54-9acf-d6edbbe8baed
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/ctrl/", "title": "CTRL", "tokens": 1027, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"69c85a2c-6250-4d54-9acf-d6edbbe8baed\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/ctrl/\", \"title\": \"CTRL\", \"tokens\": 1027, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c7ad9229-4b27-4547-a68e-95bf3414bdd1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/ctrl/\", \"title\": \"CTRL\", \"tokens\": 1027, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"98edeae81590169ba84bec73f31bdd7e0d7e418715887c74f87214fda04dc417\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a52b2faa-3fc9-4220-9003-968805848ec2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"38303f5ac0f88881c98e64927f80261931afd54ef45b4dae38fd068a8640a888\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2864, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c7ad9229-4b27-4547-a68e-95bf3414bdd1", "doc_id": "c7ad9229-4b27-4547-a68e-95bf3414bdd1", "ref_doc_id": "c7ad9229-4b27-4547-a68e-95bf3414bdd1", "chroma:document": "# CTRL\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=ctrl\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-ctrl-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/tiny-ctrl\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nCTRL model was proposed in [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://arxiv.org/abs/1909.05858) by Nitish Shirish Keskar*, Bryan McCann*, Lav R. Varshney, Caiming Xiong and\nRichard Socher. It's a causal (unidirectional) transformer pre-trained using language modeling on a very large corpus\nof ~140 GB of text data with the first token reserved as a control code (such as Links, Books, Wikipedia etc.).\n\nThe abstract from the paper is the following:\n\n*Large-scale language models show promising text generation capabilities, but users cannot easily control particular\naspects of the generated text. We release CTRL, a 1.63 billion-parameter conditional transformer language model,\ntrained to condition on control codes that govern style, content, and task-specific behavior. Control codes were\nderived from structure that naturally co-occurs with raw text, preserving the advantages of unsupervised learning while\nproviding more explicit control over text generation. These codes also allow CTRL to predict which parts of the\ntraining data are most likely given a sequence. This provides a potential method for analyzing large amounts of data\nvia model-based source attribution.*\n\nThis model was contributed by [keskarnitishr](https://huggingface.co/keskarnitishr). The original code can be found\n[here](https://github.com/salesforce/ctrl).\n\n## Usage tips\n\n- CTRL makes use of control codes to generate text: it requires generations to be started by certain words, sentences\n  or links to generate coherent text. Refer to the [original implementation](https://github.com/salesforce/ctrl) for\n  more information.\n- CTRL is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than\n  the left.\n- CTRL was trained with a causal language modeling (CLM) objective and is therefore powerful at predicting the next\n  token in a sequence. Leveraging this feature allows CTRL to generate syntactically coherent text as it can be\n  observed in the *run_generation.py* example script.\n- The PyTorch models can take the `past_key_values` as input, which is the previously computed key/value attention pairs.\n  TensorFlow models accepts `past` as input. Using the `past_key_values` value prevents the model from re-computing\n  pre-computed values in the context of text generation. See the [`forward`](model_doc/ctrl#transformers.CTRLModel.forward)\n  method for more information on the usage of this argument."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: a52b2faa-3fc9-4220-9003-968805848ec2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/ctrl/", "title": "CTRL", "tokens": 1027, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"a52b2faa-3fc9-4220-9003-968805848ec2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/ctrl/\", \"title\": \"CTRL\", \"tokens\": 1027, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c7ad9229-4b27-4547-a68e-95bf3414bdd1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/ctrl/\", \"title\": \"CTRL\", \"tokens\": 1027, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"98edeae81590169ba84bec73f31bdd7e0d7e418715887c74f87214fda04dc417\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"69c85a2c-6250-4d54-9acf-d6edbbe8baed\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/ctrl/\", \"title\": \"CTRL\", \"tokens\": 1027, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"34691692b54443ed90155d0ea3bc5676f14b8053a3d0503f9b19cf8cb03a0987\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2867, \"end_char_idx\": 3600, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c7ad9229-4b27-4547-a68e-95bf3414bdd1", "doc_id": "c7ad9229-4b27-4547-a68e-95bf3414bdd1", "ref_doc_id": "c7ad9229-4b27-4547-a68e-95bf3414bdd1", "chroma:document": "## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Causal language modeling task guide](../tasks/language_modeling)\n\n## CTRLConfig\n\n[[autodoc]] CTRLConfig\n\n## CTRLTokenizer\n\n[[autodoc]] CTRLTokenizer\n    - save_vocabulary\n\n<frameworkcontent>\n<pt>\n\n## CTRLModel\n\n[[autodoc]] CTRLModel\n    - forward\n\n## CTRLLMHeadModel\n\n[[autodoc]] CTRLLMHeadModel\n    - forward\n\n## CTRLForSequenceClassification\n\n[[autodoc]] CTRLForSequenceClassification\n    - forward\n\n</pt>\n<tf>\n\n## TFCTRLModel\n\n[[autodoc]] TFCTRLModel\n    - call\n\n## TFCTRLLMHeadModel\n\n[[autodoc]] TFCTRLLMHeadModel\n    - call\n\n## TFCTRLForSequenceClassification\n\n[[autodoc]] TFCTRLForSequenceClassification\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: d4fe0adf-449d-4a55-abe6-e9eae7684b12
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/git/", "title": "GIT", "tokens": 888, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"d4fe0adf-449d-4a55-abe6-e9eae7684b12\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/git/\", \"title\": \"GIT\", \"tokens\": 888, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"728dd6d5-7c2b-48ad-b88f-7acedde5a47d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/git/\", \"title\": \"GIT\", \"tokens\": 888, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"9991feabf760a4a60cc6de008ed23be22b321f655538112c74dfd493ec6beb73\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3136, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "728dd6d5-7c2b-48ad-b88f-7acedde5a47d", "doc_id": "728dd6d5-7c2b-48ad-b88f-7acedde5a47d", "ref_doc_id": "728dd6d5-7c2b-48ad-b88f-7acedde5a47d", "chroma:document": "# GIT\n\n## Overview\n\nThe GIT model was proposed in [GIT: A Generative Image-to-text Transformer for Vision and Language](https://arxiv.org/abs/2205.14100) by\nJianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, Lijuan Wang. GIT is a decoder-only Transformer\nthat leverages [CLIP](clip)'s vision encoder to condition the model on vision inputs besides text. The model obtains state-of-the-art results on\nimage captioning and visual question answering benchmarks.\n\nThe abstract from the paper is the following:\n\n*In this paper, we design and train a Generative Image-to-text Transformer, GIT, to unify vision-language tasks such as image/video captioning and question answering. While generative models provide a consistent network architecture between pre-training and fine-tuning, existing work typically contains complex structures (uni/multi-modal encoder/decoder) and depends on external modules such as object detectors/taggers and optical character recognition (OCR). In GIT, we simplify the architecture as one image encoder and one text decoder under a single language modeling task. We also scale up the pre-training data and the model size to boost the model performance. Without bells and whistles, our GIT establishes new state of the arts on 12 challenging benchmarks with a large margin. For instance, our model surpasses the human performance for the first time on TextCaps (138.2 vs. 125.5 in CIDEr). Furthermore, we present a new scheme of generation-based image classification and scene text recognition, achieving decent performance on standard benchmarks.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/git_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> GIT architecture. Taken from the <a href=\"https://arxiv.org/abs/2205.14100\" target=\"_blank\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/microsoft/GenerativeImage2Text).\n\n## Usage tips\n\n- GIT is implemented in a very similar way to GPT-2, the only difference being that the model is also conditioned on `pixel_values`.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with GIT.\n\n- Demo notebooks regarding inference + fine-tuning GIT on custom data can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/GIT).\n- See also: [Causal language modeling task guide](../tasks/language_modeling)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it.\nThe resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## GitVisionConfig\n\n[[autodoc]] GitVisionConfig\n\n## GitVisionModel\n\n[[autodoc]] GitVisionModel\n    - forward\n\n## GitConfig\n\n[[autodoc]] GitConfig\n    - all\n\n## GitProcessor\n\n[[autodoc]] GitProcessor\n    - __call__\n\n## GitModel\n\n[[autodoc]] GitModel\n    - forward\n\n## GitForCausalLM\n\n[[autodoc]] GitForCausalLM\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 31e7d217-5213-447b-bb49-63089328d058
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/clap/", "title": "CLAP", "tokens": 857, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"31e7d217-5213-447b-bb49-63089328d058\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clap/\", \"title\": \"CLAP\", \"tokens\": 857, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"640990f0-1b8f-4f25-9444-e37d467ec908\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clap/\", \"title\": \"CLAP\", \"tokens\": 857, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"1bc00e50582335e4f69118cb16cdc805a169326d826e17529a3c3ac861344f17\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3011, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "640990f0-1b8f-4f25-9444-e37d467ec908", "doc_id": "640990f0-1b8f-4f25-9444-e37d467ec908", "ref_doc_id": "640990f0-1b8f-4f25-9444-e37d467ec908", "chroma:document": "# CLAP\n\n## Overview\n\nThe CLAP model was proposed in [Large Scale Contrastive Language-Audio pretraining with\nfeature fusion and keyword-to-caption augmentation](https://arxiv.org/pdf/2211.06687.pdf) by Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, Shlomo Dubnov.\n\nCLAP (Contrastive Language-Audio Pretraining) is a neural network trained on a variety of (audio, text) pairs. It can be instructed in to predict the most relevant text snippet, given an audio, without directly optimizing for the task. The CLAP model uses a SWINTransformer to get audio features from a log-Mel spectrogram input, and a RoBERTa model to get text features. Both the text and audio features are then projected to a latent space with identical dimension. The dot product between the projected audio and text features is then used as a similar score.\n\nThe abstract from the paper is the following:\n\n*Contrastive learning has shown remarkable success in the field of multimodal representation learning. In this paper, we propose a pipeline of contrastive language-audio pretraining to develop an audio representation by combining audio data with natural language descriptions. To accomplish this target, we first release LAION-Audio-630K, a large collection of 633,526 audio-text pairs from different data sources. Second, we construct a contrastive language-audio pretraining model by considering different audio encoders and text encoders. We incorporate the feature fusion mechanism and keyword-to-caption augmentation into the model design to further enable the model to process audio inputs of variable lengths and enhance the performance. Third, we perform comprehensive experiments to evaluate our model across three tasks: text-to-audio retrieval, zero-shot audio classification, and supervised audio classification. The results demonstrate that our model achieves superior performance in text-to-audio retrieval task. In audio classification tasks, the model achieves state-of-the-art performance in the zeroshot setting and is able to obtain performance comparable to models' results in the non-zero-shot setting. LAION-Audio-6*\n\nThis model was contributed by [Younes Belkada](https://huggingface.co/ybelkada) and [Arthur Zucker](https://huggingface.co/ArthurZ) .\nThe original code can be found [here](https://github.com/LAION-AI/Clap).\n\n## ClapConfig\n\n[[autodoc]] ClapConfig\n    - from_text_audio_configs\n\n## ClapTextConfig\n\n[[autodoc]] ClapTextConfig\n\n## ClapAudioConfig\n\n[[autodoc]] ClapAudioConfig\n\n## ClapFeatureExtractor\n\n[[autodoc]] ClapFeatureExtractor\n\n## ClapProcessor\n\n[[autodoc]] ClapProcessor\n\n## ClapModel\n\n[[autodoc]] ClapModel\n    - forward\n    - get_text_features\n    - get_audio_features\n\n## ClapTextModel\n\n[[autodoc]] ClapTextModel\n    - forward\n\n## ClapTextModelWithProjection\n\n[[autodoc]] ClapTextModelWithProjection\n    - forward\n\n## ClapAudioModel\n\n[[autodoc]] ClapAudioModel\n    - forward\n\n## ClapAudioModelWithProjection\n\n[[autodoc]] ClapAudioModelWithProjection\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: ae6ef57d-9e12-4cf6-a137-cd9e724db0d7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/clvp/", "title": "CLVP", "tokens": 1096, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"ae6ef57d-9e12-4cf6-a137-cd9e724db0d7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clvp/\", \"title\": \"CLVP\", \"tokens\": 1096, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e1a34160-fa37-49ce-8bd2-d5433b2e2520\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clvp/\", \"title\": \"CLVP\", \"tokens\": 1096, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"50aede24859417b94ac4baee45e0dc1414e7c2c3ce127d5086c85ff672e3255e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2156a421-3b4e-48a5-8627-dc6549d3b816\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"10b1b79716168c7d477c667436e603efe4a3b6755e957eb60bc18aafe6014e1d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3218, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e1a34160-fa37-49ce-8bd2-d5433b2e2520", "doc_id": "e1a34160-fa37-49ce-8bd2-d5433b2e2520", "ref_doc_id": "e1a34160-fa37-49ce-8bd2-d5433b2e2520", "chroma:document": "# CLVP\n\n## Overview\n\nThe CLVP (Contrastive Language-Voice Pretrained Transformer) model was proposed in [Better speech synthesis through scaling](https://arxiv.org/abs/2305.07243) by James Betker.\n\nThe abstract from the paper is the following:\n\n*In recent years, the field of image generation has been revolutionized by the application of autoregressive transformers and DDPMs. These approaches model the process of image generation as a step-wise probabilistic processes and leverage large amounts of compute and data to learn the image distribution. This methodology of improving performance need not be confined to images. This paper describes a way to apply advances in the image generative domain to speech synthesis. The result is TorToise - an expressive, multi-voice text-to-speech system.*\n\n\nThis model was contributed by [Susnato Dhar](https://huggingface.co/susnato).\nThe original code can be found [here](https://github.com/neonbjb/tortoise-tts).\n\n\n## Usage tips\n\n1. CLVP is an integral part of the Tortoise TTS model.\n2. CLVP can be used to compare different generated speech candidates with the provided text, and the best speech tokens are forwarded to the diffusion model.\n3. The use of the [`ClvpModelForConditionalGeneration.generate()`] method is strongly recommended for tortoise usage.\n4. Note that the CLVP model expects the audio to be sampled at 22.05 kHz contrary to other audio models which expects 16 kHz. \n\n\n## Brief Explanation:\n\n- The [`ClvpTokenizer`] tokenizes the text input, and the [`ClvpFeatureExtractor`] extracts the log mel-spectrogram from the desired audio.\n- [`ClvpConditioningEncoder`] takes those text tokens and audio representations and converts them into embeddings conditioned on the text and audio.\n- The [`ClvpForCausalLM`] uses those embeddings to generate multiple speech candidates.\n- Each speech candidate is passed through the speech encoder ([`ClvpEncoder`]) which converts them into a vector representation, and the text encoder ([`ClvpEncoder`]) converts the text tokens into the same latent space. \n- At the end, we compare each speech vector with the text vector to see which speech vector is most similar to the text vector. \n- [`ClvpModelForConditionalGeneration.generate()`] compresses all of the logic described above into a single method.  \n\n\nExample :\n\n```python\n>>> import datasets\n>>> from transformers import ClvpProcessor, ClvpModelForConditionalGeneration\n\n>>> # Define the Text and Load the Audio (We are taking an audio example from HuggingFace Hub using `datasets` library).\n>>> text = \"This is an example text.\"\n\n>>> ds = datasets.load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\n>>> ds = ds.cast_column(\"audio\", datasets.Audio(sampling_rate=22050))\n>>> sample = ds[0][\"audio\"]\n\n>>> # Define processor and model.\n>>> processor = ClvpProcessor.from_pretrained(\"susnato/clvp_dev\")\n>>> model = ClvpModelForConditionalGeneration.from_pretrained(\"susnato/clvp_dev\")\n\n>>> # Generate processor output and model output.\n>>> processor_output = processor(raw_speech=sample[\"array\"], sampling_rate=sample[\"sampling_rate\"], text=text, return_tensors=\"pt\")\n>>> generated_output = model.generate(**processor_output)\n```"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 2156a421-3b4e-48a5-8627-dc6549d3b816
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/clvp/", "title": "CLVP", "tokens": 1096, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"2156a421-3b4e-48a5-8627-dc6549d3b816\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clvp/\", \"title\": \"CLVP\", \"tokens\": 1096, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e1a34160-fa37-49ce-8bd2-d5433b2e2520\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clvp/\", \"title\": \"CLVP\", \"tokens\": 1096, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"50aede24859417b94ac4baee45e0dc1414e7c2c3ce127d5086c85ff672e3255e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ae6ef57d-9e12-4cf6-a137-cd9e724db0d7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clvp/\", \"title\": \"CLVP\", \"tokens\": 1096, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"939ae63c5cf46bbe4f9ff653c4c3a29697074cd2d91cedda40eed169be9c0882\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1436, \"end_char_idx\": 3958, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e1a34160-fa37-49ce-8bd2-d5433b2e2520", "doc_id": "e1a34160-fa37-49ce-8bd2-d5433b2e2520", "ref_doc_id": "e1a34160-fa37-49ce-8bd2-d5433b2e2520", "chroma:document": "## Brief Explanation:\n\n- The [`ClvpTokenizer`] tokenizes the text input, and the [`ClvpFeatureExtractor`] extracts the log mel-spectrogram from the desired audio.\n- [`ClvpConditioningEncoder`] takes those text tokens and audio representations and converts them into embeddings conditioned on the text and audio.\n- The [`ClvpForCausalLM`] uses those embeddings to generate multiple speech candidates.\n- Each speech candidate is passed through the speech encoder ([`ClvpEncoder`]) which converts them into a vector representation, and the text encoder ([`ClvpEncoder`]) converts the text tokens into the same latent space. \n- At the end, we compare each speech vector with the text vector to see which speech vector is most similar to the text vector. \n- [`ClvpModelForConditionalGeneration.generate()`] compresses all of the logic described above into a single method.  \n\n\nExample :\n\n```python\n>>> import datasets\n>>> from transformers import ClvpProcessor, ClvpModelForConditionalGeneration\n\n>>> # Define the Text and Load the Audio (We are taking an audio example from HuggingFace Hub using `datasets` library).\n>>> text = \"This is an example text.\"\n\n>>> ds = datasets.load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\n>>> ds = ds.cast_column(\"audio\", datasets.Audio(sampling_rate=22050))\n>>> sample = ds[0][\"audio\"]\n\n>>> # Define processor and model.\n>>> processor = ClvpProcessor.from_pretrained(\"susnato/clvp_dev\")\n>>> model = ClvpModelForConditionalGeneration.from_pretrained(\"susnato/clvp_dev\")\n\n>>> # Generate processor output and model output.\n>>> processor_output = processor(raw_speech=sample[\"array\"], sampling_rate=sample[\"sampling_rate\"], text=text, return_tensors=\"pt\")\n>>> generated_output = model.generate(**processor_output)\n```\n\n\n## ClvpConfig\n\n[[autodoc]] ClvpConfig\n    - from_sub_model_configs\n\n## ClvpEncoderConfig\n\n[[autodoc]] ClvpEncoderConfig\n\n## ClvpDecoderConfig\n\n[[autodoc]] ClvpDecoderConfig\n\n## ClvpTokenizer\n\n[[autodoc]] ClvpTokenizer\n    - save_vocabulary\n\n## ClvpFeatureExtractor\n\n[[autodoc]] ClvpFeatureExtractor\n    - __call__\n\n## ClvpProcessor\n\n[[autodoc]] ClvpProcessor\n    - __call__\n    - decode\n    - batch_decode\n\n## ClvpModelForConditionalGeneration\n\n[[autodoc]] ClvpModelForConditionalGeneration\n    - forward\n    - generate\n    - get_text_features\n    - get_speech_features\n\n## ClvpForCausalLM\n\n[[autodoc]] ClvpForCausalLM\n\n## ClvpModel\n\n[[autodoc]] ClvpModel\n\n## ClvpEncoder\n\n[[autodoc]] ClvpEncoder\n\n## ClvpDecoder\n\n[[autodoc]] ClvpDecoder"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: d2d23c93-155b-48f8-94b1-f9ef225b89ae
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/recurrent_gemma/", "title": "RecurrentGemma", "tokens": 505, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"d2d23c93-155b-48f8-94b1-f9ef225b89ae\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/recurrent_gemma/\", \"title\": \"RecurrentGemma\", \"tokens\": 505, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f0fbb907-d2ef-4eff-898f-d1b2886e477d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/recurrent_gemma/\", \"title\": \"RecurrentGemma\", \"tokens\": 505, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f01fa09aca8de0f18304c2e8a6b3fd118660ee4f49d5b2e47e2afb0d82af8532\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1507, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f0fbb907-d2ef-4eff-898f-d1b2886e477d", "doc_id": "f0fbb907-d2ef-4eff-898f-d1b2886e477d", "ref_doc_id": "f0fbb907-d2ef-4eff-898f-d1b2886e477d", "chroma:document": "# RecurrentGemma\n\n## Overview\n\nThe Recurrent Gemma model was proposed in [RecurrentGemma: Moving Past Transformers for Efficient Open Language Models](https://storage.googleapis.com/deepmind-media/gemma/recurrentgemma-report.pdf) by the Griffin, RLHF and Gemma Teams of Google.\n\nThe abstract from the paper is the following:\n\n*We introduce RecurrentGemma, an open language model which uses Google\u2019s novel Griffin architecture. Griffin combines linear recurrences with local attention to achieve excellent performance on language. It has a fixed-sized state, which reduces memory use and enables efficient inference on long sequences. We provide a pre-trained model with 2B non-embedding parameters, and an instruction tuned variant. Both models achieve comparable performance to Gemma-2B despite being trained on fewer tokens.*\n\nTips:\n\n- The original checkpoints can be converted using the conversion script [`src/transformers/models/recurrent_gemma/convert_recurrent_gemma_weights_to_hf.py`](https://github.com/huggingface/transformers/blob/main/src/transformers/models/recurrent_gemma/convert_recurrent_gemma_to_hf.py). \n\nThis model was contributed by [Arthur Zucker](https://huggingface.co/ArthurZ). The original code can be found [here](https://github.com/google-deepmind/recurrentgemma).\n\n\n## RecurrentGemmaConfig\n\n[[autodoc]] RecurrentGemmaConfig\n\n\n## RecurrentGemmaModel\n\n[[autodoc]] RecurrentGemmaModel\n    - forward\n\n## RecurrentGemmaForCausalLM\n\n[[autodoc]] RecurrentGemmaForCausalLM\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 9846223e-4b23-478b-a3aa-c23901bac116
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/detr/", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"9846223e-4b23-478b-a3aa-c23901bac116\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr/\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3ecda19b-c73b-45f4-a33c-29c52600449d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr/\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0dac43705e07d97e882b63b49d3cdd4663b5abf70534257a4dc77f60e93c2fa4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fe6066a3-1cea-462f-ba1d-7f2b772b55e8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7cc0bb5b80177fc43726039da8bbe670c7e2fc7261cb0c4965eaced23a0ba463\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3289, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3ecda19b-c73b-45f4-a33c-29c52600449d", "doc_id": "3ecda19b-c73b-45f4-a33c-29c52600449d", "ref_doc_id": "3ecda19b-c73b-45f4-a33c-29c52600449d", "chroma:document": "# DETR\n\n## Overview\n\nThe DETR model was proposed in [End-to-End Object Detection with Transformers](https://arxiv.org/abs/2005.12872) by\nNicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov and Sergey Zagoruyko. DETR\nconsists of a convolutional backbone followed by an encoder-decoder Transformer which can be trained end-to-end for\nobject detection. It greatly simplifies a lot of the complexity of models like Faster-R-CNN and Mask-R-CNN, which use\nthings like region proposals, non-maximum suppression procedure and anchor generation. Moreover, DETR can also be\nnaturally extended to perform panoptic segmentation, by simply adding a mask head on top of the decoder outputs.\n\nThe abstract from the paper is the following:\n\n*We present a new method that views object detection as a direct set prediction problem. Our approach streamlines the\ndetection pipeline, effectively removing the need for many hand-designed components like a non-maximum suppression\nprocedure or anchor generation that explicitly encode our prior knowledge about the task. The main ingredients of the\nnew framework, called DEtection TRansformer or DETR, are a set-based global loss that forces unique predictions via\nbipartite matching, and a transformer encoder-decoder architecture. Given a fixed small set of learned object queries,\nDETR reasons about the relations of the objects and the global image context to directly output the final set of\npredictions in parallel. The new model is conceptually simple and does not require a specialized library, unlike many\nother modern detectors. DETR demonstrates accuracy and run-time performance on par with the well-established and\nhighly-optimized Faster RCNN baseline on the challenging COCO object detection dataset. Moreover, DETR can be easily\ngeneralized to produce panoptic segmentation in a unified manner. We show that it significantly outperforms competitive\nbaselines.*\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/facebookresearch/detr).\n\n## How DETR works\n\nHere's a TLDR explaining how [`~transformers.DetrForObjectDetection`] works:\n\nFirst, an image is sent through a pre-trained convolutional backbone (in the paper, the authors use\nResNet-50/ResNet-101). Let's assume we also add a batch dimension. This means that the input to the backbone is a\ntensor of shape `(batch_size, 3, height, width)`, assuming the image has 3 color channels (RGB). The CNN backbone\noutputs a new lower-resolution feature map, typically of shape `(batch_size, 2048, height/32, width/32)`. This is\nthen projected to match the hidden dimension of the Transformer of DETR, which is `256` by default, using a\n`nn.Conv2D` layer. So now, we have a tensor of shape `(batch_size, 256, height/32, width/32).` Next, the\nfeature map is flattened and transposed to obtain a tensor of shape `(batch_size, seq_len, d_model)` =\n`(batch_size, width/32*height/32, 256)`. So a difference with NLP models is that the sequence length is actually\nlonger than usual, but with a smaller `d_model` (which in NLP is typically 768 or higher).\n\nNext, this is sent through the encoder, outputting `encoder_hidden_states` of the same shape (you can consider\nthese as image features)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: fe6066a3-1cea-462f-ba1d-7f2b772b55e8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/detr/", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"fe6066a3-1cea-462f-ba1d-7f2b772b55e8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr/\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3ecda19b-c73b-45f4-a33c-29c52600449d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr/\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0dac43705e07d97e882b63b49d3cdd4663b5abf70534257a4dc77f60e93c2fa4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9846223e-4b23-478b-a3aa-c23901bac116\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr/\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"9862c18c83e422aa3cd14d1b205dc353eb833b18eae0beb2f98fa704b28a9b40\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"169a465d-4e47-4e76-9fab-72f0d13fab1c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"23941821e25ce2b1aa060ae85a540c4ab9aca2358db261fdbc09e5e92424a9fd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1778, \"end_char_idx\": 5025, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3ecda19b-c73b-45f4-a33c-29c52600449d", "doc_id": "3ecda19b-c73b-45f4-a33c-29c52600449d", "ref_doc_id": "3ecda19b-c73b-45f4-a33c-29c52600449d", "chroma:document": "Moreover, DETR can be easily\ngeneralized to produce panoptic segmentation in a unified manner. We show that it significantly outperforms competitive\nbaselines.*\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/facebookresearch/detr).\n\n## How DETR works\n\nHere's a TLDR explaining how [`~transformers.DetrForObjectDetection`] works:\n\nFirst, an image is sent through a pre-trained convolutional backbone (in the paper, the authors use\nResNet-50/ResNet-101). Let's assume we also add a batch dimension. This means that the input to the backbone is a\ntensor of shape `(batch_size, 3, height, width)`, assuming the image has 3 color channels (RGB). The CNN backbone\noutputs a new lower-resolution feature map, typically of shape `(batch_size, 2048, height/32, width/32)`. This is\nthen projected to match the hidden dimension of the Transformer of DETR, which is `256` by default, using a\n`nn.Conv2D` layer. So now, we have a tensor of shape `(batch_size, 256, height/32, width/32).` Next, the\nfeature map is flattened and transposed to obtain a tensor of shape `(batch_size, seq_len, d_model)` =\n`(batch_size, width/32*height/32, 256)`. So a difference with NLP models is that the sequence length is actually\nlonger than usual, but with a smaller `d_model` (which in NLP is typically 768 or higher).\n\nNext, this is sent through the encoder, outputting `encoder_hidden_states` of the same shape (you can consider\nthese as image features). Next, so-called **object queries** are sent through the decoder. This is a tensor of shape\n`(batch_size, num_queries, d_model)`, with `num_queries` typically set to 100 and initialized with zeros.\nThese input embeddings are learnt positional encodings that the authors refer to as object queries, and similarly to\nthe encoder, they are added to the input of each attention layer. Each object query will look for a particular object\nin the image. The decoder updates these embeddings through multiple self-attention and encoder-decoder attention layers\nto output `decoder_hidden_states` of the same shape: `(batch_size, num_queries, d_model)`. Next, two heads\nare added on top for object detection: a linear layer for classifying each object query into one of the objects or \"no\nobject\", and a MLP to predict bounding boxes for each query.\n\nThe model is trained using a **bipartite matching loss**: so what we actually do is compare the predicted classes +\nbounding boxes of each of the N = 100 object queries to the ground truth annotations, padded up to the same length N\n(so if an image only contains 4 objects, 96 annotations will just have a \"no object\" as class and \"no bounding box\" as\nbounding box). The [Hungarian matching algorithm](https://en.wikipedia.org/wiki/Hungarian_algorithm) is used to find\nan optimal one-to-one mapping of each of the N queries to each of the N annotations. Next, standard cross-entropy (for\nthe classes) and a linear combination of the L1 and [generalized IoU loss](https://giou.stanford.edu/) (for the\nbounding boxes) are used to optimize the parameters of the model.\n\nDETR can be naturally extended to perform panoptic segmentation (which unifies semantic segmentation and instance\nsegmentation)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 169a465d-4e47-4e76-9fab-72f0d13fab1c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/detr/", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"169a465d-4e47-4e76-9fab-72f0d13fab1c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr/\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3ecda19b-c73b-45f4-a33c-29c52600449d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr/\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0dac43705e07d97e882b63b49d3cdd4663b5abf70534257a4dc77f60e93c2fa4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"fe6066a3-1cea-462f-ba1d-7f2b772b55e8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr/\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"c8ed1a285dcab7ed43636e6a91018b2714e23a26e9acc031303f0d7276949d2d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8e8d2ce8-7ff2-4cbc-8c59-2447ed395b8f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"981778274d11ee1aaa6676a47f03930acc7a7e47bf9b12c3828c00c246a9aa6f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3290, \"end_char_idx\": 6686, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3ecda19b-c73b-45f4-a33c-29c52600449d", "doc_id": "3ecda19b-c73b-45f4-a33c-29c52600449d", "ref_doc_id": "3ecda19b-c73b-45f4-a33c-29c52600449d", "chroma:document": "Next, so-called **object queries** are sent through the decoder. This is a tensor of shape\n`(batch_size, num_queries, d_model)`, with `num_queries` typically set to 100 and initialized with zeros.\nThese input embeddings are learnt positional encodings that the authors refer to as object queries, and similarly to\nthe encoder, they are added to the input of each attention layer. Each object query will look for a particular object\nin the image. The decoder updates these embeddings through multiple self-attention and encoder-decoder attention layers\nto output `decoder_hidden_states` of the same shape: `(batch_size, num_queries, d_model)`. Next, two heads\nare added on top for object detection: a linear layer for classifying each object query into one of the objects or \"no\nobject\", and a MLP to predict bounding boxes for each query.\n\nThe model is trained using a **bipartite matching loss**: so what we actually do is compare the predicted classes +\nbounding boxes of each of the N = 100 object queries to the ground truth annotations, padded up to the same length N\n(so if an image only contains 4 objects, 96 annotations will just have a \"no object\" as class and \"no bounding box\" as\nbounding box). The [Hungarian matching algorithm](https://en.wikipedia.org/wiki/Hungarian_algorithm) is used to find\nan optimal one-to-one mapping of each of the N queries to each of the N annotations. Next, standard cross-entropy (for\nthe classes) and a linear combination of the L1 and [generalized IoU loss](https://giou.stanford.edu/) (for the\nbounding boxes) are used to optimize the parameters of the model.\n\nDETR can be naturally extended to perform panoptic segmentation (which unifies semantic segmentation and instance\nsegmentation). [`~transformers.DetrForSegmentation`] adds a segmentation mask head on top of\n[`~transformers.DetrForObjectDetection`]. The mask head can be trained either jointly, or in a two steps process,\nwhere one first trains a [`~transformers.DetrForObjectDetection`] model to detect bounding boxes around both\n\"things\" (instances) and \"stuff\" (background things like trees, roads, sky), then freeze all the weights and train only\nthe mask head for 25 epochs. Experimentally, these two approaches give similar results. Note that predicting boxes is\nrequired for the training to be possible, since the Hungarian matching is computed using distances between boxes.\n\n## Usage tips\n\n- DETR uses so-called **object queries** to detect objects in an image. The number of queries determines the maximum\n  number of objects that can be detected in a single image, and is set to 100 by default (see parameter\n  `num_queries` of [`~transformers.DetrConfig`]). Note that it's good to have some slack (in COCO, the\n  authors used 100, while the maximum number of objects in a COCO image is ~70).\n- The decoder of DETR updates the query embeddings in parallel. This is different from language models like GPT-2,\n  which use autoregressive decoding instead of parallel. Hence, no causal attention mask is used.\n- DETR adds position embeddings to the hidden states at each self-attention and cross-attention layer before projecting\n  to queries and keys. For the position embeddings of the image, one can choose between fixed sinusoidal or learned\n  absolute position embeddings. By default, the parameter `position_embedding_type` of\n  [`~transformers.DetrConfig`] is set to `\"sine\"`."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 8e8d2ce8-7ff2-4cbc-8c59-2447ed395b8f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/detr/", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"8e8d2ce8-7ff2-4cbc-8c59-2447ed395b8f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr/\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3ecda19b-c73b-45f4-a33c-29c52600449d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr/\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0dac43705e07d97e882b63b49d3cdd4663b5abf70534257a4dc77f60e93c2fa4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"169a465d-4e47-4e76-9fab-72f0d13fab1c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr/\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"3702a37c3f6cf4844a5e0a6c785b0a858d47dbaf0a6396232e9dacc9785159c7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e7402523-1d0a-4be4-99c6-ebce49150bab\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e8abbb78a57232433b5617e627f1746cdae35cb8de3362664cd2bfdd92f3f0bc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5026, \"end_char_idx\": 8320, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3ecda19b-c73b-45f4-a33c-29c52600449d", "doc_id": "3ecda19b-c73b-45f4-a33c-29c52600449d", "ref_doc_id": "3ecda19b-c73b-45f4-a33c-29c52600449d", "chroma:document": "[`~transformers.DetrForSegmentation`] adds a segmentation mask head on top of\n[`~transformers.DetrForObjectDetection`]. The mask head can be trained either jointly, or in a two steps process,\nwhere one first trains a [`~transformers.DetrForObjectDetection`] model to detect bounding boxes around both\n\"things\" (instances) and \"stuff\" (background things like trees, roads, sky), then freeze all the weights and train only\nthe mask head for 25 epochs. Experimentally, these two approaches give similar results. Note that predicting boxes is\nrequired for the training to be possible, since the Hungarian matching is computed using distances between boxes.\n\n## Usage tips\n\n- DETR uses so-called **object queries** to detect objects in an image. The number of queries determines the maximum\n  number of objects that can be detected in a single image, and is set to 100 by default (see parameter\n  `num_queries` of [`~transformers.DetrConfig`]). Note that it's good to have some slack (in COCO, the\n  authors used 100, while the maximum number of objects in a COCO image is ~70).\n- The decoder of DETR updates the query embeddings in parallel. This is different from language models like GPT-2,\n  which use autoregressive decoding instead of parallel. Hence, no causal attention mask is used.\n- DETR adds position embeddings to the hidden states at each self-attention and cross-attention layer before projecting\n  to queries and keys. For the position embeddings of the image, one can choose between fixed sinusoidal or learned\n  absolute position embeddings. By default, the parameter `position_embedding_type` of\n  [`~transformers.DetrConfig`] is set to `\"sine\"`.\n- During training, the authors of DETR did find it helpful to use auxiliary losses in the decoder, especially to help\n  the model output the correct number of objects of each class. If you set the parameter `auxiliary_loss` of\n  [`~transformers.DetrConfig`] to `True`, then prediction feedforward neural networks and Hungarian losses\n  are added after each decoder layer (with the FFNs sharing parameters).\n- If you want to train the model in a distributed environment across multiple nodes, then one should update the\n  _num_boxes_ variable in the _DetrLoss_ class of _modeling_detr.py_. When training on multiple nodes, this should be\n  set to the average number of target boxes across all nodes, as can be seen in the original implementation [here](https://github.com/facebookresearch/detr/blob/a54b77800eb8e64e3ad0d8237789fcbf2f8350c5/models/detr.py#L227-L232).\n- [`~transformers.DetrForObjectDetection`] and [`~transformers.DetrForSegmentation`] can be initialized with\n  any convolutional backbone available in the [timm library](https://github.com/rwightman/pytorch-image-models).\n  Initializing with a MobileNet backbone for example can be done by setting the `backbone` attribute of\n  [`~transformers.DetrConfig`] to `\"tf_mobilenetv3_small_075\"`, and then initializing the model with that\n  config.\n- DETR resizes the input images such that the shortest side is at least a certain amount of pixels while the longest is\n  at most 1333 pixels. At training time, scale augmentation is used such that the shortest side is randomly set to at\n  least 480 and at most 800 pixels. At inference time, the shortest side is set to 800."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: e7402523-1d0a-4be4-99c6-ebce49150bab
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/detr/", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"e7402523-1d0a-4be4-99c6-ebce49150bab\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr/\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3ecda19b-c73b-45f4-a33c-29c52600449d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr/\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0dac43705e07d97e882b63b49d3cdd4663b5abf70534257a4dc77f60e93c2fa4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8e8d2ce8-7ff2-4cbc-8c59-2447ed395b8f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr/\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"dafec49496f3a7a1ff2a3371b153c4f49b686f057c3b51118e0aa366b212fb5e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"59201cc9-7d7f-411a-bcdd-4dbe8d1eda45\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6659784809a4f14b87e33413e6ae2022780aaaae3758eabb9f9574ab3e1e0b60\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6869, \"end_char_idx\": 10172, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3ecda19b-c73b-45f4-a33c-29c52600449d", "doc_id": "3ecda19b-c73b-45f4-a33c-29c52600449d", "ref_doc_id": "3ecda19b-c73b-45f4-a33c-29c52600449d", "chroma:document": "If you set the parameter `auxiliary_loss` of\n  [`~transformers.DetrConfig`] to `True`, then prediction feedforward neural networks and Hungarian losses\n  are added after each decoder layer (with the FFNs sharing parameters).\n- If you want to train the model in a distributed environment across multiple nodes, then one should update the\n  _num_boxes_ variable in the _DetrLoss_ class of _modeling_detr.py_. When training on multiple nodes, this should be\n  set to the average number of target boxes across all nodes, as can be seen in the original implementation [here](https://github.com/facebookresearch/detr/blob/a54b77800eb8e64e3ad0d8237789fcbf2f8350c5/models/detr.py#L227-L232).\n- [`~transformers.DetrForObjectDetection`] and [`~transformers.DetrForSegmentation`] can be initialized with\n  any convolutional backbone available in the [timm library](https://github.com/rwightman/pytorch-image-models).\n  Initializing with a MobileNet backbone for example can be done by setting the `backbone` attribute of\n  [`~transformers.DetrConfig`] to `\"tf_mobilenetv3_small_075\"`, and then initializing the model with that\n  config.\n- DETR resizes the input images such that the shortest side is at least a certain amount of pixels while the longest is\n  at most 1333 pixels. At training time, scale augmentation is used such that the shortest side is randomly set to at\n  least 480 and at most 800 pixels. At inference time, the shortest side is set to 800. One can use\n  [`~transformers.DetrImageProcessor`] to prepare images (and optional annotations in COCO format) for the\n  model. Due to this resizing, images in a batch can have different sizes. DETR solves this by padding images up to the\n  largest size in a batch, and by creating a pixel mask that indicates which pixels are real/which are padding.\n  Alternatively, one can also define a custom `collate_fn` in order to batch images together, using\n  [`~transformers.DetrImageProcessor.pad_and_create_pixel_mask`].\n- The size of the images will determine the amount of memory being used, and will thus determine the `batch_size`.\n  It is advised to use a batch size of 2 per GPU. See [this Github thread](https://github.com/facebookresearch/detr/issues/150) for more info.\n\nThere are three ways to instantiate a DETR model (depending on what you prefer):\n\nOption 1: Instantiate DETR with pre-trained weights for entire model\n```py\n>>> from transformers import DetrForObjectDetection\n\n>>> model = DetrForObjectDetection.from_pretrained(\"facebook/detr-resnet-50\")\n```\n\nOption 2: Instantiate DETR with randomly initialized weights for Transformer, but pre-trained weights for backbone\n```py\n>>> from transformers import DetrConfig, DetrForObjectDetection\n\n>>> config = DetrConfig()\n>>> model = DetrForObjectDetection(config)\n```\nOption 3: Instantiate DETR with randomly initialized weights for backbone + Transformer\n```py\n>>> config = DetrConfig(use_pretrained_backbone=False)\n>>> model = DetrForObjectDetection(config)\n```\n\nAs a summary, consider the following table:\n\n| Task | Object detection | Instance segmentation | Panoptic segmentation |\n|------|------------------|-----------------------|-----------------------|\n| **Description** | Predicting bounding boxes and class labels around objects in an image | Predicting masks around objects (i.e."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 59201cc9-7d7f-411a-bcdd-4dbe8d1eda45
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/detr/", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"59201cc9-7d7f-411a-bcdd-4dbe8d1eda45\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr/\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3ecda19b-c73b-45f4-a33c-29c52600449d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr/\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0dac43705e07d97e882b63b49d3cdd4663b5abf70534257a4dc77f60e93c2fa4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e7402523-1d0a-4be4-99c6-ebce49150bab\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr/\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"362717a3b94b0ea18caaa071a40544a8d38859e4e878009159ac29a92dc7f919\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f82c71aa-9290-4650-832d-756ef4e5e456\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c095f323596f810559cad014578152e749735cd92decc98e9f4cd0d7b7291f54\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8449, \"end_char_idx\": 11155, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3ecda19b-c73b-45f4-a33c-29c52600449d", "doc_id": "3ecda19b-c73b-45f4-a33c-29c52600449d", "ref_doc_id": "3ecda19b-c73b-45f4-a33c-29c52600449d", "chroma:document": "Due to this resizing, images in a batch can have different sizes. DETR solves this by padding images up to the\n  largest size in a batch, and by creating a pixel mask that indicates which pixels are real/which are padding.\n  Alternatively, one can also define a custom `collate_fn` in order to batch images together, using\n  [`~transformers.DetrImageProcessor.pad_and_create_pixel_mask`].\n- The size of the images will determine the amount of memory being used, and will thus determine the `batch_size`.\n  It is advised to use a batch size of 2 per GPU. See [this Github thread](https://github.com/facebookresearch/detr/issues/150) for more info.\n\nThere are three ways to instantiate a DETR model (depending on what you prefer):\n\nOption 1: Instantiate DETR with pre-trained weights for entire model\n```py\n>>> from transformers import DetrForObjectDetection\n\n>>> model = DetrForObjectDetection.from_pretrained(\"facebook/detr-resnet-50\")\n```\n\nOption 2: Instantiate DETR with randomly initialized weights for Transformer, but pre-trained weights for backbone\n```py\n>>> from transformers import DetrConfig, DetrForObjectDetection\n\n>>> config = DetrConfig()\n>>> model = DetrForObjectDetection(config)\n```\nOption 3: Instantiate DETR with randomly initialized weights for backbone + Transformer\n```py\n>>> config = DetrConfig(use_pretrained_backbone=False)\n>>> model = DetrForObjectDetection(config)\n```\n\nAs a summary, consider the following table:\n\n| Task | Object detection | Instance segmentation | Panoptic segmentation |\n|------|------------------|-----------------------|-----------------------|\n| **Description** | Predicting bounding boxes and class labels around objects in an image | Predicting masks around objects (i.e. instances) in an image | Predicting masks around both objects (i.e. instances) as well as \"stuff\" (i.e. background things like trees and roads) in an image |\n| **Model** | [`~transformers.DetrForObjectDetection`] | [`~transformers.DetrForSegmentation`] | [`~transformers.DetrForSegmentation`] |\n| **Example dataset** | COCO detection | COCO detection, COCO panoptic | COCO panoptic  |                                                                        |\n| **Format of annotations to provide to**  [`~transformers.DetrImageProcessor`] | {'image_id': `int`, 'annotations': `List[Dict]`} each Dict being a COCO object annotation  | {'image_id': `int`, 'annotations': `List[Dict]`}  (in case of COCO detection) or {'file_name': `str`, 'image_id': `int`, 'segments_info': `List[Dict]`} (in case of COCO panoptic) | {'file_name': `str`, 'image_id': `int`, 'segments_info': `List[Dict]`} and masks_path (path to directory containing PNG files of the masks) |\n| **Postprocessing** (i.e."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: f82c71aa-9290-4650-832d-756ef4e5e456
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/detr/", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"f82c71aa-9290-4650-832d-756ef4e5e456\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr/\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3ecda19b-c73b-45f4-a33c-29c52600449d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr/\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0dac43705e07d97e882b63b49d3cdd4663b5abf70534257a4dc77f60e93c2fa4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"59201cc9-7d7f-411a-bcdd-4dbe8d1eda45\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr/\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"3add9354cf0f8a9ed3d326240908388467b1be762fa047f8f5508bff1146f3fe\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"187612dd-2941-46b1-bc41-e060c3578239\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"160e1d9ce59d6d183ea4fafeaca9384dca7efdad9bec24cfb6e4c439882240ac\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10173, \"end_char_idx\": 13285, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3ecda19b-c73b-45f4-a33c-29c52600449d", "doc_id": "3ecda19b-c73b-45f4-a33c-29c52600449d", "ref_doc_id": "3ecda19b-c73b-45f4-a33c-29c52600449d", "chroma:document": "instances) in an image | Predicting masks around both objects (i.e. instances) as well as \"stuff\" (i.e. background things like trees and roads) in an image |\n| **Model** | [`~transformers.DetrForObjectDetection`] | [`~transformers.DetrForSegmentation`] | [`~transformers.DetrForSegmentation`] |\n| **Example dataset** | COCO detection | COCO detection, COCO panoptic | COCO panoptic  |                                                                        |\n| **Format of annotations to provide to**  [`~transformers.DetrImageProcessor`] | {'image_id': `int`, 'annotations': `List[Dict]`} each Dict being a COCO object annotation  | {'image_id': `int`, 'annotations': `List[Dict]`}  (in case of COCO detection) or {'file_name': `str`, 'image_id': `int`, 'segments_info': `List[Dict]`} (in case of COCO panoptic) | {'file_name': `str`, 'image_id': `int`, 'segments_info': `List[Dict]`} and masks_path (path to directory containing PNG files of the masks) |\n| **Postprocessing** (i.e. converting the output of the model to Pascal VOC format) | [`~transformers.DetrImageProcessor.post_process`] | [`~transformers.DetrImageProcessor.post_process_segmentation`] | [`~transformers.DetrImageProcessor.post_process_segmentation`], [`~transformers.DetrImageProcessor.post_process_panoptic`] |\n| **evaluators** | `CocoEvaluator` with `iou_types=\"bbox\"` | `CocoEvaluator` with `iou_types=\"bbox\"` or `\"segm\"` | `CocoEvaluator` with `iou_tupes=\"bbox\"` or `\"segm\"`, `PanopticEvaluator` |\n\nIn short, one should prepare the data either in COCO detection or COCO panoptic format, then use\n[`~transformers.DetrImageProcessor`] to create `pixel_values`, `pixel_mask` and optional\n`labels`, which can then be used to train (or fine-tune) a model. For evaluation, one should first convert the\noutputs of the model using one of the postprocessing methods of [`~transformers.DetrImageProcessor`]. These can\nbe be provided to either `CocoEvaluator` or `PanopticEvaluator`, which allow you to calculate metrics like\nmean Average Precision (mAP) and Panoptic Quality (PQ). The latter objects are implemented in the [original repository](https://github.com/facebookresearch/detr). See the [example notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DETR) for more info regarding evaluation.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with DETR.\n\n<PipelineTag pipeline=\"object-detection\"/>\n\n- All example notebooks illustrating fine-tuning [`DetrForObjectDetection`] and [`DetrForSegmentation`] on a custom dataset can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DETR).\n- Scripts for finetuning [`DetrForObjectDetection`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/object-detection).\n- See also: [Object detection task guide](../tasks/object_detection).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it!"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 187612dd-2941-46b1-bc41-e060c3578239
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/detr/", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"187612dd-2941-46b1-bc41-e060c3578239\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr/\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3ecda19b-c73b-45f4-a33c-29c52600449d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr/\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0dac43705e07d97e882b63b49d3cdd4663b5abf70534257a4dc77f60e93c2fa4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f82c71aa-9290-4650-832d-756ef4e5e456\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/detr/\", \"title\": \"DETR\", \"tokens\": 3494, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"4f8ed5a8e6ba5bde625d23ed6f7673feb6512adeceb9e1d23c84da6ce20ae5e2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11900, \"end_char_idx\": 14294, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3ecda19b-c73b-45f4-a33c-29c52600449d", "doc_id": "3ecda19b-c73b-45f4-a33c-29c52600449d", "ref_doc_id": "3ecda19b-c73b-45f4-a33c-29c52600449d", "chroma:document": "For evaluation, one should first convert the\noutputs of the model using one of the postprocessing methods of [`~transformers.DetrImageProcessor`]. These can\nbe be provided to either `CocoEvaluator` or `PanopticEvaluator`, which allow you to calculate metrics like\nmean Average Precision (mAP) and Panoptic Quality (PQ). The latter objects are implemented in the [original repository](https://github.com/facebookresearch/detr). See the [example notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DETR) for more info regarding evaluation.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with DETR.\n\n<PipelineTag pipeline=\"object-detection\"/>\n\n- All example notebooks illustrating fine-tuning [`DetrForObjectDetection`] and [`DetrForSegmentation`] on a custom dataset can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DETR).\n- Scripts for finetuning [`DetrForObjectDetection`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/object-detection).\n- See also: [Object detection task guide](../tasks/object_detection).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## DetrConfig\n\n[[autodoc]] DetrConfig\n\n## DetrImageProcessor\n\n[[autodoc]] DetrImageProcessor\n    - preprocess\n    - post_process_object_detection\n    - post_process_semantic_segmentation\n    - post_process_instance_segmentation\n    - post_process_panoptic_segmentation\n\n## DetrFeatureExtractor\n\n[[autodoc]] DetrFeatureExtractor\n    - __call__\n    - post_process_object_detection\n    - post_process_semantic_segmentation\n    - post_process_instance_segmentation\n    - post_process_panoptic_segmentation\n\n## DETR specific outputs\n\n[[autodoc]] models.detr.modeling_detr.DetrModelOutput\n\n[[autodoc]] models.detr.modeling_detr.DetrObjectDetectionOutput\n\n[[autodoc]] models.detr.modeling_detr.DetrSegmentationOutput\n\n## DetrModel\n\n[[autodoc]] DetrModel\n    - forward\n\n## DetrForObjectDetection\n\n[[autodoc]] DetrForObjectDetection\n    - forward\n\n## DetrForSegmentation\n\n[[autodoc]] DetrForSegmentation\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 814b4690-5a4e-4bf0-9e02-46e9386442bd
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xlnet/", "title": "XLNet", "tokens": 1707, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"814b4690-5a4e-4bf0-9e02-46e9386442bd\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlnet/\", \"title\": \"XLNet\", \"tokens\": 1707, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e66800f8-85ff-4e3f-91f0-74ee952b4c62\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlnet/\", \"title\": \"XLNet\", \"tokens\": 1707, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"1f09815cf50e614ea530014995f54175ff337f6d66bdff9a638df0ca1088d1da\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5e57bc7b-cda4-4171-8424-c262a0ae44f8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6852a40e77a730b72069745d1c2dca914293210f9552d58125fa06d09af8b6de\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3283, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e66800f8-85ff-4e3f-91f0-74ee952b4c62", "doc_id": "e66800f8-85ff-4e3f-91f0-74ee952b4c62", "ref_doc_id": "e66800f8-85ff-4e3f-91f0-74ee952b4c62", "chroma:document": "# XLNet\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=xlnet\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-xlnet-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/xlnet-base-cased\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nThe XLNet model was proposed in [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://arxiv.org/abs/1906.08237) by Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov,\nQuoc V. Le. XLnet is an extension of the Transformer-XL model pre-trained using an autoregressive method to learn\nbidirectional contexts by maximizing the expected likelihood over all permutations of the input sequence factorization\norder.\n\nThe abstract from the paper is the following:\n\n*With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves\nbetter performance than pretraining approaches based on autoregressive language modeling. However, relying on\ncorrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a\npretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive\npretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all\npermutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive\nformulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into\npretraining. Empirically, under comparable experiment settings, XLNet outperforms BERT on 20 tasks, often by a large\nmargin, including question answering, natural language inference, sentiment analysis, and document ranking.*\n\nThis model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/zihangdai/xlnet/).\n\n## Usage tips\n\n- The specific attention pattern can be controlled at training and test time using the `perm_mask` input.\n- Due to the difficulty of training a fully auto-regressive model over various factorization order, XLNet is pretrained\n  using only a sub-set of the output tokens as target which are selected with the `target_mapping` input.\n- To use XLNet for sequential decoding (i.e. not in fully bi-directional setting), use the `perm_mask` and\n  `target_mapping` inputs to control the attention span and outputs (see examples in\n  *examples/pytorch/text-generation/run_generation.py*)\n- XLNet is one of the few models that has no sequence length limit.\n- XLNet is not a traditional autoregressive model but uses a training strategy that builds on that. It permutes the tokens in the sentence, then allows the model to use the last n tokens to predict the token n+1. Since this is all done with a mask, the sentence is actually fed in the model in the right order, but instead of masking the first n tokens for n+1, XLNet uses a mask that hides the previous tokens in some given permutation of 1,\u2026,sequence length.\n- XLNet also uses the same recurrence mechanism as Transformer-XL to build long-term dependencies."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 5e57bc7b-cda4-4171-8424-c262a0ae44f8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xlnet/", "title": "XLNet", "tokens": 1707, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"5e57bc7b-cda4-4171-8424-c262a0ae44f8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlnet/\", \"title\": \"XLNet\", \"tokens\": 1707, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e66800f8-85ff-4e3f-91f0-74ee952b4c62\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlnet/\", \"title\": \"XLNet\", \"tokens\": 1707, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"1f09815cf50e614ea530014995f54175ff337f6d66bdff9a638df0ca1088d1da\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"814b4690-5a4e-4bf0-9e02-46e9386442bd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlnet/\", \"title\": \"XLNet\", \"tokens\": 1707, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"2544b40c5569547b412b232afddea4f71f6b6b25ed62c3e9049a992317bced35\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1577, \"end_char_idx\": 5969, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e66800f8-85ff-4e3f-91f0-74ee952b4c62", "doc_id": "e66800f8-85ff-4e3f-91f0-74ee952b4c62", "ref_doc_id": "e66800f8-85ff-4e3f-91f0-74ee952b4c62", "chroma:document": "Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into\npretraining. Empirically, under comparable experiment settings, XLNet outperforms BERT on 20 tasks, often by a large\nmargin, including question answering, natural language inference, sentiment analysis, and document ranking.*\n\nThis model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/zihangdai/xlnet/).\n\n## Usage tips\n\n- The specific attention pattern can be controlled at training and test time using the `perm_mask` input.\n- Due to the difficulty of training a fully auto-regressive model over various factorization order, XLNet is pretrained\n  using only a sub-set of the output tokens as target which are selected with the `target_mapping` input.\n- To use XLNet for sequential decoding (i.e. not in fully bi-directional setting), use the `perm_mask` and\n  `target_mapping` inputs to control the attention span and outputs (see examples in\n  *examples/pytorch/text-generation/run_generation.py*)\n- XLNet is one of the few models that has no sequence length limit.\n- XLNet is not a traditional autoregressive model but uses a training strategy that builds on that. It permutes the tokens in the sentence, then allows the model to use the last n tokens to predict the token n+1. Since this is all done with a mask, the sentence is actually fed in the model in the right order, but instead of masking the first n tokens for n+1, XLNet uses a mask that hides the previous tokens in some given permutation of 1,\u2026,sequence length.\n- XLNet also uses the same recurrence mechanism as Transformer-XL to build long-term dependencies.\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## XLNetConfig\n\n[[autodoc]] XLNetConfig\n\n## XLNetTokenizer\n\n[[autodoc]] XLNetTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## XLNetTokenizerFast\n\n[[autodoc]] XLNetTokenizerFast\n\n## XLNet specific outputs\n\n[[autodoc]] models.xlnet.modeling_xlnet.XLNetModelOutput\n\n[[autodoc]] models.xlnet.modeling_xlnet.XLNetLMHeadModelOutput\n\n[[autodoc]] models.xlnet.modeling_xlnet.XLNetForSequenceClassificationOutput\n\n[[autodoc]] models.xlnet.modeling_xlnet.XLNetForMultipleChoiceOutput\n\n[[autodoc]] models.xlnet.modeling_xlnet.XLNetForTokenClassificationOutput\n\n[[autodoc]] models.xlnet.modeling_xlnet.XLNetForQuestionAnsweringSimpleOutput\n\n[[autodoc]] models.xlnet.modeling_xlnet.XLNetForQuestionAnsweringOutput\n\n[[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetModelOutput\n\n[[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetLMHeadModelOutput\n\n[[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetForSequenceClassificationOutput\n\n[[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetForMultipleChoiceOutput\n\n[[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetForTokenClassificationOutput\n\n[[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetForQuestionAnsweringSimpleOutput\n\n<frameworkcontent>\n<pt>\n\n## XLNetModel\n\n[[autodoc]] XLNetModel\n    - forward\n\n## XLNetLMHeadModel\n\n[[autodoc]] XLNetLMHeadModel\n    - forward\n\n## XLNetForSequenceClassification\n\n[[autodoc]] XLNetForSequenceClassification\n    - forward\n\n## XLNetForMultipleChoice\n\n[[autodoc]] XLNetForMultipleChoice\n    - forward\n\n## XLNetForTokenClassification\n\n[[autodoc]] XLNetForTokenClassification\n    - forward\n\n## XLNetForQuestionAnsweringSimple\n\n[[autodoc]] XLNetForQuestionAnsweringSimple\n    - forward\n\n## XLNetForQuestionAnswering\n\n[[autodoc]] XLNetForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFXLNetModel\n\n[[autodoc]] TFXLNetModel\n    - call\n\n## TFXLNetLMHeadModel\n\n[[autodoc]] TFXLNetLMHeadModel\n    - call\n\n## TFXLNetForSequenceClassification\n\n[[autodoc]] TFXLNetForSequenceClassification\n    - call\n\n## TFLNetForMultipleChoice\n\n[[autodoc]] TFXLNetForMultipleChoice\n    - call\n\n## TFXLNetForTokenClassification\n\n[[autodoc]] TFXLNetForTokenClassification\n    - call\n\n## TFXLNetForQuestionAnsweringSimple\n\n[[autodoc]] TFXLNetForQuestionAnsweringSimple\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 78e43ddd-3213-4268-a5f6-ccb808e3cf43
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/esm/", "title": "ESM", "tokens": 1680, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"78e43ddd-3213-4268-a5f6-ccb808e3cf43\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/esm/\", \"title\": \"ESM\", \"tokens\": 1680, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1e0eb0e7-a29c-45eb-bf9d-ddc680960939\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/esm/\", \"title\": \"ESM\", \"tokens\": 1680, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"7d65f87bda1d85b5f7510d98b948f32ec8528c9d507578cdac6cca58ba536b45\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"50d7ceed-a11d-4bf9-86df-39c236274b6a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0b564a1142000f05d18fb6d5c78e5f38ee094ba146a5af9cefc2af2d15d4a299\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3707, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1e0eb0e7-a29c-45eb-bf9d-ddc680960939", "doc_id": "1e0eb0e7-a29c-45eb-bf9d-ddc680960939", "ref_doc_id": "1e0eb0e7-a29c-45eb-bf9d-ddc680960939", "chroma:document": "# ESM\n\n## Overview\n\nThis page provides code and pre-trained weights for Transformer protein language models from Meta AI's Fundamental \nAI Research Team, providing the state-of-the-art ESMFold and ESM-2, and the previously released ESM-1b and ESM-1v.\nTransformer protein language models were introduced in the paper [Biological structure and function emerge from scaling\nunsupervised learning to 250 million protein sequences](https://www.pnas.org/content/118/15/e2016239118) by \nAlexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, \nC. Lawrence Zitnick, Jerry Ma, and Rob Fergus.\nThe first version of this paper was [preprinted in 2019](https://www.biorxiv.org/content/10.1101/622803v1?versioned=true).\n\nESM-2 outperforms all tested single-sequence protein language models across a range of structure prediction tasks,\nand enables atomic resolution structure prediction.\nIt was released with the paper [Language models of protein sequences at the scale of evolution enable accurate\nstructure prediction](https://doi.org/10.1101/2022.07.20.500902) by Zeming Lin, Halil Akin, Roshan Rao, Brian Hie,\nZhongkai Zhu, Wenting Lu, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Sal Candido and Alexander Rives.\n\nAlso introduced in this paper was ESMFold. It uses an ESM-2 stem with a head that can predict folded protein\nstructures with state-of-the-art accuracy. Unlike [AlphaFold2](https://www.nature.com/articles/s41586-021-03819-2),\nit relies on the token embeddings from the large pre-trained protein language model stem and does not perform a multiple\nsequence alignment (MSA) step at inference time, which means that ESMFold checkpoints are fully \"standalone\" -\nthey do not require a database of known protein sequences and structures with associated external query tools\nto make predictions, and are much faster as a result.\n\n\nThe abstract from \n\"Biological structure and function emerge from scaling unsupervised learning to 250 \nmillion protein sequences\" is\n\n\n*In the field of artificial intelligence, a combination of scale in data and model capacity enabled by unsupervised\nlearning has led to major advances in representation learning and statistical generation. In the life sciences, the\nanticipated growth of sequencing promises unprecedented data on natural sequence diversity. Protein language modeling\nat the scale of evolution is a logical step toward predictive and generative artificial intelligence for biology. To\nthis end, we use unsupervised learning to train a deep contextual language model on 86 billion amino acids across 250\nmillion protein sequences spanning evolutionary diversity. The resulting model contains information about biological\nproperties in its representations. The representations are learned from sequence data alone. The learned representation\nspace has a multiscale organization reflecting structure from the level of biochemical properties of amino acids to\nremote homology of proteins. Information about secondary and tertiary structure is encoded in the representations and\ncan be identified by linear projections. Representation learning produces features that generalize across a range of\napplications, enabling state-of-the-art supervised prediction of mutational effect and secondary structure and\nimproving state-of-the-art features for long-range contact prediction.*\n\n\nThe abstract from\n\"Language models of protein sequences at the scale of evolution enable accurate structure prediction\" is\n\n*Large language models have recently been shown to develop emergent capabilities with scale, going beyond\nsimple pattern matching to perform higher level reasoning and generate lifelike images and text."}


