# embeddings_queue

created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 58dbaaec-acf8-4430-9132-d386d18f811a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mixtral/", "title": "Mixtral", "tokens": 2917, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"58dbaaec-acf8-4430-9132-d386d18f811a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mixtral/\", \"title\": \"Mixtral\", \"tokens\": 2917, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d78e4927-3627-44c1-a2b6-974170c15b8f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mixtral/\", \"title\": \"Mixtral\", \"tokens\": 2917, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"731148c4922aae35ef1f94d656dac38ca59e0bd09610e1e8e358b60d8a3c9f88\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f28142b1-e954-473f-8dea-aa08eba17884\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mixtral/\", \"title\": \"Mixtral\", \"tokens\": 2917, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"544f8cbb1c1c0f2a815189b4a69ff923d77f845fbd9d833d0cccaca1680d8292\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8b365fac-f5b6-48dc-a514-7c531b810557\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"957be057f0f06ad3fa9d2efe0777af5b226d5de8bf362b27111e1582957176d5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6352, \"end_char_idx\": 9452, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d78e4927-3627-44c1-a2b6-974170c15b8f", "doc_id": "d78e4927-3627-44c1-a2b6-974170c15b8f", "ref_doc_id": "d78e4927-3627-44c1-a2b6-974170c15b8f", "chroma:document": "<div style=\"text-align: center\">\n<img src=\"https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/mixtral-7b-inference-large-seqlen.png\">\n</div>\n\n### Sliding window Attention\n\nThe current implementation supports the sliding window attention mechanism and memory efficient cache management. \nTo enable sliding window attention, just make sure to have a `flash-attn` version that is compatible with sliding window attention (`>=2.3.0`). \n\nThe Flash Attention-2 model uses also a more memory efficient cache slicing mechanism - as recommended per the official implementation of Mistral model that use rolling cache mechanism we keep the cache size fixed (`self.config.sliding_window`), support batched generation only for `padding_side=\"left\"` and use the absolute position of the current token to compute the positional embedding.\n\n## Shrinking down Mixtral using quantization\n\nAs the Mixtral model has 45 billion parameters, that would require about 90GB of GPU RAM in half precision (float16), since each parameter is stored in 2 bytes. However, one can shrink down the size of the model using [quantization](../quantization.md). If the model is quantized to 4 bits (or half a byte per parameter), a single A100 with 40GB of RAM is enough to fit the entire model, as in that case only about 27 GB of RAM is required.\n\nQuantizing a model is as simple as passing a `quantization_config` to the model. Below, we'll leverage the BitsAndyBytes quantization (but refer to [this page](../quantization.md) for other quantization methods):\n\n```python\n>>> import torch\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\n\n>>> # specify how to quantize the model\n>>> quantization_config = BitsAndBytesConfig(\n...         load_in_4bit=True,\n...         bnb_4bit_quant_type=\"nf4\",\n...         bnb_4bit_compute_dtype=\"torch.float16\",\n... )\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"mistralai/Mixtral-8x7B-Instruct-v0.1\", quantization_config=True, device_map=\"auto\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"mistralai/Mixtral-8x7B-Instruct-v0.1\")\n\n>>> prompt = \"My favourite condiment is\"\n\n>>> messages = [\n...     {\"role\": \"user\", \"content\": \"What is your favourite condiment?\"},\n...     {\"role\": \"assistant\", \"content\": \"Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!\"},\n...     {\"role\": \"user\", \"content\": \"Do you have mayonnaise recipes?\"}\n... ]\n\n>>> model_inputs = tokenizer.apply_chat_template(messages, return_tensors=\"pt\").to(\"cuda\")\n\n>>> generated_ids = model.generate(model_inputs, max_new_tokens=100, do_sample=True)\n>>> tokenizer.batch_decode(generated_ids)[0]\n\"The expected output\"\n```\n\nThis model was contributed by [Younes Belkada](https://huggingface.co/ybelkada) and [Arthur Zucker](https://huggingface.co/ArthurZ) .\nThe original code can be found [here](https://github.com/mistralai/mistral-src).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Mixtral."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 8b365fac-f5b6-48dc-a514-7c531b810557
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mixtral/", "title": "Mixtral", "tokens": 2917, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"8b365fac-f5b6-48dc-a514-7c531b810557\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mixtral/\", \"title\": \"Mixtral\", \"tokens\": 2917, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d78e4927-3627-44c1-a2b6-974170c15b8f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mixtral/\", \"title\": \"Mixtral\", \"tokens\": 2917, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"731148c4922aae35ef1f94d656dac38ca59e0bd09610e1e8e358b60d8a3c9f88\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"58dbaaec-acf8-4430-9132-d386d18f811a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mixtral/\", \"title\": \"Mixtral\", \"tokens\": 2917, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"3ecfedcef4dbc091c3622ee64bd0b16c3b6ef448d65dae5b46ab25608b3ff5e8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8582, \"end_char_idx\": 10843, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d78e4927-3627-44c1-a2b6-974170c15b8f", "doc_id": "d78e4927-3627-44c1-a2b6-974170c15b8f", "ref_doc_id": "d78e4927-3627-44c1-a2b6-974170c15b8f", "chroma:document": "\"},\n...     {\"role\": \"assistant\", \"content\": \"Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!\"},\n...     {\"role\": \"user\", \"content\": \"Do you have mayonnaise recipes?\"}\n... ]\n\n>>> model_inputs = tokenizer.apply_chat_template(messages, return_tensors=\"pt\").to(\"cuda\")\n\n>>> generated_ids = model.generate(model_inputs, max_new_tokens=100, do_sample=True)\n>>> tokenizer.batch_decode(generated_ids)[0]\n\"The expected output\"\n```\n\nThis model was contributed by [Younes Belkada](https://huggingface.co/ybelkada) and [Arthur Zucker](https://huggingface.co/ArthurZ) .\nThe original code can be found [here](https://github.com/mistralai/mistral-src).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Mixtral. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-generation\"/>\n\n- A demo notebook to perform supervised fine-tuning (SFT) of Mixtral-8x7B can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/Mistral/Supervised_fine_tuning_(SFT)_of_an_LLM_using_Hugging_Face_tooling.ipynb). \ud83c\udf0e\n- A [blog post](https://medium.com/@prakharsaxena11111/finetuning-mixtral-7bx8-6071b0ebf114) on fine-tuning Mixtral-8x7B using PEFT. \ud83c\udf0e\n- The [Alignment Handbook](https://github.com/huggingface/alignment-handbook) by Hugging Face includes scripts and recipes to perform supervised fine-tuning (SFT) and direct preference optimization with Mistral-7B. This includes scripts for full fine-tuning, QLoRa on a single GPU as well as multi-GPU fine-tuning.\n- [Causal language modeling task guide](../tasks/language_modeling)\n\n## MixtralConfig\n\n[[autodoc]] MixtralConfig\n\n## MixtralModel\n\n[[autodoc]] MixtralModel\n    - forward\n\n## MixtralForCausalLM\n\n[[autodoc]] MixtralForCausalLM\n    - forward\n\n## MixtralForSequenceClassification\n\n[[autodoc]] MixtralForSequenceClassification\n    - forward\n\n## MixtralForTokenClassification\n\n[[autodoc]] MixtralForTokenClassification\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 5bd30481-324c-47c1-b2fa-f930b26a72b7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/graphormer/", "title": "Graphormer", "tokens": 737, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"5bd30481-324c-47c1-b2fa-f930b26a72b7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/graphormer/\", \"title\": \"Graphormer\", \"tokens\": 737, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5b410778-9606-4f0b-a642-4f732cb3ca92\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/graphormer/\", \"title\": \"Graphormer\", \"tokens\": 737, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a9640fa7dacec800b1f402cbb509fd7ab5e64fec0c7d1009ae417db20e9aaa10\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2780, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5b410778-9606-4f0b-a642-4f732cb3ca92", "doc_id": "5b410778-9606-4f0b-a642-4f732cb3ca92", "ref_doc_id": "5b410778-9606-4f0b-a642-4f732cb3ca92", "chroma:document": "# Graphormer\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we don't accept any new PRs changing its code.\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.\nYou can do so by running the following command: `pip install -U transformers==4.40.2`.\n\n</Tip>\n\n## Overview\n\nThe Graphormer model was proposed in [Do Transformers Really Perform Bad for Graph Representation?](https://arxiv.org/abs/2106.05234)  by\nChengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen and Tie-Yan Liu. It is a Graph Transformer model, modified to allow computations on graphs instead of text sequences by generating embeddings and features of interest during preprocessing and collation, then using a modified attention.\n\nThe abstract from the paper is the following:\n\n*The Transformer architecture has become a dominant choice in many domains, such as natural language processing and computer vision. Yet, it has not achieved competitive performance on popular leaderboards of graph-level prediction compared to mainstream GNN variants. Therefore, it remains a mystery how Transformers could perform well for graph representation learning. In this paper, we solve this mystery by presenting Graphormer, which is built upon the standard Transformer architecture, and could attain excellent results on a broad range of graph representation learning tasks, especially on the recent OGB Large-Scale Challenge. Our key insight to utilizing Transformer in the graph is the necessity of effectively encoding the structural information of a graph into the model. To this end, we propose several simple yet effective structural encoding methods to help Graphormer better model graph-structured data. Besides, we mathematically characterize the expressive power of Graphormer and exhibit that with our ways of encoding the structural information of graphs, many popular GNN variants could be covered as the special cases of Graphormer.*\n\nThis model was contributed by [clefourrier](https://huggingface.co/clefourrier). The original code can be found [here](https://github.com/microsoft/Graphormer).\n\n## Usage tips\n\nThis model will not work well on large graphs (more than 100 nodes/edges), as it will make the memory explode.\nYou can reduce the batch size, increase your RAM, or decrease the `UNREACHABLE_NODE_DISTANCE` parameter in algos_graphormer.pyx, but it will be hard to go above 700 nodes/edges.\n\nThis model does not use a tokenizer, but instead a special collator during training.\n\n## GraphormerConfig\n\n[[autodoc]] GraphormerConfig\n\n## GraphormerModel\n\n[[autodoc]] GraphormerModel\n    - forward\n\n## GraphormerForGraphClassification\n\n[[autodoc]] GraphormerForGraphClassification\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 8bbeb50a-8450-4c5c-a880-7961671581d0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/deberta/", "title": "DeBERTa", "tokens": 2023, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"8bbeb50a-8450-4c5c-a880-7961671581d0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deberta/\", \"title\": \"DeBERTa\", \"tokens\": 2023, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bcf10896-b1df-49bf-9a0c-468e229287e2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deberta/\", \"title\": \"DeBERTa\", \"tokens\": 2023, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"d20555427c7d06ef1560be85e8f5d9cdbd5425601ede3ecc8ad192b29b27dfea\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"619924d6-aa77-4983-8140-00717de70fb5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1cd4d1c1c29961e9d79825a1731ce00bded17e7695152c853719ec3a857dc5dd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2980, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bcf10896-b1df-49bf-9a0c-468e229287e2", "doc_id": "bcf10896-b1df-49bf-9a0c-468e229287e2", "ref_doc_id": "bcf10896-b1df-49bf-9a0c-468e229287e2", "chroma:document": "# DeBERTa\n\n## Overview\n\nThe DeBERTa model was proposed in [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://arxiv.org/abs/2006.03654) by Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen It is based on Google's\nBERT model released in 2018 and Facebook's RoBERTa model released in 2019.\n\nIt builds on RoBERTa with disentangled attention and enhanced mask decoder training with half of the data used in\nRoBERTa.\n\nThe abstract from the paper is the following:\n\n*Recent progress in pre-trained neural language models has significantly improved the performance of many natural\nlanguage processing (NLP) tasks. In this paper we propose a new model architecture DeBERTa (Decoding-enhanced BERT with\ndisentangled attention) that improves the BERT and RoBERTa models using two novel techniques. The first is the\ndisentangled attention mechanism, where each word is represented using two vectors that encode its content and\nposition, respectively, and the attention weights among words are computed using disentangled matrices on their\ncontents and relative positions. Second, an enhanced mask decoder is used to replace the output softmax layer to\npredict the masked tokens for model pretraining. We show that these two techniques significantly improve the efficiency\nof model pretraining and performance of downstream tasks. Compared to RoBERTa-Large, a DeBERTa model trained on half of\nthe training data performs consistently better on a wide range of NLP tasks, achieving improvements on MNLI by +0.9%\n(90.2% vs. 91.1%), on SQuAD v2.0 by +2.3% (88.4% vs. 90.7%) and RACE by +3.6% (83.2% vs. 86.8%). The DeBERTa code and\npre-trained models will be made publicly available at https://github.com/microsoft/DeBERTa.*\n\n\nThis model was contributed by [DeBERTa](https://huggingface.co/DeBERTa). This model TF 2.0 implementation was\ncontributed by [kamalkraj](https://huggingface.co/kamalkraj) . The original code can be found [here](https://github.com/microsoft/DeBERTa).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with DeBERTa. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- A blog post on how to [Accelerate Large Model Training using DeepSpeed](https://huggingface.co/blog/accelerate-deepspeed) with DeBERTa.\n- A blog post on [Supercharged Customer Service with Machine Learning](https://huggingface.co/blog/supercharge-customer-service-with-machine-learning) with DeBERTa.\n- [`DebertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification.ipynb)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 619924d6-aa77-4983-8140-00717de70fb5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/deberta/", "title": "DeBERTa", "tokens": 2023, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"619924d6-aa77-4983-8140-00717de70fb5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deberta/\", \"title\": \"DeBERTa\", \"tokens\": 2023, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bcf10896-b1df-49bf-9a0c-468e229287e2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deberta/\", \"title\": \"DeBERTa\", \"tokens\": 2023, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"d20555427c7d06ef1560be85e8f5d9cdbd5425601ede3ecc8ad192b29b27dfea\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8bbeb50a-8450-4c5c-a880-7961671581d0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deberta/\", \"title\": \"DeBERTa\", \"tokens\": 2023, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"38fc3b9266f01eafb7daefc064becde3c6a8088c48622ecadc5535736f94ab41\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"65f25769-aa3e-492e-bf26-0bde31668d0a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"af747be1a8ae039b7671897aec2447e6c53c8110750de1bea44b7840769049e2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1733, \"end_char_idx\": 4954, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bcf10896-b1df-49bf-9a0c-468e229287e2", "doc_id": "bcf10896-b1df-49bf-9a0c-468e229287e2", "ref_doc_id": "bcf10896-b1df-49bf-9a0c-468e229287e2", "chroma:document": "This model was contributed by [DeBERTa](https://huggingface.co/DeBERTa). This model TF 2.0 implementation was\ncontributed by [kamalkraj](https://huggingface.co/kamalkraj) . The original code can be found [here](https://github.com/microsoft/DeBERTa).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with DeBERTa. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- A blog post on how to [Accelerate Large Model Training using DeepSpeed](https://huggingface.co/blog/accelerate-deepspeed) with DeBERTa.\n- A blog post on [Supercharged Customer Service with Machine Learning](https://huggingface.co/blog/supercharge-customer-service-with-machine-learning) with DeBERTa.\n- [`DebertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification.ipynb).\n- [`TFDebertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification-tf.ipynb).\n- [Text classification task guide](../tasks/sequence_classification)\n\n<PipelineTag pipeline=\"token-classification\" />\n\n- [`DebertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification.ipynb).\n- [`TFDebertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification-tf.ipynb).\n- [Token classification](https://huggingface.co/course/chapter7/2?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Byte-Pair Encoding tokenization](https://huggingface.co/course/chapter6/5?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Token classification task guide](../tasks/token_classification)\n\n<PipelineTag pipeline=\"fill-mask\"/>\n\n- [`DebertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#robertabertdistilbert-and-masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n- [`TFDebertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_mlmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 65f25769-aa3e-492e-bf26-0bde31668d0a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/deberta/", "title": "DeBERTa", "tokens": 2023, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"65f25769-aa3e-492e-bf26-0bde31668d0a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deberta/\", \"title\": \"DeBERTa\", \"tokens\": 2023, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bcf10896-b1df-49bf-9a0c-468e229287e2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deberta/\", \"title\": \"DeBERTa\", \"tokens\": 2023, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"d20555427c7d06ef1560be85e8f5d9cdbd5425601ede3ecc8ad192b29b27dfea\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"619924d6-aa77-4983-8140-00717de70fb5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deberta/\", \"title\": \"DeBERTa\", \"tokens\": 2023, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"abe08a7dbfce93980862f4126d32d20000a47efdc1efe07e2cfec09616f8b0ad\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e78c2825-375b-464a-bd22-03c52c93041e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b224725574b674797b49b71f1f949e24703afcdffed3f22b70a9dfd0801359c8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3695, \"end_char_idx\": 5882, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bcf10896-b1df-49bf-9a0c-468e229287e2", "doc_id": "bcf10896-b1df-49bf-9a0c-468e229287e2", "ref_doc_id": "bcf10896-b1df-49bf-9a0c-468e229287e2", "chroma:document": "- [`TFDebertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification-tf.ipynb).\n- [Token classification](https://huggingface.co/course/chapter7/2?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Byte-Pair Encoding tokenization](https://huggingface.co/course/chapter6/5?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Token classification task guide](../tasks/token_classification)\n\n<PipelineTag pipeline=\"fill-mask\"/>\n\n- [`DebertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#robertabertdistilbert-and-masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n- [`TFDebertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_mlmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- [`DebertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).\n- [`TFDebertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).\n- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: e78c2825-375b-464a-bd22-03c52c93041e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/deberta/", "title": "DeBERTa", "tokens": 2023, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"e78c2825-375b-464a-bd22-03c52c93041e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deberta/\", \"title\": \"DeBERTa\", \"tokens\": 2023, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bcf10896-b1df-49bf-9a0c-468e229287e2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deberta/\", \"title\": \"DeBERTa\", \"tokens\": 2023, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"d20555427c7d06ef1560be85e8f5d9cdbd5425601ede3ecc8ad192b29b27dfea\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"65f25769-aa3e-492e-bf26-0bde31668d0a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deberta/\", \"title\": \"DeBERTa\", \"tokens\": 2023, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"6e670eab1fce994f43bb557b53fd237b5ceaf9d53dedf64bc885c428f52177e5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4661, \"end_char_idx\": 7328, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bcf10896-b1df-49bf-9a0c-468e229287e2", "doc_id": "bcf10896-b1df-49bf-9a0c-468e229287e2", "ref_doc_id": "bcf10896-b1df-49bf-9a0c-468e229287e2", "chroma:document": "- [`TFDebertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_mlmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- [`DebertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).\n- [`TFDebertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).\n- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Question answering task guide](../tasks/question_answering)\n\n## DebertaConfig\n\n[[autodoc]] DebertaConfig\n\n## DebertaTokenizer\n\n[[autodoc]] DebertaTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## DebertaTokenizerFast\n\n[[autodoc]] DebertaTokenizerFast\n    - build_inputs_with_special_tokens\n    - create_token_type_ids_from_sequences\n\n<frameworkcontent>\n<pt>\n\n## DebertaModel\n\n[[autodoc]] DebertaModel\n    - forward\n\n## DebertaPreTrainedModel\n\n[[autodoc]] DebertaPreTrainedModel\n\n## DebertaForMaskedLM\n\n[[autodoc]] DebertaForMaskedLM\n    - forward\n\n## DebertaForSequenceClassification\n\n[[autodoc]] DebertaForSequenceClassification\n    - forward\n\n## DebertaForTokenClassification\n\n[[autodoc]] DebertaForTokenClassification\n    - forward\n\n## DebertaForQuestionAnswering\n\n[[autodoc]] DebertaForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFDebertaModel\n\n[[autodoc]] TFDebertaModel\n    - call\n\n## TFDebertaPreTrainedModel\n\n[[autodoc]] TFDebertaPreTrainedModel\n    - call\n\n## TFDebertaForMaskedLM\n\n[[autodoc]] TFDebertaForMaskedLM\n    - call\n\n## TFDebertaForSequenceClassification\n\n[[autodoc]] TFDebertaForSequenceClassification\n    - call\n\n## TFDebertaForTokenClassification\n\n[[autodoc]] TFDebertaForTokenClassification\n    - call\n\n## TFDebertaForQuestionAnswering\n\n[[autodoc]] TFDebertaForQuestionAnswering\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 3728c11e-11ff-40e3-8df5-b1753e94fbbd
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vit_hybrid/", "title": "Hybrid Vision Transformer (ViT Hybrid)", "tokens": 1347, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"3728c11e-11ff-40e3-8df5-b1753e94fbbd\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_hybrid/\", \"title\": \"Hybrid Vision Transformer (ViT Hybrid)\", \"tokens\": 1347, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"420afacf-5aaa-4368-9917-633ebcdd4620\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_hybrid/\", \"title\": \"Hybrid Vision Transformer (ViT Hybrid)\", \"tokens\": 1347, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"4cc056672836ddff427be12532b7a3908e7dec2a118a83a8c0af3b3da75ef735\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f5bbd57a-e188-47a7-80a0-ba4e30627244\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e07996bf2a19e697395b9508e3db45c7c92be03318f6429761c16d0f6cfeca27\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3288, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "420afacf-5aaa-4368-9917-633ebcdd4620", "doc_id": "420afacf-5aaa-4368-9917-633ebcdd4620", "ref_doc_id": "420afacf-5aaa-4368-9917-633ebcdd4620", "chroma:document": "# Hybrid Vision Transformer (ViT Hybrid)\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we don't accept any new PRs changing its code.\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.\nYou can do so by running the following command: `pip install -U transformers==4.40.2`.\n\n</Tip>\n\n## Overview\n\nThe hybrid Vision Transformer (ViT) model was proposed in [An Image is Worth 16x16 Words: Transformers for Image Recognition\nat Scale](https://arxiv.org/abs/2010.11929) by Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk\nWeissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob\nUszkoreit, Neil Houlsby. It's the first paper that successfully trains a Transformer encoder on ImageNet, attaining\nvery good results compared to familiar convolutional architectures. ViT hybrid is a slight variant of the [plain Vision Transformer](vit),\nby leveraging a convolutional backbone (specifically, [BiT](bit)) whose features are used as initial \"tokens\" for the Transformer.\n\nThe abstract from the paper is the following:\n\n*While the Transformer architecture has become the de-facto standard for natural language processing tasks, its\napplications to computer vision remain limited. In vision, attention is either applied in conjunction with\nconvolutional networks, or used to replace certain components of convolutional networks while keeping their overall\nstructure in place. We show that this reliance on CNNs is not necessary and a pure transformer applied directly to\nsequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of\ndata and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.),\nVision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring\nsubstantially fewer computational resources to train.*\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code (written in JAX) can be\nfound [here](https://github.com/google-research/vision_transformer).\n\n## Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```\nfrom transformers import ViTHybridForImageClassification\nmodel = ViTHybridForImageClassification.from_pretrained(\"google/vit-hybrid-base-bit-384\", attn_implementation=\"sdpa\", torch_dtype=torch.float16)\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: f5bbd57a-e188-47a7-80a0-ba4e30627244
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vit_hybrid/", "title": "Hybrid Vision Transformer (ViT Hybrid)", "tokens": 1347, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"f5bbd57a-e188-47a7-80a0-ba4e30627244\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_hybrid/\", \"title\": \"Hybrid Vision Transformer (ViT Hybrid)\", \"tokens\": 1347, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"420afacf-5aaa-4368-9917-633ebcdd4620\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_hybrid/\", \"title\": \"Hybrid Vision Transformer (ViT Hybrid)\", \"tokens\": 1347, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"4cc056672836ddff427be12532b7a3908e7dec2a118a83a8c0af3b3da75ef735\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3728c11e-11ff-40e3-8df5-b1753e94fbbd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_hybrid/\", \"title\": \"Hybrid Vision Transformer (ViT Hybrid)\", \"tokens\": 1347, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"83cb24aeefe9749500eff912515721222edf09f5249213b1038eae1acc0aed18\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dc77fd8e-a55b-4357-baae-b18302980ff6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b200e2f090e2f29b50347bd78dd1c7213575730b3b0b2c2114833452ae213b2d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1693, \"end_char_idx\": 5044, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "420afacf-5aaa-4368-9917-633ebcdd4620", "doc_id": "420afacf-5aaa-4368-9917-633ebcdd4620", "ref_doc_id": "420afacf-5aaa-4368-9917-633ebcdd4620", "chroma:document": "When pre-trained on large amounts of\ndata and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.),\nVision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring\nsubstantially fewer computational resources to train.*\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code (written in JAX) can be\nfound [here](https://github.com/google-research/vision_transformer).\n\n## Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```\nfrom transformers import ViTHybridForImageClassification\nmodel = ViTHybridForImageClassification.from_pretrained(\"google/vit-hybrid-base-bit-384\", attn_implementation=\"sdpa\", torch_dtype=torch.float16)\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `google/vit-hybrid-base-bit-384` model, we saw the following speedups during inference.\n\n|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |\n|--------------|-------------------------------------------|-------------------------------------------|------------------------------|\n|            1 |                                        29 |                                        18 |                      1.61 |\n|            2 |                                        26 |                                        18 |                      1.44 |\n|            4 |                                        25 |                                        18 |                      1.39 |\n|            8 |                                        34 |                                        24 |                      1.42 |\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with ViT Hybrid.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`ViTHybridForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: dc77fd8e-a55b-4357-baae-b18302980ff6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vit_hybrid/", "title": "Hybrid Vision Transformer (ViT Hybrid)", "tokens": 1347, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"dc77fd8e-a55b-4357-baae-b18302980ff6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_hybrid/\", \"title\": \"Hybrid Vision Transformer (ViT Hybrid)\", \"tokens\": 1347, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"420afacf-5aaa-4368-9917-633ebcdd4620\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_hybrid/\", \"title\": \"Hybrid Vision Transformer (ViT Hybrid)\", \"tokens\": 1347, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"4cc056672836ddff427be12532b7a3908e7dec2a118a83a8c0af3b3da75ef735\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f5bbd57a-e188-47a7-80a0-ba4e30627244\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit_hybrid/\", \"title\": \"Hybrid Vision Transformer (ViT Hybrid)\", \"tokens\": 1347, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8ce606e69e6a18ac610c9c68d24ac6e0633a7c66168cab43b04c3ff38bc2897a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3251, \"end_char_idx\": 5331, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "420afacf-5aaa-4368-9917-633ebcdd4620", "doc_id": "420afacf-5aaa-4368-9917-633ebcdd4620", "ref_doc_id": "420afacf-5aaa-4368-9917-633ebcdd4620", "chroma:document": "`torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `google/vit-hybrid-base-bit-384` model, we saw the following speedups during inference.\n\n|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |\n|--------------|-------------------------------------------|-------------------------------------------|------------------------------|\n|            1 |                                        29 |                                        18 |                      1.61 |\n|            2 |                                        26 |                                        18 |                      1.44 |\n|            4 |                                        25 |                                        18 |                      1.39 |\n|            8 |                                        34 |                                        24 |                      1.42 |\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with ViT Hybrid.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`ViTHybridForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## ViTHybridConfig\n\n[[autodoc]] ViTHybridConfig\n\n## ViTHybridImageProcessor\n\n[[autodoc]] ViTHybridImageProcessor\n    - preprocess\n\n## ViTHybridModel\n\n[[autodoc]] ViTHybridModel\n    - forward\n\n## ViTHybridForImageClassification\n\n[[autodoc]] ViTHybridForImageClassification\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 70ef534c-fbeb-4666-ac6a-1c884ff69caf
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/data2vec/", "title": "Data2Vec", "tokens": 1654, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"70ef534c-fbeb-4666-ac6a-1c884ff69caf\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/data2vec/\", \"title\": \"Data2Vec\", \"tokens\": 1654, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2a9d2ff0-18ff-430c-aa15-56c1f4bd86bf\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/data2vec/\", \"title\": \"Data2Vec\", \"tokens\": 1654, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"ab1fce3182e1a997bb435898124fb6f4d74f456556038efacab9143fdf3171c0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"495a88b5-a987-4818-82cd-209b529f7ed7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f3fda78d8cec731a0a25155df42ed80eae6a45d0874dedcb8401dfeb080669a6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3324, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2a9d2ff0-18ff-430c-aa15-56c1f4bd86bf", "doc_id": "2a9d2ff0-18ff-430c-aa15-56c1f4bd86bf", "ref_doc_id": "2a9d2ff0-18ff-430c-aa15-56c1f4bd86bf", "chroma:document": "# Data2Vec\n\n## Overview\n\nThe Data2Vec model was proposed in [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://arxiv.org/pdf/2202.03555) by Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu and Michael Auli.\nData2Vec proposes a unified framework for self-supervised learning across different data modalities - text, audio and images.\nImportantly, predicted targets for pre-training are contextualized latent representations of the inputs, rather than modality-specific, context-independent targets.\n\nThe abstract from the paper is the following:\n\n*While the general idea of self-supervised learning is identical across modalities, the actual algorithms and\nobjectives differ widely because they were developed with a single modality in mind. To get us closer to general\nself-supervised learning, we present data2vec, a framework that uses the same learning method for either speech,\nNLP or computer vision. The core idea is to predict latent representations of the full input data based on a\nmasked view of the input in a selfdistillation setup using a standard Transformer architecture.\nInstead of predicting modality-specific targets such as words, visual tokens or units of human speech which\nare local in nature, data2vec predicts contextualized latent representations that contain information from\nthe entire input. Experiments on the major benchmarks of speech recognition, image classification, and\nnatural language understanding demonstrate a new state of the art or competitive performance to predominant approaches.\nModels and code are available at www.github.com/pytorch/fairseq/tree/master/examples/data2vec.*\n\nThis model was contributed by [edugp](https://huggingface.co/edugp) and [patrickvonplaten](https://huggingface.co/patrickvonplaten).\n[sayakpaul](https://github.com/sayakpaul) and [Rocketknight1](https://github.com/Rocketknight1) contributed Data2Vec for vision in TensorFlow.\n\nThe original code (for NLP and Speech) can be found [here](https://github.com/pytorch/fairseq/tree/main/examples/data2vec).\nThe original code for vision can be found [here](https://github.com/facebookresearch/data2vec_vision/tree/main/beit).\n\n## Usage tips\n\n- Data2VecAudio, Data2VecText, and Data2VecVision have all been trained using the same self-supervised learning method.\n- For Data2VecAudio, preprocessing is identical to [`Wav2Vec2Model`], including feature extraction\n- For Data2VecText, preprocessing is identical to [`RobertaModel`], including tokenization.\n- For Data2VecVision, preprocessing is identical to [`BeitModel`], including feature extraction.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Data2Vec.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`Data2VecVisionForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- To fine-tune [`TFData2VecVisionForImageClassification`] on a custom dataset, see [this notebook](https://colab.research.google.com/github/sayakpaul/TF-2.0-Hacks/blob/master/data2vec_vision_image_classification.ipynb)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 495a88b5-a987-4818-82cd-209b529f7ed7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/data2vec/", "title": "Data2Vec", "tokens": 1654, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"495a88b5-a987-4818-82cd-209b529f7ed7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/data2vec/\", \"title\": \"Data2Vec\", \"tokens\": 1654, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2a9d2ff0-18ff-430c-aa15-56c1f4bd86bf\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/data2vec/\", \"title\": \"Data2Vec\", \"tokens\": 1654, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"ab1fce3182e1a997bb435898124fb6f4d74f456556038efacab9143fdf3171c0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"70ef534c-fbeb-4666-ac6a-1c884ff69caf\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/data2vec/\", \"title\": \"Data2Vec\", \"tokens\": 1654, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"76a6a12c78c1c565baa470d366b4f9cf8d79dc905c5825ad2d397dc20936bca7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"61dc0c8f-915d-4af9-a302-01555a738ce2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"46711f5e54e329fc98770ff2f6837bd8470a8bf0867e251fa035b949cbe3826c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1819, \"end_char_idx\": 4319, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2a9d2ff0-18ff-430c-aa15-56c1f4bd86bf", "doc_id": "2a9d2ff0-18ff-430c-aa15-56c1f4bd86bf", "ref_doc_id": "2a9d2ff0-18ff-430c-aa15-56c1f4bd86bf", "chroma:document": "[sayakpaul](https://github.com/sayakpaul) and [Rocketknight1](https://github.com/Rocketknight1) contributed Data2Vec for vision in TensorFlow.\n\nThe original code (for NLP and Speech) can be found [here](https://github.com/pytorch/fairseq/tree/main/examples/data2vec).\nThe original code for vision can be found [here](https://github.com/facebookresearch/data2vec_vision/tree/main/beit).\n\n## Usage tips\n\n- Data2VecAudio, Data2VecText, and Data2VecVision have all been trained using the same self-supervised learning method.\n- For Data2VecAudio, preprocessing is identical to [`Wav2Vec2Model`], including feature extraction\n- For Data2VecText, preprocessing is identical to [`RobertaModel`], including tokenization.\n- For Data2VecVision, preprocessing is identical to [`BeitModel`], including feature extraction.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Data2Vec.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`Data2VecVisionForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- To fine-tune [`TFData2VecVisionForImageClassification`] on a custom dataset, see [this notebook](https://colab.research.google.com/github/sayakpaul/TF-2.0-Hacks/blob/master/data2vec_vision_image_classification.ipynb).\n\n**Data2VecText documentation resources**\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n**Data2VecAudio documentation resources**\n- [Audio classification task guide](../tasks/audio_classification)\n- [Automatic speech recognition task guide](../tasks/asr)\n\n**Data2VecVision documentation resources**\n- [Image classification](../tasks/image_classification)\n- [Semantic segmentation](../tasks/semantic_segmentation)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 61dc0c8f-915d-4af9-a302-01555a738ce2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/data2vec/", "title": "Data2Vec", "tokens": 1654, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"61dc0c8f-915d-4af9-a302-01555a738ce2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/data2vec/\", \"title\": \"Data2Vec\", \"tokens\": 1654, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2a9d2ff0-18ff-430c-aa15-56c1f4bd86bf\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/data2vec/\", \"title\": \"Data2Vec\", \"tokens\": 1654, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"ab1fce3182e1a997bb435898124fb6f4d74f456556038efacab9143fdf3171c0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"495a88b5-a987-4818-82cd-209b529f7ed7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/data2vec/\", \"title\": \"Data2Vec\", \"tokens\": 1654, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"c89c220baba1bebe0abe31c6e4a48a1e78c8e4b4a98e6176fec6dce159cc5a40\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2532, \"end_char_idx\": 6185, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2a9d2ff0-18ff-430c-aa15-56c1f4bd86bf", "doc_id": "2a9d2ff0-18ff-430c-aa15-56c1f4bd86bf", "ref_doc_id": "2a9d2ff0-18ff-430c-aa15-56c1f4bd86bf", "chroma:document": "- For Data2VecVision, preprocessing is identical to [`BeitModel`], including feature extraction.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Data2Vec.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`Data2VecVisionForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- To fine-tune [`TFData2VecVisionForImageClassification`] on a custom dataset, see [this notebook](https://colab.research.google.com/github/sayakpaul/TF-2.0-Hacks/blob/master/data2vec_vision_image_classification.ipynb).\n\n**Data2VecText documentation resources**\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n**Data2VecAudio documentation resources**\n- [Audio classification task guide](../tasks/audio_classification)\n- [Automatic speech recognition task guide](../tasks/asr)\n\n**Data2VecVision documentation resources**\n- [Image classification](../tasks/image_classification)\n- [Semantic segmentation](../tasks/semantic_segmentation)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## Data2VecTextConfig\n\n[[autodoc]] Data2VecTextConfig\n\n## Data2VecAudioConfig\n\n[[autodoc]] Data2VecAudioConfig\n\n## Data2VecVisionConfig\n\n[[autodoc]] Data2VecVisionConfig\n\n<frameworkcontent>\n<pt>\n\n## Data2VecAudioModel\n\n[[autodoc]] Data2VecAudioModel\n    - forward\n\n## Data2VecAudioForAudioFrameClassification\n\n[[autodoc]] Data2VecAudioForAudioFrameClassification\n    - forward\n\n## Data2VecAudioForCTC\n\n[[autodoc]] Data2VecAudioForCTC\n    - forward\n\n## Data2VecAudioForSequenceClassification\n\n[[autodoc]] Data2VecAudioForSequenceClassification\n    - forward\n\n## Data2VecAudioForXVector\n\n[[autodoc]] Data2VecAudioForXVector\n    - forward\n\n## Data2VecTextModel\n\n[[autodoc]] Data2VecTextModel\n    - forward\n\n## Data2VecTextForCausalLM\n\n[[autodoc]] Data2VecTextForCausalLM\n    - forward\n\n## Data2VecTextForMaskedLM\n\n[[autodoc]] Data2VecTextForMaskedLM\n    - forward\n\n## Data2VecTextForSequenceClassification\n\n[[autodoc]] Data2VecTextForSequenceClassification\n    - forward\n\n## Data2VecTextForMultipleChoice\n\n[[autodoc]] Data2VecTextForMultipleChoice\n    - forward\n\n## Data2VecTextForTokenClassification\n\n[[autodoc]] Data2VecTextForTokenClassification\n    - forward\n\n## Data2VecTextForQuestionAnswering\n\n[[autodoc]] Data2VecTextForQuestionAnswering\n    - forward\n\n## Data2VecVisionModel\n\n[[autodoc]] Data2VecVisionModel\n    - forward\n\n## Data2VecVisionForImageClassification\n\n[[autodoc]] Data2VecVisionForImageClassification\n    - forward\n\n## Data2VecVisionForSemanticSegmentation\n\n[[autodoc]] Data2VecVisionForSemanticSegmentation\n    - forward\n\n</pt>\n<tf>\n\n## TFData2VecVisionModel\n\n[[autodoc]] TFData2VecVisionModel\n    - call\n\n## TFData2VecVisionForImageClassification\n\n[[autodoc]] TFData2VecVisionForImageClassification\n    - call\n\n## TFData2VecVisionForSemanticSegmentation\n\n[[autodoc]] TFData2VecVisionForSemanticSegmentation\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 959babe3-7367-4a8c-9195-1783d04c94c4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/ul2/", "title": "UL2", "tokens": 766, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"959babe3-7367-4a8c-9195-1783d04c94c4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/ul2/\", \"title\": \"UL2\", \"tokens\": 766, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b6b8f586-823e-4225-859e-1177623b701b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/ul2/\", \"title\": \"UL2\", \"tokens\": 766, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0201d8bbf114223ab53e50da61bf5fb30b85f47c61a74118a684a76bd8ae34d5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2727, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b6b8f586-823e-4225-859e-1177623b701b", "doc_id": "b6b8f586-823e-4225-859e-1177623b701b", "ref_doc_id": "b6b8f586-823e-4225-859e-1177623b701b", "chroma:document": "# UL2\n\n## Overview\n\nThe T5 model was presented in [Unifying Language Learning Paradigms](https://arxiv.org/pdf/2205.05131v1.pdf) by Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, Donald Metzler.\n\nThe abstract from the paper is the following:\n\n*Existing pre-trained models are generally geared towards a particular class of problems. To date, there seems to be still no consensus on what the right architecture and pre-training setup should be. This paper presents a unified framework for pre-training models that are universally effective across datasets and setups. We begin by disentangling architectural archetypes with pre-training objectives -- two concepts that are commonly conflated. Next, we present a generalized and unified perspective for self-supervision in NLP and show how different pre-training objectives can be cast as one another and how interpolating between different objectives can be effective. We then propose Mixture-of-Denoisers (MoD), a pre-training objective that combines diverse pre-training paradigms together. We furthermore introduce a notion of mode switching, wherein downstream fine-tuning is associated with specific pre-training schemes. We conduct extensive ablative experiments to compare multiple pre-training objectives and find that our method pushes the Pareto-frontier by outperforming T5 and/or GPT-like models across multiple diverse setups. Finally, by scaling our model up to 20B parameters, we achieve SOTA performance on 50 well-established supervised NLP tasks ranging from language generation (with automated and human evaluation), language understanding, text classification, question answering, commonsense reasoning, long text reasoning, structured knowledge grounding and information retrieval. Our model also achieve strong results at in-context learning, outperforming 175B GPT-3 on zero-shot SuperGLUE and tripling the performance of T5-XXL on one-shot summarization.*\n\nThis model was contributed by [DanielHesslow](https://huggingface.co/Seledorn). The original code can be found [here](https://github.com/google-research/google-research/tree/master/ul2).\n\n## Usage tips\n\n- UL2 is an encoder-decoder model pre-trained on a mixture of denoising functions as well as fine-tuned on an array of downstream tasks.\n- UL2 has the same architecture as [T5v1.1](t5v1.1) but uses the Gated-SiLU activation function instead of Gated-GELU.\n- The authors release checkpoints of one architecture which can be seen [here](https://huggingface.co/google/ul2)\n\n<Tip> \n\nAs UL2 has the same architecture as T5v1.1,  refer to [T5's documentation page](t5) for API reference, tips, code examples and notebooks.\n\n</Tip>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 1d59be88-0144-46c4-98f1-939d4f049a5e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/olmo/", "title": "OLMo", "tokens": 782, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"1d59be88-0144-46c4-98f1-939d4f049a5e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/olmo/\", \"title\": \"OLMo\", \"tokens\": 782, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ebe78337-af10-4878-988c-36330f9ea478\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/olmo/\", \"title\": \"OLMo\", \"tokens\": 782, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5b492a33f3a831713c53113f201f001e0f93dd1a57ffe05b7438e64598eb8dd8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2473, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ebe78337-af10-4878-988c-36330f9ea478", "doc_id": "ebe78337-af10-4878-988c-36330f9ea478", "ref_doc_id": "ebe78337-af10-4878-988c-36330f9ea478", "chroma:document": "# OLMo\n\n## Overview\n\nThe OLMo model was proposed in [OLMo: Accelerating the Science of Language Models](https://arxiv.org/abs/2402.00838) by Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, Hannaneh Hajishirzi.\n\nOLMo is a series of **O**pen **L**anguage **Mo**dels designed to enable the science of language models. The OLMo models are trained on the Dolma dataset. We release all code, checkpoints, logs (coming soon), and details involved in training these models.\n\nThe abstract from the paper is the following:\n\n*Language models (LMs) have become ubiquitous in both NLP research and in commercial product offerings. As their commercial importance has surged, the most powerful models have become closed off, gated behind proprietary interfaces, with important details of their training data, architectures, and development undisclosed. Given the importance of these details in scientifically studying these models, including their biases and potential risks, we believe it is essential for the research community to have access to powerful, truly open LMs. To this end, this technical report details the first release of OLMo, a state-of-the-art, truly Open Language Model and its framework to build and study the science of language modeling. Unlike most prior efforts that have only released model weights and inference code, we release OLMo and the whole framework, including training data and training and evaluation code. We hope this release will empower and strengthen the open research community and inspire a new wave of innovation.*\n\nThis model was contributed by [shanearora](https://huggingface.co/shanearora).\nThe original code can be found [here](https://github.com/allenai/OLMo/tree/main/olmo).\n\n\n## OlmoConfig\n\n[[autodoc]] OlmoConfig\n\n## OlmoModel\n\n[[autodoc]] OlmoModel\n    - forward\n\n## OlmoForCausalLM\n\n[[autodoc]] OlmoForCausalLM\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: c0de2f4c-38c7-4872-b72f-5aaf8d881ed0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta/", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"c0de2f4c-38c7-4872-b72f-5aaf8d881ed0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-roberta/\", \"title\": \"XLM-RoBERTa\", \"tokens\": 3116, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ec72c6e1-2d2c-47e8-9147-747727b3f23d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-roberta/\", \"title\": \"XLM-RoBERTa\", \"tokens\": 3116, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5abb04d433e42ab23a29df9030a27a2570210c8221e397aa5eda73f1c1771eba\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0e0cd590-04b0-4346-a26a-bbd8dfdc939c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"99e106084bac918565a3c12c00bc69a426deb4a3c463552a4a4b211adde43dc8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2887, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ec72c6e1-2d2c-47e8-9147-747727b3f23d", "doc_id": "ec72c6e1-2d2c-47e8-9147-747727b3f23d", "ref_doc_id": "ec72c6e1-2d2c-47e8-9147-747727b3f23d", "chroma:document": "# XLM-RoBERTa\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=xlm-roberta\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-xlm--roberta-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/xlm-roberta-base\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nThe XLM-RoBERTa model was proposed in [Unsupervised Cross-lingual Representation Learning at Scale](https://arxiv.org/abs/1911.02116) by Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume\nWenzek, Francisco Guzm\u00e1n, Edouard Grave, Myle Ott, Luke Zettlemoyer and Veselin Stoyanov. It is based on Facebook's\nRoBERTa model released in 2019. It is a large multi-lingual language model, trained on 2.5TB of filtered CommonCrawl\ndata.\n\nThe abstract from the paper is the following:\n\n*This paper shows that pretraining multilingual language models at scale leads to significant performance gains for a\nwide range of cross-lingual transfer tasks. We train a Transformer-based masked language model on one hundred\nlanguages, using more than two terabytes of filtered CommonCrawl data. Our model, dubbed XLM-R, significantly\noutperforms multilingual BERT (mBERT) on a variety of cross-lingual benchmarks, including +13.8% average accuracy on\nXNLI, +12.3% average F1 score on MLQA, and +2.1% average F1 score on NER. XLM-R performs particularly well on\nlow-resource languages, improving 11.8% in XNLI accuracy for Swahili and 9.2% for Urdu over the previous XLM model. We\nalso present a detailed empirical evaluation of the key factors that are required to achieve these gains, including the\ntrade-offs between (1) positive transfer and capacity dilution and (2) the performance of high and low resource\nlanguages at scale. Finally, we show, for the first time, the possibility of multilingual modeling without sacrificing\nper-language performance; XLM-Ris very competitive with strong monolingual models on the GLUE and XNLI benchmarks. We\nwill make XLM-R code, data, and models publicly available.*\n\nThis model was contributed by [stefan-it](https://huggingface.co/stefan-it). The original code can be found [here](https://github.com/pytorch/fairseq/tree/master/examples/xlmr).\n\n## Usage tips\n\n- XLM-RoBERTa is a multilingual model trained on 100 different languages. Unlike some XLM multilingual models, it does\n  not require `lang` tensors to understand which language is used, and should be able to determine the correct\n  language from the input ids.\n- Uses RoBERTa tricks on the XLM approach, but does not use the translation language modeling objective. It only uses masked language modeling on sentences coming from one language.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with XLM-RoBERTa."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 0e0cd590-04b0-4346-a26a-bbd8dfdc939c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta/", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"0e0cd590-04b0-4346-a26a-bbd8dfdc939c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-roberta/\", \"title\": \"XLM-RoBERTa\", \"tokens\": 3116, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ec72c6e1-2d2c-47e8-9147-747727b3f23d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-roberta/\", \"title\": \"XLM-RoBERTa\", \"tokens\": 3116, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5abb04d433e42ab23a29df9030a27a2570210c8221e397aa5eda73f1c1771eba\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c0de2f4c-38c7-4872-b72f-5aaf8d881ed0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-roberta/\", \"title\": \"XLM-RoBERTa\", \"tokens\": 3116, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"194aa1dca6a4f103cbe1d09384b8ec40836604187c54d11753773bc908bd92bb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2a12262a-3230-4afe-a818-f9522339b542\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"05d9756023dcb8700a311a9568141a8b78c722816122eb1d5f8c2bcb55ece4be\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1436, \"end_char_idx\": 4777, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ec72c6e1-2d2c-47e8-9147-747727b3f23d", "doc_id": "ec72c6e1-2d2c-47e8-9147-747727b3f23d", "ref_doc_id": "ec72c6e1-2d2c-47e8-9147-747727b3f23d", "chroma:document": "XLM-R performs particularly well on\nlow-resource languages, improving 11.8% in XNLI accuracy for Swahili and 9.2% for Urdu over the previous XLM model. We\nalso present a detailed empirical evaluation of the key factors that are required to achieve these gains, including the\ntrade-offs between (1) positive transfer and capacity dilution and (2) the performance of high and low resource\nlanguages at scale. Finally, we show, for the first time, the possibility of multilingual modeling without sacrificing\nper-language performance; XLM-Ris very competitive with strong monolingual models on the GLUE and XNLI benchmarks. We\nwill make XLM-R code, data, and models publicly available.*\n\nThis model was contributed by [stefan-it](https://huggingface.co/stefan-it). The original code can be found [here](https://github.com/pytorch/fairseq/tree/master/examples/xlmr).\n\n## Usage tips\n\n- XLM-RoBERTa is a multilingual model trained on 100 different languages. Unlike some XLM multilingual models, it does\n  not require `lang` tensors to understand which language is used, and should be able to determine the correct\n  language from the input ids.\n- Uses RoBERTa tricks on the XLM approach, but does not use the translation language modeling objective. It only uses masked language modeling on sentences coming from one language.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with XLM-RoBERTa. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- A blog post on how to [finetune XLM RoBERTa for multiclass classification with Habana Gaudi on AWS](https://www.philschmid.de/habana-distributed-training)\n- [`XLMRobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification.ipynb).\n- [`TFXLMRobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification-tf.ipynb).\n- [`FlaxXLMRobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification_flax.ipynb).\n- [Text classification](https://huggingface.co/docs/transformers/tasks/sequence_classification) chapter of the \ud83e\udd17 Hugging Face Task Guides.\n- [Text classification task guide](../tasks/sequence_classification)\n\n<PipelineTag pipeline=\"token-classification\"/>\n\n- [`XLMRobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification.ipynb)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 2a12262a-3230-4afe-a818-f9522339b542
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta/", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"2a12262a-3230-4afe-a818-f9522339b542\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-roberta/\", \"title\": \"XLM-RoBERTa\", \"tokens\": 3116, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ec72c6e1-2d2c-47e8-9147-747727b3f23d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-roberta/\", \"title\": \"XLM-RoBERTa\", \"tokens\": 3116, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5abb04d433e42ab23a29df9030a27a2570210c8221e397aa5eda73f1c1771eba\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0e0cd590-04b0-4346-a26a-bbd8dfdc939c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-roberta/\", \"title\": \"XLM-RoBERTa\", \"tokens\": 3116, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5a15191606f2aa35e2e646861c3172a2a308fab04a290fef882ab6ff99e7b9a0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ee6f4056-10a4-43fc-9453-28d1e531119a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3dd21a269c51f98ea7d3ae33c38a42845f150db3e8dd08c84ada06a61bbb894c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3016, \"end_char_idx\": 6328, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ec72c6e1-2d2c-47e8-9147-747727b3f23d", "doc_id": "ec72c6e1-2d2c-47e8-9147-747727b3f23d", "ref_doc_id": "ec72c6e1-2d2c-47e8-9147-747727b3f23d", "chroma:document": "The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- A blog post on how to [finetune XLM RoBERTa for multiclass classification with Habana Gaudi on AWS](https://www.philschmid.de/habana-distributed-training)\n- [`XLMRobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification.ipynb).\n- [`TFXLMRobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification-tf.ipynb).\n- [`FlaxXLMRobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification_flax.ipynb).\n- [Text classification](https://huggingface.co/docs/transformers/tasks/sequence_classification) chapter of the \ud83e\udd17 Hugging Face Task Guides.\n- [Text classification task guide](../tasks/sequence_classification)\n\n<PipelineTag pipeline=\"token-classification\"/>\n\n- [`XLMRobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification.ipynb).\n- [`TFXLMRobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification-tf.ipynb).\n- [`FlaxXLMRobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/token-classification).\n- [Token classification](https://huggingface.co/course/chapter7/2?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Token classification task guide](../tasks/token_classification)\n\n<PipelineTag pipeline=\"text-generation\"/>\n\n- [`XLMRobertaForCausalLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n- [Causal language modeling](https://huggingface.co/docs/transformers/tasks/language_modeling) chapter of the \ud83e\udd17 Hugging Face Task Guides.\n- [Causal language modeling task guide](../tasks/language_modeling)\n\n<PipelineTag pipeline=\"fill-mask\"/>\n\n- [`XLMRobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#robertabertdistilbert-and-masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: ee6f4056-10a4-43fc-9453-28d1e531119a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta/", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"ee6f4056-10a4-43fc-9453-28d1e531119a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-roberta/\", \"title\": \"XLM-RoBERTa\", \"tokens\": 3116, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ec72c6e1-2d2c-47e8-9147-747727b3f23d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-roberta/\", \"title\": \"XLM-RoBERTa\", \"tokens\": 3116, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5abb04d433e42ab23a29df9030a27a2570210c8221e397aa5eda73f1c1771eba\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2a12262a-3230-4afe-a818-f9522339b542\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-roberta/\", \"title\": \"XLM-RoBERTa\", \"tokens\": 3116, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"874778e9731cb2a21cf54a9135d743b9ba3c7f79dc5045a464dfc2cc639a24ab\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ce40e96e-6496-4bbe-8af9-7536113658dc\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7ac9b404f8f972cd197f72bcc8325504c1a132e5b041419085aecc7b3a5bca1f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4778, \"end_char_idx\": 7924, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ec72c6e1-2d2c-47e8-9147-747727b3f23d", "doc_id": "ec72c6e1-2d2c-47e8-9147-747727b3f23d", "ref_doc_id": "ec72c6e1-2d2c-47e8-9147-747727b3f23d", "chroma:document": "- [`TFXLMRobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification-tf.ipynb).\n- [`FlaxXLMRobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/token-classification).\n- [Token classification](https://huggingface.co/course/chapter7/2?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Token classification task guide](../tasks/token_classification)\n\n<PipelineTag pipeline=\"text-generation\"/>\n\n- [`XLMRobertaForCausalLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n- [Causal language modeling](https://huggingface.co/docs/transformers/tasks/language_modeling) chapter of the \ud83e\udd17 Hugging Face Task Guides.\n- [Causal language modeling task guide](../tasks/language_modeling)\n\n<PipelineTag pipeline=\"fill-mask\"/>\n\n- [`XLMRobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#robertabertdistilbert-and-masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n- [`TFXLMRobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_mlmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- [`FlaxXLMRobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).\n- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Masked language modeling](../tasks/masked_language_modeling)\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- [`XLMRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).\n- [`TFXLMRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).\n- [`FlaxXLMRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: ce40e96e-6496-4bbe-8af9-7536113658dc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta/", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"ce40e96e-6496-4bbe-8af9-7536113658dc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-roberta/\", \"title\": \"XLM-RoBERTa\", \"tokens\": 3116, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ec72c6e1-2d2c-47e8-9147-747727b3f23d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-roberta/\", \"title\": \"XLM-RoBERTa\", \"tokens\": 3116, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5abb04d433e42ab23a29df9030a27a2570210c8221e397aa5eda73f1c1771eba\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ee6f4056-10a4-43fc-9453-28d1e531119a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-roberta/\", \"title\": \"XLM-RoBERTa\", \"tokens\": 3116, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"51195d9e6f4abbed3e3f31ae4ea2b2f8567ebb1bf43be0f5d97d7fb4a090ddc1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"147ad559-1dca-4d86-b6b0-7befc6565fe4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"abbe03ec8c65de72df62e0564b7606f3650996465bbe73078598c3a41a1ed85f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6626, \"end_char_idx\": 9091, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ec72c6e1-2d2c-47e8-9147-747727b3f23d", "doc_id": "ec72c6e1-2d2c-47e8-9147-747727b3f23d", "ref_doc_id": "ec72c6e1-2d2c-47e8-9147-747727b3f23d", "chroma:document": "- [`FlaxXLMRobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).\n- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Masked language modeling](../tasks/masked_language_modeling)\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- [`XLMRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).\n- [`TFXLMRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).\n- [`FlaxXLMRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering).\n- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Question answering task guide](../tasks/question_answering)\n\n**Multiple choice**\n\n- [`XLMRobertaForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb).\n- [`TFXLMRobertaForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice-tf.ipynb).\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n\ud83d\ude80 Deploy\n\n- A blog post on how to [Deploy Serverless XLM RoBERTa on AWS Lambda](https://www.philschmid.de/multilingual-serverless-xlm-roberta-with-huggingface).\n\n<Tip> \n\nThis implementation is the same as RoBERTa. Refer to the [documentation of RoBERTa](roberta) for usage examples as well as the information relative to the inputs and outputs."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 147ad559-1dca-4d86-b6b0-7befc6565fe4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta/", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"147ad559-1dca-4d86-b6b0-7befc6565fe4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-roberta/\", \"title\": \"XLM-RoBERTa\", \"tokens\": 3116, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ec72c6e1-2d2c-47e8-9147-747727b3f23d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-roberta/\", \"title\": \"XLM-RoBERTa\", \"tokens\": 3116, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5abb04d433e42ab23a29df9030a27a2570210c8221e397aa5eda73f1c1771eba\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ce40e96e-6496-4bbe-8af9-7536113658dc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-roberta/\", \"title\": \"XLM-RoBERTa\", \"tokens\": 3116, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"15d72ed7bc08bf1edfcf37722663f4f7fd2774da4112cff299307e14246fcdd1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7757, \"end_char_idx\": 11332, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ec72c6e1-2d2c-47e8-9147-747727b3f23d", "doc_id": "ec72c6e1-2d2c-47e8-9147-747727b3f23d", "ref_doc_id": "ec72c6e1-2d2c-47e8-9147-747727b3f23d", "chroma:document": "- [`FlaxXLMRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering).\n- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Question answering task guide](../tasks/question_answering)\n\n**Multiple choice**\n\n- [`XLMRobertaForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb).\n- [`TFXLMRobertaForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice-tf.ipynb).\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n\ud83d\ude80 Deploy\n\n- A blog post on how to [Deploy Serverless XLM RoBERTa on AWS Lambda](https://www.philschmid.de/multilingual-serverless-xlm-roberta-with-huggingface).\n\n<Tip> \n\nThis implementation is the same as RoBERTa. Refer to the [documentation of RoBERTa](roberta) for usage examples as well as the information relative to the inputs and outputs.\n</Tip>\n\n## XLMRobertaConfig\n\n[[autodoc]] XLMRobertaConfig\n\n## XLMRobertaTokenizer\n\n[[autodoc]] XLMRobertaTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## XLMRobertaTokenizerFast\n\n[[autodoc]] XLMRobertaTokenizerFast\n\n<frameworkcontent>\n<pt>\n\n## XLMRobertaModel\n\n[[autodoc]] XLMRobertaModel\n    - forward\n\n## XLMRobertaForCausalLM\n\n[[autodoc]] XLMRobertaForCausalLM\n    - forward\n\n## XLMRobertaForMaskedLM\n\n[[autodoc]] XLMRobertaForMaskedLM\n    - forward\n\n## XLMRobertaForSequenceClassification\n\n[[autodoc]] XLMRobertaForSequenceClassification\n    - forward\n\n## XLMRobertaForMultipleChoice\n\n[[autodoc]] XLMRobertaForMultipleChoice\n    - forward\n\n## XLMRobertaForTokenClassification\n\n[[autodoc]] XLMRobertaForTokenClassification\n    - forward\n\n## XLMRobertaForQuestionAnswering\n\n[[autodoc]] XLMRobertaForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFXLMRobertaModel\n\n[[autodoc]] TFXLMRobertaModel\n    - call\n\n## TFXLMRobertaForCausalLM\n\n[[autodoc]] TFXLMRobertaForCausalLM\n    - call\n\n## TFXLMRobertaForMaskedLM\n\n[[autodoc]] TFXLMRobertaForMaskedLM\n    - call\n\n## TFXLMRobertaForSequenceClassification\n\n[[autodoc]] TFXLMRobertaForSequenceClassification\n    - call\n\n## TFXLMRobertaForMultipleChoice\n\n[[autodoc]] TFXLMRobertaForMultipleChoice\n    - call\n\n## TFXLMRobertaForTokenClassification\n\n[[autodoc]] TFXLMRobertaForTokenClassification\n    - call\n\n## TFXLMRobertaForQuestionAnswering\n\n[[autodoc]] TFXLMRobertaForQuestionAnswering\n    - call\n\n</tf>\n<jax>\n\n## FlaxXLMRobertaModel\n\n[[autodoc]] FlaxXLMRobertaModel\n    - __call__\n\n## FlaxXLMRobertaForCausalLM\n\n[[autodoc]] FlaxXLMRobertaForCausalLM\n    - __call__\n\n## FlaxXLMRobertaForMaskedLM\n\n[[autodoc]] FlaxXLMRobertaForMaskedLM\n    - __call__\n\n## FlaxXLMRobertaForSequenceClassification\n\n[[autodoc]] FlaxXLMRobertaForSequenceClassification\n    - __call__\n\n## FlaxXLMRobertaForMultipleChoice\n\n[[autodoc]] FlaxXLMRobertaForMultipleChoice\n    - __call__\n\n## FlaxXLMRobertaForTokenClassification\n\n[[autodoc]] FlaxXLMRobertaForTokenClassification\n    - __call__\n\n## FlaxXLMRobertaForQuestionAnswering\n\n[[autodoc]] FlaxXLMRobertaForQuestionAnswering\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 575109e3-5669-495d-9f58-52c09626e4bb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/dit/", "title": "DiT", "tokens": 1220, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"575109e3-5669-495d-9f58-52c09626e4bb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dit/\", \"title\": \"DiT\", \"tokens\": 1220, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"804960b7-ded9-413f-b978-3c3fdad00144\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dit/\", \"title\": \"DiT\", \"tokens\": 1220, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8f35a716ae70f031165b8c7617e41160b66c3dee6fb431a505db44bcd0588e37\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"096aea10-bae3-4274-ab0e-20efab546dd5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"876660f27fc4b678cb5a0b4b1901bc313f02d3a85b06db95b633449bd9ef95fa\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2671, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "804960b7-ded9-413f-b978-3c3fdad00144", "doc_id": "804960b7-ded9-413f-b978-3c3fdad00144", "ref_doc_id": "804960b7-ded9-413f-b978-3c3fdad00144", "chroma:document": "# DiT\n\n## Overview\n\nDiT was proposed in [DiT: Self-supervised Pre-training for Document Image Transformer](https://arxiv.org/abs/2203.02378) by Junlong Li, Yiheng Xu, Tengchao Lv, Lei Cui, Cha Zhang, Furu Wei.\nDiT applies the self-supervised objective of [BEiT](beit) (BERT pre-training of Image Transformers) to 42 million document images, allowing for state-of-the-art results on tasks including:\n\n- document image classification: the [RVL-CDIP](https://www.cs.cmu.edu/~aharley/rvl-cdip/) dataset (a collection of\n  400,000 images belonging to one of 16 classes).\n- document layout analysis: the [PubLayNet](https://github.com/ibm-aur-nlp/PubLayNet) dataset (a collection of more\n  than 360,000 document images constructed by automatically parsing PubMed XML files).\n- table detection: the [ICDAR 2019 cTDaR](https://github.com/cndplab-founder/ICDAR2019_cTDaR) dataset (a collection of\n  600 training images and 240 testing images).\n\nThe abstract from the paper is the following:\n\n*Image Transformer has recently achieved significant progress for natural image understanding, either using supervised (ViT, DeiT, etc.) or self-supervised (BEiT, MAE, etc.) pre-training techniques. In this paper, we propose DiT, a self-supervised pre-trained Document Image Transformer model using large-scale unlabeled text images for Document AI tasks, which is essential since no supervised counterparts ever exist due to the lack of human labeled document images. We leverage DiT as the backbone network in a variety of vision-based Document AI tasks, including document image classification, document layout analysis, as well as table detection. Experiment results have illustrated that the self-supervised pre-trained DiT model achieves new state-of-the-art results on these downstream tasks, e.g. document image classification (91.11 \u2192 92.69), document layout analysis (91.0 \u2192 94.9) and table detection (94.23 \u2192 96.55). *\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/dit_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/> \n\n<small> Summary of the approach. Taken from the [original paper](https://arxiv.org/abs/2203.02378). </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/microsoft/unilm/tree/master/dit).\n\n## Usage tips\n\nOne can directly use the weights of DiT with the AutoModel API:\n\n```python\nfrom transformers import AutoModel\n\nmodel = AutoModel.from_pretrained(\"microsoft/dit-base\")\n```\n\nThis will load the model pre-trained on masked image modeling. Note that this won't include the language modeling head on top, used to predict visual tokens."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 096aea10-bae3-4274-ab0e-20efab546dd5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/dit/", "title": "DiT", "tokens": 1220, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"096aea10-bae3-4274-ab0e-20efab546dd5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dit/\", \"title\": \"DiT\", \"tokens\": 1220, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"804960b7-ded9-413f-b978-3c3fdad00144\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dit/\", \"title\": \"DiT\", \"tokens\": 1220, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8f35a716ae70f031165b8c7617e41160b66c3dee6fb431a505db44bcd0588e37\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"575109e3-5669-495d-9f58-52c09626e4bb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dit/\", \"title\": \"DiT\", \"tokens\": 1220, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"e9c8f8ad59d7450ac4ec650f324213ea435f36d7b092cea74c8d00a2a6d636ae\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f0e18bd2-2c6a-4670-8cac-224896c68a48\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8c6628c7afc940abd54804257dbf62455498a0e0ee6e9b3119b0c8272d6f26ff\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 936, \"end_char_idx\": 4283, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "804960b7-ded9-413f-b978-3c3fdad00144", "doc_id": "804960b7-ded9-413f-b978-3c3fdad00144", "ref_doc_id": "804960b7-ded9-413f-b978-3c3fdad00144", "chroma:document": "The abstract from the paper is the following:\n\n*Image Transformer has recently achieved significant progress for natural image understanding, either using supervised (ViT, DeiT, etc.) or self-supervised (BEiT, MAE, etc.) pre-training techniques. In this paper, we propose DiT, a self-supervised pre-trained Document Image Transformer model using large-scale unlabeled text images for Document AI tasks, which is essential since no supervised counterparts ever exist due to the lack of human labeled document images. We leverage DiT as the backbone network in a variety of vision-based Document AI tasks, including document image classification, document layout analysis, as well as table detection. Experiment results have illustrated that the self-supervised pre-trained DiT model achieves new state-of-the-art results on these downstream tasks, e.g. document image classification (91.11 \u2192 92.69), document layout analysis (91.0 \u2192 94.9) and table detection (94.23 \u2192 96.55). *\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/dit_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/> \n\n<small> Summary of the approach. Taken from the [original paper](https://arxiv.org/abs/2203.02378). </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/microsoft/unilm/tree/master/dit).\n\n## Usage tips\n\nOne can directly use the weights of DiT with the AutoModel API:\n\n```python\nfrom transformers import AutoModel\n\nmodel = AutoModel.from_pretrained(\"microsoft/dit-base\")\n```\n\nThis will load the model pre-trained on masked image modeling. Note that this won't include the language modeling head on top, used to predict visual tokens.\n\nTo include the head, you can load the weights into a `BeitForMaskedImageModeling` model, like so:\n\n```python\nfrom transformers import BeitForMaskedImageModeling\n\nmodel = BeitForMaskedImageModeling.from_pretrained(\"microsoft/dit-base\")\n```\n\nYou can also load a fine-tuned model from the [hub](https://huggingface.co/models?other=dit), like so:\n\n```python\nfrom transformers import AutoModelForImageClassification\n\nmodel = AutoModelForImageClassification.from_pretrained(\"microsoft/dit-base-finetuned-rvlcdip\")\n```\n\nThis particular checkpoint was fine-tuned on [RVL-CDIP](https://www.cs.cmu.edu/~aharley/rvl-cdip/), an important benchmark for document image classification.\nA notebook that illustrates inference for document image classification can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/DiT/Inference_with_DiT_(Document_Image_Transformer)_for_document_image_classification.ipynb).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with DiT.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`BeitForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: f0e18bd2-2c6a-4670-8cac-224896c68a48
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/dit/", "title": "DiT", "tokens": 1220, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"f0e18bd2-2c6a-4670-8cac-224896c68a48\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dit/\", \"title\": \"DiT\", \"tokens\": 1220, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"804960b7-ded9-413f-b978-3c3fdad00144\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dit/\", \"title\": \"DiT\", \"tokens\": 1220, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8f35a716ae70f031165b8c7617e41160b66c3dee6fb431a505db44bcd0588e37\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"096aea10-bae3-4274-ab0e-20efab546dd5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dit/\", \"title\": \"DiT\", \"tokens\": 1220, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5f4b9a1e09ac60a1969d147a350724aff6b1d05b9384a0ca9dd0c061e7581f64\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2577, \"end_char_idx\": 4447, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "804960b7-ded9-413f-b978-3c3fdad00144", "doc_id": "804960b7-ded9-413f-b978-3c3fdad00144", "ref_doc_id": "804960b7-ded9-413f-b978-3c3fdad00144", "chroma:document": "Note that this won't include the language modeling head on top, used to predict visual tokens.\n\nTo include the head, you can load the weights into a `BeitForMaskedImageModeling` model, like so:\n\n```python\nfrom transformers import BeitForMaskedImageModeling\n\nmodel = BeitForMaskedImageModeling.from_pretrained(\"microsoft/dit-base\")\n```\n\nYou can also load a fine-tuned model from the [hub](https://huggingface.co/models?other=dit), like so:\n\n```python\nfrom transformers import AutoModelForImageClassification\n\nmodel = AutoModelForImageClassification.from_pretrained(\"microsoft/dit-base-finetuned-rvlcdip\")\n```\n\nThis particular checkpoint was fine-tuned on [RVL-CDIP](https://www.cs.cmu.edu/~aharley/rvl-cdip/), an important benchmark for document image classification.\nA notebook that illustrates inference for document image classification can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/DiT/Inference_with_DiT_(Document_Image_Transformer)_for_document_image_classification.ipynb).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with DiT.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`BeitForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<Tip>\n\n  As DiT's architecture is equivalent to that of BEiT, one can refer to [BEiT's documentation page](beit) for all tips, code examples and notebooks.\n</Tip>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 8361256b-1b9d-41cd-8096-8ea20a603eb8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tvlt/", "title": "TVLT", "tokens": 1057, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"8361256b-1b9d-41cd-8096-8ea20a603eb8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tvlt/\", \"title\": \"TVLT\", \"tokens\": 1057, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"42ad87aa-2e8b-400b-adc8-4073ca04b6e0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tvlt/\", \"title\": \"TVLT\", \"tokens\": 1057, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f06c5aa6215ca581d7876eb3a315c5cbb77d79f49e4c29c6ff19b6a1eb19a21a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"50f96435-4e96-422c-aec3-6146c19e9e78\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"da9fa83b82e1fb342718f431af64877b257802249ba234567691f7a88565c418\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3356, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "42ad87aa-2e8b-400b-adc8-4073ca04b6e0", "doc_id": "42ad87aa-2e8b-400b-adc8-4073ca04b6e0", "ref_doc_id": "42ad87aa-2e8b-400b-adc8-4073ca04b6e0", "chroma:document": "# TVLT\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we don't accept any new PRs changing its code.\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.\nYou can do so by running the following command: `pip install -U transformers==4.40.2`.\n\n</Tip>\n\n## Overview\n\nThe TVLT model was proposed in [TVLT: Textless Vision-Language Transformer](https://arxiv.org/abs/2209.14156)\nby Zineng Tang, Jaemin Cho, Yixin Nie, Mohit Bansal (the first three authors contributed equally). The Textless Vision-Language Transformer (TVLT) is a model that uses raw visual and audio inputs for vision-and-language representation learning, without using text-specific modules such as tokenization or automatic speech recognition (ASR). It can perform various audiovisual and vision-language tasks like retrieval, question answering, etc.\n\nThe abstract from the paper is the following:\n\n*In this work, we present the Textless Vision-Language Transformer (TVLT), where homogeneous transformer blocks take raw visual and audio inputs for vision-and-language representation learning with minimal modality-specific design, and do not use text-specific modules such as tokenization or automatic speech recognition (ASR). TVLT is trained by reconstructing masked patches of continuous video frames and audio spectrograms (masked autoencoding) and contrastive modeling to align video and audio. TVLT attains performance comparable to its text-based counterpart on various multimodal tasks, such as visual question answering, image retrieval, video retrieval, and multimodal sentiment analysis, with 28x faster inference speed and only 1/3 of the parameters. Our findings suggest the possibility of learning compact and efficient visual-linguistic representations from low-level visual and audio signals without assuming the prior existence of text.*\n\n<p align=\"center\">\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/tvlt_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n</p>\n\n<small> TVLT architecture. Taken from the <a href=\"[https://arxiv.org/abs/2102.03334](https://arxiv.org/abs/2209.14156)\">original paper</a>. </small>\n\nThe original code can be found [here](https://github.com/zinengtang/TVLT). This model was contributed by [Zineng Tang](https://huggingface.co/ZinengTang).\n\n## Usage tips\n\n- TVLT is a model that takes both `pixel_values` and `audio_values` as input. One can use [`TvltProcessor`] to prepare data for the model.\n  This processor wraps an image processor (for the image/video modality) and an audio feature extractor (for the audio modality) into one.\n- TVLT is trained with images/videos and audios of various sizes: the authors resize and crop the input images/videos to 224 and limit the length of audio spectrogram to 2048. To make batching of videos and audios possible, the authors use a `pixel_mask` that indicates which pixels are real/padding and `audio_mask` that indicates which audio values are real/padding.\n- The design of TVLT is very similar to that of a standard Vision Transformer (ViT) and masked autoencoder (MAE) as in [ViTMAE](vitmae). The difference is that the model includes embedding layers for the audio modality.\n- The PyTorch version of this model is only available in torch 1.10 and higher."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 50f96435-4e96-422c-aec3-6146c19e9e78
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tvlt/", "title": "TVLT", "tokens": 1057, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"50f96435-4e96-422c-aec3-6146c19e9e78\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tvlt/\", \"title\": \"TVLT\", \"tokens\": 1057, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"42ad87aa-2e8b-400b-adc8-4073ca04b6e0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tvlt/\", \"title\": \"TVLT\", \"tokens\": 1057, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f06c5aa6215ca581d7876eb3a315c5cbb77d79f49e4c29c6ff19b6a1eb19a21a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8361256b-1b9d-41cd-8096-8ea20a603eb8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tvlt/\", \"title\": \"TVLT\", \"tokens\": 1057, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"61489dab0240d6339e85adf3d818b2397ead727c4f71c17495b3d1f2d8a2e331\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1898, \"end_char_idx\": 3818, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "42ad87aa-2e8b-400b-adc8-4073ca04b6e0", "doc_id": "42ad87aa-2e8b-400b-adc8-4073ca04b6e0", "ref_doc_id": "42ad87aa-2e8b-400b-adc8-4073ca04b6e0", "chroma:document": "*\n\n<p align=\"center\">\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/tvlt_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n</p>\n\n<small> TVLT architecture. Taken from the <a href=\"[https://arxiv.org/abs/2102.03334](https://arxiv.org/abs/2209.14156)\">original paper</a>. </small>\n\nThe original code can be found [here](https://github.com/zinengtang/TVLT). This model was contributed by [Zineng Tang](https://huggingface.co/ZinengTang).\n\n## Usage tips\n\n- TVLT is a model that takes both `pixel_values` and `audio_values` as input. One can use [`TvltProcessor`] to prepare data for the model.\n  This processor wraps an image processor (for the image/video modality) and an audio feature extractor (for the audio modality) into one.\n- TVLT is trained with images/videos and audios of various sizes: the authors resize and crop the input images/videos to 224 and limit the length of audio spectrogram to 2048. To make batching of videos and audios possible, the authors use a `pixel_mask` that indicates which pixels are real/padding and `audio_mask` that indicates which audio values are real/padding.\n- The design of TVLT is very similar to that of a standard Vision Transformer (ViT) and masked autoencoder (MAE) as in [ViTMAE](vitmae). The difference is that the model includes embedding layers for the audio modality.\n- The PyTorch version of this model is only available in torch 1.10 and higher.\n\n## TvltConfig\n\n[[autodoc]] TvltConfig\n\n## TvltProcessor\n\n[[autodoc]] TvltProcessor\n    - __call__\n\n## TvltImageProcessor\n\n[[autodoc]] TvltImageProcessor\n    - preprocess\n\n## TvltFeatureExtractor\n\n[[autodoc]] TvltFeatureExtractor\n    - __call__\n\n## TvltModel\n\n[[autodoc]] TvltModel\n    - forward\n\n## TvltForPreTraining\n\n[[autodoc]] TvltForPreTraining\n    - forward\n\n## TvltForAudioVisualClassification\n\n[[autodoc]] TvltForAudioVisualClassification\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 0bf49dae-8efe-46c9-ae07-62817455722b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/time_series_transformer/", "title": "Time Series Transformer", "tokens": 1177, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"0bf49dae-8efe-46c9-ae07-62817455722b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/time_series_transformer/\", \"title\": \"Time Series Transformer\", \"tokens\": 1177, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9da60dba-231a-4caa-9871-9f4c46815182\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/time_series_transformer/\", \"title\": \"Time Series Transformer\", \"tokens\": 1177, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"95196beee01a5062805679a751d1f257843c080d132f819afed806afca42606d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a7f3cc52-01c1-4de4-bf91-00eee6c70aba\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8494db7d7bea4a45969be730d77aa597aea130a6b8af4b027529345a48f9b07d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3283, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9da60dba-231a-4caa-9871-9f4c46815182", "doc_id": "9da60dba-231a-4caa-9871-9f4c46815182", "ref_doc_id": "9da60dba-231a-4caa-9871-9f4c46815182", "chroma:document": "# Time Series Transformer\n\n## Overview\n\nThe Time Series Transformer model is a vanilla encoder-decoder Transformer for time series forecasting.\nThis model was contributed by [kashif](https://huggingface.co/kashif).\n\n## Usage tips\n\n- Similar to other models in the library, [`TimeSeriesTransformerModel`] is the raw Transformer without any head on top, and [`TimeSeriesTransformerForPrediction`]\nadds a distribution head on top of the former, which can be used for time-series forecasting. Note that this is a so-called probabilistic forecasting model, not a\npoint forecasting model. This means that the model learns a distribution, from which one can sample. The model doesn't directly output values.\n- [`TimeSeriesTransformerForPrediction`] consists of 2 blocks: an encoder, which takes a `context_length` of time series values as input (called `past_values`),\nand a decoder, which predicts a `prediction_length` of time series values into the future (called `future_values`). During training, one needs to provide\npairs of (`past_values` and `future_values`) to the model.\n- In addition to the raw (`past_values` and `future_values`), one typically provides additional features to the model. These can be the following:\n    - `past_time_features`: temporal features which the model will add to `past_values`. These serve as \"positional encodings\" for the Transformer encoder.\n    Examples are \"day of the month\", \"month of the year\", etc. as scalar values (and then stacked together as a vector).\n    e.g. if a given time-series value was obtained on the 11th of August, then one could have [11, 8] as time feature vector (11 being \"day of the month\", 8 being \"month of the year\").\n    - `future_time_features`: temporal features which the model will add to `future_values`. These serve as \"positional encodings\" for the Transformer decoder.\n    Examples are \"day of the month\", \"month of the year\", etc. as scalar values (and then stacked together as a vector).\n    e.g. if a given time-series value was obtained on the 11th of August, then one could have [11, 8] as time feature vector (11 being \"day of the month\", 8 being \"month of the year\").\n    - `static_categorical_features`: categorical features which are static over time (i.e., have the same value for all `past_values` and `future_values`).\n    An example here is the store ID or region ID that identifies a given time-series.\n    Note that these features need to be known for ALL data points (also those in the future).\n    - `static_real_features`: real-valued features which are static over time (i.e., have the same value for all `past_values` and `future_values`).\n    An example here is the image representation of the product for which you have the time-series values (like the [ResNet](resnet) embedding of a \"shoe\" picture,\n    if your time-series is about the sales of shoes).\n    Note that these features need to be known for ALL data points (also those in the future).\n- The model is trained using \"teacher-forcing\", similar to how a Transformer is trained for machine translation. This means that, during training, one shifts the\n`future_values` one position to the right as input to the decoder, prepended by the last value of `past_values`. At each time step, the model needs to predict the\nnext target."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: a7f3cc52-01c1-4de4-bf91-00eee6c70aba
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/time_series_transformer/", "title": "Time Series Transformer", "tokens": 1177, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"a7f3cc52-01c1-4de4-bf91-00eee6c70aba\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/time_series_transformer/\", \"title\": \"Time Series Transformer\", \"tokens\": 1177, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9da60dba-231a-4caa-9871-9f4c46815182\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/time_series_transformer/\", \"title\": \"Time Series Transformer\", \"tokens\": 1177, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"95196beee01a5062805679a751d1f257843c080d132f819afed806afca42606d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0bf49dae-8efe-46c9-ae07-62817455722b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/time_series_transformer/\", \"title\": \"Time Series Transformer\", \"tokens\": 1177, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8f00b0fa0134acbcacd5f6870a84019d68387d0e2cd677972e8bc98e18169387\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1688, \"end_char_idx\": 4552, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9da60dba-231a-4caa-9871-9f4c46815182", "doc_id": "9da60dba-231a-4caa-9871-9f4c46815182", "ref_doc_id": "9da60dba-231a-4caa-9871-9f4c46815182", "chroma:document": "- `future_time_features`: temporal features which the model will add to `future_values`. These serve as \"positional encodings\" for the Transformer decoder.\n    Examples are \"day of the month\", \"month of the year\", etc. as scalar values (and then stacked together as a vector).\n    e.g. if a given time-series value was obtained on the 11th of August, then one could have [11, 8] as time feature vector (11 being \"day of the month\", 8 being \"month of the year\").\n    - `static_categorical_features`: categorical features which are static over time (i.e., have the same value for all `past_values` and `future_values`).\n    An example here is the store ID or region ID that identifies a given time-series.\n    Note that these features need to be known for ALL data points (also those in the future).\n    - `static_real_features`: real-valued features which are static over time (i.e., have the same value for all `past_values` and `future_values`).\n    An example here is the image representation of the product for which you have the time-series values (like the [ResNet](resnet) embedding of a \"shoe\" picture,\n    if your time-series is about the sales of shoes).\n    Note that these features need to be known for ALL data points (also those in the future).\n- The model is trained using \"teacher-forcing\", similar to how a Transformer is trained for machine translation. This means that, during training, one shifts the\n`future_values` one position to the right as input to the decoder, prepended by the last value of `past_values`. At each time step, the model needs to predict the\nnext target. So the set-up of training is similar to a GPT model for language, except that there's no notion of `decoder_start_token_id` (we just use the last value\nof the context as initial input for the decoder).\n- At inference time, we give the final value of the `past_values` as input to the decoder. Next, we can sample from the model to make a prediction at the next time step,\nwhich is then fed to the decoder in order to make the next prediction (also called autoregressive generation).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n- Check out the Time Series Transformer blog-post in HuggingFace blog: [Probabilistic Time Series Forecasting with \ud83e\udd17 Transformers](https://huggingface.co/blog/time-series-transformers)\n\n\n## TimeSeriesTransformerConfig\n\n[[autodoc]] TimeSeriesTransformerConfig\n\n## TimeSeriesTransformerModel\n\n[[autodoc]] TimeSeriesTransformerModel\n    - forward\n\n## TimeSeriesTransformerForPrediction\n\n[[autodoc]] TimeSeriesTransformerForPrediction\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: b8ccdc9e-b964-4494-968f-4aa2376406a5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neo/", "title": "GPT Neo", "tokens": 1260, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"b8ccdc9e-b964-4494-968f-4aa2376406a5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neo/\", \"title\": \"GPT Neo\", \"tokens\": 1260, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1376d593-049c-4024-a87d-03c2e9ae5fcb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neo/\", \"title\": \"GPT Neo\", \"tokens\": 1260, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"bb5061f9a5e15d210df6378dc0efdc31d2b07e7f57492d2175b52ebdb122c328\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"05aba05a-31a8-458c-a571-71d456d18f20\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"be364c94058186bd36c8af94c1aae17905e36bb9ddd49274a1afb06915b5065d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2840, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1376d593-049c-4024-a87d-03c2e9ae5fcb", "doc_id": "1376d593-049c-4024-a87d-03c2e9ae5fcb", "ref_doc_id": "1376d593-049c-4024-a87d-03c2e9ae5fcb", "chroma:document": "# GPT Neo\n\n## Overview\n\nThe GPTNeo model was released in the [EleutherAI/gpt-neo](https://github.com/EleutherAI/gpt-neo) repository by Sid\nBlack, Stella Biderman, Leo Gao, Phil Wang and Connor Leahy. It is a GPT2 like causal language model trained on the\n[Pile](https://pile.eleuther.ai/) dataset.\n\nThe architecture is similar to GPT2 except that GPT Neo uses local attention in every other layer with a window size of\n256 tokens.\n\nThis model was contributed by [valhalla](https://huggingface.co/valhalla).\n\n## Usage example\n\nThe `generate()` method can be used to generate text using GPT Neo model.\n\n```python\n>>> from transformers import GPTNeoForCausalLM, GPT2Tokenizer\n\n>>> model = GPTNeoForCausalLM.from_pretrained(\"EleutherAI/gpt-neo-1.3B\")\n>>> tokenizer = GPT2Tokenizer.from_pretrained(\"EleutherAI/gpt-neo-1.3B\")\n\n>>> prompt = (\n...     \"In a shocking finding, scientists discovered a herd of unicorns living in a remote, \"\n...     \"previously unexplored valley, in the Andes Mountains. Even more surprising to the \"\n...     \"researchers was the fact that the unicorns spoke perfect English.\"\n... )\n\n>>> input_ids = tokenizer(prompt, return_tensors=\"pt\").input_ids\n\n>>> gen_tokens = model.generate(\n...     input_ids,\n...     do_sample=True,\n...     temperature=0.9,\n...     max_length=100,\n... )\n>>> gen_text = tokenizer.batch_decode(gen_tokens)[0]\n```\n\n## Combining GPT-Neo and Flash Attention 2\n\nFirst, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature, and make sure your hardware is compatible with Flash-Attention 2. More details are available [here](https://huggingface.co/docs/transformers/perf_infer_gpu_one#flashattention-2) concerning the installation.\n\nMake sure as well to load your model in half-precision (e.g. `torch.float16`).\n\nTo load and run a model using Flash Attention 2, refer to the snippet below:\n\n```python\n>>> import torch\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n>>> device = \"cuda\" # the device to load the model onto\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"EleutherAI/gpt-neo-2.7B\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"EleutherAI/gpt-neo-2.7B\")\n\n>>> prompt = \"def hello_world():\"\n\n>>> model_inputs = tokenizer([prompt], return_tensors=\"pt\").to(device)\n>>> model.to(device)\n\n>>> generated_ids = model.generate(**model_inputs, max_new_tokens=100, do_sample=True)\n>>> tokenizer.batch_decode(generated_ids)[0]\n\"def hello_world():\\n    >>> run_script(\"hello.py\")\\n    >>> exit(0)\\n<|endoftext|>\"\n```\n\n### Expected speedups\n\nBelow is an expected speedup diagram that compares pure inference time between the native implementation in transformers using `EleutherAI/gpt-neo-2.7B` checkpoint and the Flash Attention 2 version of the model."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 05aba05a-31a8-458c-a571-71d456d18f20
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neo/", "title": "GPT Neo", "tokens": 1260, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"05aba05a-31a8-458c-a571-71d456d18f20\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neo/\", \"title\": \"GPT Neo\", \"tokens\": 1260, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1376d593-049c-4024-a87d-03c2e9ae5fcb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neo/\", \"title\": \"GPT Neo\", \"tokens\": 1260, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"bb5061f9a5e15d210df6378dc0efdc31d2b07e7f57492d2175b52ebdb122c328\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b8ccdc9e-b964-4494-968f-4aa2376406a5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neo/\", \"title\": \"GPT Neo\", \"tokens\": 1260, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"7d570bc28ea89da1e60dbd6dc9fa6882f76164d3a81d2fae02760dd263ea0a10\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1589, \"end_char_idx\": 4065, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1376d593-049c-4024-a87d-03c2e9ae5fcb", "doc_id": "1376d593-049c-4024-a87d-03c2e9ae5fcb", "ref_doc_id": "1376d593-049c-4024-a87d-03c2e9ae5fcb", "chroma:document": "More details are available [here](https://huggingface.co/docs/transformers/perf_infer_gpu_one#flashattention-2) concerning the installation.\n\nMake sure as well to load your model in half-precision (e.g. `torch.float16`).\n\nTo load and run a model using Flash Attention 2, refer to the snippet below:\n\n```python\n>>> import torch\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n>>> device = \"cuda\" # the device to load the model onto\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"EleutherAI/gpt-neo-2.7B\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"EleutherAI/gpt-neo-2.7B\")\n\n>>> prompt = \"def hello_world():\"\n\n>>> model_inputs = tokenizer([prompt], return_tensors=\"pt\").to(device)\n>>> model.to(device)\n\n>>> generated_ids = model.generate(**model_inputs, max_new_tokens=100, do_sample=True)\n>>> tokenizer.batch_decode(generated_ids)[0]\n\"def hello_world():\\n    >>> run_script(\"hello.py\")\\n    >>> exit(0)\\n<|endoftext|>\"\n```\n\n### Expected speedups\n\nBelow is an expected speedup diagram that compares pure inference time between the native implementation in transformers using `EleutherAI/gpt-neo-2.7B` checkpoint and the Flash Attention 2 version of the model.\nNote that for GPT-Neo it is not possible to train / run on very long context as the max [position embeddings](https://huggingface.co/EleutherAI/gpt-neo-2.7B/blob/main/config.json#L58 ) is limited to 2048 - but this is applicable to all gpt-neo models and not specific to FA-2\n\n<div style=\"text-align: center\">\n<img src=\"https://user-images.githubusercontent.com/49240599/272241893-b1c66b75-3a48-4265-bc47-688448568b3d.png\">\n</div>\n\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Causal language modeling task guide](../tasks/language_modeling)\n\n## GPTNeoConfig\n\n[[autodoc]] GPTNeoConfig\n\n\n<frameworkcontent>\n<pt>\n\n## GPTNeoModel\n\n[[autodoc]] GPTNeoModel\n    - forward\n\n## GPTNeoForCausalLM\n\n[[autodoc]] GPTNeoForCausalLM\n    - forward\n\n## GPTNeoForQuestionAnswering\n\n[[autodoc]] GPTNeoForQuestionAnswering\n    - forward\n\n## GPTNeoForSequenceClassification\n\n[[autodoc]] GPTNeoForSequenceClassification\n    - forward\n\n## GPTNeoForTokenClassification\n\n[[autodoc]] GPTNeoForTokenClassification\n    - forward\n\n</pt>\n<jax>\n\n## FlaxGPTNeoModel\n\n[[autodoc]] FlaxGPTNeoModel\n    - __call__\n\n## FlaxGPTNeoForCausalLM\n\n[[autodoc]] FlaxGPTNeoForCausalLM\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 339d8a31-5bff-433a-9f8e-71372d5a2b21
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/hubert/", "title": "Hubert", "tokens": 1248, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"339d8a31-5bff-433a-9f8e-71372d5a2b21\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/hubert/\", \"title\": \"Hubert\", \"tokens\": 1248, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b7f1072e-1a91-414a-aea9-430297cc1d76\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/hubert/\", \"title\": \"Hubert\", \"tokens\": 1248, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a344f45d49d80cc8eb1bd2f1887412ddeed92e89e50c256005d3d5e8574bc725\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e0076444-1033-4492-a73c-c5dea72ecd82\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"98c38ade1627c5ba36db2365fbeed9b914d7ddc61f9d73fc2cc204a14afa749b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2064, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b7f1072e-1a91-414a-aea9-430297cc1d76", "doc_id": "b7f1072e-1a91-414a-aea9-430297cc1d76", "ref_doc_id": "b7f1072e-1a91-414a-aea9-430297cc1d76", "chroma:document": "# Hubert\n\n## Overview\n\nHubert was proposed in [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://arxiv.org/abs/2106.07447) by Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan\nSalakhutdinov, Abdelrahman Mohamed.\n\nThe abstract from the paper is the following:\n\n*Self-supervised approaches for speech representation learning are challenged by three unique problems: (1) there are\nmultiple sound units in each input utterance, (2) there is no lexicon of input sound units during the pre-training\nphase, and (3) sound units have variable lengths with no explicit segmentation. To deal with these three problems, we\npropose the Hidden-Unit BERT (HuBERT) approach for self-supervised speech representation learning, which utilizes an\noffline clustering step to provide aligned target labels for a BERT-like prediction loss. A key ingredient of our\napproach is applying the prediction loss over the masked regions only, which forces the model to learn a combined\nacoustic and language model over the continuous inputs. HuBERT relies primarily on the consistency of the unsupervised\nclustering step rather than the intrinsic quality of the assigned cluster labels. Starting with a simple k-means\nteacher of 100 clusters, and using two iterations of clustering, the HuBERT model either matches or improves upon the\nstate-of-the-art wav2vec 2.0 performance on the Librispeech (960h) and Libri-light (60,000h) benchmarks with 10min, 1h,\n10h, 100h, and 960h fine-tuning subsets. Using a 1B parameter model, HuBERT shows up to 19% and 13% relative WER\nreduction on the more challenging dev-other and test-other evaluation subsets.*\n\nThis model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten).\n\n# Usage tips\n\n- Hubert is a speech model that accepts a float array corresponding to the raw waveform of the speech signal.\n- Hubert model was fine-tuned using connectionist temporal classification (CTC) so the model output has to be decoded\n  using [`Wav2Vec2CTCTokenizer`]."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: e0076444-1033-4492-a73c-c5dea72ecd82
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/hubert/", "title": "Hubert", "tokens": 1248, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"e0076444-1033-4492-a73c-c5dea72ecd82\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/hubert/\", \"title\": \"Hubert\", \"tokens\": 1248, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b7f1072e-1a91-414a-aea9-430297cc1d76\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/hubert/\", \"title\": \"Hubert\", \"tokens\": 1248, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a344f45d49d80cc8eb1bd2f1887412ddeed92e89e50c256005d3d5e8574bc725\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"339d8a31-5bff-433a-9f8e-71372d5a2b21\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/hubert/\", \"title\": \"Hubert\", \"tokens\": 1248, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b32923070a57857e4902f2f7c8d8f2eadccba7a599fa150bfec923a0f2d2f511\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2067, \"end_char_idx\": 4488, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b7f1072e-1a91-414a-aea9-430297cc1d76", "doc_id": "b7f1072e-1a91-414a-aea9-430297cc1d76", "ref_doc_id": "b7f1072e-1a91-414a-aea9-430297cc1d76", "chroma:document": "## Using Flash Attention 2\n\nFlash Attention 2 is an faster, optimized version of the model.\n\n### Installation \n\nFirst, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features). If your hardware is not compatible with Flash Attention 2, you can still benefit from attention kernel optimisations through Better Transformer support covered [above](https://huggingface.co/docs/transformers/main/en/model_doc/bark#using-better-transformer).\n\nNext, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\n### Usage\n\nBelow is an expected speedup diagram comparing the pure inference time between the native implementation in transformers of `facebook/hubert-large-ls960-ft`, the flash-attention-2 and the sdpa (scale-dot-product-attention) version. We show the average speedup obtained on the `librispeech_asr` `clean` validation split: \n\n```python\n>>> from transformers import Wav2Vec2Model\n\nmodel = Wav2Vec2Model.from_pretrained(\"facebook/hubert-large-ls960-ft\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\").to(device)\n...\n```\n\n### Expected speedups\n\nBelow is an expected speedup diagram comparing the pure inference time between the native implementation in transformers of the `facebook/hubert-large-ls960-ft` model and the flash-attention-2 and sdpa (scale-dot-product-attention) versions. . We show the average speedup obtained on the `librispeech_asr` `clean` validation split: \n\n\n<div style=\"text-align: center\">\n<img src=\"https://huggingface.co/datasets/kamilakesbi/transformers_image_doc/resolve/main/data/Hubert_speedup.png\">\n</div>\n\n\n## Resources\n\n- [Audio classification task guide](../tasks/audio_classification)\n- [Automatic speech recognition task guide](../tasks/asr)\n\n## HubertConfig\n\n[[autodoc]] HubertConfig\n\n<frameworkcontent>\n<pt>\n\n## HubertModel\n\n[[autodoc]] HubertModel\n    - forward\n\n## HubertForCTC\n\n[[autodoc]] HubertForCTC\n    - forward\n\n## HubertForSequenceClassification\n\n[[autodoc]] HubertForSequenceClassification\n    - forward\n\n</pt>\n<tf>\n\n## TFHubertModel\n\n[[autodoc]] TFHubertModel\n    - call\n\n## TFHubertForCTC\n\n[[autodoc]] TFHubertForCTC\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 9dd3246c-e0ef-4d93-9461-93335965d280
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/qwen2/", "title": "Qwen2", "tokens": 795, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"9dd3246c-e0ef-4d93-9461-93335965d280\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qwen2/\", \"title\": \"Qwen2\", \"tokens\": 795, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1699721f-2a3e-4992-81ea-d23f85d0021a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qwen2/\", \"title\": \"Qwen2\", \"tokens\": 795, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"234695ff88b3ee9c50eecb379c684cd86169511124aa96a08f38653b976f2420\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2374, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1699721f-2a3e-4992-81ea-d23f85d0021a", "doc_id": "1699721f-2a3e-4992-81ea-d23f85d0021a", "ref_doc_id": "1699721f-2a3e-4992-81ea-d23f85d0021a", "chroma:document": "# Qwen2\n\n## Overview\n\nQwen2 is the new model series of large language models from the Qwen team. Previously, we released the Qwen series, including Qwen2-0.5B, Qwen2-1.5B, Qwen2-7B, Qwen2-57B-A14B, Qwen2-72B, Qwen2-Audio, etc.\n\n### Model Details\n\nQwen2 is a language model series including decoder language models of different model sizes. For each size, we release the base language model and the aligned chat model. It is based on the Transformer architecture with SwiGLU activation, attention QKV bias, group query attention, mixture of sliding window attention and full attention, etc. Additionally, we have an improved tokenizer adaptive to multiple natural languages and codes.\n\n\n## Usage tips\n\n`Qwen2-7B` and `Qwen2-7B-Instruct` can be found on the [Huggingface Hub](https://huggingface.co/Qwen)\n\nIn the following, we demonstrate how to use `Qwen2-7B-Instruct` for the inference. Note that we have used the ChatML format for dialog, in this demo we show how to leverage `apply_chat_template` for this purpose.\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n>>> device = \"cuda\" # the device to load the model onto\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen2-7B-Instruct\", device_map=\"auto\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen2-7B-Instruct\")\n\n>>> prompt = \"Give me a short introduction to large language model.\"\n\n>>> messages = [{\"role\": \"user\", \"content\": prompt}]\n\n>>> text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)\n\n>>> model_inputs = tokenizer([text], return_tensors=\"pt\").to(device)\n\n>>> generated_ids = model.generate(model_inputs.input_ids, max_new_tokens=512, do_sample=True)\n\n>>> generated_ids = [output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)]\n\n>>> response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]\n```\n\n## Qwen2Config\n\n[[autodoc]] Qwen2Config\n\n## Qwen2Tokenizer\n\n[[autodoc]] Qwen2Tokenizer\n    - save_vocabulary\n\n## Qwen2TokenizerFast\n\n[[autodoc]] Qwen2TokenizerFast\n\n## Qwen2Model\n\n[[autodoc]] Qwen2Model\n    - forward\n\n## Qwen2ForCausalLM\n\n[[autodoc]] Qwen2ForCausalLM\n    - forward\n\n## Qwen2ForSequenceClassification\n\n[[autodoc]] Qwen2ForSequenceClassification\n    - forward\n\n## Qwen2ForTokenClassification\n\n[[autodoc]] Qwen2ForTokenClassification\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 0717aa44-58bc-4731-a4bc-b709baba8f5b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/zoedepth/", "title": "ZoeDepth", "tokens": 1267, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"0717aa44-58bc-4731-a4bc-b709baba8f5b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/zoedepth/\", \"title\": \"ZoeDepth\", \"tokens\": 1267, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2cff7bf1-974b-4112-a580-43cbd47a147b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/zoedepth/\", \"title\": \"ZoeDepth\", \"tokens\": 1267, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"9a669a856c177f9e627d0397a7b1d305ab7ac019e35461418a9e5bca959f4b66\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7b0f5836-ed6a-4ea1-860a-bbc7ad32336d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"df4e7cbd0e87004a0f349e98dbbafbfff98526700d153be245c358a6a08a2f21\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2839, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2cff7bf1-974b-4112-a580-43cbd47a147b", "doc_id": "2cff7bf1-974b-4112-a580-43cbd47a147b", "ref_doc_id": "2cff7bf1-974b-4112-a580-43cbd47a147b", "chroma:document": "# ZoeDepth\n\n## Overview\n\nThe ZoeDepth model was proposed in [ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth](https://arxiv.org/abs/2302.12288) by Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, Matthias M\u00fcller. ZoeDepth extends the [DPT](dpt) framework for metric (also called absolute) depth estimation. ZoeDepth is pre-trained on 12 datasets using relative depth and fine-tuned on two domains (NYU and KITTI) using metric depth. A lightweight head is used with a novel bin adjustment design called metric bins module for each domain. During inference, each input image is automatically routed to the appropriate head using a latent classifier.\n\nThe abstract from the paper is the following:\n\n*This paper tackles the problem of depth estimation from a single image. Existing work either focuses on generalization performance disregarding metric scale, i.e. relative depth estimation, or state-of-the-art results on specific datasets, i.e. metric depth estimation. We propose the first approach that combines both worlds, leading to a model with excellent generalization performance while maintaining metric scale. Our flagship model, ZoeD-M12-NK, is pre-trained on 12 datasets using relative depth and fine-tuned on two datasets using metric depth. We use a lightweight head with a novel bin adjustment design called metric bins module for each domain. During inference, each input image is automatically routed to the appropriate head using a latent classifier. Our framework admits multiple configurations depending on the datasets used for relative depth pre-training and metric fine-tuning. Without pre-training, we can already significantly improve the state of the art (SOTA) on the NYU Depth v2 indoor dataset. Pre-training on twelve datasets and fine-tuning on the NYU Depth v2 indoor dataset, we can further improve SOTA for a total of 21% in terms of relative absolute error (REL). Finally, ZoeD-M12-NK is the first model that can jointly train on multiple datasets (NYU Depth v2 and KITTI) without a significant drop in performance and achieve unprecedented zero-shot generalization performance to eight unseen datasets from both indoor and outdoor domains.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/zoedepth_architecture_bis.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> ZoeDepth architecture. Taken from the <a href=\"https://arxiv.org/abs/2302.12288\">original paper.</a> </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/isl-org/ZoeDepth).\n\n## Usage tips\n\n- ZoeDepth is an absolute (also called metric) depth estimation model, unlike DPT which is a relative depth estimation model. This means that ZoeDepth is able to estimate depth in metric units like meters."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 7b0f5836-ed6a-4ea1-860a-bbc7ad32336d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/zoedepth/", "title": "ZoeDepth", "tokens": 1267, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"7b0f5836-ed6a-4ea1-860a-bbc7ad32336d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/zoedepth/\", \"title\": \"ZoeDepth\", \"tokens\": 1267, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2cff7bf1-974b-4112-a580-43cbd47a147b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/zoedepth/\", \"title\": \"ZoeDepth\", \"tokens\": 1267, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"9a669a856c177f9e627d0397a7b1d305ab7ac019e35461418a9e5bca959f4b66\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0717aa44-58bc-4731-a4bc-b709baba8f5b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/zoedepth/\", \"title\": \"ZoeDepth\", \"tokens\": 1267, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"2b7b945d440e9eff7cdf83bda6d9d1748cdfad95442eaa771683f504ca2b8187\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"774d3826-d218-479e-bf3e-9b9b92d5e0f8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8a8a8d0a1620e5cab6b9d0fb8a5e4dbc071d483f8a97b8424f5332e689fe9ada\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1147, \"end_char_idx\": 4454, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2cff7bf1-974b-4112-a580-43cbd47a147b", "doc_id": "2cff7bf1-974b-4112-a580-43cbd47a147b", "ref_doc_id": "2cff7bf1-974b-4112-a580-43cbd47a147b", "chroma:document": "Our flagship model, ZoeD-M12-NK, is pre-trained on 12 datasets using relative depth and fine-tuned on two datasets using metric depth. We use a lightweight head with a novel bin adjustment design called metric bins module for each domain. During inference, each input image is automatically routed to the appropriate head using a latent classifier. Our framework admits multiple configurations depending on the datasets used for relative depth pre-training and metric fine-tuning. Without pre-training, we can already significantly improve the state of the art (SOTA) on the NYU Depth v2 indoor dataset. Pre-training on twelve datasets and fine-tuning on the NYU Depth v2 indoor dataset, we can further improve SOTA for a total of 21% in terms of relative absolute error (REL). Finally, ZoeD-M12-NK is the first model that can jointly train on multiple datasets (NYU Depth v2 and KITTI) without a significant drop in performance and achieve unprecedented zero-shot generalization performance to eight unseen datasets from both indoor and outdoor domains.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/zoedepth_architecture_bis.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> ZoeDepth architecture. Taken from the <a href=\"https://arxiv.org/abs/2302.12288\">original paper.</a> </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/isl-org/ZoeDepth).\n\n## Usage tips\n\n- ZoeDepth is an absolute (also called metric) depth estimation model, unlike DPT which is a relative depth estimation model. This means that ZoeDepth is able to estimate depth in metric units like meters.\n\nThe easiest to perform inference with ZoeDepth is by leveraging the [pipeline API](../main_classes/pipelines.md):\n\n```python\nfrom transformers import pipeline\nfrom PIL import Image\nimport requests\n\nurl = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nimage = Image.open(requests.get(url, stream=True).raw)\n\npipe = pipeline(task=\"depth-estimation\", model=\"Intel/zoedepth-nyu-kitti\")\nresult = pipe(image)\ndepth = result[\"depth\"]\n```\n\nAlternatively, one can also perform inference using the classes:\n\n```python\nfrom transformers import AutoImageProcessor, ZoeDepthForDepthEstimation\nimport torch\nimport numpy as np\nfrom PIL import Image\nimport requests\n\nurl = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nimage = Image.open(requests.get(url, stream=True).raw)\n\nimage_processor = AutoImageProcessor.from_pretrained(\"Intel/zoedepth-nyu-kitti\")\nmodel = ZoeDepthForDepthEstimation.from_pretrained(\"Intel/zoedepth-nyu-kitti\")\n\n# prepare image for the model\ninputs = image_processor(images=image, return_tensors=\"pt\")\n\nwith torch.no_grad():\n    outputs = model(**inputs)\n    predicted_depth = outputs.predicted_depth\n\n# interpolate to original size\nprediction = torch.nn.functional.interpolate(\n    predicted_depth.unsqueeze(1),\n    size=image.size[::-1],\n    mode=\"bicubic\",\n    align_corners=False,\n)\n\n# visualize the prediction\noutput = prediction.squeeze().cpu().numpy()\nformatted = (output * 255 / np.max(output)).astype(\"uint8\")\ndepth = Image.fromarray(formatted)\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with ZoeDepth."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 774d3826-d218-479e-bf3e-9b9b92d5e0f8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/zoedepth/", "title": "ZoeDepth", "tokens": 1267, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"774d3826-d218-479e-bf3e-9b9b92d5e0f8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/zoedepth/\", \"title\": \"ZoeDepth\", \"tokens\": 1267, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2cff7bf1-974b-4112-a580-43cbd47a147b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/zoedepth/\", \"title\": \"ZoeDepth\", \"tokens\": 1267, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"9a669a856c177f9e627d0397a7b1d305ab7ac019e35461418a9e5bca959f4b66\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7b0f5836-ed6a-4ea1-860a-bbc7ad32336d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/zoedepth/\", \"title\": \"ZoeDepth\", \"tokens\": 1267, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b8e6df1fb2e66b38cf4a456db8b2b19cab9ab81237fd37a5c0449ccb8e38bb57\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2841, \"end_char_idx\": 4824, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2cff7bf1-974b-4112-a580-43cbd47a147b", "doc_id": "2cff7bf1-974b-4112-a580-43cbd47a147b", "ref_doc_id": "2cff7bf1-974b-4112-a580-43cbd47a147b", "chroma:document": "The easiest to perform inference with ZoeDepth is by leveraging the [pipeline API](../main_classes/pipelines.md):\n\n```python\nfrom transformers import pipeline\nfrom PIL import Image\nimport requests\n\nurl = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nimage = Image.open(requests.get(url, stream=True).raw)\n\npipe = pipeline(task=\"depth-estimation\", model=\"Intel/zoedepth-nyu-kitti\")\nresult = pipe(image)\ndepth = result[\"depth\"]\n```\n\nAlternatively, one can also perform inference using the classes:\n\n```python\nfrom transformers import AutoImageProcessor, ZoeDepthForDepthEstimation\nimport torch\nimport numpy as np\nfrom PIL import Image\nimport requests\n\nurl = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nimage = Image.open(requests.get(url, stream=True).raw)\n\nimage_processor = AutoImageProcessor.from_pretrained(\"Intel/zoedepth-nyu-kitti\")\nmodel = ZoeDepthForDepthEstimation.from_pretrained(\"Intel/zoedepth-nyu-kitti\")\n\n# prepare image for the model\ninputs = image_processor(images=image, return_tensors=\"pt\")\n\nwith torch.no_grad():\n    outputs = model(**inputs)\n    predicted_depth = outputs.predicted_depth\n\n# interpolate to original size\nprediction = torch.nn.functional.interpolate(\n    predicted_depth.unsqueeze(1),\n    size=image.size[::-1],\n    mode=\"bicubic\",\n    align_corners=False,\n)\n\n# visualize the prediction\noutput = prediction.squeeze().cpu().numpy()\nformatted = (output * 255 / np.max(output)).astype(\"uint8\")\ndepth = Image.fromarray(formatted)\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with ZoeDepth.\n\n- A demo notebook regarding inference with ZoeDepth models can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/ZoeDepth). \ud83c\udf0e\n\n## ZoeDepthConfig\n\n[[autodoc]] ZoeDepthConfig\n\n## ZoeDepthImageProcessor\n\n[[autodoc]] ZoeDepthImageProcessor\n    - preprocess\n\n## ZoeDepthForDepthEstimation\n\n[[autodoc]] ZoeDepthForDepthEstimation\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: f81640ef-bc31-4b18-9018-29eff1106da1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlm/", "title": "LayoutLM", "tokens": 1747, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"f81640ef-bc31-4b18-9018-29eff1106da1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlm/\", \"title\": \"LayoutLM\", \"tokens\": 1747, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"68e22d16-a9b3-4fab-b09a-f422f4a19f60\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlm/\", \"title\": \"LayoutLM\", \"tokens\": 1747, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f74403395d60d2134d753a0f4a3ceef2949d8b2ef44dfe90917b552c89bdeaaf\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"aadc4286-2bf3-4a2f-b3f5-11d299ce261e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"23cfd99685570d19c96c88ce554adc2b21fc5d0abab896e08d10eb4d6cdd35d6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2823, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "68e22d16-a9b3-4fab-b09a-f422f4a19f60", "doc_id": "68e22d16-a9b3-4fab-b09a-f422f4a19f60", "ref_doc_id": "68e22d16-a9b3-4fab-b09a-f422f4a19f60", "chroma:document": "# LayoutLM\n\n<a id='Overview'></a>\n\n## Overview\n\nThe LayoutLM model was proposed in the paper [LayoutLM: Pre-training of Text and Layout for Document Image\nUnderstanding](https://arxiv.org/abs/1912.13318) by Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and\nMing Zhou. It's a simple but effective pretraining method of text and layout for document image understanding and\ninformation extraction tasks, such as form understanding and receipt understanding. It obtains state-of-the-art results\non several downstream tasks:\n\n- form understanding: the [FUNSD](https://guillaumejaume.github.io/FUNSD/) dataset (a collection of 199 annotated\n  forms comprising more than 30,000 words).\n- receipt understanding: the [SROIE](https://rrc.cvc.uab.es/?ch=13) dataset (a collection of 626 receipts for\n  training and 347 receipts for testing).\n- document image classification: the [RVL-CDIP](https://www.cs.cmu.edu/~aharley/rvl-cdip/) dataset (a collection of\n  400,000 images belonging to one of 16 classes).\n\nThe abstract from the paper is the following:\n\n*Pre-training techniques have been verified successfully in a variety of NLP tasks in recent years. Despite the\nwidespread use of pretraining models for NLP applications, they almost exclusively focus on text-level manipulation,\nwhile neglecting layout and style information that is vital for document image understanding. In this paper, we propose\nthe LayoutLM to jointly model interactions between text and layout information across scanned document images, which is\nbeneficial for a great number of real-world document image understanding tasks such as information extraction from\nscanned documents. Furthermore, we also leverage image features to incorporate words' visual information into LayoutLM.\nTo the best of our knowledge, this is the first time that text and layout are jointly learned in a single framework for\ndocument-level pretraining. It achieves new state-of-the-art results in several downstream tasks, including form\nunderstanding (from 70.72 to 79.27), receipt understanding (from 94.02 to 95.24) and document image classification\n(from 93.07 to 94.42).*\n\n## Usage tips\n\n- In addition to *input_ids*, [`~transformers.LayoutLMModel.forward`] also expects the input `bbox`, which are\n  the bounding boxes (i.e. 2D-positions) of the input tokens. These can be obtained using an external OCR engine such\n  as Google's [Tesseract](https://github.com/tesseract-ocr/tesseract) (there's a [Python wrapper](https://pypi.org/project/pytesseract/) available). Each bounding box should be in (x0, y0, x1, y1) format, where\n  (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the\n  position of the lower right corner. Note that one first needs to normalize the bounding boxes to be on a 0-1000\n  scale."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: aadc4286-2bf3-4a2f-b3f5-11d299ce261e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlm/", "title": "LayoutLM", "tokens": 1747, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"aadc4286-2bf3-4a2f-b3f5-11d299ce261e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlm/\", \"title\": \"LayoutLM\", \"tokens\": 1747, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"68e22d16-a9b3-4fab-b09a-f422f4a19f60\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlm/\", \"title\": \"LayoutLM\", \"tokens\": 1747, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f74403395d60d2134d753a0f4a3ceef2949d8b2ef44dfe90917b552c89bdeaaf\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f81640ef-bc31-4b18-9018-29eff1106da1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlm/\", \"title\": \"LayoutLM\", \"tokens\": 1747, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"92acb183c1972de08d4409d6f03541c4b4bea483a0552e068095df3ecdd379cc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0637c265-7899-4744-851f-a7407f814fd4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a3d8f9c18c0d0ecda730ba4098dd00f999f0471a8239e6b17aed453831835dfc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1156, \"end_char_idx\": 3859, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "68e22d16-a9b3-4fab-b09a-f422f4a19f60", "doc_id": "68e22d16-a9b3-4fab-b09a-f422f4a19f60", "ref_doc_id": "68e22d16-a9b3-4fab-b09a-f422f4a19f60", "chroma:document": "Despite the\nwidespread use of pretraining models for NLP applications, they almost exclusively focus on text-level manipulation,\nwhile neglecting layout and style information that is vital for document image understanding. In this paper, we propose\nthe LayoutLM to jointly model interactions between text and layout information across scanned document images, which is\nbeneficial for a great number of real-world document image understanding tasks such as information extraction from\nscanned documents. Furthermore, we also leverage image features to incorporate words' visual information into LayoutLM.\nTo the best of our knowledge, this is the first time that text and layout are jointly learned in a single framework for\ndocument-level pretraining. It achieves new state-of-the-art results in several downstream tasks, including form\nunderstanding (from 70.72 to 79.27), receipt understanding (from 94.02 to 95.24) and document image classification\n(from 93.07 to 94.42).*\n\n## Usage tips\n\n- In addition to *input_ids*, [`~transformers.LayoutLMModel.forward`] also expects the input `bbox`, which are\n  the bounding boxes (i.e. 2D-positions) of the input tokens. These can be obtained using an external OCR engine such\n  as Google's [Tesseract](https://github.com/tesseract-ocr/tesseract) (there's a [Python wrapper](https://pypi.org/project/pytesseract/) available). Each bounding box should be in (x0, y0, x1, y1) format, where\n  (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the\n  position of the lower right corner. Note that one first needs to normalize the bounding boxes to be on a 0-1000\n  scale. To normalize, you can use the following function:\n\n```python\ndef normalize_bbox(bbox, width, height):\n    return [\n        int(1000 * (bbox[0] / width)),\n        int(1000 * (bbox[1] / height)),\n        int(1000 * (bbox[2] / width)),\n        int(1000 * (bbox[3] / height)),\n    ]\n```\n\nHere, `width` and `height` correspond to the width and height of the original document in which the token\noccurs. Those can be obtained using the Python Image Library (PIL) library for example, as follows:\n\n```python\nfrom PIL import Image\n\n# Document can be a png, jpg, etc. PDFs must be converted to images.\nimage = Image.open(name_of_your_document).convert(\"RGB\")\n\nwidth, height = image.size\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with LayoutLM. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 0637c265-7899-4744-851f-a7407f814fd4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlm/", "title": "LayoutLM", "tokens": 1747, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"0637c265-7899-4744-851f-a7407f814fd4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlm/\", \"title\": \"LayoutLM\", \"tokens\": 1747, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"68e22d16-a9b3-4fab-b09a-f422f4a19f60\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlm/\", \"title\": \"LayoutLM\", \"tokens\": 1747, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f74403395d60d2134d753a0f4a3ceef2949d8b2ef44dfe90917b552c89bdeaaf\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"aadc4286-2bf3-4a2f-b3f5-11d299ce261e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlm/\", \"title\": \"LayoutLM\", \"tokens\": 1747, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"062437da17e8f91e3521994bf9f4447b44aeae1fd048a28d17a10f3406fafe75\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2321, \"end_char_idx\": 6448, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "68e22d16-a9b3-4fab-b09a-f422f4a19f60", "doc_id": "68e22d16-a9b3-4fab-b09a-f422f4a19f60", "ref_doc_id": "68e22d16-a9b3-4fab-b09a-f422f4a19f60", "chroma:document": "These can be obtained using an external OCR engine such\n  as Google's [Tesseract](https://github.com/tesseract-ocr/tesseract) (there's a [Python wrapper](https://pypi.org/project/pytesseract/) available). Each bounding box should be in (x0, y0, x1, y1) format, where\n  (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the\n  position of the lower right corner. Note that one first needs to normalize the bounding boxes to be on a 0-1000\n  scale. To normalize, you can use the following function:\n\n```python\ndef normalize_bbox(bbox, width, height):\n    return [\n        int(1000 * (bbox[0] / width)),\n        int(1000 * (bbox[1] / height)),\n        int(1000 * (bbox[2] / width)),\n        int(1000 * (bbox[3] / height)),\n    ]\n```\n\nHere, `width` and `height` correspond to the width and height of the original document in which the token\noccurs. Those can be obtained using the Python Image Library (PIL) library for example, as follows:\n\n```python\nfrom PIL import Image\n\n# Document can be a png, jpg, etc. PDFs must be converted to images.\nimage = Image.open(name_of_your_document).convert(\"RGB\")\n\nwidth, height = image.size\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with LayoutLM. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n\n<PipelineTag pipeline=\"document-question-answering\" />\n\n- A blog post on [fine-tuning\n  LayoutLM for document-understanding using Keras & Hugging Face\n  Transformers](https://www.philschmid.de/fine-tuning-layoutlm-keras).\n\n- A blog post on how to [fine-tune LayoutLM for document-understanding using only Hugging Face Transformers](https://www.philschmid.de/fine-tuning-layoutlm).\n\n- A notebook on how to [fine-tune LayoutLM on the FUNSD dataset with image embeddings](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLM/Add_image_embeddings_to_LayoutLM.ipynb).\n\n- See also: [Document question answering task guide](../tasks/document_question_answering)\n\n<PipelineTag pipeline=\"text-classification\" />\n\n- A notebook on how to [fine-tune LayoutLM for sequence classification on the RVL-CDIP dataset](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLM/Fine_tuning_LayoutLMForSequenceClassification_on_RVL_CDIP.ipynb).\n- [Text classification task guide](../tasks/sequence_classification)\n\n<PipelineTag pipeline=\"token-classification\" />\n\n- A notebook on how to [ fine-tune LayoutLM for token classification on the FUNSD dataset](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/LayoutLM/Fine_tuning_LayoutLMForTokenClassification_on_FUNSD.ipynb).\n- [Token classification task guide](../tasks/token_classification)\n\n**Other resources**\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n\n\ud83d\ude80 Deploy\n\n- A blog post on how to [Deploy LayoutLM with Hugging Face Inference Endpoints](https://www.philschmid.de/inference-endpoints-layoutlm).\n\n## LayoutLMConfig\n\n[[autodoc]] LayoutLMConfig\n\n## LayoutLMTokenizer\n\n[[autodoc]] LayoutLMTokenizer\n\n## LayoutLMTokenizerFast\n\n[[autodoc]] LayoutLMTokenizerFast\n\n<frameworkcontent>\n<pt>\n\n## LayoutLMModel\n\n[[autodoc]] LayoutLMModel\n\n## LayoutLMForMaskedLM\n\n[[autodoc]] LayoutLMForMaskedLM\n\n## LayoutLMForSequenceClassification\n\n[[autodoc]] LayoutLMForSequenceClassification\n\n## LayoutLMForTokenClassification\n\n[[autodoc]] LayoutLMForTokenClassification\n\n## LayoutLMForQuestionAnswering\n\n[[autodoc]] LayoutLMForQuestionAnswering\n\n</pt>\n<tf>\n\n## TFLayoutLMModel\n\n[[autodoc]] TFLayoutLMModel\n\n## TFLayoutLMForMaskedLM\n\n[[autodoc]] TFLayoutLMForMaskedLM\n\n## TFLayoutLMForSequenceClassification\n\n[[autodoc]] TFLayoutLMForSequenceClassification\n\n## TFLayoutLMForTokenClassification\n\n[[autodoc]] TFLayoutLMForTokenClassification\n\n## TFLayoutLMForQuestionAnswering\n\n[[autodoc]] TFLayoutLMForQuestionAnswering\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: e842e107-d624-4cd6-ae4e-c8c30f370f0f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/table-transformer/", "title": "Table Transformer", "tokens": 839, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"e842e107-d624-4cd6-ae4e-c8c30f370f0f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/table-transformer/\", \"title\": \"Table Transformer\", \"tokens\": 839, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"39109427-0540-4b7b-b504-975c493b17cd\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/table-transformer/\", \"title\": \"Table Transformer\", \"tokens\": 839, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"94b36822bd9d66c2664fe8a660ea2b6b94c23d34a94fd88f08e17c378a29f78a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3322, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "39109427-0540-4b7b-b504-975c493b17cd", "doc_id": "39109427-0540-4b7b-b504-975c493b17cd", "ref_doc_id": "39109427-0540-4b7b-b504-975c493b17cd", "chroma:document": "# Table Transformer\n\n## Overview\n\nThe Table Transformer model was proposed in [PubTables-1M: Towards comprehensive table extraction from unstructured documents](https://arxiv.org/abs/2110.00061) by\nBrandon Smock, Rohith Pesala, Robin Abraham. The authors introduce a new dataset, PubTables-1M, to benchmark progress in table extraction from unstructured documents,\nas well as table structure recognition and functional analysis. The authors train 2 [DETR](detr) models, one for table detection and one for table structure recognition, dubbed Table Transformers.\n\nThe abstract from the paper is the following:\n\n*Recently, significant progress has been made applying machine learning to the problem of table structure inference and extraction from unstructured documents.\nHowever, one of the greatest challenges remains the creation of datasets with complete, unambiguous ground truth at scale. To address this, we develop a new, more\ncomprehensive dataset for table extraction, called PubTables-1M. PubTables-1M contains nearly one million tables from scientific articles, supports multiple input\nmodalities, and contains detailed header and location information for table structures, making it useful for a wide variety of modeling approaches. It also addresses a significant\nsource of ground truth inconsistency observed in prior datasets called oversegmentation, using a novel canonicalization procedure. We demonstrate that these improvements lead to a\nsignificant increase in training performance and a more reliable estimate of model performance at evaluation for table structure recognition. Further, we show that transformer-based\nobject detection models trained on PubTables-1M produce excellent results for all three tasks of detection, structure recognition, and functional analysis without the need for any\nspecial customization for these tasks.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/table_transformer_architecture.jpeg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Table detection and table structure recognition clarified. Taken from the <a href=\"https://arxiv.org/abs/2110.00061\">original paper</a>. </small>\n\nThe authors released 2 models, one for [table detection](https://huggingface.co/microsoft/table-transformer-detection) in \ndocuments, one for [table structure recognition](https://huggingface.co/microsoft/table-transformer-structure-recognition) \n(the task of recognizing the individual rows, columns etc. in a table).\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be\nfound [here](https://github.com/microsoft/table-transformer).\n\n## Resources\n\n<PipelineTag pipeline=\"object-detection\"/>\n\n- A demo notebook for the Table Transformer can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Table%20Transformer).\n- It turns out padding of images is quite important for detection. An interesting Github thread with replies from the authors can be found [here](https://github.com/microsoft/table-transformer/issues/68).\n\n## TableTransformerConfig\n\n[[autodoc]] TableTransformerConfig\n\n## TableTransformerModel\n\n[[autodoc]] TableTransformerModel\n    - forward\n\n## TableTransformerForObjectDetection\n\n[[autodoc]] TableTransformerForObjectDetection\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: fb12c947-f701-4c9a-bc96-398155fba898
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/lilt/", "title": "LiLT", "tokens": 1084, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"fb12c947-f701-4c9a-bc96-398155fba898\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/lilt/\", \"title\": \"LiLT\", \"tokens\": 1084, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ffb089a4-974a-4963-ab3a-51ea9db018df\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/lilt/\", \"title\": \"LiLT\", \"tokens\": 1084, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"955035da74407edb70bb93665a563f627331283001afe87825ec70a8026d36cc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"805a4db8-9a73-4bea-89a7-768d87dfcf5e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"65a7dca4b06655e142721d57bda860399cb9caa0f2ec0ed005b35822edc88417\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3229, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ffb089a4-974a-4963-ab3a-51ea9db018df", "doc_id": "ffb089a4-974a-4963-ab3a-51ea9db018df", "ref_doc_id": "ffb089a4-974a-4963-ab3a-51ea9db018df", "chroma:document": "# LiLT\n\n## Overview\n\nThe LiLT model was proposed in [LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understanding](https://arxiv.org/abs/2202.13669) by Jiapeng Wang, Lianwen Jin, Kai Ding.\nLiLT allows to combine any pre-trained RoBERTa text encoder with a lightweight Layout Transformer, to enable [LayoutLM](layoutlm)-like document understanding for many\nlanguages.\n\nThe abstract from the paper is the following:\n\n*Structured document understanding has attracted considerable attention and made significant progress recently, owing to its crucial role in intelligent document processing. However, most existing related models can only deal with the document data of specific language(s) (typically English) included in the pre-training collection, which is extremely limited. To address this issue, we propose a simple yet effective Language-independent Layout Transformer (LiLT) for structured document understanding. LiLT can be pre-trained on the structured documents of a single language and then directly fine-tuned on other languages with the corresponding off-the-shelf monolingual/multilingual pre-trained textual models. Experimental results on eight languages have shown that LiLT can achieve competitive or even superior performance on diverse widely-used downstream benchmarks, which enables language-independent benefit from the pre-training of document layout structure.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/lilt_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> LiLT architecture. Taken from the <a href=\"https://arxiv.org/abs/2202.13669\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/jpwang/lilt).\n\n## Usage tips\n\n- To combine the Language-Independent Layout Transformer with a new RoBERTa checkpoint from the [hub](https://huggingface.co/models?search=roberta), refer to [this guide](https://github.com/jpWang/LiLT#or-generate-your-own-checkpoint-optional).\nThe script will result in `config.json` and `pytorch_model.bin` files being stored locally. After doing this, one can do the following (assuming you're logged in with your HuggingFace account):\n\n```python\nfrom transformers import LiltModel\n\nmodel = LiltModel.from_pretrained(\"path_to_your_files\")\nmodel.push_to_hub(\"name_of_repo_on_the_hub\")\n```\n\n- When preparing data for the model, make sure to use the token vocabulary that corresponds to the RoBERTa checkpoint you combined with the Layout Transformer.\n- As [lilt-roberta-en-base](https://huggingface.co/SCUT-DLVCLab/lilt-roberta-en-base) uses the same vocabulary as [LayoutLMv3](layoutlmv3), one can use [`LayoutLMv3TokenizerFast`] to prepare data for the model.\nThe same is true for [lilt-roberta-en-base](https://huggingface.co/SCUT-DLVCLab/lilt-infoxlm-base): one can use [`LayoutXLMTokenizerFast`] for that model.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with LiLT.\n\n- Demo notebooks for LiLT can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/LiLT)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 805a4db8-9a73-4bea-89a7-768d87dfcf5e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/lilt/", "title": "LiLT", "tokens": 1084, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"805a4db8-9a73-4bea-89a7-768d87dfcf5e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/lilt/\", \"title\": \"LiLT\", \"tokens\": 1084, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ffb089a4-974a-4963-ab3a-51ea9db018df\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/lilt/\", \"title\": \"LiLT\", \"tokens\": 1084, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"955035da74407edb70bb93665a563f627331283001afe87825ec70a8026d36cc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"fb12c947-f701-4c9a-bc96-398155fba898\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/lilt/\", \"title\": \"LiLT\", \"tokens\": 1084, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"ef50d05fa6261009793813412b71c1f6a8d47d9ab104da2cf4f9755e4b0b5107\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1783, \"end_char_idx\": 4032, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ffb089a4-974a-4963-ab3a-51ea9db018df", "doc_id": "ffb089a4-974a-4963-ab3a-51ea9db018df", "ref_doc_id": "ffb089a4-974a-4963-ab3a-51ea9db018df", "chroma:document": "The original code can be found [here](https://github.com/jpwang/lilt).\n\n## Usage tips\n\n- To combine the Language-Independent Layout Transformer with a new RoBERTa checkpoint from the [hub](https://huggingface.co/models?search=roberta), refer to [this guide](https://github.com/jpWang/LiLT#or-generate-your-own-checkpoint-optional).\nThe script will result in `config.json` and `pytorch_model.bin` files being stored locally. After doing this, one can do the following (assuming you're logged in with your HuggingFace account):\n\n```python\nfrom transformers import LiltModel\n\nmodel = LiltModel.from_pretrained(\"path_to_your_files\")\nmodel.push_to_hub(\"name_of_repo_on_the_hub\")\n```\n\n- When preparing data for the model, make sure to use the token vocabulary that corresponds to the RoBERTa checkpoint you combined with the Layout Transformer.\n- As [lilt-roberta-en-base](https://huggingface.co/SCUT-DLVCLab/lilt-roberta-en-base) uses the same vocabulary as [LayoutLMv3](layoutlmv3), one can use [`LayoutLMv3TokenizerFast`] to prepare data for the model.\nThe same is true for [lilt-roberta-en-base](https://huggingface.co/SCUT-DLVCLab/lilt-infoxlm-base): one can use [`LayoutXLMTokenizerFast`] for that model.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with LiLT.\n\n- Demo notebooks for LiLT can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/LiLT).\n\n**Documentation resources**\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## LiltConfig\n\n[[autodoc]] LiltConfig\n\n## LiltModel\n\n[[autodoc]] LiltModel\n    - forward\n\n## LiltForSequenceClassification\n\n[[autodoc]] LiltForSequenceClassification\n    - forward\n\n## LiltForTokenClassification\n\n[[autodoc]] LiltForTokenClassification\n    - forward\n\n## LiltForQuestionAnswering\n\n[[autodoc]] LiltForQuestionAnswering\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: dae3e06f-035d-4ee3-b248-26ce542d1c73
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/m2m_100/", "title": "M2M100", "tokens": 1860, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"dae3e06f-035d-4ee3-b248-26ce542d1c73\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/m2m_100/\", \"title\": \"M2M100\", \"tokens\": 1860, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"eab38459-be22-4c3c-88f9-cff7d4c9a552\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/m2m_100/\", \"title\": \"M2M100\", \"tokens\": 1860, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"dc059b2533fb4d78c12696464875c9caecff885e2c8c357b4f36a5b7d9f7565d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2beb0da6-905e-4335-99bf-4681bc391b69\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f2ed00b16198c0e1c433a3b2aceee0fd3c152faa917a9aeb417e6de4d34427ad\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3162, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "eab38459-be22-4c3c-88f9-cff7d4c9a552", "doc_id": "eab38459-be22-4c3c-88f9-cff7d4c9a552", "ref_doc_id": "eab38459-be22-4c3c-88f9-cff7d4c9a552", "chroma:document": "# M2M100\n\n## Overview\n\nThe M2M100 model was proposed in [Beyond English-Centric Multilingual Machine Translation](https://arxiv.org/abs/2010.11125) by Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky,\nSiddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy\nLiptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, Armand Joulin.\n\nThe abstract from the paper is the following:\n\n*Existing work in translation demonstrated the potential of massively multilingual machine translation by training a\nsingle model able to translate between any pair of languages. However, much of this work is English-Centric by training\nonly on data which was translated from or to English. While this is supported by large sources of training data, it\ndoes not reflect translation needs worldwide. In this work, we create a true Many-to-Many multilingual translation\nmodel that can translate directly between any pair of 100 languages. We build and open source a training dataset that\ncovers thousands of language directions with supervised data, created through large-scale mining. Then, we explore how\nto effectively increase model capacity through a combination of dense scaling and language-specific sparse parameters\nto create high quality models. Our focus on non-English-Centric models brings gains of more than 10 BLEU when directly\ntranslating between non-English directions while performing competitively to the best single systems of WMT. We\nopen-source our scripts so that others may reproduce the data, evaluation, and final M2M-100 model.*\n\nThis model was contributed by [valhalla](https://huggingface.co/valhalla).\n\n\n## Usage tips and examples\n\nM2M100 is a multilingual encoder-decoder (seq-to-seq) model primarily intended for translation tasks. As the model is\nmultilingual it expects the sequences in a certain format: A special language id token is used as prefix in both the\nsource and target text. The source text format is `[lang_code] X [eos]`, where `lang_code` is source language\nid for source text and target language id for target text, with `X` being the source or target text.\n\nThe [`M2M100Tokenizer`] depends on `sentencepiece` so be sure to install it before running the\nexamples. To install `sentencepiece` run `pip install sentencepiece`.\n\n**Supervised Training**\n\n```python\nfrom transformers import M2M100Config, M2M100ForConditionalGeneration, M2M100Tokenizer\n\nmodel = M2M100ForConditionalGeneration.from_pretrained(\"facebook/m2m100_418M\")\ntokenizer = M2M100Tokenizer.from_pretrained(\"facebook/m2m100_418M\", src_lang=\"en\", tgt_lang=\"fr\")\n\nsrc_text = \"Life is like a box of chocolates.\"\ntgt_text = \"La vie est comme une bo\u00eete de chocolat.\"\n\nmodel_inputs = tokenizer(src_text, text_target=tgt_text, return_tensors=\"pt\")\n\nloss = model(**model_inputs).loss  # forward pass\n```\n\n**Generation**\n\nM2M100 uses the `eos_token_id` as the `decoder_start_token_id` for generation with the target language id \nbeing forced as the first generated token. To force the target language id as the first generated token, pass the \n*forced_bos_token_id* parameter to the *generate* method."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 2beb0da6-905e-4335-99bf-4681bc391b69
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/m2m_100/", "title": "M2M100", "tokens": 1860, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"2beb0da6-905e-4335-99bf-4681bc391b69\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/m2m_100/\", \"title\": \"M2M100\", \"tokens\": 1860, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"eab38459-be22-4c3c-88f9-cff7d4c9a552\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/m2m_100/\", \"title\": \"M2M100\", \"tokens\": 1860, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"dc059b2533fb4d78c12696464875c9caecff885e2c8c357b4f36a5b7d9f7565d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dae3e06f-035d-4ee3-b248-26ce542d1c73\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/m2m_100/\", \"title\": \"M2M100\", \"tokens\": 1860, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"15e33478638ffb83c3305a6492aa934f181bf03f7661637bd126b14ba85569a0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7bcc5bde-6e75-4fcd-b529-b17957268da5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4d1d84674c6d4a8e0a3c44dd72b321d370a79a8de96ad0844cbdd1f9a2422100\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1690, \"end_char_idx\": 4294, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "eab38459-be22-4c3c-88f9-cff7d4c9a552", "doc_id": "eab38459-be22-4c3c-88f9-cff7d4c9a552", "ref_doc_id": "eab38459-be22-4c3c-88f9-cff7d4c9a552", "chroma:document": "## Usage tips and examples\n\nM2M100 is a multilingual encoder-decoder (seq-to-seq) model primarily intended for translation tasks. As the model is\nmultilingual it expects the sequences in a certain format: A special language id token is used as prefix in both the\nsource and target text. The source text format is `[lang_code] X [eos]`, where `lang_code` is source language\nid for source text and target language id for target text, with `X` being the source or target text.\n\nThe [`M2M100Tokenizer`] depends on `sentencepiece` so be sure to install it before running the\nexamples. To install `sentencepiece` run `pip install sentencepiece`.\n\n**Supervised Training**\n\n```python\nfrom transformers import M2M100Config, M2M100ForConditionalGeneration, M2M100Tokenizer\n\nmodel = M2M100ForConditionalGeneration.from_pretrained(\"facebook/m2m100_418M\")\ntokenizer = M2M100Tokenizer.from_pretrained(\"facebook/m2m100_418M\", src_lang=\"en\", tgt_lang=\"fr\")\n\nsrc_text = \"Life is like a box of chocolates.\"\ntgt_text = \"La vie est comme une bo\u00eete de chocolat.\"\n\nmodel_inputs = tokenizer(src_text, text_target=tgt_text, return_tensors=\"pt\")\n\nloss = model(**model_inputs).loss  # forward pass\n```\n\n**Generation**\n\nM2M100 uses the `eos_token_id` as the `decoder_start_token_id` for generation with the target language id \nbeing forced as the first generated token. To force the target language id as the first generated token, pass the \n*forced_bos_token_id* parameter to the *generate* method. The following example shows how to translate between \nHindi to French and Chinese to English using the *facebook/m2m100_418M* checkpoint.\n\n```python\n>>> from transformers import M2M100ForConditionalGeneration, M2M100Tokenizer\n\n>>> hi_text = \"\u091c\u0940\u0935\u0928 \u090f\u0915 \u091a\u0949\u0915\u0932\u0947\u091f \u092c\u0949\u0915\u094d\u0938 \u0915\u0940 \u0924\u0930\u0939 \u0939\u0948\u0964\"\n>>> chinese_text = \"\u751f\u6d3b\u5c31\u50cf\u4e00\u76d2\u5de7\u514b\u529b\u3002\"\n\n>>> model = M2M100ForConditionalGeneration.from_pretrained(\"facebook/m2m100_418M\")\n>>> tokenizer = M2M100Tokenizer.from_pretrained(\"facebook/m2m100_418M\")\n\n>>> # translate Hindi to French\n>>> tokenizer.src_lang = \"hi\"\n>>> encoded_hi = tokenizer(hi_text, return_tensors=\"pt\")\n>>> generated_tokens = model.generate(**encoded_hi, forced_bos_token_id=tokenizer.get_lang_id(\"fr\"))\n>>> tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)\n\"La vie est comme une bo\u00eete de chocolat.\"\n\n>>> # translate Chinese to English\n>>> tokenizer.src_lang = \"zh\"\n>>> encoded_zh = tokenizer(chinese_text, return_tensors=\"pt\")\n>>> generated_tokens = model.generate(**encoded_zh, forced_bos_token_id=tokenizer.get_lang_id(\"en\"))\n>>> tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)\n\"Life is like a box of chocolate.\""}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 7bcc5bde-6e75-4fcd-b529-b17957268da5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/m2m_100/", "title": "M2M100", "tokens": 1860, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"7bcc5bde-6e75-4fcd-b529-b17957268da5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/m2m_100/\", \"title\": \"M2M100\", \"tokens\": 1860, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"eab38459-be22-4c3c-88f9-cff7d4c9a552\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/m2m_100/\", \"title\": \"M2M100\", \"tokens\": 1860, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"dc059b2533fb4d78c12696464875c9caecff885e2c8c357b4f36a5b7d9f7565d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2beb0da6-905e-4335-99bf-4681bc391b69\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/m2m_100/\", \"title\": \"M2M100\", \"tokens\": 1860, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0f8e4c930f7cf31a96dd548315c9481927fdf1200c2ab37ee35343a977205d1d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fbf0a780-4a42-4462-8770-e6354cb7526e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e06ce49ff041b581891a189a4ed13b464ed7b51df174ef64942a966d09a09b3b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3033, \"end_char_idx\": 5700, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "eab38459-be22-4c3c-88f9-cff7d4c9a552", "doc_id": "eab38459-be22-4c3c-88f9-cff7d4c9a552", "ref_doc_id": "eab38459-be22-4c3c-88f9-cff7d4c9a552", "chroma:document": "To force the target language id as the first generated token, pass the \n*forced_bos_token_id* parameter to the *generate* method. The following example shows how to translate between \nHindi to French and Chinese to English using the *facebook/m2m100_418M* checkpoint.\n\n```python\n>>> from transformers import M2M100ForConditionalGeneration, M2M100Tokenizer\n\n>>> hi_text = \"\u091c\u0940\u0935\u0928 \u090f\u0915 \u091a\u0949\u0915\u0932\u0947\u091f \u092c\u0949\u0915\u094d\u0938 \u0915\u0940 \u0924\u0930\u0939 \u0939\u0948\u0964\"\n>>> chinese_text = \"\u751f\u6d3b\u5c31\u50cf\u4e00\u76d2\u5de7\u514b\u529b\u3002\"\n\n>>> model = M2M100ForConditionalGeneration.from_pretrained(\"facebook/m2m100_418M\")\n>>> tokenizer = M2M100Tokenizer.from_pretrained(\"facebook/m2m100_418M\")\n\n>>> # translate Hindi to French\n>>> tokenizer.src_lang = \"hi\"\n>>> encoded_hi = tokenizer(hi_text, return_tensors=\"pt\")\n>>> generated_tokens = model.generate(**encoded_hi, forced_bos_token_id=tokenizer.get_lang_id(\"fr\"))\n>>> tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)\n\"La vie est comme une bo\u00eete de chocolat.\"\n\n>>> # translate Chinese to English\n>>> tokenizer.src_lang = \"zh\"\n>>> encoded_zh = tokenizer(chinese_text, return_tensors=\"pt\")\n>>> generated_tokens = model.generate(**encoded_zh, forced_bos_token_id=tokenizer.get_lang_id(\"en\"))\n>>> tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)\n\"Life is like a box of chocolate.\"\n```\n\n## Resources\n\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n## M2M100Config\n\n[[autodoc]] M2M100Config\n\n## M2M100Tokenizer\n\n[[autodoc]] M2M100Tokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## M2M100Model\n\n[[autodoc]] M2M100Model\n    - forward\n\n## M2M100ForConditionalGeneration\n\n[[autodoc]] M2M100ForConditionalGeneration\n    - forward\n\n## Using Flash Attention 2\n\nFlash Attention 2 is a faster, optimized version of the attention scores computation which relies on `cuda` kernels.\n\n### Installation \n\nFirst, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features).\n\nNext, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\n### Usage\n\nTo load a model using Flash Attention 2, we can pass the argument `attn_implementation=\"flash_attention_2\"` to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). You can use either `torch.float16` or `torch.bfloat16` precision."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: fbf0a780-4a42-4462-8770-e6354cb7526e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/m2m_100/", "title": "M2M100", "tokens": 1860, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"fbf0a780-4a42-4462-8770-e6354cb7526e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/m2m_100/\", \"title\": \"M2M100\", \"tokens\": 1860, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"eab38459-be22-4c3c-88f9-cff7d4c9a552\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/m2m_100/\", \"title\": \"M2M100\", \"tokens\": 1860, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"dc059b2533fb4d78c12696464875c9caecff885e2c8c357b4f36a5b7d9f7565d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7bcc5bde-6e75-4fcd-b529-b17957268da5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/m2m_100/\", \"title\": \"M2M100\", \"tokens\": 1860, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8890f18a71fceb647aba02ed57b9719d104f1aafb427bba565f47e207a1cf63f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4295, \"end_char_idx\": 6748, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "eab38459-be22-4c3c-88f9-cff7d4c9a552", "doc_id": "eab38459-be22-4c3c-88f9-cff7d4c9a552", "ref_doc_id": "eab38459-be22-4c3c-88f9-cff7d4c9a552", "chroma:document": "```\n\n## Resources\n\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n## M2M100Config\n\n[[autodoc]] M2M100Config\n\n## M2M100Tokenizer\n\n[[autodoc]] M2M100Tokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## M2M100Model\n\n[[autodoc]] M2M100Model\n    - forward\n\n## M2M100ForConditionalGeneration\n\n[[autodoc]] M2M100ForConditionalGeneration\n    - forward\n\n## Using Flash Attention 2\n\nFlash Attention 2 is a faster, optimized version of the attention scores computation which relies on `cuda` kernels.\n\n### Installation \n\nFirst, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features).\n\nNext, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\n### Usage\n\nTo load a model using Flash Attention 2, we can pass the argument `attn_implementation=\"flash_attention_2\"` to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). You can use either `torch.float16` or `torch.bfloat16` precision.\n\n```python\n>>> import torch\n>>> from transformers import M2M100ForConditionalGeneration, M2M100Tokenizer\n\n>>> model = M2M100ForConditionalGeneration.from_pretrained(\"facebook/m2m100_418M\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\").to(\"cuda\").eval()\n>>> tokenizer = M2M100Tokenizer.from_pretrained(\"facebook/m2m100_418M\")\n\n>>> # translate Hindi to French\n>>> hi_text = \"\u091c\u0940\u0935\u0928 \u090f\u0915 \u091a\u0949\u0915\u0932\u0947\u091f \u092c\u0949\u0915\u094d\u0938 \u0915\u0940 \u0924\u0930\u0939 \u0939\u0948\u0964\"\n>>> tokenizer.src_lang = \"hi\"\n>>> encoded_hi = tokenizer(hi_text, return_tensors=\"pt\").to(\"cuda\")\n>>> generated_tokens = model.generate(**encoded_hi, forced_bos_token_id=tokenizer.get_lang_id(\"fr\"))\n>>> tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)\n\"La vie est comme une bo\u00eete de chocolat.\"\n```\n\n### Expected speedups\n\nBelow is an expected speedup diagram that compares pure inference time between the native implementation and the Flash Attention 2.\n\n<div style=\"text-align: center\">\n<img src=\"https://huggingface.co/datasets/visheratin/documentation-images/resolve/main/nllb-speedup.webp\">\n</div>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: e79b4879-58c7-4a13-9370-0cc46c252453
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/instructblipvideo/", "title": "InstructBlipVideo", "tokens": 901, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"e79b4879-58c7-4a13-9370-0cc46c252453\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/instructblipvideo/\", \"title\": \"InstructBlipVideo\", \"tokens\": 901, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a61de56e-c670-4eb7-a23e-ec31fc3b5d71\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/instructblipvideo/\", \"title\": \"InstructBlipVideo\", \"tokens\": 901, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"7d4bd3747ed458479ea86843f6fec8a430bab829f285e0e554bb972de1ca2356\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3345, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a61de56e-c670-4eb7-a23e-ec31fc3b5d71", "doc_id": "a61de56e-c670-4eb7-a23e-ec31fc3b5d71", "ref_doc_id": "a61de56e-c670-4eb7-a23e-ec31fc3b5d71", "chroma:document": "# InstructBlipVideo\n\n## Overview\n\n## Overview\n\nThe InstructBLIPVideo is an extension of the models proposed in [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://arxiv.org/abs/2305.06500) by Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, Steven Hoi.\nInstructBLIPVideo uses the same architecture as [InstructBLIP](instructblip) and works with the same checkpoints as [InstructBLIP](instructblip). The only difference is the ability to process videos.\n\nThe abstract from the paper is the following:\n\n*General-purpose language models that can solve various language-domain tasks have emerged driven by the pre-training and instruction-tuning pipeline. However, building general-purpose vision-language models is challenging due to the increased task discrepancy introduced by the additional visual input. Although vision-language pre-training has been widely studied, vision-language instruction tuning remains relatively less explored. In this paper, we conduct a systematic and comprehensive study on vision-language instruction tuning based on the pre-trained BLIP-2 models. We gather a wide variety of 26 publicly available datasets, transform them into instruction tuning format and categorize them into two clusters for held-in instruction tuning and held-out zero-shot evaluation. Additionally, we introduce instruction-aware visual feature extraction, a crucial method that enables the model to extract informative features tailored to the given instruction. The resulting InstructBLIP models achieve state-of-the-art zero-shot performance across all 13 held-out datasets, substantially outperforming BLIP-2 and the larger Flamingo. Our models also lead to state-of-the-art performance when finetuned on individual downstream tasks (e.g., 90.7% accuracy on ScienceQA IMG). Furthermore, we qualitatively demonstrate the advantages of InstructBLIP over concurrent multimodal models.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/instructblip_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> InstructBLIPVideo architecture. Taken from the <a href=\"https://arxiv.org/abs/2305.06500\">original paper.</a> </small>\n\nThis model was contributed by [RaushanTurganbay](https://huggingface.co/RaushanTurganbay).\nThe original code can be found [here](https://github.com/salesforce/LAVIS/tree/main/projects/instructblip).\n\n## Usage tips\n\n- The model was trained by sampling 4 frames per video, so it's recommended to sample 4 frames\n\n## InstructBlipVideoConfig\n\n[[autodoc]] InstructBlipVideoConfig\n    - from_vision_qformer_text_configs\n\n## InstructBlipVideoVisionConfig\n\n[[autodoc]] InstructBlipVideoVisionConfig\n\n## InstructBlipVideoQFormerConfig\n\n[[autodoc]] InstructBlipVideoQFormerConfig\n\n## InstructBlipVideoProcessor\n\n[[autodoc]] InstructBlipVideoProcessor\n\n## InstructBlipVideoImageProcessor\n\n[[autodoc]] InstructBlipVideoImageProcessor\n    - preprocess\n\n## InstructBlipVideoVisionModel\n\n[[autodoc]] InstructBlipVideoVisionModel\n    - forward\n\n## InstructBlipVideoQFormerModel\n\n[[autodoc]] InstructBlipVideoQFormerModel\n    - forward\n\n## InstructBlipVideoForConditionalGeneration\n\n[[autodoc]] InstructBlipVideoForConditionalGeneration\n    - forward\n    - generate"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: cff6783f-e970-4522-b28c-2c72e4ddc181
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/owlv2/", "title": "OWLv2", "tokens": 1744, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"cff6783f-e970-4522-b28c-2c72e4ddc181\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/owlv2/\", \"title\": \"OWLv2\", \"tokens\": 1744, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c4a71f9d-2a40-4e81-9581-f6b2589d53c9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/owlv2/\", \"title\": \"OWLv2\", \"tokens\": 1744, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f86ebc0ad8323d2222c1e2648a297da824bd8ca654568bb2d0c012639908b707\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"50591176-9971-4c09-8d10-a7b88aeb6281\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c6faede5bac43651ab761145da80d9b11ae36c3346ba3b2d62900780f13b5e2c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3367, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c4a71f9d-2a40-4e81-9581-f6b2589d53c9", "doc_id": "c4a71f9d-2a40-4e81-9581-f6b2589d53c9", "ref_doc_id": "c4a71f9d-2a40-4e81-9581-f6b2589d53c9", "chroma:document": "# OWLv2\n\n## Overview\n\nOWLv2 was proposed in [Scaling Open-Vocabulary Object Detection](https://arxiv.org/abs/2306.09683) by Matthias Minderer, Alexey Gritsenko, Neil Houlsby. OWLv2 scales up [OWL-ViT](owlvit) using self-training, which uses an existing detector to generate pseudo-box annotations on image-text pairs. This results in large gains over the previous state-of-the-art for zero-shot object detection.\n\nThe abstract from the paper is the following:\n\n*Open-vocabulary object detection has benefited greatly from pretrained vision-language models, but is still limited by the amount of available detection training data. While detection training data can be expanded by using Web image-text pairs as weak supervision, this has not been done at scales comparable to image-level pretraining. Here, we scale up detection data with self-training, which uses an existing detector to generate pseudo-box annotations on image-text pairs. Major challenges in scaling self-training are the choice of label space, pseudo-annotation filtering, and training efficiency. We present the OWLv2 model and OWL-ST self-training recipe, which address these challenges. OWLv2 surpasses the performance of previous state-of-the-art open-vocabulary detectors already at comparable training scales (~10M examples). However, with OWL-ST, we can scale to over 1B examples, yielding further large improvement: With an L/14 architecture, OWL-ST improves AP on LVIS rare classes, for which the model has seen no human box annotations, from 31.2% to 44.6% (43% relative improvement). OWL-ST unlocks Web-scale training for open-world localization, similar to what has been seen for image classification and language modelling.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/owlv2_overview.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> OWLv2 high-level overview. Taken from the <a href=\"https://arxiv.org/abs/2306.09683\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit).\n\n## Usage example\n\nOWLv2 is, just like its predecessor [OWL-ViT](owlvit), a zero-shot text-conditioned object detection model. OWL-ViT uses [CLIP](clip) as its multi-modal backbone, with a ViT-like Transformer to get visual features and a causal language model to get the text features. To use CLIP for detection, OWL-ViT removes the final token pooling layer of the vision model and attaches a lightweight classification and box head to each transformer output token. Open-vocabulary classification is enabled by replacing the fixed classification layer weights with the class-name embeddings obtained from the text model. The authors first train CLIP from scratch and fine-tune it end-to-end with the classification and box heads on standard detection datasets using a bipartite matching loss. One or multiple text queries per image can be used to perform zero-shot text-conditioned object detection.\n\n[`Owlv2ImageProcessor`] can be used to resize (or rescale) and normalize images for the model and [`CLIPTokenizer`] is used to encode the text. [`Owlv2Processor`] wraps [`Owlv2ImageProcessor`] and [`CLIPTokenizer`] into a single instance to both encode the text and prepare the images."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 50591176-9971-4c09-8d10-a7b88aeb6281
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/owlv2/", "title": "OWLv2", "tokens": 1744, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"50591176-9971-4c09-8d10-a7b88aeb6281\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/owlv2/\", \"title\": \"OWLv2\", \"tokens\": 1744, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c4a71f9d-2a40-4e81-9581-f6b2589d53c9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/owlv2/\", \"title\": \"OWLv2\", \"tokens\": 1744, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f86ebc0ad8323d2222c1e2648a297da824bd8ca654568bb2d0c012639908b707\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cff6783f-e970-4522-b28c-2c72e4ddc181\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/owlv2/\", \"title\": \"OWLv2\", \"tokens\": 1744, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"80c2fb60aa8945ec16df89b52acef75b94e3bf616a91b85f31e7ed4f4771499a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"455d1950-f421-4a54-be2a-c9b557b5c74a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6260cc23f840e5d00db96a8448f3ff1d99e9f2dbd99311442ed6e84719240bb4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1705, \"end_char_idx\": 3485, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c4a71f9d-2a40-4e81-9581-f6b2589d53c9", "doc_id": "c4a71f9d-2a40-4e81-9581-f6b2589d53c9", "ref_doc_id": "c4a71f9d-2a40-4e81-9581-f6b2589d53c9", "chroma:document": "*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/owlv2_overview.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> OWLv2 high-level overview. Taken from the <a href=\"https://arxiv.org/abs/2306.09683\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit).\n\n## Usage example\n\nOWLv2 is, just like its predecessor [OWL-ViT](owlvit), a zero-shot text-conditioned object detection model. OWL-ViT uses [CLIP](clip) as its multi-modal backbone, with a ViT-like Transformer to get visual features and a causal language model to get the text features. To use CLIP for detection, OWL-ViT removes the final token pooling layer of the vision model and attaches a lightweight classification and box head to each transformer output token. Open-vocabulary classification is enabled by replacing the fixed classification layer weights with the class-name embeddings obtained from the text model. The authors first train CLIP from scratch and fine-tune it end-to-end with the classification and box heads on standard detection datasets using a bipartite matching loss. One or multiple text queries per image can be used to perform zero-shot text-conditioned object detection.\n\n[`Owlv2ImageProcessor`] can be used to resize (or rescale) and normalize images for the model and [`CLIPTokenizer`] is used to encode the text. [`Owlv2Processor`] wraps [`Owlv2ImageProcessor`] and [`CLIPTokenizer`] into a single instance to both encode the text and prepare the images. The following example shows how to perform object detection using [`Owlv2Processor`] and [`Owlv2ForObjectDetection`]."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 455d1950-f421-4a54-be2a-c9b557b5c74a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/owlv2/", "title": "OWLv2", "tokens": 1744, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"455d1950-f421-4a54-be2a-c9b557b5c74a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/owlv2/\", \"title\": \"OWLv2\", \"tokens\": 1744, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c4a71f9d-2a40-4e81-9581-f6b2589d53c9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/owlv2/\", \"title\": \"OWLv2\", \"tokens\": 1744, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f86ebc0ad8323d2222c1e2648a297da824bd8ca654568bb2d0c012639908b707\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"50591176-9971-4c09-8d10-a7b88aeb6281\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/owlv2/\", \"title\": \"OWLv2\", \"tokens\": 1744, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"6cc4501babd715a1b99022a7578caf1c4f7cd69561ce17e93dd6583bda7e43c8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9332ba2e-761f-4f6c-bbcc-3c13aa9527d0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8d1b944ae73367fdc90c704e54ef8a3ba35dd2b3ea9c7b66c61ad002c0c24d8e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1903, \"end_char_idx\": 5229, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c4a71f9d-2a40-4e81-9581-f6b2589d53c9", "doc_id": "c4a71f9d-2a40-4e81-9581-f6b2589d53c9", "ref_doc_id": "c4a71f9d-2a40-4e81-9581-f6b2589d53c9", "chroma:document": "Taken from the <a href=\"https://arxiv.org/abs/2306.09683\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit).\n\n## Usage example\n\nOWLv2 is, just like its predecessor [OWL-ViT](owlvit), a zero-shot text-conditioned object detection model. OWL-ViT uses [CLIP](clip) as its multi-modal backbone, with a ViT-like Transformer to get visual features and a causal language model to get the text features. To use CLIP for detection, OWL-ViT removes the final token pooling layer of the vision model and attaches a lightweight classification and box head to each transformer output token. Open-vocabulary classification is enabled by replacing the fixed classification layer weights with the class-name embeddings obtained from the text model. The authors first train CLIP from scratch and fine-tune it end-to-end with the classification and box heads on standard detection datasets using a bipartite matching loss. One or multiple text queries per image can be used to perform zero-shot text-conditioned object detection.\n\n[`Owlv2ImageProcessor`] can be used to resize (or rescale) and normalize images for the model and [`CLIPTokenizer`] is used to encode the text. [`Owlv2Processor`] wraps [`Owlv2ImageProcessor`] and [`CLIPTokenizer`] into a single instance to both encode the text and prepare the images. The following example shows how to perform object detection using [`Owlv2Processor`] and [`Owlv2ForObjectDetection`].\n\n```python\n>>> import requests\n>>> from PIL import Image\n>>> import torch\n\n>>> from transformers import Owlv2Processor, Owlv2ForObjectDetection\n\n>>> processor = Owlv2Processor.from_pretrained(\"google/owlv2-base-patch16-ensemble\")\n>>> model = Owlv2ForObjectDetection.from_pretrained(\"google/owlv2-base-patch16-ensemble\")\n\n>>> url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n>>> texts = [[\"a photo of a cat\", \"a photo of a dog\"]]\n>>> inputs = processor(text=texts, images=image, return_tensors=\"pt\")\n>>> outputs = model(**inputs)\n\n>>> # Target image sizes (height, width) to rescale box predictions [batch_size, 2]\n>>> target_sizes = torch.Tensor([image.size[::-1]])\n>>> # Convert outputs (bounding boxes and class logits) to Pascal VOC Format (xmin, ymin, xmax, ymax)\n>>> results = processor.post_process_object_detection(outputs=outputs, target_sizes=target_sizes, threshold=0.1)\n>>> i = 0  # Retrieve predictions for the first image for the corresponding text queries\n>>> text = texts[i]\n>>> boxes, scores, labels = results[i][\"boxes\"], results[i][\"scores\"], results[i][\"labels\"]\n>>> for box, score, label in zip(boxes, scores, labels):\n...     box = [round(i, 2) for i in box.tolist()]\n...     print(f\"Detected {text[label]} with confidence {round(score.item(), 3)} at location {box}\")\nDetected a photo of a cat with confidence 0.614 at location [341.67, 23.39, 642.32, 371.35]\nDetected a photo of a cat with confidence 0.665 at location [6.75, 51.96, 326.62, 473.13]\n```\n\n## Resources\n\n- A demo notebook on using OWLv2 for zero- and one-shot (image-guided) object detection can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/OWLv2)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 9332ba2e-761f-4f6c-bbcc-3c13aa9527d0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/owlv2/", "title": "OWLv2", "tokens": 1744, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"9332ba2e-761f-4f6c-bbcc-3c13aa9527d0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/owlv2/\", \"title\": \"OWLv2\", \"tokens\": 1744, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c4a71f9d-2a40-4e81-9581-f6b2589d53c9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/owlv2/\", \"title\": \"OWLv2\", \"tokens\": 1744, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f86ebc0ad8323d2222c1e2648a297da824bd8ca654568bb2d0c012639908b707\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"455d1950-f421-4a54-be2a-c9b557b5c74a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/owlv2/\", \"title\": \"OWLv2\", \"tokens\": 1744, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"99b9df51e44b27882bf05afa98588b59d6deb89c0ce79c74b94d0b62a3af69b0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5230, \"end_char_idx\": 6493, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c4a71f9d-2a40-4e81-9581-f6b2589d53c9", "doc_id": "c4a71f9d-2a40-4e81-9581-f6b2589d53c9", "ref_doc_id": "c4a71f9d-2a40-4e81-9581-f6b2589d53c9", "chroma:document": "- [Zero-shot object detection task guide](../tasks/zero_shot_object_detection)\n\n<Tip>\n\nThe architecture of OWLv2 is identical to [OWL-ViT](owlvit), however the object detection head now also includes an objectness classifier, which predicts the (query-agnostic) likelihood that a predicted box contains an object (as opposed to background). The objectness score can be used to rank or filter predictions independently of text queries.\nUsage of OWLv2 is identical to [OWL-ViT](owlvit) with a new, updated image processor ([`Owlv2ImageProcessor`]).\n\n</Tip>\n\n## Owlv2Config\n\n[[autodoc]] Owlv2Config\n    - from_text_vision_configs\n\n## Owlv2TextConfig\n\n[[autodoc]] Owlv2TextConfig\n\n## Owlv2VisionConfig\n\n[[autodoc]] Owlv2VisionConfig\n\n## Owlv2ImageProcessor\n\n[[autodoc]] Owlv2ImageProcessor\n    - preprocess\n    - post_process_object_detection\n    - post_process_image_guided_detection\n\n## Owlv2Processor\n\n[[autodoc]] Owlv2Processor\n\n## Owlv2Model\n\n[[autodoc]] Owlv2Model\n    - forward\n    - get_text_features\n    - get_image_features\n\n## Owlv2TextModel\n\n[[autodoc]] Owlv2TextModel\n    - forward\n\n## Owlv2VisionModel\n\n[[autodoc]] Owlv2VisionModel\n    - forward\n\n## Owlv2ForObjectDetection\n\n[[autodoc]] Owlv2ForObjectDetection\n    - forward\n    - image_guided_detection"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: fa25e367-3f56-4f34-bfed-22fb1dd45517
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/funnel/", "title": "Funnel Transformer", "tokens": 1572, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"fa25e367-3f56-4f34-bfed-22fb1dd45517\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/funnel/\", \"title\": \"Funnel Transformer\", \"tokens\": 1572, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1814850e-b84c-4bf4-a1f9-8e826622d95d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/funnel/\", \"title\": \"Funnel Transformer\", \"tokens\": 1572, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"72e3497f2c1024aa8fec3d5d1e6644b3ca65b1faa83d212329850068a7472035\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dc0aebd1-1b92-4efc-a5f8-626d16dcf2aa\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6e7d5d122c7543867a5533232c5b311e149775a95169dd271d73f10dd1b6ab8b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 393, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1814850e-b84c-4bf4-a1f9-8e826622d95d", "doc_id": "1814850e-b84c-4bf4-a1f9-8e826622d95d", "ref_doc_id": "1814850e-b84c-4bf4-a1f9-8e826622d95d", "chroma:document": "# Funnel Transformer\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=funnel\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-funnel-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/funnel-transformer-small\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: dc0aebd1-1b92-4efc-a5f8-626d16dcf2aa
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/funnel/", "title": "Funnel Transformer", "tokens": 1572, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"dc0aebd1-1b92-4efc-a5f8-626d16dcf2aa\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/funnel/\", \"title\": \"Funnel Transformer\", \"tokens\": 1572, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1814850e-b84c-4bf4-a1f9-8e826622d95d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/funnel/\", \"title\": \"Funnel Transformer\", \"tokens\": 1572, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"72e3497f2c1024aa8fec3d5d1e6644b3ca65b1faa83d212329850068a7472035\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"fa25e367-3f56-4f34-bfed-22fb1dd45517\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/funnel/\", \"title\": \"Funnel Transformer\", \"tokens\": 1572, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f3368ab1cbf7faa39bd6600cc0dc8928c0dc511324b1c290a3d3bea1d186aeda\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f23f71d0-e36a-4b65-8d08-fd2bbb1fec49\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e9f96315b912f044856534ffc6a09d39fe3c7070fbc2a4a5bec6d5d4626a43b9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 4081, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1814850e-b84c-4bf4-a1f9-8e826622d95d", "doc_id": "1814850e-b84c-4bf4-a1f9-8e826622d95d", "ref_doc_id": "1814850e-b84c-4bf4-a1f9-8e826622d95d", "chroma:document": "# Funnel Transformer\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=funnel\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-funnel-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/funnel-transformer-small\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n\n## Overview\n\nThe Funnel Transformer model was proposed in the paper [Funnel-Transformer: Filtering out Sequential Redundancy for\nEfficient Language Processing](https://arxiv.org/abs/2006.03236). It is a bidirectional transformer model, like\nBERT, but with a pooling operation after each block of layers, a bit like in traditional convolutional neural networks\n(CNN) in computer vision.\n\nThe abstract from the paper is the following:\n\n*With the success of language pretraining, it is highly desirable to develop more efficient architectures of good\nscalability that can exploit the abundant unlabeled data at a lower cost. To improve the efficiency, we examine the\nmuch-overlooked redundancy in maintaining a full-length token-level presentation, especially for tasks that only\nrequire a single-vector presentation of the sequence. With this intuition, we propose Funnel-Transformer which\ngradually compresses the sequence of hidden states to a shorter one and hence reduces the computation cost. More\nimportantly, by re-investing the saved FLOPs from length reduction in constructing a deeper or wider model, we further\nimprove the model capacity. In addition, to perform token-level predictions as required by common pretraining\nobjectives, Funnel-Transformer is able to recover a deep representation for each token from the reduced hidden sequence\nvia a decoder. Empirically, with comparable or fewer FLOPs, Funnel-Transformer outperforms the standard Transformer on\na wide variety of sequence-level prediction tasks, including text classification, language understanding, and reading\ncomprehension.*\n\nThis model was contributed by [sgugger](https://huggingface.co/sgugger). The original code can be found [here](https://github.com/laiguokun/Funnel-Transformer).\n\n## Usage tips\n\n- Since Funnel Transformer uses pooling, the sequence length of the hidden states changes after each block of layers. This way, their length is divided by 2, which speeds up the computation of the next hidden states.\n  The base model therefore has a final sequence length that is a quarter of the original one. This model can be used\n  directly for tasks that just require a sentence summary (like sequence classification or multiple choice). For other\n  tasks, the full model is used; this full model has a decoder that upsamples the final hidden states to the same\n  sequence length as the input.\n- For tasks such as classification, this is not a problem, but for tasks like masked language modeling or token classification, we need a hidden state with the same sequence length as the original input. In those cases, the final hidden states are upsampled to the input sequence length and go through two additional layers. That's why there are two versions of each checkpoint. The version suffixed with \u201c-base\u201d contains only the three blocks, while the version without that suffix contains the three blocks and the upsampling head with its additional layers.\n- The Funnel Transformer checkpoints are all available with a full version and a base version. The first ones should be\n  used for [`FunnelModel`], [`FunnelForPreTraining`],\n  [`FunnelForMaskedLM`], [`FunnelForTokenClassification`] and\n  [`FunnelForQuestionAnswering`]. The second ones should be used for\n  [`FunnelBaseModel`], [`FunnelForSequenceClassification`] and\n  [`FunnelForMultipleChoice`].\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: f23f71d0-e36a-4b65-8d08-fd2bbb1fec49
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/funnel/", "title": "Funnel Transformer", "tokens": 1572, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"f23f71d0-e36a-4b65-8d08-fd2bbb1fec49\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/funnel/\", \"title\": \"Funnel Transformer\", \"tokens\": 1572, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1814850e-b84c-4bf4-a1f9-8e826622d95d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/funnel/\", \"title\": \"Funnel Transformer\", \"tokens\": 1572, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"72e3497f2c1024aa8fec3d5d1e6644b3ca65b1faa83d212329850068a7472035\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dc0aebd1-1b92-4efc-a5f8-626d16dcf2aa\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/funnel/\", \"title\": \"Funnel Transformer\", \"tokens\": 1572, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"97986fe909fca7be2052059b2ba450797c90158dfcc29ad374e32e6f7192e65f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4084, \"end_char_idx\": 5832, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1814850e-b84c-4bf4-a1f9-8e826622d95d", "doc_id": "1814850e-b84c-4bf4-a1f9-8e826622d95d", "ref_doc_id": "1814850e-b84c-4bf4-a1f9-8e826622d95d", "chroma:document": "## FunnelConfig\n\n[[autodoc]] FunnelConfig\n\n## FunnelTokenizer\n\n[[autodoc]] FunnelTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## FunnelTokenizerFast\n\n[[autodoc]] FunnelTokenizerFast\n\n## Funnel specific outputs\n\n[[autodoc]] models.funnel.modeling_funnel.FunnelForPreTrainingOutput\n\n[[autodoc]] models.funnel.modeling_tf_funnel.TFFunnelForPreTrainingOutput\n\n<frameworkcontent>\n<pt>\n\n## FunnelBaseModel\n\n[[autodoc]] FunnelBaseModel\n    - forward\n\n## FunnelModel\n\n[[autodoc]] FunnelModel\n    - forward\n\n## FunnelModelForPreTraining\n\n[[autodoc]] FunnelForPreTraining\n    - forward\n\n## FunnelForMaskedLM\n\n[[autodoc]] FunnelForMaskedLM\n    - forward\n\n## FunnelForSequenceClassification\n\n[[autodoc]] FunnelForSequenceClassification\n    - forward\n\n## FunnelForMultipleChoice\n\n[[autodoc]] FunnelForMultipleChoice\n    - forward\n\n## FunnelForTokenClassification\n\n[[autodoc]] FunnelForTokenClassification\n    - forward\n\n## FunnelForQuestionAnswering\n\n[[autodoc]] FunnelForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFFunnelBaseModel\n\n[[autodoc]] TFFunnelBaseModel\n    - call\n\n## TFFunnelModel\n\n[[autodoc]] TFFunnelModel\n    - call\n\n## TFFunnelModelForPreTraining\n\n[[autodoc]] TFFunnelForPreTraining\n    - call\n\n## TFFunnelForMaskedLM\n\n[[autodoc]] TFFunnelForMaskedLM\n    - call\n\n## TFFunnelForSequenceClassification\n\n[[autodoc]] TFFunnelForSequenceClassification\n    - call\n\n## TFFunnelForMultipleChoice\n\n[[autodoc]] TFFunnelForMultipleChoice\n    - call\n\n## TFFunnelForTokenClassification\n\n[[autodoc]] TFFunnelForTokenClassification\n    - call\n\n## TFFunnelForQuestionAnswering\n\n[[autodoc]] TFFunnelForQuestionAnswering\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: a347359f-5dea-4718-9e78-d7a4af250952
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llama2/", "title": "Llama2", "tokens": 2821, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"a347359f-5dea-4718-9e78-d7a4af250952\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama2/\", \"title\": \"Llama2\", \"tokens\": 2821, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"58df5025-e638-496c-8c3e-5368bcf4376f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama2/\", \"title\": \"Llama2\", \"tokens\": 2821, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5938d926f7e281fe082e27f5501999084a6b851cd99182cd40b695dbe49fd0ea\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"be46eb81-28c9-48d0-98f3-78d470216de0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"20adad9b1102008813e6aff516481b568d5262f92f50b5a4f9ed224b0041f43b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2668, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "58df5025-e638-496c-8c3e-5368bcf4376f", "doc_id": "58df5025-e638-496c-8c3e-5368bcf4376f", "ref_doc_id": "58df5025-e638-496c-8c3e-5368bcf4376f", "chroma:document": "# Llama2\n\n## Overview\n\nThe Llama2 model was proposed in [LLaMA: Open Foundation and Fine-Tuned Chat Models](https://ai.meta.com/research/publications/llama-2-open-foundation-and-fine-tuned-chat-models/) by Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushka rMishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing EllenTan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, Thomas Scialom. It is a collection of foundation language models ranging from 7B to 70B parameters, with checkpoints finetuned for chat application!\n\nThe abstract from the paper is the following:\n\n*In this work, we develop and release Llama 2, a collection of pretrained and fine-tuned large language models (LLMs) ranging in scale from 7 billion to 70 billion parameters. Our fine-tuned LLMs, called Llama 2-Chat, are optimized for dialogue use cases. Our models outperform open-source chat models on most benchmarks we tested, and based on our human evaluations for helpfulness and safety, may be a suitable substitute for closed-source models. We provide a detailed description of our approach to fine-tuning and safety improvements of Llama 2-Chat in order to enable the community to build on our work and contribute to the responsible development of LLMs.*\n\nCheckout all Llama2 model checkpoints [here](https://huggingface.co/models?search=llama2).\nThis model was contributed by [Arthur Zucker](https://huggingface.co/ArthurZ) with contributions from [Lysandre Debut](https://huggingface.co/lysandre). The code of the implementation in Hugging Face is based on GPT-NeoX [here](https://github.com/EleutherAI/gpt-neox). The original code of the authors can be found [here](https://github.com/facebookresearch/llama).\n\n## Usage tips\n\n<Tip warning={true}>\n\nThe `Llama2` models were trained using `bfloat16`, but the original inference uses `float16`."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: be46eb81-28c9-48d0-98f3-78d470216de0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llama2/", "title": "Llama2", "tokens": 2821, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"be46eb81-28c9-48d0-98f3-78d470216de0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama2/\", \"title\": \"Llama2\", \"tokens\": 2821, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"58df5025-e638-496c-8c3e-5368bcf4376f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama2/\", \"title\": \"Llama2\", \"tokens\": 2821, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5938d926f7e281fe082e27f5501999084a6b851cd99182cd40b695dbe49fd0ea\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a347359f-5dea-4718-9e78-d7a4af250952\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama2/\", \"title\": \"Llama2\", \"tokens\": 2821, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"109bf69a61b92f43c158931be49f33e14952184745966d393c6ba95065a3046a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4c329852-007c-4af1-8c57-dbff6d07b141\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c5ec7006ecb8e91eff4b7d46e7234753e6015b976ba4f9a08be1700e73df63a3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1233, \"end_char_idx\": 4272, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "58df5025-e638-496c-8c3e-5368bcf4376f", "doc_id": "58df5025-e638-496c-8c3e-5368bcf4376f", "ref_doc_id": "58df5025-e638-496c-8c3e-5368bcf4376f", "chroma:document": "It is a collection of foundation language models ranging from 7B to 70B parameters, with checkpoints finetuned for chat application!\n\nThe abstract from the paper is the following:\n\n*In this work, we develop and release Llama 2, a collection of pretrained and fine-tuned large language models (LLMs) ranging in scale from 7 billion to 70 billion parameters. Our fine-tuned LLMs, called Llama 2-Chat, are optimized for dialogue use cases. Our models outperform open-source chat models on most benchmarks we tested, and based on our human evaluations for helpfulness and safety, may be a suitable substitute for closed-source models. We provide a detailed description of our approach to fine-tuning and safety improvements of Llama 2-Chat in order to enable the community to build on our work and contribute to the responsible development of LLMs.*\n\nCheckout all Llama2 model checkpoints [here](https://huggingface.co/models?search=llama2).\nThis model was contributed by [Arthur Zucker](https://huggingface.co/ArthurZ) with contributions from [Lysandre Debut](https://huggingface.co/lysandre). The code of the implementation in Hugging Face is based on GPT-NeoX [here](https://github.com/EleutherAI/gpt-neox). The original code of the authors can be found [here](https://github.com/facebookresearch/llama).\n\n## Usage tips\n\n<Tip warning={true}>\n\nThe `Llama2` models were trained using `bfloat16`, but the original inference uses `float16`. The checkpoints uploaded on the Hub use `torch_dtype = 'float16'`, which will be\nused by the `AutoModel` API to cast the checkpoints from `torch.float32` to `torch.float16`. \n\nThe `dtype` of the online weights is mostly irrelevant unless you are using `torch_dtype=\"auto\"` when initializing a model using `model = AutoModelForCausalLM.from_pretrained(\"path\", torch_dtype = \"auto\")`. The reason is that the model will first be downloaded ( using the `dtype` of the checkpoints online), then it will be casted to the default `dtype` of `torch` (becomes `torch.float32`), and finally, if there is a `torch_dtype` provided in the config, it will be used. \n\nTraining the model in `float16` is not recommended and is known to produce `nan`; as such, the model should be trained in `bfloat16`.\n\n</Tip>\n\nTips:\n\n- Weights for the Llama2 models can be obtained by filling out [this form](https://ai.meta.com/resources/models-and-libraries/llama-downloads/)\n- The architecture is very similar to the first Llama, with the addition of Grouped Query Attention (GQA) following this [paper](https://arxiv.org/pdf/2305.13245.pdf)\n- Setting `config.pretraining_tp` to a value different than 1 will activate the more accurate but slower computation of the linear layers, which should better match the original logits.\n- The original model uses `pad_id = -1` which means that there is no padding token. We can't have the same logic, make sure to add a padding token using `tokenizer.add_special_tokens({\"pad_token\":\"<pad>\"})` and resize the token embedding accordingly. You should also set the `model.config.pad_token_id`."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 4c329852-007c-4af1-8c57-dbff6d07b141
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llama2/", "title": "Llama2", "tokens": 2821, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"4c329852-007c-4af1-8c57-dbff6d07b141\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama2/\", \"title\": \"Llama2\", \"tokens\": 2821, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"58df5025-e638-496c-8c3e-5368bcf4376f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama2/\", \"title\": \"Llama2\", \"tokens\": 2821, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5938d926f7e281fe082e27f5501999084a6b851cd99182cd40b695dbe49fd0ea\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"be46eb81-28c9-48d0-98f3-78d470216de0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama2/\", \"title\": \"Llama2\", \"tokens\": 2821, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"ab54c79cff4c2b9bb0f3d5fe601de42782e5102c21f60594ec27cc9fbbc7024d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"23f24e03-f23b-4596-986b-494b62430447\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8c4dc8371e1ae68ea66892214dbd8c46b33b9dfc40a9d52fa9e1c393cb9305a8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2669, \"end_char_idx\": 5970, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "58df5025-e638-496c-8c3e-5368bcf4376f", "doc_id": "58df5025-e638-496c-8c3e-5368bcf4376f", "ref_doc_id": "58df5025-e638-496c-8c3e-5368bcf4376f", "chroma:document": "The checkpoints uploaded on the Hub use `torch_dtype = 'float16'`, which will be\nused by the `AutoModel` API to cast the checkpoints from `torch.float32` to `torch.float16`. \n\nThe `dtype` of the online weights is mostly irrelevant unless you are using `torch_dtype=\"auto\"` when initializing a model using `model = AutoModelForCausalLM.from_pretrained(\"path\", torch_dtype = \"auto\")`. The reason is that the model will first be downloaded ( using the `dtype` of the checkpoints online), then it will be casted to the default `dtype` of `torch` (becomes `torch.float32`), and finally, if there is a `torch_dtype` provided in the config, it will be used. \n\nTraining the model in `float16` is not recommended and is known to produce `nan`; as such, the model should be trained in `bfloat16`.\n\n</Tip>\n\nTips:\n\n- Weights for the Llama2 models can be obtained by filling out [this form](https://ai.meta.com/resources/models-and-libraries/llama-downloads/)\n- The architecture is very similar to the first Llama, with the addition of Grouped Query Attention (GQA) following this [paper](https://arxiv.org/pdf/2305.13245.pdf)\n- Setting `config.pretraining_tp` to a value different than 1 will activate the more accurate but slower computation of the linear layers, which should better match the original logits.\n- The original model uses `pad_id = -1` which means that there is no padding token. We can't have the same logic, make sure to add a padding token using `tokenizer.add_special_tokens({\"pad_token\":\"<pad>\"})` and resize the token embedding accordingly. You should also set the `model.config.pad_token_id`. The `embed_tokens` layer of the model is initialized with `self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.config.padding_idx)`, which makes sure that encoding the padding token will output zeros, so passing it when initializing is recommended.\n- After filling out the form and gaining access to the model checkpoints, you should be able to use the already converted checkpoints. Otherwise, if you are converting your own model, feel free to use the [conversion script](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/convert_llama_weights_to_hf.py). The script can be called with the following (example) command:\n\n```bash\npython src/transformers/models/llama/convert_llama_weights_to_hf.py \\\n    --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path\n```\n\n- After conversion, the model and tokenizer can be loaded via:\n\n```python\nfrom transformers import LlamaForCausalLM, LlamaTokenizer\n\ntokenizer = LlamaTokenizer.from_pretrained(\"/output/path\")\nmodel = LlamaForCausalLM.from_pretrained(\"/output/path\")\n```\n\nNote that executing the script requires enough CPU RAM to host the whole model in float16 precision (even if the biggest versions\ncome in several checkpoints they each contain a part of each weight of the model, so we need to load them all in RAM). For the 75B model, it's thus 145GB of RAM needed.\n\n- The LLaMA tokenizer is a BPE model based on [sentencepiece](https://github.com/google/sentencepiece). One quirk of sentencepiece is that when decoding a sequence, if the first token is the start of the word (e.g. \"Banana\"), the tokenizer does not prepend the prefix space to the string."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 23f24e03-f23b-4596-986b-494b62430447
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llama2/", "title": "Llama2", "tokens": 2821, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"23f24e03-f23b-4596-986b-494b62430447\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama2/\", \"title\": \"Llama2\", \"tokens\": 2821, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"58df5025-e638-496c-8c3e-5368bcf4376f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama2/\", \"title\": \"Llama2\", \"tokens\": 2821, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5938d926f7e281fe082e27f5501999084a6b851cd99182cd40b695dbe49fd0ea\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4c329852-007c-4af1-8c57-dbff6d07b141\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama2/\", \"title\": \"Llama2\", \"tokens\": 2821, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a0974b61cd19f9ea5ab5c3bc1240811adaea18464ea48a6b352b5f3a82c3b179\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e49b3e86-f46c-4186-a251-506473ac800f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9c8fe80139f963b263e96614110165e3b131990ef025b9676a4d0f59b4dc8e6c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4273, \"end_char_idx\": 7277, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "58df5025-e638-496c-8c3e-5368bcf4376f", "doc_id": "58df5025-e638-496c-8c3e-5368bcf4376f", "ref_doc_id": "58df5025-e638-496c-8c3e-5368bcf4376f", "chroma:document": "The `embed_tokens` layer of the model is initialized with `self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.config.padding_idx)`, which makes sure that encoding the padding token will output zeros, so passing it when initializing is recommended.\n- After filling out the form and gaining access to the model checkpoints, you should be able to use the already converted checkpoints. Otherwise, if you are converting your own model, feel free to use the [conversion script](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/convert_llama_weights_to_hf.py). The script can be called with the following (example) command:\n\n```bash\npython src/transformers/models/llama/convert_llama_weights_to_hf.py \\\n    --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path\n```\n\n- After conversion, the model and tokenizer can be loaded via:\n\n```python\nfrom transformers import LlamaForCausalLM, LlamaTokenizer\n\ntokenizer = LlamaTokenizer.from_pretrained(\"/output/path\")\nmodel = LlamaForCausalLM.from_pretrained(\"/output/path\")\n```\n\nNote that executing the script requires enough CPU RAM to host the whole model in float16 precision (even if the biggest versions\ncome in several checkpoints they each contain a part of each weight of the model, so we need to load them all in RAM). For the 75B model, it's thus 145GB of RAM needed.\n\n- The LLaMA tokenizer is a BPE model based on [sentencepiece](https://github.com/google/sentencepiece). One quirk of sentencepiece is that when decoding a sequence, if the first token is the start of the word (e.g. \"Banana\"), the tokenizer does not prepend the prefix space to the string.\n\n- When using Flash Attention 2 via `attn_implementation=\"flash_attention_2\"`, don't pass `torch_dtype` to the `from_pretrained` class method and use Automatic Mixed-Precision training. When using `Trainer`, it is simply specifying either `fp16` or `bf16` to `True`. Otherwise, make sure you are using `torch.autocast`. This is required because the Flash Attention only support `fp16` and `bf16` data type.\n\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with LLaMA2. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n- [Llama 2 is here - get it on Hugging Face](https://huggingface.co/blog/llama2), a blog post about Llama 2 and how to use it with \ud83e\udd17 Transformers and \ud83e\udd17 PEFT.\n- [LLaMA 2 - Every Resource you need](https://www.philschmid.de/llama-2), a compilation of relevant resources to learn about LLaMA 2 and how to get started quickly.\n\n<PipelineTag pipeline=\"text-generation\"/>\n\n- A [notebook](https://colab.research.google.com/drive/1PEQyJO1-f6j0S_XJ8DV50NkpzasXkrzd?usp=sharing) on how to fine-tune Llama 2 in Google Colab using QLoRA and 4-bit precision."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: e49b3e86-f46c-4186-a251-506473ac800f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llama2/", "title": "Llama2", "tokens": 2821, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"e49b3e86-f46c-4186-a251-506473ac800f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama2/\", \"title\": \"Llama2\", \"tokens\": 2821, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"58df5025-e638-496c-8c3e-5368bcf4376f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama2/\", \"title\": \"Llama2\", \"tokens\": 2821, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5938d926f7e281fe082e27f5501999084a6b851cd99182cd40b695dbe49fd0ea\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"23f24e03-f23b-4596-986b-494b62430447\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama2/\", \"title\": \"Llama2\", \"tokens\": 2821, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"cb258f54569cc0bd10dfc1772e3dd2cae34ddfd68ae190c60a32a727176ad73f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6fdc3d7b-7b79-4d10-a3e0-832544872a0e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b45673d818dcab6095c8cf5d18d9ee8bf386c86b1fc8c27ec942c6e498d0fc95\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5897, \"end_char_idx\": 8335, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "58df5025-e638-496c-8c3e-5368bcf4376f", "doc_id": "58df5025-e638-496c-8c3e-5368bcf4376f", "ref_doc_id": "58df5025-e638-496c-8c3e-5368bcf4376f", "chroma:document": "\"Banana\"), the tokenizer does not prepend the prefix space to the string.\n\n- When using Flash Attention 2 via `attn_implementation=\"flash_attention_2\"`, don't pass `torch_dtype` to the `from_pretrained` class method and use Automatic Mixed-Precision training. When using `Trainer`, it is simply specifying either `fp16` or `bf16` to `True`. Otherwise, make sure you are using `torch.autocast`. This is required because the Flash Attention only support `fp16` and `bf16` data type.\n\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with LLaMA2. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n- [Llama 2 is here - get it on Hugging Face](https://huggingface.co/blog/llama2), a blog post about Llama 2 and how to use it with \ud83e\udd17 Transformers and \ud83e\udd17 PEFT.\n- [LLaMA 2 - Every Resource you need](https://www.philschmid.de/llama-2), a compilation of relevant resources to learn about LLaMA 2 and how to get started quickly.\n\n<PipelineTag pipeline=\"text-generation\"/>\n\n- A [notebook](https://colab.research.google.com/drive/1PEQyJO1-f6j0S_XJ8DV50NkpzasXkrzd?usp=sharing) on how to fine-tune Llama 2 in Google Colab using QLoRA and 4-bit precision. \ud83c\udf0e\n- A [notebook](https://colab.research.google.com/drive/134o_cXcMe_lsvl15ZE_4Y75Kstepsntu?usp=sharing) on how to fine-tune the \"Llama-v2-7b-guanaco\" model with 4-bit QLoRA and generate Q&A datasets from PDFs. \ud83c\udf0e\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- A [notebook](https://colab.research.google.com/drive/1ggaa2oRFphdBmqIjSEbnb_HGkcIRC2ZB?usp=sharing) on how to fine-tune the Llama 2 model with QLoRa, TRL, and Korean text classification dataset. \ud83c\udf0e\ud83c\uddf0\ud83c\uddf7\n\n\u2697\ufe0f Optimization\n- [Fine-tune Llama 2 with DPO](https://huggingface.co/blog/dpo-trl), a guide to using the TRL library's DPO method to fine tune Llama 2 on a specific dataset.\n- [Extended Guide: Instruction-tune Llama 2](https://www.philschmid.de/instruction-tune-llama-2), a guide to training Llama 2 to generate instructions from inputs, transforming the model from instruction-following to instruction-giving.\n- A [notebook](https://colab.research.google.com/drive/1SYpgFpcmtIUzdE7pxqknrM4ArCASfkFQ?usp=sharing) on how to fine-tune the Llama 2 model on a personal computer using QLoRa and TRL."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 6fdc3d7b-7b79-4d10-a3e0-832544872a0e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llama2/", "title": "Llama2", "tokens": 2821, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"6fdc3d7b-7b79-4d10-a3e0-832544872a0e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama2/\", \"title\": \"Llama2\", \"tokens\": 2821, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"58df5025-e638-496c-8c3e-5368bcf4376f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama2/\", \"title\": \"Llama2\", \"tokens\": 2821, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5938d926f7e281fe082e27f5501999084a6b851cd99182cd40b695dbe49fd0ea\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e49b3e86-f46c-4186-a251-506473ac800f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama2/\", \"title\": \"Llama2\", \"tokens\": 2821, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"3cf5ed1600d5b35a386935cfd4b689bb571891daf5f0538d2b4719811e97782d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ff93541e-b452-4c47-9b24-71f0bda8000a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c18d822fb17a7cd8885b892614de55873e6ce1227443d22b527ff2548f95a187\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7278, \"end_char_idx\": 9606, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "58df5025-e638-496c-8c3e-5368bcf4376f", "doc_id": "58df5025-e638-496c-8c3e-5368bcf4376f", "ref_doc_id": "58df5025-e638-496c-8c3e-5368bcf4376f", "chroma:document": "\ud83c\udf0e\n- A [notebook](https://colab.research.google.com/drive/134o_cXcMe_lsvl15ZE_4Y75Kstepsntu?usp=sharing) on how to fine-tune the \"Llama-v2-7b-guanaco\" model with 4-bit QLoRA and generate Q&A datasets from PDFs. \ud83c\udf0e\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- A [notebook](https://colab.research.google.com/drive/1ggaa2oRFphdBmqIjSEbnb_HGkcIRC2ZB?usp=sharing) on how to fine-tune the Llama 2 model with QLoRa, TRL, and Korean text classification dataset. \ud83c\udf0e\ud83c\uddf0\ud83c\uddf7\n\n\u2697\ufe0f Optimization\n- [Fine-tune Llama 2 with DPO](https://huggingface.co/blog/dpo-trl), a guide to using the TRL library's DPO method to fine tune Llama 2 on a specific dataset.\n- [Extended Guide: Instruction-tune Llama 2](https://www.philschmid.de/instruction-tune-llama-2), a guide to training Llama 2 to generate instructions from inputs, transforming the model from instruction-following to instruction-giving.\n- A [notebook](https://colab.research.google.com/drive/1SYpgFpcmtIUzdE7pxqknrM4ArCASfkFQ?usp=sharing) on how to fine-tune the Llama 2 model on a personal computer using QLoRa and TRL. \ud83c\udf0e\n\n\u26a1\ufe0f Inference\n- A [notebook](https://colab.research.google.com/drive/1TC56ArKerXUpbgRy5vM3woRsbTEVNq7h?usp=sharing) on how to quantize the Llama 2 model using GPTQ from the AutoGPTQ library. \ud83c\udf0e\n- A [notebook](https://colab.research.google.com/drive/1X1z9Q6domMKl2CnEM0QGHNwidLfR4dW2?usp=sharing) on how to run the Llama 2 Chat Model with 4-bit quantization on a local computer or Google Colab. \ud83c\udf0e\n\n\ud83d\ude80 Deploy\n- [Fine-tune LLaMA 2 (7-70B) on Amazon SageMaker](https://www.philschmid.de/sagemaker-llama2-qlora), a complete guide from setup to QLoRA fine-tuning and deployment on Amazon SageMaker.\n- [Deploy Llama 2 7B/13B/70B on Amazon SageMaker](https://www.philschmid.de/sagemaker-llama-llm), a guide on using Hugging Face's LLM DLC container for secure and scalable deployment.\n\n\n## LlamaConfig\n\n[[autodoc]] LlamaConfig\n\n\n## LlamaTokenizer\n\n[[autodoc]] LlamaTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## LlamaTokenizerFast\n\n[[autodoc]] LlamaTokenizerFast\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - update_post_processor\n    - save_vocabulary\n\n## LlamaModel\n\n[[autodoc]] LlamaModel\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: ff93541e-b452-4c47-9b24-71f0bda8000a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llama2/", "title": "Llama2", "tokens": 2821, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"ff93541e-b452-4c47-9b24-71f0bda8000a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama2/\", \"title\": \"Llama2\", \"tokens\": 2821, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"58df5025-e638-496c-8c3e-5368bcf4376f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama2/\", \"title\": \"Llama2\", \"tokens\": 2821, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5938d926f7e281fe082e27f5501999084a6b851cd99182cd40b695dbe49fd0ea\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6fdc3d7b-7b79-4d10-a3e0-832544872a0e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama2/\", \"title\": \"Llama2\", \"tokens\": 2821, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"fb60da1f31f59bc9d123b9b0a82ac58b7dbfc973f9e4826b532a1b0f2637f1f2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8529, \"end_char_idx\": 9765, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "58df5025-e638-496c-8c3e-5368bcf4376f", "doc_id": "58df5025-e638-496c-8c3e-5368bcf4376f", "ref_doc_id": "58df5025-e638-496c-8c3e-5368bcf4376f", "chroma:document": "\ud83c\udf0e\n- A [notebook](https://colab.research.google.com/drive/1X1z9Q6domMKl2CnEM0QGHNwidLfR4dW2?usp=sharing) on how to run the Llama 2 Chat Model with 4-bit quantization on a local computer or Google Colab. \ud83c\udf0e\n\n\ud83d\ude80 Deploy\n- [Fine-tune LLaMA 2 (7-70B) on Amazon SageMaker](https://www.philschmid.de/sagemaker-llama2-qlora), a complete guide from setup to QLoRA fine-tuning and deployment on Amazon SageMaker.\n- [Deploy Llama 2 7B/13B/70B on Amazon SageMaker](https://www.philschmid.de/sagemaker-llama-llm), a guide on using Hugging Face's LLM DLC container for secure and scalable deployment.\n\n\n## LlamaConfig\n\n[[autodoc]] LlamaConfig\n\n\n## LlamaTokenizer\n\n[[autodoc]] LlamaTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## LlamaTokenizerFast\n\n[[autodoc]] LlamaTokenizerFast\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - update_post_processor\n    - save_vocabulary\n\n## LlamaModel\n\n[[autodoc]] LlamaModel\n    - forward\n\n\n## LlamaForCausalLM\n\n[[autodoc]] LlamaForCausalLM\n    - forward\n\n## LlamaForSequenceClassification\n\n[[autodoc]] LlamaForSequenceClassification\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 89306e40-ba1a-4eec-91fb-6a59c7924ad6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mctct/", "title": "M-CTC-T", "tokens": 810, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"89306e40-ba1a-4eec-91fb-6a59c7924ad6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mctct/\", \"title\": \"M-CTC-T\", \"tokens\": 810, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"31cdcbce-093a-411a-bc56-ea30b2ae38a2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mctct/\", \"title\": \"M-CTC-T\", \"tokens\": 810, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"db46f791e17f42066b7a75c2edc6ee4cc2eb3d80521fb5c58fcd92170692649f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2593, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "31cdcbce-093a-411a-bc56-ea30b2ae38a2", "doc_id": "31cdcbce-093a-411a-bc56-ea30b2ae38a2", "ref_doc_id": "31cdcbce-093a-411a-bc56-ea30b2ae38a2", "chroma:document": "# M-CTC-T\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, so we won't accept any new PRs changing its code.\n\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.30.0.\nYou can do so by running the following command: `pip install -U transformers==4.30.0`.\n\n</Tip>\n\n## Overview\n\nThe M-CTC-T model was proposed in [Pseudo-Labeling For Massively Multilingual Speech Recognition](https://arxiv.org/abs/2111.00161) by Loren Lugosch, Tatiana Likhomanenko, Gabriel Synnaeve, and Ronan Collobert. The model is a 1B-param transformer encoder, with a CTC head over 8065 character labels and a language identification head over 60 language ID labels. It is trained on Common Voice (version 6.1, December 2020 release) and VoxPopuli. After training on Common Voice and VoxPopuli, the model is trained on Common Voice only. The labels are unnormalized character-level transcripts (punctuation and capitalization are not removed). The model takes as input Mel filterbank features from a 16Khz audio signal.\n\nThe abstract from the paper is the following:\n\n*Semi-supervised learning through pseudo-labeling has become a staple of state-of-the-art monolingual\nspeech recognition systems. In this work, we extend pseudo-labeling to massively multilingual speech\nrecognition with 60 languages. We propose a simple pseudo-labeling recipe that works well even\nwith low-resource languages: train a supervised multilingual model, fine-tune it with semi-supervised\nlearning on a target language, generate pseudo-labels for that language, and train a final model using\npseudo-labels for all languages, either from scratch or by fine-tuning. Experiments on the labeled\nCommon Voice and unlabeled VoxPopuli datasets show that our recipe can yield a model with better\nperformance for many languages that also transfers well to LibriSpeech.*\n\nThis model was contributed by [cwkeam](https://huggingface.co/cwkeam). The original code can be found [here](https://github.com/flashlight/wav2letter/tree/main/recipes/mling_pl).\n\n## Usage tips\n\nThe PyTorch version of this model is only available in torch 1.9 and higher.\n\n## Resources\n\n- [Automatic speech recognition task guide](../tasks/asr)\n\n## MCTCTConfig\n\n[[autodoc]] MCTCTConfig\n\n## MCTCTFeatureExtractor\n\n[[autodoc]] MCTCTFeatureExtractor\n    - __call__\n\n## MCTCTProcessor\n\n[[autodoc]] MCTCTProcessor\n    - __call__\n    - from_pretrained\n    - save_pretrained\n    - batch_decode\n    - decode\n\n## MCTCTModel\n\n[[autodoc]] MCTCTModel\n    - forward\n\n## MCTCTForCTC\n\n[[autodoc]] MCTCTForCTC\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: f8a2b86c-f132-4ee2-9d38-a1ccb1bbedba
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/blenderbot-small/", "title": "Blenderbot Small", "tokens": 976, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"f8a2b86c-f132-4ee2-9d38-a1ccb1bbedba\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blenderbot-small/\", \"title\": \"Blenderbot Small\", \"tokens\": 976, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d7c278b6-5cba-4391-a0fb-892b3f8a8821\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blenderbot-small/\", \"title\": \"Blenderbot Small\", \"tokens\": 976, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0207e6e260b55a0ee67552465e35528e5640122fcf7d220500691f6e5a289eeb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"80a941e0-cf7b-4ad4-bd85-c8119b23d2b4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"def884d2e42ca979f1be2a5d3a151172a9cf63fea68255e68ee098c15f7b3528\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2114, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d7c278b6-5cba-4391-a0fb-892b3f8a8821", "doc_id": "d7c278b6-5cba-4391-a0fb-892b3f8a8821", "ref_doc_id": "d7c278b6-5cba-4391-a0fb-892b3f8a8821", "chroma:document": "# Blenderbot Small\n\nNote that [`BlenderbotSmallModel`] and\n[`BlenderbotSmallForConditionalGeneration`] are only used in combination with the checkpoint\n[facebook/blenderbot-90M](https://huggingface.co/facebook/blenderbot-90M). Larger Blenderbot checkpoints should\ninstead be used with [`BlenderbotModel`] and\n[`BlenderbotForConditionalGeneration`]\n\n## Overview\n\nThe Blender chatbot model was proposed in [Recipes for building an open-domain chatbot](https://arxiv.org/pdf/2004.13637.pdf) Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu,\nJing Xu, Myle Ott, Kurt Shuster, Eric M. Smith, Y-Lan Boureau, Jason Weston on 30 Apr 2020.\n\nThe abstract of the paper is the following:\n\n*Building open-domain chatbots is a challenging area for machine learning research. While prior work has shown that\nscaling neural models in the number of parameters and the size of the data they are trained on gives improved results,\nwe show that other ingredients are important for a high-performing chatbot. Good conversation requires a number of\nskills that an expert conversationalist blends in a seamless way: providing engaging talking points and listening to\ntheir partners, and displaying knowledge, empathy and personality appropriately, while maintaining a consistent\npersona. We show that large scale models can learn these skills when given appropriate training data and choice of\ngeneration strategy. We build variants of these recipes with 90M, 2.7B and 9.4B parameter models, and make our models\nand code publicly available. Human evaluations show our best models are superior to existing approaches in multi-turn\ndialogue in terms of engagingness and humanness measurements. We then discuss the limitations of this work by analyzing\nfailure cases of our models.*\n\nThis model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The authors' code can be\nfound [here](https://github.com/facebookresearch/ParlAI).\n\n## Usage tips\n\nBlenderbot Small is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than \nthe left."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 80a941e0-cf7b-4ad4-bd85-c8119b23d2b4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/blenderbot-small/", "title": "Blenderbot Small", "tokens": 976, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"80a941e0-cf7b-4ad4-bd85-c8119b23d2b4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blenderbot-small/\", \"title\": \"Blenderbot Small\", \"tokens\": 976, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d7c278b6-5cba-4391-a0fb-892b3f8a8821\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blenderbot-small/\", \"title\": \"Blenderbot Small\", \"tokens\": 976, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0207e6e260b55a0ee67552465e35528e5640122fcf7d220500691f6e5a289eeb\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f8a2b86c-f132-4ee2-9d38-a1ccb1bbedba\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blenderbot-small/\", \"title\": \"Blenderbot Small\", \"tokens\": 976, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5f0733644d89801d445a45cf0ce03868cb284fae74c5bc9fd61b38edc1538317\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2117, \"end_char_idx\": 3419, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d7c278b6-5cba-4391-a0fb-892b3f8a8821", "doc_id": "d7c278b6-5cba-4391-a0fb-892b3f8a8821", "ref_doc_id": "d7c278b6-5cba-4391-a0fb-892b3f8a8821", "chroma:document": "## Resources\n\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n## BlenderbotSmallConfig\n\n[[autodoc]] BlenderbotSmallConfig\n\n## BlenderbotSmallTokenizer\n\n[[autodoc]] BlenderbotSmallTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## BlenderbotSmallTokenizerFast\n\n[[autodoc]] BlenderbotSmallTokenizerFast\n\n<frameworkcontent>\n<pt>\n\n## BlenderbotSmallModel\n\n[[autodoc]] BlenderbotSmallModel\n    - forward\n\n## BlenderbotSmallForConditionalGeneration\n\n[[autodoc]] BlenderbotSmallForConditionalGeneration\n    - forward\n\n## BlenderbotSmallForCausalLM\n\n[[autodoc]] BlenderbotSmallForCausalLM\n    - forward\n\n</pt>\n<tf>\n\n## TFBlenderbotSmallModel\n\n[[autodoc]] TFBlenderbotSmallModel\n    - call\n\n## TFBlenderbotSmallForConditionalGeneration\n\n[[autodoc]] TFBlenderbotSmallForConditionalGeneration\n    - call\n\n</tf>\n<jax>\n\n## FlaxBlenderbotSmallModel\n\n[[autodoc]] FlaxBlenderbotSmallModel\n    - __call__\n    - encode\n    - decode\n\n## FlaxBlenderbotForConditionalGeneration\n\n[[autodoc]] FlaxBlenderbotSmallForConditionalGeneration\n    - __call__\n    - encode\n    - decode\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: c9066892-716a-4a6f-83f5-cd8ba0af8a71
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vitmatte/", "title": "ViTMatte", "tokens": 784, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"c9066892-716a-4a6f-83f5-cd8ba0af8a71\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vitmatte/\", \"title\": \"ViTMatte\", \"tokens\": 784, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"18f23925-16bb-4965-8f59-19f6c03a07bd\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vitmatte/\", \"title\": \"ViTMatte\", \"tokens\": 784, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8eccfeb8163e92d18315ee8cafd0241a7b52a962252f770d466d91240a97e8ae\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2804, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "18f23925-16bb-4965-8f59-19f6c03a07bd", "doc_id": "18f23925-16bb-4965-8f59-19f6c03a07bd", "ref_doc_id": "18f23925-16bb-4965-8f59-19f6c03a07bd", "chroma:document": "# ViTMatte\n\n## Overview\n\nThe ViTMatte model was proposed in [Boosting Image Matting with Pretrained Plain Vision Transformers](https://arxiv.org/abs/2305.15272) by Jingfeng Yao, Xinggang Wang, Shusheng Yang, Baoyuan Wang.\nViTMatte leverages plain [Vision Transformers](vit) for the task of image matting, which is the process of accurately estimating the foreground object in images and videos.\n\nThe abstract from the paper is the following:\n\n*Recently, plain vision Transformers (ViTs) have shown impressive performance on various computer vision tasks, thanks to their strong modeling capacity and large-scale pretraining. However, they have not yet conquered the problem of image matting. We hypothesize that image matting could also be boosted by ViTs and present a new efficient and robust ViT-based matting system, named ViTMatte. Our method utilizes (i) a hybrid attention mechanism combined with a convolution neck to help ViTs achieve an excellent performance-computation trade-off in matting tasks. (ii) Additionally, we introduce the detail capture module, which just consists of simple lightweight convolutions to complement the detailed information required by matting. To the best of our knowledge, ViTMatte is the first work to unleash the potential of ViT on image matting with concise adaptation. It inherits many superior properties from ViT to matting, including various pretraining strategies, concise architecture design, and flexible inference strategies. We evaluate ViTMatte on Composition-1k and Distinctions-646, the most commonly used benchmark for image matting, our method achieves state-of-the-art performance and outperforms prior matting works by a large margin.*\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/hustvl/ViTMatte).\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/vitmatte_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> ViTMatte high-level overview. Taken from the <a href=\"https://arxiv.org/abs/2305.15272\">original paper.</a> </small>\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with ViTMatte.\n\n- A demo notebook regarding inference with [`VitMatteForImageMatting`], including background replacement, can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/ViTMatte).\n\n<Tip>\n\nThe model expects both the image and trimap (concatenated) as input. Use [`ViTMatteImageProcessor`] for this purpose.\n</Tip>\n\n## VitMatteConfig\n\n[[autodoc]] VitMatteConfig\n\n## VitMatteImageProcessor\n\n[[autodoc]] VitMatteImageProcessor\n    - preprocess\n\n## VitMatteForImageMatting\n\n[[autodoc]] VitMatteForImageMatting\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: eaf28b54-cfe9-4685-a64e-6974176f8a51
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bloom/", "title": "BLOOM", "tokens": 954, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"eaf28b54-cfe9-4685-a64e-6974176f8a51\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bloom/\", \"title\": \"BLOOM\", \"tokens\": 954, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"55f19f9f-902f-4bc1-b764-d6e4c5a18194\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bloom/\", \"title\": \"BLOOM\", \"tokens\": 954, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"50b8542db90a4da2f8f5d6fb589bbc8b694349271a58aa0b817fe99e71ad9f20\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"aea74cf9-f0d4-4c3b-a8b0-391d5f35b3a6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d54478e8ce727dbf775c2dc7b72a0970d59cd293bb6020770cc330ad26a7074e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2486, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "55f19f9f-902f-4bc1-b764-d6e4c5a18194", "doc_id": "55f19f9f-902f-4bc1-b764-d6e4c5a18194", "ref_doc_id": "55f19f9f-902f-4bc1-b764-d6e4c5a18194", "chroma:document": "# BLOOM\n\n## Overview\n\nThe BLOOM model has been proposed with its various versions through the [BigScience Workshop](https://bigscience.huggingface.co/). BigScience is inspired by other open science initiatives where researchers have pooled their time and resources to collectively achieve a higher impact.\nThe architecture of BLOOM is essentially similar to GPT3 (auto-regressive model for next token prediction), but has been trained on 46 different languages and 13 programming languages.\nSeveral smaller versions of the models have been trained on the same dataset. BLOOM is available in the following versions:\n\n- [bloom-560m](https://huggingface.co/bigscience/bloom-560m)\n- [bloom-1b1](https://huggingface.co/bigscience/bloom-1b1)\n- [bloom-1b7](https://huggingface.co/bigscience/bloom-1b7)\n- [bloom-3b](https://huggingface.co/bigscience/bloom-3b)\n- [bloom-7b1](https://huggingface.co/bigscience/bloom-7b1)\n- [bloom](https://huggingface.co/bigscience/bloom) (176B parameters)\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with BLOOM. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-generation\"/>\n\n- [`BloomForCausalLM`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#gpt-2gpt-and-causal-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n\nSee also:\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n\n\n\u26a1\ufe0f Inference\n- A blog on [Optimization story: Bloom inference](https://huggingface.co/blog/bloom-inference-optimization).\n- A blog on [Incredibly Fast BLOOM Inference with DeepSpeed and Accelerate](https://huggingface.co/blog/bloom-inference-pytorch-scripts).\n\n\u2699\ufe0f Training\n- A blog on [The Technology Behind BLOOM Training](https://huggingface.co/blog/bloom-megatron-deepspeed).\n\n## BloomConfig\n\n[[autodoc]] BloomConfig\n    - all\n\n## BloomTokenizerFast\n\n[[autodoc]] BloomTokenizerFast\n    - all"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: aea74cf9-f0d4-4c3b-a8b0-391d5f35b3a6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bloom/", "title": "BLOOM", "tokens": 954, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"aea74cf9-f0d4-4c3b-a8b0-391d5f35b3a6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bloom/\", \"title\": \"BLOOM\", \"tokens\": 954, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"55f19f9f-902f-4bc1-b764-d6e4c5a18194\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bloom/\", \"title\": \"BLOOM\", \"tokens\": 954, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"50b8542db90a4da2f8f5d6fb589bbc8b694349271a58aa0b817fe99e71ad9f20\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"eaf28b54-cfe9-4685-a64e-6974176f8a51\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bloom/\", \"title\": \"BLOOM\", \"tokens\": 954, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0286e482b0504bc7ce29676b7673df36f16006210c17c02059f8a21c57797de3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1992, \"end_char_idx\": 3070, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "55f19f9f-902f-4bc1-b764-d6e4c5a18194", "doc_id": "55f19f9f-902f-4bc1-b764-d6e4c5a18194", "ref_doc_id": "55f19f9f-902f-4bc1-b764-d6e4c5a18194", "chroma:document": "\u26a1\ufe0f Inference\n- A blog on [Optimization story: Bloom inference](https://huggingface.co/blog/bloom-inference-optimization).\n- A blog on [Incredibly Fast BLOOM Inference with DeepSpeed and Accelerate](https://huggingface.co/blog/bloom-inference-pytorch-scripts).\n\n\u2699\ufe0f Training\n- A blog on [The Technology Behind BLOOM Training](https://huggingface.co/blog/bloom-megatron-deepspeed).\n\n## BloomConfig\n\n[[autodoc]] BloomConfig\n    - all\n\n## BloomTokenizerFast\n\n[[autodoc]] BloomTokenizerFast\n    - all\n\n\n<frameworkcontent>\n<pt>\n\n## BloomModel\n\n[[autodoc]] BloomModel\n    - forward\n\n## BloomForCausalLM\n\n[[autodoc]] BloomForCausalLM\n    - forward\n\n## BloomForSequenceClassification\n\n[[autodoc]] BloomForSequenceClassification\n    - forward\n\n## BloomForTokenClassification\n\n[[autodoc]] BloomForTokenClassification\n    - forward\n\n## BloomForQuestionAnswering\n\n[[autodoc]] BloomForQuestionAnswering\n    - forward\n\n</pt>\n<jax>\n\n## FlaxBloomModel\n\n[[autodoc]] FlaxBloomModel\n    - __call__\n\n## FlaxBloomForCausalLM\n\n[[autodoc]] FlaxBloomForCausalLM\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 6496c3c4-d330-43c7-a362-9d5904bf007d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/speech_to_text_2/", "title": "Speech2Text2", "tokens": 1275, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"6496c3c4-d330-43c7-a362-9d5904bf007d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/speech_to_text_2/\", \"title\": \"Speech2Text2\", \"tokens\": 1275, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5de10106-3c02-416b-b708-1e107d095b4b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/speech_to_text_2/\", \"title\": \"Speech2Text2\", \"tokens\": 1275, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"85afe69fffd4bfeb14d8ca2027a11e3929b492a154448e332469a70f002fecca\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a512eb01-14f1-47e5-b0a4-a979fe9bbd19\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"baa69438b0f3770996ef35a1512dfbe14e961adc2c6696d605b33ddd36bad7d4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2755, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5de10106-3c02-416b-b708-1e107d095b4b", "doc_id": "5de10106-3c02-416b-b708-1e107d095b4b", "ref_doc_id": "5de10106-3c02-416b-b708-1e107d095b4b", "chroma:document": "# Speech2Text2\n\n  <Tip warning={true}>\n\n  This model is in maintenance mode only, we don't accept any new PRs changing its code.\n  If you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.\n  You can do so by running the following command: `pip install -U transformers==4.40.2`.\n\n  </Tip>\n\n## Overview\n\nThe Speech2Text2 model is used together with [Wav2Vec2](wav2vec2) for Speech Translation models proposed in\n[Large-Scale Self- and Semi-Supervised Learning for Speech Translation](https://arxiv.org/abs/2104.06678) by\nChanghan Wang, Anne Wu, Juan Pino, Alexei Baevski, Michael Auli, Alexis Conneau.\n\nSpeech2Text2 is a *decoder-only* transformer model that can be used with any speech *encoder-only*, such as\n[Wav2Vec2](wav2vec2) or [HuBERT](hubert) for Speech-to-Text tasks. Please refer to the\n[SpeechEncoderDecoder](speech-encoder-decoder) class on how to combine Speech2Text2 with any speech *encoder-only*\nmodel.\n\nThis model was contributed by [Patrick von Platen](https://huggingface.co/patrickvonplaten).\n\nThe original code can be found [here](https://github.com/pytorch/fairseq/blob/1f7ef9ed1e1061f8c7f88f8b94c7186834398690/fairseq/models/wav2vec/wav2vec2_asr.py#L266).\n\n## Usage tips\n\n- Speech2Text2 achieves state-of-the-art results on the CoVoST Speech Translation dataset. For more information, see\n  the [official models](https://huggingface.co/models?other=speech2text2) .\n- Speech2Text2 is always used within the [SpeechEncoderDecoder](speech-encoder-decoder) framework.\n- Speech2Text2's tokenizer is based on [fastBPE](https://github.com/glample/fastBPE).\n\n## Inference\n\nSpeech2Text2's [`SpeechEncoderDecoderModel`] model accepts raw waveform input values from speech and\nmakes use of [`~generation.GenerationMixin.generate`] to translate the input speech\nautoregressively to the target language.\n\nThe [`Wav2Vec2FeatureExtractor`] class is responsible for preprocessing the input speech and\n[`Speech2Text2Tokenizer`] decodes the generated target tokens to the target string. The\n[`Speech2Text2Processor`] wraps [`Wav2Vec2FeatureExtractor`] and\n[`Speech2Text2Tokenizer`] into a single instance to both extract the input features and decode the\npredicted token ids.\n\n- Step-by-step Speech Translation\n\n```python\n>>> import torch\n>>> from transformers import Speech2Text2Processor, SpeechEncoderDecoderModel\n>>> from datasets import load_dataset\n>>> import soundfile as sf\n\n>>> model = SpeechEncoderDecoderModel.from_pretrained(\"facebook/s2t-wav2vec2-large-en-de\")\n>>> processor = Speech2Text2Processor.from_pretrained(\"facebook/s2t-wav2vec2-large-en-de\")\n\n\n>>> def map_to_array(batch):\n...     speech, _ = sf.read(batch[\"file\"])\n...     batch[\"speech\"] = speech\n...     return batch"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: a512eb01-14f1-47e5-b0a4-a979fe9bbd19
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/speech_to_text_2/", "title": "Speech2Text2", "tokens": 1275, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"a512eb01-14f1-47e5-b0a4-a979fe9bbd19\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/speech_to_text_2/\", \"title\": \"Speech2Text2\", \"tokens\": 1275, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5de10106-3c02-416b-b708-1e107d095b4b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/speech_to_text_2/\", \"title\": \"Speech2Text2\", \"tokens\": 1275, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"85afe69fffd4bfeb14d8ca2027a11e3929b492a154448e332469a70f002fecca\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6496c3c4-d330-43c7-a362-9d5904bf007d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/speech_to_text_2/\", \"title\": \"Speech2Text2\", \"tokens\": 1275, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b0e815a090bb37a244d7937a7ee10faa26863b72f091ca813f8306df368e79cc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2630, \"end_char_idx\": 4308, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5de10106-3c02-416b-b708-1e107d095b4b", "doc_id": "5de10106-3c02-416b-b708-1e107d095b4b", "ref_doc_id": "5de10106-3c02-416b-b708-1e107d095b4b", "chroma:document": ">>> def map_to_array(batch):\n...     speech, _ = sf.read(batch[\"file\"])\n...     batch[\"speech\"] = speech\n...     return batch\n\n\n>>> ds = load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\n>>> ds = ds.map(map_to_array)\n\n>>> inputs = processor(ds[\"speech\"][0], sampling_rate=16_000, return_tensors=\"pt\")\n>>> generated_ids = model.generate(inputs=inputs[\"input_values\"], attention_mask=inputs[\"attention_mask\"])\n\n>>> transcription = processor.batch_decode(generated_ids)\n```\n\n- Speech Translation via Pipelines\n\n  The automatic speech recognition pipeline can also be used to translate speech in just a couple lines of code\n\n```python\n>>> from datasets import load_dataset\n>>> from transformers import pipeline\n\n>>> librispeech_en = load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\n>>> asr = pipeline(\n...     \"automatic-speech-recognition\",\n...     model=\"facebook/s2t-wav2vec2-large-en-de\",\n...     feature_extractor=\"facebook/s2t-wav2vec2-large-en-de\",\n... )\n\n>>> translation_de = asr(librispeech_en[0][\"file\"])\n```\n\nSee [model hub](https://huggingface.co/models?filter=speech2text2) to look for Speech2Text2 checkpoints.\n\n## Resources\n\n- [Causal language modeling task guide](../tasks/language_modeling)\n\n## Speech2Text2Config\n\n[[autodoc]] Speech2Text2Config\n\n## Speech2TextTokenizer\n\n[[autodoc]] Speech2Text2Tokenizer\n    - batch_decode\n    - decode\n    - save_vocabulary\n\n## Speech2Text2Processor\n\n[[autodoc]] Speech2Text2Processor\n    - __call__\n    - from_pretrained\n    - save_pretrained\n    - batch_decode\n    - decode\n\n## Speech2Text2ForCausalLM\n\n[[autodoc]] Speech2Text2ForCausalLM\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 27b87048-1494-4fd5-bb75-c54ee503e3bf
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutxlm/", "title": "LayoutXLM", "tokens": 832, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"27b87048-1494-4fd5-bb75-c54ee503e3bf\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutxlm/\", \"title\": \"LayoutXLM\", \"tokens\": 832, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2d6ac778-8275-4801-9506-1675e47edc86\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutxlm/\", \"title\": \"LayoutXLM\", \"tokens\": 832, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"bbcb3209d19a6bdade5fdb80164c842e3b1b4538fd252784a223416a436cfb04\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2724, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2d6ac778-8275-4801-9506-1675e47edc86", "doc_id": "2d6ac778-8275-4801-9506-1675e47edc86", "ref_doc_id": "2d6ac778-8275-4801-9506-1675e47edc86", "chroma:document": "# LayoutXLM\n\n## Overview\n\nLayoutXLM was proposed in [LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding](https://arxiv.org/abs/2104.08836) by Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha\nZhang, Furu Wei. It's a multilingual extension of the [LayoutLMv2 model](https://arxiv.org/abs/2012.14740) trained\non 53 languages.\n\nThe abstract from the paper is the following:\n\n*Multimodal pre-training with text, layout, and image has achieved SOTA performance for visually-rich document\nunderstanding tasks recently, which demonstrates the great potential for joint learning across different modalities. In\nthis paper, we present LayoutXLM, a multimodal pre-trained model for multilingual document understanding, which aims to\nbridge the language barriers for visually-rich document understanding. To accurately evaluate LayoutXLM, we also\nintroduce a multilingual form understanding benchmark dataset named XFUN, which includes form understanding samples in\n7 languages (Chinese, Japanese, Spanish, French, Italian, German, Portuguese), and key-value pairs are manually labeled\nfor each language. Experiment results show that the LayoutXLM model has significantly outperformed the existing SOTA\ncross-lingual pre-trained models on the XFUN dataset.*\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/microsoft/unilm).\n\n## Usage tips and examples\n\nOne can directly plug in the weights of LayoutXLM into a LayoutLMv2 model, like so:\n\n```python\nfrom transformers import LayoutLMv2Model\n\nmodel = LayoutLMv2Model.from_pretrained(\"microsoft/layoutxlm-base\")\n```\n\nNote that LayoutXLM has its own tokenizer, based on\n[`LayoutXLMTokenizer`]/[`LayoutXLMTokenizerFast`]. You can initialize it as\nfollows:\n\n```python\nfrom transformers import LayoutXLMTokenizer\n\ntokenizer = LayoutXLMTokenizer.from_pretrained(\"microsoft/layoutxlm-base\")\n```\n\nSimilar to LayoutLMv2, you can use [`LayoutXLMProcessor`] (which internally applies\n[`LayoutLMv2ImageProcessor`] and\n[`LayoutXLMTokenizer`]/[`LayoutXLMTokenizerFast`] in sequence) to prepare all\ndata for the model.\n\n<Tip>\n\nAs LayoutXLM's architecture is equivalent to that of LayoutLMv2, one can refer to [LayoutLMv2's documentation page](layoutlmv2) for all tips, code examples and notebooks.\n</Tip>\n\n## LayoutXLMTokenizer\n\n[[autodoc]] LayoutXLMTokenizer\n    - __call__\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## LayoutXLMTokenizerFast\n\n[[autodoc]] LayoutXLMTokenizerFast\n    - __call__\n\n## LayoutXLMProcessor\n\n[[autodoc]] LayoutXLMProcessor\n    - __call__"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: ae414d61-28fa-4fa3-8c80-db96cbff8698
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/megatron-bert/", "title": "MegatronBERT", "tokens": 1498, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"ae414d61-28fa-4fa3-8c80-db96cbff8698\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/megatron-bert/\", \"title\": \"MegatronBERT\", \"tokens\": 1498, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0342c6d1-f187-4c16-ac61-2dca8dd80c83\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/megatron-bert/\", \"title\": \"MegatronBERT\", \"tokens\": 1498, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"7e2ca6cd24c127fa488b93621ecf7824dd206794f4c4c6c996b958479918bb8f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"78fd18ba-b36a-4781-b662-434a54da6283\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"cb6d4636a7e2f8a902f0ba299c3550383d1b19657c4f17f6438af1f26a29487c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2853, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0342c6d1-f187-4c16-ac61-2dca8dd80c83", "doc_id": "0342c6d1-f187-4c16-ac61-2dca8dd80c83", "ref_doc_id": "0342c6d1-f187-4c16-ac61-2dca8dd80c83", "chroma:document": "# MegatronBERT\n\n## Overview\n\nThe MegatronBERT model was proposed in [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model\nParallelism](https://arxiv.org/abs/1909.08053) by Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley,\nJared Casper and Bryan Catanzaro.\n\nThe abstract from the paper is the following:\n\n*Recent work in language modeling demonstrates that training large transformer models advances the state of the art in\nNatural Language Processing applications. However, very large models can be quite difficult to train due to memory\nconstraints. In this work, we present our techniques for training very large transformer models and implement a simple,\nefficient intra-layer model parallel approach that enables training transformer models with billions of parameters. Our\napproach does not require a new compiler or library changes, is orthogonal and complimentary to pipeline model\nparallelism, and can be fully implemented with the insertion of a few communication operations in native PyTorch. We\nillustrate this approach by converging transformer based models up to 8.3 billion parameters using 512 GPUs. We sustain\n15.1 PetaFLOPs across the entire application with 76% scaling efficiency when compared to a strong single GPU baseline\nthat sustains 39 TeraFLOPs, which is 30% of peak FLOPs. To demonstrate that large language models can further advance\nthe state of the art (SOTA), we train an 8.3 billion parameter transformer language model similar to GPT-2 and a 3.9\nbillion parameter model similar to BERT. We show that careful attention to the placement of layer normalization in\nBERT-like models is critical to achieving increased performance as the model size grows. Using the GPT-2 model we\nachieve SOTA results on the WikiText103 (10.8 compared to SOTA perplexity of 15.8) and LAMBADA (66.5% compared to SOTA\naccuracy of 63.2%) datasets. Our BERT model achieves SOTA results on the RACE dataset (90.9% compared to SOTA accuracy\nof 89.4%).*\n\nThis model was contributed by [jdemouth](https://huggingface.co/jdemouth). The original code can be found [here](https://github.com/NVIDIA/Megatron-LM). \nThat repository contains a multi-GPU and multi-node implementation of the Megatron Language models. In particular, \nit contains a hybrid model parallel approach using \"tensor parallel\" and \"pipeline parallel\" techniques.\n\n## Usage tips\n\nWe have provided pretrained [BERT-345M](https://ngc.nvidia.com/catalog/models/nvidia:megatron_bert_345m) checkpoints\nfor use to evaluate or finetuning downstream tasks.\n\nTo access these checkpoints, first [sign up](https://ngc.nvidia.com/signup) for and setup the NVIDIA GPU Cloud (NGC)\nRegistry CLI. Further documentation for downloading models can be found in the [NGC documentation](https://docs.nvidia.com/dgx/ngc-registry-cli-user-guide/index.html#topic_6_4_1)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 78fd18ba-b36a-4781-b662-434a54da6283
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/megatron-bert/", "title": "MegatronBERT", "tokens": 1498, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"78fd18ba-b36a-4781-b662-434a54da6283\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/megatron-bert/\", \"title\": \"MegatronBERT\", \"tokens\": 1498, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0342c6d1-f187-4c16-ac61-2dca8dd80c83\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/megatron-bert/\", \"title\": \"MegatronBERT\", \"tokens\": 1498, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"7e2ca6cd24c127fa488b93621ecf7824dd206794f4c4c6c996b958479918bb8f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ae414d61-28fa-4fa3-8c80-db96cbff8698\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/megatron-bert/\", \"title\": \"MegatronBERT\", \"tokens\": 1498, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"74656ad0849e21725ef3157f1d1f7dd4298489dac1d65d8be8261ec651760fbd\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"22879963-be49-4725-933b-134369eace4d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3cdd11a072a95f91e7e95f426ae5b33469743a147584db1713803c99bf23491c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1337, \"end_char_idx\": 3537, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0342c6d1-f187-4c16-ac61-2dca8dd80c83", "doc_id": "0342c6d1-f187-4c16-ac61-2dca8dd80c83", "ref_doc_id": "0342c6d1-f187-4c16-ac61-2dca8dd80c83", "chroma:document": "To demonstrate that large language models can further advance\nthe state of the art (SOTA), we train an 8.3 billion parameter transformer language model similar to GPT-2 and a 3.9\nbillion parameter model similar to BERT. We show that careful attention to the placement of layer normalization in\nBERT-like models is critical to achieving increased performance as the model size grows. Using the GPT-2 model we\nachieve SOTA results on the WikiText103 (10.8 compared to SOTA perplexity of 15.8) and LAMBADA (66.5% compared to SOTA\naccuracy of 63.2%) datasets. Our BERT model achieves SOTA results on the RACE dataset (90.9% compared to SOTA accuracy\nof 89.4%).*\n\nThis model was contributed by [jdemouth](https://huggingface.co/jdemouth). The original code can be found [here](https://github.com/NVIDIA/Megatron-LM). \nThat repository contains a multi-GPU and multi-node implementation of the Megatron Language models. In particular, \nit contains a hybrid model parallel approach using \"tensor parallel\" and \"pipeline parallel\" techniques.\n\n## Usage tips\n\nWe have provided pretrained [BERT-345M](https://ngc.nvidia.com/catalog/models/nvidia:megatron_bert_345m) checkpoints\nfor use to evaluate or finetuning downstream tasks.\n\nTo access these checkpoints, first [sign up](https://ngc.nvidia.com/signup) for and setup the NVIDIA GPU Cloud (NGC)\nRegistry CLI. Further documentation for downloading models can be found in the [NGC documentation](https://docs.nvidia.com/dgx/ngc-registry-cli-user-guide/index.html#topic_6_4_1).\n\nAlternatively, you can directly download the checkpoints using:\n\nBERT-345M-uncased:\n\n```bash\nwget --content-disposition https://api.ngc.nvidia.com/v2/models/nvidia/megatron_bert_345m/versions/v0.1_uncased/zip\n-O megatron_bert_345m_v0_1_uncased.zip\n```\n\nBERT-345M-cased:\n\n```bash\nwget --content-disposition https://api.ngc.nvidia.com/v2/models/nvidia/megatron_bert_345m/versions/v0.1_cased/zip -O\nmegatron_bert_345m_v0_1_cased.zip\n```\n\nOnce you have obtained the checkpoints from NVIDIA GPU Cloud (NGC), you have to convert them to a format that will\neasily be loaded by Hugging Face Transformers and our port of the BERT code.\n\nThe following commands allow you to do the conversion."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 22879963-be49-4725-933b-134369eace4d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/megatron-bert/", "title": "MegatronBERT", "tokens": 1498, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"22879963-be49-4725-933b-134369eace4d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/megatron-bert/\", \"title\": \"MegatronBERT\", \"tokens\": 1498, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0342c6d1-f187-4c16-ac61-2dca8dd80c83\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/megatron-bert/\", \"title\": \"MegatronBERT\", \"tokens\": 1498, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"7e2ca6cd24c127fa488b93621ecf7824dd206794f4c4c6c996b958479918bb8f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"78fd18ba-b36a-4781-b662-434a54da6283\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/megatron-bert/\", \"title\": \"MegatronBERT\", \"tokens\": 1498, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"4491b69022f6e925515e625467be0384de5900cfced6421af4a263d7a8a3f48b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2071, \"end_char_idx\": 5249, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0342c6d1-f187-4c16-ac61-2dca8dd80c83", "doc_id": "0342c6d1-f187-4c16-ac61-2dca8dd80c83", "ref_doc_id": "0342c6d1-f187-4c16-ac61-2dca8dd80c83", "chroma:document": "The original code can be found [here](https://github.com/NVIDIA/Megatron-LM). \nThat repository contains a multi-GPU and multi-node implementation of the Megatron Language models. In particular, \nit contains a hybrid model parallel approach using \"tensor parallel\" and \"pipeline parallel\" techniques.\n\n## Usage tips\n\nWe have provided pretrained [BERT-345M](https://ngc.nvidia.com/catalog/models/nvidia:megatron_bert_345m) checkpoints\nfor use to evaluate or finetuning downstream tasks.\n\nTo access these checkpoints, first [sign up](https://ngc.nvidia.com/signup) for and setup the NVIDIA GPU Cloud (NGC)\nRegistry CLI. Further documentation for downloading models can be found in the [NGC documentation](https://docs.nvidia.com/dgx/ngc-registry-cli-user-guide/index.html#topic_6_4_1).\n\nAlternatively, you can directly download the checkpoints using:\n\nBERT-345M-uncased:\n\n```bash\nwget --content-disposition https://api.ngc.nvidia.com/v2/models/nvidia/megatron_bert_345m/versions/v0.1_uncased/zip\n-O megatron_bert_345m_v0_1_uncased.zip\n```\n\nBERT-345M-cased:\n\n```bash\nwget --content-disposition https://api.ngc.nvidia.com/v2/models/nvidia/megatron_bert_345m/versions/v0.1_cased/zip -O\nmegatron_bert_345m_v0_1_cased.zip\n```\n\nOnce you have obtained the checkpoints from NVIDIA GPU Cloud (NGC), you have to convert them to a format that will\neasily be loaded by Hugging Face Transformers and our port of the BERT code.\n\nThe following commands allow you to do the conversion. We assume that the folder `models/megatron_bert` contains\n`megatron_bert_345m_v0_1_{cased, uncased}.zip` and that the commands are run from inside that folder:\n\n```bash\npython3 $PATH_TO_TRANSFORMERS/models/megatron_bert/convert_megatron_bert_checkpoint.py megatron_bert_345m_v0_1_uncased.zip\n```\n\n```bash\npython3 $PATH_TO_TRANSFORMERS/models/megatron_bert/convert_megatron_bert_checkpoint.py megatron_bert_345m_v0_1_cased.zip\n```\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## MegatronBertConfig\n\n[[autodoc]] MegatronBertConfig\n\n## MegatronBertModel\n\n[[autodoc]] MegatronBertModel\n    - forward\n\n## MegatronBertForMaskedLM\n\n[[autodoc]] MegatronBertForMaskedLM\n    - forward\n\n## MegatronBertForCausalLM\n\n[[autodoc]] MegatronBertForCausalLM\n    - forward\n\n## MegatronBertForNextSentencePrediction\n\n[[autodoc]] MegatronBertForNextSentencePrediction\n    - forward\n\n## MegatronBertForPreTraining\n\n[[autodoc]] MegatronBertForPreTraining\n    - forward\n\n## MegatronBertForSequenceClassification\n\n[[autodoc]] MegatronBertForSequenceClassification\n    - forward\n\n## MegatronBertForMultipleChoice\n\n[[autodoc]] MegatronBertForMultipleChoice\n    - forward\n\n## MegatronBertForTokenClassification\n\n[[autodoc]] MegatronBertForTokenClassification\n    - forward\n\n## MegatronBertForQuestionAnswering\n\n[[autodoc]] MegatronBertForQuestionAnswering\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 8398a623-a13b-40b1-b195-36f28930f5a2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xlm-prophetnet/", "title": "XLM-ProphetNet", "tokens": 1060, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"8398a623-a13b-40b1-b195-36f28930f5a2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-prophetnet/\", \"title\": \"XLM-ProphetNet\", \"tokens\": 1060, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"574ea399-f653-4292-99e4-a5a18e7fa21e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-prophetnet/\", \"title\": \"XLM-ProphetNet\", \"tokens\": 1060, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"20e7f023af247e8fff7169b5199d9498a307f47c17e2c251fd6553aca2cbb49e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"36f4e08c-9a72-4c0c-b7ac-62191b724d99\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d20385311eda57a0037cf62219d3e6514f0aafad268755f4b951cdd63ddfa2c7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 940, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "574ea399-f653-4292-99e4-a5a18e7fa21e", "doc_id": "574ea399-f653-4292-99e4-a5a18e7fa21e", "ref_doc_id": "574ea399-f653-4292-99e4-a5a18e7fa21e", "chroma:document": "# XLM-ProphetNet\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we don't accept any new PRs changing its code.\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.\nYou can do so by running the following command: `pip install -U transformers==4.40.2`.\n\n</Tip>\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=xprophetnet\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-xprophetnet-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/xprophetnet-large-wiki100-cased-xglue-ntg\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n**DISCLAIMER:** If you see something strange, file a [Github Issue](https://github.com/huggingface/transformers/issues/new?assignees=&labels=&template=bug-report.md&title) and assign\n@patrickvonplaten"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 36f4e08c-9a72-4c0c-b7ac-62191b724d99
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xlm-prophetnet/", "title": "XLM-ProphetNet", "tokens": 1060, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"36f4e08c-9a72-4c0c-b7ac-62191b724d99\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-prophetnet/\", \"title\": \"XLM-ProphetNet\", \"tokens\": 1060, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"574ea399-f653-4292-99e4-a5a18e7fa21e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-prophetnet/\", \"title\": \"XLM-ProphetNet\", \"tokens\": 1060, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"20e7f023af247e8fff7169b5199d9498a307f47c17e2c251fd6553aca2cbb49e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8398a623-a13b-40b1-b195-36f28930f5a2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-prophetnet/\", \"title\": \"XLM-ProphetNet\", \"tokens\": 1060, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"4838dd57243fdc49f453dbcc367d65ec487da11f32512109f26aa2b5bb4d39e4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3442, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "574ea399-f653-4292-99e4-a5a18e7fa21e", "doc_id": "574ea399-f653-4292-99e4-a5a18e7fa21e", "ref_doc_id": "574ea399-f653-4292-99e4-a5a18e7fa21e", "chroma:document": "# XLM-ProphetNet\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we don't accept any new PRs changing its code.\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.\nYou can do so by running the following command: `pip install -U transformers==4.40.2`.\n\n</Tip>\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=xprophetnet\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-xprophetnet-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/xprophetnet-large-wiki100-cased-xglue-ntg\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n**DISCLAIMER:** If you see something strange, file a [Github Issue](https://github.com/huggingface/transformers/issues/new?assignees=&labels=&template=bug-report.md&title) and assign\n@patrickvonplaten\n\n\n## Overview\n\nThe XLM-ProphetNet model was proposed in [ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training,](https://arxiv.org/abs/2001.04063) by Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei\nZhang, Ming Zhou on 13 Jan, 2020.\n\nXLM-ProphetNet is an encoder-decoder model and can predict n-future tokens for \"ngram\" language modeling instead of\njust the next token. Its architecture is identical to ProhpetNet, but the model was trained on the multi-lingual\n\"wiki100\" Wikipedia dump. XLM-ProphetNet's model architecture and pretraining objective is same as ProphetNet, but XLM-ProphetNet was pre-trained on the cross-lingual dataset XGLUE.\n\nThe abstract from the paper is the following:\n\n*In this paper, we present a new sequence-to-sequence pretraining model called ProphetNet, which introduces a novel\nself-supervised objective named future n-gram prediction and the proposed n-stream self-attention mechanism. Instead of\nthe optimization of one-step ahead prediction in traditional sequence-to-sequence model, the ProphetNet is optimized by\nn-step ahead prediction which predicts the next n tokens simultaneously based on previous context tokens at each time\nstep. The future n-gram prediction explicitly encourages the model to plan for the future tokens and prevent\noverfitting on strong local correlations. We pre-train ProphetNet using a base scale dataset (16GB) and a large scale\ndataset (160GB) respectively. Then we conduct experiments on CNN/DailyMail, Gigaword, and SQuAD 1.1 benchmarks for\nabstractive summarization and question generation tasks. Experimental results show that ProphetNet achieves new\nstate-of-the-art results on all these datasets compared to the models using the same scale pretraining corpus.*\n\nThe Authors' code can be found [here](https://github.com/microsoft/ProphetNet).\n\n## Resources\n\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n## XLMProphetNetConfig\n\n[[autodoc]] XLMProphetNetConfig\n\n## XLMProphetNetTokenizer\n\n[[autodoc]] XLMProphetNetTokenizer\n\n## XLMProphetNetModel\n\n[[autodoc]] XLMProphetNetModel\n\n## XLMProphetNetEncoder\n\n[[autodoc]] XLMProphetNetEncoder\n\n## XLMProphetNetDecoder\n\n[[autodoc]] XLMProphetNetDecoder\n\n## XLMProphetNetForConditionalGeneration\n\n[[autodoc]] XLMProphetNetForConditionalGeneration\n\n## XLMProphetNetForCausalLM\n\n[[autodoc]] XLMProphetNetForCausalLM"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 224e2a1c-1ec1-4f29-bab8-a7a760650043
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/open-llama/", "title": "Open-Llama", "tokens": 523, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"224e2a1c-1ec1-4f29-bab8-a7a760650043\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/open-llama/\", \"title\": \"Open-Llama\", \"tokens\": 523, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"520413dd-f5dc-4cc5-bcd4-02146709b826\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/open-llama/\", \"title\": \"Open-Llama\", \"tokens\": 523, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0106fb1aac56c680d183a2aed5b36c45554ec9027ac50b664a6c4172523ab05e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1418, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "520413dd-f5dc-4cc5-bcd4-02146709b826", "doc_id": "520413dd-f5dc-4cc5-bcd4-02146709b826", "ref_doc_id": "520413dd-f5dc-4cc5-bcd4-02146709b826", "chroma:document": "# Open-Llama\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we don't accept any new PRs changing its code.\n\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.31.0.\nYou can do so by running the following command: `pip install -U transformers==4.31.0`.\n\n</Tip>\n\n<Tip warning={true}>\n\nThis model differs from the [OpenLLaMA models](https://huggingface.co/models?search=openllama) on the Hugging Face Hub, which primarily use the [LLaMA](llama) architecture.\n\n</Tip>\n\n## Overview\n\nThe Open-Llama model was proposed in the open source Open-Llama project by community developer s-JoL.\n\nThe model is mainly based on LLaMA with some modifications, incorporating memory-efficient attention from Xformers, stable embedding from Bloom, and shared input-output embedding from PaLM.\nAnd the model is pre-trained on both Chinese and English, which gives it better performance on Chinese language tasks.\n\nThis model was contributed by [s-JoL](https://huggingface.co/s-JoL).\nThe original code was released on GitHub by [s-JoL](https://github.com/s-JoL), but is now removed.\n\n## OpenLlamaConfig\n\n[[autodoc]] OpenLlamaConfig\n\n## OpenLlamaModel\n\n[[autodoc]] OpenLlamaModel\n    - forward\n\n## OpenLlamaForCausalLM\n\n[[autodoc]] OpenLlamaForCausalLM\n    - forward\n\n## OpenLlamaForSequenceClassification\n\n[[autodoc]] OpenLlamaForSequenceClassification\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: d8d5359e-27d1-4eac-92e8-7e02864431e9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/phi/", "title": "Phi", "tokens": 2257, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"d8d5359e-27d1-4eac-92e8-7e02864431e9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi/\", \"title\": \"Phi\", \"tokens\": 2257, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d7bcf86e-63d9-40b6-8228-96e11346af67\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi/\", \"title\": \"Phi\", \"tokens\": 2257, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"7f2cdb1772281b2a1b626df6464a7d73fc394859c8782d62a5d50cb69247beb5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fba8fb8b-1532-4627-925b-d99b26cd929d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"74284ebc51ce0dc77778d818d996671bd4a235357803d538f2f8e275cfd84d97\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2863, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d7bcf86e-63d9-40b6-8228-96e11346af67", "doc_id": "d7bcf86e-63d9-40b6-8228-96e11346af67", "ref_doc_id": "d7bcf86e-63d9-40b6-8228-96e11346af67", "chroma:document": "# Phi\n\n## Overview\n\nThe Phi-1 model was proposed in [Textbooks Are All You Need](https://arxiv.org/abs/2306.11644) by Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio C\u00e9sar Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, S\u00e9bastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee and Yuanzhi Li.\n\nThe Phi-1.5 model was proposed in [Textbooks Are All You Need II: phi-1.5 technical report](https://arxiv.org/abs/2309.05463) by Yuanzhi Li, S\u00e9bastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar and Yin Tat Lee.\n\n### Summary\n\nIn Phi-1 and Phi-1.5 papers, the authors showed how important the quality of the data is in training relative to the model size.\nThey selected high quality \"textbook\" data alongside with synthetically generated data for training their small sized Transformer\nbased model Phi-1 with 1.3B parameters. Despite this small scale, phi-1 attains pass@1 accuracy 50.6% on HumanEval and 55.5% on MBPP.\nThey follow the same strategy for Phi-1.5 and created another 1.3B parameter model with performance on natural language tasks comparable\nto models 5x larger, and surpassing most non-frontier LLMs. Phi-1.5 exhibits many of the traits of much larger LLMs such as the ability\nto \u201cthink step by step\u201d or perform some rudimentary in-context learning.\nWith these two experiments the authors successfully showed the huge impact of quality of training data when training machine learning models.\n\nThe abstract from the Phi-1 paper is the following:\n\n*We introduce phi-1, a new large language model for code, with significantly smaller size than\ncompeting models: phi-1 is a Transformer-based model with 1.3B parameters, trained for 4 days on\n8 A100s, using a selection of \u201ctextbook quality\u201d data from the web (6B tokens) and synthetically\ngenerated textbooks and exercises with GPT-3.5 (1B tokens). Despite this small scale, phi-1 attains\npass@1 accuracy 50.6% on HumanEval and 55.5% on MBPP. It also displays surprising emergent\nproperties compared to phi-1-base, our model before our finetuning stage on a dataset of coding\nexercises, and phi-1-small, a smaller model with 350M parameters trained with the same pipeline as\nphi-1 that still achieves 45% on HumanEval.*\n\nThe abstract from the Phi-1.5 paper is the following:\n\n*We continue the investigation into the power of smaller Transformer-based language models as\ninitiated by TinyStories \u2013 a 10 million parameter model that can produce coherent English \u2013 and\nthe follow-up work on phi-1, a 1.3 billion parameter model with Python coding performance close\nto the state-of-the-art. The latter work proposed to use existing Large Language Models (LLMs) to\ngenerate \u201ctextbook quality\u201d data as a way to enhance the learning process compared to traditional\nweb data."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: fba8fb8b-1532-4627-925b-d99b26cd929d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/phi/", "title": "Phi", "tokens": 2257, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"fba8fb8b-1532-4627-925b-d99b26cd929d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi/\", \"title\": \"Phi\", \"tokens\": 2257, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d7bcf86e-63d9-40b6-8228-96e11346af67\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi/\", \"title\": \"Phi\", \"tokens\": 2257, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"7f2cdb1772281b2a1b626df6464a7d73fc394859c8782d62a5d50cb69247beb5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d8d5359e-27d1-4eac-92e8-7e02864431e9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi/\", \"title\": \"Phi\", \"tokens\": 2257, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"ed3836333ca50d6bfb263ffb78261a87d571fbb38bdb342924f3955ba507784d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"73089be3-8576-4d00-9fe1-e5a2ef833f5f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"dc38f22066d503c6187ba9898e841dd2c7173ed4fa013fdfe4680c2fb4924e86\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1251, \"end_char_idx\": 4372, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d7bcf86e-63d9-40b6-8228-96e11346af67", "doc_id": "d7bcf86e-63d9-40b6-8228-96e11346af67", "ref_doc_id": "d7bcf86e-63d9-40b6-8228-96e11346af67", "chroma:document": "Phi-1.5 exhibits many of the traits of much larger LLMs such as the ability\nto \u201cthink step by step\u201d or perform some rudimentary in-context learning.\nWith these two experiments the authors successfully showed the huge impact of quality of training data when training machine learning models.\n\nThe abstract from the Phi-1 paper is the following:\n\n*We introduce phi-1, a new large language model for code, with significantly smaller size than\ncompeting models: phi-1 is a Transformer-based model with 1.3B parameters, trained for 4 days on\n8 A100s, using a selection of \u201ctextbook quality\u201d data from the web (6B tokens) and synthetically\ngenerated textbooks and exercises with GPT-3.5 (1B tokens). Despite this small scale, phi-1 attains\npass@1 accuracy 50.6% on HumanEval and 55.5% on MBPP. It also displays surprising emergent\nproperties compared to phi-1-base, our model before our finetuning stage on a dataset of coding\nexercises, and phi-1-small, a smaller model with 350M parameters trained with the same pipeline as\nphi-1 that still achieves 45% on HumanEval.*\n\nThe abstract from the Phi-1.5 paper is the following:\n\n*We continue the investigation into the power of smaller Transformer-based language models as\ninitiated by TinyStories \u2013 a 10 million parameter model that can produce coherent English \u2013 and\nthe follow-up work on phi-1, a 1.3 billion parameter model with Python coding performance close\nto the state-of-the-art. The latter work proposed to use existing Large Language Models (LLMs) to\ngenerate \u201ctextbook quality\u201d data as a way to enhance the learning process compared to traditional\nweb data. We follow the \u201cTextbooks Are All You Need\u201d approach, focusing this time on common\nsense reasoning in natural language, and create a new 1.3 billion parameter model named phi-1.5,\nwith performance on natural language tasks comparable to models 5x larger, and surpassing most\nnon-frontier LLMs on more complex reasoning tasks such as grade-school mathematics and basic\ncoding. More generally, phi-1.5 exhibits many of the traits of much larger LLMs, both good \u2013such\nas the ability to \u201cthink step by step\u201d or perform some rudimentary in-context learning\u2013 and bad,\nincluding hallucinations and the potential for toxic and biased generations \u2013encouragingly though, we\nare seeing improvement on that front thanks to the absence of web data. We open-source phi-1.5 to\npromote further research on these urgent topics.*\n\nThis model was contributed by [Susnato Dhar](https://huggingface.co/susnato).\n\nThe original code for Phi-1, Phi-1.5 and Phi-2 can be found [here](https://huggingface.co/microsoft/phi-1), [here](https://huggingface.co/microsoft/phi-1_5) and [here](https://huggingface.co/microsoft/phi-2), respectively.\n\n## Usage tips\n\n- This model is quite similar to `Llama` with the main difference in [`PhiDecoderLayer`], where they used [`PhiAttention`] and [`PhiMLP`] layers in parallel configuration.\n- The tokenizer used for this model is identical to the [`CodeGenTokenizer`].\n\n## How to use Phi-2\n\n<Tip warning={true}>\n\nPhi-2 has been integrated in the development version (4.37.0.dev) of `transformers`."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 73089be3-8576-4d00-9fe1-e5a2ef833f5f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/phi/", "title": "Phi", "tokens": 2257, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"73089be3-8576-4d00-9fe1-e5a2ef833f5f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi/\", \"title\": \"Phi\", \"tokens\": 2257, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d7bcf86e-63d9-40b6-8228-96e11346af67\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi/\", \"title\": \"Phi\", \"tokens\": 2257, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"7f2cdb1772281b2a1b626df6464a7d73fc394859c8782d62a5d50cb69247beb5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"fba8fb8b-1532-4627-925b-d99b26cd929d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi/\", \"title\": \"Phi\", \"tokens\": 2257, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"29070afd3d7189797b5573393230d237c7c9424b6d0030dc8345a085e7e05802\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"25b97187-c3f6-46a7-8869-dc12a8e6ba4a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"638d20c204657d5239234b3165a3b06de54a47d483c94110b44c8a4dbbdffdb7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2864, \"end_char_idx\": 5975, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d7bcf86e-63d9-40b6-8228-96e11346af67", "doc_id": "d7bcf86e-63d9-40b6-8228-96e11346af67", "ref_doc_id": "d7bcf86e-63d9-40b6-8228-96e11346af67", "chroma:document": "We follow the \u201cTextbooks Are All You Need\u201d approach, focusing this time on common\nsense reasoning in natural language, and create a new 1.3 billion parameter model named phi-1.5,\nwith performance on natural language tasks comparable to models 5x larger, and surpassing most\nnon-frontier LLMs on more complex reasoning tasks such as grade-school mathematics and basic\ncoding. More generally, phi-1.5 exhibits many of the traits of much larger LLMs, both good \u2013such\nas the ability to \u201cthink step by step\u201d or perform some rudimentary in-context learning\u2013 and bad,\nincluding hallucinations and the potential for toxic and biased generations \u2013encouragingly though, we\nare seeing improvement on that front thanks to the absence of web data. We open-source phi-1.5 to\npromote further research on these urgent topics.*\n\nThis model was contributed by [Susnato Dhar](https://huggingface.co/susnato).\n\nThe original code for Phi-1, Phi-1.5 and Phi-2 can be found [here](https://huggingface.co/microsoft/phi-1), [here](https://huggingface.co/microsoft/phi-1_5) and [here](https://huggingface.co/microsoft/phi-2), respectively.\n\n## Usage tips\n\n- This model is quite similar to `Llama` with the main difference in [`PhiDecoderLayer`], where they used [`PhiAttention`] and [`PhiMLP`] layers in parallel configuration.\n- The tokenizer used for this model is identical to the [`CodeGenTokenizer`].\n\n## How to use Phi-2\n\n<Tip warning={true}>\n\nPhi-2 has been integrated in the development version (4.37.0.dev) of `transformers`. Until the official version is released through `pip`, ensure that you are doing one of the following:\n\n* When loading the model, ensure that `trust_remote_code=True` is passed as an argument of the `from_pretrained()` function.\n\n* Update your local `transformers` to the development version: `pip uninstall -y transformers && pip install git+https://github.com/huggingface/transformers`. The previous command is an alternative to cloning and installing from the source.\n\n</Tip>\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"microsoft/phi-2\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"microsoft/phi-2\")\n\n>>> inputs = tokenizer('Can you help me write a formal email to a potential business partner proposing a joint venture?', return_tensors=\"pt\", return_attention_mask=False)\n\n>>> outputs = model.generate(**inputs, max_length=30)\n>>> text = tokenizer.batch_decode(outputs)[0]\n>>> print(text)\nCan you help me write a formal email to a potential business partner proposing a joint venture?\nInput: Company A: ABC Inc.\nCompany B\n```\n\n### Example :\n\n```python\n>>> from transformers import PhiForCausalLM, AutoTokenizer\n\n>>> # define the model and tokenizer.\n>>> model = PhiForCausalLM.from_pretrained(\"microsoft/phi-1_5\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"microsoft/phi-1_5\")\n\n>>> # feel free to change the prompt to your liking.\n>>> prompt = \"If I were an AI that had just achieved\"\n\n>>> # apply the tokenizer.\n>>> tokens = tokenizer(prompt, return_tensors=\"pt\")\n\n>>> # use the model to generate new tokens."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 25b97187-c3f6-46a7-8869-dc12a8e6ba4a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/phi/", "title": "Phi", "tokens": 2257, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"25b97187-c3f6-46a7-8869-dc12a8e6ba4a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi/\", \"title\": \"Phi\", \"tokens\": 2257, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d7bcf86e-63d9-40b6-8228-96e11346af67\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi/\", \"title\": \"Phi\", \"tokens\": 2257, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"7f2cdb1772281b2a1b626df6464a7d73fc394859c8782d62a5d50cb69247beb5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"73089be3-8576-4d00-9fe1-e5a2ef833f5f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi/\", \"title\": \"Phi\", \"tokens\": 2257, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a694c90d8040728187239da80aa012bf8a85a26970f5e09ce879b2bf8fb3619b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3f813090-440c-41b7-8c79-73ef3443be49\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7b1844cd0723baaef3ce814dcfa12aaf633198619363a9910fb7bb4dd8ba8548\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4373, \"end_char_idx\": 7378, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d7bcf86e-63d9-40b6-8228-96e11346af67", "doc_id": "d7bcf86e-63d9-40b6-8228-96e11346af67", "ref_doc_id": "d7bcf86e-63d9-40b6-8228-96e11346af67", "chroma:document": "Until the official version is released through `pip`, ensure that you are doing one of the following:\n\n* When loading the model, ensure that `trust_remote_code=True` is passed as an argument of the `from_pretrained()` function.\n\n* Update your local `transformers` to the development version: `pip uninstall -y transformers && pip install git+https://github.com/huggingface/transformers`. The previous command is an alternative to cloning and installing from the source.\n\n</Tip>\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"microsoft/phi-2\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"microsoft/phi-2\")\n\n>>> inputs = tokenizer('Can you help me write a formal email to a potential business partner proposing a joint venture?', return_tensors=\"pt\", return_attention_mask=False)\n\n>>> outputs = model.generate(**inputs, max_length=30)\n>>> text = tokenizer.batch_decode(outputs)[0]\n>>> print(text)\nCan you help me write a formal email to a potential business partner proposing a joint venture?\nInput: Company A: ABC Inc.\nCompany B\n```\n\n### Example :\n\n```python\n>>> from transformers import PhiForCausalLM, AutoTokenizer\n\n>>> # define the model and tokenizer.\n>>> model = PhiForCausalLM.from_pretrained(\"microsoft/phi-1_5\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"microsoft/phi-1_5\")\n\n>>> # feel free to change the prompt to your liking.\n>>> prompt = \"If I were an AI that had just achieved\"\n\n>>> # apply the tokenizer.\n>>> tokens = tokenizer(prompt, return_tensors=\"pt\")\n\n>>> # use the model to generate new tokens.\n>>> generated_output = model.generate(**tokens, use_cache=True, max_new_tokens=10)\n\n>>> tokenizer.batch_decode(generated_output)[0]\n'If I were an AI that had just achieved a breakthrough in machine learning, I would be thrilled'\n```\n\n## Combining Phi and Flash Attention 2\n\nFirst, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature.\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nMake also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of flash-attn repository. Make also sure to load your model in half-precision (e.g. `torch.float16``)\n\nTo load and run a model using Flash Attention 2, refer to the snippet below:\n\n```python\n>>> import torch\n>>> from transformers import PhiForCausalLM, AutoTokenizer\n\n>>> # define the model and tokenizer and push the model and tokens to the GPU.\n>>> model = PhiForCausalLM.from_pretrained(\"microsoft/phi-1_5\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\").to(\"cuda\")  # doctest: +SKIP\n>>> tokenizer = AutoTokenizer.from_pretrained(\"microsoft/phi-1_5\")\n\n>>> # feel free to change the prompt to your liking.\n>>> prompt = \"If I were an AI that had just achieved\"\n\n>>> # apply the tokenizer.\n>>> tokens = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda\")\n\n>>> # use the model to generate new tokens."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 3f813090-440c-41b7-8c79-73ef3443be49
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/phi/", "title": "Phi", "tokens": 2257, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"3f813090-440c-41b7-8c79-73ef3443be49\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi/\", \"title\": \"Phi\", \"tokens\": 2257, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d7bcf86e-63d9-40b6-8228-96e11346af67\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi/\", \"title\": \"Phi\", \"tokens\": 2257, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"7f2cdb1772281b2a1b626df6464a7d73fc394859c8782d62a5d50cb69247beb5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"25b97187-c3f6-46a7-8869-dc12a8e6ba4a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi/\", \"title\": \"Phi\", \"tokens\": 2257, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"62b1380a926374b91ba2b6f1c36a9e7a1c61d2522078762deaff1fafa95e6301\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5797, \"end_char_idx\": 8444, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d7bcf86e-63d9-40b6-8228-96e11346af67", "doc_id": "d7bcf86e-63d9-40b6-8228-96e11346af67", "ref_doc_id": "d7bcf86e-63d9-40b6-8228-96e11346af67", "chroma:document": ">>> prompt = \"If I were an AI that had just achieved\"\n\n>>> # apply the tokenizer.\n>>> tokens = tokenizer(prompt, return_tensors=\"pt\")\n\n>>> # use the model to generate new tokens.\n>>> generated_output = model.generate(**tokens, use_cache=True, max_new_tokens=10)\n\n>>> tokenizer.batch_decode(generated_output)[0]\n'If I were an AI that had just achieved a breakthrough in machine learning, I would be thrilled'\n```\n\n## Combining Phi and Flash Attention 2\n\nFirst, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature.\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nMake also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of flash-attn repository. Make also sure to load your model in half-precision (e.g. `torch.float16``)\n\nTo load and run a model using Flash Attention 2, refer to the snippet below:\n\n```python\n>>> import torch\n>>> from transformers import PhiForCausalLM, AutoTokenizer\n\n>>> # define the model and tokenizer and push the model and tokens to the GPU.\n>>> model = PhiForCausalLM.from_pretrained(\"microsoft/phi-1_5\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\").to(\"cuda\")  # doctest: +SKIP\n>>> tokenizer = AutoTokenizer.from_pretrained(\"microsoft/phi-1_5\")\n\n>>> # feel free to change the prompt to your liking.\n>>> prompt = \"If I were an AI that had just achieved\"\n\n>>> # apply the tokenizer.\n>>> tokens = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda\")\n\n>>> # use the model to generate new tokens.\n>>> generated_output = model.generate(**tokens, use_cache=True, max_new_tokens=10)  # doctest: +SKIP\n\n>>> tokenizer.batch_decode(generated_output)[0]  # doctest: +SKIP\n'If I were an AI that had just achieved a breakthrough in machine learning, I would be thrilled'\n```\n\n### Expected speedups\n\nBelow is an expected speedup diagram that compares pure inference time between the native implementation in transformers using `microsoft/phi-1` checkpoint and the Flash Attention 2 version of the model using a sequence length of 2048.\n\n<div style=\"text-align: center\">\n<img src=\"https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/phi_1_speedup_plot.jpg\">\n</div>\n\n## PhiConfig\n\n[[autodoc]] PhiConfig\n\n<frameworkcontent>\n<pt>\n\n## PhiModel\n\n[[autodoc]] PhiModel\n    - forward\n\n## PhiForCausalLM\n\n[[autodoc]] PhiForCausalLM\n    - forward\n    - generate\n\n## PhiForSequenceClassification\n\n[[autodoc]] PhiForSequenceClassification\n    - forward\n\n## PhiForTokenClassification\n\n[[autodoc]] PhiForTokenClassification\n    - forward\n\n</pt>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: df633edc-16fb-4d84-9cf4-0bd0f0444885
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/auto/", "title": "Auto Classes", "tokens": 2236, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"df633edc-16fb-4d84-9cf4-0bd0f0444885\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/auto/\", \"title\": \"Auto Classes\", \"tokens\": 2236, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c7afa274-5d52-4e69-a47a-785ed276650c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/auto/\", \"title\": \"Auto Classes\", \"tokens\": 2236, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"92953ab82047ace848d6e82cf3ce41c5cef8da0d6cb71cfea3db6cfb938c4ada\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cb2944b7-3216-4f4d-bb9e-ecbe29ae2c91\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b480bec7f264a3e1595308ceeaf69c432724d85b3d574536faab7df3c82178d7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2618, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c7afa274-5d52-4e69-a47a-785ed276650c", "doc_id": "c7afa274-5d52-4e69-a47a-785ed276650c", "ref_doc_id": "c7afa274-5d52-4e69-a47a-785ed276650c", "chroma:document": "# Auto Classes\n\nIn many cases, the architecture you want to use can be guessed from the name or the path of the pretrained model you\nare supplying to the `from_pretrained()` method. AutoClasses are here to do this job for you so that you\nautomatically retrieve the relevant model given the name/path to the pretrained weights/config/vocabulary.\n\nInstantiating one of [`AutoConfig`], [`AutoModel`], and\n[`AutoTokenizer`] will directly create a class of the relevant architecture. For instance\n\n\n```python\nmodel = AutoModel.from_pretrained(\"google-bert/bert-base-cased\")\n```\n\nwill create a model that is an instance of [`BertModel`].\n\nThere is one class of `AutoModel` for each task, and for each backend (PyTorch, TensorFlow, or Flax).\n\n## Extending the Auto Classes\n\nEach of the auto classes has a method to be extended with your custom classes. For instance, if you have defined a\ncustom class of model `NewModel`, make sure you have a `NewModelConfig` then you can add those to the auto\nclasses like this:\n\n```python\nfrom transformers import AutoConfig, AutoModel\n\nAutoConfig.register(\"new-model\", NewModelConfig)\nAutoModel.register(NewModelConfig, NewModel)\n```\n\nYou will then be able to use the auto classes like you would usually do!\n\n<Tip warning={true}>\n\nIf your `NewModelConfig` is a subclass of [`~transformers.PretrainedConfig`], make sure its\n`model_type` attribute is set to the same key you use when registering the config (here `\"new-model\"`).\n\nLikewise, if your `NewModel` is a subclass of [`PreTrainedModel`], make sure its\n`config_class` attribute is set to the same class you use when registering the model (here\n`NewModelConfig`).\n\n</Tip>\n\n## AutoConfig\n\n[[autodoc]] AutoConfig\n\n## AutoTokenizer\n\n[[autodoc]] AutoTokenizer\n\n## AutoFeatureExtractor\n\n[[autodoc]] AutoFeatureExtractor\n\n## AutoImageProcessor\n\n[[autodoc]] AutoImageProcessor\n\n## AutoProcessor\n\n[[autodoc]] AutoProcessor\n\n## Generic model classes\n\nThe following auto classes are available for instantiating a base model class without a specific head.\n\n### AutoModel\n\n[[autodoc]] AutoModel\n\n### TFAutoModel\n\n[[autodoc]] TFAutoModel\n\n### FlaxAutoModel\n\n[[autodoc]] FlaxAutoModel\n\n## Generic pretraining classes\n\nThe following auto classes are available for instantiating a model with a pretraining head.\n\n### AutoModelForPreTraining\n\n[[autodoc]] AutoModelForPreTraining\n\n### TFAutoModelForPreTraining\n\n[[autodoc]] TFAutoModelForPreTraining\n\n### FlaxAutoModelForPreTraining\n\n[[autodoc]] FlaxAutoModelForPreTraining\n\n## Natural Language Processing\n\nThe following auto classes are available for the following natural language processing tasks."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: cb2944b7-3216-4f4d-bb9e-ecbe29ae2c91
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/auto/", "title": "Auto Classes", "tokens": 2236, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"cb2944b7-3216-4f4d-bb9e-ecbe29ae2c91\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/auto/\", \"title\": \"Auto Classes\", \"tokens\": 2236, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c7afa274-5d52-4e69-a47a-785ed276650c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/auto/\", \"title\": \"Auto Classes\", \"tokens\": 2236, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"92953ab82047ace848d6e82cf3ce41c5cef8da0d6cb71cfea3db6cfb938c4ada\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"df633edc-16fb-4d84-9cf4-0bd0f0444885\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/auto/\", \"title\": \"Auto Classes\", \"tokens\": 2236, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"34459354cd0f73e30087e49dcfd40fc6df41d989d5376178a85ea67896c129b8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9bdc1aea-2675-4072-8007-29d16561bb0e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"30ae3c78bd44c2ef7cfe2313accaa334224d87f03ecaadd1f7793697758e7a77\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1240, \"end_char_idx\": 4875, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c7afa274-5d52-4e69-a47a-785ed276650c", "doc_id": "c7afa274-5d52-4e69-a47a-785ed276650c", "ref_doc_id": "c7afa274-5d52-4e69-a47a-785ed276650c", "chroma:document": "<Tip warning={true}>\n\nIf your `NewModelConfig` is a subclass of [`~transformers.PretrainedConfig`], make sure its\n`model_type` attribute is set to the same key you use when registering the config (here `\"new-model\"`).\n\nLikewise, if your `NewModel` is a subclass of [`PreTrainedModel`], make sure its\n`config_class` attribute is set to the same class you use when registering the model (here\n`NewModelConfig`).\n\n</Tip>\n\n## AutoConfig\n\n[[autodoc]] AutoConfig\n\n## AutoTokenizer\n\n[[autodoc]] AutoTokenizer\n\n## AutoFeatureExtractor\n\n[[autodoc]] AutoFeatureExtractor\n\n## AutoImageProcessor\n\n[[autodoc]] AutoImageProcessor\n\n## AutoProcessor\n\n[[autodoc]] AutoProcessor\n\n## Generic model classes\n\nThe following auto classes are available for instantiating a base model class without a specific head.\n\n### AutoModel\n\n[[autodoc]] AutoModel\n\n### TFAutoModel\n\n[[autodoc]] TFAutoModel\n\n### FlaxAutoModel\n\n[[autodoc]] FlaxAutoModel\n\n## Generic pretraining classes\n\nThe following auto classes are available for instantiating a model with a pretraining head.\n\n### AutoModelForPreTraining\n\n[[autodoc]] AutoModelForPreTraining\n\n### TFAutoModelForPreTraining\n\n[[autodoc]] TFAutoModelForPreTraining\n\n### FlaxAutoModelForPreTraining\n\n[[autodoc]] FlaxAutoModelForPreTraining\n\n## Natural Language Processing\n\nThe following auto classes are available for the following natural language processing tasks.\n\n### AutoModelForCausalLM\n\n[[autodoc]] AutoModelForCausalLM\n\n### TFAutoModelForCausalLM\n\n[[autodoc]] TFAutoModelForCausalLM\n\n### FlaxAutoModelForCausalLM\n\n[[autodoc]] FlaxAutoModelForCausalLM\n\n### AutoModelForMaskedLM\n\n[[autodoc]] AutoModelForMaskedLM\n\n### TFAutoModelForMaskedLM\n\n[[autodoc]] TFAutoModelForMaskedLM\n\n### FlaxAutoModelForMaskedLM\n\n[[autodoc]] FlaxAutoModelForMaskedLM\n\n### AutoModelForMaskGeneration\n\n[[autodoc]] AutoModelForMaskGeneration\n\n### TFAutoModelForMaskGeneration\n\n[[autodoc]] TFAutoModelForMaskGeneration\n\n### AutoModelForSeq2SeqLM\n\n[[autodoc]] AutoModelForSeq2SeqLM\n\n### TFAutoModelForSeq2SeqLM\n\n[[autodoc]] TFAutoModelForSeq2SeqLM\n\n### FlaxAutoModelForSeq2SeqLM\n\n[[autodoc]] FlaxAutoModelForSeq2SeqLM\n\n### AutoModelForSequenceClassification\n\n[[autodoc]] AutoModelForSequenceClassification\n\n### TFAutoModelForSequenceClassification\n\n[[autodoc]] TFAutoModelForSequenceClassification\n\n### FlaxAutoModelForSequenceClassification\n\n[[autodoc]] FlaxAutoModelForSequenceClassification\n\n### AutoModelForMultipleChoice\n\n[[autodoc]] AutoModelForMultipleChoice\n\n### TFAutoModelForMultipleChoice\n\n[[autodoc]] TFAutoModelForMultipleChoice\n\n### FlaxAutoModelForMultipleChoice\n\n[[autodoc]] FlaxAutoModelForMultipleChoice\n\n### AutoModelForNextSentencePrediction\n\n[[autodoc]] AutoModelForNextSentencePrediction\n\n### TFAutoModelForNextSentencePrediction\n\n[[autodoc]] TFAutoModelForNextSentencePrediction\n\n### FlaxAutoModelForNextSentencePrediction\n\n[[autodoc]] FlaxAutoModelForNextSentencePrediction\n\n### AutoModelForTokenClassification\n\n[[autodoc]] AutoModelForTokenClassification\n\n### TFAutoModelForTokenClassification\n\n[[autodoc]] TFAutoModelForTokenClassification\n\n### FlaxAutoModelForTokenClassification\n\n[[autodoc]] FlaxAutoModelForTokenClassification\n\n### AutoModelForQuestionAnswering\n\n[[autodoc]] AutoModelForQuestionAnswering\n\n### TFAutoModelForQuestionAnswering\n\n[[autodoc]] TFAutoModelForQuestionAnswering\n\n### FlaxAutoModelForQuestionAnswering\n\n[[autodoc]] FlaxAutoModelForQuestionAnswering\n\n### AutoModelForTextEncoding\n\n[[autodoc]] AutoModelForTextEncoding\n\n### TFAutoModelForTextEncoding\n\n[[autodoc]] TFAutoModelForTextEncoding\n\n## Computer vision\n\nThe following auto classes are available for the following computer vision tasks."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 9bdc1aea-2675-4072-8007-29d16561bb0e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/auto/", "title": "Auto Classes", "tokens": 2236, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"9bdc1aea-2675-4072-8007-29d16561bb0e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/auto/\", \"title\": \"Auto Classes\", \"tokens\": 2236, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c7afa274-5d52-4e69-a47a-785ed276650c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/auto/\", \"title\": \"Auto Classes\", \"tokens\": 2236, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"92953ab82047ace848d6e82cf3ce41c5cef8da0d6cb71cfea3db6cfb938c4ada\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cb2944b7-3216-4f4d-bb9e-ecbe29ae2c91\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/auto/\", \"title\": \"Auto Classes\", \"tokens\": 2236, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b500cb4b4dc8e4fca3a7e680c218ea4bc2bad86972196a534fcd6df2447e7620\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ab8997a4-6b6b-442b-93f8-56916d21f01c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"483b737e38141b09087203cb31518ffd892e9da53950bfdd4658aacf7413e188\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4877, \"end_char_idx\": 7318, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c7afa274-5d52-4e69-a47a-785ed276650c", "doc_id": "c7afa274-5d52-4e69-a47a-785ed276650c", "ref_doc_id": "c7afa274-5d52-4e69-a47a-785ed276650c", "chroma:document": "### AutoModelForDepthEstimation\n\n[[autodoc]] AutoModelForDepthEstimation\n\n### AutoModelForImageClassification\n\n[[autodoc]] AutoModelForImageClassification\n\n### TFAutoModelForImageClassification\n\n[[autodoc]] TFAutoModelForImageClassification\n\n### FlaxAutoModelForImageClassification\n\n[[autodoc]] FlaxAutoModelForImageClassification\n\n### AutoModelForVideoClassification\n\n[[autodoc]] AutoModelForVideoClassification\n\n### AutoModelForKeypointDetection\n\n[[autodoc]] AutoModelForKeypointDetection\n\n### AutoModelForMaskedImageModeling\n\n[[autodoc]] AutoModelForMaskedImageModeling\n\n### TFAutoModelForMaskedImageModeling\n\n[[autodoc]] TFAutoModelForMaskedImageModeling\n\n### AutoModelForObjectDetection\n\n[[autodoc]] AutoModelForObjectDetection\n\n### AutoModelForImageSegmentation\n\n[[autodoc]] AutoModelForImageSegmentation\n\n### AutoModelForImageToImage\n\n[[autodoc]] AutoModelForImageToImage\n\n### AutoModelForSemanticSegmentation\n\n[[autodoc]] AutoModelForSemanticSegmentation\n\n### TFAutoModelForSemanticSegmentation\n\n[[autodoc]] TFAutoModelForSemanticSegmentation\n\n### AutoModelForInstanceSegmentation\n\n[[autodoc]] AutoModelForInstanceSegmentation\n\n### AutoModelForUniversalSegmentation\n\n[[autodoc]] AutoModelForUniversalSegmentation\n\n### AutoModelForZeroShotImageClassification\n\n[[autodoc]] AutoModelForZeroShotImageClassification\n\n### TFAutoModelForZeroShotImageClassification\n\n[[autodoc]] TFAutoModelForZeroShotImageClassification\n\n### AutoModelForZeroShotObjectDetection\n\n[[autodoc]] AutoModelForZeroShotObjectDetection\n\n## Audio\n\nThe following auto classes are available for the following audio tasks.\n\n### AutoModelForAudioClassification\n\n[[autodoc]] AutoModelForAudioClassification\n\n### AutoModelForAudioFrameClassification\n\n[[autodoc]] TFAutoModelForAudioClassification\n\n### TFAutoModelForAudioFrameClassification\n\n[[autodoc]] AutoModelForAudioFrameClassification\n\n### AutoModelForCTC\n\n[[autodoc]] AutoModelForCTC\n\n### AutoModelForSpeechSeq2Seq\n\n[[autodoc]] AutoModelForSpeechSeq2Seq\n\n### TFAutoModelForSpeechSeq2Seq\n\n[[autodoc]] TFAutoModelForSpeechSeq2Seq\n\n### FlaxAutoModelForSpeechSeq2Seq\n\n[[autodoc]] FlaxAutoModelForSpeechSeq2Seq\n\n### AutoModelForAudioXVector\n\n[[autodoc]] AutoModelForAudioXVector\n\n### AutoModelForTextToSpectrogram\n\n[[autodoc]] AutoModelForTextToSpectrogram\n\n### AutoModelForTextToWaveform\n\n[[autodoc]] AutoModelForTextToWaveform\n\n## Multimodal\n\nThe following auto classes are available for the following multimodal tasks."}


