# embeddings_queue

created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 9e65cc41-a3ea-4c12-8261-cf109238e920
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/jamba/", "title": "Jamba", "tokens": 1474, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"9e65cc41-a3ea-4c12-8261-cf109238e920\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jamba/\", \"title\": \"Jamba\", \"tokens\": 1474, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ba4d23dc-0291-4ff1-a039-151d6d8c1f76\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jamba/\", \"title\": \"Jamba\", \"tokens\": 1474, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"2f1de855611fd39ebefcc95df232dc471fd6837d9eb4711f7fb8c8d60d5887c9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d6d13c29-5f80-46f5-bade-bec899c2cdbc\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5f6d9d44d2a0a2dc8f7dff3d4017a86ae4e29dfd3a5d2a5baba17cb5056c4053\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3224, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ba4d23dc-0291-4ff1-a039-151d6d8c1f76", "doc_id": "ba4d23dc-0291-4ff1-a039-151d6d8c1f76", "ref_doc_id": "ba4d23dc-0291-4ff1-a039-151d6d8c1f76", "chroma:document": "# Jamba\n\n## Overview\n\nJamba is a state-of-the-art, hybrid SSM-Transformer LLM. It is the first production-scale Mamba implementation, which opens up interesting research and application opportunities. While this initial experimentation shows encouraging gains, we expect these to be further enhanced with future optimizations and explorations.\n\nFor full details of this model please read the [release blog post](https://www.ai21.com/blog/announcing-jamba).\n\n### Model Details\n\nJamba is a pretrained, mixture-of-experts (MoE) generative text model, with 12B active parameters and an overall of 52B parameters across all experts. It supports a 256K context length, and can fit up to 140K tokens on a single 80GB GPU.\n\nAs depicted in the diagram below, Jamba's architecture features a blocks-and-layers approach that allows Jamba to successfully integrate Transformer and Mamba architectures altogether. Each Jamba block contains either an attention or a Mamba layer, followed by a multi-layer perceptron (MLP), producing an overall ratio of one Transformer layer out of every eight total layers.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/jamba_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n## Usage\n\n### Presequities\n\nJamba requires you use `transformers` version 4.39.0 or higher:\n```bash\npip install transformers>=4.39.0\n```\n\nIn order to run optimized Mamba implementations, you first need to install `mamba-ssm` and `causal-conv1d`:\n```bash\npip install mamba-ssm causal-conv1d>=1.2.0\n```\nYou also have to have the model on a CUDA device.\n\nYou can run the model not using the optimized Mamba kernels, but it is **not** recommended as it will result in significantly lower latencies. In order to do that, you'll need to specify `use_mamba_kernels=False` when loading the model.\n\n### Run the model\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel = AutoModelForCausalLM.from_pretrained(\"ai21labs/Jamba-v0.1\")\ntokenizer = AutoTokenizer.from_pretrained(\"ai21labs/Jamba-v0.1\")\n\ninput_ids = tokenizer(\"In the recent Super Bowl LVIII,\", return_tensors='pt').to(model.device)[\"input_ids\"]\n\noutputs = model.generate(input_ids, max_new_tokens=216)\n\nprint(tokenizer.batch_decode(outputs))\n# [\"<|startoftext|>In the recent Super Bowl LVIII, the Kansas City Chiefs emerged victorious, defeating the San Francisco 49ers in a thrilling overtime showdown. The game was a nail-biter, with both teams showcasing their skills and determination.\\n\\nThe Chiefs, led by their star quarterback Patrick Mahomes, displayed their offensive prowess, while the 49ers, led by their strong defense, put up a tough fight. The game went into overtime, with the Chiefs ultimately securing the win with a touchdown.\\n\\nThe victory marked the Chiefs' second Super Bowl win in four years, solidifying their status as one of the top teams in the NFL. The game was a testament to the skill and talent of both teams, and a thrilling end to the NFL season.\\n\\nThe Super Bowl is not just about the game itself, but also about the halftime show and the commercials. This year's halftime show featured a star-studded lineup, including Usher, Alicia Keys, and Lil Jon."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: d6d13c29-5f80-46f5-bade-bec899c2cdbc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/jamba/", "title": "Jamba", "tokens": 1474, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"d6d13c29-5f80-46f5-bade-bec899c2cdbc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jamba/\", \"title\": \"Jamba\", \"tokens\": 1474, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ba4d23dc-0291-4ff1-a039-151d6d8c1f76\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jamba/\", \"title\": \"Jamba\", \"tokens\": 1474, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"2f1de855611fd39ebefcc95df232dc471fd6837d9eb4711f7fb8c8d60d5887c9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9e65cc41-a3ea-4c12-8261-cf109238e920\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jamba/\", \"title\": \"Jamba\", \"tokens\": 1474, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a18ab883e7bcafff35eec5789f4918e51a52569d88f24d3979f7da9e15ba5906\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"654b7cee-40ab-4fc4-afae-3f0aeaa6be18\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"adbba3a1979e614f15bbb5d39c2e92546abab509692eecddb39203c978a41ee5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1611, \"end_char_idx\": 4816, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ba4d23dc-0291-4ff1-a039-151d6d8c1f76", "doc_id": "ba4d23dc-0291-4ff1-a039-151d6d8c1f76", "ref_doc_id": "ba4d23dc-0291-4ff1-a039-151d6d8c1f76", "chroma:document": "You can run the model not using the optimized Mamba kernels, but it is **not** recommended as it will result in significantly lower latencies. In order to do that, you'll need to specify `use_mamba_kernels=False` when loading the model.\n\n### Run the model\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel = AutoModelForCausalLM.from_pretrained(\"ai21labs/Jamba-v0.1\")\ntokenizer = AutoTokenizer.from_pretrained(\"ai21labs/Jamba-v0.1\")\n\ninput_ids = tokenizer(\"In the recent Super Bowl LVIII,\", return_tensors='pt').to(model.device)[\"input_ids\"]\n\noutputs = model.generate(input_ids, max_new_tokens=216)\n\nprint(tokenizer.batch_decode(outputs))\n# [\"<|startoftext|>In the recent Super Bowl LVIII, the Kansas City Chiefs emerged victorious, defeating the San Francisco 49ers in a thrilling overtime showdown. The game was a nail-biter, with both teams showcasing their skills and determination.\\n\\nThe Chiefs, led by their star quarterback Patrick Mahomes, displayed their offensive prowess, while the 49ers, led by their strong defense, put up a tough fight. The game went into overtime, with the Chiefs ultimately securing the win with a touchdown.\\n\\nThe victory marked the Chiefs' second Super Bowl win in four years, solidifying their status as one of the top teams in the NFL. The game was a testament to the skill and talent of both teams, and a thrilling end to the NFL season.\\n\\nThe Super Bowl is not just about the game itself, but also about the halftime show and the commercials. This year's halftime show featured a star-studded lineup, including Usher, Alicia Keys, and Lil Jon. The show was a spectacle of music and dance, with the performers delivering an energetic and entertaining performance.\\n\"]\n```\n\n<details>\n<summary><strong>Loading the model in half precision</strong></summary>\n\nThe published checkpoint is saved in BF16. In order to load it into RAM in BF16/FP16, you need to specify `torch_dtype`:\n\n```python\nfrom transformers import AutoModelForCausalLM\nimport torch\nmodel = AutoModelForCausalLM.from_pretrained(\"ai21labs/Jamba-v0.1\", torch_dtype=torch.bfloat16)\n# you can also use torch_dtype=torch.float16\n```\n\nWhen using half precision, you can enable the [FlashAttention2](https://github.com/Dao-AILab/flash-attention) implementation of the Attention blocks. In order to use it, you also need the model on a CUDA device. Since in this precision the model is to big to fit on a single 80GB GPU, you'll also need to parallelize it using [accelerate](https://huggingface.co/docs/accelerate/index):\n```python\nfrom transformers import AutoModelForCausalLM\nimport torch\nmodel = AutoModelForCausalLM.from_pretrained(\"ai21labs/Jamba-v0.1\",\n                                             torch_dtype=torch.bfloat16,\n                                             attn_implementation=\"flash_attention_2\",\n                                             device_map=\"auto\")\n```\n\n</details>\n<details><summary><strong>Load the model in 8-bit</strong></summary>\n\n**Using 8-bit precision, it is possible to fit up to 140K sequence lengths on a single 80GB GPU.** You can easily quantize the model to 8-bit using [bitsandbytes](https://huggingface.co/docs/bitsandbytes/index)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 654b7cee-40ab-4fc4-afae-3f0aeaa6be18
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/jamba/", "title": "Jamba", "tokens": 1474, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"654b7cee-40ab-4fc4-afae-3f0aeaa6be18\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jamba/\", \"title\": \"Jamba\", \"tokens\": 1474, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ba4d23dc-0291-4ff1-a039-151d6d8c1f76\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jamba/\", \"title\": \"Jamba\", \"tokens\": 1474, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"2f1de855611fd39ebefcc95df232dc471fd6837d9eb4711f7fb8c8d60d5887c9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d6d13c29-5f80-46f5-bade-bec899c2cdbc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jamba/\", \"title\": \"Jamba\", \"tokens\": 1474, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5bf41292d646d59a65463ffaaece1ae4c26d63ad0ed154fbcaa63ba1ac32e987\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3124, \"end_char_idx\": 5556, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ba4d23dc-0291-4ff1-a039-151d6d8c1f76", "doc_id": "ba4d23dc-0291-4ff1-a039-151d6d8c1f76", "ref_doc_id": "ba4d23dc-0291-4ff1-a039-151d6d8c1f76", "chroma:document": "This year's halftime show featured a star-studded lineup, including Usher, Alicia Keys, and Lil Jon. The show was a spectacle of music and dance, with the performers delivering an energetic and entertaining performance.\\n\"]\n```\n\n<details>\n<summary><strong>Loading the model in half precision</strong></summary>\n\nThe published checkpoint is saved in BF16. In order to load it into RAM in BF16/FP16, you need to specify `torch_dtype`:\n\n```python\nfrom transformers import AutoModelForCausalLM\nimport torch\nmodel = AutoModelForCausalLM.from_pretrained(\"ai21labs/Jamba-v0.1\", torch_dtype=torch.bfloat16)\n# you can also use torch_dtype=torch.float16\n```\n\nWhen using half precision, you can enable the [FlashAttention2](https://github.com/Dao-AILab/flash-attention) implementation of the Attention blocks. In order to use it, you also need the model on a CUDA device. Since in this precision the model is to big to fit on a single 80GB GPU, you'll also need to parallelize it using [accelerate](https://huggingface.co/docs/accelerate/index):\n```python\nfrom transformers import AutoModelForCausalLM\nimport torch\nmodel = AutoModelForCausalLM.from_pretrained(\"ai21labs/Jamba-v0.1\",\n                                             torch_dtype=torch.bfloat16,\n                                             attn_implementation=\"flash_attention_2\",\n                                             device_map=\"auto\")\n```\n\n</details>\n<details><summary><strong>Load the model in 8-bit</strong></summary>\n\n**Using 8-bit precision, it is possible to fit up to 140K sequence lengths on a single 80GB GPU.** You can easily quantize the model to 8-bit using [bitsandbytes](https://huggingface.co/docs/bitsandbytes/index). In order to not degrade model quality, we recommend to exclude the Mamba blocks from the quantization:\n\n```python\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\nquantization_config = BitsAndBytesConfig(load_in_8bit=True, llm_int8_skip_modules=[\"mamba\"])\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"ai21labs/Jamba-v0.1\", torch_dtype=torch.bfloat16, attn_implementation=\"flash_attention_2\", quantization_config=quantization_config\n)\n```\n</details>\n\n## JambaConfig\n\n[[autodoc]] JambaConfig\n\n\n## JambaModel\n\n[[autodoc]] JambaModel\n    - forward\n\n\n## JambaForCausalLM\n\n[[autodoc]] JambaForCausalLM\n    - forward\n\n\n## JambaForSequenceClassification\n\n[[autodoc]] transformers.JambaForSequenceClassification\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 68128d92-ed6a-43cd-8e09-b3ad2db639a7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llama/", "title": "LLaMA", "tokens": 2025, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"68128d92-ed6a-43cd-8e09-b3ad2db639a7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama/\", \"title\": \"LLaMA\", \"tokens\": 2025, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2c507092-e7eb-41bb-b00e-400f03b5926e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama/\", \"title\": \"LLaMA\", \"tokens\": 2025, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f62a223c451fc0528fa08c5f5caa090f6b64e784d5709a666d43ca7fee36973a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5345d5aa-64d5-473b-83cb-44601b1e8a65\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2ba5d0c0df263d545be96b6e21a428d756dcc06322811b8f2408ab72258c6b33\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2927, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2c507092-e7eb-41bb-b00e-400f03b5926e", "doc_id": "2c507092-e7eb-41bb-b00e-400f03b5926e", "ref_doc_id": "2c507092-e7eb-41bb-b00e-400f03b5926e", "chroma:document": "# LLaMA\n\n## Overview\n\nThe LLaMA model was proposed in [LLaMA: Open and Efficient Foundation Language Models](https://arxiv.org/abs/2302.13971) by Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth\u00e9e Lacroix, Baptiste Rozi\u00e8re, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, Guillaume Lample. It is a collection of foundation language models ranging from 7B to 65B parameters.\n\nThe abstract from the paper is the following:\n\n*We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B. We release all our models to the research community. *\n\nThis model was contributed by [zphang](https://huggingface.co/zphang) with contributions from [BlackSamorez](https://huggingface.co/BlackSamorez). The code of the implementation in Hugging Face is based on GPT-NeoX [here](https://github.com/EleutherAI/gpt-neox). The original code of the authors can be found [here](https://github.com/facebookresearch/llama).\n\n## Usage tips\n\n- Weights for the LLaMA models can be obtained from by filling out [this form](https://docs.google.com/forms/d/e/1FAIpQLSfqNECQnMkycAp2jP4Z9TFX0cGR4uf7b_fBxjY_OjhJILlKGA/viewform?usp=send_form)\n- After downloading the weights, they will need to be converted to the Hugging Face Transformers format using the [conversion script](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/convert_llama_weights_to_hf.py). The script can be called with the following (example) command:\n\n```bash\npython src/transformers/models/llama/convert_llama_weights_to_hf.py \\\n    --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path\n```\n\n- After conversion, the model and tokenizer can be loaded via:\n\n```python\nfrom transformers import LlamaForCausalLM, LlamaTokenizer\n\ntokenizer = LlamaTokenizer.from_pretrained(\"/output/path\")\nmodel = LlamaForCausalLM.from_pretrained(\"/output/path\")\n```\n\nNote that executing the script requires enough CPU RAM to host the whole model in float16 precision (even if the biggest versions\ncome in several checkpoints they each contain a part of each weight of the model, so we need to load them all in RAM). For the 65B model, it's thus 130GB of RAM needed.\n\n- The LLaMA tokenizer is a BPE model based on [sentencepiece](https://github.com/google/sentencepiece). One quirk of sentencepiece is that when decoding a sequence, if the first token is the start of the word (e.g. \"Banana\"), the tokenizer does not prepend the prefix space to the string."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 5345d5aa-64d5-473b-83cb-44601b1e8a65
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llama/", "title": "LLaMA", "tokens": 2025, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"5345d5aa-64d5-473b-83cb-44601b1e8a65\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama/\", \"title\": \"LLaMA\", \"tokens\": 2025, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2c507092-e7eb-41bb-b00e-400f03b5926e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama/\", \"title\": \"LLaMA\", \"tokens\": 2025, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f62a223c451fc0528fa08c5f5caa090f6b64e784d5709a666d43ca7fee36973a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"68128d92-ed6a-43cd-8e09-b3ad2db639a7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama/\", \"title\": \"LLaMA\", \"tokens\": 2025, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"2a0d63f7047f05e2b85fe9fdd3b2e96ec43c2d2defae35fc86f5136735c5cb2c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ae0bc18b-f66c-4ed6-a968-809d88e4b432\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ea5b533443d7a5392adef42f8c712c0e6b5e43149fd5ac37142c44f2f4b5d94a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1846, \"end_char_idx\": 4707, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2c507092-e7eb-41bb-b00e-400f03b5926e", "doc_id": "2c507092-e7eb-41bb-b00e-400f03b5926e", "ref_doc_id": "2c507092-e7eb-41bb-b00e-400f03b5926e", "chroma:document": "The script can be called with the following (example) command:\n\n```bash\npython src/transformers/models/llama/convert_llama_weights_to_hf.py \\\n    --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path\n```\n\n- After conversion, the model and tokenizer can be loaded via:\n\n```python\nfrom transformers import LlamaForCausalLM, LlamaTokenizer\n\ntokenizer = LlamaTokenizer.from_pretrained(\"/output/path\")\nmodel = LlamaForCausalLM.from_pretrained(\"/output/path\")\n```\n\nNote that executing the script requires enough CPU RAM to host the whole model in float16 precision (even if the biggest versions\ncome in several checkpoints they each contain a part of each weight of the model, so we need to load them all in RAM). For the 65B model, it's thus 130GB of RAM needed.\n\n- The LLaMA tokenizer is a BPE model based on [sentencepiece](https://github.com/google/sentencepiece). One quirk of sentencepiece is that when decoding a sequence, if the first token is the start of the word (e.g. \"Banana\"), the tokenizer does not prepend the prefix space to the string.\n\nThis model was contributed by [zphang](https://huggingface.co/zphang) with contributions from [BlackSamorez](https://huggingface.co/BlackSamorez). The code of the implementation in Hugging Face is based on GPT-NeoX [here](https://github.com/EleutherAI/gpt-neox). The original code of the authors can be found [here](https://github.com/facebookresearch/llama). The Flax version of the implementation was contributed by [afmck](https://huggingface.co/afmck) with the code in the implementation based on Hugging Face's Flax GPT-Neo.\n\n\nBased on the original LLaMA model, Meta AI has released some follow-up works:\n\n- **Llama2**: Llama2 is an improved version of Llama with some architectural tweaks (Grouped Query Attention), and is pre-trained on 2Trillion tokens. Refer to the documentation of Llama2 which can be found [here](llama2).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with LLaMA. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- A [notebook](https://colab.research.google.com/github/bigscience-workshop/petals/blob/main/examples/prompt-tuning-sst2.ipynb#scrollTo=f04ba4d2) on how to use prompt tuning to adapt the LLaMA model for text classification task. \ud83c\udf0e\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- [StackLLaMA: A hands-on guide to train LLaMA with RLHF](https://huggingface.co/blog/stackllama#stackllama-a-hands-on-guide-to-train-llama-with-rlhf), a blog post about how to train LLaMA to answer questions on [Stack Exchange](https://stackexchange.com/) with RLHF."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: ae0bc18b-f66c-4ed6-a968-809d88e4b432
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llama/", "title": "LLaMA", "tokens": 2025, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"ae0bc18b-f66c-4ed6-a968-809d88e4b432\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama/\", \"title\": \"LLaMA\", \"tokens\": 2025, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2c507092-e7eb-41bb-b00e-400f03b5926e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama/\", \"title\": \"LLaMA\", \"tokens\": 2025, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f62a223c451fc0528fa08c5f5caa090f6b64e784d5709a666d43ca7fee36973a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5345d5aa-64d5-473b-83cb-44601b1e8a65\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama/\", \"title\": \"LLaMA\", \"tokens\": 2025, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"6aa2869385279387e282e09cd72d699c92b03bfadff9aaad4707ce22c42996c8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9b3db112-b282-472c-825a-6cfb8c332c49\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f5e831bddeb404184352ad69569fcb49951a541f4cc776139a81b7fb9d6feeec\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3192, \"end_char_idx\": 5795, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2c507092-e7eb-41bb-b00e-400f03b5926e", "doc_id": "2c507092-e7eb-41bb-b00e-400f03b5926e", "ref_doc_id": "2c507092-e7eb-41bb-b00e-400f03b5926e", "chroma:document": "The original code of the authors can be found [here](https://github.com/facebookresearch/llama). The Flax version of the implementation was contributed by [afmck](https://huggingface.co/afmck) with the code in the implementation based on Hugging Face's Flax GPT-Neo.\n\n\nBased on the original LLaMA model, Meta AI has released some follow-up works:\n\n- **Llama2**: Llama2 is an improved version of Llama with some architectural tweaks (Grouped Query Attention), and is pre-trained on 2Trillion tokens. Refer to the documentation of Llama2 which can be found [here](llama2).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with LLaMA. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- A [notebook](https://colab.research.google.com/github/bigscience-workshop/petals/blob/main/examples/prompt-tuning-sst2.ipynb#scrollTo=f04ba4d2) on how to use prompt tuning to adapt the LLaMA model for text classification task. \ud83c\udf0e\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- [StackLLaMA: A hands-on guide to train LLaMA with RLHF](https://huggingface.co/blog/stackllama#stackllama-a-hands-on-guide-to-train-llama-with-rlhf), a blog post about how to train LLaMA to answer questions on [Stack Exchange](https://stackexchange.com/) with RLHF.\n\n\u2697\ufe0f Optimization\n- A [notebook](https://colab.research.google.com/drive/1SQUXq1AMZPSLD4mk3A3swUIc6Y2dclme?usp=sharing) on how to fine-tune LLaMA model using xturing library on GPU which has limited memory. \ud83c\udf0e \n\n\u26a1\ufe0f Inference\n- A [notebook](https://colab.research.google.com/github/DominguesM/alpaca-lora-ptbr-7b/blob/main/notebooks/02%20-%20Evaluate.ipynb) on how to run the LLaMA Model using PeftModel from the \ud83e\udd17 PEFT library. \ud83c\udf0e \n- A [notebook](https://colab.research.google.com/drive/1l2GiSSPbajVyp2Nk3CFT4t3uH6-5TiBe?usp=sharing) on how to load a PEFT adapter LLaMA model with LangChain. \ud83c\udf0e\n\n\ud83d\ude80 Deploy\n- A [notebook](https://colab.research.google.com/github/lxe/simple-llama-finetuner/blob/master/Simple_LLaMA_FineTuner.ipynb#scrollTo=3PM_DilAZD8T) on how to fine-tune LLaMA model using LoRA method via the \ud83e\udd17 PEFT library with intuitive UI. \ud83c\udf0e \n- A [notebook](https://github.com/aws/amazon-sagemaker-examples/blob/main/introduction_to_amazon_algorithms/jumpstart-foundation-models/text-generation-open-llama.ipynb) on how to deploy Open-LLaMA model for text generation on Amazon SageMaker."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 9b3db112-b282-472c-825a-6cfb8c332c49
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llama/", "title": "LLaMA", "tokens": 2025, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"9b3db112-b282-472c-825a-6cfb8c332c49\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama/\", \"title\": \"LLaMA\", \"tokens\": 2025, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2c507092-e7eb-41bb-b00e-400f03b5926e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama/\", \"title\": \"LLaMA\", \"tokens\": 2025, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f62a223c451fc0528fa08c5f5caa090f6b64e784d5709a666d43ca7fee36973a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ae0bc18b-f66c-4ed6-a968-809d88e4b432\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama/\", \"title\": \"LLaMA\", \"tokens\": 2025, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f907d39ba7ad6a726f38b9dace630fbfd8063c8698d3c0e9c6d4659f3046f7bc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4709, \"end_char_idx\": 6754, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2c507092-e7eb-41bb-b00e-400f03b5926e", "doc_id": "2c507092-e7eb-41bb-b00e-400f03b5926e", "ref_doc_id": "2c507092-e7eb-41bb-b00e-400f03b5926e", "chroma:document": "\u2697\ufe0f Optimization\n- A [notebook](https://colab.research.google.com/drive/1SQUXq1AMZPSLD4mk3A3swUIc6Y2dclme?usp=sharing) on how to fine-tune LLaMA model using xturing library on GPU which has limited memory. \ud83c\udf0e \n\n\u26a1\ufe0f Inference\n- A [notebook](https://colab.research.google.com/github/DominguesM/alpaca-lora-ptbr-7b/blob/main/notebooks/02%20-%20Evaluate.ipynb) on how to run the LLaMA Model using PeftModel from the \ud83e\udd17 PEFT library. \ud83c\udf0e \n- A [notebook](https://colab.research.google.com/drive/1l2GiSSPbajVyp2Nk3CFT4t3uH6-5TiBe?usp=sharing) on how to load a PEFT adapter LLaMA model with LangChain. \ud83c\udf0e\n\n\ud83d\ude80 Deploy\n- A [notebook](https://colab.research.google.com/github/lxe/simple-llama-finetuner/blob/master/Simple_LLaMA_FineTuner.ipynb#scrollTo=3PM_DilAZD8T) on how to fine-tune LLaMA model using LoRA method via the \ud83e\udd17 PEFT library with intuitive UI. \ud83c\udf0e \n- A [notebook](https://github.com/aws/amazon-sagemaker-examples/blob/main/introduction_to_amazon_algorithms/jumpstart-foundation-models/text-generation-open-llama.ipynb) on how to deploy Open-LLaMA model for text generation on Amazon SageMaker. \ud83c\udf0e \n\n## LlamaConfig\n\n[[autodoc]] LlamaConfig\n\n## LlamaTokenizer\n\n[[autodoc]] LlamaTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## LlamaTokenizerFast\n\n[[autodoc]] LlamaTokenizerFast\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - update_post_processor\n    - save_vocabulary\n\n## LlamaModel\n\n[[autodoc]] LlamaModel\n    - forward\n\n## LlamaForCausalLM\n\n[[autodoc]] LlamaForCausalLM\n    - forward\n\n## LlamaForSequenceClassification\n\n[[autodoc]] LlamaForSequenceClassification\n    - forward\n\n## LlamaForQuestionAnswering\n\n[[autodoc]] LlamaForQuestionAnswering\n    - forward\n\n## LlamaForTokenClassification\n\n[[autodoc]] LlamaForTokenClassification\n    - forward\n\n## FlaxLlamaModel\n\n[[autodoc]] FlaxLlamaModel\n    - __call__\n\n## FlaxLlamaForCausalLM\n\n[[autodoc]] FlaxLlamaForCausalLM\n    - __call__"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 45cf7c4d-0350-48ca-bf62-b647dd644b99
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xlsr_wav2vec2/", "title": "XLSR-Wav2Vec2", "tokens": 728, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"45cf7c4d-0350-48ca-bf62-b647dd644b99\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlsr_wav2vec2/\", \"title\": \"XLSR-Wav2Vec2\", \"tokens\": 728, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"63597afb-7202-4db0-a12d-d43da382200f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlsr_wav2vec2/\", \"title\": \"XLSR-Wav2Vec2\", \"tokens\": 728, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"9b6a5add459e1b4202d63d63136e3eddca3800439d9250ab60833e66b3eaa69c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2269, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "63597afb-7202-4db0-a12d-d43da382200f", "doc_id": "63597afb-7202-4db0-a12d-d43da382200f", "ref_doc_id": "63597afb-7202-4db0-a12d-d43da382200f", "chroma:document": "# XLSR-Wav2Vec2\n\n## Overview\n\nThe XLSR-Wav2Vec2 model was proposed in [Unsupervised Cross-Lingual Representation Learning For Speech Recognition](https://arxiv.org/abs/2006.13979) by Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, Michael\nAuli.\n\nThe abstract from the paper is the following:\n\n*This paper presents XLSR which learns cross-lingual speech representations by pretraining a single model from the raw\nwaveform of speech in multiple languages. We build on wav2vec 2.0 which is trained by solving a contrastive task over\nmasked latent speech representations and jointly learns a quantization of the latents shared across languages. The\nresulting model is fine-tuned on labeled data and experiments show that cross-lingual pretraining significantly\noutperforms monolingual pretraining. On the CommonVoice benchmark, XLSR shows a relative phoneme error rate reduction\nof 72% compared to the best known results. On BABEL, our approach improves word error rate by 16% relative compared to\na comparable system. Our approach enables a single multilingual speech recognition model which is competitive to strong\nindividual models. Analysis shows that the latent discrete speech representations are shared across languages with\nincreased sharing for related languages. We hope to catalyze research in low-resource speech understanding by releasing\nXLSR-53, a large model pretrained in 53 languages.*\n\nThe original code can be found [here](https://github.com/pytorch/fairseq/tree/master/fairseq/models/wav2vec).\n\nNote: Meta (FAIR) released a new version of [Wav2Vec2-BERT 2.0](https://huggingface.co/docs/transformers/en/model_doc/wav2vec2-bert) - it's pretrained on 4.5M hours of audio. We especially recommend using it for fine-tuning tasks, e.g. as per [this guide](https://huggingface.co/blog/fine-tune-w2v2-bert).\n\n## Usage tips\n\n- XLSR-Wav2Vec2 is a speech model that accepts a float array corresponding to the raw waveform of the speech signal.\n- XLSR-Wav2Vec2 model was trained using connectionist temporal classification (CTC) so the model output has to be\n  decoded using [`Wav2Vec2CTCTokenizer`].\n\n<Tip>\n\nXLSR-Wav2Vec2's architecture is based on the Wav2Vec2 model, so one can refer to [Wav2Vec2's documentation page](wav2vec2).\n\n</Tip>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 7b55b72a-56e2-4d19-9d6a-927375404508
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mvp/", "title": "MVP", "tokens": 1508, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"7b55b72a-56e2-4d19-9d6a-927375404508\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mvp/\", \"title\": \"MVP\", \"tokens\": 1508, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c414b956-fc8a-4d47-8f8d-ff2f158d48bd\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mvp/\", \"title\": \"MVP\", \"tokens\": 1508, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"e857decc007b05183614299dc205789cd9b5e1e0734fca3d3a5557d9ca0e9ccc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ce757168-5edd-4a40-abd7-6105284964d1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7f3234812f55db0630f62818f2dfca439c0b58df9d761d6c19610cccc52b3107\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2883, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c414b956-fc8a-4d47-8f8d-ff2f158d48bd", "doc_id": "c414b956-fc8a-4d47-8f8d-ff2f158d48bd", "ref_doc_id": "c414b956-fc8a-4d47-8f8d-ff2f158d48bd", "chroma:document": "# MVP\n\n## Overview\n\nThe MVP model was proposed in [MVP: Multi-task Supervised Pre-training for Natural Language Generation](https://arxiv.org/abs/2206.12131) by Tianyi Tang, Junyi Li, Wayne Xin Zhao and Ji-Rong Wen.\n\n\nAccording to the abstract,\n\n- MVP follows a standard Transformer encoder-decoder architecture.\n- MVP is supervised pre-trained using labeled datasets.\n- MVP also has task-specific soft prompts to stimulate the model's capacity in performing a certain task.\n- MVP is specially designed for natural language generation and can be adapted to a wide range of generation tasks, including but not limited to summarization, data-to-text generation, open-ended dialogue system, story generation, question answering, question generation, task-oriented dialogue system, commonsense generation, paraphrase generation, text style transfer, and text simplification. Our model can also be adapted to natural language understanding tasks such as sequence classification and (extractive) question answering.\n\nThis model was contributed by [Tianyi Tang](https://huggingface.co/StevenTang). The detailed information and instructions can be found [here](https://github.com/RUCAIBox/MVP).\n\n## Usage tips\n\n- We have released a series of models [here](https://huggingface.co/models?filter=mvp), including MVP, MVP with task-specific prompts, and multi-task pre-trained variants.\n- If you want to use a model without prompts (standard Transformer), you can load it through `MvpForConditionalGeneration.from_pretrained('RUCAIBox/mvp')`.\n- If you want to use a model with task-specific prompts, such as summarization, you can load it through `MvpForConditionalGeneration.from_pretrained('RUCAIBox/mvp-summarization')`.\n- Our model supports lightweight prompt tuning following [Prefix-tuning](https://arxiv.org/abs/2101.00190) with method `set_lightweight_tuning()`.\n\n## Usage examples\n\nFor summarization, it is an example to use MVP and MVP with summarization-specific prompts.\n\n```python\n>>> from transformers import MvpTokenizer, MvpForConditionalGeneration\n\n>>> tokenizer = MvpTokenizer.from_pretrained(\"RUCAIBox/mvp\")\n>>> model = MvpForConditionalGeneration.from_pretrained(\"RUCAIBox/mvp\")\n>>> model_with_prompt = MvpForConditionalGeneration.from_pretrained(\"RUCAIBox/mvp-summarization\")\n\n>>> inputs = tokenizer(\n...     \"Summarize: You may want to stick it to your boss and leave your job, but don't do it if these are your reasons.\",\n...     return_tensors=\"pt\",\n... )\n>>> generated_ids = model.generate(**inputs)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)\n[\"Why You Shouldn't Quit Your Job\"]\n\n>>> generated_ids = model_with_prompt.generate(**inputs)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)\n[\"Don't do it if these are your reasons\"]\n```\n\nFor data-to-text generation, it is an example to use MVP and multi-task pre-trained variants."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: ce757168-5edd-4a40-abd7-6105284964d1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mvp/", "title": "MVP", "tokens": 1508, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"ce757168-5edd-4a40-abd7-6105284964d1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mvp/\", \"title\": \"MVP\", \"tokens\": 1508, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c414b956-fc8a-4d47-8f8d-ff2f158d48bd\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mvp/\", \"title\": \"MVP\", \"tokens\": 1508, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"e857decc007b05183614299dc205789cd9b5e1e0734fca3d3a5557d9ca0e9ccc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7b55b72a-56e2-4d19-9d6a-927375404508\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mvp/\", \"title\": \"MVP\", \"tokens\": 1508, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"4ebb18fb1ce8d29e8fe3ee99c61b127f22ee675b19fd0a19ea4fac948f7fd274\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d0d8fc61-4ee8-476e-87a5-79928f215a07\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4a633ea8106f4a3a4fbb4c31e5b159e1d3da56e1224fa656f6444362ce601d17\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1188, \"end_char_idx\": 4079, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c414b956-fc8a-4d47-8f8d-ff2f158d48bd", "doc_id": "c414b956-fc8a-4d47-8f8d-ff2f158d48bd", "ref_doc_id": "c414b956-fc8a-4d47-8f8d-ff2f158d48bd", "chroma:document": "## Usage tips\n\n- We have released a series of models [here](https://huggingface.co/models?filter=mvp), including MVP, MVP with task-specific prompts, and multi-task pre-trained variants.\n- If you want to use a model without prompts (standard Transformer), you can load it through `MvpForConditionalGeneration.from_pretrained('RUCAIBox/mvp')`.\n- If you want to use a model with task-specific prompts, such as summarization, you can load it through `MvpForConditionalGeneration.from_pretrained('RUCAIBox/mvp-summarization')`.\n- Our model supports lightweight prompt tuning following [Prefix-tuning](https://arxiv.org/abs/2101.00190) with method `set_lightweight_tuning()`.\n\n## Usage examples\n\nFor summarization, it is an example to use MVP and MVP with summarization-specific prompts.\n\n```python\n>>> from transformers import MvpTokenizer, MvpForConditionalGeneration\n\n>>> tokenizer = MvpTokenizer.from_pretrained(\"RUCAIBox/mvp\")\n>>> model = MvpForConditionalGeneration.from_pretrained(\"RUCAIBox/mvp\")\n>>> model_with_prompt = MvpForConditionalGeneration.from_pretrained(\"RUCAIBox/mvp-summarization\")\n\n>>> inputs = tokenizer(\n...     \"Summarize: You may want to stick it to your boss and leave your job, but don't do it if these are your reasons.\",\n...     return_tensors=\"pt\",\n... )\n>>> generated_ids = model.generate(**inputs)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)\n[\"Why You Shouldn't Quit Your Job\"]\n\n>>> generated_ids = model_with_prompt.generate(**inputs)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)\n[\"Don't do it if these are your reasons\"]\n```\n\nFor data-to-text generation, it is an example to use MVP and multi-task pre-trained variants.\n```python\n>>> from transformers import MvpTokenizerFast, MvpForConditionalGeneration\n\n>>> tokenizer = MvpTokenizerFast.from_pretrained(\"RUCAIBox/mvp\")\n>>> model = MvpForConditionalGeneration.from_pretrained(\"RUCAIBox/mvp\")\n>>> model_with_mtl = MvpForConditionalGeneration.from_pretrained(\"RUCAIBox/mtl-data-to-text\")\n\n>>> inputs = tokenizer(\n...     \"Describe the following data: Iron Man | instance of | Superhero [SEP] Stan Lee | creator | Iron Man\",\n...     return_tensors=\"pt\",\n... )\n>>> generated_ids = model.generate(**inputs)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)\n['Stan Lee created the character of Iron Man, a fictional superhero appearing in American comic']\n\n>>> generated_ids = model_with_mtl.generate(**inputs)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)\n['Iron Man is a fictional superhero appearing in American comic books published by Marvel Comics.']\n```\n\nFor lightweight tuning, *i.e.*, fixing the model and only tuning prompts, you can load MVP with randomly initialized prompts or with task-specific prompts. Our code also supports Prefix-tuning with BART following the [original paper](https://arxiv.org/abs/2101.00190)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: d0d8fc61-4ee8-476e-87a5-79928f215a07
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mvp/", "title": "MVP", "tokens": 1508, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"d0d8fc61-4ee8-476e-87a5-79928f215a07\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mvp/\", \"title\": \"MVP\", \"tokens\": 1508, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c414b956-fc8a-4d47-8f8d-ff2f158d48bd\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mvp/\", \"title\": \"MVP\", \"tokens\": 1508, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"e857decc007b05183614299dc205789cd9b5e1e0734fca3d3a5557d9ca0e9ccc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ce757168-5edd-4a40-abd7-6105284964d1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mvp/\", \"title\": \"MVP\", \"tokens\": 1508, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"6e071bc96ed733b6278d6343bd3569a3e0f59a2ae58755ae90fb10f5247916bf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2430, \"end_char_idx\": 5837, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c414b956-fc8a-4d47-8f8d-ff2f158d48bd", "doc_id": "c414b956-fc8a-4d47-8f8d-ff2f158d48bd", "ref_doc_id": "c414b956-fc8a-4d47-8f8d-ff2f158d48bd", "chroma:document": "\",\n...     return_tensors=\"pt\",\n... )\n>>> generated_ids = model.generate(**inputs)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)\n[\"Why You Shouldn't Quit Your Job\"]\n\n>>> generated_ids = model_with_prompt.generate(**inputs)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)\n[\"Don't do it if these are your reasons\"]\n```\n\nFor data-to-text generation, it is an example to use MVP and multi-task pre-trained variants.\n```python\n>>> from transformers import MvpTokenizerFast, MvpForConditionalGeneration\n\n>>> tokenizer = MvpTokenizerFast.from_pretrained(\"RUCAIBox/mvp\")\n>>> model = MvpForConditionalGeneration.from_pretrained(\"RUCAIBox/mvp\")\n>>> model_with_mtl = MvpForConditionalGeneration.from_pretrained(\"RUCAIBox/mtl-data-to-text\")\n\n>>> inputs = tokenizer(\n...     \"Describe the following data: Iron Man | instance of | Superhero [SEP] Stan Lee | creator | Iron Man\",\n...     return_tensors=\"pt\",\n... )\n>>> generated_ids = model.generate(**inputs)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)\n['Stan Lee created the character of Iron Man, a fictional superhero appearing in American comic']\n\n>>> generated_ids = model_with_mtl.generate(**inputs)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)\n['Iron Man is a fictional superhero appearing in American comic books published by Marvel Comics.']\n```\n\nFor lightweight tuning, *i.e.*, fixing the model and only tuning prompts, you can load MVP with randomly initialized prompts or with task-specific prompts. Our code also supports Prefix-tuning with BART following the [original paper](https://arxiv.org/abs/2101.00190).\n\n```python\n>>> from transformers import MvpForConditionalGeneration\n\n>>> model = MvpForConditionalGeneration.from_pretrained(\"RUCAIBox/mvp\", use_prompt=True)\n>>> # the number of trainable parameters (full tuning)\n>>> sum(p.numel() for p in model.parameters() if p.requires_grad)\n468116832\n\n>>> # lightweight tuning with randomly initialized prompts\n>>> model.set_lightweight_tuning()\n>>> # the number of trainable parameters (lightweight tuning)\n>>> sum(p.numel() for p in model.parameters() if p.requires_grad)\n61823328\n\n>>> # lightweight tuning with task-specific prompts\n>>> model = MvpForConditionalGeneration.from_pretrained(\"RUCAIBox/mtl-data-to-text\")\n>>> model.set_lightweight_tuning()\n>>> # original lightweight Prefix-tuning\n>>> model = MvpForConditionalGeneration.from_pretrained(\"facebook/bart-large\", use_prompt=True)\n>>> model.set_lightweight_tuning()\n```\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n## MvpConfig\n\n[[autodoc]] MvpConfig\n\n## MvpTokenizer\n\n[[autodoc]] MvpTokenizer\n\n## MvpTokenizerFast\n\n[[autodoc]] MvpTokenizerFast\n\n## MvpModel\n\n[[autodoc]] MvpModel\n    - forward\n\n## MvpForConditionalGeneration\n\n[[autodoc]] MvpForConditionalGeneration\n    - forward\n\n## MvpForSequenceClassification\n\n[[autodoc]] MvpForSequenceClassification\n    - forward\n\n## MvpForQuestionAnswering\n\n[[autodoc]] MvpForQuestionAnswering\n    - forward\n\n## MvpForCausalLM\n\n[[autodoc]] MvpForCausalLM\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: d3343876-b217-4c2d-b36b-3ca085ed023c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mbart/", "title": "MBart and MBart-50", "tokens": 2452, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"d3343876-b217-4c2d-b36b-3ca085ed023c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mbart/\", \"title\": \"MBart and MBart-50\", \"tokens\": 2452, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4eaabdc9-104a-4d67-90ec-c68b442af250\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mbart/\", \"title\": \"MBart and MBart-50\", \"tokens\": 2452, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b1d5932ea9bed5df529ac53dfdc450a26f301a5abbf9a203bf33f1f23e627621\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0e17bee3-815a-4ed0-8c9a-aed7fe4fcb11\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8f2925fd6c46cee35196f44f97ef807ca7309c72b612a73365cb45d2fb96a125\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2768, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4eaabdc9-104a-4d67-90ec-c68b442af250", "doc_id": "4eaabdc9-104a-4d67-90ec-c68b442af250", "ref_doc_id": "4eaabdc9-104a-4d67-90ec-c68b442af250", "chroma:document": "# MBart and MBart-50\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=mbart\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-mbart-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/mbart-large-50-one-to-many-mmt\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n\n## Overview of MBart\n\nThe MBart model was presented in [Multilingual Denoising Pre-training for Neural Machine Translation](https://arxiv.org/abs/2001.08210) by Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov Marjan\nGhazvininejad, Mike Lewis, Luke Zettlemoyer.\n\nAccording to the abstract, MBART is a sequence-to-sequence denoising auto-encoder pretrained on large-scale monolingual\ncorpora in many languages using the BART objective. mBART is one of the first methods for pretraining a complete\nsequence-to-sequence model by denoising full texts in multiple languages, while previous approaches have focused only\non the encoder, decoder, or reconstructing parts of the text.\n\nThis model was contributed by [valhalla](https://huggingface.co/valhalla). The Authors' code can be found [here](https://github.com/pytorch/fairseq/tree/master/examples/mbart)\n\n### Training of MBart\n\nMBart is a multilingual encoder-decoder (sequence-to-sequence) model primarily intended for translation task. As the\nmodel is multilingual it expects the sequences in a different format. A special language id token is added in both the\nsource and target text. The source text format is `X [eos, src_lang_code]` where `X` is the source text. The\ntarget text format is `[tgt_lang_code] X [eos]`. `bos` is never used.\n\nThe regular [`~MBartTokenizer.__call__`] will encode source text format passed as first argument or with the `text`\nkeyword, and target text format passed with the `text_label` keyword argument.\n\n- Supervised training\n\n```python\n>>> from transformers import MBartForConditionalGeneration, MBartTokenizer\n\n>>> tokenizer = MBartTokenizer.from_pretrained(\"facebook/mbart-large-en-ro\", src_lang=\"en_XX\", tgt_lang=\"ro_RO\")\n>>> example_english_phrase = \"UN Chief Says There Is No Military Solution in Syria\"\n>>> expected_translation_romanian = \"\u015eeful ONU declar\u0103 c\u0103 nu exist\u0103 o solu\u0163ie militar\u0103 \u00een Siria\"\n\n>>> inputs = tokenizer(example_english_phrase, text_target=expected_translation_romanian, return_tensors=\"pt\")\n\n>>> model = MBartForConditionalGeneration.from_pretrained(\"facebook/mbart-large-en-ro\")\n>>> # forward pass\n>>> model(**inputs)\n```\n\n- Generation\n\n  While generating the target text set the `decoder_start_token_id` to the target language id. The following\n  example shows how to translate English to Romanian using the *facebook/mbart-large-en-ro* model."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 0e17bee3-815a-4ed0-8c9a-aed7fe4fcb11
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mbart/", "title": "MBart and MBart-50", "tokens": 2452, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"0e17bee3-815a-4ed0-8c9a-aed7fe4fcb11\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mbart/\", \"title\": \"MBart and MBart-50\", \"tokens\": 2452, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4eaabdc9-104a-4d67-90ec-c68b442af250\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mbart/\", \"title\": \"MBart and MBart-50\", \"tokens\": 2452, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b1d5932ea9bed5df529ac53dfdc450a26f301a5abbf9a203bf33f1f23e627621\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d3343876-b217-4c2d-b36b-3ca085ed023c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mbart/\", \"title\": \"MBart and MBart-50\", \"tokens\": 2452, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0f399790a83076d57e9de3ad4f19e64024358a825787973cc8629228a7c63097\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4c1818e6-5a0b-4d9d-830b-581fdb3a689b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7d6a9301c89809ed9f5de04da89188082c482b01652de5c58c0a0388581f63ec\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1162, \"end_char_idx\": 4380, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4eaabdc9-104a-4d67-90ec-c68b442af250", "doc_id": "4eaabdc9-104a-4d67-90ec-c68b442af250", "ref_doc_id": "4eaabdc9-104a-4d67-90ec-c68b442af250", "chroma:document": "The Authors' code can be found [here](https://github.com/pytorch/fairseq/tree/master/examples/mbart)\n\n### Training of MBart\n\nMBart is a multilingual encoder-decoder (sequence-to-sequence) model primarily intended for translation task. As the\nmodel is multilingual it expects the sequences in a different format. A special language id token is added in both the\nsource and target text. The source text format is `X [eos, src_lang_code]` where `X` is the source text. The\ntarget text format is `[tgt_lang_code] X [eos]`. `bos` is never used.\n\nThe regular [`~MBartTokenizer.__call__`] will encode source text format passed as first argument or with the `text`\nkeyword, and target text format passed with the `text_label` keyword argument.\n\n- Supervised training\n\n```python\n>>> from transformers import MBartForConditionalGeneration, MBartTokenizer\n\n>>> tokenizer = MBartTokenizer.from_pretrained(\"facebook/mbart-large-en-ro\", src_lang=\"en_XX\", tgt_lang=\"ro_RO\")\n>>> example_english_phrase = \"UN Chief Says There Is No Military Solution in Syria\"\n>>> expected_translation_romanian = \"\u015eeful ONU declar\u0103 c\u0103 nu exist\u0103 o solu\u0163ie militar\u0103 \u00een Siria\"\n\n>>> inputs = tokenizer(example_english_phrase, text_target=expected_translation_romanian, return_tensors=\"pt\")\n\n>>> model = MBartForConditionalGeneration.from_pretrained(\"facebook/mbart-large-en-ro\")\n>>> # forward pass\n>>> model(**inputs)\n```\n\n- Generation\n\n  While generating the target text set the `decoder_start_token_id` to the target language id. The following\n  example shows how to translate English to Romanian using the *facebook/mbart-large-en-ro* model.\n\n```python\n>>> from transformers import MBartForConditionalGeneration, MBartTokenizer\n\n>>> tokenizer = MBartTokenizer.from_pretrained(\"facebook/mbart-large-en-ro\", src_lang=\"en_XX\")\n>>> article = \"UN Chief Says There Is No Military Solution in Syria\"\n>>> inputs = tokenizer(article, return_tensors=\"pt\")\n>>> translated_tokens = model.generate(**inputs, decoder_start_token_id=tokenizer.lang_code_to_id[\"ro_RO\"])\n>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]\n\"\u015eeful ONU declar\u0103 c\u0103 nu exist\u0103 o solu\u0163ie militar\u0103 \u00een Siria\"\n```\n\n## Overview of MBart-50\n\nMBart-50 was introduced in the [Multilingual Translation with Extensible Multilingual Pretraining and Finetuning](https://arxiv.org/abs/2008.00401) paper by Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav\nChaudhary, Jiatao Gu, Angela Fan. MBart-50 is created using the original *mbart-large-cc25* checkpoint by extendeding\nits embedding layers with randomly initialized vectors for an extra set of 25 language tokens and then pretrained on 50\nlanguages.\n\nAccording to the abstract\n\n*Multilingual translation models can be created through multilingual finetuning. Instead of finetuning on one\ndirection, a pretrained model is finetuned on many directions at the same time. It demonstrates that pretrained models\ncan be extended to incorporate additional languages without loss of performance. Multilingual finetuning improves on\naverage 1 BLEU over the strongest baselines (being either multilingual from scratch or bilingual finetuning) while\nimproving 9.3 BLEU on average over bilingual baselines from scratch.*"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 4c1818e6-5a0b-4d9d-830b-581fdb3a689b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mbart/", "title": "MBart and MBart-50", "tokens": 2452, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"4c1818e6-5a0b-4d9d-830b-581fdb3a689b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mbart/\", \"title\": \"MBart and MBart-50\", \"tokens\": 2452, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4eaabdc9-104a-4d67-90ec-c68b442af250\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mbart/\", \"title\": \"MBart and MBart-50\", \"tokens\": 2452, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b1d5932ea9bed5df529ac53dfdc450a26f301a5abbf9a203bf33f1f23e627621\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0e17bee3-815a-4ed0-8c9a-aed7fe4fcb11\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mbart/\", \"title\": \"MBart and MBart-50\", \"tokens\": 2452, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"08344431ca9dc5e11529eb489120562e98d1831737d18bc4cf900752e081961d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2e488e56-1ba9-4575-99fa-f83a3061ac6a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1c7ed3d03c5d33f8f421f6e8976baf8920d60ea504d690d142b74a05b6920824\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2770, \"end_char_idx\": 5870, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4eaabdc9-104a-4d67-90ec-c68b442af250", "doc_id": "4eaabdc9-104a-4d67-90ec-c68b442af250", "ref_doc_id": "4eaabdc9-104a-4d67-90ec-c68b442af250", "chroma:document": "```python\n>>> from transformers import MBartForConditionalGeneration, MBartTokenizer\n\n>>> tokenizer = MBartTokenizer.from_pretrained(\"facebook/mbart-large-en-ro\", src_lang=\"en_XX\")\n>>> article = \"UN Chief Says There Is No Military Solution in Syria\"\n>>> inputs = tokenizer(article, return_tensors=\"pt\")\n>>> translated_tokens = model.generate(**inputs, decoder_start_token_id=tokenizer.lang_code_to_id[\"ro_RO\"])\n>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]\n\"\u015eeful ONU declar\u0103 c\u0103 nu exist\u0103 o solu\u0163ie militar\u0103 \u00een Siria\"\n```\n\n## Overview of MBart-50\n\nMBart-50 was introduced in the [Multilingual Translation with Extensible Multilingual Pretraining and Finetuning](https://arxiv.org/abs/2008.00401) paper by Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav\nChaudhary, Jiatao Gu, Angela Fan. MBart-50 is created using the original *mbart-large-cc25* checkpoint by extendeding\nits embedding layers with randomly initialized vectors for an extra set of 25 language tokens and then pretrained on 50\nlanguages.\n\nAccording to the abstract\n\n*Multilingual translation models can be created through multilingual finetuning. Instead of finetuning on one\ndirection, a pretrained model is finetuned on many directions at the same time. It demonstrates that pretrained models\ncan be extended to incorporate additional languages without loss of performance. Multilingual finetuning improves on\naverage 1 BLEU over the strongest baselines (being either multilingual from scratch or bilingual finetuning) while\nimproving 9.3 BLEU on average over bilingual baselines from scratch.*\n\n\n### Training of MBart-50\n\nThe text format for MBart-50 is slightly different from mBART. For MBart-50 the language id token is used as a prefix\nfor both source and target text i.e the text format is `[lang_code] X [eos]`, where `lang_code` is source\nlanguage id for source text and target language id for target text, with `X` being the source or target text\nrespectively.\n\n\nMBart-50 has its own tokenizer [`MBart50Tokenizer`].\n\n-  Supervised training\n\n```python\nfrom transformers import MBartForConditionalGeneration, MBart50TokenizerFast\n\nmodel = MBartForConditionalGeneration.from_pretrained(\"facebook/mbart-large-50\")\ntokenizer = MBart50TokenizerFast.from_pretrained(\"facebook/mbart-large-50\", src_lang=\"en_XX\", tgt_lang=\"ro_RO\")\n\nsrc_text = \" UN Chief Says There Is No Military Solution in Syria\"\ntgt_text = \"\u015eeful ONU declar\u0103 c\u0103 nu exist\u0103 o solu\u0163ie militar\u0103 \u00een Siria\"\n\nmodel_inputs = tokenizer(src_text, text_target=tgt_text, return_tensors=\"pt\")\n\nmodel(**model_inputs)  # forward pass\n```\n\n- Generation\n\n  To generate using the mBART-50 multilingual translation models, `eos_token_id` is used as the\n  `decoder_start_token_id` and the target language id is forced as the first generated token. To force the\n  target language id as the first generated token, pass the *forced_bos_token_id* parameter to the *generate* method.\n  The following example shows how to translate between Hindi to French and Arabic to English using the\n  *facebook/mbart-50-large-many-to-many* checkpoint."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 2e488e56-1ba9-4575-99fa-f83a3061ac6a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mbart/", "title": "MBart and MBart-50", "tokens": 2452, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"2e488e56-1ba9-4575-99fa-f83a3061ac6a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mbart/\", \"title\": \"MBart and MBart-50\", \"tokens\": 2452, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4eaabdc9-104a-4d67-90ec-c68b442af250\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mbart/\", \"title\": \"MBart and MBart-50\", \"tokens\": 2452, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b1d5932ea9bed5df529ac53dfdc450a26f301a5abbf9a203bf33f1f23e627621\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4c1818e6-5a0b-4d9d-830b-581fdb3a689b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mbart/\", \"title\": \"MBart and MBart-50\", \"tokens\": 2452, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"2b5aa08a06181bedee7408ee7a2d1c5699a7ef8fbdd3c6626fa3309965d86545\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"90aed525-49ef-4a5c-9eab-efd72f114f44\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4346abfd61c417c7387c8b4b974d86dcdcb2d0fe3045e27dc26fb974396f5007\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4379, \"end_char_idx\": 7071, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4eaabdc9-104a-4d67-90ec-c68b442af250", "doc_id": "4eaabdc9-104a-4d67-90ec-c68b442af250", "ref_doc_id": "4eaabdc9-104a-4d67-90ec-c68b442af250", "chroma:document": "*\n\n\n### Training of MBart-50\n\nThe text format for MBart-50 is slightly different from mBART. For MBart-50 the language id token is used as a prefix\nfor both source and target text i.e the text format is `[lang_code] X [eos]`, where `lang_code` is source\nlanguage id for source text and target language id for target text, with `X` being the source or target text\nrespectively.\n\n\nMBart-50 has its own tokenizer [`MBart50Tokenizer`].\n\n-  Supervised training\n\n```python\nfrom transformers import MBartForConditionalGeneration, MBart50TokenizerFast\n\nmodel = MBartForConditionalGeneration.from_pretrained(\"facebook/mbart-large-50\")\ntokenizer = MBart50TokenizerFast.from_pretrained(\"facebook/mbart-large-50\", src_lang=\"en_XX\", tgt_lang=\"ro_RO\")\n\nsrc_text = \" UN Chief Says There Is No Military Solution in Syria\"\ntgt_text = \"\u015eeful ONU declar\u0103 c\u0103 nu exist\u0103 o solu\u0163ie militar\u0103 \u00een Siria\"\n\nmodel_inputs = tokenizer(src_text, text_target=tgt_text, return_tensors=\"pt\")\n\nmodel(**model_inputs)  # forward pass\n```\n\n- Generation\n\n  To generate using the mBART-50 multilingual translation models, `eos_token_id` is used as the\n  `decoder_start_token_id` and the target language id is forced as the first generated token. To force the\n  target language id as the first generated token, pass the *forced_bos_token_id* parameter to the *generate* method.\n  The following example shows how to translate between Hindi to French and Arabic to English using the\n  *facebook/mbart-50-large-many-to-many* checkpoint.\n\n```python\nfrom transformers import MBartForConditionalGeneration, MBart50TokenizerFast\n\narticle_hi = \"\u0938\u0902\u092f\u0941\u0915\u094d\u0924 \u0930\u093e\u0937\u094d\u091f\u094d\u0930 \u0915\u0947 \u092a\u094d\u0930\u092e\u0941\u0916 \u0915\u093e \u0915\u0939\u0928\u093e \u0939\u0948 \u0915\u093f \u0938\u0940\u0930\u093f\u092f\u093e \u092e\u0947\u0902 \u0915\u094b\u0908 \u0938\u0948\u0928\u094d\u092f \u0938\u092e\u093e\u0927\u093e\u0928 \u0928\u0939\u0940\u0902 \u0939\u0948\"\narticle_ar = \"\u0627\u0644\u0623\u0645\u064a\u0646 \u0627\u0644\u0639\u0627\u0645 \u0644\u0644\u0623\u0645\u0645 \u0627\u0644\u0645\u062a\u062d\u062f\u0629 \u064a\u0642\u0648\u0644 \u0625\u0646\u0647 \u0644\u0627 \u064a\u0648\u062c\u062f \u062d\u0644 \u0639\u0633\u0643\u0631\u064a \u0641\u064a \u0633\u0648\u0631\u064a\u0627.\"\n\nmodel = MBartForConditionalGeneration.from_pretrained(\"facebook/mbart-large-50-many-to-many-mmt\")\ntokenizer = MBart50TokenizerFast.from_pretrained(\"facebook/mbart-large-50-many-to-many-mmt\")\n\n# translate Hindi to French\ntokenizer.src_lang = \"hi_IN\"\nencoded_hi = tokenizer(article_hi, return_tensors=\"pt\")\ngenerated_tokens = model.generate(**encoded_hi, forced_bos_token_id=tokenizer.lang_code_to_id[\"fr_XX\"])\ntokenizer.batch_decode(generated_tokens, skip_special_tokens=True)\n# => \"Le chef de l 'ONU affirme qu 'il n 'y a pas de solution militaire en Syria.\"\n\n# translate Arabic to English\ntokenizer.src_lang = \"ar_AR\"\nencoded_ar = tokenizer(article_ar, return_tensors=\"pt\")\ngenerated_tokens = model.generate(**encoded_ar, forced_bos_token_id=tokenizer.lang_code_to_id[\"en_XX\"])\ntokenizer.batch_decode(generated_tokens, skip_special_tokens=True)\n# => \"The Secretary-General of the United Nations says there is no military solution in Syria.\""}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 90aed525-49ef-4a5c-9eab-efd72f114f44
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mbart/", "title": "MBart and MBart-50", "tokens": 2452, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"90aed525-49ef-4a5c-9eab-efd72f114f44\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mbart/\", \"title\": \"MBart and MBart-50\", \"tokens\": 2452, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4eaabdc9-104a-4d67-90ec-c68b442af250\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mbart/\", \"title\": \"MBart and MBart-50\", \"tokens\": 2452, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b1d5932ea9bed5df529ac53dfdc450a26f301a5abbf9a203bf33f1f23e627621\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2e488e56-1ba9-4575-99fa-f83a3061ac6a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mbart/\", \"title\": \"MBart and MBart-50\", \"tokens\": 2452, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a8cf37e26b1f4c7c0a311b264b2a3c30b5ecd5a7e8d671fc50439b51763c4cd1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5717, \"end_char_idx\": 8786, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4eaabdc9-104a-4d67-90ec-c68b442af250", "doc_id": "4eaabdc9-104a-4d67-90ec-c68b442af250", "ref_doc_id": "4eaabdc9-104a-4d67-90ec-c68b442af250", "chroma:document": "The following example shows how to translate between Hindi to French and Arabic to English using the\n  *facebook/mbart-50-large-many-to-many* checkpoint.\n\n```python\nfrom transformers import MBartForConditionalGeneration, MBart50TokenizerFast\n\narticle_hi = \"\u0938\u0902\u092f\u0941\u0915\u094d\u0924 \u0930\u093e\u0937\u094d\u091f\u094d\u0930 \u0915\u0947 \u092a\u094d\u0930\u092e\u0941\u0916 \u0915\u093e \u0915\u0939\u0928\u093e \u0939\u0948 \u0915\u093f \u0938\u0940\u0930\u093f\u092f\u093e \u092e\u0947\u0902 \u0915\u094b\u0908 \u0938\u0948\u0928\u094d\u092f \u0938\u092e\u093e\u0927\u093e\u0928 \u0928\u0939\u0940\u0902 \u0939\u0948\"\narticle_ar = \"\u0627\u0644\u0623\u0645\u064a\u0646 \u0627\u0644\u0639\u0627\u0645 \u0644\u0644\u0623\u0645\u0645 \u0627\u0644\u0645\u062a\u062d\u062f\u0629 \u064a\u0642\u0648\u0644 \u0625\u0646\u0647 \u0644\u0627 \u064a\u0648\u062c\u062f \u062d\u0644 \u0639\u0633\u0643\u0631\u064a \u0641\u064a \u0633\u0648\u0631\u064a\u0627.\"\n\nmodel = MBartForConditionalGeneration.from_pretrained(\"facebook/mbart-large-50-many-to-many-mmt\")\ntokenizer = MBart50TokenizerFast.from_pretrained(\"facebook/mbart-large-50-many-to-many-mmt\")\n\n# translate Hindi to French\ntokenizer.src_lang = \"hi_IN\"\nencoded_hi = tokenizer(article_hi, return_tensors=\"pt\")\ngenerated_tokens = model.generate(**encoded_hi, forced_bos_token_id=tokenizer.lang_code_to_id[\"fr_XX\"])\ntokenizer.batch_decode(generated_tokens, skip_special_tokens=True)\n# => \"Le chef de l 'ONU affirme qu 'il n 'y a pas de solution militaire en Syria.\"\n\n# translate Arabic to English\ntokenizer.src_lang = \"ar_AR\"\nencoded_ar = tokenizer(article_ar, return_tensors=\"pt\")\ngenerated_tokens = model.generate(**encoded_ar, forced_bos_token_id=tokenizer.lang_code_to_id[\"en_XX\"])\ntokenizer.batch_decode(generated_tokens, skip_special_tokens=True)\n# => \"The Secretary-General of the United Nations says there is no military solution in Syria.\"\n```\n\n## Documentation resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n## MBartConfig\n\n[[autodoc]] MBartConfig\n\n## MBartTokenizer\n\n[[autodoc]] MBartTokenizer\n    - build_inputs_with_special_tokens\n\n## MBartTokenizerFast\n\n[[autodoc]] MBartTokenizerFast\n\n## MBart50Tokenizer\n\n[[autodoc]] MBart50Tokenizer\n\n## MBart50TokenizerFast\n\n[[autodoc]] MBart50TokenizerFast\n\n<frameworkcontent>\n<pt>\n\n## MBartModel\n\n[[autodoc]] MBartModel\n\n## MBartForConditionalGeneration\n\n[[autodoc]] MBartForConditionalGeneration\n\n## MBartForQuestionAnswering\n\n[[autodoc]] MBartForQuestionAnswering\n\n## MBartForSequenceClassification\n\n[[autodoc]] MBartForSequenceClassification\n\n## MBartForCausalLM\n\n[[autodoc]] MBartForCausalLM\n    - forward\n\n</pt>\n<tf>\n\n## TFMBartModel\n\n[[autodoc]] TFMBartModel\n    - call\n\n## TFMBartForConditionalGeneration\n\n[[autodoc]] TFMBartForConditionalGeneration\n    - call\n\n</tf>\n<jax>\n\n## FlaxMBartModel\n\n[[autodoc]] FlaxMBartModel\n    - __call__\n    - encode\n    - decode\n\n## FlaxMBartForConditionalGeneration\n\n[[autodoc]] FlaxMBartForConditionalGeneration\n    - __call__\n    - encode\n    - decode\n\n## FlaxMBartForSequenceClassification\n\n[[autodoc]] FlaxMBartForSequenceClassification\n    - __call__\n    - encode\n    - decode\n\n## FlaxMBartForQuestionAnswering\n\n[[autodoc]] FlaxMBartForQuestionAnswering\n    - __call__\n    - encode\n    - decode\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 81f1b70a-0c84-44aa-af17-6c505c0b0270
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/whisper/", "title": "Whisper", "tokens": 2030, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"81f1b70a-0c84-44aa-af17-6c505c0b0270\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/whisper/\", \"title\": \"Whisper\", \"tokens\": 2030, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8ee7081d-3551-415f-99a9-89ab44e495f8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/whisper/\", \"title\": \"Whisper\", \"tokens\": 2030, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"d15fbb51f93c9e4642ead83896dd73fdeb7d98928a383850bc61b54c1bc3f933\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"07b48d94-e46e-4cd1-9fb2-5d9fef3fefda\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7de80ccb92d34fc122a27f2f31c89b8c5ca764dcae0db3e1221a6aa5599c27d5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3543, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8ee7081d-3551-415f-99a9-89ab44e495f8", "doc_id": "8ee7081d-3551-415f-99a9-89ab44e495f8", "ref_doc_id": "8ee7081d-3551-415f-99a9-89ab44e495f8", "chroma:document": "# Whisper\n\n## Overview\n\nThe Whisper model was proposed in [Robust Speech Recognition via Large-Scale Weak Supervision](https://cdn.openai.com/papers/whisper.pdf) by Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, Ilya Sutskever.\n\nThe abstract from the paper is the following:\n\n*We study the capabilities of speech processing systems trained simply to predict large amounts of transcripts of audio on the internet. When scaled to 680,000 hours of multilingual and multitask supervision, the resulting models generalize well to standard benchmarks and are often competitive with prior fully supervised results but in a zeroshot transfer setting without the need for any finetuning. When compared to humans, the models approach their accuracy and robustness. We are releasing models and inference code to serve as a foundation for further work on robust speech processing.*\n\nThis model was contributed by [Arthur Zucker](https://huggingface.co/ArthurZ). The Tensorflow version of this model was contributed by [amyeroberts](https://huggingface.co/amyeroberts).\nThe original code can be found [here](https://github.com/openai/whisper).\n\n## Usage tips\n\n- The model usually performs well without requiring any finetuning.\n- The architecture follows a classic encoder-decoder architecture, which means that it relies on the [`~generation.GenerationMixin.generate`] function for inference.\n- One can use [`WhisperProcessor`] to prepare audio for the model, and decode the predicted ID's back into text.\n\n- To convert the model and the processor, we recommend using the following:\n\n```bash\npython src/transformers/models/whisper/convert_openai_to_hf.py --checkpoint_path \"\" --pytorch_dump_folder_path \"Arthur/whisper-3\" --convert_preprocessor True\n```\nThe script will automatically determine all necessary parameters from the OpenAI checkpoint. A `tiktoken` library needs to be installed\nto perform the conversion of the OpenAI tokenizer to the `tokenizers` version.\n\n## Inference\n\nHere is a step-by-step guide to transcribing an audio sample using a pre-trained Whisper model:\n\n```python\n>>> from datasets import load_dataset\n>>> from transformers import WhisperProcessor, WhisperForConditionalGeneration\n\n>>> # Select an audio file and read it:\n>>> ds = load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\n>>> audio_sample = ds[0][\"audio\"]\n\n>>> # Load the Whisper model in Hugging Face format:\n>>> processor = WhisperProcessor.from_pretrained(\"openai/whisper-tiny.en\")\n>>> model = WhisperForConditionalGeneration.from_pretrained(\"openai/whisper-tiny.en\")\n\n>>> # Use the model and processor to transcribe the audio:\n>>> input_features = processor(\n...     audio_sample[\"array\"], sampling_rate=audio_sample[\"sampling_rate\"], return_tensors=\"pt\"\n... ).input_features\n\n>>> # Generate token ids\n>>> predicted_ids = model.generate(input_features)\n\n>>> # Decode token ids to text\n>>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)\n\n>>> transcription[0]\n' Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.'\n```\n\nWhisper is compatible with the following optimisations:\n- [PyTorch Scaled Dot Product Attention (SDPA)](../perf_infer_gpu_one#pytorch-scaled-dot-product-attention): flash attention and memory-efficient attention kernels. Enabled by default for `torch>=2.1.1`.\n- [Flash Attention 2](../perf_infer_gpu_one#flashattention-2): improved implementation of flash attention through better parallelism and work partitioning."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 07b48d94-e46e-4cd1-9fb2-5d9fef3fefda
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/whisper/", "title": "Whisper", "tokens": 2030, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"07b48d94-e46e-4cd1-9fb2-5d9fef3fefda\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/whisper/\", \"title\": \"Whisper\", \"tokens\": 2030, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8ee7081d-3551-415f-99a9-89ab44e495f8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/whisper/\", \"title\": \"Whisper\", \"tokens\": 2030, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"d15fbb51f93c9e4642ead83896dd73fdeb7d98928a383850bc61b54c1bc3f933\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"81f1b70a-0c84-44aa-af17-6c505c0b0270\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/whisper/\", \"title\": \"Whisper\", \"tokens\": 2030, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"d077dfdd690533c98255a7073d25d0564d9cd6cfa95ace4d0773721737ffb12b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e7790a4a-19b3-403b-82fe-2f288985a1e1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e452d31f7b5859347406ff70bd08a31d376ff972fa38f1213697ae39d6231dda\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1860, \"end_char_idx\": 5356, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8ee7081d-3551-415f-99a9-89ab44e495f8", "doc_id": "8ee7081d-3551-415f-99a9-89ab44e495f8", "ref_doc_id": "8ee7081d-3551-415f-99a9-89ab44e495f8", "chroma:document": "A `tiktoken` library needs to be installed\nto perform the conversion of the OpenAI tokenizer to the `tokenizers` version.\n\n## Inference\n\nHere is a step-by-step guide to transcribing an audio sample using a pre-trained Whisper model:\n\n```python\n>>> from datasets import load_dataset\n>>> from transformers import WhisperProcessor, WhisperForConditionalGeneration\n\n>>> # Select an audio file and read it:\n>>> ds = load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\n>>> audio_sample = ds[0][\"audio\"]\n\n>>> # Load the Whisper model in Hugging Face format:\n>>> processor = WhisperProcessor.from_pretrained(\"openai/whisper-tiny.en\")\n>>> model = WhisperForConditionalGeneration.from_pretrained(\"openai/whisper-tiny.en\")\n\n>>> # Use the model and processor to transcribe the audio:\n>>> input_features = processor(\n...     audio_sample[\"array\"], sampling_rate=audio_sample[\"sampling_rate\"], return_tensors=\"pt\"\n... ).input_features\n\n>>> # Generate token ids\n>>> predicted_ids = model.generate(input_features)\n\n>>> # Decode token ids to text\n>>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)\n\n>>> transcription[0]\n' Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.'\n```\n\nWhisper is compatible with the following optimisations:\n- [PyTorch Scaled Dot Product Attention (SDPA)](../perf_infer_gpu_one#pytorch-scaled-dot-product-attention): flash attention and memory-efficient attention kernels. Enabled by default for `torch>=2.1.1`.\n- [Flash Attention 2](../perf_infer_gpu_one#flashattention-2): improved implementation of flash attention through better parallelism and work partitioning. \n- [torch.compile](../llm_optims#static-kv-cache-and-torchcompile): JIT-compile the forward pass to dispatch to efficient fused kernels.\n\nAs an example, the following codesnippet enables SDPA and `torch.compile` for up to 5x faster inference:\n\n```python\n>>> from datasets import load_dataset\n>>> from transformers import WhisperProcessor, WhisperForConditionalGeneration\n\n>>> # Select an audio file and read it:\n>>> ds = load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\n>>> audio_sample = ds[0][\"audio\"]\n\n>>> # Load the Whisper model with SDPA attention\n>>> processor = WhisperProcessor.from_pretrained(\"openai/whisper-tiny.en\")\n>>> model = WhisperForConditionalGeneration.from_pretrained(\"openai/whisper-tiny.en\", attn_implementation=\"sdpa\")\n\n>>> # Enable static cache and compile the forward pass\n>>> model.generation_config.cache_implementation = \"static\"\n>>> model.forward = torch.compile(model.forward, mode=\"reduce-overhead\", fullgraph=True)\n\n>>> # Use the model and processor to transcribe the audio:\n>>> input_features = processor(\n...     audio_sample[\"array\"], sampling_rate=audio_sample[\"sampling_rate\"], return_tensors=\"pt\"\n... ).input_features\n\n>>> # Compile the forward pass\n>>> _ = model.generate(input_features)\n\n>>> # Generate token ids using compiled graph (fast!)\n>>> predicted_ids = model.generate(input_features)\n\n>>> # Decode token ids to text\n>>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)\n\n>>> transcription[0]\n' Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.'\n```\n\nFor more details on each optimisation, refer to the documentation linked above.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Whisper."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: e7790a4a-19b3-403b-82fe-2f288985a1e1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/whisper/", "title": "Whisper", "tokens": 2030, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"e7790a4a-19b3-403b-82fe-2f288985a1e1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/whisper/\", \"title\": \"Whisper\", \"tokens\": 2030, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8ee7081d-3551-415f-99a9-89ab44e495f8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/whisper/\", \"title\": \"Whisper\", \"tokens\": 2030, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"d15fbb51f93c9e4642ead83896dd73fdeb7d98928a383850bc61b54c1bc3f933\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"07b48d94-e46e-4cd1-9fb2-5d9fef3fefda\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/whisper/\", \"title\": \"Whisper\", \"tokens\": 2030, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"e86e4ddb13a7140cd81936582564d56c7c4ae22aec2ab3bf93f099243018dbb3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"33a736f6-2a7b-49d0-be7b-e9fe5b18d6ab\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"92efc6becf9843ea07cb93a5f1260f8908068d112a667e6e2178ddbf707b2ed6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3682, \"end_char_idx\": 6199, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8ee7081d-3551-415f-99a9-89ab44e495f8", "doc_id": "8ee7081d-3551-415f-99a9-89ab44e495f8", "ref_doc_id": "8ee7081d-3551-415f-99a9-89ab44e495f8", "chroma:document": "As an example, the following codesnippet enables SDPA and `torch.compile` for up to 5x faster inference:\n\n```python\n>>> from datasets import load_dataset\n>>> from transformers import WhisperProcessor, WhisperForConditionalGeneration\n\n>>> # Select an audio file and read it:\n>>> ds = load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\n>>> audio_sample = ds[0][\"audio\"]\n\n>>> # Load the Whisper model with SDPA attention\n>>> processor = WhisperProcessor.from_pretrained(\"openai/whisper-tiny.en\")\n>>> model = WhisperForConditionalGeneration.from_pretrained(\"openai/whisper-tiny.en\", attn_implementation=\"sdpa\")\n\n>>> # Enable static cache and compile the forward pass\n>>> model.generation_config.cache_implementation = \"static\"\n>>> model.forward = torch.compile(model.forward, mode=\"reduce-overhead\", fullgraph=True)\n\n>>> # Use the model and processor to transcribe the audio:\n>>> input_features = processor(\n...     audio_sample[\"array\"], sampling_rate=audio_sample[\"sampling_rate\"], return_tensors=\"pt\"\n... ).input_features\n\n>>> # Compile the forward pass\n>>> _ = model.generate(input_features)\n\n>>> # Generate token ids using compiled graph (fast!)\n>>> predicted_ids = model.generate(input_features)\n\n>>> # Decode token ids to text\n>>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)\n\n>>> transcription[0]\n' Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.'\n```\n\nFor more details on each optimisation, refer to the documentation linked above.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Whisper. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n- [Fine-tune Whisper](https://huggingface.co/blog/fine-tune-whisper) on your own dataset for better downstream performance.\n- [Distil-Whisper](https://huggingface.co/distil-whisper): Upto 6x faster, 2x smaller distilled Whisper models for English. We release the [model checkpoints](https://huggingface.co/distil-whisper), and [distillation code](https://github.com/huggingface/distil-whisper).\n- A fork with a script to [convert a Whisper model in Hugging Face format to OpenAI format](https://github.com/zuazo-forks/transformers/blob/convert_hf_to_openai/src/transformers/models/whisper/convert_hf_to_openai.py)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 33a736f6-2a7b-49d0-be7b-e9fe5b18d6ab
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/whisper/", "title": "Whisper", "tokens": 2030, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"33a736f6-2a7b-49d0-be7b-e9fe5b18d6ab\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/whisper/\", \"title\": \"Whisper\", \"tokens\": 2030, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8ee7081d-3551-415f-99a9-89ab44e495f8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/whisper/\", \"title\": \"Whisper\", \"tokens\": 2030, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"d15fbb51f93c9e4642ead83896dd73fdeb7d98928a383850bc61b54c1bc3f933\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e7790a4a-19b3-403b-82fe-2f288985a1e1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/whisper/\", \"title\": \"Whisper\", \"tokens\": 2030, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"925a6687679f011c7c8946a7cd65491def380b9a16ae85eb80abdb947f64c848\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4865, \"end_char_idx\": 8066, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8ee7081d-3551-415f-99a9-89ab44e495f8", "doc_id": "8ee7081d-3551-415f-99a9-89ab44e495f8", "ref_doc_id": "8ee7081d-3551-415f-99a9-89ab44e495f8", "chroma:document": ">>> predicted_ids = model.generate(input_features)\n\n>>> # Decode token ids to text\n>>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)\n\n>>> transcription[0]\n' Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.'\n```\n\nFor more details on each optimisation, refer to the documentation linked above.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Whisper. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n- [Fine-tune Whisper](https://huggingface.co/blog/fine-tune-whisper) on your own dataset for better downstream performance.\n- [Distil-Whisper](https://huggingface.co/distil-whisper): Upto 6x faster, 2x smaller distilled Whisper models for English. We release the [model checkpoints](https://huggingface.co/distil-whisper), and [distillation code](https://github.com/huggingface/distil-whisper).\n- A fork with a script to [convert a Whisper model in Hugging Face format to OpenAI format](https://github.com/zuazo-forks/transformers/blob/convert_hf_to_openai/src/transformers/models/whisper/convert_hf_to_openai.py). \ud83c\udf0e\nUsage example:\n```bash\npip install -U openai-whisper\npython convert_hf_to_openai.py \\\n    --checkpoint openai/whisper-tiny \\\n    --whisper_dump_path whisper-tiny-openai.pt\n```\n\n## WhisperConfig\n\n[[autodoc]] WhisperConfig\n\n## WhisperTokenizer\n\n[[autodoc]] WhisperTokenizer\n    - set_prefix_tokens\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n    - batch_decode\n    - decode\n    - basic_normalize\n    - normalize\n\n## WhisperTokenizerFast\n\n[[autodoc]] WhisperTokenizerFast\n    - set_prefix_tokens\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n    - batch_decode\n    - decode\n    - basic_normalize\n    - normalize\n\n## WhisperFeatureExtractor\n\n[[autodoc]] WhisperFeatureExtractor\n    - __call__\n\n## WhisperProcessor\n\n[[autodoc]] WhisperProcessor\n    - __call__\n    - from_pretrained\n    - save_pretrained\n    - batch_decode\n    - decode\n\n<frameworkcontent>\n<pt>\n\n## WhisperModel\n\n[[autodoc]] WhisperModel\n    - forward\n    - _mask_input_features\n\n## WhisperForConditionalGeneration\n\n[[autodoc]] WhisperForConditionalGeneration\n    - forward\n    - generate\n\n## WhisperForCausalLM\n\n[[autodoc]] WhisperForCausalLM\n    - forward\n\n## WhisperForAudioClassification\n\n[[autodoc]] WhisperForAudioClassification\n    - forward\n\n</pt>\n<tf>\n\n## TFWhisperModel\n\n[[autodoc]] TFWhisperModel\n    - call\n\n## TFWhisperForConditionalGeneration\n\n[[autodoc]] TFWhisperForConditionalGeneration\n    - call\n\n</tf>\n<jax>\n\n## FlaxWhisperModel\n\n[[autodoc]] FlaxWhisperModel\n    - __call__\n\n## FlaxWhisperForConditionalGeneration\n\n[[autodoc]] FlaxWhisperForConditionalGeneration\n    - __call__\n\n## FlaxWhisperForAudioClassification\n\n[[autodoc]] FlaxWhisperForAudioClassification\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: dbde2f0e-f6aa-49ed-9fe9-1a84b7c3f453
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv3/", "title": "LayoutLMv3", "tokens": 2018, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"dbde2f0e-f6aa-49ed-9fe9-1a84b7c3f453\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv3/\", \"title\": \"LayoutLMv3\", \"tokens\": 2018, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52fec52e-5049-4577-be8b-d75378fd9b9e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv3/\", \"title\": \"LayoutLMv3\", \"tokens\": 2018, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"3fb4b688c89a531475ebcf232aabc69d0f36e52207859ac6c829307d0e15f0f9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"381fc445-eaa7-408b-b23e-7370e99e6d15\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"dffe5e2e6b910278b431176684075a98e64351ae11c1fbe3912ec5934959ba0a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3255, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52fec52e-5049-4577-be8b-d75378fd9b9e", "doc_id": "52fec52e-5049-4577-be8b-d75378fd9b9e", "ref_doc_id": "52fec52e-5049-4577-be8b-d75378fd9b9e", "chroma:document": "# LayoutLMv3\n\n## Overview\n\nThe LayoutLMv3 model was proposed in [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://arxiv.org/abs/2204.08387) by Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, Furu Wei.\nLayoutLMv3 simplifies [LayoutLMv2](layoutlmv2) by using patch embeddings (as in [ViT](vit)) instead of leveraging a CNN backbone, and pre-trains the model on 3 objectives: masked language modeling (MLM), masked image modeling (MIM)\nand word-patch alignment (WPA).\n\nThe abstract from the paper is the following:\n\n*Self-supervised pre-training techniques have achieved remarkable progress in Document AI. Most multimodal pre-trained models use a masked language modeling objective to learn bidirectional representations on the text modality, but they differ in pre-training objectives for the image modality. This discrepancy adds difficulty to multimodal representation learning. In this paper, we propose LayoutLMv3 to pre-train multimodal Transformers for Document AI with unified text and image masking. Additionally, LayoutLMv3 is pre-trained with a word-patch alignment objective to learn cross-modal alignment by predicting whether the corresponding image patch of a text word is masked. The simple unified architecture and training objectives make LayoutLMv3 a general-purpose pre-trained model for both text-centric and image-centric Document AI tasks. Experimental results show that LayoutLMv3 achieves state-of-the-art performance not only in text-centric tasks, including form understanding, receipt understanding, and document visual question answering, but also in image-centric tasks such as document image classification and document layout analysis.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/layoutlmv3_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> LayoutLMv3 architecture. Taken from the <a href=\"https://arxiv.org/abs/2204.08387\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The TensorFlow version of this model was added by [chriskoo](https://huggingface.co/chriskoo), [tokec](https://huggingface.co/tokec), and [lre](https://huggingface.co/lre). The original code can be found [here](https://github.com/microsoft/unilm/tree/master/layoutlmv3).\n\n## Usage tips\n\n- In terms of data processing, LayoutLMv3 is identical to its predecessor [LayoutLMv2](layoutlmv2), except that:\n    - images need to be resized and normalized with channels in regular RGB format. LayoutLMv2 on the other hand normalizes the images internally and expects the channels in BGR format.\n    - text is tokenized using byte-pair encoding (BPE), as opposed to WordPiece.\n  Due to these differences in data preprocessing, one can use [`LayoutLMv3Processor`] which internally combines a [`LayoutLMv3ImageProcessor`] (for the image modality) and a [`LayoutLMv3Tokenizer`]/[`LayoutLMv3TokenizerFast`] (for the text modality) to prepare all data for the model.\n- Regarding usage of [`LayoutLMv3Processor`], we refer to the [usage guide](layoutlmv2#usage-layoutlmv2processor) of its predecessor.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with LayoutLMv3."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 381fc445-eaa7-408b-b23e-7370e99e6d15
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv3/", "title": "LayoutLMv3", "tokens": 2018, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"381fc445-eaa7-408b-b23e-7370e99e6d15\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv3/\", \"title\": \"LayoutLMv3\", \"tokens\": 2018, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52fec52e-5049-4577-be8b-d75378fd9b9e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv3/\", \"title\": \"LayoutLMv3\", \"tokens\": 2018, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"3fb4b688c89a531475ebcf232aabc69d0f36e52207859ac6c829307d0e15f0f9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dbde2f0e-f6aa-49ed-9fe9-1a84b7c3f453\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv3/\", \"title\": \"LayoutLMv3\", \"tokens\": 2018, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"faf926c7788f1a0b1b4599833d856ba61f4968d392c4f25f5023be6af4b8b487\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fde0d019-258f-4000-a687-3fe54167ae3a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1aae3006127329e62b24f53eafb906414bce3f9146f9bacb111d67e0a09c6031\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1882, \"end_char_idx\": 4747, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52fec52e-5049-4577-be8b-d75378fd9b9e", "doc_id": "52fec52e-5049-4577-be8b-d75378fd9b9e", "ref_doc_id": "52fec52e-5049-4577-be8b-d75378fd9b9e", "chroma:document": "Taken from the <a href=\"https://arxiv.org/abs/2204.08387\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The TensorFlow version of this model was added by [chriskoo](https://huggingface.co/chriskoo), [tokec](https://huggingface.co/tokec), and [lre](https://huggingface.co/lre). The original code can be found [here](https://github.com/microsoft/unilm/tree/master/layoutlmv3).\n\n## Usage tips\n\n- In terms of data processing, LayoutLMv3 is identical to its predecessor [LayoutLMv2](layoutlmv2), except that:\n    - images need to be resized and normalized with channels in regular RGB format. LayoutLMv2 on the other hand normalizes the images internally and expects the channels in BGR format.\n    - text is tokenized using byte-pair encoding (BPE), as opposed to WordPiece.\n  Due to these differences in data preprocessing, one can use [`LayoutLMv3Processor`] which internally combines a [`LayoutLMv3ImageProcessor`] (for the image modality) and a [`LayoutLMv3Tokenizer`]/[`LayoutLMv3TokenizerFast`] (for the text modality) to prepare all data for the model.\n- Regarding usage of [`LayoutLMv3Processor`], we refer to the [usage guide](layoutlmv2#usage-layoutlmv2processor) of its predecessor.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with LayoutLMv3. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<Tip>\n\nLayoutLMv3 is nearly identical to LayoutLMv2, so we've also included LayoutLMv2 resources you can adapt for LayoutLMv3 tasks. For these notebooks, take care to use [`LayoutLMv2Processor`] instead when preparing data for the model!\n\n</Tip>\n\n- Demo notebooks for LayoutLMv3 can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/LayoutLMv3).\n- Demo scripts can be found [here](https://github.com/huggingface/transformers/tree/main/examples/research_projects/layoutlmv3).\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- [`LayoutLMv2ForSequenceClassification`] is supported by this [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/RVL-CDIP/Fine_tuning_LayoutLMv2ForSequenceClassification_on_RVL_CDIP.ipynb).\n- [Text classification task guide](../tasks/sequence_classification)\n\n<PipelineTag pipeline=\"token-classification\"/>\n\n- [`LayoutLMv3ForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/research_projects/layoutlmv3) and [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv3/Fine_tune_LayoutLMv3_on_FUNSD_(HuggingFace_Trainer).ipynb)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: fde0d019-258f-4000-a687-3fe54167ae3a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv3/", "title": "LayoutLMv3", "tokens": 2018, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"fde0d019-258f-4000-a687-3fe54167ae3a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv3/\", \"title\": \"LayoutLMv3\", \"tokens\": 2018, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52fec52e-5049-4577-be8b-d75378fd9b9e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv3/\", \"title\": \"LayoutLMv3\", \"tokens\": 2018, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"3fb4b688c89a531475ebcf232aabc69d0f36e52207859ac6c829307d0e15f0f9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"381fc445-eaa7-408b-b23e-7370e99e6d15\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv3/\", \"title\": \"LayoutLMv3\", \"tokens\": 2018, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"42db908d795850504f1d385921fcfd91a767679bc2e40e66c4900cbd7bc22036\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"90523dcd-2d5d-4be8-af23-0133abd6f82f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2488f3cc1b0f99cdce812989fb0d66b521eebe10dd9ced5cab542bbd25139b0c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3128, \"end_char_idx\": 5922, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52fec52e-5049-4577-be8b-d75378fd9b9e", "doc_id": "52fec52e-5049-4577-be8b-d75378fd9b9e", "ref_doc_id": "52fec52e-5049-4577-be8b-d75378fd9b9e", "chroma:document": "## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with LayoutLMv3. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<Tip>\n\nLayoutLMv3 is nearly identical to LayoutLMv2, so we've also included LayoutLMv2 resources you can adapt for LayoutLMv3 tasks. For these notebooks, take care to use [`LayoutLMv2Processor`] instead when preparing data for the model!\n\n</Tip>\n\n- Demo notebooks for LayoutLMv3 can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/LayoutLMv3).\n- Demo scripts can be found [here](https://github.com/huggingface/transformers/tree/main/examples/research_projects/layoutlmv3).\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- [`LayoutLMv2ForSequenceClassification`] is supported by this [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/RVL-CDIP/Fine_tuning_LayoutLMv2ForSequenceClassification_on_RVL_CDIP.ipynb).\n- [Text classification task guide](../tasks/sequence_classification)\n\n<PipelineTag pipeline=\"token-classification\"/>\n\n- [`LayoutLMv3ForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/research_projects/layoutlmv3) and [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv3/Fine_tune_LayoutLMv3_on_FUNSD_(HuggingFace_Trainer).ipynb).\n- A [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/FUNSD/Inference_with_LayoutLMv2ForTokenClassification.ipynb) for how to perform inference with [`LayoutLMv2ForTokenClassification`] and a [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/FUNSD/True_inference_with_LayoutLMv2ForTokenClassification_%2B_Gradio_demo.ipynb) for how to perform inference when no labels are available with [`LayoutLMv2ForTokenClassification`].\n- A [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/FUNSD/Fine_tuning_LayoutLMv2ForTokenClassification_on_FUNSD_using_HuggingFace_Trainer.ipynb) for how to finetune [`LayoutLMv2ForTokenClassification`] with the \ud83e\udd17 Trainer.\n- [Token classification task guide](../tasks/token_classification)\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- [`LayoutLMv2ForQuestionAnswering`] is supported by this [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/DocVQA/Fine_tuning_LayoutLMv2ForQuestionAnswering_on_DocVQA.ipynb)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 90523dcd-2d5d-4be8-af23-0133abd6f82f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv3/", "title": "LayoutLMv3", "tokens": 2018, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"90523dcd-2d5d-4be8-af23-0133abd6f82f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv3/\", \"title\": \"LayoutLMv3\", \"tokens\": 2018, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52fec52e-5049-4577-be8b-d75378fd9b9e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv3/\", \"title\": \"LayoutLMv3\", \"tokens\": 2018, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"3fb4b688c89a531475ebcf232aabc69d0f36e52207859ac6c829307d0e15f0f9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"fde0d019-258f-4000-a687-3fe54167ae3a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv3/\", \"title\": \"LayoutLMv3\", \"tokens\": 2018, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5d0939dfad3cdbbcfae2efad140183a39e20b11bb0d39358da7b58963366abe7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4748, \"end_char_idx\": 7346, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52fec52e-5049-4577-be8b-d75378fd9b9e", "doc_id": "52fec52e-5049-4577-be8b-d75378fd9b9e", "ref_doc_id": "52fec52e-5049-4577-be8b-d75378fd9b9e", "chroma:document": "- A [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/FUNSD/Inference_with_LayoutLMv2ForTokenClassification.ipynb) for how to perform inference with [`LayoutLMv2ForTokenClassification`] and a [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/FUNSD/True_inference_with_LayoutLMv2ForTokenClassification_%2B_Gradio_demo.ipynb) for how to perform inference when no labels are available with [`LayoutLMv2ForTokenClassification`].\n- A [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/FUNSD/Fine_tuning_LayoutLMv2ForTokenClassification_on_FUNSD_using_HuggingFace_Trainer.ipynb) for how to finetune [`LayoutLMv2ForTokenClassification`] with the \ud83e\udd17 Trainer.\n- [Token classification task guide](../tasks/token_classification)\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- [`LayoutLMv2ForQuestionAnswering`] is supported by this [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/DocVQA/Fine_tuning_LayoutLMv2ForQuestionAnswering_on_DocVQA.ipynb).\n- [Question answering task guide](../tasks/question_answering)\n\n**Document question answering**\n- [Document question answering task guide](../tasks/document_question_answering)\n\n## LayoutLMv3Config\n\n[[autodoc]] LayoutLMv3Config\n\n## LayoutLMv3FeatureExtractor\n\n[[autodoc]] LayoutLMv3FeatureExtractor\n    - __call__\n\n## LayoutLMv3ImageProcessor\n\n[[autodoc]] LayoutLMv3ImageProcessor\n    - preprocess\n\n## LayoutLMv3Tokenizer\n\n[[autodoc]] LayoutLMv3Tokenizer\n    - __call__\n    - save_vocabulary\n\n## LayoutLMv3TokenizerFast\n\n[[autodoc]] LayoutLMv3TokenizerFast\n    - __call__\n\n## LayoutLMv3Processor\n\n[[autodoc]] LayoutLMv3Processor\n    - __call__\n\n<frameworkcontent>\n<pt>\n\n## LayoutLMv3Model\n\n[[autodoc]] LayoutLMv3Model\n    - forward\n\n## LayoutLMv3ForSequenceClassification\n\n[[autodoc]] LayoutLMv3ForSequenceClassification\n    - forward\n\n## LayoutLMv3ForTokenClassification\n\n[[autodoc]] LayoutLMv3ForTokenClassification\n    - forward\n\n## LayoutLMv3ForQuestionAnswering\n\n[[autodoc]] LayoutLMv3ForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFLayoutLMv3Model\n\n[[autodoc]] TFLayoutLMv3Model\n    - call\n\n## TFLayoutLMv3ForSequenceClassification\n\n[[autodoc]] TFLayoutLMv3ForSequenceClassification\n    - call\n\n## TFLayoutLMv3ForTokenClassification\n\n[[autodoc]] TFLayoutLMv3ForTokenClassification\n    - call\n\n## TFLayoutLMv3ForQuestionAnswering\n\n[[autodoc]] TFLayoutLMv3ForQuestionAnswering\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 202862dd-d7ad-4e29-a542-9e0f4e800f44
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/deformable_detr/", "title": "Deformable DETR", "tokens": 934, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"202862dd-d7ad-4e29-a542-9e0f4e800f44\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deformable_detr/\", \"title\": \"Deformable DETR\", \"tokens\": 934, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7faf5277-dfb5-4297-abd2-ca25b05190b0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deformable_detr/\", \"title\": \"Deformable DETR\", \"tokens\": 934, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"fc9973e848fb897a4489f3254926fa635b3137ed06de6741237d8b0cdafcadf5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3267, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7faf5277-dfb5-4297-abd2-ca25b05190b0", "doc_id": "7faf5277-dfb5-4297-abd2-ca25b05190b0", "ref_doc_id": "7faf5277-dfb5-4297-abd2-ca25b05190b0", "chroma:document": "# Deformable DETR\n\n## Overview\n\nThe Deformable DETR model was proposed in [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://arxiv.org/abs/2010.04159) by Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, Jifeng Dai.\nDeformable DETR mitigates the slow convergence issues and limited feature spatial resolution of the original [DETR](detr) by leveraging a new deformable attention module which only attends to a small set of key sampling points around a reference.\n\nThe abstract from the paper is the following:\n\n*DETR has been recently proposed to eliminate the need for many hand-designed components in object detection while demonstrating good performance. However, it suffers from slow convergence and limited feature spatial resolution, due to the limitation of Transformer attention modules in processing image feature maps. To mitigate these issues, we proposed Deformable DETR, whose attention modules only attend to a small set of key sampling points around a reference. Deformable DETR can achieve better performance than DETR (especially on small objects) with 10 times less training epochs. Extensive experiments on the COCO benchmark demonstrate the effectiveness of our approach.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/deformable_detr_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Deformable DETR architecture. Taken from the <a href=\"https://arxiv.org/abs/2010.04159\">original paper</a>.</small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/fundamentalvision/Deformable-DETR).\n\n## Usage tips\n\n- Training Deformable DETR is equivalent to training the original [DETR](detr) model. See the [resources](#resources) section below for demo notebooks.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Deformable DETR.\n\n<PipelineTag pipeline=\"object-detection\"/>\n\n- Demo notebooks regarding inference + fine-tuning on a custom dataset for [`DeformableDetrForObjectDetection`] can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Deformable-DETR).\n- Scripts for finetuning [`DeformableDetrForObjectDetection`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/object-detection).\n- See also: [Object detection task guide](../tasks/object_detection).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## DeformableDetrImageProcessor\n\n[[autodoc]] DeformableDetrImageProcessor\n    - preprocess\n    - post_process_object_detection\n\n## DeformableDetrFeatureExtractor\n\n[[autodoc]] DeformableDetrFeatureExtractor\n    - __call__\n    - post_process_object_detection\n\n## DeformableDetrConfig\n\n[[autodoc]] DeformableDetrConfig\n\n## DeformableDetrModel\n\n[[autodoc]] DeformableDetrModel\n    - forward\n\n## DeformableDetrForObjectDetection\n\n[[autodoc]] DeformableDetrForObjectDetection\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 3bf9ee9d-3eb3-4a8a-9bac-2a70589068e1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/blip/", "title": "BLIP", "tokens": 1084, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"3bf9ee9d-3eb3-4a8a-9bac-2a70589068e1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blip/\", \"title\": \"BLIP\", \"tokens\": 1084, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"894905c4-afa3-49da-8570-ebbc790ffd67\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blip/\", \"title\": \"BLIP\", \"tokens\": 1084, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"e0a3e947dab9053c93e8ba7e1290e00212d529121769afc321ff8fd83470a8af\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0285f15d-2aa7-4ebf-9802-0d27c9a62348\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7061fb6d69c3beec448cbbb28a29e6878f2b3e380e1f2312c4f5a95746853acf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2492, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "894905c4-afa3-49da-8570-ebbc790ffd67", "doc_id": "894905c4-afa3-49da-8570-ebbc790ffd67", "ref_doc_id": "894905c4-afa3-49da-8570-ebbc790ffd67", "chroma:document": "# BLIP\n\n## Overview\n\nThe BLIP model was proposed in [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://arxiv.org/abs/2201.12086) by Junnan Li, Dongxu Li, Caiming Xiong, Steven Hoi.\n\nBLIP is a model that is able to perform various multi-modal tasks including:\n- Visual Question Answering \n- Image-Text retrieval (Image-text matching)\n- Image Captioning\n\nThe abstract from the paper is the following:\n\n*Vision-Language Pre-training (VLP) has advanced the performance for many vision-language tasks. \nHowever, most existing pre-trained models only excel in either understanding-based tasks or generation-based tasks. Furthermore, performance improvement has been largely achieved by scaling up the dataset with noisy image-text pairs collected from the web, which is a suboptimal source of supervision. In this paper, we propose BLIP, a new VLP framework which transfers flexibly to both vision-language understanding and generation tasks. BLIP effectively utilizes the noisy web data by bootstrapping the captions, where a captioner generates synthetic captions and a filter removes the noisy ones. We achieve state-of-the-art results on a wide range of vision-language tasks, such as image-text retrieval (+2.7% in average recall@1), image captioning (+2.8% in CIDEr), and VQA (+1.6% in VQA score). BLIP also demonstrates strong generalization ability when directly transferred to videolanguage tasks in a zero-shot manner. Code, models, and datasets are released.*\n\n![BLIP.gif](https://cdn-uploads.huggingface.co/production/uploads/1670928184033-62441d1d9fdefb55a0b7d12c.gif)\n\nThis model was contributed by [ybelkada](https://huggingface.co/ybelkada).\nThe original code can be found [here](https://github.com/salesforce/BLIP).\n\n## Resources\n\n- [Jupyter notebook](https://github.com/huggingface/notebooks/blob/main/examples/image_captioning_blip.ipynb) on how to fine-tune BLIP for image captioning on a custom dataset\n\n## BlipConfig\n\n[[autodoc]] BlipConfig\n    - from_text_vision_configs\n\n## BlipTextConfig\n\n[[autodoc]] BlipTextConfig\n\n## BlipVisionConfig\n\n[[autodoc]] BlipVisionConfig\n\n## BlipProcessor\n\n[[autodoc]] BlipProcessor\n\n## BlipImageProcessor\n\n[[autodoc]] BlipImageProcessor\n    - preprocess\n\n<frameworkcontent>\n<pt>\n\n## BlipModel\n\n`BlipModel` is going to be deprecated in future versions, please use `BlipForConditionalGeneration`, `BlipForImageTextRetrieval` or `BlipForQuestionAnswering` depending on your usecase."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 0285f15d-2aa7-4ebf-9802-0d27c9a62348
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/blip/", "title": "BLIP", "tokens": 1084, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"0285f15d-2aa7-4ebf-9802-0d27c9a62348\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blip/\", \"title\": \"BLIP\", \"tokens\": 1084, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"894905c4-afa3-49da-8570-ebbc790ffd67\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blip/\", \"title\": \"BLIP\", \"tokens\": 1084, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"e0a3e947dab9053c93e8ba7e1290e00212d529121769afc321ff8fd83470a8af\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3bf9ee9d-3eb3-4a8a-9bac-2a70589068e1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blip/\", \"title\": \"BLIP\", \"tokens\": 1084, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"fdf0e4566630744497a57648e293f93a694bd7022d95d28d669164b257074ef8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1161, \"end_char_idx\": 3472, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "894905c4-afa3-49da-8570-ebbc790ffd67", "doc_id": "894905c4-afa3-49da-8570-ebbc790ffd67", "ref_doc_id": "894905c4-afa3-49da-8570-ebbc790ffd67", "chroma:document": "We achieve state-of-the-art results on a wide range of vision-language tasks, such as image-text retrieval (+2.7% in average recall@1), image captioning (+2.8% in CIDEr), and VQA (+1.6% in VQA score). BLIP also demonstrates strong generalization ability when directly transferred to videolanguage tasks in a zero-shot manner. Code, models, and datasets are released.*\n\n![BLIP.gif](https://cdn-uploads.huggingface.co/production/uploads/1670928184033-62441d1d9fdefb55a0b7d12c.gif)\n\nThis model was contributed by [ybelkada](https://huggingface.co/ybelkada).\nThe original code can be found [here](https://github.com/salesforce/BLIP).\n\n## Resources\n\n- [Jupyter notebook](https://github.com/huggingface/notebooks/blob/main/examples/image_captioning_blip.ipynb) on how to fine-tune BLIP for image captioning on a custom dataset\n\n## BlipConfig\n\n[[autodoc]] BlipConfig\n    - from_text_vision_configs\n\n## BlipTextConfig\n\n[[autodoc]] BlipTextConfig\n\n## BlipVisionConfig\n\n[[autodoc]] BlipVisionConfig\n\n## BlipProcessor\n\n[[autodoc]] BlipProcessor\n\n## BlipImageProcessor\n\n[[autodoc]] BlipImageProcessor\n    - preprocess\n\n<frameworkcontent>\n<pt>\n\n## BlipModel\n\n`BlipModel` is going to be deprecated in future versions, please use `BlipForConditionalGeneration`, `BlipForImageTextRetrieval` or `BlipForQuestionAnswering` depending on your usecase.\n\n[[autodoc]] BlipModel\n    - forward\n    - get_text_features\n    - get_image_features\n\n## BlipTextModel\n\n[[autodoc]] BlipTextModel\n    - forward\n\n## BlipVisionModel\n\n[[autodoc]] BlipVisionModel\n    - forward\n\n## BlipForConditionalGeneration\n\n[[autodoc]] BlipForConditionalGeneration\n    - forward\n\n## BlipForImageTextRetrieval\n\n[[autodoc]] BlipForImageTextRetrieval\n    - forward\n\n## BlipForQuestionAnswering\n\n[[autodoc]] BlipForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFBlipModel\n\n[[autodoc]] TFBlipModel\n    - call\n    - get_text_features\n    - get_image_features\n\n## TFBlipTextModel\n\n[[autodoc]] TFBlipTextModel\n    - call\n\n## TFBlipVisionModel\n\n[[autodoc]] TFBlipVisionModel\n    - call\n\n## TFBlipForConditionalGeneration\n\n[[autodoc]] TFBlipForConditionalGeneration\n    - call\n\n## TFBlipForImageTextRetrieval\n\n[[autodoc]] TFBlipForImageTextRetrieval\n    - call\n\n## TFBlipForQuestionAnswering\n\n[[autodoc]] TFBlipForQuestionAnswering\n    - call\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 37d26964-4ab8-4c33-9263-19c0f9b7537c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/persimmon/", "title": "Persimmon", "tokens": 1355, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"37d26964-4ab8-4c33-9263-19c0f9b7537c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/persimmon/\", \"title\": \"Persimmon\", \"tokens\": 1355, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"075c2dd1-2f8a-4c0f-a008-afb73905f958\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/persimmon/\", \"title\": \"Persimmon\", \"tokens\": 1355, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"c6831407f5f706840b590b42d4c1269b2a65054c1ffe379b888999a3314a973a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d1ec8a85-85a3-4c6f-8f93-a4fb77a57bc8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f71057f60e0b274766089ee37f984314fd3ac146e8cb211542163f6dca0c3bbf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2719, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "075c2dd1-2f8a-4c0f-a008-afb73905f958", "doc_id": "075c2dd1-2f8a-4c0f-a008-afb73905f958", "ref_doc_id": "075c2dd1-2f8a-4c0f-a008-afb73905f958", "chroma:document": "# Persimmon\n\n## Overview\n\nThe Persimmon model was created by [ADEPT](https://www.adept.ai/blog/persimmon-8b), and authored by Erich Elsen, Augustus Odena, Maxwell Nye, Sa\u011fnak Ta\u015f\u0131rlar, Tri Dao, Curtis Hawthorne, Deepak Moparthi, Arushi Somani.\n\nThe authors introduced Persimmon-8B, a decoder model based on the classic transformers architecture, with query and key normalization. Persimmon-8B is a fully permissively-licensed model with approximately 8 billion parameters, released under the Apache license.  Some of the key attributes of Persimmon-8B are long context size (16K), performance, and capabilities for multimodal extensions.\n\nThe authors showcase their approach to model evaluation, focusing on practical text generation, mirroring how users interact with language models. The work also includes a comparative analysis, pitting Persimmon-8B against other prominent models (MPT 7B Instruct and Llama 2 Base 7B 1-Shot), across various evaluation tasks. The results demonstrate Persimmon-8B's competitive performance, even with limited training data.\n\nIn terms of model details, the work outlines the architecture and training methodology of Persimmon-8B, providing insights into its design choices, sequence length, and dataset composition. The authors present a fast inference code that outperforms traditional implementations through operator fusion and CUDA graph utilization while maintaining code coherence. They express their anticipation of how the community will leverage this contribution to drive innovation, hinting at further upcoming releases as part of an ongoing series of developments.\n\nThis model was contributed by [ArthurZ](https://huggingface.co/ArthurZ).\nThe original code can be found [here](https://github.com/persimmon-ai-labs/adept-inference).\n\n## Usage tips\n\n<Tip warning={true}>\n\nThe `Persimmon` models were trained using `bfloat16`, but the original inference uses `float16` The checkpoints uploaded on the hub use `torch_dtype = 'float16'` which will be\nused by the `AutoModel` API to cast the checkpoints from `torch.float32` to `torch.float16`. \n\nThe `dtype` of the online weights is mostly irrelevant, unless you are using `torch_dtype=\"auto\"` when initializing a model using `model = AutoModelForCausalLM.from_pretrained(\"path\", torch_dtype = \"auto\")`. The reason is that the model will first be downloaded ( using the `dtype` of the checkpoints online) then it will be cast to the default `dtype` of `torch` (becomes `torch.float32`). Users should specify the `torch_dtype` they want, and if they don't it will be `torch.float32`.\n\nFinetuning the model in `float16` is not recommended and known to produce `nan`, as such the model should be fine-tuned in `bfloat16`.\n\n</Tip>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: d1ec8a85-85a3-4c6f-8f93-a4fb77a57bc8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/persimmon/", "title": "Persimmon", "tokens": 1355, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"d1ec8a85-85a3-4c6f-8f93-a4fb77a57bc8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/persimmon/\", \"title\": \"Persimmon\", \"tokens\": 1355, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"075c2dd1-2f8a-4c0f-a008-afb73905f958\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/persimmon/\", \"title\": \"Persimmon\", \"tokens\": 1355, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"c6831407f5f706840b590b42d4c1269b2a65054c1ffe379b888999a3314a973a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"37d26964-4ab8-4c33-9263-19c0f9b7537c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/persimmon/\", \"title\": \"Persimmon\", \"tokens\": 1355, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"4278df8c16c01c4106fa30d1efb675e591aa5fec6adce594087fce9eab30286f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2722, \"end_char_idx\": 4708, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "075c2dd1-2f8a-4c0f-a008-afb73905f958", "doc_id": "075c2dd1-2f8a-4c0f-a008-afb73905f958", "ref_doc_id": "075c2dd1-2f8a-4c0f-a008-afb73905f958", "chroma:document": "Tips:\n\n- To convert the model, you need to clone the original repository using `git clone https://github.com/persimmon-ai-labs/adept-inference`, then get the checkpoints:\n\n```bash\ngit clone https://github.com/persimmon-ai-labs/adept-inference\nwget https://axtkn4xl5cip.objectstorage.us-phoenix-1.oci.customer-oci.com/n/axtkn4xl5cip/b/adept-public-data/o/8b_base_model_release.tar\ntar -xvf 8b_base_model_release.tar\npython src/transformers/models/persimmon/convert_persimmon_weights_to_hf.py  --input_dir /path/to/downloaded/persimmon/weights/ --output_dir /output/path \\\n    --pt_model_path /path/to/8b_chat_model_release/iter_0001251/mp_rank_00/model_optim_rng.pt\n    --ada_lib_path /path/to/adept-inference\n```\n\nFor the chat model:\n```bash\nwget https://axtkn4xl5cip.objectstorage.us-phoenix-1.oci.customer-oci.com/n/axtkn4xl5cip/b/adept-public-data/o/8b_chat_model_release.tar\ntar -xvf 8b_base_model_release.tar\n```\n\nThereafter, models can be loaded via:\n\n```py\nfrom transformers import PersimmonForCausalLM, PersimmonTokenizer\n\nmodel = PersimmonForCausalLM.from_pretrained(\"/output/path\")\ntokenizer = PersimmonTokenizer.from_pretrained(\"/output/path\")\n```\n\n\n- Perismmon uses a `sentencepiece` based tokenizer, with a `Unigram` model. It supports bytefallback, which is only available in `tokenizers==0.14.0` for the fast tokenizer.\nThe `LlamaTokenizer` is used as it is a standard wrapper around sentencepiece. The `chat` template will be updated with the templating functions in a follow up PR!\n\n- The authors suggest to use the following prompt format for the chat mode: `f\"human: {prompt}\\n\\nadept:\"`\n\n\n## PersimmonConfig\n\n[[autodoc]] PersimmonConfig\n\n## PersimmonModel\n\n[[autodoc]] PersimmonModel\n    - forward\n\n## PersimmonForCausalLM\n\n[[autodoc]] PersimmonForCausalLM\n    - forward\n\n## PersimmonForSequenceClassification\n\n[[autodoc]] PersimmonForSequenceClassification\n    - forward\n\n## PersimmonForTokenClassification\n\n[[autodoc]] PersimmonForTokenClassification\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: c929e918-5de8-4664-9a0a-e246befd48e7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/deplot/", "title": "DePlot", "tokens": 993, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"c929e918-5de8-4664-9a0a-e246befd48e7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deplot/\", \"title\": \"DePlot\", \"tokens\": 993, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4cab7d28-0d99-4f6d-b9a7-098e957ef7d7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deplot/\", \"title\": \"DePlot\", \"tokens\": 993, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8a791e24d35828ba8228bf84c609a81bf6117f201656c9874e21d578a207eae4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f5238e07-c801-4484-9136-fed78168e2df\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bf66f3b03383cac5b0b17b826cf800ed42b73ac9bb35089422818eb6dc5dd505\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2167, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4cab7d28-0d99-4f6d-b9a7-098e957ef7d7", "doc_id": "4cab7d28-0d99-4f6d-b9a7-098e957ef7d7", "ref_doc_id": "4cab7d28-0d99-4f6d-b9a7-098e957ef7d7", "chroma:document": "# DePlot\n\n## Overview \n\nDePlot was proposed in the paper [DePlot: One-shot visual language reasoning by plot-to-table translation](https://arxiv.org/abs/2212.10505) from Fangyu Liu, Julian Martin Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, Yasemin Altun.\n\nThe abstract of the paper states the following:\n\n*Visual language such as charts and plots is ubiquitous in the human world. Comprehending plots and charts requires strong reasoning skills. Prior state-of-the-art (SOTA) models require at least tens of thousands of training examples and their reasoning capabilities are still much limited, especially on complex human-written queries. This paper presents the first one-shot solution to visual language reasoning. We decompose the challenge of visual language reasoning into two steps: (1) plot-to-text translation, and (2) reasoning over the translated text. The key in this method is a modality conversion module, named as DePlot, which translates the image of a plot or chart to a linearized table. The output of DePlot can then be directly used to prompt a pretrained large language model (LLM), exploiting the few-shot reasoning capabilities of LLMs. To obtain DePlot, we standardize the plot-to-table task by establishing unified task formats and metrics, and train DePlot end-to-end on this task. DePlot can then be used off-the-shelf together with LLMs in a plug-and-play fashion. Compared with a SOTA model finetuned on more than >28k data points, DePlot+LLM with just one-shot prompting achieves a 24.0% improvement over finetuned SOTA on human-written queries from the task of chart QA.*\n\nDePlot is a model that is trained using `Pix2Struct` architecture. You can find more information about `Pix2Struct` in the [Pix2Struct documentation](https://huggingface.co/docs/transformers/main/en/model_doc/pix2struct).\nDePlot is a Visual Question Answering subset of `Pix2Struct` architecture. It renders the input question on the image and predicts the answer.\n\n## Usage example\n\nCurrently one checkpoint is available for DePlot:\n\n- `google/deplot`: DePlot fine-tuned on ChartQA dataset"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: f5238e07-c801-4484-9136-fed78168e2df
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/deplot/", "title": "DePlot", "tokens": 993, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"f5238e07-c801-4484-9136-fed78168e2df\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deplot/\", \"title\": \"DePlot\", \"tokens\": 993, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4cab7d28-0d99-4f6d-b9a7-098e957ef7d7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deplot/\", \"title\": \"DePlot\", \"tokens\": 993, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8a791e24d35828ba8228bf84c609a81bf6117f201656c9874e21d578a207eae4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c929e918-5de8-4664-9a0a-e246befd48e7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deplot/\", \"title\": \"DePlot\", \"tokens\": 993, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"82b1b4472cfb18349d9570db46d62a52f45431921370d4db8f60f332d5591eff\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2171, \"end_char_idx\": 3610, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4cab7d28-0d99-4f6d-b9a7-098e957ef7d7", "doc_id": "4cab7d28-0d99-4f6d-b9a7-098e957ef7d7", "ref_doc_id": "4cab7d28-0d99-4f6d-b9a7-098e957ef7d7", "chroma:document": "```python\nfrom transformers import AutoProcessor, Pix2StructForConditionalGeneration\nimport requests\nfrom PIL import Image\n\nmodel = Pix2StructForConditionalGeneration.from_pretrained(\"google/deplot\")\nprocessor = AutoProcessor.from_pretrained(\"google/deplot\")\nurl = \"https://raw.githubusercontent.com/vis-nlp/ChartQA/main/ChartQA%20Dataset/val/png/5090.png\"\nimage = Image.open(requests.get(url, stream=True).raw)\n\ninputs = processor(images=image, text=\"Generate underlying data table of the figure below:\", return_tensors=\"pt\")\npredictions = model.generate(**inputs, max_new_tokens=512)\nprint(processor.decode(predictions[0], skip_special_tokens=True))\n```\n\n## Fine-tuning\n\nTo fine-tune DePlot, refer to the pix2struct [fine-tuning notebook](https://github.com/huggingface/notebooks/blob/main/examples/image_captioning_pix2struct.ipynb). For `Pix2Struct` models, we have found out that fine-tuning the model with Adafactor and cosine learning rate scheduler leads to faster convergence:\n```python\nfrom transformers.optimization import Adafactor, get_cosine_schedule_with_warmup\n\noptimizer = Adafactor(self.parameters(), scale_parameter=False, relative_step=False, lr=0.01, weight_decay=1e-05)\nscheduler = get_cosine_schedule_with_warmup(optimizer, num_warmup_steps=1000, num_training_steps=40000)\n```\n\n<Tip>\n\nDePlot is a model trained using `Pix2Struct` architecture. For API reference, see [`Pix2Struct` documentation](pix2struct).\n\n</Tip>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: e90101d2-7982-4106-9167-48f0161f6b34
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gemma2/", "title": "Gemma2", "tokens": 539, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"e90101d2-7982-4106-9167-48f0161f6b34\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gemma2/\", \"title\": \"Gemma2\", \"tokens\": 539, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"111075a7-5cad-45bb-bb0e-54c6d433d773\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gemma2/\", \"title\": \"Gemma2\", \"tokens\": 539, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"d05355dd92297974e522179e75eecd4665a6265635028b1c56ef19c3713b9178\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1469, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "111075a7-5cad-45bb-bb0e-54c6d433d773", "doc_id": "111075a7-5cad-45bb-bb0e-54c6d433d773", "ref_doc_id": "111075a7-5cad-45bb-bb0e-54c6d433d773", "chroma:document": "# Gemma2\n\n## Overview\n\nThe Gemma2 model was proposed in [Gemma2: Open Models Based on Gemini Technology and Research](https://blog.google/technology/developers/google-gemma-2/) by Gemma2 Team, Google.\nTwo Gemma2 models are released, with parameters sizes of 9 billion (9B) and 27 billion (27B).\n\nThe abstract from the blog post is the following:\n\n*Now we\u2019re officially releasing Gemma 2 to researchers and developers globally. Available in both 9 billion (9B) and 27 billion (27B) parameter sizes, Gemma 2 is higher-performing and more efficient at inference than the first generation, with significant safety advancements built in. In fact, at 27B, it offers competitive alternatives to models more than twice its size, delivering the kind of performance that was only possible with proprietary models as recently as December.*\n\nTips:\n\n- The original checkpoints can be converted using the conversion script `src/transformers/models/Gemma2/convert_Gemma2_weights_to_hf.py` \n\nThis model was contributed by [Arthur Zucker](https://huggingface.co/ArthurZ), [Pedro Cuenca](https://huggingface.co/pcuenq) and [Tom Arsen]().\n\n\n## Gemma2Config\n\n[[autodoc]] Gemma2Config\n\n## Gemma2Model\n\n[[autodoc]] Gemma2Model\n    - forward\n\n## Gemma2ForCausalLM\n\n[[autodoc]] Gemma2ForCausalLM\n    - forward\n\n## Gemma2ForSequenceClassification\n\n[[autodoc]] Gemma2ForSequenceClassification\n    - forward\n\n## Gemma2ForTokenClassification\n\n[[autodoc]] Gemma2ForTokenClassification\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 5d900ce8-c9c1-4646-8180-f10a94cf796b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/maskformer/", "title": "MaskFormer", "tokens": 1272, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"5d900ce8-c9c1-4646-8180-f10a94cf796b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/maskformer/\", \"title\": \"MaskFormer\", \"tokens\": 1272, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"17a151a2-ff46-4128-bca9-1e79e59da360\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/maskformer/\", \"title\": \"MaskFormer\", \"tokens\": 1272, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"496907c9986952544070c3eee07521854c5e8f594d1bb119f49a47fb5dfb26d3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fccd4ea1-131c-49df-a053-78c8d73130cc\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"21600adbb10d029becae8e0ef62881c7d29a3ba2dfaba20c69ce1df8e54ecd2a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3519, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "17a151a2-ff46-4128-bca9-1e79e59da360", "doc_id": "17a151a2-ff46-4128-bca9-1e79e59da360", "ref_doc_id": "17a151a2-ff46-4128-bca9-1e79e59da360", "chroma:document": "# MaskFormer\n\n<Tip>\n\nThis is a recently introduced model so the API hasn't been tested extensively. There may be some bugs or slight\nbreaking changes to fix it in the future. If you see something strange, file a [Github Issue](https://github.com/huggingface/transformers/issues/new?assignees=&labels=&template=bug-report.md&title).\n\n</Tip>\n\n## Overview\n\nThe MaskFormer model was proposed in [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://arxiv.org/abs/2107.06278) by Bowen Cheng, Alexander G. Schwing, Alexander Kirillov. MaskFormer addresses semantic segmentation with a mask classification paradigm instead of performing classic pixel-level classification.\n\nThe abstract from the paper is the following:\n\n*Modern approaches typically formulate semantic segmentation as a per-pixel classification task, while instance-level segmentation is handled with an alternative mask classification. Our key insight: mask classification is sufficiently general to solve both semantic- and instance-level segmentation tasks in a unified manner using the exact same model, loss, and training procedure. Following this observation, we propose MaskFormer, a simple mask classification model which predicts a set of binary masks, each associated with a single global class label prediction. Overall, the proposed mask classification-based method simplifies the landscape of effective approaches to semantic and panoptic segmentation tasks and shows excellent empirical results. In particular, we observe that MaskFormer outperforms per-pixel classification baselines when the number of classes is large. Our mask classification-based method outperforms both current state-of-the-art semantic (55.6 mIoU on ADE20K) and panoptic segmentation (52.7 PQ on COCO) models.*\n\nThe figure below illustrates the architecture of MaskFormer. Taken from the [original paper](https://arxiv.org/abs/2107.06278).\n\n<img width=\"600\" src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/maskformer_architecture.png\"/>\n\nThis model was contributed by [francesco](https://huggingface.co/francesco). The original code can be found [here](https://github.com/facebookresearch/MaskFormer).\n\n## Usage tips\n\n-  MaskFormer's Transformer decoder is identical to the decoder of [DETR](detr). During training, the authors of DETR did find it helpful to use auxiliary losses in the decoder, especially to help the model output the correct number of objects of each class. If you set the parameter `use_auxiliary_loss` of [`MaskFormerConfig`] to `True`, then prediction feedforward neural networks and Hungarian losses are added after each decoder layer (with the FFNs sharing parameters).\n- If you want to train the model in a distributed environment across multiple nodes, then one should update the\n  `get_num_masks` function inside in the `MaskFormerLoss` class of `modeling_maskformer.py`. When training on multiple nodes, this should be\n  set to the average number of target masks across all nodes, as can be seen in the original implementation [here](https://github.com/facebookresearch/MaskFormer/blob/da3e60d85fdeedcb31476b5edd7d328826ce56cc/mask_former/modeling/criterion.py#L169).\n- One can use [`MaskFormerImageProcessor`] to prepare images for the model and optional targets for the model.\n- To get the final segmentation, depending on the task, you can call [`~MaskFormerImageProcessor.post_process_semantic_segmentation`] or [`~MaskFormerImageProcessor.post_process_panoptic_segmentation`]."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: fccd4ea1-131c-49df-a053-78c8d73130cc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/maskformer/", "title": "MaskFormer", "tokens": 1272, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"fccd4ea1-131c-49df-a053-78c8d73130cc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/maskformer/\", \"title\": \"MaskFormer\", \"tokens\": 1272, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"17a151a2-ff46-4128-bca9-1e79e59da360\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/maskformer/\", \"title\": \"MaskFormer\", \"tokens\": 1272, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"496907c9986952544070c3eee07521854c5e8f594d1bb119f49a47fb5dfb26d3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5d900ce8-c9c1-4646-8180-f10a94cf796b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/maskformer/\", \"title\": \"MaskFormer\", \"tokens\": 1272, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"7f9750f592c61c676cabfe9cdb86597d4da6fb1d07343c5744fc67f8d0578a1c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1848, \"end_char_idx\": 5083, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "17a151a2-ff46-4128-bca9-1e79e59da360", "doc_id": "17a151a2-ff46-4128-bca9-1e79e59da360", "ref_doc_id": "17a151a2-ff46-4128-bca9-1e79e59da360", "chroma:document": "Taken from the [original paper](https://arxiv.org/abs/2107.06278).\n\n<img width=\"600\" src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/maskformer_architecture.png\"/>\n\nThis model was contributed by [francesco](https://huggingface.co/francesco). The original code can be found [here](https://github.com/facebookresearch/MaskFormer).\n\n## Usage tips\n\n-  MaskFormer's Transformer decoder is identical to the decoder of [DETR](detr). During training, the authors of DETR did find it helpful to use auxiliary losses in the decoder, especially to help the model output the correct number of objects of each class. If you set the parameter `use_auxiliary_loss` of [`MaskFormerConfig`] to `True`, then prediction feedforward neural networks and Hungarian losses are added after each decoder layer (with the FFNs sharing parameters).\n- If you want to train the model in a distributed environment across multiple nodes, then one should update the\n  `get_num_masks` function inside in the `MaskFormerLoss` class of `modeling_maskformer.py`. When training on multiple nodes, this should be\n  set to the average number of target masks across all nodes, as can be seen in the original implementation [here](https://github.com/facebookresearch/MaskFormer/blob/da3e60d85fdeedcb31476b5edd7d328826ce56cc/mask_former/modeling/criterion.py#L169).\n- One can use [`MaskFormerImageProcessor`] to prepare images for the model and optional targets for the model.\n- To get the final segmentation, depending on the task, you can call [`~MaskFormerImageProcessor.post_process_semantic_segmentation`] or [`~MaskFormerImageProcessor.post_process_panoptic_segmentation`]. Both tasks can be solved using [`MaskFormerForInstanceSegmentation`] output, panoptic segmentation accepts an optional `label_ids_to_fuse` argument to fuse instances of the target object/s (e.g. sky) together.\n\n## Resources\n\n<PipelineTag pipeline=\"image-segmentation\"/>\n\n- All notebooks that illustrate inference as well as fine-tuning on custom data with MaskFormer can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/MaskFormer).\n- Scripts for finetuning [`MaskFormer`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/instance-segmentation).\n\n## MaskFormer specific outputs\n\n[[autodoc]] models.maskformer.modeling_maskformer.MaskFormerModelOutput\n\n[[autodoc]] models.maskformer.modeling_maskformer.MaskFormerForInstanceSegmentationOutput\n\n## MaskFormerConfig\n\n[[autodoc]] MaskFormerConfig\n\n## MaskFormerImageProcessor\n\n[[autodoc]] MaskFormerImageProcessor\n    - preprocess\n    - encode_inputs\n    - post_process_semantic_segmentation\n    - post_process_instance_segmentation\n    - post_process_panoptic_segmentation\n\n## MaskFormerFeatureExtractor\n\n[[autodoc]] MaskFormerFeatureExtractor\n    - __call__\n    - encode_inputs\n    - post_process_semantic_segmentation\n    - post_process_instance_segmentation\n    - post_process_panoptic_segmentation\n\n## MaskFormerModel\n\n[[autodoc]] MaskFormerModel\n    - forward\n\n## MaskFormerForInstanceSegmentation\n\n[[autodoc]] MaskFormerForInstanceSegmentation\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: d5516631-b72c-4aab-b728-346e27a3ec07
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/kosmos-2/", "title": "KOSMOS-2", "tokens": 1310, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"d5516631-b72c-4aab-b728-346e27a3ec07\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/kosmos-2/\", \"title\": \"KOSMOS-2\", \"tokens\": 1310, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4e9394f4-7df9-4045-a6da-84eea72d148e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/kosmos-2/\", \"title\": \"KOSMOS-2\", \"tokens\": 1310, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0ba73be3ce59231f65f1d46ad9f83263bdc7f2e9b4fdc3086ac1cb23bdc5c5b8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9dbce1f9-5c94-47f2-af20-daf2d9dd1323\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fbdbd3fbf8d1165877f9b1a4377b2ab5653c71138ab696d44704998c1a27a9c4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2491, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4e9394f4-7df9-4045-a6da-84eea72d148e", "doc_id": "4e9394f4-7df9-4045-a6da-84eea72d148e", "ref_doc_id": "4e9394f4-7df9-4045-a6da-84eea72d148e", "chroma:document": "# KOSMOS-2\n\n## Overview\n\nThe KOSMOS-2 model was proposed in [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://arxiv.org/abs/2306.14824) by Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, Furu Wei.\n\nKOSMOS-2 is a Transformer-based causal language model and is trained using the next-word prediction task on a web-scale\ndataset of grounded image-text pairs [GRIT](https://huggingface.co/datasets/zzliang/GRIT). The spatial coordinates of\nthe bounding boxes in the dataset are converted to a sequence of location tokens, which are appended to their respective\nentity text spans (for example, `a snowman` followed by `<patch_index_0044><patch_index_0863>`). The data format is\nsimilar to \u201chyperlinks\u201d that connect the object regions in an image to their text span in the corresponding caption.\n\nThe abstract from the paper is the following:\n\n*We introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enabling new capabilities of perceiving object descriptions (e.g., bounding boxes) and grounding text to the visual world. Specifically, we represent refer expressions as links in Markdown, i.e., ``[text span](bounding boxes)'', where object descriptions are sequences of location tokens. Together with multimodal corpora, we construct large-scale data of grounded image-text pairs (called GrIT) to train the model. In addition to the existing capabilities of MLLMs (e.g., perceiving general modalities, following instructions, and performing in-context learning), Kosmos-2 integrates the grounding capability into downstream applications. We evaluate Kosmos-2 on a wide range of tasks, including (i) multimodal grounding, such as referring expression comprehension, and phrase grounding, (ii) multimodal referring, such as referring expression generation, (iii) perception-language tasks, and (iv) language understanding and generation. This work lays out the foundation for the development of Embodiment AI and sheds light on the big convergence of language, multimodal perception, action, and world modeling, which is a key step toward artificial general intelligence. Code and pretrained models are available at https://aka.ms/kosmos-2.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/kosmos_2_overview.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Overview of tasks that KOSMOS-2 can handle. Taken from the <a href=\"https://arxiv.org/abs/2306.14824\">original paper</a>."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 9dbce1f9-5c94-47f2-af20-daf2d9dd1323
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/kosmos-2/", "title": "KOSMOS-2", "tokens": 1310, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"9dbce1f9-5c94-47f2-af20-daf2d9dd1323\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/kosmos-2/\", \"title\": \"KOSMOS-2\", \"tokens\": 1310, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4e9394f4-7df9-4045-a6da-84eea72d148e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/kosmos-2/\", \"title\": \"KOSMOS-2\", \"tokens\": 1310, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0ba73be3ce59231f65f1d46ad9f83263bdc7f2e9b4fdc3086ac1cb23bdc5c5b8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d5516631-b72c-4aab-b728-346e27a3ec07\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/kosmos-2/\", \"title\": \"KOSMOS-2\", \"tokens\": 1310, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a7f4c7cf2fba27c072fa88c3b44b42e603bec05df2897ae4b9c1c3d3e77c3325\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7be49271-cc66-4dc4-93db-200cd93864fc\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"989c840e4edc78718d1651302ce8a3570ce373969d02f00bc09ebe110edf467d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 706, \"end_char_idx\": 3956, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4e9394f4-7df9-4045-a6da-84eea72d148e", "doc_id": "4e9394f4-7df9-4045-a6da-84eea72d148e", "ref_doc_id": "4e9394f4-7df9-4045-a6da-84eea72d148e", "chroma:document": "The data format is\nsimilar to \u201chyperlinks\u201d that connect the object regions in an image to their text span in the corresponding caption.\n\nThe abstract from the paper is the following:\n\n*We introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enabling new capabilities of perceiving object descriptions (e.g., bounding boxes) and grounding text to the visual world. Specifically, we represent refer expressions as links in Markdown, i.e., ``[text span](bounding boxes)'', where object descriptions are sequences of location tokens. Together with multimodal corpora, we construct large-scale data of grounded image-text pairs (called GrIT) to train the model. In addition to the existing capabilities of MLLMs (e.g., perceiving general modalities, following instructions, and performing in-context learning), Kosmos-2 integrates the grounding capability into downstream applications. We evaluate Kosmos-2 on a wide range of tasks, including (i) multimodal grounding, such as referring expression comprehension, and phrase grounding, (ii) multimodal referring, such as referring expression generation, (iii) perception-language tasks, and (iv) language understanding and generation. This work lays out the foundation for the development of Embodiment AI and sheds light on the big convergence of language, multimodal perception, action, and world modeling, which is a key step toward artificial general intelligence. Code and pretrained models are available at https://aka.ms/kosmos-2.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/kosmos_2_overview.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Overview of tasks that KOSMOS-2 can handle. Taken from the <a href=\"https://arxiv.org/abs/2306.14824\">original paper</a>. </small>\n\n## Example\n\n```python\n>>> from PIL import Image\n>>> import requests\n>>> from transformers import AutoProcessor, Kosmos2ForConditionalGeneration\n\n>>> model = Kosmos2ForConditionalGeneration.from_pretrained(\"microsoft/kosmos-2-patch14-224\")\n>>> processor = AutoProcessor.from_pretrained(\"microsoft/kosmos-2-patch14-224\")\n\n>>> url = \"https://huggingface.co/microsoft/kosmos-2-patch14-224/resolve/main/snowman.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n\n>>> prompt = \"<grounding> An image of\"\n\n>>> inputs = processor(text=prompt, images=image, return_tensors=\"pt\")\n\n>>> generated_ids = model.generate(\n...     pixel_values=inputs[\"pixel_values\"],\n...     input_ids=inputs[\"input_ids\"],\n...     attention_mask=inputs[\"attention_mask\"],\n...     image_embeds=None,\n...     image_embeds_position_mask=inputs[\"image_embeds_position_mask\"],\n...     use_cache=True,\n...     max_new_tokens=64,\n... )\n>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]\n>>> processed_text = processor.post_process_generation(generated_text, cleanup_and_extract=False)\n>>> processed_text\n'<grounding> An image of<phrase> a snowman</phrase><object><patch_index_0044><patch_index_0863></object> warming himself by<phrase> a fire</phrase><object><patch_index_0005><patch_index_0911></object>.'\n\n>>> caption, entities = processor.post_process_generation(generated_text)\n>>> caption\n'An image of a snowman warming himself by a fire.'"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 7be49271-cc66-4dc4-93db-200cd93864fc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/kosmos-2/", "title": "KOSMOS-2", "tokens": 1310, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"7be49271-cc66-4dc4-93db-200cd93864fc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/kosmos-2/\", \"title\": \"KOSMOS-2\", \"tokens\": 1310, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4e9394f4-7df9-4045-a6da-84eea72d148e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/kosmos-2/\", \"title\": \"KOSMOS-2\", \"tokens\": 1310, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0ba73be3ce59231f65f1d46ad9f83263bdc7f2e9b4fdc3086ac1cb23bdc5c5b8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9dbce1f9-5c94-47f2-af20-daf2d9dd1323\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/kosmos-2/\", \"title\": \"KOSMOS-2\", \"tokens\": 1310, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"3febef9af337a548bd2a990944e204189e0f03b7544d9a052e9104d0ac872f78\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2414, \"end_char_idx\": 4575, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4e9394f4-7df9-4045-a6da-84eea72d148e", "doc_id": "4e9394f4-7df9-4045-a6da-84eea72d148e", "ref_doc_id": "4e9394f4-7df9-4045-a6da-84eea72d148e", "chroma:document": "Taken from the <a href=\"https://arxiv.org/abs/2306.14824\">original paper</a>. </small>\n\n## Example\n\n```python\n>>> from PIL import Image\n>>> import requests\n>>> from transformers import AutoProcessor, Kosmos2ForConditionalGeneration\n\n>>> model = Kosmos2ForConditionalGeneration.from_pretrained(\"microsoft/kosmos-2-patch14-224\")\n>>> processor = AutoProcessor.from_pretrained(\"microsoft/kosmos-2-patch14-224\")\n\n>>> url = \"https://huggingface.co/microsoft/kosmos-2-patch14-224/resolve/main/snowman.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n\n>>> prompt = \"<grounding> An image of\"\n\n>>> inputs = processor(text=prompt, images=image, return_tensors=\"pt\")\n\n>>> generated_ids = model.generate(\n...     pixel_values=inputs[\"pixel_values\"],\n...     input_ids=inputs[\"input_ids\"],\n...     attention_mask=inputs[\"attention_mask\"],\n...     image_embeds=None,\n...     image_embeds_position_mask=inputs[\"image_embeds_position_mask\"],\n...     use_cache=True,\n...     max_new_tokens=64,\n... )\n>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]\n>>> processed_text = processor.post_process_generation(generated_text, cleanup_and_extract=False)\n>>> processed_text\n'<grounding> An image of<phrase> a snowman</phrase><object><patch_index_0044><patch_index_0863></object> warming himself by<phrase> a fire</phrase><object><patch_index_0005><patch_index_0911></object>.'\n\n>>> caption, entities = processor.post_process_generation(generated_text)\n>>> caption\n'An image of a snowman warming himself by a fire.'\n\n>>> entities\n[('a snowman', (12, 21), [(0.390625, 0.046875, 0.984375, 0.828125)]), ('a fire', (41, 47), [(0.171875, 0.015625, 0.484375, 0.890625)])]\n```\n\nThis model was contributed by [Yih-Dar SHIEH](https://huggingface.co/ydshieh). The original code can be found [here](https://github.com/microsoft/unilm/tree/master/kosmos-2).\n\n## Kosmos2Config\n\n[[autodoc]] Kosmos2Config\n\n## Kosmos2ImageProcessor\n\n## Kosmos2Processor\n\n[[autodoc]] Kosmos2Processor\n    - __call__\n\n## Kosmos2Model\n\n[[autodoc]] Kosmos2Model\n    - forward\n\n## Kosmos2ForConditionalGeneration\n\n[[autodoc]] Kosmos2ForConditionalGeneration\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 3aefcc41-2506-4692-ad44-52f85e36e8a8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2/", "title": "Wav2Vec2", "tokens": 3049, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"3aefcc41-2506-4692-ad44-52f85e36e8a8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2/\", \"title\": \"Wav2Vec2\", \"tokens\": 3049, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1854bc00-e71c-43a1-a5cc-cdfe2c63b1cf\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2/\", \"title\": \"Wav2Vec2\", \"tokens\": 3049, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"348a691cf3244b01f4fa5e5586abc4a56c0ad49e3523e599cf0d8bfccbb408e9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1bfe3985-f979-4d3c-80a2-6832f3c4af42\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a7556100accb5fb4b0ccab26eed5def11201dfcbaccf7da88aa300f1dcd36d9d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2939, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1854bc00-e71c-43a1-a5cc-cdfe2c63b1cf", "doc_id": "1854bc00-e71c-43a1-a5cc-cdfe2c63b1cf", "ref_doc_id": "1854bc00-e71c-43a1-a5cc-cdfe2c63b1cf", "chroma:document": "# Wav2Vec2\n\n## Overview\n\nThe Wav2Vec2 model was proposed in [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://arxiv.org/abs/2006.11477) by Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, Michael Auli.\n\nThe abstract from the paper is the following:\n\n*We show for the first time that learning powerful representations from speech audio alone followed by fine-tuning on\ntranscribed speech can outperform the best semi-supervised methods while being conceptually simpler. wav2vec 2.0 masks\nthe speech input in the latent space and solves a contrastive task defined over a quantization of the latent\nrepresentations which are jointly learned. Experiments using all labeled data of Librispeech achieve 1.8/3.3 WER on the\nclean/other test sets. When lowering the amount of labeled data to one hour, wav2vec 2.0 outperforms the previous state\nof the art on the 100 hour subset while using 100 times less labeled data. Using just ten minutes of labeled data and\npre-training on 53k hours of unlabeled data still achieves 4.8/8.2 WER. This demonstrates the feasibility of speech\nrecognition with limited amounts of labeled data.*\n\nThis model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten).\n\nNote: Meta (FAIR) released a new version of [Wav2Vec2-BERT 2.0](https://huggingface.co/docs/transformers/en/model_doc/wav2vec2-bert) - it's pretrained on 4.5M hours of audio. We especially recommend using it for fine-tuning tasks, e.g. as per [this guide](https://huggingface.co/blog/fine-tune-w2v2-bert).\n\n## Usage tips\n\n- Wav2Vec2 is a speech model that accepts a float array corresponding to the raw waveform of the speech signal.\n- Wav2Vec2 model was trained using connectionist temporal classification (CTC) so the model output has to be decoded\n  using [`Wav2Vec2CTCTokenizer`].\n\n## Using Flash Attention 2\n\nFlash Attention 2 is an faster, optimized version of the model.\n\n### Installation \n\nFirst, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features). If your hardware is not compatible with Flash Attention 2, you can still benefit from attention kernel optimisations through Better Transformer support covered [above](https://huggingface.co/docs/transformers/main/en/model_doc/bark#using-better-transformer).\n\nNext, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\n### Usage\n\nTo load a model using Flash Attention 2, we can pass the argument `attn_implementation=\"flash_attention_2\"` to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). We'll also load the model in half-precision (e.g."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 1bfe3985-f979-4d3c-80a2-6832f3c4af42
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2/", "title": "Wav2Vec2", "tokens": 3049, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"1bfe3985-f979-4d3c-80a2-6832f3c4af42\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2/\", \"title\": \"Wav2Vec2\", \"tokens\": 3049, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1854bc00-e71c-43a1-a5cc-cdfe2c63b1cf\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2/\", \"title\": \"Wav2Vec2\", \"tokens\": 3049, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"348a691cf3244b01f4fa5e5586abc4a56c0ad49e3523e599cf0d8bfccbb408e9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3aefcc41-2506-4692-ad44-52f85e36e8a8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2/\", \"title\": \"Wav2Vec2\", \"tokens\": 3049, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"1afe9bf5b61a418cc80c0acc8c3d5bfd804db1f1aa990f4aec7bcab139527c35\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a314aa02-1cd8-4b6e-affc-4651a88c140f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7db575ae64e35bb846ad4825283746b02ea946b1ccf23258397cebc31ba4f15a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1429, \"end_char_idx\": 4452, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1854bc00-e71c-43a1-a5cc-cdfe2c63b1cf", "doc_id": "1854bc00-e71c-43a1-a5cc-cdfe2c63b1cf", "ref_doc_id": "1854bc00-e71c-43a1-a5cc-cdfe2c63b1cf", "chroma:document": "We especially recommend using it for fine-tuning tasks, e.g. as per [this guide](https://huggingface.co/blog/fine-tune-w2v2-bert).\n\n## Usage tips\n\n- Wav2Vec2 is a speech model that accepts a float array corresponding to the raw waveform of the speech signal.\n- Wav2Vec2 model was trained using connectionist temporal classification (CTC) so the model output has to be decoded\n  using [`Wav2Vec2CTCTokenizer`].\n\n## Using Flash Attention 2\n\nFlash Attention 2 is an faster, optimized version of the model.\n\n### Installation \n\nFirst, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features). If your hardware is not compatible with Flash Attention 2, you can still benefit from attention kernel optimisations through Better Transformer support covered [above](https://huggingface.co/docs/transformers/main/en/model_doc/bark#using-better-transformer).\n\nNext, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\n### Usage\n\nTo load a model using Flash Attention 2, we can pass the argument `attn_implementation=\"flash_attention_2\"` to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). We'll also load the model in half-precision (e.g. `torch.float16`), since it results in almost no degradation to audio quality but significantly lower memory usage and faster inference:\n\n```python\n>>> from transformers import Wav2Vec2Model\n\nmodel = Wav2Vec2Model.from_pretrained(\"facebook/wav2vec2-large-960h-lv60-self\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\").to(device)\n...\n```\n\n### Expected speedups\n\nBelow is an expected speedup diagram comparing the pure inference time between the native implementation in transformers of the `facebook/wav2vec2-large-960h-lv60-self` model and the flash-attention-2 and sdpa (scale-dot-product-attention) versions. . We show the average speedup obtained on the `librispeech_asr` `clean` validation split: \n\n\n<div style=\"text-align: center\">\n<img src=\"https://huggingface.co/datasets/kamilakesbi/transformers_image_doc/resolve/main/data/Wav2Vec2_speedup.png\">\n</div>\n\n\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Wav2Vec2. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"audio-classification\"/>\n\n- A notebook on how to [leverage a pretrained Wav2Vec2 model for emotion classification](https://colab.research.google.com/github/m3hrdadfi/soxan/blob/main/notebooks/Emotion_recognition_in_Greek_speech_using_Wav2Vec2.ipynb)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: a314aa02-1cd8-4b6e-affc-4651a88c140f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2/", "title": "Wav2Vec2", "tokens": 3049, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"a314aa02-1cd8-4b6e-affc-4651a88c140f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2/\", \"title\": \"Wav2Vec2\", \"tokens\": 3049, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1854bc00-e71c-43a1-a5cc-cdfe2c63b1cf\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2/\", \"title\": \"Wav2Vec2\", \"tokens\": 3049, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"348a691cf3244b01f4fa5e5586abc4a56c0ad49e3523e599cf0d8bfccbb408e9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1bfe3985-f979-4d3c-80a2-6832f3c4af42\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2/\", \"title\": \"Wav2Vec2\", \"tokens\": 3049, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"7a7055016cb32086a483fbeccca09881e494f6a9932b63e745f7195a0913cc49\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d260ceb9-81b8-4d6d-aca0-d009343d6aaa\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6bcccf13fca0b66112af214c50b34a8ce1dd6fa699a3a6e2143502a6b5dc8435\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2890, \"end_char_idx\": 5854, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1854bc00-e71c-43a1-a5cc-cdfe2c63b1cf", "doc_id": "1854bc00-e71c-43a1-a5cc-cdfe2c63b1cf", "ref_doc_id": "1854bc00-e71c-43a1-a5cc-cdfe2c63b1cf", "chroma:document": "We'll also load the model in half-precision (e.g. `torch.float16`), since it results in almost no degradation to audio quality but significantly lower memory usage and faster inference:\n\n```python\n>>> from transformers import Wav2Vec2Model\n\nmodel = Wav2Vec2Model.from_pretrained(\"facebook/wav2vec2-large-960h-lv60-self\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\").to(device)\n...\n```\n\n### Expected speedups\n\nBelow is an expected speedup diagram comparing the pure inference time between the native implementation in transformers of the `facebook/wav2vec2-large-960h-lv60-self` model and the flash-attention-2 and sdpa (scale-dot-product-attention) versions. . We show the average speedup obtained on the `librispeech_asr` `clean` validation split: \n\n\n<div style=\"text-align: center\">\n<img src=\"https://huggingface.co/datasets/kamilakesbi/transformers_image_doc/resolve/main/data/Wav2Vec2_speedup.png\">\n</div>\n\n\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Wav2Vec2. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"audio-classification\"/>\n\n- A notebook on how to [leverage a pretrained Wav2Vec2 model for emotion classification](https://colab.research.google.com/github/m3hrdadfi/soxan/blob/main/notebooks/Emotion_recognition_in_Greek_speech_using_Wav2Vec2.ipynb). \ud83c\udf0e\n- [`Wav2Vec2ForCTC`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/audio-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/audio_classification.ipynb).\n- [Audio classification task guide](../tasks/audio_classification)\n\n<PipelineTag pipeline=\"automatic-speech-recognition\"/>\n\n- A blog post on [boosting Wav2Vec2 with n-grams in \ud83e\udd17 Transformers](https://huggingface.co/blog/wav2vec2-with-ngram).\n- A blog post on how to [finetune Wav2Vec2 for English ASR with \ud83e\udd17 Transformers](https://huggingface.co/blog/fine-tune-wav2vec2-english).\n- A blog post on [finetuning XLS-R for Multi-Lingual ASR with \ud83e\udd17 Transformers](https://huggingface.co/blog/fine-tune-xlsr-wav2vec2).\n- A notebook on how to [create YouTube captions from any video by transcribing audio with Wav2Vec2](https://colab.research.google.com/github/Muennighoff/ytclipcc/blob/main/wav2vec_youtube_captions.ipynb). \ud83c\udf0e\n- [`Wav2Vec2ForCTC`] is supported by a notebook on [how to finetune a speech recognition model in English](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/speech_recognition.ipynb), and [how to finetune a speech recognition model in any language](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multi_lingual_speech_recognition.ipynb)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: d260ceb9-81b8-4d6d-aca0-d009343d6aaa
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2/", "title": "Wav2Vec2", "tokens": 3049, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"d260ceb9-81b8-4d6d-aca0-d009343d6aaa\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2/\", \"title\": \"Wav2Vec2\", \"tokens\": 3049, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1854bc00-e71c-43a1-a5cc-cdfe2c63b1cf\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2/\", \"title\": \"Wav2Vec2\", \"tokens\": 3049, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"348a691cf3244b01f4fa5e5586abc4a56c0ad49e3523e599cf0d8bfccbb408e9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a314aa02-1cd8-4b6e-affc-4651a88c140f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2/\", \"title\": \"Wav2Vec2\", \"tokens\": 3049, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"44e8a38c1f59d7db133befe8406b1761605a3fc09c5a002998855ad8f637d52f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d83cde30-3b60-412b-9931-e65401df1023\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"cb5249c5093421b669e5163ed416f3eea27e3262955d9020962f1ecd2fee5a64\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4453, \"end_char_idx\": 7142, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1854bc00-e71c-43a1-a5cc-cdfe2c63b1cf", "doc_id": "1854bc00-e71c-43a1-a5cc-cdfe2c63b1cf", "ref_doc_id": "1854bc00-e71c-43a1-a5cc-cdfe2c63b1cf", "chroma:document": "\ud83c\udf0e\n- [`Wav2Vec2ForCTC`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/audio-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/audio_classification.ipynb).\n- [Audio classification task guide](../tasks/audio_classification)\n\n<PipelineTag pipeline=\"automatic-speech-recognition\"/>\n\n- A blog post on [boosting Wav2Vec2 with n-grams in \ud83e\udd17 Transformers](https://huggingface.co/blog/wav2vec2-with-ngram).\n- A blog post on how to [finetune Wav2Vec2 for English ASR with \ud83e\udd17 Transformers](https://huggingface.co/blog/fine-tune-wav2vec2-english).\n- A blog post on [finetuning XLS-R for Multi-Lingual ASR with \ud83e\udd17 Transformers](https://huggingface.co/blog/fine-tune-xlsr-wav2vec2).\n- A notebook on how to [create YouTube captions from any video by transcribing audio with Wav2Vec2](https://colab.research.google.com/github/Muennighoff/ytclipcc/blob/main/wav2vec_youtube_captions.ipynb). \ud83c\udf0e\n- [`Wav2Vec2ForCTC`] is supported by a notebook on [how to finetune a speech recognition model in English](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/speech_recognition.ipynb), and [how to finetune a speech recognition model in any language](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multi_lingual_speech_recognition.ipynb).\n- [Automatic speech recognition task guide](../tasks/asr)\n\n\ud83d\ude80 Deploy\n\n- A blog post on how to deploy Wav2Vec2 for [Automatic Speech Recognition with Hugging Face's Transformers & Amazon SageMaker](https://www.philschmid.de/automatic-speech-recognition-sagemaker).\n\n## Wav2Vec2Config\n\n[[autodoc]] Wav2Vec2Config\n\n## Wav2Vec2CTCTokenizer\n\n[[autodoc]] Wav2Vec2CTCTokenizer\n    - __call__\n    - save_vocabulary\n    - decode\n    - batch_decode\n    - set_target_lang\n\n## Wav2Vec2FeatureExtractor\n\n[[autodoc]] Wav2Vec2FeatureExtractor\n    - __call__\n\n## Wav2Vec2Processor\n\n[[autodoc]] Wav2Vec2Processor\n    - __call__\n    - pad\n    - from_pretrained\n    - save_pretrained\n    - batch_decode\n    - decode\n\n## Wav2Vec2ProcessorWithLM\n\n[[autodoc]] Wav2Vec2ProcessorWithLM\n    - __call__\n    - pad\n    - from_pretrained\n    - save_pretrained\n    - batch_decode\n    - decode\n\n### Decoding multiple audios\n\nIf you are planning to decode multiple batches of audios, you should consider using [`~Wav2Vec2ProcessorWithLM.batch_decode`] and passing an instantiated `multiprocessing.Pool`.\nOtherwise, [`~Wav2Vec2ProcessorWithLM.batch_decode`] performance will be slower than calling [`~Wav2Vec2ProcessorWithLM.decode`] for each audio individually, as it internally instantiates a new `Pool` for every call."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: d83cde30-3b60-412b-9931-e65401df1023
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2/", "title": "Wav2Vec2", "tokens": 3049, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"d83cde30-3b60-412b-9931-e65401df1023\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2/\", \"title\": \"Wav2Vec2\", \"tokens\": 3049, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1854bc00-e71c-43a1-a5cc-cdfe2c63b1cf\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2/\", \"title\": \"Wav2Vec2\", \"tokens\": 3049, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"348a691cf3244b01f4fa5e5586abc4a56c0ad49e3523e599cf0d8bfccbb408e9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d260ceb9-81b8-4d6d-aca0-d009343d6aaa\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2/\", \"title\": \"Wav2Vec2\", \"tokens\": 3049, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"458db8b4b9065c5003c7bf24097394107b9271b8f07e7607a1d5975a78b96e0c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ec06d96c-0605-409a-a369-5496bdfafa34\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7cb29b1fc564efa70e0b6542d82929556ce1233fc00a29d8734cc2e556116205\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5855, \"end_char_idx\": 8546, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1854bc00-e71c-43a1-a5cc-cdfe2c63b1cf", "doc_id": "1854bc00-e71c-43a1-a5cc-cdfe2c63b1cf", "ref_doc_id": "1854bc00-e71c-43a1-a5cc-cdfe2c63b1cf", "chroma:document": "- [Automatic speech recognition task guide](../tasks/asr)\n\n\ud83d\ude80 Deploy\n\n- A blog post on how to deploy Wav2Vec2 for [Automatic Speech Recognition with Hugging Face's Transformers & Amazon SageMaker](https://www.philschmid.de/automatic-speech-recognition-sagemaker).\n\n## Wav2Vec2Config\n\n[[autodoc]] Wav2Vec2Config\n\n## Wav2Vec2CTCTokenizer\n\n[[autodoc]] Wav2Vec2CTCTokenizer\n    - __call__\n    - save_vocabulary\n    - decode\n    - batch_decode\n    - set_target_lang\n\n## Wav2Vec2FeatureExtractor\n\n[[autodoc]] Wav2Vec2FeatureExtractor\n    - __call__\n\n## Wav2Vec2Processor\n\n[[autodoc]] Wav2Vec2Processor\n    - __call__\n    - pad\n    - from_pretrained\n    - save_pretrained\n    - batch_decode\n    - decode\n\n## Wav2Vec2ProcessorWithLM\n\n[[autodoc]] Wav2Vec2ProcessorWithLM\n    - __call__\n    - pad\n    - from_pretrained\n    - save_pretrained\n    - batch_decode\n    - decode\n\n### Decoding multiple audios\n\nIf you are planning to decode multiple batches of audios, you should consider using [`~Wav2Vec2ProcessorWithLM.batch_decode`] and passing an instantiated `multiprocessing.Pool`.\nOtherwise, [`~Wav2Vec2ProcessorWithLM.batch_decode`] performance will be slower than calling [`~Wav2Vec2ProcessorWithLM.decode`] for each audio individually, as it internally instantiates a new `Pool` for every call. See the example below:\n\n```python\n>>> # Let's see how to use a user-managed pool for batch decoding multiple audios\n>>> from multiprocessing import get_context\n>>> from transformers import AutoTokenizer, AutoProcessor, AutoModelForCTC\n>>> from datasets import load_dataset\n>>> import datasets\n>>> import torch\n\n>>> # import model, feature extractor, tokenizer\n>>> model = AutoModelForCTC.from_pretrained(\"patrickvonplaten/wav2vec2-base-100h-with-lm\").to(\"cuda\")\n>>> processor = AutoProcessor.from_pretrained(\"patrickvonplaten/wav2vec2-base-100h-with-lm\")\n\n>>> # load example dataset\n>>> dataset = load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\n>>> dataset = dataset.cast_column(\"audio\", datasets.Audio(sampling_rate=16_000))\n\n\n>>> def map_to_array(batch):\n...     batch[\"speech\"] = batch[\"audio\"][\"array\"]\n...     return batch\n\n\n>>> # prepare speech data for batch inference\n>>> dataset = dataset.map(map_to_array, remove_columns=[\"audio\"])\n\n\n>>> def map_to_pred(batch, pool):\n...     inputs = processor(batch[\"speech\"], sampling_rate=16_000, padding=True, return_tensors=\"pt\")\n...     inputs = {k: v.to(\"cuda\") for k, v in inputs.items()}\n\n...     with torch.no_grad():\n...         logits = model(**inputs).logits\n\n...     transcription = processor.batch_decode(logits.cpu().numpy(), pool).text\n...     batch[\"transcription\"] = transcription\n...     return batch"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: ec06d96c-0605-409a-a369-5496bdfafa34
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2/", "title": "Wav2Vec2", "tokens": 3049, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"ec06d96c-0605-409a-a369-5496bdfafa34\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2/\", \"title\": \"Wav2Vec2\", \"tokens\": 3049, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1854bc00-e71c-43a1-a5cc-cdfe2c63b1cf\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2/\", \"title\": \"Wav2Vec2\", \"tokens\": 3049, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"348a691cf3244b01f4fa5e5586abc4a56c0ad49e3523e599cf0d8bfccbb408e9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d83cde30-3b60-412b-9931-e65401df1023\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2/\", \"title\": \"Wav2Vec2\", \"tokens\": 3049, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"3eab8b57be27f59c8aedc11cc1429eaa456bb34c4f91d0861877b0a0b727b4a1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6926, \"end_char_idx\": 10652, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1854bc00-e71c-43a1-a5cc-cdfe2c63b1cf", "doc_id": "1854bc00-e71c-43a1-a5cc-cdfe2c63b1cf", "ref_doc_id": "1854bc00-e71c-43a1-a5cc-cdfe2c63b1cf", "chroma:document": "Otherwise, [`~Wav2Vec2ProcessorWithLM.batch_decode`] performance will be slower than calling [`~Wav2Vec2ProcessorWithLM.decode`] for each audio individually, as it internally instantiates a new `Pool` for every call. See the example below:\n\n```python\n>>> # Let's see how to use a user-managed pool for batch decoding multiple audios\n>>> from multiprocessing import get_context\n>>> from transformers import AutoTokenizer, AutoProcessor, AutoModelForCTC\n>>> from datasets import load_dataset\n>>> import datasets\n>>> import torch\n\n>>> # import model, feature extractor, tokenizer\n>>> model = AutoModelForCTC.from_pretrained(\"patrickvonplaten/wav2vec2-base-100h-with-lm\").to(\"cuda\")\n>>> processor = AutoProcessor.from_pretrained(\"patrickvonplaten/wav2vec2-base-100h-with-lm\")\n\n>>> # load example dataset\n>>> dataset = load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\n>>> dataset = dataset.cast_column(\"audio\", datasets.Audio(sampling_rate=16_000))\n\n\n>>> def map_to_array(batch):\n...     batch[\"speech\"] = batch[\"audio\"][\"array\"]\n...     return batch\n\n\n>>> # prepare speech data for batch inference\n>>> dataset = dataset.map(map_to_array, remove_columns=[\"audio\"])\n\n\n>>> def map_to_pred(batch, pool):\n...     inputs = processor(batch[\"speech\"], sampling_rate=16_000, padding=True, return_tensors=\"pt\")\n...     inputs = {k: v.to(\"cuda\") for k, v in inputs.items()}\n\n...     with torch.no_grad():\n...         logits = model(**inputs).logits\n\n...     transcription = processor.batch_decode(logits.cpu().numpy(), pool).text\n...     batch[\"transcription\"] = transcription\n...     return batch\n\n\n>>> # note: pool should be instantiated *after* `Wav2Vec2ProcessorWithLM`.\n>>> #       otherwise, the LM won't be available to the pool's sub-processes\n>>> # select number of processes and batch_size based on number of CPU cores available and on dataset size\n>>> with get_context(\"fork\").Pool(processes=2) as pool:\n...     result = dataset.map(\n...         map_to_pred, batched=True, batch_size=2, fn_kwargs={\"pool\": pool}, remove_columns=[\"speech\"]\n...     )\n\n>>> result[\"transcription\"][:2]\n['MISTER QUILTER IS THE APOSTLE OF THE MIDDLE CLASSES AND WE ARE GLAD TO WELCOME HIS GOSPEL', \"NOR IS MISTER COULTER'S MANNER LESS INTERESTING THAN HIS MATTER\"]\n```\n\n## Wav2Vec2 specific outputs\n\n[[autodoc]] models.wav2vec2_with_lm.processing_wav2vec2_with_lm.Wav2Vec2DecoderWithLMOutput\n\n[[autodoc]] models.wav2vec2.modeling_wav2vec2.Wav2Vec2BaseModelOutput\n\n[[autodoc]] models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForPreTrainingOutput\n\n[[autodoc]] models.wav2vec2.modeling_flax_wav2vec2.FlaxWav2Vec2BaseModelOutput\n\n[[autodoc]] models.wav2vec2.modeling_flax_wav2vec2.FlaxWav2Vec2ForPreTrainingOutput\n\n<frameworkcontent>\n<pt>\n\n## Wav2Vec2Model\n\n[[autodoc]] Wav2Vec2Model\n    - forward\n\n## Wav2Vec2ForCTC\n\n[[autodoc]] Wav2Vec2ForCTC\n    - forward\n    - load_adapter\n\n## Wav2Vec2ForSequenceClassification\n\n[[autodoc]] Wav2Vec2ForSequenceClassification\n    - forward\n\n## Wav2Vec2ForAudioFrameClassification\n\n[[autodoc]] Wav2Vec2ForAudioFrameClassification\n    - forward\n\n## Wav2Vec2ForXVector\n\n[[autodoc]] Wav2Vec2ForXVector\n    - forward\n\n## Wav2Vec2ForPreTraining\n\n[[autodoc]] Wav2Vec2ForPreTraining\n    - forward\n\n</pt>\n<tf>\n\n## TFWav2Vec2Model\n\n[[autodoc]] TFWav2Vec2Model\n    - call\n\n## TFWav2Vec2ForSequenceClassification\n\n[[autodoc]] TFWav2Vec2ForSequenceClassification\n    - call\n\n## TFWav2Vec2ForCTC\n\n[[autodoc]] TFWav2Vec2ForCTC\n    - call\n\n</tf>\n<jax>\n\n## FlaxWav2Vec2Model\n\n[[autodoc]] FlaxWav2Vec2Model\n    - __call__\n\n## FlaxWav2Vec2ForCTC\n\n[[autodoc]] FlaxWav2Vec2ForCTC\n    - __call__\n\n## FlaxWav2Vec2ForPreTraining\n\n[[autodoc]] FlaxWav2Vec2ForPreTraining\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 896a6511-433c-4d78-9a06-592f8d92eafc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt-sw3/", "title": "GPT-Sw3", "tokens": 727, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"896a6511-433c-4d78-9a06-592f8d92eafc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt-sw3/\", \"title\": \"GPT-Sw3\", \"tokens\": 727, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"03f3d8c2-2af6-4865-a9e7-6b55d87f8040\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt-sw3/\", \"title\": \"GPT-Sw3\", \"tokens\": 727, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"cda1bd487d3cedb2afed1cabb4526d76674c3b4fc2b1d5be4c1c64c0ea5d76e7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2216, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "03f3d8c2-2af6-4865-a9e7-6b55d87f8040", "doc_id": "03f3d8c2-2af6-4865-a9e7-6b55d87f8040", "ref_doc_id": "03f3d8c2-2af6-4865-a9e7-6b55d87f8040", "chroma:document": "# GPT-Sw3\n\n## Overview\n\nThe GPT-Sw3 model was first proposed in\n[Lessons Learned from GPT-SW3: Building the First Large-Scale Generative Language Model for Swedish](http://www.lrec-conf.org/proceedings/lrec2022/pdf/2022.lrec-1.376.pdf)\nby Ariel Ekgren, Amaru Cuba Gyllensten, Evangelia Gogoulou, Alice Heiman, Severine Verlinden, Joey \u00d6hman,\nFredrik Carlsson, Magnus Sahlgren.\n\nSince that first paper the authors have extended their work and trained new models on their new 1.2TB corpora named The Nordic Pile.\n\nGPT-Sw3 is a collection of large decoder-only pretrained transformer language models that were developed by AI Sweden\nin collaboration with RISE and the WASP WARA for Media and Language. GPT-Sw3 has been trained on a dataset containing\n320B tokens in Swedish, Norwegian, Danish, Icelandic, English, and programming code. The model was pretrained using a\ncausal language modeling (CLM) objective utilizing the NeMo Megatron GPT implementation.\n\nThis model was contributed by [AI Sweden Models](https://huggingface.co/AI-Sweden-Models).\n\n## Usage example\n\n```python\n>>> from transformers import AutoTokenizer, AutoModelForCausalLM\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"AI-Sweden-Models/gpt-sw3-356m\")\n>>> model = AutoModelForCausalLM.from_pretrained(\"AI-Sweden-Models/gpt-sw3-356m\")\n\n>>> input_ids = tokenizer(\"Tr\u00e4d \u00e4r fina f\u00f6r att\", return_tensors=\"pt\")[\"input_ids\"]\n\n>>> generated_token_ids = model.generate(inputs=input_ids, max_new_tokens=10, do_sample=True)[0]\n\n>>> print(tokenizer.decode(generated_token_ids))\nTr\u00e4d \u00e4r fina f\u00f6r att de \u00e4r f\u00e4rgstarka. Men ibland \u00e4r det fint\n```\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Causal language modeling task guide](../tasks/language_modeling)\n\n<Tip>\n\nThe implementation uses the `GPT2Model` coupled with our `GPTSw3Tokenizer`. Refer to [GPT2Model documentation](gpt2) \nfor API reference and examples.  \n\nNote that sentencepiece is required to use our tokenizer and can be installed with `pip install transformers[sentencepiece]` or `pip install sentencepiece`\n\n</Tip>\n\n## GPTSw3Tokenizer\n\n[[autodoc]] GPTSw3Tokenizer\n    - save_vocabulary"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 2e3765a6-dc7d-4f41-81e5-8279b9c2b8a1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vivit/", "title": "Video Vision Transformer (ViViT)", "tokens": 546, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"2e3765a6-dc7d-4f41-81e5-8279b9c2b8a1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vivit/\", \"title\": \"Video Vision Transformer (ViViT)\", \"tokens\": 546, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"44e57e4f-d6ec-4d13-93ef-4d8bb3ecdf1f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vivit/\", \"title\": \"Video Vision Transformer (ViViT)\", \"tokens\": 546, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"9e1344ac789f589b36666af2c3b753c56f45f7f2687c64ec6cb2006cbc257d78\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1878, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "44e57e4f-d6ec-4d13-93ef-4d8bb3ecdf1f", "doc_id": "44e57e4f-d6ec-4d13-93ef-4d8bb3ecdf1f", "ref_doc_id": "44e57e4f-d6ec-4d13-93ef-4d8bb3ecdf1f", "chroma:document": "# Video Vision Transformer (ViViT)\n\n## Overview\n\nThe Vivit model was proposed in [ViViT: A Video Vision Transformer](https://arxiv.org/abs/2103.15691) by Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lu\u010di\u0107, Cordelia Schmid.\nThe paper proposes one of the first successful pure-transformer based set of models for video understanding.\n\nThe abstract from the paper is the following:\n\n*We present pure-transformer based models for video classification, drawing upon the recent success of such models in image classification. Our model extracts spatio-temporal tokens from the input video, which are then encoded by a series of transformer layers. In order to handle the long sequences of tokens encountered in video, we propose several, efficient variants of our model which factorise the spatial- and temporal-dimensions of the input. Although transformer-based models are known to only be effective when large training datasets are available, we show how we can effectively regularise the model during training and leverage pretrained image models to be able to train on comparatively small datasets. We conduct thorough ablation studies, and achieve state-of-the-art results on multiple video classification benchmarks including Kinetics 400 and 600, Epic Kitchens, Something-Something v2 and Moments in Time, outperforming prior methods based on deep 3D convolutional networks.*\n\nThis model was contributed by [jegormeister](https://huggingface.co/jegormeister). The original code (written in JAX) can be found [here](https://github.com/google-research/scenic/tree/main/scenic/projects/vivit).\n\n## VivitConfig\n\n[[autodoc]] VivitConfig\n\n## VivitImageProcessor\n\n[[autodoc]] VivitImageProcessor\n    - preprocess\n\n## VivitModel\n\n[[autodoc]] VivitModel\n    - forward\n\n## VivitForVideoClassification\n\n[[autodoc]] transformers.VivitForVideoClassification\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 21a06dd9-73b9-49eb-b661-9f837f0da651
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/resnet/", "title": "ResNet", "tokens": 1080, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"21a06dd9-73b9-49eb-b661-9f837f0da651\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/resnet/\", \"title\": \"ResNet\", \"tokens\": 1080, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0e516bc6-b701-47c0-aed5-4b910536676d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/resnet/\", \"title\": \"ResNet\", \"tokens\": 1080, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"9a518f7acbd3d5a0dccce553bf52ee9ccc32ef6bcf7a238787b34bfe04fae61c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"33ab68de-58c8-4a19-9b4e-3b7162bf6567\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1cb00cc3a9758088f1fa4dca3c0879a6333099dad9a822387e59dde449ef047c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3320, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0e516bc6-b701-47c0-aed5-4b910536676d", "doc_id": "0e516bc6-b701-47c0-aed5-4b910536676d", "ref_doc_id": "0e516bc6-b701-47c0-aed5-4b910536676d", "chroma:document": "# ResNet\n\n## Overview\n\nThe ResNet model was proposed in [Deep Residual Learning for Image Recognition](https://arxiv.org/abs/1512.03385) by Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun. Our implementation follows the small changes made by [Nvidia](https://catalog.ngc.nvidia.com/orgs/nvidia/resources/resnet_50_v1_5_for_pytorch), we apply the `stride=2` for downsampling in bottleneck's `3x3` conv and not in the first `1x1`. This is generally known as \"ResNet v1.5\".\n\nResNet introduced residual connections, they allow to train networks with an unseen number of layers (up to 1000). ResNet won the 2015 ILSVRC & COCO competition, one important milestone in deep computer vision.\n\nThe abstract from the paper is the following:\n\n*Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers---8x deeper than VGG nets but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers.\nThe depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.*\n\nThe figure below illustrates the architecture of ResNet. Taken from the [original paper](https://arxiv.org/abs/1512.03385).\n\n<img width=\"600\" src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/resnet_architecture.png\"/>\n\nThis model was contributed by [Francesco](https://huggingface.co/Francesco). The TensorFlow version of this model was added by [amyeroberts](https://huggingface.co/amyeroberts). The original code can be found [here](https://github.com/KaimingHe/deep-residual-networks).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with ResNet.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`ResNetForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 33ab68de-58c8-4a19-9b4e-3b7162bf6567
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/resnet/", "title": "ResNet", "tokens": 1080, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"33ab68de-58c8-4a19-9b4e-3b7162bf6567\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/resnet/\", \"title\": \"ResNet\", \"tokens\": 1080, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0e516bc6-b701-47c0-aed5-4b910536676d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/resnet/\", \"title\": \"ResNet\", \"tokens\": 1080, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"9a518f7acbd3d5a0dccce553bf52ee9ccc32ef6bcf7a238787b34bfe04fae61c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"21a06dd9-73b9-49eb-b661-9f837f0da651\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/resnet/\", \"title\": \"ResNet\", \"tokens\": 1080, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"212c3b401ce3be93e6bb57c3e8e89f190d9489efe1851fd3caa5c61b5a847275\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1590, \"end_char_idx\": 3893, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0e516bc6-b701-47c0-aed5-4b910536676d", "doc_id": "0e516bc6-b701-47c0-aed5-4b910536676d", "ref_doc_id": "0e516bc6-b701-47c0-aed5-4b910536676d", "chroma:document": "The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.*\n\nThe figure below illustrates the architecture of ResNet. Taken from the [original paper](https://arxiv.org/abs/1512.03385).\n\n<img width=\"600\" src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/resnet_architecture.png\"/>\n\nThis model was contributed by [Francesco](https://huggingface.co/Francesco). The TensorFlow version of this model was added by [amyeroberts](https://huggingface.co/amyeroberts). The original code can be found [here](https://github.com/KaimingHe/deep-residual-networks).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with ResNet.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`ResNetForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## ResNetConfig\n\n[[autodoc]] ResNetConfig\n\n<frameworkcontent>\n<pt>\n\n## ResNetModel\n\n[[autodoc]] ResNetModel\n    - forward\n\n## ResNetForImageClassification\n\n[[autodoc]] ResNetForImageClassification\n    - forward\n\n</pt>\n<tf>\n\n## TFResNetModel\n\n[[autodoc]] TFResNetModel\n    - call\n\n## TFResNetForImageClassification\n\n[[autodoc]] TFResNetForImageClassification\n    - call\n\n</tf>\n<jax>\n\n## FlaxResNetModel\n\n[[autodoc]] FlaxResNetModel\n    - __call__\n\n## FlaxResNetForImageClassification\n\n[[autodoc]] FlaxResNetForImageClassification\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: cc71e0da-f74d-44f6-ab0d-bb4a1469ebb7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/van/", "title": "VAN", "tokens": 910, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"cc71e0da-f74d-44f6-ab0d-bb4a1469ebb7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/van/\", \"title\": \"VAN\", \"tokens\": 910, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1c15659f-5a57-4763-981d-55c586520f32\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/van/\", \"title\": \"VAN\", \"tokens\": 910, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"2aecbdb3674f289be77cac6b160357901e094a81d736201faae7b3c7db8cc4bb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3445, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1c15659f-5a57-4763-981d-55c586520f32", "doc_id": "1c15659f-5a57-4763-981d-55c586520f32", "ref_doc_id": "1c15659f-5a57-4763-981d-55c586520f32", "chroma:document": "# VAN\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we don't accept any new PRs changing its code.\n\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.30.0.\nYou can do so by running the following command: `pip install -U transformers==4.30.0`.\n\n</Tip>\n\n## Overview\n\nThe VAN model was proposed in [Visual Attention Network](https://arxiv.org/abs/2202.09741) by Meng-Hao Guo, Cheng-Ze Lu, Zheng-Ning Liu, Ming-Ming Cheng, Shi-Min Hu.\n\nThis paper introduces a new attention layer based on convolution operations able to capture both local and distant relationships. This is done by combining normal and large kernel convolution layers. The latter uses a dilated convolution to capture distant correlations.\n\nThe abstract from the paper is the following:\n\n*While originally designed for natural language processing tasks, the self-attention mechanism has recently taken various computer vision areas by storm. However, the 2D nature of images brings three challenges for applying self-attention in computer vision. (1) Treating images as 1D sequences neglects their 2D structures. (2) The quadratic complexity is too expensive for high-resolution images. (3) It only captures spatial adaptability but ignores channel adaptability. In this paper, we propose a novel large kernel attention (LKA) module to enable self-adaptive and long-range correlations in self-attention while avoiding the above issues. We further introduce a novel neural network based on LKA, namely Visual Attention Network (VAN). While extremely simple, VAN outperforms the state-of-the-art vision transformers and convolutional neural networks with a large margin in extensive experiments, including image classification, object detection, semantic segmentation, instance segmentation, etc. Code is available at [this https URL](https://github.com/Visual-Attention-Network/VAN-Classification).*\n\nTips:\n\n- VAN does not have an embedding layer, thus the `hidden_states` will have a length equal to the number of stages.\n\nThe figure below illustrates the architecture of a Visual Attention Layer. Taken from the [original paper](https://arxiv.org/abs/2202.09741).\n\n<img width=\"600\" src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/van_architecture.png\"/>\n\nThis model was contributed by [Francesco](https://huggingface.co/Francesco). The original code can be found [here](https://github.com/Visual-Attention-Network/VAN-Classification).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with VAN.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`VanForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## VanConfig\n\n[[autodoc]] VanConfig\n\n## VanModel\n\n[[autodoc]] VanModel\n    - forward\n\n## VanForImageClassification\n\n[[autodoc]] VanForImageClassification\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 7902132e-d33a-4a18-a3ec-7dbb660afe21
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/flaubert/", "title": "FlauBERT", "tokens": 1184, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"7902132e-d33a-4a18-a3ec-7dbb660afe21\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/flaubert/\", \"title\": \"FlauBERT\", \"tokens\": 1184, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"66e7a93a-7657-4100-b67e-b8461f10f111\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/flaubert/\", \"title\": \"FlauBERT\", \"tokens\": 1184, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"dd96c379122536ccd9148fd2543e646858a1aa22250276609ecbb9afc6abd94e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"97c3b2db-1fac-4516-bb3a-f8239d3a4088\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8af9a48db5e31ca8071c07a3de21b1a02f51533548a9c6c829c8fb3dce797b10\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2269, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "66e7a93a-7657-4100-b67e-b8461f10f111", "doc_id": "66e7a93a-7657-4100-b67e-b8461f10f111", "ref_doc_id": "66e7a93a-7657-4100-b67e-b8461f10f111", "chroma:document": "# FlauBERT\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=flaubert\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-flaubert-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/flaubert_small_cased\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nThe FlauBERT model was proposed in the paper [FlauBERT: Unsupervised Language Model Pre-training for French](https://arxiv.org/abs/1912.05372) by Hang Le et al. It's a transformer model pretrained using a masked language\nmodeling (MLM) objective (like BERT).\n\nThe abstract from the paper is the following:\n\n*Language models have become a key step to achieve state-of-the art results in many different Natural Language\nProcessing (NLP) tasks. Leveraging the huge amount of unlabeled texts nowadays available, they provide an efficient way\nto pre-train continuous word representations that can be fine-tuned for a downstream task, along with their\ncontextualization at the sentence level. This has been widely demonstrated for English using contextualized\nrepresentations (Dai and Le, 2015; Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al.,\n2019; Yang et al., 2019b). In this paper, we introduce and share FlauBERT, a model learned on a very large and\nheterogeneous French corpus. Models of different sizes are trained using the new CNRS (French National Centre for\nScientific Research) Jean Zay supercomputer. We apply our French language models to diverse NLP tasks (text\nclassification, paraphrasing, natural language inference, parsing, word sense disambiguation) and show that most of the\ntime they outperform other pretraining approaches. Different versions of FlauBERT as well as a unified evaluation\nprotocol for the downstream tasks, called FLUE (French Language Understanding Evaluation), are shared to the research\ncommunity for further reproducible experiments in French NLP.*\n\nThis model was contributed by [formiel](https://huggingface.co/formiel). The original code can be found [here](https://github.com/getalp/Flaubert).\n\nTips:\n- Like RoBERTa, without the sentence ordering prediction (so just trained on the MLM objective)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 97c3b2db-1fac-4516-bb3a-f8239d3a4088
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/flaubert/", "title": "FlauBERT", "tokens": 1184, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"97c3b2db-1fac-4516-bb3a-f8239d3a4088\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/flaubert/\", \"title\": \"FlauBERT\", \"tokens\": 1184, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"66e7a93a-7657-4100-b67e-b8461f10f111\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/flaubert/\", \"title\": \"FlauBERT\", \"tokens\": 1184, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"dd96c379122536ccd9148fd2543e646858a1aa22250276609ecbb9afc6abd94e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7902132e-d33a-4a18-a3ec-7dbb660afe21\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/flaubert/\", \"title\": \"FlauBERT\", \"tokens\": 1184, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"499c6a64db3d5847364ea3b9cf4315c9c2c2b556e6551881846861e589ce1c6c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 559, \"end_char_idx\": 3902, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "66e7a93a-7657-4100-b67e-b8461f10f111", "doc_id": "66e7a93a-7657-4100-b67e-b8461f10f111", "ref_doc_id": "66e7a93a-7657-4100-b67e-b8461f10f111", "chroma:document": "It's a transformer model pretrained using a masked language\nmodeling (MLM) objective (like BERT).\n\nThe abstract from the paper is the following:\n\n*Language models have become a key step to achieve state-of-the art results in many different Natural Language\nProcessing (NLP) tasks. Leveraging the huge amount of unlabeled texts nowadays available, they provide an efficient way\nto pre-train continuous word representations that can be fine-tuned for a downstream task, along with their\ncontextualization at the sentence level. This has been widely demonstrated for English using contextualized\nrepresentations (Dai and Le, 2015; Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al.,\n2019; Yang et al., 2019b). In this paper, we introduce and share FlauBERT, a model learned on a very large and\nheterogeneous French corpus. Models of different sizes are trained using the new CNRS (French National Centre for\nScientific Research) Jean Zay supercomputer. We apply our French language models to diverse NLP tasks (text\nclassification, paraphrasing, natural language inference, parsing, word sense disambiguation) and show that most of the\ntime they outperform other pretraining approaches. Different versions of FlauBERT as well as a unified evaluation\nprotocol for the downstream tasks, called FLUE (French Language Understanding Evaluation), are shared to the research\ncommunity for further reproducible experiments in French NLP.*\n\nThis model was contributed by [formiel](https://huggingface.co/formiel). The original code can be found [here](https://github.com/getalp/Flaubert).\n\nTips:\n- Like RoBERTa, without the sentence ordering prediction (so just trained on the MLM objective).\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## FlaubertConfig\n\n[[autodoc]] FlaubertConfig\n\n## FlaubertTokenizer\n\n[[autodoc]] FlaubertTokenizer\n\n<frameworkcontent>\n<pt>\n\n## FlaubertModel\n\n[[autodoc]] FlaubertModel\n    - forward\n\n## FlaubertWithLMHeadModel\n\n[[autodoc]] FlaubertWithLMHeadModel\n    - forward\n\n## FlaubertForSequenceClassification\n\n[[autodoc]] FlaubertForSequenceClassification\n    - forward\n\n## FlaubertForMultipleChoice\n\n[[autodoc]] FlaubertForMultipleChoice\n    - forward\n\n## FlaubertForTokenClassification\n\n[[autodoc]] FlaubertForTokenClassification\n    - forward\n\n## FlaubertForQuestionAnsweringSimple\n\n[[autodoc]] FlaubertForQuestionAnsweringSimple\n    - forward\n\n## FlaubertForQuestionAnswering\n\n[[autodoc]] FlaubertForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFFlaubertModel\n\n[[autodoc]] TFFlaubertModel\n    - call\n\n## TFFlaubertWithLMHeadModel\n\n[[autodoc]] TFFlaubertWithLMHeadModel\n    - call\n\n## TFFlaubertForSequenceClassification\n\n[[autodoc]] TFFlaubertForSequenceClassification\n    - call\n\n## TFFlaubertForMultipleChoice\n\n[[autodoc]] TFFlaubertForMultipleChoice\n    - call\n\n## TFFlaubertForTokenClassification\n\n[[autodoc]] TFFlaubertForTokenClassification\n    - call\n\n## TFFlaubertForQuestionAnsweringSimple\n\n[[autodoc]] TFFlaubertForQuestionAnsweringSimple\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 1c6b9b76-ea0a-49ac-a0e5-82a145d7b7ba
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tvp/", "title": "TVP", "tokens": 2307, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"1c6b9b76-ea0a-49ac-a0e5-82a145d7b7ba\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tvp/\", \"title\": \"TVP\", \"tokens\": 2307, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"81d142dc-de24-4cea-8f32-397d460b42a3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tvp/\", \"title\": \"TVP\", \"tokens\": 2307, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0abd09683ed72070347ad35b41e34c742fd4aa9c9f9f3fcbdc13dd3af10f02a4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6e3ed60a-610b-4493-81cf-d00bc3efbbec\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"52c6c2d35a21e77b21af6968aed82910cc28c1ea417ea194e450e40142b01820\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3338, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "81d142dc-de24-4cea-8f32-397d460b42a3", "doc_id": "81d142dc-de24-4cea-8f32-397d460b42a3", "ref_doc_id": "81d142dc-de24-4cea-8f32-397d460b42a3", "chroma:document": "# TVP\n\n## Overview\n\nThe text-visual prompting (TVP) framework was proposed in the paper [Text-Visual Prompting for Efficient 2D Temporal Video Grounding](https://arxiv.org/abs/2303.04995) by Yimeng Zhang, Xin Chen, Jinghan Jia, Sijia Liu, Ke Ding.\n\nThe abstract from the paper is the following:\n\n*In this paper, we study the problem of temporal video grounding (TVG), which aims to predict the starting/ending time points of moments described by a text sentence within a long untrimmed video. Benefiting from fine-grained 3D visual features, the TVG techniques have achieved remarkable progress in recent years. However, the high complexity of 3D convolutional neural networks (CNNs) makes extracting dense 3D visual features time-consuming, which calls for intensive memory and computing resources. Towards efficient TVG, we propose a novel text-visual prompting (TVP) framework, which incorporates optimized perturbation patterns (that we call \u2018prompts\u2019) into both visual inputs and textual features of a TVG model. In sharp contrast to 3D CNNs, we show that TVP allows us to effectively co-train vision encoder and language encoder in a 2D TVG model and improves the performance of cross-modal feature fusion using only low-complexity sparse 2D visual features. Further, we propose a Temporal-Distance IoU (TDIoU) loss for efficient learning of TVG. Experiments on two benchmark datasets, Charades-STA and ActivityNet Captions datasets, empirically show that the proposed TVP significantly boosts the performance of 2D TVG (e.g., 9.79% improvement on Charades-STA and 30.77% improvement on ActivityNet Captions) and achieves 5\u00d7 inference acceleration over TVG using 3D visual features.*\n\nThis research addresses temporal video grounding (TVG), which is the process of pinpointing the start and end times of specific events in a long video, as described by a text sentence. Text-visual prompting (TVP), is proposed to enhance TVG. TVP involves integrating specially designed patterns, known as 'prompts', into both the visual (image-based) and textual (word-based) input components of a TVG model. These prompts provide additional spatial-temporal context, improving the model's ability to accurately determine event timings in the video. The approach employs 2D visual inputs in place of 3D ones. Although 3D inputs offer more spatial-temporal detail, they are also more time-consuming to process. The use of 2D inputs with the prompting method aims to provide similar levels of context and accuracy more efficiently.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/tvp_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> TVP architecture. Taken from the <a href=\"https://arxiv.org/abs/2303.04995\">original paper.</a> </small>\n\nThis model was contributed by [Jiqing Feng](https://huggingface.co/Jiqing). The original code can be found [here](https://github.com/intel/TVP).\n\n## Usage tips and examples\n\nPrompts are optimized perturbation patterns, which would be added to input video frames or text features. Universal set refers to using the same exact set of prompts for any input, this means that these prompts are added consistently to all video frames and text features, regardless of the input's content.\n\nTVP consists of a visual encoder and cross-modal encoder."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 6e3ed60a-610b-4493-81cf-d00bc3efbbec
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tvp/", "title": "TVP", "tokens": 2307, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"6e3ed60a-610b-4493-81cf-d00bc3efbbec\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tvp/\", \"title\": \"TVP\", \"tokens\": 2307, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"81d142dc-de24-4cea-8f32-397d460b42a3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tvp/\", \"title\": \"TVP\", \"tokens\": 2307, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0abd09683ed72070347ad35b41e34c742fd4aa9c9f9f3fcbdc13dd3af10f02a4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1c6b9b76-ea0a-49ac-a0e5-82a145d7b7ba\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tvp/\", \"title\": \"TVP\", \"tokens\": 2307, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8b525d284742225121aedef02347b2eb409a3089479721f281db0fc97d36de9a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"056c7bb8-aeea-470b-9d94-086c9d44f498\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"04e76286dc9cbeb8bc68dc4421ef5f180059f9b77b56d0abc68d68b3523c4af5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1688, \"end_char_idx\": 4266, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "81d142dc-de24-4cea-8f32-397d460b42a3", "doc_id": "81d142dc-de24-4cea-8f32-397d460b42a3", "ref_doc_id": "81d142dc-de24-4cea-8f32-397d460b42a3", "chroma:document": "*\n\nThis research addresses temporal video grounding (TVG), which is the process of pinpointing the start and end times of specific events in a long video, as described by a text sentence. Text-visual prompting (TVP), is proposed to enhance TVG. TVP involves integrating specially designed patterns, known as 'prompts', into both the visual (image-based) and textual (word-based) input components of a TVG model. These prompts provide additional spatial-temporal context, improving the model's ability to accurately determine event timings in the video. The approach employs 2D visual inputs in place of 3D ones. Although 3D inputs offer more spatial-temporal detail, they are also more time-consuming to process. The use of 2D inputs with the prompting method aims to provide similar levels of context and accuracy more efficiently.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/tvp_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> TVP architecture. Taken from the <a href=\"https://arxiv.org/abs/2303.04995\">original paper.</a> </small>\n\nThis model was contributed by [Jiqing Feng](https://huggingface.co/Jiqing). The original code can be found [here](https://github.com/intel/TVP).\n\n## Usage tips and examples\n\nPrompts are optimized perturbation patterns, which would be added to input video frames or text features. Universal set refers to using the same exact set of prompts for any input, this means that these prompts are added consistently to all video frames and text features, regardless of the input's content.\n\nTVP consists of a visual encoder and cross-modal encoder. A universal set of visual prompts and text prompts to be integrated into sampled video frames and textual features, respectively. Specially, a set of different visual prompts are applied to uniformly-sampled frames of one untrimmed video in order.\n\nThe goal of this model is to incorporate trainable prompts into both visual inputs and textual features to temporal video grounding(TVG) problems.\nIn principle, one can apply any visual, cross-modal encoder in the proposed architecture.\n\nThe [`TvpProcessor`] wraps [`BertTokenizer`] and [`TvpImageProcessor`] into a single instance to both\nencode the text and prepare the images respectively.\n\nThe following example shows how to run temporal video grounding using [`TvpProcessor`] and [`TvpForVideoGrounding`].\n```python\nimport av\nimport cv2\nimport numpy as np\nimport torch\nfrom huggingface_hub import hf_hub_download\nfrom transformers import AutoProcessor, TvpForVideoGrounding"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 056c7bb8-aeea-470b-9d94-086c9d44f498
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tvp/", "title": "TVP", "tokens": 2307, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"056c7bb8-aeea-470b-9d94-086c9d44f498\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tvp/\", \"title\": \"TVP\", \"tokens\": 2307, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"81d142dc-de24-4cea-8f32-397d460b42a3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tvp/\", \"title\": \"TVP\", \"tokens\": 2307, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0abd09683ed72070347ad35b41e34c742fd4aa9c9f9f3fcbdc13dd3af10f02a4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6e3ed60a-610b-4493-81cf-d00bc3efbbec\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tvp/\", \"title\": \"TVP\", \"tokens\": 2307, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"83147e35a255b53da2f7372b5f228c88c41bd8e26e6e52da84ca51853ff76103\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ba496f4d-df59-4383-a6dd-a59b9946966a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6f83deaeaed929a8cc079e479a2187704f78b0faa245c7a72eef01c44edc7a2e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2522, \"end_char_idx\": 6125, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "81d142dc-de24-4cea-8f32-397d460b42a3", "doc_id": "81d142dc-de24-4cea-8f32-397d460b42a3", "ref_doc_id": "81d142dc-de24-4cea-8f32-397d460b42a3", "chroma:document": "<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/tvp_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> TVP architecture. Taken from the <a href=\"https://arxiv.org/abs/2303.04995\">original paper.</a> </small>\n\nThis model was contributed by [Jiqing Feng](https://huggingface.co/Jiqing). The original code can be found [here](https://github.com/intel/TVP).\n\n## Usage tips and examples\n\nPrompts are optimized perturbation patterns, which would be added to input video frames or text features. Universal set refers to using the same exact set of prompts for any input, this means that these prompts are added consistently to all video frames and text features, regardless of the input's content.\n\nTVP consists of a visual encoder and cross-modal encoder. A universal set of visual prompts and text prompts to be integrated into sampled video frames and textual features, respectively. Specially, a set of different visual prompts are applied to uniformly-sampled frames of one untrimmed video in order.\n\nThe goal of this model is to incorporate trainable prompts into both visual inputs and textual features to temporal video grounding(TVG) problems.\nIn principle, one can apply any visual, cross-modal encoder in the proposed architecture.\n\nThe [`TvpProcessor`] wraps [`BertTokenizer`] and [`TvpImageProcessor`] into a single instance to both\nencode the text and prepare the images respectively.\n\nThe following example shows how to run temporal video grounding using [`TvpProcessor`] and [`TvpForVideoGrounding`].\n```python\nimport av\nimport cv2\nimport numpy as np\nimport torch\nfrom huggingface_hub import hf_hub_download\nfrom transformers import AutoProcessor, TvpForVideoGrounding\n\n\ndef pyav_decode(container, sampling_rate, num_frames, clip_idx, num_clips, target_fps):\n    '''\n    Convert the video from its original fps to the target_fps and decode the video with PyAV decoder.\n    Args:\n        container (container): pyav container.\n        sampling_rate (int): frame sampling rate (interval between two sampled frames).\n        num_frames (int): number of frames to sample.\n        clip_idx (int): if clip_idx is -1, perform random temporal sampling.\n            If clip_idx is larger than -1, uniformly split the video to num_clips\n            clips, and select the clip_idx-th video clip.\n        num_clips (int): overall number of clips to uniformly sample from the given video.\n        target_fps (int): the input video may have different fps, convert it to\n            the target video fps before frame sampling.\n    Returns:\n        frames (tensor): decoded frames from the video. Return None if the no\n            video stream was found.\n        fps (float): the number of frames per second of the video.\n    '''\n    video = container.streams.video[0]\n    fps = float(video.average_rate)\n    clip_size = sampling_rate * num_frames / target_fps * fps\n    delta = max(num_frames - clip_size, 0)\n    start_idx = delta * clip_idx / num_clips\n    end_idx = start_idx + clip_size - 1\n    timebase = video.duration / num_frames\n    video_start_pts = int(start_idx * timebase)\n    video_end_pts = int(end_idx * timebase)\n    seek_offset = max(video_start_pts - 1024, 0)\n    container.seek(seek_offset, any_frame=False, backward=True, stream=video)\n    frames = {}\n    for frame in container.decode(video=0):\n        if frame.pts < video_start_pts:\n            continue\n        frames[frame.pts] = frame\n        if frame.pts > video_end_pts:\n            break\n    frames = [frames[pts] for pts in sorted(frames)]\n    return frames, fps"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: ba496f4d-df59-4383-a6dd-a59b9946966a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tvp/", "title": "TVP", "tokens": 2307, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"ba496f4d-df59-4383-a6dd-a59b9946966a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tvp/\", \"title\": \"TVP\", \"tokens\": 2307, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"81d142dc-de24-4cea-8f32-397d460b42a3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tvp/\", \"title\": \"TVP\", \"tokens\": 2307, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0abd09683ed72070347ad35b41e34c742fd4aa9c9f9f3fcbdc13dd3af10f02a4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"056c7bb8-aeea-470b-9d94-086c9d44f498\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tvp/\", \"title\": \"TVP\", \"tokens\": 2307, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"698714e29e942e36988b0302f69baa734081849735925bc277894abf464d66c9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9b5d9323-afa7-40fc-8ecd-ea320c09d62d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1d008542f6e1250f373683d4b89bb1e9a5cc15926d23a73c36c8aa343b0a84eb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6128, \"end_char_idx\": 9006, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "81d142dc-de24-4cea-8f32-397d460b42a3", "doc_id": "81d142dc-de24-4cea-8f32-397d460b42a3", "ref_doc_id": "81d142dc-de24-4cea-8f32-397d460b42a3", "chroma:document": "def decode(container, sampling_rate, num_frames, clip_idx, num_clips, target_fps):\n    '''\n    Decode the video and perform temporal sampling.\n    Args:\n        container (container): pyav container.\n        sampling_rate (int): frame sampling rate (interval between two sampled frames).\n        num_frames (int): number of frames to sample.\n        clip_idx (int): if clip_idx is -1, perform random temporal sampling.\n            If clip_idx is larger than -1, uniformly split the video to num_clips\n            clips, and select the clip_idx-th video clip.\n        num_clips (int): overall number of clips to uniformly sample from the given video.\n        target_fps (int): the input video may have different fps, convert it to\n            the target video fps before frame sampling.\n    Returns:\n        frames (tensor): decoded frames from the video.\n    '''\n    assert clip_idx >= -2, \"Not a valied clip_idx {}\".format(clip_idx)\n    frames, fps = pyav_decode(container, sampling_rate, num_frames, clip_idx, num_clips, target_fps)\n    clip_size = sampling_rate * num_frames / target_fps * fps\n    index = np.linspace(0, clip_size - 1, num_frames)\n    index = np.clip(index, 0, len(frames) - 1).astype(np.int64)\n    frames = np.array([frames[idx].to_rgb().to_ndarray() for idx in index])\n    frames = frames.transpose(0, 3, 1, 2)\n    return frames\n\n\nfile = hf_hub_download(repo_id=\"Intel/tvp_demo\", filename=\"AK2KG.mp4\", repo_type=\"dataset\")\nmodel = TvpForVideoGrounding.from_pretrained(\"Intel/tvp-base\")\n\ndecoder_kwargs = dict(\n    container=av.open(file, metadata_errors=\"ignore\"),\n    sampling_rate=1,\n    num_frames=model.config.num_frames,\n    clip_idx=0,\n    num_clips=1,\n    target_fps=3,\n)\nraw_sampled_frms = decode(**decoder_kwargs)\n\ntext = \"a person is sitting on a bed.\"\nprocessor = AutoProcessor.from_pretrained(\"Intel/tvp-base\")\nmodel_inputs = processor(\n    text=[text], videos=list(raw_sampled_frms), return_tensors=\"pt\", max_text_length=100#, size=size\n)\n\nmodel_inputs[\"pixel_values\"] = model_inputs[\"pixel_values\"].to(model.dtype)\noutput = model(**model_inputs)\n\ndef get_video_duration(filename):\n    cap = cv2.VideoCapture(filename)\n    if cap.isOpened():\n        rate = cap.get(5)\n        frame_num = cap.get(7)\n        duration = frame_num/rate\n        return duration\n    return -1\n\nduration = get_video_duration(file)\nstart, end = processor.post_process_video_grounding(output.logits, duration)\n\nprint(f\"The time slot of the video corresponding to the text \\\"{text}\\\" is from {start}s to {end}s\")\n```\n\nTips:\n\n- This implementation of TVP uses [`BertTokenizer`] to generate text embeddings and Resnet-50 model to compute visual embeddings.\n- Checkpoints for pre-trained [tvp-base](https://huggingface.co/Intel/tvp-base) is released.\n- Please refer to [Table 2](https://arxiv.org/pdf/2303.04995.pdf) for TVP's performance on Temporal Video Grounding task."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 9b5d9323-afa7-40fc-8ecd-ea320c09d62d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tvp/", "title": "TVP", "tokens": 2307, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"9b5d9323-afa7-40fc-8ecd-ea320c09d62d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tvp/\", \"title\": \"TVP\", \"tokens\": 2307, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"81d142dc-de24-4cea-8f32-397d460b42a3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tvp/\", \"title\": \"TVP\", \"tokens\": 2307, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0abd09683ed72070347ad35b41e34c742fd4aa9c9f9f3fcbdc13dd3af10f02a4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ba496f4d-df59-4383-a6dd-a59b9946966a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tvp/\", \"title\": \"TVP\", \"tokens\": 2307, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8f4d944c8109c58944ade423fea1de9d189c16c229613834953823b290abfec5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7481, \"end_char_idx\": 9294, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "81d142dc-de24-4cea-8f32-397d460b42a3", "doc_id": "81d142dc-de24-4cea-8f32-397d460b42a3", "ref_doc_id": "81d142dc-de24-4cea-8f32-397d460b42a3", "chroma:document": "file = hf_hub_download(repo_id=\"Intel/tvp_demo\", filename=\"AK2KG.mp4\", repo_type=\"dataset\")\nmodel = TvpForVideoGrounding.from_pretrained(\"Intel/tvp-base\")\n\ndecoder_kwargs = dict(\n    container=av.open(file, metadata_errors=\"ignore\"),\n    sampling_rate=1,\n    num_frames=model.config.num_frames,\n    clip_idx=0,\n    num_clips=1,\n    target_fps=3,\n)\nraw_sampled_frms = decode(**decoder_kwargs)\n\ntext = \"a person is sitting on a bed.\"\nprocessor = AutoProcessor.from_pretrained(\"Intel/tvp-base\")\nmodel_inputs = processor(\n    text=[text], videos=list(raw_sampled_frms), return_tensors=\"pt\", max_text_length=100#, size=size\n)\n\nmodel_inputs[\"pixel_values\"] = model_inputs[\"pixel_values\"].to(model.dtype)\noutput = model(**model_inputs)\n\ndef get_video_duration(filename):\n    cap = cv2.VideoCapture(filename)\n    if cap.isOpened():\n        rate = cap.get(5)\n        frame_num = cap.get(7)\n        duration = frame_num/rate\n        return duration\n    return -1\n\nduration = get_video_duration(file)\nstart, end = processor.post_process_video_grounding(output.logits, duration)\n\nprint(f\"The time slot of the video corresponding to the text \\\"{text}\\\" is from {start}s to {end}s\")\n```\n\nTips:\n\n- This implementation of TVP uses [`BertTokenizer`] to generate text embeddings and Resnet-50 model to compute visual embeddings.\n- Checkpoints for pre-trained [tvp-base](https://huggingface.co/Intel/tvp-base) is released.\n- Please refer to [Table 2](https://arxiv.org/pdf/2303.04995.pdf) for TVP's performance on Temporal Video Grounding task.\n\n\n## TvpConfig\n\n[[autodoc]] TvpConfig\n\n## TvpImageProcessor\n\n[[autodoc]] TvpImageProcessor\n    - preprocess\n\n## TvpProcessor\n\n[[autodoc]] TvpProcessor\n    - __call__\n\n## TvpModel\n\n[[autodoc]] TvpModel\n    - forward\n\n## TvpForVideoGrounding\n\n[[autodoc]] TvpForVideoGrounding\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: f772bfbe-8f77-46ce-b196-962f320d6ed7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/fnet/", "title": "FNet", "tokens": 979, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"f772bfbe-8f77-46ce-b196-962f320d6ed7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fnet/\", \"title\": \"FNet\", \"tokens\": 979, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f533a8fd-6d14-452f-a415-3b779480c270\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fnet/\", \"title\": \"FNet\", \"tokens\": 979, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"c86a63e10dde447cd5b4be40679b13780a02b11a4cfb1cb887a4661dd2c4f559\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"22ce4ec0-65b3-43d3-8d12-e21d161229b4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"efdf391577bc1f2dc0c44dbc8bf420acfd67b596e35ddc5964401d114c481485\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2285, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f533a8fd-6d14-452f-a415-3b779480c270", "doc_id": "f533a8fd-6d14-452f-a415-3b779480c270", "ref_doc_id": "f533a8fd-6d14-452f-a415-3b779480c270", "chroma:document": "# FNet\n\n## Overview\n\nThe FNet model was proposed in [FNet: Mixing Tokens with Fourier Transforms](https://arxiv.org/abs/2105.03824) by\nJames Lee-Thorp, Joshua Ainslie, Ilya Eckstein, Santiago Ontanon. The model replaces the self-attention layer in a BERT\nmodel with a fourier transform which returns only the real parts of the transform. The model is significantly faster\nthan the BERT model because it has fewer parameters and is more memory efficient. The model achieves about 92-97%\naccuracy of BERT counterparts on GLUE benchmark, and trains much faster than the BERT model. The abstract from the\npaper is the following:\n\n*We show that Transformer encoder architectures can be sped up, with limited accuracy costs, by replacing the\nself-attention sublayers with simple linear transformations that \"mix\" input tokens. These linear mixers, along with\nstandard nonlinearities in feed-forward layers, prove competent at modeling semantic relationships in several text\nclassification tasks. Most surprisingly, we find that replacing the self-attention sublayer in a Transformer encoder\nwith a standard, unparameterized Fourier Transform achieves 92-97% of the accuracy of BERT counterparts on the GLUE\nbenchmark, but trains 80% faster on GPUs and 70% faster on TPUs at standard 512 input lengths. At longer input lengths,\nour FNet model is significantly faster: when compared to the \"efficient\" Transformers on the Long Range Arena\nbenchmark, FNet matches the accuracy of the most accurate models, while outpacing the fastest models across all\nsequence lengths on GPUs (and across relatively shorter lengths on TPUs). Finally, FNet has a light memory footprint\nand is particularly efficient at smaller model sizes; for a fixed speed and accuracy budget, small FNet models\noutperform Transformer counterparts.*\n\nThis model was contributed by [gchhablani](https://huggingface.co/gchhablani). The original code can be found [here](https://github.com/google-research/google-research/tree/master/f_net).\n\n## Usage tips\n\nThe model was trained without an attention mask as it is based on Fourier Transform. The model was trained with \nmaximum sequence length 512 which includes pad tokens. Hence, it is highly recommended to use the same maximum \nsequence length for fine-tuning and inference."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 22ce4ec0-65b3-43d3-8d12-e21d161229b4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/fnet/", "title": "FNet", "tokens": 979, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"22ce4ec0-65b3-43d3-8d12-e21d161229b4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fnet/\", \"title\": \"FNet\", \"tokens\": 979, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f533a8fd-6d14-452f-a415-3b779480c270\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fnet/\", \"title\": \"FNet\", \"tokens\": 979, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"c86a63e10dde447cd5b4be40679b13780a02b11a4cfb1cb887a4661dd2c4f559\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f772bfbe-8f77-46ce-b196-962f320d6ed7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fnet/\", \"title\": \"FNet\", \"tokens\": 979, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"69f47258d28ef3974ce08422b9375eda82f577f23023f7ac7c8f1c86c284d2b2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 454, \"end_char_idx\": 3508, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f533a8fd-6d14-452f-a415-3b779480c270", "doc_id": "f533a8fd-6d14-452f-a415-3b779480c270", "ref_doc_id": "f533a8fd-6d14-452f-a415-3b779480c270", "chroma:document": "The model achieves about 92-97%\naccuracy of BERT counterparts on GLUE benchmark, and trains much faster than the BERT model. The abstract from the\npaper is the following:\n\n*We show that Transformer encoder architectures can be sped up, with limited accuracy costs, by replacing the\nself-attention sublayers with simple linear transformations that \"mix\" input tokens. These linear mixers, along with\nstandard nonlinearities in feed-forward layers, prove competent at modeling semantic relationships in several text\nclassification tasks. Most surprisingly, we find that replacing the self-attention sublayer in a Transformer encoder\nwith a standard, unparameterized Fourier Transform achieves 92-97% of the accuracy of BERT counterparts on the GLUE\nbenchmark, but trains 80% faster on GPUs and 70% faster on TPUs at standard 512 input lengths. At longer input lengths,\nour FNet model is significantly faster: when compared to the \"efficient\" Transformers on the Long Range Arena\nbenchmark, FNet matches the accuracy of the most accurate models, while outpacing the fastest models across all\nsequence lengths on GPUs (and across relatively shorter lengths on TPUs). Finally, FNet has a light memory footprint\nand is particularly efficient at smaller model sizes; for a fixed speed and accuracy budget, small FNet models\noutperform Transformer counterparts.*\n\nThis model was contributed by [gchhablani](https://huggingface.co/gchhablani). The original code can be found [here](https://github.com/google-research/google-research/tree/master/f_net).\n\n## Usage tips\n\nThe model was trained without an attention mask as it is based on Fourier Transform. The model was trained with \nmaximum sequence length 512 which includes pad tokens. Hence, it is highly recommended to use the same maximum \nsequence length for fine-tuning and inference.\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## FNetConfig\n\n[[autodoc]] FNetConfig\n\n## FNetTokenizer\n\n[[autodoc]] FNetTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## FNetTokenizerFast\n\n[[autodoc]] FNetTokenizerFast\n\n## FNetModel\n\n[[autodoc]] FNetModel\n    - forward\n\n## FNetForPreTraining\n\n[[autodoc]] FNetForPreTraining\n    - forward\n\n## FNetForMaskedLM\n\n[[autodoc]] FNetForMaskedLM\n    - forward\n\n## FNetForNextSentencePrediction\n\n[[autodoc]] FNetForNextSentencePrediction\n    - forward\n\n## FNetForSequenceClassification\n\n[[autodoc]] FNetForSequenceClassification\n    - forward\n\n## FNetForMultipleChoice\n\n[[autodoc]] FNetForMultipleChoice\n    - forward\n\n## FNetForTokenClassification\n\n[[autodoc]] FNetForTokenClassification\n    - forward\n\n## FNetForQuestionAnswering\n\n[[autodoc]] FNetForQuestionAnswering\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: ea3804ad-65d0-4a57-9154-43a5a7c9dd77
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/unispeech/", "title": "UniSpeech", "tokens": 731, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"ea3804ad-65d0-4a57-9154-43a5a7c9dd77\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/unispeech/\", \"title\": \"UniSpeech\", \"tokens\": 731, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3314c1f9-b088-40d4-a920-af33546eee35\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/unispeech/\", \"title\": \"UniSpeech\", \"tokens\": 731, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"16135194de13690dfbf87cdd0fabb3b5fd5bac88bc002d37d4c1bacffa863dd7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2468, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3314c1f9-b088-40d4-a920-af33546eee35", "doc_id": "3314c1f9-b088-40d4-a920-af33546eee35", "ref_doc_id": "3314c1f9-b088-40d4-a920-af33546eee35", "chroma:document": "# UniSpeech\n\n## Overview\n\nThe UniSpeech model was proposed in [UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data](https://arxiv.org/abs/2101.07597) by Chengyi Wang, Yu Wu, Yao Qian, Kenichi Kumatani, Shujie Liu, Furu Wei, Michael\nZeng, Xuedong Huang .\n\nThe abstract from the paper is the following:\n\n*In this paper, we propose a unified pre-training approach called UniSpeech to learn speech representations with both\nunlabeled and labeled data, in which supervised phonetic CTC learning and phonetically-aware contrastive\nself-supervised learning are conducted in a multi-task learning manner. The resultant representations can capture\ninformation more correlated with phonetic structures and improve the generalization across languages and domains. We\nevaluate the effectiveness of UniSpeech for cross-lingual representation learning on public CommonVoice corpus. The\nresults show that UniSpeech outperforms self-supervised pretraining and supervised transfer learning for speech\nrecognition by a maximum of 13.4% and 17.8% relative phone error rate reductions respectively (averaged over all\ntesting languages). The transferability of UniSpeech is also demonstrated on a domain-shift speech recognition task,\ni.e., a relative word error rate reduction of 6% against the previous approach.*\n\nThis model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The Authors' code can be\nfound [here](https://github.com/microsoft/UniSpeech/tree/main/UniSpeech).\n\n## Usage tips\n\n- UniSpeech is a speech model that accepts a float array corresponding to the raw waveform of the speech signal. Please\n  use [`Wav2Vec2Processor`] for the feature extraction.\n- UniSpeech model can be fine-tuned using connectionist temporal classification (CTC) so the model output has to be\n  decoded using [`Wav2Vec2CTCTokenizer`].\n\n## Resources\n\n- [Audio classification task guide](../tasks/audio_classification)\n- [Automatic speech recognition task guide](../tasks/asr)\n\n## UniSpeechConfig\n\n[[autodoc]] UniSpeechConfig\n\n## UniSpeech specific outputs\n\n[[autodoc]] models.unispeech.modeling_unispeech.UniSpeechForPreTrainingOutput\n\n## UniSpeechModel\n\n[[autodoc]] UniSpeechModel\n    - forward\n\n## UniSpeechForCTC\n\n[[autodoc]] UniSpeechForCTC\n    - forward\n\n## UniSpeechForSequenceClassification\n\n[[autodoc]] UniSpeechForSequenceClassification\n    - forward\n\n## UniSpeechForPreTraining\n\n[[autodoc]] UniSpeechForPreTraining\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 7efaf734-0f2a-4c66-b031-81f24eb4d267
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/cpmant/", "title": "CPMAnt", "tokens": 457, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"7efaf734-0f2a-4c66-b031-81f24eb4d267\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cpmant/\", \"title\": \"CPMAnt\", \"tokens\": 457, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"489da474-3367-4593-8307-5e4f2006387e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cpmant/\", \"title\": \"CPMAnt\", \"tokens\": 457, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"aba1c6f3a6097d895b5810cb32be9adbe46eb0157880889d341ce7df6ad0bee1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1034, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "489da474-3367-4593-8307-5e4f2006387e", "doc_id": "489da474-3367-4593-8307-5e4f2006387e", "ref_doc_id": "489da474-3367-4593-8307-5e4f2006387e", "chroma:document": "# CPMAnt\n\n## Overview\n\nCPM-Ant is an open-source Chinese pre-trained language model (PLM) with 10B parameters. It is also the first milestone of the live training process of CPM-Live. The training process is cost-effective and environment-friendly. CPM-Ant also achieves promising results with delta tuning on the CUGE benchmark. Besides the full model, we also provide various compressed versions to meet the requirements of different hardware configurations. [See more](https://github.com/OpenBMB/CPM-Live/tree/cpm-ant/cpm-live)\n\nThis model was contributed by [OpenBMB](https://huggingface.co/openbmb). The original code can be found [here](https://github.com/OpenBMB/CPM-Live/tree/cpm-ant/cpm-live).\n\n## Resources\n\n- A tutorial on [CPM-Live](https://github.com/OpenBMB/CPM-Live/tree/cpm-ant/cpm-live).\n\n## CpmAntConfig\n\n[[autodoc]] CpmAntConfig\n    - all\n\n## CpmAntTokenizer\n\n[[autodoc]] CpmAntTokenizer\n    - all\n\n## CpmAntModel\n\n[[autodoc]] CpmAntModel\n    - all\n    \n## CpmAntForCausalLM\n\n[[autodoc]] CpmAntForCausalLM\n    - all"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 2d29d3fa-4059-40c4-bae4-65008f516223
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/speech-encoder-decoder/", "title": "Speech Encoder Decoder Models", "tokens": 1588, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"2d29d3fa-4059-40c4-bae4-65008f516223\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/speech-encoder-decoder/\", \"title\": \"Speech Encoder Decoder Models\", \"tokens\": 1588, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8f6b8aee-981b-438c-88f1-6239d4a7d4c0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/speech-encoder-decoder/\", \"title\": \"Speech Encoder Decoder Models\", \"tokens\": 1588, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"c331fc107012a59df6a5039d6e8028796765bd95d750678ed3da52f227260336\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7b09cff6-5f5c-4989-a8e2-e2376ddd21b1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"806ebcc676b29644c1f0c8f225715545b12e0fa28b80648c10c9618f90dd60ab\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3294, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8f6b8aee-981b-438c-88f1-6239d4a7d4c0", "doc_id": "8f6b8aee-981b-438c-88f1-6239d4a7d4c0", "ref_doc_id": "8f6b8aee-981b-438c-88f1-6239d4a7d4c0", "chroma:document": "# Speech Encoder Decoder Models\n\nThe [`SpeechEncoderDecoderModel`] can be used to initialize a speech-to-text model\nwith any pretrained speech autoencoding model as the encoder (*e.g.* [Wav2Vec2](wav2vec2), [Hubert](hubert)) and any pretrained autoregressive model as the decoder.\n\nThe effectiveness of initializing speech-sequence-to-text-sequence models with pretrained checkpoints for speech\nrecognition and speech translation has *e.g.* been shown in [Large-Scale Self- and Semi-Supervised Learning for Speech\nTranslation](https://arxiv.org/abs/2104.06678) by Changhan Wang, Anne Wu, Juan Pino, Alexei Baevski, Michael Auli,\nAlexis Conneau.\n\nAn example of how to use a [`SpeechEncoderDecoderModel`] for inference can be seen in [Speech2Text2](speech_to_text_2).\n\n## Randomly initializing `SpeechEncoderDecoderModel` from model configurations.\n\n[`SpeechEncoderDecoderModel`] can be randomly initialized from an encoder and a decoder config. In the following example, we show how to do this using the default [`Wav2Vec2Model`] configuration for the encoder\nand the default [`BertForCausalLM`] configuration for the decoder.\n\n```python\n>>> from transformers import BertConfig, Wav2Vec2Config, SpeechEncoderDecoderConfig, SpeechEncoderDecoderModel\n\n>>> config_encoder = Wav2Vec2Config()\n>>> config_decoder = BertConfig()\n\n>>> config = SpeechEncoderDecoderConfig.from_encoder_decoder_configs(config_encoder, config_decoder)\n>>> model = SpeechEncoderDecoderModel(config=config)\n```\n\n## Initialising `SpeechEncoderDecoderModel` from a pretrained encoder and a pretrained decoder.\n\n[`SpeechEncoderDecoderModel`] can be initialized from a pretrained encoder checkpoint and a pretrained decoder checkpoint. Note that any pretrained Transformer-based speech model, *e.g.* [Wav2Vec2](wav2vec2), [Hubert](hubert) can serve as the encoder and both pretrained auto-encoding models, *e.g.* BERT, pretrained causal language models, *e.g.* GPT2, as well as the pretrained decoder part of sequence-to-sequence models, *e.g.* decoder of BART, can be used as the decoder.\nDepending on which architecture you choose as the decoder, the cross-attention layers might be randomly initialized.\nInitializing [`SpeechEncoderDecoderModel`] from a pretrained encoder and decoder checkpoint requires the model to be fine-tuned on a downstream task, as has been shown in [the *Warm-starting-encoder-decoder blog post*](https://huggingface.co/blog/warm-starting-encoder-decoder).\nTo do so, the `SpeechEncoderDecoderModel` class provides a [`SpeechEncoderDecoderModel.from_encoder_decoder_pretrained`] method.\n\n```python\n>>> from transformers import SpeechEncoderDecoderModel\n\n>>> model = SpeechEncoderDecoderModel.from_encoder_decoder_pretrained(\n...     \"facebook/hubert-large-ll60k\", \"google-bert/bert-base-uncased\"\n... )\n```\n\n## Loading an existing `SpeechEncoderDecoderModel` checkpoint and perform inference.\n\nTo load fine-tuned checkpoints of the `SpeechEncoderDecoderModel` class, [`SpeechEncoderDecoderModel`] provides the `from_pretrained(...)` method just like any other model architecture in Transformers.\n\nTo perform inference, one uses the [`generate`] method, which allows to autoregressively generate text. This method supports various forms of decoding, such as greedy, beam search and multinomial sampling."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 7b09cff6-5f5c-4989-a8e2-e2376ddd21b1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/speech-encoder-decoder/", "title": "Speech Encoder Decoder Models", "tokens": 1588, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"7b09cff6-5f5c-4989-a8e2-e2376ddd21b1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/speech-encoder-decoder/\", \"title\": \"Speech Encoder Decoder Models\", \"tokens\": 1588, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8f6b8aee-981b-438c-88f1-6239d4a7d4c0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/speech-encoder-decoder/\", \"title\": \"Speech Encoder Decoder Models\", \"tokens\": 1588, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"c331fc107012a59df6a5039d6e8028796765bd95d750678ed3da52f227260336\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2d29d3fa-4059-40c4-bae4-65008f516223\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/speech-encoder-decoder/\", \"title\": \"Speech Encoder Decoder Models\", \"tokens\": 1588, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b5086302af39aa3ce394b14377a773fd7731cb8ec48bda3bf28cfc5467c01d13\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9fa9539e-77c2-4250-9303-1d248feeaf8e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bb337b06be225d89ce8f742e4eee3192c49066638352bb63d8c29a6cc64d43ae\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1578, \"end_char_idx\": 4693, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8f6b8aee-981b-438c-88f1-6239d4a7d4c0", "doc_id": "8f6b8aee-981b-438c-88f1-6239d4a7d4c0", "ref_doc_id": "8f6b8aee-981b-438c-88f1-6239d4a7d4c0", "chroma:document": "[`SpeechEncoderDecoderModel`] can be initialized from a pretrained encoder checkpoint and a pretrained decoder checkpoint. Note that any pretrained Transformer-based speech model, *e.g.* [Wav2Vec2](wav2vec2), [Hubert](hubert) can serve as the encoder and both pretrained auto-encoding models, *e.g.* BERT, pretrained causal language models, *e.g.* GPT2, as well as the pretrained decoder part of sequence-to-sequence models, *e.g.* decoder of BART, can be used as the decoder.\nDepending on which architecture you choose as the decoder, the cross-attention layers might be randomly initialized.\nInitializing [`SpeechEncoderDecoderModel`] from a pretrained encoder and decoder checkpoint requires the model to be fine-tuned on a downstream task, as has been shown in [the *Warm-starting-encoder-decoder blog post*](https://huggingface.co/blog/warm-starting-encoder-decoder).\nTo do so, the `SpeechEncoderDecoderModel` class provides a [`SpeechEncoderDecoderModel.from_encoder_decoder_pretrained`] method.\n\n```python\n>>> from transformers import SpeechEncoderDecoderModel\n\n>>> model = SpeechEncoderDecoderModel.from_encoder_decoder_pretrained(\n...     \"facebook/hubert-large-ll60k\", \"google-bert/bert-base-uncased\"\n... )\n```\n\n## Loading an existing `SpeechEncoderDecoderModel` checkpoint and perform inference.\n\nTo load fine-tuned checkpoints of the `SpeechEncoderDecoderModel` class, [`SpeechEncoderDecoderModel`] provides the `from_pretrained(...)` method just like any other model architecture in Transformers.\n\nTo perform inference, one uses the [`generate`] method, which allows to autoregressively generate text. This method supports various forms of decoding, such as greedy, beam search and multinomial sampling.\n\n```python\n>>> from transformers import Wav2Vec2Processor, SpeechEncoderDecoderModel\n>>> from datasets import load_dataset\n>>> import torch\n\n>>> # load a fine-tuned speech translation model and corresponding processor\n>>> model = SpeechEncoderDecoderModel.from_pretrained(\"facebook/wav2vec2-xls-r-300m-en-to-15\")\n>>> processor = Wav2Vec2Processor.from_pretrained(\"facebook/wav2vec2-xls-r-300m-en-to-15\")\n\n>>> # let's perform inference on a piece of English speech (which we'll translate to German)\n>>> ds = load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\n>>> input_values = processor(ds[0][\"audio\"][\"array\"], return_tensors=\"pt\").input_values\n\n>>> # autoregressively generate transcription (uses greedy decoding by default)\n>>> generated_ids = model.generate(input_values)\n>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]\n>>> print(generated_text)\nMr. Quilter ist der Apostel der Mittelschicht und wir freuen uns, sein Evangelium willkommen hei\u00dfen zu k\u00f6nnen.\n```\n\n## Training\n\nOnce the model is created, it can be fine-tuned similar to BART, T5 or any other encoder-decoder model on a dataset of (speech, text) pairs.\nAs you can see, only 2 inputs are required for the model in order to compute a loss: `input_values` (which are the\nspeech inputs) and `labels` (which are the `input_ids` of the encoded target sequence)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 9fa9539e-77c2-4250-9303-1d248feeaf8e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/speech-encoder-decoder/", "title": "Speech Encoder Decoder Models", "tokens": 1588, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"9fa9539e-77c2-4250-9303-1d248feeaf8e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/speech-encoder-decoder/\", \"title\": \"Speech Encoder Decoder Models\", \"tokens\": 1588, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8f6b8aee-981b-438c-88f1-6239d4a7d4c0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/speech-encoder-decoder/\", \"title\": \"Speech Encoder Decoder Models\", \"tokens\": 1588, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"c331fc107012a59df6a5039d6e8028796765bd95d750678ed3da52f227260336\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7b09cff6-5f5c-4989-a8e2-e2376ddd21b1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/speech-encoder-decoder/\", \"title\": \"Speech Encoder Decoder Models\", \"tokens\": 1588, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"32dfdd10feeaf7dcb36f97827dcdf8fcdfa4efe65c875c935a3ef241df896dc7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3089, \"end_char_idx\": 6300, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8f6b8aee-981b-438c-88f1-6239d4a7d4c0", "doc_id": "8f6b8aee-981b-438c-88f1-6239d4a7d4c0", "ref_doc_id": "8f6b8aee-981b-438c-88f1-6239d4a7d4c0", "chroma:document": "To perform inference, one uses the [`generate`] method, which allows to autoregressively generate text. This method supports various forms of decoding, such as greedy, beam search and multinomial sampling.\n\n```python\n>>> from transformers import Wav2Vec2Processor, SpeechEncoderDecoderModel\n>>> from datasets import load_dataset\n>>> import torch\n\n>>> # load a fine-tuned speech translation model and corresponding processor\n>>> model = SpeechEncoderDecoderModel.from_pretrained(\"facebook/wav2vec2-xls-r-300m-en-to-15\")\n>>> processor = Wav2Vec2Processor.from_pretrained(\"facebook/wav2vec2-xls-r-300m-en-to-15\")\n\n>>> # let's perform inference on a piece of English speech (which we'll translate to German)\n>>> ds = load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\n>>> input_values = processor(ds[0][\"audio\"][\"array\"], return_tensors=\"pt\").input_values\n\n>>> # autoregressively generate transcription (uses greedy decoding by default)\n>>> generated_ids = model.generate(input_values)\n>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]\n>>> print(generated_text)\nMr. Quilter ist der Apostel der Mittelschicht und wir freuen uns, sein Evangelium willkommen hei\u00dfen zu k\u00f6nnen.\n```\n\n## Training\n\nOnce the model is created, it can be fine-tuned similar to BART, T5 or any other encoder-decoder model on a dataset of (speech, text) pairs.\nAs you can see, only 2 inputs are required for the model in order to compute a loss: `input_values` (which are the\nspeech inputs) and `labels` (which are the `input_ids` of the encoded target sequence).\n\n```python\n>>> from transformers import AutoTokenizer, AutoFeatureExtractor, SpeechEncoderDecoderModel\n>>> from datasets import load_dataset\n\n>>> encoder_id = \"facebook/wav2vec2-base-960h\"  # acoustic model encoder\n>>> decoder_id = \"google-bert/bert-base-uncased\"  # text decoder\n\n>>> feature_extractor = AutoFeatureExtractor.from_pretrained(encoder_id)\n>>> tokenizer = AutoTokenizer.from_pretrained(decoder_id)\n>>> # Combine pre-trained encoder and pre-trained decoder to form a Seq2Seq model\n>>> model = SpeechEncoderDecoderModel.from_encoder_decoder_pretrained(encoder_id, decoder_id)\n\n>>> model.config.decoder_start_token_id = tokenizer.cls_token_id\n>>> model.config.pad_token_id = tokenizer.pad_token_id\n\n>>> # load an audio input and pre-process (normalise mean/std to 0/1)\n>>> ds = load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\n>>> input_values = feature_extractor(ds[0][\"audio\"][\"array\"], return_tensors=\"pt\").input_values\n\n>>> # load its corresponding transcription and tokenize to generate labels\n>>> labels = tokenizer(ds[0][\"text\"], return_tensors=\"pt\").input_ids\n\n>>> # the forward function automatically creates the correct decoder_input_ids\n>>> loss = model(input_values=input_values, labels=labels).loss\n>>> loss.backward()\n```\n\n## SpeechEncoderDecoderConfig\n\n[[autodoc]] SpeechEncoderDecoderConfig\n\n## SpeechEncoderDecoderModel\n\n[[autodoc]] SpeechEncoderDecoderModel\n    - forward\n    - from_encoder_decoder_pretrained\n\n## FlaxSpeechEncoderDecoderModel\n\n[[autodoc]] FlaxSpeechEncoderDecoderModel\n    - __call__\n    - from_encoder_decoder_pretrained"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 133e8457-9cd2-480c-864e-9eed4b08ab65
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mms/", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"133e8457-9cd2-480c-864e-9eed4b08ab65\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mms/\", \"title\": \"MMS\", \"tokens\": 3875, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8a220030-9b9f-410d-973b-d78b25fd6a05\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mms/\", \"title\": \"MMS\", \"tokens\": 3875, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"92a8679f4ccf17656b5d94f595f12f0dd2ec07ad9f2ca58da45e5b7ee3a1a824\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a5476e9b-86c8-4464-9319-b80af39b87ac\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e529b13c45eeaa1989b74de85b2102fd684c907ea89523f926d5ad5c752c53ea\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3006, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8a220030-9b9f-410d-973b-d78b25fd6a05", "doc_id": "8a220030-9b9f-410d-973b-d78b25fd6a05", "ref_doc_id": "8a220030-9b9f-410d-973b-d78b25fd6a05", "chroma:document": "# MMS\n\n## Overview\n\nThe MMS model was proposed in [Scaling Speech Technology to 1,000+ Languages](https://arxiv.org/abs/2305.13516) \nby Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, Alexei Baevski, Yossi Adi, Xiaohui Zhang, Wei-Ning Hsu, Alexis Conneau, Michael Auli\n\nThe abstract from the paper is the following:\n\n*Expanding the language coverage of speech technology has the potential to improve access to information for many more people. \nHowever, current speech technology is restricted to about one hundred languages which is a small fraction of the over 7,000\nlanguages spoken around the world. \nThe Massively Multilingual Speech (MMS) project increases the number of supported languages by 10-40x, depending on the task. \nThe main ingredients are a new dataset based on readings of publicly available religious texts and effectively leveraging\nself-supervised learning. We built pre-trained wav2vec 2.0 models covering 1,406 languages, \na single multilingual automatic speech recognition model for 1,107 languages, speech synthesis models \nfor the same number of languages, as well as a language identification model for 4,017 languages. \nExperiments show that our multilingual speech recognition model more than halves the word error rate of \nWhisper on 54 languages of the FLEURS benchmark while being trained on a small fraction of the labeled data.*\n\nHere are the different models open sourced in the MMS project. The models and code are originally released [here](https://github.com/facebookresearch/fairseq/tree/main/examples/mms). We have add them to the `transformers` framework, making them easier to use.\n\n### Automatic Speech Recognition (ASR)\n\nThe ASR model checkpoints  can be found here : [mms-1b-fl102](https://huggingface.co/facebook/mms-1b-fl102), [mms-1b-l1107](https://huggingface.co/facebook/mms-1b-l1107), [mms-1b-all](https://huggingface.co/facebook/mms-1b-all). For best accuracy, use the `mms-1b-all` model. \n\nTips:\n\n- All ASR models accept a float array corresponding to the raw waveform of the speech signal. The raw waveform should be pre-processed with [`Wav2Vec2FeatureExtractor`].\n- The models were trained using connectionist temporal classification (CTC) so the model output has to be decoded using\n  [`Wav2Vec2CTCTokenizer`].\n- You can load different language adapter weights for different languages via [`~Wav2Vec2PreTrainedModel.load_adapter`]. Language adapters only consists of roughly 2 million parameters \n  and can therefore be efficiently loaded on the fly when needed.\n\n#### Loading\n\nBy default MMS loads adapter weights for English. If you want to load adapter weights of another language \nmake sure to specify `target_lang=<your-chosen-target-lang>` as well as `\"ignore_mismatched_sizes=True`.\nThe `ignore_mismatched_sizes=True` keyword has to be passed to allow the language model head to be resized according\nto the vocabulary of the specified language."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: a5476e9b-86c8-4464-9319-b80af39b87ac
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mms/", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"a5476e9b-86c8-4464-9319-b80af39b87ac\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mms/\", \"title\": \"MMS\", \"tokens\": 3875, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8a220030-9b9f-410d-973b-d78b25fd6a05\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mms/\", \"title\": \"MMS\", \"tokens\": 3875, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"92a8679f4ccf17656b5d94f595f12f0dd2ec07ad9f2ca58da45e5b7ee3a1a824\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"133e8457-9cd2-480c-864e-9eed4b08ab65\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mms/\", \"title\": \"MMS\", \"tokens\": 3875, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"581f5bb03065568134f1d12457fa306f769825ce1a167b52431c12ea3574c7cd\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"59807261-8b48-4aa5-bc54-b6d4c448fbb4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9e7cd13ea839e3fdfd8f76b6a01d738cf7d27a7e66fb174d8b6ef0bba58c89ec\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1459, \"end_char_idx\": 4514, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8a220030-9b9f-410d-973b-d78b25fd6a05", "doc_id": "8a220030-9b9f-410d-973b-d78b25fd6a05", "ref_doc_id": "8a220030-9b9f-410d-973b-d78b25fd6a05", "chroma:document": "*\n\nHere are the different models open sourced in the MMS project. The models and code are originally released [here](https://github.com/facebookresearch/fairseq/tree/main/examples/mms). We have add them to the `transformers` framework, making them easier to use.\n\n### Automatic Speech Recognition (ASR)\n\nThe ASR model checkpoints  can be found here : [mms-1b-fl102](https://huggingface.co/facebook/mms-1b-fl102), [mms-1b-l1107](https://huggingface.co/facebook/mms-1b-l1107), [mms-1b-all](https://huggingface.co/facebook/mms-1b-all). For best accuracy, use the `mms-1b-all` model. \n\nTips:\n\n- All ASR models accept a float array corresponding to the raw waveform of the speech signal. The raw waveform should be pre-processed with [`Wav2Vec2FeatureExtractor`].\n- The models were trained using connectionist temporal classification (CTC) so the model output has to be decoded using\n  [`Wav2Vec2CTCTokenizer`].\n- You can load different language adapter weights for different languages via [`~Wav2Vec2PreTrainedModel.load_adapter`]. Language adapters only consists of roughly 2 million parameters \n  and can therefore be efficiently loaded on the fly when needed.\n\n#### Loading\n\nBy default MMS loads adapter weights for English. If you want to load adapter weights of another language \nmake sure to specify `target_lang=<your-chosen-target-lang>` as well as `\"ignore_mismatched_sizes=True`.\nThe `ignore_mismatched_sizes=True` keyword has to be passed to allow the language model head to be resized according\nto the vocabulary of the specified language.\nSimilarly, the processor should be loaded with the same target language\n\n```py\nfrom transformers import Wav2Vec2ForCTC, AutoProcessor\n\nmodel_id = \"facebook/mms-1b-all\"\ntarget_lang = \"fra\"\n\nprocessor = AutoProcessor.from_pretrained(model_id, target_lang=target_lang)\nmodel = Wav2Vec2ForCTC.from_pretrained(model_id, target_lang=target_lang, ignore_mismatched_sizes=True)\n```\n\n<Tip>\n\nYou can safely ignore a warning such as:\n\n```text\nSome weights of Wav2Vec2ForCTC were not initialized from the model checkpoint at facebook/mms-1b-all and are newly initialized because the shapes did not match:\n- lm_head.bias: found shape torch.Size([154]) in the checkpoint and torch.Size([314]) in the model instantiated\n- lm_head.weight: found shape torch.Size([154, 1280]) in the checkpoint and torch.Size([314, 1280]) in the model instantiated\nYou should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n```\n\n</Tip>\n\nIf you want to use the ASR pipeline, you can load your chosen target language as such:\n\n```py\nfrom transformers import pipeline\n\nmodel_id = \"facebook/mms-1b-all\"\ntarget_lang = \"fra\"\n\npipe = pipeline(model=model_id, model_kwargs={\"target_lang\": \"fra\", \"ignore_mismatched_sizes\": True})\n```\n\n#### Inference\n\nNext, let's look at how we can run MMS in inference and change adapter layers after having called [`~PretrainedModel.from_pretrained`]\nFirst, we load audio data in different languages using the [Datasets](https://github.com/huggingface/datasets)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 59807261-8b48-4aa5-bc54-b6d4c448fbb4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mms/", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"59807261-8b48-4aa5-bc54-b6d4c448fbb4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mms/\", \"title\": \"MMS\", \"tokens\": 3875, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8a220030-9b9f-410d-973b-d78b25fd6a05\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mms/\", \"title\": \"MMS\", \"tokens\": 3875, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"92a8679f4ccf17656b5d94f595f12f0dd2ec07ad9f2ca58da45e5b7ee3a1a824\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a5476e9b-86c8-4464-9319-b80af39b87ac\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mms/\", \"title\": \"MMS\", \"tokens\": 3875, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"6aa92a3a06736885b526a0af67f26dea51a1e3d3a99b6eea70230beae5f33e64\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7ae13ec4-da40-486c-80fd-919592800a4f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"da14434dc71b510f4eb6af4105017014096815b855aac5313fd5fda323e50f75\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2845, \"end_char_idx\": 6000, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8a220030-9b9f-410d-973b-d78b25fd6a05", "doc_id": "8a220030-9b9f-410d-973b-d78b25fd6a05", "ref_doc_id": "8a220030-9b9f-410d-973b-d78b25fd6a05", "chroma:document": "The `ignore_mismatched_sizes=True` keyword has to be passed to allow the language model head to be resized according\nto the vocabulary of the specified language.\nSimilarly, the processor should be loaded with the same target language\n\n```py\nfrom transformers import Wav2Vec2ForCTC, AutoProcessor\n\nmodel_id = \"facebook/mms-1b-all\"\ntarget_lang = \"fra\"\n\nprocessor = AutoProcessor.from_pretrained(model_id, target_lang=target_lang)\nmodel = Wav2Vec2ForCTC.from_pretrained(model_id, target_lang=target_lang, ignore_mismatched_sizes=True)\n```\n\n<Tip>\n\nYou can safely ignore a warning such as:\n\n```text\nSome weights of Wav2Vec2ForCTC were not initialized from the model checkpoint at facebook/mms-1b-all and are newly initialized because the shapes did not match:\n- lm_head.bias: found shape torch.Size([154]) in the checkpoint and torch.Size([314]) in the model instantiated\n- lm_head.weight: found shape torch.Size([154, 1280]) in the checkpoint and torch.Size([314, 1280]) in the model instantiated\nYou should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n```\n\n</Tip>\n\nIf you want to use the ASR pipeline, you can load your chosen target language as such:\n\n```py\nfrom transformers import pipeline\n\nmodel_id = \"facebook/mms-1b-all\"\ntarget_lang = \"fra\"\n\npipe = pipeline(model=model_id, model_kwargs={\"target_lang\": \"fra\", \"ignore_mismatched_sizes\": True})\n```\n\n#### Inference\n\nNext, let's look at how we can run MMS in inference and change adapter layers after having called [`~PretrainedModel.from_pretrained`]\nFirst, we load audio data in different languages using the [Datasets](https://github.com/huggingface/datasets).\n\n```py\nfrom datasets import load_dataset, Audio\n\n# English\nstream_data = load_dataset(\"mozilla-foundation/common_voice_13_0\", \"en\", split=\"test\", streaming=True)\nstream_data = stream_data.cast_column(\"audio\", Audio(sampling_rate=16000))\nen_sample = next(iter(stream_data))[\"audio\"][\"array\"]\n\n# French\nstream_data = load_dataset(\"mozilla-foundation/common_voice_13_0\", \"fr\", split=\"test\", streaming=True)\nstream_data = stream_data.cast_column(\"audio\", Audio(sampling_rate=16000))\nfr_sample = next(iter(stream_data))[\"audio\"][\"array\"]\n```\n\nNext, we load the model and processor\n\n```py\nfrom transformers import Wav2Vec2ForCTC, AutoProcessor\nimport torch\n\nmodel_id = \"facebook/mms-1b-all\"\n\nprocessor = AutoProcessor.from_pretrained(model_id)\nmodel = Wav2Vec2ForCTC.from_pretrained(model_id)\n```\n\nNow we process the audio data, pass the processed audio data to the model and transcribe the model output,\njust like we usually do for [`Wav2Vec2ForCTC`].\n\n```py\ninputs = processor(en_sample, sampling_rate=16_000, return_tensors=\"pt\")\n\nwith torch.no_grad():\n    outputs = model(**inputs).logits\n\nids = torch.argmax(outputs, dim=-1)[0]\ntranscription = processor.decode(ids)\n# 'joe keton disapproved of films and buster also had reservations about the media'\n```\n\nWe can now keep the same model in memory and simply switch out the language adapters by\ncalling the convenient [`~Wav2Vec2ForCTC.load_adapter`] function for the model and [`~Wav2Vec2CTCTokenizer.set_target_lang`] for the tokenizer."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 7ae13ec4-da40-486c-80fd-919592800a4f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mms/", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"7ae13ec4-da40-486c-80fd-919592800a4f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mms/\", \"title\": \"MMS\", \"tokens\": 3875, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8a220030-9b9f-410d-973b-d78b25fd6a05\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mms/\", \"title\": \"MMS\", \"tokens\": 3875, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"92a8679f4ccf17656b5d94f595f12f0dd2ec07ad9f2ca58da45e5b7ee3a1a824\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"59807261-8b48-4aa5-bc54-b6d4c448fbb4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mms/\", \"title\": \"MMS\", \"tokens\": 3875, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"fd45500545fd334fd596d2adf230f1f678e97274e9949335b0bba211a064441a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6b19aaae-4447-49ab-b30a-cc9a08ef011d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c656f3b8740c8719af7639754c5581ea0baa358736b66cd62e6f39334e7a39a8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4516, \"end_char_idx\": 7556, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8a220030-9b9f-410d-973b-d78b25fd6a05", "doc_id": "8a220030-9b9f-410d-973b-d78b25fd6a05", "ref_doc_id": "8a220030-9b9f-410d-973b-d78b25fd6a05", "chroma:document": "```py\nfrom datasets import load_dataset, Audio\n\n# English\nstream_data = load_dataset(\"mozilla-foundation/common_voice_13_0\", \"en\", split=\"test\", streaming=True)\nstream_data = stream_data.cast_column(\"audio\", Audio(sampling_rate=16000))\nen_sample = next(iter(stream_data))[\"audio\"][\"array\"]\n\n# French\nstream_data = load_dataset(\"mozilla-foundation/common_voice_13_0\", \"fr\", split=\"test\", streaming=True)\nstream_data = stream_data.cast_column(\"audio\", Audio(sampling_rate=16000))\nfr_sample = next(iter(stream_data))[\"audio\"][\"array\"]\n```\n\nNext, we load the model and processor\n\n```py\nfrom transformers import Wav2Vec2ForCTC, AutoProcessor\nimport torch\n\nmodel_id = \"facebook/mms-1b-all\"\n\nprocessor = AutoProcessor.from_pretrained(model_id)\nmodel = Wav2Vec2ForCTC.from_pretrained(model_id)\n```\n\nNow we process the audio data, pass the processed audio data to the model and transcribe the model output,\njust like we usually do for [`Wav2Vec2ForCTC`].\n\n```py\ninputs = processor(en_sample, sampling_rate=16_000, return_tensors=\"pt\")\n\nwith torch.no_grad():\n    outputs = model(**inputs).logits\n\nids = torch.argmax(outputs, dim=-1)[0]\ntranscription = processor.decode(ids)\n# 'joe keton disapproved of films and buster also had reservations about the media'\n```\n\nWe can now keep the same model in memory and simply switch out the language adapters by\ncalling the convenient [`~Wav2Vec2ForCTC.load_adapter`] function for the model and [`~Wav2Vec2CTCTokenizer.set_target_lang`] for the tokenizer.\nWe pass the target language as an input - `\"fra\"` for French.\n\n```py\nprocessor.tokenizer.set_target_lang(\"fra\")\nmodel.load_adapter(\"fra\")\n\ninputs = processor(fr_sample, sampling_rate=16_000, return_tensors=\"pt\")\n\nwith torch.no_grad():\n    outputs = model(**inputs).logits\n\nids = torch.argmax(outputs, dim=-1)[0]\ntranscription = processor.decode(ids)\n# \"ce dernier est vol\u00e9 tout au long de l'histoire romaine\"\n```\n\nIn the same way the language can be switched out for all other supported languages. Please have a look at:\n\n```py\nprocessor.tokenizer.vocab.keys()\n```\n\nto see all supported languages.\n\nTo further improve performance from ASR models, language model decoding can be used. See the documentation [here](https://huggingface.co/facebook/mms-1b-all) for further details.  \n\n### Speech Synthesis (TTS)\n\nMMS-TTS uses the same model architecture as VITS, which was added to \ud83e\udd17 Transformers in v4.33. MMS trains a separate \nmodel checkpoint for each of the 1100+ languages in the project. All available checkpoints can be found on the Hugging \nFace Hub: [facebook/mms-tts](https://huggingface.co/models?sort=trending&search=facebook%2Fmms-tts), and the inference \ndocumentation under [VITS](https://huggingface.co/docs/transformers/main/en/model_doc/vits).\n\n#### Inference\n\nTo use the MMS model, first update to the latest version of the Transformers library:\n\n```bash\npip install --upgrade transformers accelerate\n```\n\nSince the flow-based model in VITS is non-deterministic, it is good practice to set a seed to ensure reproducibility of \nthe outputs."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 6b19aaae-4447-49ab-b30a-cc9a08ef011d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mms/", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"6b19aaae-4447-49ab-b30a-cc9a08ef011d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mms/\", \"title\": \"MMS\", \"tokens\": 3875, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8a220030-9b9f-410d-973b-d78b25fd6a05\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mms/\", \"title\": \"MMS\", \"tokens\": 3875, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"92a8679f4ccf17656b5d94f595f12f0dd2ec07ad9f2ca58da45e5b7ee3a1a824\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7ae13ec4-da40-486c-80fd-919592800a4f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mms/\", \"title\": \"MMS\", \"tokens\": 3875, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"20047a2f9763994ad91941bade2a64dc4fe4bb1c6e80bc18f7c329a7b2eeddfb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1ee6c7d1-0589-4631-9ed5-13eedf824c8d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7063073d6cde19b5b1b1dea12007f459b1c143072d89fe3fb1d62c640cd7e455\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6001, \"end_char_idx\": 9230, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8a220030-9b9f-410d-973b-d78b25fd6a05", "doc_id": "8a220030-9b9f-410d-973b-d78b25fd6a05", "ref_doc_id": "8a220030-9b9f-410d-973b-d78b25fd6a05", "chroma:document": "We pass the target language as an input - `\"fra\"` for French.\n\n```py\nprocessor.tokenizer.set_target_lang(\"fra\")\nmodel.load_adapter(\"fra\")\n\ninputs = processor(fr_sample, sampling_rate=16_000, return_tensors=\"pt\")\n\nwith torch.no_grad():\n    outputs = model(**inputs).logits\n\nids = torch.argmax(outputs, dim=-1)[0]\ntranscription = processor.decode(ids)\n# \"ce dernier est vol\u00e9 tout au long de l'histoire romaine\"\n```\n\nIn the same way the language can be switched out for all other supported languages. Please have a look at:\n\n```py\nprocessor.tokenizer.vocab.keys()\n```\n\nto see all supported languages.\n\nTo further improve performance from ASR models, language model decoding can be used. See the documentation [here](https://huggingface.co/facebook/mms-1b-all) for further details.  \n\n### Speech Synthesis (TTS)\n\nMMS-TTS uses the same model architecture as VITS, which was added to \ud83e\udd17 Transformers in v4.33. MMS trains a separate \nmodel checkpoint for each of the 1100+ languages in the project. All available checkpoints can be found on the Hugging \nFace Hub: [facebook/mms-tts](https://huggingface.co/models?sort=trending&search=facebook%2Fmms-tts), and the inference \ndocumentation under [VITS](https://huggingface.co/docs/transformers/main/en/model_doc/vits).\n\n#### Inference\n\nTo use the MMS model, first update to the latest version of the Transformers library:\n\n```bash\npip install --upgrade transformers accelerate\n```\n\nSince the flow-based model in VITS is non-deterministic, it is good practice to set a seed to ensure reproducibility of \nthe outputs. \n\n- For languages with a Roman alphabet, such as English or French, the tokenizer can be used directly to \npre-process the text inputs. The following code example runs a forward pass using the MMS-TTS English checkpoint:\n\n```python\nimport torch\nfrom transformers import VitsTokenizer, VitsModel, set_seed\n\ntokenizer = VitsTokenizer.from_pretrained(\"facebook/mms-tts-eng\")\nmodel = VitsModel.from_pretrained(\"facebook/mms-tts-eng\")\n\ninputs = tokenizer(text=\"Hello - my dog is cute\", return_tensors=\"pt\")\n\nset_seed(555)  # make deterministic\n\nwith torch.no_grad():\n   outputs = model(**inputs)\n\nwaveform = outputs.waveform[0]\n```\n\nThe resulting waveform can be saved as a `.wav` file:\n\n```python\nimport scipy\n\nscipy.io.wavfile.write(\"synthesized_speech.wav\", rate=model.config.sampling_rate, data=waveform)\n```\n\nOr displayed in a Jupyter Notebook / Google Colab:\n\n```python\nfrom IPython.display import Audio\n\nAudio(waveform, rate=model.config.sampling_rate)\n```\n\nFor certain languages with non-Roman alphabets, such as Arabic, Mandarin or Hindi, the [`uroman`](https://github.com/isi-nlp/uroman) \nperl package is required to pre-process the text inputs to the Roman alphabet.\n\nYou can check whether you require the `uroman` package for your language by inspecting the `is_uroman` attribute of \nthe pre-trained `tokenizer`:\n\n```python\nfrom transformers import VitsTokenizer\n\ntokenizer = VitsTokenizer.from_pretrained(\"facebook/mms-tts-eng\")\nprint(tokenizer.is_uroman)\n```\n\nIf required, you should apply the uroman package to your text inputs **prior** to passing them to the `VitsTokenizer`, \nsince currently the tokenizer does not support performing the pre-processing itself."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 1ee6c7d1-0589-4631-9ed5-13eedf824c8d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mms/", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"1ee6c7d1-0589-4631-9ed5-13eedf824c8d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mms/\", \"title\": \"MMS\", \"tokens\": 3875, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8a220030-9b9f-410d-973b-d78b25fd6a05\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mms/\", \"title\": \"MMS\", \"tokens\": 3875, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"92a8679f4ccf17656b5d94f595f12f0dd2ec07ad9f2ca58da45e5b7ee3a1a824\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6b19aaae-4447-49ab-b30a-cc9a08ef011d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mms/\", \"title\": \"MMS\", \"tokens\": 3875, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"af1e4b4c82d1613584d3ea46692ceb2758c7890693818c586d8675750a96a084\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4dd1b54e-3ec3-44ff-831f-ff511a74c06e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8036d7e7f1203f73693f30b0bca793c7d885bf407371735cbc736fd969315bc4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7559, \"end_char_idx\": 10050, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8a220030-9b9f-410d-973b-d78b25fd6a05", "doc_id": "8a220030-9b9f-410d-973b-d78b25fd6a05", "ref_doc_id": "8a220030-9b9f-410d-973b-d78b25fd6a05", "chroma:document": "- For languages with a Roman alphabet, such as English or French, the tokenizer can be used directly to \npre-process the text inputs. The following code example runs a forward pass using the MMS-TTS English checkpoint:\n\n```python\nimport torch\nfrom transformers import VitsTokenizer, VitsModel, set_seed\n\ntokenizer = VitsTokenizer.from_pretrained(\"facebook/mms-tts-eng\")\nmodel = VitsModel.from_pretrained(\"facebook/mms-tts-eng\")\n\ninputs = tokenizer(text=\"Hello - my dog is cute\", return_tensors=\"pt\")\n\nset_seed(555)  # make deterministic\n\nwith torch.no_grad():\n   outputs = model(**inputs)\n\nwaveform = outputs.waveform[0]\n```\n\nThe resulting waveform can be saved as a `.wav` file:\n\n```python\nimport scipy\n\nscipy.io.wavfile.write(\"synthesized_speech.wav\", rate=model.config.sampling_rate, data=waveform)\n```\n\nOr displayed in a Jupyter Notebook / Google Colab:\n\n```python\nfrom IPython.display import Audio\n\nAudio(waveform, rate=model.config.sampling_rate)\n```\n\nFor certain languages with non-Roman alphabets, such as Arabic, Mandarin or Hindi, the [`uroman`](https://github.com/isi-nlp/uroman) \nperl package is required to pre-process the text inputs to the Roman alphabet.\n\nYou can check whether you require the `uroman` package for your language by inspecting the `is_uroman` attribute of \nthe pre-trained `tokenizer`:\n\n```python\nfrom transformers import VitsTokenizer\n\ntokenizer = VitsTokenizer.from_pretrained(\"facebook/mms-tts-eng\")\nprint(tokenizer.is_uroman)\n```\n\nIf required, you should apply the uroman package to your text inputs **prior** to passing them to the `VitsTokenizer`, \nsince currently the tokenizer does not support performing the pre-processing itself.\n\nTo do this, first clone the uroman repository to your local machine and set the bash variable `UROMAN` to the local path:\n\n```bash\ngit clone https://github.com/isi-nlp/uroman.git\ncd uroman\nexport UROMAN=$(pwd)\n```\n\nYou can then pre-process the text input using the following code snippet. You can either rely on using the bash variable \n`UROMAN` to point to the uroman repository, or you can pass the uroman directory as an argument to the `uromaize` function:\n\n```python\nimport torch\nfrom transformers import VitsTokenizer, VitsModel, set_seed\nimport os\nimport subprocess\n\ntokenizer = VitsTokenizer.from_pretrained(\"facebook/mms-tts-kor\")\nmodel = VitsModel.from_pretrained(\"facebook/mms-tts-kor\")\n\ndef uromanize(input_string, uroman_path):\n    \"\"\"Convert non-Roman strings to Roman using the `uroman` perl package.\"\"\""}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 4dd1b54e-3ec3-44ff-831f-ff511a74c06e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mms/", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"4dd1b54e-3ec3-44ff-831f-ff511a74c06e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mms/\", \"title\": \"MMS\", \"tokens\": 3875, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8a220030-9b9f-410d-973b-d78b25fd6a05\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mms/\", \"title\": \"MMS\", \"tokens\": 3875, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"92a8679f4ccf17656b5d94f595f12f0dd2ec07ad9f2ca58da45e5b7ee3a1a824\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1ee6c7d1-0589-4631-9ed5-13eedf824c8d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mms/\", \"title\": \"MMS\", \"tokens\": 3875, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a9e02ff5f00680a84c4d069338e00d9896e69359c26af683448419d884270cfe\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"156e30d8-ddfb-4b74-8a22-aab976f64093\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9b1ca43ee45fbe8d3a455ed8092172510bd5ef0a143cc283f8ab41d73eedb337\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8731, \"end_char_idx\": 11466, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8a220030-9b9f-410d-973b-d78b25fd6a05", "doc_id": "8a220030-9b9f-410d-973b-d78b25fd6a05", "ref_doc_id": "8a220030-9b9f-410d-973b-d78b25fd6a05", "chroma:document": "You can check whether you require the `uroman` package for your language by inspecting the `is_uroman` attribute of \nthe pre-trained `tokenizer`:\n\n```python\nfrom transformers import VitsTokenizer\n\ntokenizer = VitsTokenizer.from_pretrained(\"facebook/mms-tts-eng\")\nprint(tokenizer.is_uroman)\n```\n\nIf required, you should apply the uroman package to your text inputs **prior** to passing them to the `VitsTokenizer`, \nsince currently the tokenizer does not support performing the pre-processing itself.\n\nTo do this, first clone the uroman repository to your local machine and set the bash variable `UROMAN` to the local path:\n\n```bash\ngit clone https://github.com/isi-nlp/uroman.git\ncd uroman\nexport UROMAN=$(pwd)\n```\n\nYou can then pre-process the text input using the following code snippet. You can either rely on using the bash variable \n`UROMAN` to point to the uroman repository, or you can pass the uroman directory as an argument to the `uromaize` function:\n\n```python\nimport torch\nfrom transformers import VitsTokenizer, VitsModel, set_seed\nimport os\nimport subprocess\n\ntokenizer = VitsTokenizer.from_pretrained(\"facebook/mms-tts-kor\")\nmodel = VitsModel.from_pretrained(\"facebook/mms-tts-kor\")\n\ndef uromanize(input_string, uroman_path):\n    \"\"\"Convert non-Roman strings to Roman using the `uroman` perl package.\"\"\"\n    script_path = os.path.join(uroman_path, \"bin\", \"uroman.pl\")\n\n    command = [\"perl\", script_path]\n\n    process = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)\n    # Execute the perl command\n    stdout, stderr = process.communicate(input=input_string.encode())\n\n    if process.returncode != 0:\n        raise ValueError(f\"Error {process.returncode}: {stderr.decode()}\")\n\n    # Return the output as a string and skip the new-line character at the end\n    return stdout.decode()[:-1]\n\ntext = \"\uc774\ubd10 \ubb34\uc2a8 \uc77c\uc774\uc57c\"\nuromaized_text = uromanize(text, uroman_path=os.environ[\"UROMAN\"])\n\ninputs = tokenizer(text=uromaized_text, return_tensors=\"pt\")\n\nset_seed(555)  # make deterministic\nwith torch.no_grad():\n   outputs = model(inputs[\"input_ids\"])\n\nwaveform = outputs.waveform[0]\n```\n\n**Tips:**\n\n* The MMS-TTS checkpoints are trained on lower-cased, un-punctuated text. By default, the `VitsTokenizer` *normalizes* the inputs by removing any casing and punctuation, to avoid passing out-of-vocabulary characters to the model. Hence, the model is agnostic to casing and punctuation, so these should be avoided in the text prompt. You can disable normalisation by setting `normalize=False` in the call to the tokenizer, but this will lead to un-expected behaviour and is discouraged.\n* The speaking rate can be varied by setting the attribute `model.speaking_rate` to a chosen value."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 156e30d8-ddfb-4b74-8a22-aab976f64093
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mms/", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"156e30d8-ddfb-4b74-8a22-aab976f64093\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mms/\", \"title\": \"MMS\", \"tokens\": 3875, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8a220030-9b9f-410d-973b-d78b25fd6a05\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mms/\", \"title\": \"MMS\", \"tokens\": 3875, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"92a8679f4ccf17656b5d94f595f12f0dd2ec07ad9f2ca58da45e5b7ee3a1a824\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4dd1b54e-3ec3-44ff-831f-ff511a74c06e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mms/\", \"title\": \"MMS\", \"tokens\": 3875, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"78925a7b5eaeec283f1d4570b709146b7d584fec37178ea0f9c8b5ae31e083ea\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ccffd737-7279-4825-a912-8ba58df06cec\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b9a6cbc717434a4c650dcbda78e7ba5dc949295bd46e1dec5c919e0d4d879015\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10055, \"end_char_idx\": 12726, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8a220030-9b9f-410d-973b-d78b25fd6a05", "doc_id": "8a220030-9b9f-410d-973b-d78b25fd6a05", "ref_doc_id": "8a220030-9b9f-410d-973b-d78b25fd6a05", "chroma:document": "script_path = os.path.join(uroman_path, \"bin\", \"uroman.pl\")\n\n    command = [\"perl\", script_path]\n\n    process = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)\n    # Execute the perl command\n    stdout, stderr = process.communicate(input=input_string.encode())\n\n    if process.returncode != 0:\n        raise ValueError(f\"Error {process.returncode}: {stderr.decode()}\")\n\n    # Return the output as a string and skip the new-line character at the end\n    return stdout.decode()[:-1]\n\ntext = \"\uc774\ubd10 \ubb34\uc2a8 \uc77c\uc774\uc57c\"\nuromaized_text = uromanize(text, uroman_path=os.environ[\"UROMAN\"])\n\ninputs = tokenizer(text=uromaized_text, return_tensors=\"pt\")\n\nset_seed(555)  # make deterministic\nwith torch.no_grad():\n   outputs = model(inputs[\"input_ids\"])\n\nwaveform = outputs.waveform[0]\n```\n\n**Tips:**\n\n* The MMS-TTS checkpoints are trained on lower-cased, un-punctuated text. By default, the `VitsTokenizer` *normalizes* the inputs by removing any casing and punctuation, to avoid passing out-of-vocabulary characters to the model. Hence, the model is agnostic to casing and punctuation, so these should be avoided in the text prompt. You can disable normalisation by setting `normalize=False` in the call to the tokenizer, but this will lead to un-expected behaviour and is discouraged.\n* The speaking rate can be varied by setting the attribute `model.speaking_rate` to a chosen value. Likewise, the randomness of the noise is controlled by `model.noise_scale`:\n\n```python\nimport torch\nfrom transformers import VitsTokenizer, VitsModel, set_seed\n\ntokenizer = VitsTokenizer.from_pretrained(\"facebook/mms-tts-eng\")\nmodel = VitsModel.from_pretrained(\"facebook/mms-tts-eng\")\n\ninputs = tokenizer(text=\"Hello - my dog is cute\", return_tensors=\"pt\")\n\n# make deterministic\nset_seed(555)  \n\n# make speech faster and more noisy\nmodel.speaking_rate = 1.5\nmodel.noise_scale = 0.8\n\nwith torch.no_grad():\n   outputs = model(**inputs)\n```\n\n### Language Identification (LID)\n\nDifferent LID models are available based on the number of languages they can recognize - [126](https://huggingface.co/facebook/mms-lid-126), [256](https://huggingface.co/facebook/mms-lid-256), [512](https://huggingface.co/facebook/mms-lid-512), [1024](https://huggingface.co/facebook/mms-lid-1024), [2048](https://huggingface.co/facebook/mms-lid-2048), [4017](https://huggingface.co/facebook/mms-lid-4017). \n\n#### Inference\nFirst, we install transformers and some other libraries\n\n```bash\npip install torch accelerate datasets[audio]\npip install --upgrade transformers\n````\n\nNext, we load a couple of audio samples via `datasets`. Make sure that the audio data is sampled to 16000 kHz."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: ccffd737-7279-4825-a912-8ba58df06cec
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mms/", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"ccffd737-7279-4825-a912-8ba58df06cec\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mms/\", \"title\": \"MMS\", \"tokens\": 3875, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8a220030-9b9f-410d-973b-d78b25fd6a05\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mms/\", \"title\": \"MMS\", \"tokens\": 3875, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"92a8679f4ccf17656b5d94f595f12f0dd2ec07ad9f2ca58da45e5b7ee3a1a824\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"156e30d8-ddfb-4b74-8a22-aab976f64093\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mms/\", \"title\": \"MMS\", \"tokens\": 3875, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"2e4929e8c055d5778669dc1a3236f805d366bb9647047ff462845feb4d96f2be\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"88dced06-7076-4e73-b4ee-e45c55f099d5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9ac2dad7ed6dc6356ab78625514b54c459e219990aa148fcd2d0dfd0e0352dd4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11214, \"end_char_idx\": 14720, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8a220030-9b9f-410d-973b-d78b25fd6a05", "doc_id": "8a220030-9b9f-410d-973b-d78b25fd6a05", "ref_doc_id": "8a220030-9b9f-410d-973b-d78b25fd6a05", "chroma:document": "You can disable normalisation by setting `normalize=False` in the call to the tokenizer, but this will lead to un-expected behaviour and is discouraged.\n* The speaking rate can be varied by setting the attribute `model.speaking_rate` to a chosen value. Likewise, the randomness of the noise is controlled by `model.noise_scale`:\n\n```python\nimport torch\nfrom transformers import VitsTokenizer, VitsModel, set_seed\n\ntokenizer = VitsTokenizer.from_pretrained(\"facebook/mms-tts-eng\")\nmodel = VitsModel.from_pretrained(\"facebook/mms-tts-eng\")\n\ninputs = tokenizer(text=\"Hello - my dog is cute\", return_tensors=\"pt\")\n\n# make deterministic\nset_seed(555)  \n\n# make speech faster and more noisy\nmodel.speaking_rate = 1.5\nmodel.noise_scale = 0.8\n\nwith torch.no_grad():\n   outputs = model(**inputs)\n```\n\n### Language Identification (LID)\n\nDifferent LID models are available based on the number of languages they can recognize - [126](https://huggingface.co/facebook/mms-lid-126), [256](https://huggingface.co/facebook/mms-lid-256), [512](https://huggingface.co/facebook/mms-lid-512), [1024](https://huggingface.co/facebook/mms-lid-1024), [2048](https://huggingface.co/facebook/mms-lid-2048), [4017](https://huggingface.co/facebook/mms-lid-4017). \n\n#### Inference\nFirst, we install transformers and some other libraries\n\n```bash\npip install torch accelerate datasets[audio]\npip install --upgrade transformers\n````\n\nNext, we load a couple of audio samples via `datasets`. Make sure that the audio data is sampled to 16000 kHz.\n\n```py\nfrom datasets import load_dataset, Audio\n\n# English\nstream_data = load_dataset(\"mozilla-foundation/common_voice_13_0\", \"en\", split=\"test\", streaming=True)\nstream_data = stream_data.cast_column(\"audio\", Audio(sampling_rate=16000))\nen_sample = next(iter(stream_data))[\"audio\"][\"array\"]\n\n# Arabic\nstream_data = load_dataset(\"mozilla-foundation/common_voice_13_0\", \"ar\", split=\"test\", streaming=True)\nstream_data = stream_data.cast_column(\"audio\", Audio(sampling_rate=16000))\nar_sample = next(iter(stream_data))[\"audio\"][\"array\"]\n```\n\nNext, we load the model and processor\n\n```py\nfrom transformers import Wav2Vec2ForSequenceClassification, AutoFeatureExtractor\nimport torch\n\nmodel_id = \"facebook/mms-lid-126\"\n\nprocessor = AutoFeatureExtractor.from_pretrained(model_id)\nmodel = Wav2Vec2ForSequenceClassification.from_pretrained(model_id)\n```\n\nNow we process the audio data, pass the processed audio data to the model to classify it into a language, just like we usually do for Wav2Vec2 audio classification models such as [ehcalabres/wav2vec2-lg-xlsr-en-speech-emotion-recognition](https://huggingface.co/harshit345/xlsr-wav2vec-speech-emotion-recognition)\n\n```py\n# English\ninputs = processor(en_sample, sampling_rate=16_000, return_tensors=\"pt\")\n\nwith torch.no_grad():\n    outputs = model(**inputs).logits\n\nlang_id = torch.argmax(outputs, dim=-1)[0].item()\ndetected_lang = model.config.id2label[lang_id]\n# 'eng'\n\n# Arabic\ninputs = processor(ar_sample, sampling_rate=16_000, return_tensors=\"pt\")\n\nwith torch.no_grad():\n    outputs = model(**inputs).logits\n\nlang_id = torch.argmax(outputs, dim=-1)[0].item()\ndetected_lang = model.config.id2label[lang_id]\n# 'ara'\n```\n\nTo see all the supported languages of a checkpoint, you can print out the language ids as follows:\n```py\nprocessor.id2label.values()\n```\n\n### Audio Pretrained Models\n\nPretrained models are available for two different sizes - [300M](https://huggingface.co/facebook/mms-300m) , \n[1Bil](https://huggingface.co/facebook/mms-1b)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 88dced06-7076-4e73-b4ee-e45c55f099d5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mms/", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"88dced06-7076-4e73-b4ee-e45c55f099d5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mms/\", \"title\": \"MMS\", \"tokens\": 3875, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8a220030-9b9f-410d-973b-d78b25fd6a05\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mms/\", \"title\": \"MMS\", \"tokens\": 3875, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"92a8679f4ccf17656b5d94f595f12f0dd2ec07ad9f2ca58da45e5b7ee3a1a824\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ccffd737-7279-4825-a912-8ba58df06cec\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mms/\", \"title\": \"MMS\", \"tokens\": 3875, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b04f15bcb5fe82870355c928b87e67b5ebd74187cc4b5488ed6464cf5712d9f0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14723, \"end_char_idx\": 15043, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8a220030-9b9f-410d-973b-d78b25fd6a05", "doc_id": "8a220030-9b9f-410d-973b-d78b25fd6a05", "ref_doc_id": "8a220030-9b9f-410d-973b-d78b25fd6a05", "chroma:document": "<Tip>\n\nThe MMS for ASR architecture is based on the Wav2Vec2 model, refer to [Wav2Vec2's documentation page](wav2vec2) for further \ndetails on how to finetune with models for various downstream tasks.\n\nMMS-TTS uses the same model architecture as VITS, refer to [VITS's documentation page](vits) for API reference.\n</Tip>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 3b7a5f68-9ff8-46ce-a050-975f52ee9f66
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bort/", "title": "BORT", "tokens": 785, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"3b7a5f68-9ff8-46ce-a050-975f52ee9f66\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bort/\", \"title\": \"BORT\", \"tokens\": 785, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"033b5cd9-f496-4ccb-a113-327ea2f325d0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bort/\", \"title\": \"BORT\", \"tokens\": 785, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"37f64a50cec0530cb3eef51794c7fedcb9f18ccbc25a837040a3250c8069dd38\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2507, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "033b5cd9-f496-4ccb-a113-327ea2f325d0", "doc_id": "033b5cd9-f496-4ccb-a113-327ea2f325d0", "ref_doc_id": "033b5cd9-f496-4ccb-a113-327ea2f325d0", "chroma:document": "# BORT\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we do not accept any new PRs changing its code.\n\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.30.0.\nYou can do so by running the following command: `pip install -U transformers==4.30.0`.\n\n</Tip>\n\n## Overview\n\nThe BORT model was proposed in [Optimal Subarchitecture Extraction for BERT](https://arxiv.org/abs/2010.10499) by\nAdrian de Wynter and Daniel J. Perry. It is an optimal subset of architectural parameters for the BERT, which the\nauthors refer to as \"Bort\".\n\nThe abstract from the paper is the following:\n\n*We extract an optimal subset of architectural parameters for the BERT architecture from Devlin et al. (2018) by\napplying recent breakthroughs in algorithms for neural architecture search. This optimal subset, which we refer to as\n\"Bort\", is demonstrably smaller, having an effective (that is, not counting the embedding layer) size of 5.5% the\noriginal BERT-large architecture, and 16% of the net size. Bort is also able to be pretrained in 288 GPU hours, which\nis 1.2% of the time required to pretrain the highest-performing BERT parametric architectural variant, RoBERTa-large\n(Liu et al., 2019), and about 33% of that of the world-record, in GPU hours, required to train BERT-large on the same\nhardware. It is also 7.9x faster on a CPU, as well as being better performing than other compressed variants of the\narchitecture, and some of the non-compressed variants: it obtains performance improvements of between 0.3% and 31%,\nabsolute, with respect to BERT-large, on multiple public natural language understanding (NLU) benchmarks.*\n\nThis model was contributed by [stefan-it](https://huggingface.co/stefan-it). The original code can be found [here](https://github.com/alexa/bort/).\n\n## Usage tips\n\n- BORT's model architecture is based on BERT, refer to [BERT's documentation page](bert) for the\n  model's API reference as well as usage examples.\n- BORT uses the RoBERTa tokenizer instead of the BERT tokenizer, refer to [RoBERTa's documentation page](roberta) for the tokenizer's API reference as well as usage examples.\n- BORT requires a specific fine-tuning algorithm, called [Agora](https://adewynter.github.io/notes/bort_algorithms_and_applications.html#fine-tuning-with-algebraic-topology) ,\n  that is sadly not open-sourced yet. It would be very useful for the community, if someone tries to implement the\n  algorithm to make BORT fine-tuning work."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 45ded241-f9c7-430c-8cb0-e88baa7e7dd1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/squeezebert/", "title": "SqueezeBERT", "tokens": 1032, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"45ded241-f9c7-430c-8cb0-e88baa7e7dd1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/squeezebert/\", \"title\": \"SqueezeBERT\", \"tokens\": 1032, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"36747e96-2865-44c1-bd24-5bf550e508b2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/squeezebert/\", \"title\": \"SqueezeBERT\", \"tokens\": 1032, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"910ee345e98c6d06394fecd7f98ba7c7e94843120c2a408f19d8e51e9d129b8b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ab79c29b-c1ef-4956-a48a-f9f94c605f12\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d4f18e94a5fd48abc78fa694c50cb084e2c9d6758db135cb3c69da09634a1eb2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2625, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "36747e96-2865-44c1-bd24-5bf550e508b2", "doc_id": "36747e96-2865-44c1-bd24-5bf550e508b2", "ref_doc_id": "36747e96-2865-44c1-bd24-5bf550e508b2", "chroma:document": "# SqueezeBERT\n\n## Overview\n\nThe SqueezeBERT model was proposed in [SqueezeBERT: What can computer vision teach NLP about efficient neural networks?](https://arxiv.org/abs/2006.11316) by Forrest N. Iandola, Albert E. Shaw, Ravi Krishna, Kurt W. Keutzer. It's a\nbidirectional transformer similar to the BERT model. The key difference between the BERT architecture and the\nSqueezeBERT architecture is that SqueezeBERT uses [grouped convolutions](https://blog.yani.io/filter-group-tutorial)\ninstead of fully-connected layers for the Q, K, V and FFN layers.\n\nThe abstract from the paper is the following:\n\n*Humans read and write hundreds of billions of messages every day. Further, due to the availability of large datasets,\nlarge computing systems, and better neural network models, natural language processing (NLP) technology has made\nsignificant strides in understanding, proofreading, and organizing these messages. Thus, there is a significant\nopportunity to deploy NLP in myriad applications to help web users, social networks, and businesses. In particular, we\nconsider smartphones and other mobile devices as crucial platforms for deploying NLP models at scale. However, today's\nhighly-accurate NLP neural network models such as BERT and RoBERTa are extremely computationally expensive, with\nBERT-base taking 1.7 seconds to classify a text snippet on a Pixel 3 smartphone. In this work, we observe that methods\nsuch as grouped convolutions have yielded significant speedups for computer vision networks, but many of these\ntechniques have not been adopted by NLP neural network designers. We demonstrate how to replace several operations in\nself-attention layers with grouped convolutions, and we use this technique in a novel network architecture called\nSqueezeBERT, which runs 4.3x faster than BERT-base on the Pixel 3 while achieving competitive accuracy on the GLUE test\nset. The SqueezeBERT code will be released.*\n\nThis model was contributed by [forresti](https://huggingface.co/forresti).\n\n## Usage tips\n\n- SqueezeBERT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right\n  rather than the left.\n- SqueezeBERT is similar to BERT and therefore relies on the masked language modeling (MLM) objective. It is therefore\n  efficient at predicting masked tokens and at NLU in general, but is not optimal for text generation. Models trained\n  with a causal language modeling (CLM) objective are better in that regard.\n- For best results when finetuning on sequence classification tasks, it is recommended to start with the\n  *squeezebert/squeezebert-mnli-headless* checkpoint."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: ab79c29b-c1ef-4956-a48a-f9f94c605f12
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/squeezebert/", "title": "SqueezeBERT", "tokens": 1032, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"ab79c29b-c1ef-4956-a48a-f9f94c605f12\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/squeezebert/\", \"title\": \"SqueezeBERT\", \"tokens\": 1032, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"36747e96-2865-44c1-bd24-5bf550e508b2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/squeezebert/\", \"title\": \"SqueezeBERT\", \"tokens\": 1032, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"910ee345e98c6d06394fecd7f98ba7c7e94843120c2a408f19d8e51e9d129b8b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"45ded241-f9c7-430c-8cb0-e88baa7e7dd1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/squeezebert/\", \"title\": \"SqueezeBERT\", \"tokens\": 1032, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"15b04842797a23330932273568f21987bd4e9a082fa572bd173483b7db62a1be\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 916, \"end_char_idx\": 3730, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "36747e96-2865-44c1-bd24-5bf550e508b2", "doc_id": "36747e96-2865-44c1-bd24-5bf550e508b2", "ref_doc_id": "36747e96-2865-44c1-bd24-5bf550e508b2", "chroma:document": "Thus, there is a significant\nopportunity to deploy NLP in myriad applications to help web users, social networks, and businesses. In particular, we\nconsider smartphones and other mobile devices as crucial platforms for deploying NLP models at scale. However, today's\nhighly-accurate NLP neural network models such as BERT and RoBERTa are extremely computationally expensive, with\nBERT-base taking 1.7 seconds to classify a text snippet on a Pixel 3 smartphone. In this work, we observe that methods\nsuch as grouped convolutions have yielded significant speedups for computer vision networks, but many of these\ntechniques have not been adopted by NLP neural network designers. We demonstrate how to replace several operations in\nself-attention layers with grouped convolutions, and we use this technique in a novel network architecture called\nSqueezeBERT, which runs 4.3x faster than BERT-base on the Pixel 3 while achieving competitive accuracy on the GLUE test\nset. The SqueezeBERT code will be released.*\n\nThis model was contributed by [forresti](https://huggingface.co/forresti).\n\n## Usage tips\n\n- SqueezeBERT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right\n  rather than the left.\n- SqueezeBERT is similar to BERT and therefore relies on the masked language modeling (MLM) objective. It is therefore\n  efficient at predicting masked tokens and at NLU in general, but is not optimal for text generation. Models trained\n  with a causal language modeling (CLM) objective are better in that regard.\n- For best results when finetuning on sequence classification tasks, it is recommended to start with the\n  *squeezebert/squeezebert-mnli-headless* checkpoint.\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## SqueezeBertConfig\n\n[[autodoc]] SqueezeBertConfig\n\n## SqueezeBertTokenizer\n\n[[autodoc]] SqueezeBertTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## SqueezeBertTokenizerFast\n\n[[autodoc]] SqueezeBertTokenizerFast\n\n## SqueezeBertModel\n\n[[autodoc]] SqueezeBertModel\n\n## SqueezeBertForMaskedLM\n\n[[autodoc]] SqueezeBertForMaskedLM\n\n## SqueezeBertForSequenceClassification\n\n[[autodoc]] SqueezeBertForSequenceClassification\n\n## SqueezeBertForMultipleChoice\n\n[[autodoc]] SqueezeBertForMultipleChoice\n\n## SqueezeBertForTokenClassification\n\n[[autodoc]] SqueezeBertForTokenClassification\n\n## SqueezeBertForQuestionAnswering\n\n[[autodoc]] SqueezeBertForQuestionAnswering"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 9bde9e7f-1324-4527-9ff7-a16626c69a6a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2_phoneme/", "title": "Wav2Vec2Phoneme", "tokens": 748, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"9bde9e7f-1324-4527-9ff7-a16626c69a6a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2_phoneme/\", \"title\": \"Wav2Vec2Phoneme\", \"tokens\": 748, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b4e29f7e-06d6-465b-8b79-5548f761365c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2_phoneme/\", \"title\": \"Wav2Vec2Phoneme\", \"tokens\": 748, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a19b7863a93e1886c335203a6b63701b7fc8a3f8a9a3add34c0e6f563e6aa54e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2315, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b4e29f7e-06d6-465b-8b79-5548f761365c", "doc_id": "b4e29f7e-06d6-465b-8b79-5548f761365c", "ref_doc_id": "b4e29f7e-06d6-465b-8b79-5548f761365c", "chroma:document": "# Wav2Vec2Phoneme\n\n## Overview\n\nThe Wav2Vec2Phoneme model was proposed in [Simple and Effective Zero-shot Cross-lingual Phoneme Recognition (Xu et al.,\n2021](https://arxiv.org/abs/2109.11680) by Qiantong Xu, Alexei Baevski, Michael Auli.\n\nThe abstract from the paper is the following:\n\n*Recent progress in self-training, self-supervised pretraining and unsupervised learning enabled well performing speech\nrecognition systems without any labeled data. However, in many cases there is labeled data available for related\nlanguages which is not utilized by these methods. This paper extends previous work on zero-shot cross-lingual transfer\nlearning by fine-tuning a multilingually pretrained wav2vec 2.0 model to transcribe unseen languages. This is done by\nmapping phonemes of the training languages to the target language using articulatory features. Experiments show that\nthis simple method significantly outperforms prior work which introduced task-specific architectures and used only part\nof a monolingually pretrained model.*\n\nRelevant checkpoints can be found under https://huggingface.co/models?other=phoneme-recognition.\n\nThis model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten)\n\nThe original code can be found [here](https://github.com/pytorch/fairseq/tree/master/fairseq/models/wav2vec).\n\n## Usage tips\n\n- Wav2Vec2Phoneme uses the exact same architecture as Wav2Vec2\n- Wav2Vec2Phoneme is a speech model that accepts a float array corresponding to the raw waveform of the speech signal.\n- Wav2Vec2Phoneme model was trained using connectionist temporal classification (CTC) so the model output has to be\n  decoded using [`Wav2Vec2PhonemeCTCTokenizer`].\n- Wav2Vec2Phoneme can be fine-tuned on multiple language at once and decode unseen languages in a single forward pass\n  to a sequence of phonemes\n- By default, the model outputs a sequence of phonemes. In order to transform the phonemes to a sequence of words one\n  should make use of a dictionary and language model.\n\n\n<Tip>\n\nWav2Vec2Phoneme's architecture is based on the Wav2Vec2 model, for API reference, check out [`Wav2Vec2`](wav2vec2)'s documentation page \nexcept for the tokenizer.\n\n</Tip>\n\n## Wav2Vec2PhonemeCTCTokenizer\n\n[[autodoc]] Wav2Vec2PhonemeCTCTokenizer\n\t- __call__\n\t- batch_decode\n\t- decode\n\t- phonemize"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 1f499dee-6aba-4a4d-8111-2417a36107d2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/big_bird/", "title": "BigBird", "tokens": 1440, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"1f499dee-6aba-4a4d-8111-2417a36107d2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/big_bird/\", \"title\": \"BigBird\", \"tokens\": 1440, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5229ba58-8ebf-40d1-b86c-63cf76d2d585\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/big_bird/\", \"title\": \"BigBird\", \"tokens\": 1440, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"7ea51d99656e61eae27ccfd2e17bb82bd4b24750d1fce9051355f0170f84754e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f13a9fb6-8347-487e-b815-9dfcd81c61f6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b616d1f7fa0cf21b7fbac73ec7fa57d7537032c3d7d6c460588e2fe5fee80246\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3017, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5229ba58-8ebf-40d1-b86c-63cf76d2d585", "doc_id": "5229ba58-8ebf-40d1-b86c-63cf76d2d585", "ref_doc_id": "5229ba58-8ebf-40d1-b86c-63cf76d2d585", "chroma:document": "# BigBird\n\n## Overview\n\nThe BigBird model was proposed in [Big Bird: Transformers for Longer Sequences](https://arxiv.org/abs/2007.14062) by\nZaheer, Manzil and Guruganesh, Guru and Dubey, Kumar Avinava and Ainslie, Joshua and Alberti, Chris and Ontanon,\nSantiago and Pham, Philip and Ravula, Anirudh and Wang, Qifan and Yang, Li and others. BigBird, is a sparse-attention\nbased transformer which extends Transformer based models, such as BERT to much longer sequences. In addition to sparse\nattention, BigBird also applies global attention as well as random attention to the input sequence. Theoretically, it\nhas been shown that applying sparse, global, and random attention approximates full attention, while being\ncomputationally much more efficient for longer sequences. As a consequence of the capability to handle longer context,\nBigBird has shown improved performance on various long document NLP tasks, such as question answering and\nsummarization, compared to BERT or RoBERTa.\n\nThe abstract from the paper is the following:\n\n*Transformers-based models, such as BERT, have been one of the most successful deep learning models for NLP.\nUnfortunately, one of their core limitations is the quadratic dependency (mainly in terms of memory) on the sequence\nlength due to their full attention mechanism. To remedy this, we propose, BigBird, a sparse attention mechanism that\nreduces this quadratic dependency to linear. We show that BigBird is a universal approximator of sequence functions and\nis Turing complete, thereby preserving these properties of the quadratic, full attention model. Along the way, our\ntheoretical analysis reveals some of the benefits of having O(1) global tokens (such as CLS), that attend to the entire\nsequence as part of the sparse attention mechanism. The proposed sparse attention can handle sequences of length up to\n8x of what was previously possible using similar hardware. As a consequence of the capability to handle longer context,\nBigBird drastically improves performance on various NLP tasks such as question answering and summarization. We also\npropose novel applications to genomics data.*\n\nThis model was contributed by [vasudevgupta](https://huggingface.co/vasudevgupta). The original code can be found\n[here](https://github.com/google-research/bigbird).\n\n## Usage tips\n\n- For an in-detail explanation on how BigBird's attention works, see [this blog post](https://huggingface.co/blog/big-bird).\n- BigBird comes with 2 implementations: **original_full** & **block_sparse**. For the sequence length < 1024, using\n  **original_full** is advised as there is no benefit in using **block_sparse** attention.\n- The code currently uses window size of 3 blocks and 2 global blocks.\n- Sequence length must be divisible by block size.\n- Current implementation supports only **ITC**.\n- Current implementation doesn't support **num_random_blocks = 0**\n- BigBird is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than\n  the left."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: f13a9fb6-8347-487e-b815-9dfcd81c61f6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/big_bird/", "title": "BigBird", "tokens": 1440, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"f13a9fb6-8347-487e-b815-9dfcd81c61f6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/big_bird/\", \"title\": \"BigBird\", \"tokens\": 1440, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5229ba58-8ebf-40d1-b86c-63cf76d2d585\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/big_bird/\", \"title\": \"BigBird\", \"tokens\": 1440, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"7ea51d99656e61eae27ccfd2e17bb82bd4b24750d1fce9051355f0170f84754e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1f499dee-6aba-4a4d-8111-2417a36107d2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/big_bird/\", \"title\": \"BigBird\", \"tokens\": 1440, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"d4072a5d665d75d24e0c75d372714843bb88d52966e77d614c53b255082bd3f0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3020, \"end_char_idx\": 5215, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5229ba58-8ebf-40d1-b86c-63cf76d2d585", "doc_id": "5229ba58-8ebf-40d1-b86c-63cf76d2d585", "ref_doc_id": "5229ba58-8ebf-40d1-b86c-63cf76d2d585", "chroma:document": "## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## BigBirdConfig\n\n[[autodoc]] BigBirdConfig\n\n## BigBirdTokenizer\n\n[[autodoc]] BigBirdTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## BigBirdTokenizerFast\n\n[[autodoc]] BigBirdTokenizerFast\n\n## BigBird specific outputs\n\n[[autodoc]] models.big_bird.modeling_big_bird.BigBirdForPreTrainingOutput\n\n<frameworkcontent>\n<pt>\n\n## BigBirdModel\n\n[[autodoc]] BigBirdModel\n    - forward\n\n## BigBirdForPreTraining\n\n[[autodoc]] BigBirdForPreTraining\n    - forward\n\n## BigBirdForCausalLM\n\n[[autodoc]] BigBirdForCausalLM\n    - forward\n\n## BigBirdForMaskedLM\n\n[[autodoc]] BigBirdForMaskedLM\n    - forward\n\n## BigBirdForSequenceClassification\n\n[[autodoc]] BigBirdForSequenceClassification\n    - forward\n\n## BigBirdForMultipleChoice\n\n[[autodoc]] BigBirdForMultipleChoice\n    - forward\n\n## BigBirdForTokenClassification\n\n[[autodoc]] BigBirdForTokenClassification\n    - forward\n\n## BigBirdForQuestionAnswering\n\n[[autodoc]] BigBirdForQuestionAnswering\n    - forward\n\n</pt>\n<jax>\n\n## FlaxBigBirdModel\n\n[[autodoc]] FlaxBigBirdModel\n    - __call__\n\n## FlaxBigBirdForPreTraining\n\n[[autodoc]] FlaxBigBirdForPreTraining\n    - __call__\n\n## FlaxBigBirdForCausalLM\n\n[[autodoc]] FlaxBigBirdForCausalLM\n    - __call__\n\n## FlaxBigBirdForMaskedLM\n\n[[autodoc]] FlaxBigBirdForMaskedLM\n    - __call__\n\n## FlaxBigBirdForSequenceClassification\n\n[[autodoc]] FlaxBigBirdForSequenceClassification\n    - __call__\n\n## FlaxBigBirdForMultipleChoice\n\n[[autodoc]] FlaxBigBirdForMultipleChoice\n    - __call__\n\n## FlaxBigBirdForTokenClassification\n\n[[autodoc]] FlaxBigBirdForTokenClassification\n    - __call__\n\n## FlaxBigBirdForQuestionAnswering\n\n[[autodoc]] FlaxBigBirdForQuestionAnswering\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: be9facc6-3f26-4b31-87a7-94360bed6a8a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mixtral/", "title": "Mixtral", "tokens": 2917, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"be9facc6-3f26-4b31-87a7-94360bed6a8a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mixtral/\", \"title\": \"Mixtral\", \"tokens\": 2917, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d78e4927-3627-44c1-a2b6-974170c15b8f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mixtral/\", \"title\": \"Mixtral\", \"tokens\": 2917, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"731148c4922aae35ef1f94d656dac38ca59e0bd09610e1e8e358b60d8a3c9f88\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3673855d-f3cd-40ca-9d5f-f779320bb311\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"dcea0642d00f5f4cf469a1f4c002db384b3fa7033bde79248802b1856a7ca7fe\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2872, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d78e4927-3627-44c1-a2b6-974170c15b8f", "doc_id": "d78e4927-3627-44c1-a2b6-974170c15b8f", "ref_doc_id": "d78e4927-3627-44c1-a2b6-974170c15b8f", "chroma:document": "# Mixtral\n\n## Overview\n\nMixtral-8x7B was introduced in the [Mixtral of Experts blogpost](https://mistral.ai/news/mixtral-of-experts/) by Albert Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, L\u00e9lio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timoth\u00e9e Lacroix, William El Sayed.\n\nThe introduction of the blog post says:\n\n*Today, the team is proud to release Mixtral 8x7B, a high-quality sparse mixture of experts models (SMoE) with open weights. Licensed under Apache 2.0. Mixtral outperforms Llama 2 70B on most benchmarks with 6x faster inference. It is the strongest open-weight model with a permissive license and the best model overall regarding cost/performance trade-offs. In particular, it matches or outperforms GPT3.5 on most standard benchmarks.*\n\nMixtral-8x7B is the second large language model (LLM) released by [mistral.ai](https://mistral.ai/), after [Mistral-7B](mistral).\n\n### Architectural details\n\nMixtral-8x7B is a decoder-only Transformer with the following architectural choices:\n\n- Mixtral is a Mixture of Experts (MoE) model with 8 experts per MLP, with a total of 45 billion parameters. To learn more about mixture-of-experts, refer to the [blog post](https://huggingface.co/blog/moe).\n- Despite the model having 45 billion parameters,, the compute required for a single forward pass is the same as that of a 14 billion parameter model. This is because even though each of the experts have to be loaded in RAM (70B like ram requirement) each token from the hidden states are dispatched twice (top 2 routing) and thus the compute (the operation required at each forward computation) is just 2 X sequence_length. \n\nThe following implementation details are shared with Mistral AI's first model [Mistral-7B](mistral):\n- Sliding Window Attention - Trained with 8k context length and fixed cache size, with a theoretical attention span of 128K tokens\n- GQA (Grouped Query Attention) - allowing faster inference and lower cache size.\n- Byte-fallback BPE tokenizer - ensures that characters are never mapped to out of vocabulary tokens.\n\nFor more details refer to the [release blog post](https://mistral.ai/news/mixtral-of-experts/).\n\n### License\n\n`Mixtral-8x7B` is released under the Apache 2.0 license.\n\n## Usage tips\n\nThe Mistral team has released 2 checkpoints:\n- a base model, [Mixtral-8x7B-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1), which has been pre-trained to predict the next token on internet-scale data.\n- an instruction tuned model, [Mixtral-8x7B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1), which is the base model optimized for chat purposes using supervised fine-tuning (SFT) and direct preference optimization (DPO)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 3673855d-f3cd-40ca-9d5f-f779320bb311
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mixtral/", "title": "Mixtral", "tokens": 2917, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"3673855d-f3cd-40ca-9d5f-f779320bb311\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mixtral/\", \"title\": \"Mixtral\", \"tokens\": 2917, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d78e4927-3627-44c1-a2b6-974170c15b8f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mixtral/\", \"title\": \"Mixtral\", \"tokens\": 2917, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"731148c4922aae35ef1f94d656dac38ca59e0bd09610e1e8e358b60d8a3c9f88\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"be9facc6-3f26-4b31-87a7-94360bed6a8a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mixtral/\", \"title\": \"Mixtral\", \"tokens\": 2917, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"e18983bfe2eeb41d21e54896b10b3d6fcc0db5bbd05fac12750c2da1cf55df73\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e0340ef4-7555-4b67-adbe-fe17de833025\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"48a11e7c30c0c76b6e660ba8dbbb71198448ada8df9f4158ed1bd0b81f483762\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1385, \"end_char_idx\": 4158, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d78e4927-3627-44c1-a2b6-974170c15b8f", "doc_id": "d78e4927-3627-44c1-a2b6-974170c15b8f", "ref_doc_id": "d78e4927-3627-44c1-a2b6-974170c15b8f", "chroma:document": "- Despite the model having 45 billion parameters,, the compute required for a single forward pass is the same as that of a 14 billion parameter model. This is because even though each of the experts have to be loaded in RAM (70B like ram requirement) each token from the hidden states are dispatched twice (top 2 routing) and thus the compute (the operation required at each forward computation) is just 2 X sequence_length. \n\nThe following implementation details are shared with Mistral AI's first model [Mistral-7B](mistral):\n- Sliding Window Attention - Trained with 8k context length and fixed cache size, with a theoretical attention span of 128K tokens\n- GQA (Grouped Query Attention) - allowing faster inference and lower cache size.\n- Byte-fallback BPE tokenizer - ensures that characters are never mapped to out of vocabulary tokens.\n\nFor more details refer to the [release blog post](https://mistral.ai/news/mixtral-of-experts/).\n\n### License\n\n`Mixtral-8x7B` is released under the Apache 2.0 license.\n\n## Usage tips\n\nThe Mistral team has released 2 checkpoints:\n- a base model, [Mixtral-8x7B-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1), which has been pre-trained to predict the next token on internet-scale data.\n- an instruction tuned model, [Mixtral-8x7B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1), which is the base model optimized for chat purposes using supervised fine-tuning (SFT) and direct preference optimization (DPO).\n\nThe base model can be used as follows:\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"mistralai/Mixtral-8x7B-v0.1\", device_map=\"auto\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"mistralai/Mixtral-8x7B-v0.1\")\n\n>>> prompt = \"My favourite condiment is\"\n\n>>> model_inputs = tokenizer([prompt], return_tensors=\"pt\").to(\"cuda\")\n>>> model.to(device)\n\n>>> generated_ids = model.generate(**model_inputs, max_new_tokens=100, do_sample=True)\n>>> tokenizer.batch_decode(generated_ids)[0]\n\"My favourite condiment is to ...\"\n```\n\nThe instruction tuned model can be used as follows:\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"mistralai/Mixtral-8x7B-Instruct-v0.1\", device_map=\"auto\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"mistralai/Mixtral-8x7B-Instruct-v0.1\")\n\n>>> messages = [\n...     {\"role\": \"user\", \"content\": \"What is your favourite condiment?\"},\n...     {\"role\": \"assistant\", \"content\": \"Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!\"},\n...     {\"role\": \"user\", \"content\": \"Do you have mayonnaise recipes?\"}"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: e0340ef4-7555-4b67-adbe-fe17de833025
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mixtral/", "title": "Mixtral", "tokens": 2917, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"e0340ef4-7555-4b67-adbe-fe17de833025\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mixtral/\", \"title\": \"Mixtral\", \"tokens\": 2917, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d78e4927-3627-44c1-a2b6-974170c15b8f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mixtral/\", \"title\": \"Mixtral\", \"tokens\": 2917, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"731148c4922aae35ef1f94d656dac38ca59e0bd09610e1e8e358b60d8a3c9f88\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3673855d-f3cd-40ca-9d5f-f779320bb311\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mixtral/\", \"title\": \"Mixtral\", \"tokens\": 2917, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"2738d77c060fc07785d8eb74a475c7886a6ab9938baf0e257d2cfb356dc2ad6c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f28142b1-e954-473f-8dea-aa08eba17884\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b571ec0696e6d767c15269ed0be8d0fe530c73eef9db80d77e0745d5fa70c7bf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2874, \"end_char_idx\": 5384, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d78e4927-3627-44c1-a2b6-974170c15b8f", "doc_id": "d78e4927-3627-44c1-a2b6-974170c15b8f", "ref_doc_id": "d78e4927-3627-44c1-a2b6-974170c15b8f", "chroma:document": "The base model can be used as follows:\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"mistralai/Mixtral-8x7B-v0.1\", device_map=\"auto\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"mistralai/Mixtral-8x7B-v0.1\")\n\n>>> prompt = \"My favourite condiment is\"\n\n>>> model_inputs = tokenizer([prompt], return_tensors=\"pt\").to(\"cuda\")\n>>> model.to(device)\n\n>>> generated_ids = model.generate(**model_inputs, max_new_tokens=100, do_sample=True)\n>>> tokenizer.batch_decode(generated_ids)[0]\n\"My favourite condiment is to ...\"\n```\n\nThe instruction tuned model can be used as follows:\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"mistralai/Mixtral-8x7B-Instruct-v0.1\", device_map=\"auto\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"mistralai/Mixtral-8x7B-Instruct-v0.1\")\n\n>>> messages = [\n...     {\"role\": \"user\", \"content\": \"What is your favourite condiment?\"},\n...     {\"role\": \"assistant\", \"content\": \"Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!\"},\n...     {\"role\": \"user\", \"content\": \"Do you have mayonnaise recipes?\"}\n... ]\n\n>>> model_inputs = tokenizer.apply_chat_template(messages, return_tensors=\"pt\").to(\"cuda\")\n\n>>> generated_ids = model.generate(model_inputs, max_new_tokens=100, do_sample=True)\n>>> tokenizer.batch_decode(generated_ids)[0]\n\"Mayonnaise can be made as follows: (...)\"\n```\n\nAs can be seen, the instruction-tuned model requires a [chat template](../chat_templating) to be applied to make sure the inputs are prepared in the right format.\n\n## Speeding up Mixtral by using Flash Attention\n\nThe code snippets above showcase inference without any optimization tricks. However, one can drastically speed up the model by leveraging [Flash Attention](../perf_train_gpu_one.md#flash-attention-2), which is a faster implementation of the attention mechanism used inside the model.\n\nFirst, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature.\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nMake also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of the [flash attention repository](https://github.com/Dao-AILab/flash-attention). Make also sure to load your model in half-precision (e.g."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: f28142b1-e954-473f-8dea-aa08eba17884
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mixtral/", "title": "Mixtral", "tokens": 2917, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"f28142b1-e954-473f-8dea-aa08eba17884\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mixtral/\", \"title\": \"Mixtral\", \"tokens\": 2917, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d78e4927-3627-44c1-a2b6-974170c15b8f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mixtral/\", \"title\": \"Mixtral\", \"tokens\": 2917, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"731148c4922aae35ef1f94d656dac38ca59e0bd09610e1e8e358b60d8a3c9f88\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e0340ef4-7555-4b67-adbe-fe17de833025\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mixtral/\", \"title\": \"Mixtral\", \"tokens\": 2917, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"627f7c569bbf48f6dbb7a0c1c961a5dd7c07711c6a253a6d4ca1460b48b944b1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"58dbaaec-acf8-4430-9132-d386d18f811a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0a88a5c1f38288a1bc35d318b2b7f978f6ab611e6623e8a5008504232ae6ce9d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3885, \"end_char_idx\": 7200, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d78e4927-3627-44c1-a2b6-974170c15b8f", "doc_id": "d78e4927-3627-44c1-a2b6-974170c15b8f", "ref_doc_id": "d78e4927-3627-44c1-a2b6-974170c15b8f", "chroma:document": "\"},\n...     {\"role\": \"assistant\", \"content\": \"Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!\"},\n...     {\"role\": \"user\", \"content\": \"Do you have mayonnaise recipes?\"}\n... ]\n\n>>> model_inputs = tokenizer.apply_chat_template(messages, return_tensors=\"pt\").to(\"cuda\")\n\n>>> generated_ids = model.generate(model_inputs, max_new_tokens=100, do_sample=True)\n>>> tokenizer.batch_decode(generated_ids)[0]\n\"Mayonnaise can be made as follows: (...)\"\n```\n\nAs can be seen, the instruction-tuned model requires a [chat template](../chat_templating) to be applied to make sure the inputs are prepared in the right format.\n\n## Speeding up Mixtral by using Flash Attention\n\nThe code snippets above showcase inference without any optimization tricks. However, one can drastically speed up the model by leveraging [Flash Attention](../perf_train_gpu_one.md#flash-attention-2), which is a faster implementation of the attention mechanism used inside the model.\n\nFirst, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature.\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nMake also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of the [flash attention repository](https://github.com/Dao-AILab/flash-attention). Make also sure to load your model in half-precision (e.g. `torch.float16`)\n\nTo load and run a model using Flash Attention-2, refer to the snippet below:\n\n```python\n>>> import torch\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"mistralai/Mixtral-8x7B-v0.1\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\", device_map=\"auto\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"mistralai/Mixtral-8x7B-v0.1\")\n\n>>> prompt = \"My favourite condiment is\"\n\n>>> model_inputs = tokenizer([prompt], return_tensors=\"pt\").to(\"cuda\")\n>>> model.to(device)\n\n>>> generated_ids = model.generate(**model_inputs, max_new_tokens=100, do_sample=True)\n>>> tokenizer.batch_decode(generated_ids)[0]\n\"The expected output\"\n```\n\n### Expected speedups\n\nBelow is a expected speedup diagram that compares pure inference time between the native implementation in transformers using `mistralai/Mixtral-8x7B-v0.1` checkpoint and the Flash Attention 2 version of the model.\n\n<div style=\"text-align: center\">\n<img src=\"https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/mixtral-7b-inference-large-seqlen.png\">\n</div>\n\n### Sliding window Attention\n\nThe current implementation supports the sliding window attention mechanism and memory efficient cache management. \nTo enable sliding window attention, just make sure to have a `flash-attn` version that is compatible with sliding window attention (`>=2.3.0`). \n\nThe Flash Attention-2 model uses also a more memory efficient cache slicing mechanism - as recommended per the official implementation of Mistral model that use rolling cache mechanism we keep the cache size fixed (`self.config.sliding_window`), support batched generation only for `padding_side=\"left\"` and use the absolute position of the current token to compute the positional embedding."}


