# embeddings_queue

created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: b7efe0e6-6f64-4db5-ae21-76218801cf38
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vits", "title": "VITS", "tokens": 1725, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b7efe0e6-6f64-4db5-ae21-76218801cf38\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vits\", \"title\": \"VITS\", \"tokens\": 1725, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c087beb7-a151-4bb0-bdec-f6c6d3daf2db\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vits\", \"title\": \"VITS\", \"tokens\": 1725, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"18c5ace19d8381498c51c2737ea594a3c52d8a5427063c2c044cc4a7896cf4ed\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a3ecd987-bce7-4ee2-82c5-5ef9872f3295\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vits\", \"title\": \"VITS\", \"tokens\": 1725, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c5f9fcb682d21ecf3915464905be5f66ea5e5791434c3ee6f2d6c6e3653a9725\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"170473ee-10c0-4fb3-be72-4d6818b346da\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"806fd3a612d618bd38917f134a7783c2a7c12e1cf5f3266c9d5763c5b1382419\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1695, \"end_char_idx\": 5110, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c087beb7-a151-4bb0-bdec-f6c6d3daf2db", "doc_id": "c087beb7-a151-4bb0-bdec-f6c6d3daf2db", "ref_doc_id": "c087beb7-a151-4bb0-bdec-f6c6d3daf2db", "chroma:document": "The abstract from the paper is the following:\n\n*Several recent end-to-end text-to-speech (TTS) models enabling single-stage training and parallel sampling have been proposed, but their sample quality does not match that of two-stage TTS systems. In this work, we present a parallel end-to-end TTS method that generates more natural sounding audio than current two-stage models. Our method adopts variational inference augmented with normalizing flows and an adversarial training process, which improves the expressive power of generative modeling. We also propose a stochastic duration predictor to synthesize speech with diverse rhythms from input text. With the uncertainty modeling over latent variables and the stochastic duration predictor, our method expresses the natural one-to-many relationship in which a text input can be spoken in multiple ways with different pitches and rhythms. A subjective human evaluation (mean opinion score, or MOS) on the LJ Speech, a single speaker dataset, shows that our method outperforms the best publicly available TTS systems and achieves a MOS comparable to ground truth.*\n\nThis model can also be used with TTS checkpoints from [Massively Multilingual Speech (MMS)](https://arxiv.org/abs/2305.13516) \nas these checkpoints use the same architecture and a slightly modified tokenizer.\n\nThis model was contributed by [Matthijs](https://huggingface.co/Matthijs) and [sanchit-gandhi](https://huggingface.co/sanchit-gandhi). The original code can be found [here](https://github.com/jaywalnut310/vits).\n\n## Usage examples\n\nBoth the VITS and MMS-TTS checkpoints can be used with the same API. Since the flow-based model is non-deterministic, it \nis good practice to set a seed to ensure reproducibility of the outputs. For languages with a Roman alphabet, \nsuch as English or French, the tokenizer can be used directly to pre-process the text inputs. The following code example \nruns a forward pass using the MMS-TTS English checkpoint:\n\n```python\nimport torch\nfrom transformers import VitsTokenizer, VitsModel, set_seed\n\ntokenizer = VitsTokenizer.from_pretrained(\"facebook/mms-tts-eng\")\nmodel = VitsModel.from_pretrained(\"facebook/mms-tts-eng\")\n\ninputs = tokenizer(text=\"Hello - my dog is cute\", return_tensors=\"pt\")\n\nset_seed(555)  # make deterministic\n\nwith torch.no_grad():\n   outputs = model(**inputs)\n\nwaveform = outputs.waveform[0]\n```\n\nThe resulting waveform can be saved as a `.wav` file:\n\n```python\nimport scipy\n\nscipy.io.wavfile.write(\"techno.wav\", rate=model.config.sampling_rate, data=waveform)\n```\n\nOr displayed in a Jupyter Notebook / Google Colab:\n\n```python\nfrom IPython.display import Audio\n\nAudio(waveform, rate=model.config.sampling_rate)\n```\n\nFor certain languages with a non-Roman alphabet, such as Arabic, Mandarin or Hindi, the [`uroman`](https://github.com/isi-nlp/uroman) \nperl package is required to pre-process the text inputs to the Roman alphabet.\n\nYou can check whether you require the `uroman` package for your language by inspecting the `is_uroman` attribute of \nthe pre-trained `tokenizer`:\n\n```python\nfrom transformers import VitsTokenizer\n\ntokenizer = VitsTokenizer.from_pretrained(\"facebook/mms-tts-eng\")\nprint(tokenizer.is_uroman)\n```\n\nIf required, you should apply the uroman package to your text inputs **prior** to passing them to the `VitsTokenizer`, \nsince currently the tokenizer does not support performing the pre-processing itself."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 170473ee-10c0-4fb3-be72-4d6818b346da
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vits", "title": "VITS", "tokens": 1725, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"170473ee-10c0-4fb3-be72-4d6818b346da\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vits\", \"title\": \"VITS\", \"tokens\": 1725, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c087beb7-a151-4bb0-bdec-f6c6d3daf2db\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vits\", \"title\": \"VITS\", \"tokens\": 1725, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"18c5ace19d8381498c51c2737ea594a3c52d8a5427063c2c044cc4a7896cf4ed\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b7efe0e6-6f64-4db5-ae21-76218801cf38\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vits\", \"title\": \"VITS\", \"tokens\": 1725, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3a98310dbf03d94be6673653a29cad31a923a4fcf64bb1a6e35a8eb439c7cb19\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"be1a443e-f691-449f-ba03-e264471a33e5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4a0ee5faa814964a709021f6886f6730fc5b98eeda57191c5c5f665dd4da7375\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3451, \"end_char_idx\": 5932, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c087beb7-a151-4bb0-bdec-f6c6d3daf2db", "doc_id": "c087beb7-a151-4bb0-bdec-f6c6d3daf2db", "ref_doc_id": "c087beb7-a151-4bb0-bdec-f6c6d3daf2db", "chroma:document": "For languages with a Roman alphabet, \nsuch as English or French, the tokenizer can be used directly to pre-process the text inputs. The following code example \nruns a forward pass using the MMS-TTS English checkpoint:\n\n```python\nimport torch\nfrom transformers import VitsTokenizer, VitsModel, set_seed\n\ntokenizer = VitsTokenizer.from_pretrained(\"facebook/mms-tts-eng\")\nmodel = VitsModel.from_pretrained(\"facebook/mms-tts-eng\")\n\ninputs = tokenizer(text=\"Hello - my dog is cute\", return_tensors=\"pt\")\n\nset_seed(555)  # make deterministic\n\nwith torch.no_grad():\n   outputs = model(**inputs)\n\nwaveform = outputs.waveform[0]\n```\n\nThe resulting waveform can be saved as a `.wav` file:\n\n```python\nimport scipy\n\nscipy.io.wavfile.write(\"techno.wav\", rate=model.config.sampling_rate, data=waveform)\n```\n\nOr displayed in a Jupyter Notebook / Google Colab:\n\n```python\nfrom IPython.display import Audio\n\nAudio(waveform, rate=model.config.sampling_rate)\n```\n\nFor certain languages with a non-Roman alphabet, such as Arabic, Mandarin or Hindi, the [`uroman`](https://github.com/isi-nlp/uroman) \nperl package is required to pre-process the text inputs to the Roman alphabet.\n\nYou can check whether you require the `uroman` package for your language by inspecting the `is_uroman` attribute of \nthe pre-trained `tokenizer`:\n\n```python\nfrom transformers import VitsTokenizer\n\ntokenizer = VitsTokenizer.from_pretrained(\"facebook/mms-tts-eng\")\nprint(tokenizer.is_uroman)\n```\n\nIf required, you should apply the uroman package to your text inputs **prior** to passing them to the `VitsTokenizer`, \nsince currently the tokenizer does not support performing the pre-processing itself.  \n\nTo do this, first clone the uroman repository to your local machine and set the bash variable `UROMAN` to the local path:\n\n```bash\ngit clone https://github.com/isi-nlp/uroman.git\ncd uroman\nexport UROMAN=$(pwd)\n```\n\nYou can then pre-process the text input using the following code snippet. You can either rely on using the bash variable \n`UROMAN` to point to the uroman repository, or you can pass the uroman directory as an argument to the `uromaize` function:\n\n```python\nimport torch\nfrom transformers import VitsTokenizer, VitsModel, set_seed\nimport os\nimport subprocess\n\ntokenizer = VitsTokenizer.from_pretrained(\"facebook/mms-tts-kor\")\nmodel = VitsModel.from_pretrained(\"facebook/mms-tts-kor\")\n\ndef uromanize(input_string, uroman_path):\n    \"\"\"Convert non-Roman strings to Roman using the `uroman` perl package.\"\"\""}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: be1a443e-f691-449f-ba03-e264471a33e5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vits", "title": "VITS", "tokens": 1725, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"be1a443e-f691-449f-ba03-e264471a33e5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vits\", \"title\": \"VITS\", \"tokens\": 1725, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c087beb7-a151-4bb0-bdec-f6c6d3daf2db\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vits\", \"title\": \"VITS\", \"tokens\": 1725, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"18c5ace19d8381498c51c2737ea594a3c52d8a5427063c2c044cc4a7896cf4ed\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"170473ee-10c0-4fb3-be72-4d6818b346da\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vits\", \"title\": \"VITS\", \"tokens\": 1725, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"410f05ca1a1441835687d5ee89c396d9836d8b972090693642ffc4896a186e14\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4611, \"end_char_idx\": 6922, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c087beb7-a151-4bb0-bdec-f6c6d3daf2db", "doc_id": "c087beb7-a151-4bb0-bdec-f6c6d3daf2db", "ref_doc_id": "c087beb7-a151-4bb0-bdec-f6c6d3daf2db", "chroma:document": "You can check whether you require the `uroman` package for your language by inspecting the `is_uroman` attribute of \nthe pre-trained `tokenizer`:\n\n```python\nfrom transformers import VitsTokenizer\n\ntokenizer = VitsTokenizer.from_pretrained(\"facebook/mms-tts-eng\")\nprint(tokenizer.is_uroman)\n```\n\nIf required, you should apply the uroman package to your text inputs **prior** to passing them to the `VitsTokenizer`, \nsince currently the tokenizer does not support performing the pre-processing itself.  \n\nTo do this, first clone the uroman repository to your local machine and set the bash variable `UROMAN` to the local path:\n\n```bash\ngit clone https://github.com/isi-nlp/uroman.git\ncd uroman\nexport UROMAN=$(pwd)\n```\n\nYou can then pre-process the text input using the following code snippet. You can either rely on using the bash variable \n`UROMAN` to point to the uroman repository, or you can pass the uroman directory as an argument to the `uromaize` function:\n\n```python\nimport torch\nfrom transformers import VitsTokenizer, VitsModel, set_seed\nimport os\nimport subprocess\n\ntokenizer = VitsTokenizer.from_pretrained(\"facebook/mms-tts-kor\")\nmodel = VitsModel.from_pretrained(\"facebook/mms-tts-kor\")\n\ndef uromanize(input_string, uroman_path):\n    \"\"\"Convert non-Roman strings to Roman using the `uroman` perl package.\"\"\"\n    script_path = os.path.join(uroman_path, \"bin\", \"uroman.pl\")\n\n    command = [\"perl\", script_path]\n\n    process = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)\n    # Execute the perl command\n    stdout, stderr = process.communicate(input=input_string.encode())\n\n    if process.returncode != 0:\n        raise ValueError(f\"Error {process.returncode}: {stderr.decode()}\")\n\n    # Return the output as a string and skip the new-line character at the end\n    return stdout.decode()[:-1]\n\ntext = \"\uc774\ubd10 \ubb34\uc2a8 \uc77c\uc774\uc57c\"\nuromaized_text = uromanize(text, uroman_path=os.environ[\"UROMAN\"])\n\ninputs = tokenizer(text=uromaized_text, return_tensors=\"pt\")\n\nset_seed(555)  # make deterministic\nwith torch.no_grad():\n   outputs = model(inputs[\"input_ids\"])\n\nwaveform = outputs.waveform[0]\n```\n\n## VitsConfig\n\n[[autodoc]] VitsConfig\n\n## VitsTokenizer\n\n[[autodoc]] VitsTokenizer\n    - __call__\n    - save_vocabulary\n\n## VitsModel\n\n[[autodoc]] VitsModel\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 2f7ef29d-f3b2-4d1b-ab5d-7935fac1e284
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/rag", "title": "RAG", "tokens": 1095, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2f7ef29d-f3b2-4d1b-ab5d-7935fac1e284\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rag\", \"title\": \"RAG\", \"tokens\": 1095, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4200e995-3b2a-44b1-8395-78f558834623\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rag\", \"title\": \"RAG\", \"tokens\": 1095, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"66f0f489a31b9270f6d92a744e267e2988c3f5e8881ccd9ca441948a02a4ba7b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"83dceedb-13c4-4e05-b508-8bc134013621\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8ee0b0f41e322443a262654be367d4e29f922eeefaefd2a25fa267005dd11929\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3075, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4200e995-3b2a-44b1-8395-78f558834623", "doc_id": "4200e995-3b2a-44b1-8395-78f558834623", "ref_doc_id": "4200e995-3b2a-44b1-8395-78f558834623", "chroma:document": "# RAG\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=rag\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-rag-blueviolet\">\n</a>\n</div>\n\n## Overview\n\nRetrieval-augmented generation (\"RAG\") models combine the powers of pretrained dense retrieval (DPR) and\nsequence-to-sequence models. RAG models retrieve documents, pass them to a seq2seq model, then marginalize to generate\noutputs. The retriever and seq2seq modules are initialized from pretrained models, and fine-tuned jointly, allowing\nboth retrieval and generation to adapt to downstream tasks.\n\nIt is based on the paper [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/abs/2005.11401) by Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir\nKarpukhin, Naman Goyal, Heinrich K\u00fcttler, Mike Lewis, Wen-tau Yih, Tim Rockt\u00e4schel, Sebastian Riedel, Douwe Kiela.\n\nThe abstract from the paper is the following:\n\n*Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve\nstate-of-the-art results when fine-tuned on downstream NLP tasks. However, their ability to access and precisely\nmanipulate knowledge is still limited, and hence on knowledge-intensive tasks, their performance lags behind\ntask-specific architectures. Additionally, providing provenance for their decisions and updating their world knowledge\nremain open research problems. Pre-trained models with a differentiable access mechanism to explicit nonparametric\nmemory can overcome this issue, but have so far been only investigated for extractive downstream tasks. We explore a\ngeneral-purpose fine-tuning recipe for retrieval-augmented generation (RAG) \u2014 models which combine pre-trained\nparametric and non-parametric memory for language generation. We introduce RAG models where the parametric memory is a\npre-trained seq2seq model and the non-parametric memory is a dense vector index of Wikipedia, accessed with a\npre-trained neural retriever. We compare two RAG formulations, one which conditions on the same retrieved passages\nacross the whole generated sequence, the other can use different passages per token. We fine-tune and evaluate our\nmodels on a wide range of knowledge-intensive NLP tasks and set the state-of-the-art on three open domain QA tasks,\noutperforming parametric seq2seq models and task-specific retrieve-and-extract architectures. For language generation\ntasks, we find that RAG models generate more specific, diverse and factual language than a state-of-the-art\nparametric-only seq2seq baseline.*\n\nThis model was contributed by [ola13](https://huggingface.co/ola13).\n\n## Usage tips\n\nRetrieval-augmented generation (\"RAG\") models combine the powers of pretrained dense retrieval (DPR) and Seq2Seq models. \nRAG models retrieve docs, pass them to a seq2seq model, then marginalize to generate outputs. The retriever and seq2seq \nmodules are initialized from pretrained models, and fine-tuned jointly, allowing both retrieval and generation to adapt \nto downstream tasks."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 83dceedb-13c4-4e05-b508-8bc134013621
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/rag", "title": "RAG", "tokens": 1095, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"83dceedb-13c4-4e05-b508-8bc134013621\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rag\", \"title\": \"RAG\", \"tokens\": 1095, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4200e995-3b2a-44b1-8395-78f558834623\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rag\", \"title\": \"RAG\", \"tokens\": 1095, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"66f0f489a31b9270f6d92a744e267e2988c3f5e8881ccd9ca441948a02a4ba7b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2f7ef29d-f3b2-4d1b-ab5d-7935fac1e284\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rag\", \"title\": \"RAG\", \"tokens\": 1095, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"534d5ba6ad357a6229b61dafca3abb32685881a6464d92272e2c887ca3f63e55\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1336, \"end_char_idx\": 3878, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4200e995-3b2a-44b1-8395-78f558834623", "doc_id": "4200e995-3b2a-44b1-8395-78f558834623", "ref_doc_id": "4200e995-3b2a-44b1-8395-78f558834623", "chroma:document": "Additionally, providing provenance for their decisions and updating their world knowledge\nremain open research problems. Pre-trained models with a differentiable access mechanism to explicit nonparametric\nmemory can overcome this issue, but have so far been only investigated for extractive downstream tasks. We explore a\ngeneral-purpose fine-tuning recipe for retrieval-augmented generation (RAG) \u2014 models which combine pre-trained\nparametric and non-parametric memory for language generation. We introduce RAG models where the parametric memory is a\npre-trained seq2seq model and the non-parametric memory is a dense vector index of Wikipedia, accessed with a\npre-trained neural retriever. We compare two RAG formulations, one which conditions on the same retrieved passages\nacross the whole generated sequence, the other can use different passages per token. We fine-tune and evaluate our\nmodels on a wide range of knowledge-intensive NLP tasks and set the state-of-the-art on three open domain QA tasks,\noutperforming parametric seq2seq models and task-specific retrieve-and-extract architectures. For language generation\ntasks, we find that RAG models generate more specific, diverse and factual language than a state-of-the-art\nparametric-only seq2seq baseline.*\n\nThis model was contributed by [ola13](https://huggingface.co/ola13).\n\n## Usage tips\n\nRetrieval-augmented generation (\"RAG\") models combine the powers of pretrained dense retrieval (DPR) and Seq2Seq models. \nRAG models retrieve docs, pass them to a seq2seq model, then marginalize to generate outputs. The retriever and seq2seq \nmodules are initialized from pretrained models, and fine-tuned jointly, allowing both retrieval and generation to adapt \nto downstream tasks.\n\n## RagConfig\n\n[[autodoc]] RagConfig\n\n## RagTokenizer\n\n[[autodoc]] RagTokenizer\n\n## Rag specific outputs\n\n[[autodoc]] models.rag.modeling_rag.RetrievAugLMMarginOutput\n\n[[autodoc]] models.rag.modeling_rag.RetrievAugLMOutput\n\n## RagRetriever\n\n[[autodoc]] RagRetriever\n\n<frameworkcontent>\n<pt>\n\n## RagModel\n\n[[autodoc]] RagModel\n    - forward\n\n## RagSequenceForGeneration\n\n[[autodoc]] RagSequenceForGeneration\n    - forward\n    - generate\n\n## RagTokenForGeneration\n\n[[autodoc]] RagTokenForGeneration\n    - forward\n    - generate\n\n</pt>\n<tf>\n\n## TFRagModel\n\n[[autodoc]] TFRagModel\n    - call\n\n## TFRagSequenceForGeneration\n\n[[autodoc]] TFRagSequenceForGeneration\n    - call\n    - generate\n\n## TFRagTokenForGeneration\n\n[[autodoc]] TFRagTokenForGeneration\n    - call\n    - generate\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 38e5ef3e-8baf-4153-a5a0-7783610832cc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mobilebert", "title": "MobileBERT", "tokens": 1329, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"38e5ef3e-8baf-4153-a5a0-7783610832cc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilebert\", \"title\": \"MobileBERT\", \"tokens\": 1329, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c4d9c4bf-6682-4e4c-8b1c-89334b7b08fd\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilebert\", \"title\": \"MobileBERT\", \"tokens\": 1329, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d30515b3b1df27173a8f460e680315b37a9574b5299a9d3662909a89f6dceb62\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"df2218b8-05bb-4863-b7b8-57a5253d0716\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e1fd1110ad3fe4e69c25bebca773101e3267a7868d3f71c12ac046126fb0a88c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2410, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c4d9c4bf-6682-4e4c-8b1c-89334b7b08fd", "doc_id": "c4d9c4bf-6682-4e4c-8b1c-89334b7b08fd", "ref_doc_id": "c4d9c4bf-6682-4e4c-8b1c-89334b7b08fd", "chroma:document": "# MobileBERT\n\n## Overview\n\nThe MobileBERT model was proposed in [MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://arxiv.org/abs/2004.02984) by Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny\nZhou. It's a bidirectional transformer based on the BERT model, which is compressed and accelerated using several\napproaches.\n\nThe abstract from the paper is the following:\n\n*Natural Language Processing (NLP) has recently achieved great success by using huge pre-trained models with hundreds\nof millions of parameters. However, these models suffer from heavy model sizes and high latency such that they cannot\nbe deployed to resource-limited mobile devices. In this paper, we propose MobileBERT for compressing and accelerating\nthe popular BERT model. Like the original BERT, MobileBERT is task-agnostic, that is, it can be generically applied to\nvarious downstream NLP tasks via simple fine-tuning. Basically, MobileBERT is a thin version of BERT_LARGE, while\nequipped with bottleneck structures and a carefully designed balance between self-attentions and feed-forward networks.\nTo train MobileBERT, we first train a specially designed teacher model, an inverted-bottleneck incorporated BERT_LARGE\nmodel. Then, we conduct knowledge transfer from this teacher to MobileBERT. Empirical studies show that MobileBERT is\n4.3x smaller and 5.5x faster than BERT_BASE while achieving competitive results on well-known benchmarks. On the\nnatural language inference tasks of GLUE, MobileBERT achieves a GLUEscore o 77.7 (0.6 lower than BERT_BASE), and 62 ms\nlatency on a Pixel 4 phone. On the SQuAD v1.1/v2.0 question answering task, MobileBERT achieves a dev F1 score of\n90.0/79.2 (1.5/2.1 higher than BERT_BASE).*\n\nThis model was contributed by [vshampor](https://huggingface.co/vshampor). The original code can be found [here](https://github.com/google-research/google-research/tree/master/mobilebert).\n\n## Usage tips\n\n- MobileBERT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather\n  than the left.\n- MobileBERT is similar to BERT and therefore relies on the masked language modeling (MLM) objective. It is therefore\n  efficient at predicting masked tokens and at NLU in general, but is not optimal for text generation. Models trained\n  with a causal language modeling (CLM) objective are better in that regard."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: df2218b8-05bb-4863-b7b8-57a5253d0716
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mobilebert", "title": "MobileBERT", "tokens": 1329, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"df2218b8-05bb-4863-b7b8-57a5253d0716\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilebert\", \"title\": \"MobileBERT\", \"tokens\": 1329, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c4d9c4bf-6682-4e4c-8b1c-89334b7b08fd\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilebert\", \"title\": \"MobileBERT\", \"tokens\": 1329, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d30515b3b1df27173a8f460e680315b37a9574b5299a9d3662909a89f6dceb62\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"38e5ef3e-8baf-4153-a5a0-7783610832cc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilebert\", \"title\": \"MobileBERT\", \"tokens\": 1329, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a063e637a21df6ec785bac6c22eac6c70d75cf2235bf6c0a241f7a169adc310a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2413, \"end_char_idx\": 4607, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c4d9c4bf-6682-4e4c-8b1c-89334b7b08fd", "doc_id": "c4d9c4bf-6682-4e4c-8b1c-89334b7b08fd", "ref_doc_id": "c4d9c4bf-6682-4e4c-8b1c-89334b7b08fd", "chroma:document": "## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## MobileBertConfig\n\n[[autodoc]] MobileBertConfig\n\n## MobileBertTokenizer\n\n[[autodoc]] MobileBertTokenizer\n\n## MobileBertTokenizerFast\n\n[[autodoc]] MobileBertTokenizerFast\n\n## MobileBert specific outputs\n\n[[autodoc]] models.mobilebert.modeling_mobilebert.MobileBertForPreTrainingOutput\n\n[[autodoc]] models.mobilebert.modeling_tf_mobilebert.TFMobileBertForPreTrainingOutput\n\n<frameworkcontent>\n<pt>\n\n## MobileBertModel\n\n[[autodoc]] MobileBertModel\n    - forward\n\n## MobileBertForPreTraining\n\n[[autodoc]] MobileBertForPreTraining\n    - forward\n\n## MobileBertForMaskedLM\n\n[[autodoc]] MobileBertForMaskedLM\n    - forward\n\n## MobileBertForNextSentencePrediction\n\n[[autodoc]] MobileBertForNextSentencePrediction\n    - forward\n\n## MobileBertForSequenceClassification\n\n[[autodoc]] MobileBertForSequenceClassification\n    - forward\n\n## MobileBertForMultipleChoice\n\n[[autodoc]] MobileBertForMultipleChoice\n    - forward\n\n## MobileBertForTokenClassification\n\n[[autodoc]] MobileBertForTokenClassification\n    - forward\n\n## MobileBertForQuestionAnswering\n\n[[autodoc]] MobileBertForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFMobileBertModel\n\n[[autodoc]] TFMobileBertModel\n    - call\n\n## TFMobileBertForPreTraining\n\n[[autodoc]] TFMobileBertForPreTraining\n    - call\n\n## TFMobileBertForMaskedLM\n\n[[autodoc]] TFMobileBertForMaskedLM\n    - call\n\n## TFMobileBertForNextSentencePrediction\n\n[[autodoc]] TFMobileBertForNextSentencePrediction\n    - call\n\n## TFMobileBertForSequenceClassification\n\n[[autodoc]] TFMobileBertForSequenceClassification\n    - call\n\n## TFMobileBertForMultipleChoice\n\n[[autodoc]] TFMobileBertForMultipleChoice\n    - call\n\n## TFMobileBertForTokenClassification\n\n[[autodoc]] TFMobileBertForTokenClassification\n    - call\n\n## TFMobileBertForQuestionAnswering\n\n[[autodoc]] TFMobileBertForQuestionAnswering\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: d6329e77-de7b-4885-bb0b-2795a98eef9f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/pvt_v2", "title": "Pyramid Vision Transformer V2 (PVTv2)", "tokens": 2114, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d6329e77-de7b-4885-bb0b-2795a98eef9f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pvt_v2\", \"title\": \"Pyramid Vision Transformer V2 (PVTv2)\", \"tokens\": 2114, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0e39505d-5a02-48c2-a5ad-ceaf62673ee0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pvt_v2\", \"title\": \"Pyramid Vision Transformer V2 (PVTv2)\", \"tokens\": 2114, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"07cb9bb28b2efb5f14eeda98fb8e5660f2894f9e2bef7ead8f771c91e753ad88\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8503311e-9227-4d0b-b33a-f521c027a86b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"26f5fa39308f72010911f4168a92ec9b3b44fb5a444268cb288b26b62c28445d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3435, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0e39505d-5a02-48c2-a5ad-ceaf62673ee0", "doc_id": "0e39505d-5a02-48c2-a5ad-ceaf62673ee0", "ref_doc_id": "0e39505d-5a02-48c2-a5ad-ceaf62673ee0", "chroma:document": "# Pyramid Vision Transformer V2 (PVTv2)\n\n## Overview\n\nThe PVTv2 model was proposed in\n[PVT v2: Improved Baselines with Pyramid Vision Transformer](https://arxiv.org/abs/2106.13797) by Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. As an improved variant of PVT, it eschews position embeddings, relying instead on positional information encoded through zero-padding and overlapping patch embeddings. This lack of reliance on position embeddings simplifies the architecture, and enables running inference at any resolution without needing to interpolate them.\n\nThe PVTv2 encoder structure has been successfully deployed to achieve state-of-the-art scores in [Segformer](https://arxiv.org/abs/2105.15203) for semantic segmentation, [GLPN](https://arxiv.org/abs/2201.07436) for monocular depth, and [Panoptic Segformer](https://arxiv.org/abs/2109.03814) for panoptic segmentation.\n\nPVTv2 belongs to a family of models called [hierarchical transformers](https://natecibik.medium.com/the-rise-of-vision-transformers-f623c980419f) , which make adaptations to transformer layers in order to generate multi-scale feature maps. Unlike the columnal structure of Vision Transformer ([ViT](https://arxiv.org/abs/2010.11929)) which loses fine-grained detail, multi-scale feature maps are known preserve this detail and aid performance in dense prediction tasks. In the case of PVTv2, this is achieved by generating image patch tokens using 2D convolution with overlapping kernels in each encoder layer.\n\nThe multi-scale features of hierarchical transformers allow them to be easily swapped in for traditional workhorse computer vision backbone models like ResNet in larger architectures. Both Segformer and Panoptic Segformer demonstrated that configurations using PVTv2 for a backbone consistently outperformed those with similarly sized ResNet backbones. \n\nAnother powerful feature of the PVTv2 is the complexity reduction in the self-attention layers called Spatial Reduction Attention (SRA), which uses 2D convolution layers to project hidden states to a smaller resolution before attending to them with the queries, improving the $O(n^2)$ complexity of self-attention to $O(n^2/R)$, with $R$ being the spatial reduction ratio (`sr_ratio`, aka kernel size and stride in the 2D convolution).\n\nSRA was introduced in PVT, and is the default attention complexity reduction method used in PVTv2. However, PVTv2 also introduced the option of using a self-attention mechanism with linear complexity related to image size, which they called \"Linear SRA\". This method uses average pooling to reduce the hidden states to a fixed size that is invariant to their original resolution (although this is inherently more lossy than regular SRA). This option can be enabled by setting `linear_attention` to `True` in the PVTv2Config.\n\n### Abstract from the paper:\n\n*Transformer recently has presented encouraging progress in computer vision. In this work, we present new baselines by improving the original Pyramid Vision Transformer (PVT v1) by adding three designs, including (1) linear complexity attention layer, (2) overlapping patch embedding, and (3) convolutional feed-forward network. With these modifications, PVT v2 reduces the computational complexity of PVT v1 to linear and achieves significant improvements on fundamental vision tasks such as classification, detection, and segmentation."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 8503311e-9227-4d0b-b33a-f521c027a86b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/pvt_v2", "title": "Pyramid Vision Transformer V2 (PVTv2)", "tokens": 2114, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8503311e-9227-4d0b-b33a-f521c027a86b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pvt_v2\", \"title\": \"Pyramid Vision Transformer V2 (PVTv2)\", \"tokens\": 2114, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0e39505d-5a02-48c2-a5ad-ceaf62673ee0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pvt_v2\", \"title\": \"Pyramid Vision Transformer V2 (PVTv2)\", \"tokens\": 2114, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"07cb9bb28b2efb5f14eeda98fb8e5660f2894f9e2bef7ead8f771c91e753ad88\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d6329e77-de7b-4885-bb0b-2795a98eef9f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pvt_v2\", \"title\": \"Pyramid Vision Transformer V2 (PVTv2)\", \"tokens\": 2114, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c16956542091149944d0183cc460212f3f9228083f54f59418e79e3f1435dbff\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ce600675-25cc-4d55-b587-6f16201a21f2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2ad32c3bfe2e7b8e141b1e79a96f963d361ad02e739937f2a85577344cd21f04\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1550, \"end_char_idx\": 5124, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0e39505d-5a02-48c2-a5ad-ceaf62673ee0", "doc_id": "0e39505d-5a02-48c2-a5ad-ceaf62673ee0", "ref_doc_id": "0e39505d-5a02-48c2-a5ad-ceaf62673ee0", "chroma:document": "The multi-scale features of hierarchical transformers allow them to be easily swapped in for traditional workhorse computer vision backbone models like ResNet in larger architectures. Both Segformer and Panoptic Segformer demonstrated that configurations using PVTv2 for a backbone consistently outperformed those with similarly sized ResNet backbones. \n\nAnother powerful feature of the PVTv2 is the complexity reduction in the self-attention layers called Spatial Reduction Attention (SRA), which uses 2D convolution layers to project hidden states to a smaller resolution before attending to them with the queries, improving the $O(n^2)$ complexity of self-attention to $O(n^2/R)$, with $R$ being the spatial reduction ratio (`sr_ratio`, aka kernel size and stride in the 2D convolution).\n\nSRA was introduced in PVT, and is the default attention complexity reduction method used in PVTv2. However, PVTv2 also introduced the option of using a self-attention mechanism with linear complexity related to image size, which they called \"Linear SRA\". This method uses average pooling to reduce the hidden states to a fixed size that is invariant to their original resolution (although this is inherently more lossy than regular SRA). This option can be enabled by setting `linear_attention` to `True` in the PVTv2Config.\n\n### Abstract from the paper:\n\n*Transformer recently has presented encouraging progress in computer vision. In this work, we present new baselines by improving the original Pyramid Vision Transformer (PVT v1) by adding three designs, including (1) linear complexity attention layer, (2) overlapping patch embedding, and (3) convolutional feed-forward network. With these modifications, PVT v2 reduces the computational complexity of PVT v1 to linear and achieves significant improvements on fundamental vision tasks such as classification, detection, and segmentation. Notably, the proposed PVT v2 achieves comparable or better performances than recent works such as Swin Transformer. We hope this work will facilitate state-of-the-art Transformer researches in computer vision. Code is available at https://github.com/whai362/PVT.*\n\nThis model was contributed by [FoamoftheSea](https://huggingface.co/FoamoftheSea). The original code can be found [here](https://github.com/whai362/PVT).\n\n## Usage tips\n\n- [PVTv2](https://arxiv.org/abs/2106.13797) is a hierarchical transformer model which has demonstrated powerful performance in image classification and multiple other tasks, used as a backbone for semantic segmentation in [Segformer](https://arxiv.org/abs/2105.15203), monocular depth estimation in [GLPN](https://arxiv.org/abs/2201.07436), and panoptic segmentation in [Panoptic Segformer](https://arxiv.org/abs/2109.03814), consistently showing higher performance than similar ResNet configurations.\n- Hierarchical transformers like PVTv2 achieve superior data and parameter efficiency on image data compared with pure transformer architectures by incorporating design elements of convolutional neural networks (CNNs) into their encoders. This creates a best-of-both-worlds architecture that infuses the useful inductive biases of CNNs like translation equivariance and locality into the network while still enjoying the benefits of dynamic data response and global relationship modeling provided by the self-attention mechanism of [transformers](https://arxiv.org/abs/1706.03762).\n- PVTv2 uses overlapping patch embeddings to create multi-scale feature maps, which are infused with location information using zero-padding and depth-wise convolutions."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: ce600675-25cc-4d55-b587-6f16201a21f2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/pvt_v2", "title": "Pyramid Vision Transformer V2 (PVTv2)", "tokens": 2114, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ce600675-25cc-4d55-b587-6f16201a21f2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pvt_v2\", \"title\": \"Pyramid Vision Transformer V2 (PVTv2)\", \"tokens\": 2114, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0e39505d-5a02-48c2-a5ad-ceaf62673ee0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pvt_v2\", \"title\": \"Pyramid Vision Transformer V2 (PVTv2)\", \"tokens\": 2114, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"07cb9bb28b2efb5f14eeda98fb8e5660f2894f9e2bef7ead8f771c91e753ad88\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8503311e-9227-4d0b-b33a-f521c027a86b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pvt_v2\", \"title\": \"Pyramid Vision Transformer V2 (PVTv2)\", \"tokens\": 2114, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"10e45edd72eab713eeeb4cab3a26f1fa1f089c2977e93e934b8811f854bc6e08\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"202d12e3-0fbc-40ed-9c6c-669ad32cca9e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5e500083fc5272d9381c2e888bbee8a0d9d14a1caf55e12e02a27747269df45f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3436, \"end_char_idx\": 6012, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0e39505d-5a02-48c2-a5ad-ceaf62673ee0", "doc_id": "0e39505d-5a02-48c2-a5ad-ceaf62673ee0", "ref_doc_id": "0e39505d-5a02-48c2-a5ad-ceaf62673ee0", "chroma:document": "Notably, the proposed PVT v2 achieves comparable or better performances than recent works such as Swin Transformer. We hope this work will facilitate state-of-the-art Transformer researches in computer vision. Code is available at https://github.com/whai362/PVT.*\n\nThis model was contributed by [FoamoftheSea](https://huggingface.co/FoamoftheSea). The original code can be found [here](https://github.com/whai362/PVT).\n\n## Usage tips\n\n- [PVTv2](https://arxiv.org/abs/2106.13797) is a hierarchical transformer model which has demonstrated powerful performance in image classification and multiple other tasks, used as a backbone for semantic segmentation in [Segformer](https://arxiv.org/abs/2105.15203), monocular depth estimation in [GLPN](https://arxiv.org/abs/2201.07436), and panoptic segmentation in [Panoptic Segformer](https://arxiv.org/abs/2109.03814), consistently showing higher performance than similar ResNet configurations.\n- Hierarchical transformers like PVTv2 achieve superior data and parameter efficiency on image data compared with pure transformer architectures by incorporating design elements of convolutional neural networks (CNNs) into their encoders. This creates a best-of-both-worlds architecture that infuses the useful inductive biases of CNNs like translation equivariance and locality into the network while still enjoying the benefits of dynamic data response and global relationship modeling provided by the self-attention mechanism of [transformers](https://arxiv.org/abs/1706.03762).\n- PVTv2 uses overlapping patch embeddings to create multi-scale feature maps, which are infused with location information using zero-padding and depth-wise convolutions.\n- To reduce the complexity in the attention layers, PVTv2 performs a spatial reduction on the hidden states using either strided 2D convolution (SRA) or fixed-size average pooling (Linear SRA). Although inherently more lossy, Linear SRA provides impressive performance with a linear complexity with respect to image size. To use Linear SRA in the self-attention layers, set `linear_attention=True` in the `PvtV2Config`.\n- [`PvtV2Model`] is the hierarchical transformer encoder (which is also often referred to as Mix Transformer or MiT in the literature). [`PvtV2ForImageClassification`] adds a simple classifier head on top to perform Image Classification. [`PvtV2Backbone`] can be used with the [`AutoBackbone`] system in larger architectures like Deformable DETR.\n- ImageNet pretrained weights for all model sizes can be found on the [hub](https://huggingface.co/models?other=pvt_v2)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 202d12e3-0fbc-40ed-9c6c-669ad32cca9e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/pvt_v2", "title": "Pyramid Vision Transformer V2 (PVTv2)", "tokens": 2114, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"202d12e3-0fbc-40ed-9c6c-669ad32cca9e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pvt_v2\", \"title\": \"Pyramid Vision Transformer V2 (PVTv2)\", \"tokens\": 2114, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0e39505d-5a02-48c2-a5ad-ceaf62673ee0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pvt_v2\", \"title\": \"Pyramid Vision Transformer V2 (PVTv2)\", \"tokens\": 2114, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"07cb9bb28b2efb5f14eeda98fb8e5660f2894f9e2bef7ead8f771c91e753ad88\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ce600675-25cc-4d55-b587-6f16201a21f2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pvt_v2\", \"title\": \"Pyramid Vision Transformer V2 (PVTv2)\", \"tokens\": 2114, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b99dfc8b3773d4937b2d931a773f61bc0f20075bdb8195151e99634ddb8e6378\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e1c8d33d-5188-40c8-82ab-0eb94a3dc120\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8a43301674d73bce30f4473587d937e21ad82c330b447e4e58f2f62dd70aedef\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4373, \"end_char_idx\": 8115, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0e39505d-5a02-48c2-a5ad-ceaf62673ee0", "doc_id": "0e39505d-5a02-48c2-a5ad-ceaf62673ee0", "ref_doc_id": "0e39505d-5a02-48c2-a5ad-ceaf62673ee0", "chroma:document": "- Hierarchical transformers like PVTv2 achieve superior data and parameter efficiency on image data compared with pure transformer architectures by incorporating design elements of convolutional neural networks (CNNs) into their encoders. This creates a best-of-both-worlds architecture that infuses the useful inductive biases of CNNs like translation equivariance and locality into the network while still enjoying the benefits of dynamic data response and global relationship modeling provided by the self-attention mechanism of [transformers](https://arxiv.org/abs/1706.03762).\n- PVTv2 uses overlapping patch embeddings to create multi-scale feature maps, which are infused with location information using zero-padding and depth-wise convolutions.\n- To reduce the complexity in the attention layers, PVTv2 performs a spatial reduction on the hidden states using either strided 2D convolution (SRA) or fixed-size average pooling (Linear SRA). Although inherently more lossy, Linear SRA provides impressive performance with a linear complexity with respect to image size. To use Linear SRA in the self-attention layers, set `linear_attention=True` in the `PvtV2Config`.\n- [`PvtV2Model`] is the hierarchical transformer encoder (which is also often referred to as Mix Transformer or MiT in the literature). [`PvtV2ForImageClassification`] adds a simple classifier head on top to perform Image Classification. [`PvtV2Backbone`] can be used with the [`AutoBackbone`] system in larger architectures like Deformable DETR.\n- ImageNet pretrained weights for all model sizes can be found on the [hub](https://huggingface.co/models?other=pvt_v2).\n\n The best way to get started with the PVTv2 is to load the pretrained checkpoint with the size of your choosing using `AutoModelForImageClassification`:\n```python\nimport requests\nimport torch\n\nfrom transformers import AutoModelForImageClassification, AutoImageProcessor\nfrom PIL import Image\n\nmodel = AutoModelForImageClassification.from_pretrained(\"OpenGVLab/pvt_v2_b0\")\nimage_processor = AutoImageProcessor.from_pretrained(\"OpenGVLab/pvt_v2_b0\")\nurl = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nimage = Image.open(requests.get(url, stream=True).raw)\nprocessed = image_processor(image)\noutputs = model(torch.tensor(processed[\"pixel_values\"]))\n```\n\nTo use the PVTv2 as a backbone for more complex architectures like DeformableDETR, you can use AutoBackbone (this model would need fine-tuning as you're replacing the backbone in the pretrained model):\n\n```python\nimport requests\nimport torch\n\nfrom transformers import AutoConfig, AutoModelForObjectDetection, AutoImageProcessor\nfrom PIL import Image\n\nmodel = AutoModelForObjectDetection.from_config(\n    config=AutoConfig.from_pretrained(\n        \"SenseTime/deformable-detr\",\n        backbone_config=AutoConfig.from_pretrained(\"OpenGVLab/pvt_v2_b5\"),\n        use_timm_backbone=False\n    ),\n)\n\nimage_processor = AutoImageProcessor.from_pretrained(\"SenseTime/deformable-detr\")\nurl = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nimage = Image.open(requests.get(url, stream=True).raw)\nprocessed = image_processor(image)\noutputs = model(torch.tensor(processed[\"pixel_values\"]))\n```\n\n[PVTv2](https://github.com/whai362/PVT/tree/v2) performance on ImageNet-1K by model size (B0-B5):\n\n| Method           | Size | Acc@1 | #Params (M) |\n|------------------|:----:|:-----:|:-----------:|\n| PVT-V2-B0        |  224 |  70.5 |     3.7     |\n| PVT-V2-B1        |  224 |  78.7 |     14.0    |\n| PVT-V2-B2-Linear |  224 |  82.1 |     22.6    |\n| PVT-V2-B2        |  224 |  82.0 |     25.4    |\n| PVT-V2-B3        |  224 |  83.1 |     45.2    |\n| PVT-V2-B4        |  224 |  83.6 |     62.6    |\n| PVT-V2-B5        |  224 |  83.8 |     82.0    |"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: e1c8d33d-5188-40c8-82ab-0eb94a3dc120
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/pvt_v2", "title": "Pyramid Vision Transformer V2 (PVTv2)", "tokens": 2114, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e1c8d33d-5188-40c8-82ab-0eb94a3dc120\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pvt_v2\", \"title\": \"Pyramid Vision Transformer V2 (PVTv2)\", \"tokens\": 2114, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0e39505d-5a02-48c2-a5ad-ceaf62673ee0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pvt_v2\", \"title\": \"Pyramid Vision Transformer V2 (PVTv2)\", \"tokens\": 2114, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"07cb9bb28b2efb5f14eeda98fb8e5660f2894f9e2bef7ead8f771c91e753ad88\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"202d12e3-0fbc-40ed-9c6c-669ad32cca9e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pvt_v2\", \"title\": \"Pyramid Vision Transformer V2 (PVTv2)\", \"tokens\": 2114, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9b99656f03df57fa4e38d125a226e553f3f93c30ffd0464a16bdcea889d7b301\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8118, \"end_char_idx\": 8293, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0e39505d-5a02-48c2-a5ad-ceaf62673ee0", "doc_id": "0e39505d-5a02-48c2-a5ad-ceaf62673ee0", "ref_doc_id": "0e39505d-5a02-48c2-a5ad-ceaf62673ee0", "chroma:document": "## PvtV2Config\n\n[[autodoc]] PvtV2Config\n\n## PvtForImageClassification\n\n[[autodoc]] PvtV2ForImageClassification\n    - forward\n\n## PvtModel\n\n[[autodoc]] PvtV2Model\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 349ec33e-a603-4df5-a75d-d48d8a88797a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/trocr", "title": "TrOCR", "tokens": 1704, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"349ec33e-a603-4df5-a75d-d48d8a88797a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/trocr\", \"title\": \"TrOCR\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5f0b94f7-bdd1-409a-a1ce-754d93e48f8e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/trocr\", \"title\": \"TrOCR\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7b32a4b8847a1905f6b356bb3801af7070f92f55e4c9dd38e20fe57baf8cfd09\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"906bd50e-8507-40d3-8f45-f61c928aefdb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3083a32de0faa1e89f9510ce3035b357ea3c230a6c4a61f64b3d42ffc880c1c6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3221, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5f0b94f7-bdd1-409a-a1ce-754d93e48f8e", "doc_id": "5f0b94f7-bdd1-409a-a1ce-754d93e48f8e", "ref_doc_id": "5f0b94f7-bdd1-409a-a1ce-754d93e48f8e", "chroma:document": "# TrOCR\n\n## Overview\n\nThe TrOCR model was proposed in [TrOCR: Transformer-based Optical Character Recognition with Pre-trained\nModels](https://arxiv.org/abs/2109.10282) by Minghao Li, Tengchao Lv, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang,\nZhoujun Li, Furu Wei. TrOCR consists of an image Transformer encoder and an autoregressive text Transformer decoder to\nperform [optical character recognition (OCR)](https://en.wikipedia.org/wiki/Optical_character_recognition).\n\nThe abstract from the paper is the following:\n\n*Text recognition is a long-standing research problem for document digitalization. Existing approaches for text recognition\nare usually built based on CNN for image understanding and RNN for char-level text generation. In addition, another language\nmodel is usually needed to improve the overall accuracy as a post-processing step. In this paper, we propose an end-to-end\ntext recognition approach with pre-trained image Transformer and text Transformer models, namely TrOCR, which leverages the\nTransformer architecture for both image understanding and wordpiece-level text generation. The TrOCR model is simple but\neffective, and can be pre-trained with large-scale synthetic data and fine-tuned with human-labeled datasets. Experiments\nshow that the TrOCR model outperforms the current state-of-the-art models on both printed and handwritten text recognition\ntasks.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/trocr_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> TrOCR architecture. Taken from the <a href=\"https://arxiv.org/abs/2109.10282\">original paper</a>. </small>\n\nPlease refer to the [`VisionEncoderDecoder`] class on how to use this model.\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found\n[here](https://github.com/microsoft/unilm/tree/6f60612e7cc86a2a1ae85c47231507a587ab4e01/trocr).\n\n## Usage tips\n\n- The quickest way to get started with TrOCR is by checking the [tutorial\n  notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/TrOCR), which show how to use the model\n  at inference time as well as fine-tuning on custom data.\n- TrOCR is pre-trained in 2 stages before being fine-tuned on downstream datasets. It achieves state-of-the-art results\n  on both printed (e.g. the [SROIE dataset](https://paperswithcode.com/dataset/sroie) and handwritten (e.g. the [IAM\n  Handwriting dataset](https://fki.tic.heia-fr.ch/databases/iam-handwriting-database>) text recognition tasks. For more\n  information, see the [official models](https://huggingface.co/models?other=trocr>).\n- TrOCR is always used within the [VisionEncoderDecoder](vision-encoder-decoder) framework.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with TrOCR. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- A blog post on [Accelerating Document AI](https://huggingface.co/blog/document-ai) with TrOCR."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 906bd50e-8507-40d3-8f45-f61c928aefdb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/trocr", "title": "TrOCR", "tokens": 1704, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"906bd50e-8507-40d3-8f45-f61c928aefdb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/trocr\", \"title\": \"TrOCR\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5f0b94f7-bdd1-409a-a1ce-754d93e48f8e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/trocr\", \"title\": \"TrOCR\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7b32a4b8847a1905f6b356bb3801af7070f92f55e4c9dd38e20fe57baf8cfd09\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"349ec33e-a603-4df5-a75d-d48d8a88797a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/trocr\", \"title\": \"TrOCR\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7fdf91138c5ded548f5e9a1247f45872352d7a8dce81823cd31dae612e2e57e7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"79e52d70-018f-4f14-919a-93444acd47c7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3aa8757042d4916f50e8470731435e2ba83e3964fec1c8ac69118c98cc40f045\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1726, \"end_char_idx\": 4527, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5f0b94f7-bdd1-409a-a1ce-754d93e48f8e", "doc_id": "5f0b94f7-bdd1-409a-a1ce-754d93e48f8e", "ref_doc_id": "5f0b94f7-bdd1-409a-a1ce-754d93e48f8e", "chroma:document": "This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found\n[here](https://github.com/microsoft/unilm/tree/6f60612e7cc86a2a1ae85c47231507a587ab4e01/trocr).\n\n## Usage tips\n\n- The quickest way to get started with TrOCR is by checking the [tutorial\n  notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/TrOCR), which show how to use the model\n  at inference time as well as fine-tuning on custom data.\n- TrOCR is pre-trained in 2 stages before being fine-tuned on downstream datasets. It achieves state-of-the-art results\n  on both printed (e.g. the [SROIE dataset](https://paperswithcode.com/dataset/sroie) and handwritten (e.g. the [IAM\n  Handwriting dataset](https://fki.tic.heia-fr.ch/databases/iam-handwriting-database>) text recognition tasks. For more\n  information, see the [official models](https://huggingface.co/models?other=trocr>).\n- TrOCR is always used within the [VisionEncoderDecoder](vision-encoder-decoder) framework.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with TrOCR. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- A blog post on [Accelerating Document AI](https://huggingface.co/blog/document-ai) with TrOCR.\n- A blog post on how to [Document AI](https://github.com/philschmid/document-ai-transformers) with TrOCR.\n- A notebook on how to [finetune TrOCR on IAM Handwriting Database using Seq2SeqTrainer](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/TrOCR/Fine_tune_TrOCR_on_IAM_Handwriting_Database_using_Seq2SeqTrainer.ipynb).\n- A notebook on [inference with TrOCR](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/TrOCR/Inference_with_TrOCR_%2B_Gradio_demo.ipynb) and Gradio demo.\n- A notebook on [finetune TrOCR on the IAM Handwriting Database](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/TrOCR/Fine_tune_TrOCR_on_IAM_Handwriting_Database_using_native_PyTorch.ipynb) using native PyTorch.\n- A notebook on [evaluating TrOCR on the IAM test set](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/TrOCR/Evaluating_TrOCR_base_handwritten_on_the_IAM_test_set.ipynb).\n\n<PipelineTag pipeline=\"text-generation\"/>\n\n- [Casual language modeling](https://huggingface.co/docs/transformers/tasks/language_modeling) task guide.\n\n\u26a1\ufe0f Inference\n\n- An interactive-demo on [TrOCR handwritten character recognition](https://huggingface.co/spaces/nielsr/TrOCR-handwritten)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 79e52d70-018f-4f14-919a-93444acd47c7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/trocr", "title": "TrOCR", "tokens": 1704, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"79e52d70-018f-4f14-919a-93444acd47c7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/trocr\", \"title\": \"TrOCR\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5f0b94f7-bdd1-409a-a1ce-754d93e48f8e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/trocr\", \"title\": \"TrOCR\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7b32a4b8847a1905f6b356bb3801af7070f92f55e4c9dd38e20fe57baf8cfd09\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"906bd50e-8507-40d3-8f45-f61c928aefdb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/trocr\", \"title\": \"TrOCR\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bfeabe17678bff387bfe52e3d23e80453157c3043c206e45cb46f287b196d183\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d41969da-9a9d-4396-9736-13c108a70f40\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"56b4c8050daf5f434ed000a8e2dfa18bd1e8a134ad0053ddcad8c5908ecc1e21\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3078, \"end_char_idx\": 5979, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5f0b94f7-bdd1-409a-a1ce-754d93e48f8e", "doc_id": "5f0b94f7-bdd1-409a-a1ce-754d93e48f8e", "ref_doc_id": "5f0b94f7-bdd1-409a-a1ce-754d93e48f8e", "chroma:document": "<PipelineTag pipeline=\"text-classification\"/>\n\n- A blog post on [Accelerating Document AI](https://huggingface.co/blog/document-ai) with TrOCR.\n- A blog post on how to [Document AI](https://github.com/philschmid/document-ai-transformers) with TrOCR.\n- A notebook on how to [finetune TrOCR on IAM Handwriting Database using Seq2SeqTrainer](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/TrOCR/Fine_tune_TrOCR_on_IAM_Handwriting_Database_using_Seq2SeqTrainer.ipynb).\n- A notebook on [inference with TrOCR](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/TrOCR/Inference_with_TrOCR_%2B_Gradio_demo.ipynb) and Gradio demo.\n- A notebook on [finetune TrOCR on the IAM Handwriting Database](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/TrOCR/Fine_tune_TrOCR_on_IAM_Handwriting_Database_using_native_PyTorch.ipynb) using native PyTorch.\n- A notebook on [evaluating TrOCR on the IAM test set](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/TrOCR/Evaluating_TrOCR_base_handwritten_on_the_IAM_test_set.ipynb).\n\n<PipelineTag pipeline=\"text-generation\"/>\n\n- [Casual language modeling](https://huggingface.co/docs/transformers/tasks/language_modeling) task guide.\n\n\u26a1\ufe0f Inference\n\n- An interactive-demo on [TrOCR handwritten character recognition](https://huggingface.co/spaces/nielsr/TrOCR-handwritten).\n\n## Inference\n\nTrOCR's [`VisionEncoderDecoder`] model accepts images as input and makes use of\n[`~generation.GenerationMixin.generate`] to autoregressively generate text given the input image.\n\nThe [`ViTImageProcessor`/`DeiTImageProcessor`] class is responsible for preprocessing the input image and\n[`RobertaTokenizer`/`XLMRobertaTokenizer`] decodes the generated target tokens to the target string. The\n[`TrOCRProcessor`] wraps [`ViTImageProcessor`/`DeiTImageProcessor`] and [`RobertaTokenizer`/`XLMRobertaTokenizer`]\ninto a single instance to both extract the input features and decode the predicted token ids.\n\n- Step-by-step Optical Character Recognition (OCR)\n\n``` py\n>>> from transformers import TrOCRProcessor, VisionEncoderDecoderModel\n>>> import requests\n>>> from PIL import Image\n\n>>> processor = TrOCRProcessor.from_pretrained(\"microsoft/trocr-base-handwritten\")\n>>> model = VisionEncoderDecoderModel.from_pretrained(\"microsoft/trocr-base-handwritten\")\n\n>>> # load image from the IAM dataset\n>>> url = \"https://fki.tic.heia-fr.ch/static/img/a01-122-02.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw).convert(\"RGB\")\n\n>>> pixel_values = processor(image, return_tensors=\"pt\").pixel_values\n>>> generated_ids = model.generate(pixel_values)\n\n>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]\n```\n\nSee the [model hub](https://huggingface.co/models?filter=trocr) to look for TrOCR checkpoints."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: d41969da-9a9d-4396-9736-13c108a70f40
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/trocr", "title": "TrOCR", "tokens": 1704, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d41969da-9a9d-4396-9736-13c108a70f40\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/trocr\", \"title\": \"TrOCR\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5f0b94f7-bdd1-409a-a1ce-754d93e48f8e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/trocr\", \"title\": \"TrOCR\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7b32a4b8847a1905f6b356bb3801af7070f92f55e4c9dd38e20fe57baf8cfd09\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"79e52d70-018f-4f14-919a-93444acd47c7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/trocr\", \"title\": \"TrOCR\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"020469f650433c83c362977e0d992af0b84457414f1593189c6f3c6d9416fd2d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4390, \"end_char_idx\": 6224, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5f0b94f7-bdd1-409a-a1ce-754d93e48f8e", "doc_id": "5f0b94f7-bdd1-409a-a1ce-754d93e48f8e", "ref_doc_id": "5f0b94f7-bdd1-409a-a1ce-754d93e48f8e", "chroma:document": "\u26a1\ufe0f Inference\n\n- An interactive-demo on [TrOCR handwritten character recognition](https://huggingface.co/spaces/nielsr/TrOCR-handwritten).\n\n## Inference\n\nTrOCR's [`VisionEncoderDecoder`] model accepts images as input and makes use of\n[`~generation.GenerationMixin.generate`] to autoregressively generate text given the input image.\n\nThe [`ViTImageProcessor`/`DeiTImageProcessor`] class is responsible for preprocessing the input image and\n[`RobertaTokenizer`/`XLMRobertaTokenizer`] decodes the generated target tokens to the target string. The\n[`TrOCRProcessor`] wraps [`ViTImageProcessor`/`DeiTImageProcessor`] and [`RobertaTokenizer`/`XLMRobertaTokenizer`]\ninto a single instance to both extract the input features and decode the predicted token ids.\n\n- Step-by-step Optical Character Recognition (OCR)\n\n``` py\n>>> from transformers import TrOCRProcessor, VisionEncoderDecoderModel\n>>> import requests\n>>> from PIL import Image\n\n>>> processor = TrOCRProcessor.from_pretrained(\"microsoft/trocr-base-handwritten\")\n>>> model = VisionEncoderDecoderModel.from_pretrained(\"microsoft/trocr-base-handwritten\")\n\n>>> # load image from the IAM dataset\n>>> url = \"https://fki.tic.heia-fr.ch/static/img/a01-122-02.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw).convert(\"RGB\")\n\n>>> pixel_values = processor(image, return_tensors=\"pt\").pixel_values\n>>> generated_ids = model.generate(pixel_values)\n\n>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]\n```\n\nSee the [model hub](https://huggingface.co/models?filter=trocr) to look for TrOCR checkpoints.\n\n## TrOCRConfig\n\n[[autodoc]] TrOCRConfig\n\n## TrOCRProcessor\n\n[[autodoc]] TrOCRProcessor\n    - __call__\n    - from_pretrained\n    - save_pretrained\n    - batch_decode\n    - decode\n\n## TrOCRForCausalLM\n\n[[autodoc]] TrOCRForCausalLM\n     - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: f0d3a69a-fc95-4eb9-b23d-d2989340e11d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bartpho", "title": "BARTpho", "tokens": 959, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f0d3a69a-fc95-4eb9-b23d-d2989340e11d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bartpho\", \"title\": \"BARTpho\", \"tokens\": 959, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9bc4f925-ba1c-4b6a-bbb2-f74bc65ac9b8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bartpho\", \"title\": \"BARTpho\", \"tokens\": 959, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8c44a25c732b5d6ef1588fb60f7c6dbadf4be3d9a29198625ce69077015d75ff\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0af2de4a-eb93-4bfc-aab0-739402ca0418\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"659ef2213e8b23c6080b70df8143e0c7bbcd6c6fc793770b0453b383b7db4997\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2891, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9bc4f925-ba1c-4b6a-bbb2-f74bc65ac9b8", "doc_id": "9bc4f925-ba1c-4b6a-bbb2-f74bc65ac9b8", "ref_doc_id": "9bc4f925-ba1c-4b6a-bbb2-f74bc65ac9b8", "chroma:document": "# BARTpho\n\n## Overview\n\nThe BARTpho model was proposed in [BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamese](https://arxiv.org/abs/2109.09701) by Nguyen Luong Tran, Duong Minh Le and Dat Quoc Nguyen.\n\nThe abstract from the paper is the following:\n\n*We present BARTpho with two versions -- BARTpho_word and BARTpho_syllable -- the first public large-scale monolingual\nsequence-to-sequence models pre-trained for Vietnamese. Our BARTpho uses the \"large\" architecture and pre-training\nscheme of the sequence-to-sequence denoising model BART, thus especially suitable for generative NLP tasks. Experiments\non a downstream task of Vietnamese text summarization show that in both automatic and human evaluations, our BARTpho\noutperforms the strong baseline mBART and improves the state-of-the-art. We release BARTpho to facilitate future\nresearch and applications of generative Vietnamese NLP tasks.*\n\nThis model was contributed by [dqnguyen](https://huggingface.co/dqnguyen). The original code can be found [here](https://github.com/VinAIResearch/BARTpho).\n\n## Usage example\n\n```python\n>>> import torch\n>>> from transformers import AutoModel, AutoTokenizer\n\n>>> bartpho = AutoModel.from_pretrained(\"vinai/bartpho-syllable\")\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"vinai/bartpho-syllable\")\n\n>>> line = \"Ch\u00fang t\u00f4i l\u00e0 nh\u1eefng nghi\u00ean c\u1ee9u vi\u00ean.\"\n\n>>> input_ids = tokenizer(line, return_tensors=\"pt\")\n\n>>> with torch.no_grad():\n...     features = bartpho(**input_ids)  # Models outputs are now tuples\n\n>>> # With TensorFlow 2.0+:\n>>> from transformers import TFAutoModel\n\n>>> bartpho = TFAutoModel.from_pretrained(\"vinai/bartpho-syllable\")\n>>> input_ids = tokenizer(line, return_tensors=\"tf\")\n>>> features = bartpho(**input_ids)\n```\n\n## Usage tips\n\n- Following mBART, BARTpho uses the \"large\" architecture of BART with an additional layer-normalization layer on top of\n  both the encoder and decoder. Thus, usage examples in the [documentation of BART](bart), when adapting to use\n  with BARTpho, should be adjusted by replacing the BART-specialized classes with the mBART-specialized counterparts.\n  For example:\n\n```python\n>>> from transformers import MBartForConditionalGeneration\n\n>>> bartpho = MBartForConditionalGeneration.from_pretrained(\"vinai/bartpho-syllable\")\n>>> TXT = \"Ch\u00fang t\u00f4i l\u00e0 <mask> nghi\u00ean c\u1ee9u vi\u00ean.\"\n>>> input_ids = tokenizer([TXT], return_tensors=\"pt\")[\"input_ids\"]\n>>> logits = bartpho(input_ids).logits\n>>> masked_index = (input_ids[0] == tokenizer.mask_token_id).nonzero().item()\n>>> probs = logits[0, masked_index].softmax(dim=0)\n>>> values, predictions = probs.topk(5)\n>>> print(tokenizer.decode(predictions).split())\n```\n\n- This implementation is only for tokenization: \"monolingual_vocab_file\" consists of Vietnamese-specialized types\n  extracted from the pre-trained SentencePiece model \"vocab_file\" that is available from the multilingual XLM-RoBERTa."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 0af2de4a-eb93-4bfc-aab0-739402ca0418
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bartpho", "title": "BARTpho", "tokens": 959, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0af2de4a-eb93-4bfc-aab0-739402ca0418\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bartpho\", \"title\": \"BARTpho\", \"tokens\": 959, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9bc4f925-ba1c-4b6a-bbb2-f74bc65ac9b8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bartpho\", \"title\": \"BARTpho\", \"tokens\": 959, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8c44a25c732b5d6ef1588fb60f7c6dbadf4be3d9a29198625ce69077015d75ff\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f0d3a69a-fc95-4eb9-b23d-d2989340e11d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bartpho\", \"title\": \"BARTpho\", \"tokens\": 959, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1373c9e34caffc244c05b548585112bd12ce3e81c8f70cbc2849ecd3aa1bc36f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1358, \"end_char_idx\": 3154, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9bc4f925-ba1c-4b6a-bbb2-f74bc65ac9b8", "doc_id": "9bc4f925-ba1c-4b6a-bbb2-f74bc65ac9b8", "ref_doc_id": "9bc4f925-ba1c-4b6a-bbb2-f74bc65ac9b8", "chroma:document": ">>> input_ids = tokenizer(line, return_tensors=\"pt\")\n\n>>> with torch.no_grad():\n...     features = bartpho(**input_ids)  # Models outputs are now tuples\n\n>>> # With TensorFlow 2.0+:\n>>> from transformers import TFAutoModel\n\n>>> bartpho = TFAutoModel.from_pretrained(\"vinai/bartpho-syllable\")\n>>> input_ids = tokenizer(line, return_tensors=\"tf\")\n>>> features = bartpho(**input_ids)\n```\n\n## Usage tips\n\n- Following mBART, BARTpho uses the \"large\" architecture of BART with an additional layer-normalization layer on top of\n  both the encoder and decoder. Thus, usage examples in the [documentation of BART](bart), when adapting to use\n  with BARTpho, should be adjusted by replacing the BART-specialized classes with the mBART-specialized counterparts.\n  For example:\n\n```python\n>>> from transformers import MBartForConditionalGeneration\n\n>>> bartpho = MBartForConditionalGeneration.from_pretrained(\"vinai/bartpho-syllable\")\n>>> TXT = \"Ch\u00fang t\u00f4i l\u00e0 <mask> nghi\u00ean c\u1ee9u vi\u00ean.\"\n>>> input_ids = tokenizer([TXT], return_tensors=\"pt\")[\"input_ids\"]\n>>> logits = bartpho(input_ids).logits\n>>> masked_index = (input_ids[0] == tokenizer.mask_token_id).nonzero().item()\n>>> probs = logits[0, masked_index].softmax(dim=0)\n>>> values, predictions = probs.topk(5)\n>>> print(tokenizer.decode(predictions).split())\n```\n\n- This implementation is only for tokenization: \"monolingual_vocab_file\" consists of Vietnamese-specialized types\n  extracted from the pre-trained SentencePiece model \"vocab_file\" that is available from the multilingual XLM-RoBERTa.\n  Other languages, if employing this pre-trained multilingual SentencePiece model \"vocab_file\" for subword\n  segmentation, can reuse BartphoTokenizer with their own language-specialized \"monolingual_vocab_file\".\n\n## BartphoTokenizer\n\n[[autodoc]] BartphoTokenizer"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 8eb37fb8-154f-4b56-80b8-e4d09660da46
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/biogpt", "title": "BioGPT", "tokens": 949, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8eb37fb8-154f-4b56-80b8-e4d09660da46\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/biogpt\", \"title\": \"BioGPT\", \"tokens\": 949, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d19fcb2d-f99b-4138-99fc-7e2acc11e9f6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/biogpt\", \"title\": \"BioGPT\", \"tokens\": 949, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1f0e9175b8bf15f63a0372f9f8c8b5083f311f6022d15799137b2d8d03dbad48\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"641ecdc1-c3a1-4107-80eb-9d01a5e778a2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"222cf8c3c392d7965a3fc46ada93147f1de257bfc3f673339839c8b4d0b998a0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3231, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d19fcb2d-f99b-4138-99fc-7e2acc11e9f6", "doc_id": "d19fcb2d-f99b-4138-99fc-7e2acc11e9f6", "ref_doc_id": "d19fcb2d-f99b-4138-99fc-7e2acc11e9f6", "chroma:document": "# BioGPT\n\n## Overview\n\nThe BioGPT model was proposed in [BioGPT: generative pre-trained transformer for biomedical text generation and mining](https://academic.oup.com/bib/advance-article/doi/10.1093/bib/bbac409/6713511?guestAccessKey=a66d9b5d-4f83-4017-bb52-405815c907b9) by Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon and Tie-Yan Liu. BioGPT is a domain-specific generative pre-trained Transformer language model for biomedical text generation and mining. BioGPT follows the Transformer language model backbone, and is pre-trained on 15M PubMed abstracts from scratch.\n\nThe abstract from the paper is the following:\n\n*Pre-trained language models have attracted increasing attention in the biomedical domain, inspired by their great success in the general natural language domain. Among the two main branches of pre-trained language models in the general language domain, i.e. BERT (and its variants) and GPT (and its variants), the first one has been extensively studied in the biomedical domain, such as BioBERT and PubMedBERT. While they have achieved great success on a variety of discriminative downstream biomedical tasks, the lack of generation ability constrains their application scope. In this paper, we propose BioGPT, a domain-specific generative Transformer language model pre-trained on large-scale biomedical literature. We evaluate BioGPT on six biomedical natural language processing tasks and demonstrate that our model outperforms previous models on most tasks. Especially, we get 44.98%, 38.42% and 40.76% F1 score on BC5CDR, KD-DTI and DDI end-to-end relation extraction tasks, respectively, and 78.2% accuracy on PubMedQA, creating a new record. Our case study on text generation further demonstrates the advantage of BioGPT on biomedical literature to generate fluent descriptions for biomedical terms.*\n\nThis model was contributed by [kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/microsoft/BioGPT).\n\n## Usage tips\n\n- BioGPT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than the left.\n- BioGPT was trained with a causal language modeling (CLM) objective and is therefore powerful at predicting the next token in a sequence. Leveraging this feature allows BioGPT to generate syntactically coherent text as it can be observed in the run_generation.py example script.\n- The model can take the `past_key_values` (for PyTorch) as input, which is the previously computed key/value attention pairs. Using this (past_key_values or past) value prevents the model from re-computing pre-computed values in the context of text generation. For PyTorch, see past_key_values argument of the BioGptForCausalLM.forward() method for more information on its usage.\n\n## Resources\n\n- [Causal language modeling task guide](../tasks/language_modeling)\n\n## BioGptConfig\n\n[[autodoc]] BioGptConfig\n\n\n## BioGptTokenizer\n\n[[autodoc]] BioGptTokenizer\n    - save_vocabulary\n\n\n## BioGptModel\n\n[[autodoc]] BioGptModel\n    - forward\n\n\n## BioGptForCausalLM\n\n[[autodoc]] BioGptForCausalLM\n    - forward\n\n    \n## BioGptForTokenClassification\n\n[[autodoc]] BioGptForTokenClassification\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 641ecdc1-c3a1-4107-80eb-9d01a5e778a2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/biogpt", "title": "BioGPT", "tokens": 949, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"641ecdc1-c3a1-4107-80eb-9d01a5e778a2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/biogpt\", \"title\": \"BioGPT\", \"tokens\": 949, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d19fcb2d-f99b-4138-99fc-7e2acc11e9f6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/biogpt\", \"title\": \"BioGPT\", \"tokens\": 949, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1f0e9175b8bf15f63a0372f9f8c8b5083f311f6022d15799137b2d8d03dbad48\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8eb37fb8-154f-4b56-80b8-e4d09660da46\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/biogpt\", \"title\": \"BioGPT\", \"tokens\": 949, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5310debde05321e72c201fe5a229804509bd9e5be3e9c3053951708c553b0b9f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2944, \"end_char_idx\": 3327, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d19fcb2d-f99b-4138-99fc-7e2acc11e9f6", "doc_id": "d19fcb2d-f99b-4138-99fc-7e2acc11e9f6", "ref_doc_id": "d19fcb2d-f99b-4138-99fc-7e2acc11e9f6", "chroma:document": "## BioGptTokenizer\n\n[[autodoc]] BioGptTokenizer\n    - save_vocabulary\n\n\n## BioGptModel\n\n[[autodoc]] BioGptModel\n    - forward\n\n\n## BioGptForCausalLM\n\n[[autodoc]] BioGptForCausalLM\n    - forward\n\n    \n## BioGptForTokenClassification\n\n[[autodoc]] BioGptForTokenClassification\n    - forward\n\n\n## BioGptForSequenceClassification\n\n[[autodoc]] BioGptForSequenceClassification\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 734346c9-684c-45cc-a72a-ec9c6a9d484c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/speecht5", "title": "SpeechT5", "tokens": 845, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"734346c9-684c-45cc-a72a-ec9c6a9d484c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/speecht5\", \"title\": \"SpeechT5\", \"tokens\": 845, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dea93bb8-2d04-4119-b177-55b7ff37436d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/speecht5\", \"title\": \"SpeechT5\", \"tokens\": 845, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5ab9b289ae152568516619e94dd97d860fa184aed6ffcf730e5f1035a65dffce\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2738, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dea93bb8-2d04-4119-b177-55b7ff37436d", "doc_id": "dea93bb8-2d04-4119-b177-55b7ff37436d", "ref_doc_id": "dea93bb8-2d04-4119-b177-55b7ff37436d", "chroma:document": "# SpeechT5\n\n## Overview\n\nThe SpeechT5 model was proposed in [SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing](https://arxiv.org/abs/2110.07205) by Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, Furu Wei.\n\nThe abstract from the paper is the following:\n\n*Motivated by the success of T5 (Text-To-Text Transfer Transformer) in pre-trained natural language processing models, we propose a unified-modal SpeechT5 framework that explores the encoder-decoder pre-training for self-supervised speech/text representation learning. The SpeechT5 framework consists of a shared encoder-decoder network and six modal-specific (speech/text) pre/post-nets. After preprocessing the input speech/text through the pre-nets, the shared encoder-decoder network models the sequence-to-sequence transformation, and then the post-nets generate the output in the speech/text modality based on the output of the decoder. Leveraging large-scale unlabeled speech and text data, we pre-train SpeechT5 to learn a unified-modal representation, hoping to improve the modeling capability for both speech and text. To align the textual and speech information into this unified semantic space, we propose a cross-modal vector quantization approach that randomly mixes up speech/text states with latent units as the interface between encoder and decoder. Extensive evaluations show the superiority of the proposed SpeechT5 framework on a wide variety of spoken language processing tasks, including automatic speech recognition, speech synthesis, speech translation, voice conversion, speech enhancement, and speaker identification.*\n\nThis model was contributed by [Matthijs](https://huggingface.co/Matthijs). The original code can be found [here](https://github.com/microsoft/SpeechT5).\n\n## SpeechT5Config\n\n[[autodoc]] SpeechT5Config\n\n## SpeechT5HifiGanConfig\n\n[[autodoc]] SpeechT5HifiGanConfig\n\n## SpeechT5Tokenizer\n\n[[autodoc]] SpeechT5Tokenizer\n    - __call__\n    - save_vocabulary\n    - decode\n    - batch_decode\n\n## SpeechT5FeatureExtractor\n\n[[autodoc]] SpeechT5FeatureExtractor\n    - __call__\n\n## SpeechT5Processor\n\n[[autodoc]] SpeechT5Processor\n    - __call__\n    - pad\n    - from_pretrained\n    - save_pretrained\n    - batch_decode\n    - decode\n\n## SpeechT5Model\n\n[[autodoc]] SpeechT5Model\n    - forward\n\n## SpeechT5ForSpeechToText\n\n[[autodoc]] SpeechT5ForSpeechToText\n    - forward\n\n## SpeechT5ForTextToSpeech\n\n[[autodoc]] SpeechT5ForTextToSpeech\n    - forward\n    - generate\n\n## SpeechT5ForSpeechToSpeech\n\n[[autodoc]] SpeechT5ForSpeechToSpeech\n    - forward\n    - generate_speech\n\n## SpeechT5HifiGan\n\n[[autodoc]] SpeechT5HifiGan\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 9365b1ff-7695-4eaa-a33e-0be9382c57d4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mobilenet_v1", "title": "MobileNet V1", "tokens": 1239, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9365b1ff-7695-4eaa-a33e-0be9382c57d4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilenet_v1\", \"title\": \"MobileNet V1\", \"tokens\": 1239, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dc1c8485-eb41-4663-8152-055e7fa1bf7e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilenet_v1\", \"title\": \"MobileNet V1\", \"tokens\": 1239, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cc756dcef81023f96529c34925be84d96b335510f0e706d4887bbf897364c5f5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"64291aaf-a6e2-4b3b-8ff5-8fcd309dd77e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e723db98fb83ed78eabc5540a51f436096ac48e25bc0f039e2c90155d197212c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3373, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dc1c8485-eb41-4663-8152-055e7fa1bf7e", "doc_id": "dc1c8485-eb41-4663-8152-055e7fa1bf7e", "ref_doc_id": "dc1c8485-eb41-4663-8152-055e7fa1bf7e", "chroma:document": "# MobileNet V1\n\n## Overview\n\nThe MobileNet model was proposed in [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://arxiv.org/abs/1704.04861) by Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, Hartwig Adam.\n\nThe abstract from the paper is the following:\n\n*We present a class of efficient models called MobileNets for mobile and embedded vision applications. MobileNets are based on a streamlined architecture that uses depth-wise separable convolutions to build light weight deep neural networks. We introduce two simple global hyper-parameters that efficiently trade off between latency and accuracy. These hyper-parameters allow the model builder to choose the right sized model for their application based on the constraints of the problem. We present extensive experiments on resource and accuracy tradeoffs and show strong performance compared to other popular models on ImageNet classification. We then demonstrate the effectiveness of MobileNets across a wide range of applications and use cases including object detection, finegrain classification, face attributes and large scale geo-localization.*\n\nThis model was contributed by [matthijs](https://huggingface.co/Matthijs). The original code and weights can be found [here](https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet_v1.md).\n\n## Usage tips\n\n- The checkpoints are named **mobilenet\\_v1\\_*depth*\\_*size***, for example **mobilenet\\_v1\\_1.0\\_224**, where **1.0** is the depth multiplier (sometimes also referred to as \"alpha\" or the width multiplier) and **224** is the resolution of the input images the model was trained on.\n\n- Even though the checkpoint is trained on images of specific size, the model will work on images of any size. The smallest supported image size is 32x32.\n\n- One can use [`MobileNetV1ImageProcessor`] to prepare images for the model.\n\n- The available image classification checkpoints are pre-trained on [ImageNet-1k](https://huggingface.co/datasets/imagenet-1k) (also referred to as ILSVRC 2012, a collection of 1.3 million images and 1,000 classes). However, the model predicts 1001 classes: the 1000 classes from ImageNet plus an extra \u201cbackground\u201d class (index 0).\n\n- The original TensorFlow checkpoints use different padding rules than PyTorch, requiring the model to determine the padding amount at inference time, since this depends on the input image size. To use native PyTorch padding behavior, create a [`MobileNetV1Config`] with `tf_padding = False`.\n\nUnsupported features:\n\n- The [`MobileNetV1Model`] outputs a globally pooled version of the last hidden state. In the original model it is possible to use a 7x7 average pooling layer with stride 2 instead of global pooling. For larger inputs, this gives a pooled output that is larger than 1x1 pixel. The HuggingFace implementation does not support this.\n\n- It is currently not possible to specify an `output_stride`. For smaller output strides, the original model invokes dilated convolution to prevent the spatial resolution from being reduced further. The output stride of the HuggingFace model is always 32.\n\n- The original TensorFlow checkpoints include quantized models. We do not support these models as they include additional \"FakeQuantization\" operations to unquantize the weights."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 64291aaf-a6e2-4b3b-8ff5-8fcd309dd77e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mobilenet_v1", "title": "MobileNet V1", "tokens": 1239, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"64291aaf-a6e2-4b3b-8ff5-8fcd309dd77e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilenet_v1\", \"title\": \"MobileNet V1\", \"tokens\": 1239, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dc1c8485-eb41-4663-8152-055e7fa1bf7e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilenet_v1\", \"title\": \"MobileNet V1\", \"tokens\": 1239, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cc756dcef81023f96529c34925be84d96b335510f0e706d4887bbf897364c5f5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9365b1ff-7695-4eaa-a33e-0be9382c57d4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilenet_v1\", \"title\": \"MobileNet V1\", \"tokens\": 1239, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cefc6fa4e0b6711b01ab6a5a93328d435f2e1a89adb533173b5215a8a54ea80d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1721, \"end_char_idx\": 4803, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dc1c8485-eb41-4663-8152-055e7fa1bf7e", "doc_id": "dc1c8485-eb41-4663-8152-055e7fa1bf7e", "ref_doc_id": "dc1c8485-eb41-4663-8152-055e7fa1bf7e", "chroma:document": "- Even though the checkpoint is trained on images of specific size, the model will work on images of any size. The smallest supported image size is 32x32.\n\n- One can use [`MobileNetV1ImageProcessor`] to prepare images for the model.\n\n- The available image classification checkpoints are pre-trained on [ImageNet-1k](https://huggingface.co/datasets/imagenet-1k) (also referred to as ILSVRC 2012, a collection of 1.3 million images and 1,000 classes). However, the model predicts 1001 classes: the 1000 classes from ImageNet plus an extra \u201cbackground\u201d class (index 0).\n\n- The original TensorFlow checkpoints use different padding rules than PyTorch, requiring the model to determine the padding amount at inference time, since this depends on the input image size. To use native PyTorch padding behavior, create a [`MobileNetV1Config`] with `tf_padding = False`.\n\nUnsupported features:\n\n- The [`MobileNetV1Model`] outputs a globally pooled version of the last hidden state. In the original model it is possible to use a 7x7 average pooling layer with stride 2 instead of global pooling. For larger inputs, this gives a pooled output that is larger than 1x1 pixel. The HuggingFace implementation does not support this.\n\n- It is currently not possible to specify an `output_stride`. For smaller output strides, the original model invokes dilated convolution to prevent the spatial resolution from being reduced further. The output stride of the HuggingFace model is always 32.\n\n- The original TensorFlow checkpoints include quantized models. We do not support these models as they include additional \"FakeQuantization\" operations to unquantize the weights.\n\n- It's common to extract the output from the pointwise layers at indices 5, 11, 12, 13 for downstream purposes. Using `output_hidden_states=True` returns the output from all intermediate layers. There is currently no way to limit this to specific layers.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with MobileNetV1.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`MobileNetV1ForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## MobileNetV1Config\n\n[[autodoc]] MobileNetV1Config\n\n## MobileNetV1FeatureExtractor\n\n[[autodoc]] MobileNetV1FeatureExtractor\n    - preprocess\n\n## MobileNetV1ImageProcessor\n\n[[autodoc]] MobileNetV1ImageProcessor\n    - preprocess\n\n## MobileNetV1Model\n\n[[autodoc]] MobileNetV1Model\n    - forward\n\n## MobileNetV1ForImageClassification\n\n[[autodoc]] MobileNetV1ForImageClassification\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 4daaa9ee-0811-4a14-aed3-3161b0f106e7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/jetmoe", "title": "JetMoe", "tokens": 506, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4daaa9ee-0811-4a14-aed3-3161b0f106e7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jetmoe\", \"title\": \"JetMoe\", \"tokens\": 506, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e7ecdfa4-91c0-47f7-8b8c-55a2a728893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jetmoe\", \"title\": \"JetMoe\", \"tokens\": 506, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8684ce766325fedc198ac2435f441b62b98fbbe99672150e91a13eadec9131e7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1247, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e7ecdfa4-91c0-47f7-8b8c-55a2a728893c", "doc_id": "e7ecdfa4-91c0-47f7-8b8c-55a2a728893c", "ref_doc_id": "e7ecdfa4-91c0-47f7-8b8c-55a2a728893c", "chroma:document": "# JetMoe\n\n## Overview\n\n**JetMoe-8B** is an 8B Mixture-of-Experts (MoE) language model developed by [Yikang Shen](https://scholar.google.com.hk/citations?user=qff5rRYAAAAJ) and [MyShell](https://myshell.ai/).\nJetMoe project aims to provide a LLaMA2-level performance and efficient language model with a limited budget.\nTo achieve this goal, JetMoe uses a sparsely activated architecture inspired by the [ModuleFormer](https://arxiv.org/abs/2306.04640). \nEach JetMoe block consists of two MoE layers: Mixture of Attention Heads and Mixture of MLP Experts.\nGiven the input tokens, it activates a subset of its experts to process them.\nThis sparse activation schema enables JetMoe to achieve much better training throughput than similar size dense models. \nThe training throughput of JetMoe-8B is around 100B tokens per day on a cluster of 96 H100 GPUs with a straightforward 3-way pipeline parallelism strategy.\n\nThis model was contributed by [Yikang Shen](https://huggingface.co/YikangS).\n\n\n## JetMoeConfig\n\n[[autodoc]] JetMoeConfig\n\n## JetMoeModel\n\n[[autodoc]] JetMoeModel\n    - forward\n\n## JetMoeForCausalLM\n\n[[autodoc]] JetMoeForCausalLM\n    - forward\n\n## JetMoeForSequenceClassification\n\n[[autodoc]] JetMoeForSequenceClassification\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: e5d36576-c188-4eec-bc77-88a066095af3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta-xl", "title": "XLM-RoBERTa-XL", "tokens": 849, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e5d36576-c188-4eec-bc77-88a066095af3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-roberta-xl\", \"title\": \"XLM-RoBERTa-XL\", \"tokens\": 849, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0d040ecd-57a7-479a-8645-9dcd0f2363a1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-roberta-xl\", \"title\": \"XLM-RoBERTa-XL\", \"tokens\": 849, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"75d481267b5776122b84e5856634c7c7bf373394d2081b29254048efaa23d3fe\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2614, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0d040ecd-57a7-479a-8645-9dcd0f2363a1", "doc_id": "0d040ecd-57a7-479a-8645-9dcd0f2363a1", "ref_doc_id": "0d040ecd-57a7-479a-8645-9dcd0f2363a1", "chroma:document": "# XLM-RoBERTa-XL\n\n## Overview\n\nThe XLM-RoBERTa-XL model was proposed in [Larger-Scale Transformers for Multilingual Masked Language Modeling](https://arxiv.org/abs/2105.00572) by Naman Goyal, Jingfei Du, Myle Ott, Giri Anantharaman, Alexis Conneau. \n\nThe abstract from the paper is the following:\n\n*Recent work has demonstrated the effectiveness of cross-lingual language model pretraining for cross-lingual understanding. In this study, we present the results of two larger multilingual masked language models, with 3.5B and 10.7B parameters. Our two new models dubbed XLM-R XL and XLM-R XXL outperform XLM-R by 1.8% and 2.4% average accuracy on XNLI. Our model also outperforms the RoBERTa-Large model on several English tasks of the GLUE benchmark by 0.3% on average while handling 99 more languages. This suggests pretrained models with larger capacity may obtain both strong performance on high-resource languages while greatly improving low-resource languages. We make our code and models publicly available.*\n\nThis model was contributed by [Soonhwan-Kwon](https://github.com/Soonhwan-Kwon) and [stefan-it](https://huggingface.co/stefan-it). The original code can be found [here](https://github.com/pytorch/fairseq/tree/master/examples/xlmr).\n\n## Usage tips\n\nXLM-RoBERTa-XL is a multilingual model trained on 100 different languages. Unlike some XLM multilingual models, it does \nnot require `lang` tensors to understand which language is used, and should be able to determine the correct \nlanguage from the input ids.\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## XLMRobertaXLConfig\n\n[[autodoc]] XLMRobertaXLConfig\n\n## XLMRobertaXLModel\n\n[[autodoc]] XLMRobertaXLModel\n    - forward\n\n## XLMRobertaXLForCausalLM\n\n[[autodoc]] XLMRobertaXLForCausalLM\n    - forward\n\n## XLMRobertaXLForMaskedLM\n\n[[autodoc]] XLMRobertaXLForMaskedLM\n    - forward\n\n## XLMRobertaXLForSequenceClassification\n\n[[autodoc]] XLMRobertaXLForSequenceClassification\n    - forward\n\n## XLMRobertaXLForMultipleChoice\n\n[[autodoc]] XLMRobertaXLForMultipleChoice\n    - forward\n\n## XLMRobertaXLForTokenClassification\n\n[[autodoc]] XLMRobertaXLForTokenClassification\n    - forward\n\n## XLMRobertaXLForQuestionAnswering\n\n[[autodoc]] XLMRobertaXLForQuestionAnswering\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 57ddd75e-0e74-4f13-a0e3-e7c751550566
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bit", "title": "Big Transfer (BiT)", "tokens": 890, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"57ddd75e-0e74-4f13-a0e3-e7c751550566\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bit\", \"title\": \"Big Transfer (BiT)\", \"tokens\": 890, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ad35c2e6-5356-4f5e-89d0-a60ac7fcf4e9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bit\", \"title\": \"Big Transfer (BiT)\", \"tokens\": 890, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"528ac4260d29d9fc92230c49b6f5fa694d0a5405a764fb143540ad9638c0a137\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3082, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ad35c2e6-5356-4f5e-89d0-a60ac7fcf4e9", "doc_id": "ad35c2e6-5356-4f5e-89d0-a60ac7fcf4e9", "ref_doc_id": "ad35c2e6-5356-4f5e-89d0-a60ac7fcf4e9", "chroma:document": "# Big Transfer (BiT)\n\n## Overview\n\nThe BiT model was proposed in [Big Transfer (BiT): General Visual Representation Learning](https://arxiv.org/abs/1912.11370) by Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, Neil Houlsby.\nBiT is a simple recipe for scaling up pre-training of [ResNet](resnet)-like architectures (specifically, ResNetv2). The method results in significant improvements for transfer learning.\n\nThe abstract from the paper is the following:\n\n*Transfer of pre-trained representations improves sample efficiency and simplifies hyperparameter tuning when training deep neural networks for vision. We revisit the paradigm of pre-training on large supervised datasets and fine-tuning the model on a target task. We scale up pre-training, and propose a simple recipe that we call Big Transfer (BiT). By combining a few carefully selected components, and transferring using a simple heuristic, we achieve strong performance on over 20 datasets. BiT performs well across a surprisingly wide range of data regimes -- from 1 example per class to 1M total examples. BiT achieves 87.5% top-1 accuracy on ILSVRC-2012, 99.4% on CIFAR-10, and 76.3% on the 19 task Visual Task Adaptation Benchmark (VTAB). On small datasets, BiT attains 76.8% on ILSVRC-2012 with 10 examples per class, and 97.0% on CIFAR-10 with 10 examples per class. We conduct detailed analysis of the main components that lead to high transfer performance.*\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/google-research/big_transfer).\n\n## Usage tips\n\n- BiT models are equivalent to ResNetv2 in terms of architecture, except that: 1) all batch normalization layers are replaced by [group normalization](https://arxiv.org/abs/1803.08494),\n2) [weight standardization](https://arxiv.org/abs/1903.10520) is used for convolutional layers. The authors show that the combination of both is useful for training with large batch sizes, and has a significant\nimpact on transfer learning.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with BiT.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`BitForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## BitConfig\n\n[[autodoc]] BitConfig\n\n## BitImageProcessor\n\n[[autodoc]] BitImageProcessor\n    - preprocess\n\n## BitModel\n\n[[autodoc]] BitModel\n    - forward\n\n## BitForImageClassification\n\n[[autodoc]] BitForImageClassification\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 13a27d17-9988-47fe-86b4-ac5032cf7900
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/idefics", "title": "IDEFICS", "tokens": 743, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"13a27d17-9988-47fe-86b4-ac5032cf7900\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics\", \"title\": \"IDEFICS\", \"tokens\": 743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4bae8221-9e31-42c5-846c-8620b82e6663\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics\", \"title\": \"IDEFICS\", \"tokens\": 743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b941ca0f6bc20ee29516c8e0810f9f8fe26d387aeec4c12f95043ab212cf8a90\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2319, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4bae8221-9e31-42c5-846c-8620b82e6663", "doc_id": "4bae8221-9e31-42c5-846c-8620b82e6663", "ref_doc_id": "4bae8221-9e31-42c5-846c-8620b82e6663", "chroma:document": "# IDEFICS\n\n## Overview\n\nThe IDEFICS model was proposed in [OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents\n](https://huggingface.co/papers/2306.16527\n) by Hugo Lauren\u00e7on, Lucile Saulnier, L\u00e9o Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, Victor Sanh\n\nThe abstract from the paper is the following:\n\n*Large multimodal models trained on natural documents, which interleave images and text, outperform models trained on image-text pairs on various multimodal benchmarks that require reasoning over one or multiple images to generate a text. However, the datasets used to train these models have not been released, and the collection process has not been fully specified. We introduce the OBELICS dataset, an open web-scale filtered dataset of interleaved image-text documents comprising 141 million web pages extracted from Common Crawl, 353 million associated images, and 115 billion text tokens. We describe the dataset creation process, present comprehensive filtering rules, and provide an analysis of the dataset's content. To show the viability of OBELISC, we train an 80 billion parameters vision and language model on the dataset and obtain competitive performance on various multimodal benchmarks. We release the code to reproduce the dataset along with the dataset itself.*\n\nThis model was contributed by [HuggingFaceM4](https://huggingface.co/HuggingFaceM4). The original code can be found [here](<INSERT LINK TO GITHUB REPO HERE>). (TODO: don't have a public link yet).\n\n\n<Tip warning={true}>\n\nIDEFICS modeling code in Transformers is for finetuning and inferencing the pre-trained IDEFICS models.\n\nTo train a new IDEFICS model from scratch use the m4 codebase (a link will be provided once it's made public)\n\n</Tip>\n\n\n## IdeficsConfig\n\n[[autodoc]] IdeficsConfig\n\n## IdeficsModel\n\n[[autodoc]] IdeficsModel\n    - forward\n\n## IdeficsForVisionText2Text\n\n[[autodoc]] IdeficsForVisionText2Text\n    - forward\n\n## TFIdeficsModel\n\n[[autodoc]] TFIdeficsModel\n    - call\n\n## TFIdeficsForVisionText2Text\n\n[[autodoc]] TFIdeficsForVisionText2Text\n    - call\n\n## IdeficsImageProcessor\n\n[[autodoc]] IdeficsImageProcessor\n    - preprocess\n\n## IdeficsProcessor\n\n[[autodoc]] IdeficsProcessor\n    - __call__"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: d60e01b2-b4cd-4dba-878e-f3f017a4ddee
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vilt", "title": "ViLT", "tokens": 1055, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d60e01b2-b4cd-4dba-878e-f3f017a4ddee\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vilt\", \"title\": \"ViLT\", \"tokens\": 1055, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"072ff23f-2591-4cb7-9a64-2885345bd4aa\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vilt\", \"title\": \"ViLT\", \"tokens\": 1055, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"40866bcab7c8c567834d5846571dc84f5d1ba53fb55cb6326439e5d5d4c628b8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"90ec1fed-76e0-44d2-803b-c32ca26513c3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e1bdb4b6cbed24efffabdc6e056d3eadf630777174a9729feb5d5d62dbead28d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2977, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "072ff23f-2591-4cb7-9a64-2885345bd4aa", "doc_id": "072ff23f-2591-4cb7-9a64-2885345bd4aa", "ref_doc_id": "072ff23f-2591-4cb7-9a64-2885345bd4aa", "chroma:document": "# ViLT\n\n## Overview\n\nThe ViLT model was proposed in [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://arxiv.org/abs/2102.03334)\nby Wonjae Kim, Bokyung Son, Ildoo Kim. ViLT incorporates text embeddings into a Vision Transformer (ViT), allowing it to have a minimal design\nfor Vision-and-Language Pre-training (VLP).\n\nThe abstract from the paper is the following:\n\n*Vision-and-Language Pre-training (VLP) has improved performance on various joint vision-and-language downstream tasks.\nCurrent approaches to VLP heavily rely on image feature extraction processes, most of which involve region supervision\n(e.g., object detection) and the convolutional architecture (e.g., ResNet). Although disregarded in the literature, we\nfind it problematic in terms of both (1) efficiency/speed, that simply extracting input features requires much more\ncomputation than the multimodal interaction steps; and (2) expressive power, as it is upper bounded to the expressive\npower of the visual embedder and its predefined visual vocabulary. In this paper, we present a minimal VLP model,\nVision-and-Language Transformer (ViLT), monolithic in the sense that the processing of visual inputs is drastically\nsimplified to just the same convolution-free manner that we process textual inputs. We show that ViLT is up to tens of\ntimes faster than previous VLP models, yet with competitive or better downstream task performance.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vilt_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> ViLT architecture. Taken from the <a href=\"https://arxiv.org/abs/2102.03334\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/dandelin/ViLT).\n\n## Usage tips\n\n- The quickest way to get started with ViLT is by checking the [example notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/ViLT)\n  (which showcase both inference and fine-tuning on custom data).\n- ViLT is a model that takes both `pixel_values` and `input_ids` as input. One can use [`ViltProcessor`] to prepare data for the model.\n  This processor wraps a image processor (for the image modality) and a tokenizer (for the language modality) into one.\n- ViLT is trained with images of various sizes: the authors resize the shorter edge of input images to 384 and limit the longer edge to\n  under 640 while preserving the aspect ratio. To make batching of images possible, the authors use a `pixel_mask` that indicates\n  which pixel values are real and which are padding. [`ViltProcessor`] automatically creates this for you.\n- The design of ViLT is very similar to that of a standard Vision Transformer (ViT). The only difference is that the model includes\n  additional embedding layers for the language modality.\n- The PyTorch version of this model is only available in torch 1.10 and higher."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 90ec1fed-76e0-44d2-803b-c32ca26513c3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vilt", "title": "ViLT", "tokens": 1055, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"90ec1fed-76e0-44d2-803b-c32ca26513c3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vilt\", \"title\": \"ViLT\", \"tokens\": 1055, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"072ff23f-2591-4cb7-9a64-2885345bd4aa\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vilt\", \"title\": \"ViLT\", \"tokens\": 1055, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"40866bcab7c8c567834d5846571dc84f5d1ba53fb55cb6326439e5d5d4c628b8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d60e01b2-b4cd-4dba-878e-f3f017a4ddee\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vilt\", \"title\": \"ViLT\", \"tokens\": 1055, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f0412cf16d1609611eada350244e6b16fcb969917e09179cde87e6c2b2535e18\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1442, \"end_char_idx\": 3692, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "072ff23f-2591-4cb7-9a64-2885345bd4aa", "doc_id": "072ff23f-2591-4cb7-9a64-2885345bd4aa", "ref_doc_id": "072ff23f-2591-4cb7-9a64-2885345bd4aa", "chroma:document": "*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vilt_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> ViLT architecture. Taken from the <a href=\"https://arxiv.org/abs/2102.03334\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/dandelin/ViLT).\n\n## Usage tips\n\n- The quickest way to get started with ViLT is by checking the [example notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/ViLT)\n  (which showcase both inference and fine-tuning on custom data).\n- ViLT is a model that takes both `pixel_values` and `input_ids` as input. One can use [`ViltProcessor`] to prepare data for the model.\n  This processor wraps a image processor (for the image modality) and a tokenizer (for the language modality) into one.\n- ViLT is trained with images of various sizes: the authors resize the shorter edge of input images to 384 and limit the longer edge to\n  under 640 while preserving the aspect ratio. To make batching of images possible, the authors use a `pixel_mask` that indicates\n  which pixel values are real and which are padding. [`ViltProcessor`] automatically creates this for you.\n- The design of ViLT is very similar to that of a standard Vision Transformer (ViT). The only difference is that the model includes\n  additional embedding layers for the language modality.\n- The PyTorch version of this model is only available in torch 1.10 and higher.\n\n## ViltConfig\n\n[[autodoc]] ViltConfig\n\n## ViltFeatureExtractor\n\n[[autodoc]] ViltFeatureExtractor\n    - __call__\n\n## ViltImageProcessor\n\n[[autodoc]] ViltImageProcessor\n    - preprocess\n\n## ViltProcessor\n\n[[autodoc]] ViltProcessor\n    - __call__\n\n## ViltModel\n\n[[autodoc]] ViltModel\n    - forward\n\n## ViltForMaskedLM\n\n[[autodoc]] ViltForMaskedLM\n    - forward\n\n## ViltForQuestionAnswering\n\n[[autodoc]] ViltForQuestionAnswering\n    - forward\n\n## ViltForImagesAndTextClassification\n\n[[autodoc]] ViltForImagesAndTextClassification\n    - forward\n\n## ViltForImageAndTextRetrieval\n\n[[autodoc]] ViltForImageAndTextRetrieval\n    - forward\n\n## ViltForTokenClassification\n\n[[autodoc]] ViltForTokenClassification\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 53ce51cf-0e08-4728-8869-0e8c56749a1c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mpt", "title": "MPT", "tokens": 704, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"53ce51cf-0e08-4728-8869-0e8c56749a1c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mpt\", \"title\": \"MPT\", \"tokens\": 704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3c76267e-e07e-4eaa-900f-e8cf8eaa0f5a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mpt\", \"title\": \"MPT\", \"tokens\": 704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"09a8105580481f1d98ee1bebfc0a5025d8e3363e4fc5f2d2ed794ccf4693b168\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2103, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3c76267e-e07e-4eaa-900f-e8cf8eaa0f5a", "doc_id": "3c76267e-e07e-4eaa-900f-e8cf8eaa0f5a", "ref_doc_id": "3c76267e-e07e-4eaa-900f-e8cf8eaa0f5a", "chroma:document": "# MPT\n\n## Overview\n\nThe MPT model was proposed by the [MosaicML](https://www.mosaicml.com/) team and released with multiple sizes and finetuned variants. The MPT models is a series of open source and commercially usable LLMs pre-trained on 1T tokens. \n\nMPT models are GPT-style decoder-only transformers with several improvements: performance-optimized layer implementations, architecture changes that provide greater training stability, and the elimination of context length limits by replacing positional embeddings with ALiBi. \n\n- MPT base: MPT base pre-trained models on next token prediction \n- MPT instruct: MPT base models fine-tuned on instruction based tasks\n- MPT storywriter: MPT base models fine-tuned for 2500 steps on 65k-token excerpts of fiction books contained in the books3 corpus, this enables the model to handle very long sequences\n\nThe original code is available at the  [`llm-foundry`](https://github.com/mosaicml/llm-foundry/tree/main) repository.\n\nRead more about it [in the release blogpost](https://www.mosaicml.com/blog/mpt-7b)\n\n## Usage tips\n\n- Learn more about some techniques behind training of the model [in this section of llm-foundry repository](https://github.com/mosaicml/llm-foundry/blob/main/TUTORIAL.md#faqs)\n- If you want to use the advanced version of the model (triton kernels, direct flash attention integration), you can still use the original model implementation by adding `trust_remote_code=True` when calling `from_pretrained`.\n\n## Resources\n\n- [Fine-tuning Notebook](https://colab.research.google.com/drive/1HCpQkLL7UXW8xJUJJ29X7QAeNJKO0frZ?usp=sharing) on how to fine-tune MPT-7B on a free Google Colab instance to turn the model into a Chatbot.\n\n## MptConfig\n\n[[autodoc]] MptConfig\n    - all\n\n## MptModel\n\n[[autodoc]] MptModel\n    - forward\n\n## MptForCausalLM\n\n[[autodoc]] MptForCausalLM\n    - forward\n\n## MptForSequenceClassification\n\n[[autodoc]] MptForSequenceClassification\n    - forward\n\n## MptForTokenClassification\n\n[[autodoc]] MptForTokenClassification\n    - forward\n\n## MptForQuestionAnswering\n\n[[autodoc]] MptForQuestionAnswering\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 56911fd5-ac8b-4e1b-a6a8-f142bea51f19
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/code_llama", "title": "CodeLlama", "tokens": 2064, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"56911fd5-ac8b-4e1b-a6a8-f142bea51f19\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/code_llama\", \"title\": \"CodeLlama\", \"tokens\": 2064, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"177b6cd1-07b3-4bce-882d-b7e0afc62f62\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/code_llama\", \"title\": \"CodeLlama\", \"tokens\": 2064, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"18affe0a10ce1571f11526a8082f64dd55465a1884ca565bcc3d19dd4e0db30d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"adbff70c-1b26-485e-9a13-e7dcb813fad5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"452619de1cc79c67d6bba72b39f7e7e5b71310d035d82bcd183bb01345f026ed\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2959, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "177b6cd1-07b3-4bce-882d-b7e0afc62f62", "doc_id": "177b6cd1-07b3-4bce-882d-b7e0afc62f62", "ref_doc_id": "177b6cd1-07b3-4bce-882d-b7e0afc62f62", "chroma:document": "# CodeLlama\n\n## Overview\n\nThe Code Llama model was proposed in [Code Llama: Open Foundation Models for Code](https://ai.meta.com/research/publications/code-llama-open-foundation-models-for-code/) by Baptiste Rozi\u00e8re, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, J\u00e9r\u00e9my Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre D\u00e9fossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, Gabriel Synnaeve.\n\nThe abstract from the paper is the following:\n\n*We release Code Llama, a family of large language models for code based on Llama 2 providing state-of-the-art performance among open models, infilling capabilities, support for large input contexts, and zero-shot instruction following ability for programming tasks. We provide multiple flavors to cover a wide range of applications: foundation models (Code Llama), Python specializations (Code Llama - Python), and instruction-following models (Code Llama - Instruct) with 7B, 13B and 34B parameters each. All models are trained on sequences of 16k tokens and show improvements on inputs with up to 100k tokens. 7B and 13B Code Llama and Code Llama - Instruct variants support infilling based on surrounding content. Code Llama reaches state-of-the-art performance among open models on several code benchmarks, with scores of up to 53% and 55% on HumanEval and MBPP, respectively. Notably, Code Llama - Python 7B outperforms Llama 2 70B on HumanEval and MBPP, and all our models outperform every other publicly available model on MultiPL-E. We release Code Llama under a permissive license that allows for both research and commercial use.*\n\nCheck out all Code Llama model checkpoints [here](https://huggingface.co/models?search=code_llama) and the officially released ones in the [Meta Llama org](https://huggingface.co/meta-llama).\n\nThis model was contributed by [ArthurZucker](https://huggingface.co/ArthurZ). The original code of the authors can be found [here](https://github.com/facebookresearch/llama).\n\n## Usage tips and examples\n\n<Tip warning={true}>\n\nThe `Llama2` family models, on which Code Llama is based, were trained using `bfloat16`, but the original inference uses `float16`. Let's look at the different precisions:\n\n* `float32`: PyTorch convention on model initialization is to load models in `float32`, no matter with which `dtype` the model weights were stored. `transformers` also follows this convention for consistency with PyTorch. This will be picked by default. If you want the `AutoModel` API to cast the load the checkpoints with the storage weights type, you must specify `torch_dtype=\"auto\"`, e.g. `model = AutoModelForCausalLM.from_pretrained(\"path\", torch_dtype = \"auto\")`.\n* `bfloat16`: Code Llama was trained with this precision, so we recommend using it for further training or fine-tuning."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: adbff70c-1b26-485e-9a13-e7dcb813fad5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/code_llama", "title": "CodeLlama", "tokens": 2064, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"adbff70c-1b26-485e-9a13-e7dcb813fad5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/code_llama\", \"title\": \"CodeLlama\", \"tokens\": 2064, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"177b6cd1-07b3-4bce-882d-b7e0afc62f62\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/code_llama\", \"title\": \"CodeLlama\", \"tokens\": 2064, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"18affe0a10ce1571f11526a8082f64dd55465a1884ca565bcc3d19dd4e0db30d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"56911fd5-ac8b-4e1b-a6a8-f142bea51f19\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/code_llama\", \"title\": \"CodeLlama\", \"tokens\": 2064, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"05dea4d9a0290fa0a20acc15d36be3089515b889f97f151e7aabcb4848741c7c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"27e50627-271f-4016-85d4-e628a016ac3a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2b1cbc272f1079ada86ccd7aac3e504c05ce46c243096b3f18662ae8dedf4bea\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1351, \"end_char_idx\": 4366, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "177b6cd1-07b3-4bce-882d-b7e0afc62f62", "doc_id": "177b6cd1-07b3-4bce-882d-b7e0afc62f62", "ref_doc_id": "177b6cd1-07b3-4bce-882d-b7e0afc62f62", "chroma:document": "Code Llama reaches state-of-the-art performance among open models on several code benchmarks, with scores of up to 53% and 55% on HumanEval and MBPP, respectively. Notably, Code Llama - Python 7B outperforms Llama 2 70B on HumanEval and MBPP, and all our models outperform every other publicly available model on MultiPL-E. We release Code Llama under a permissive license that allows for both research and commercial use.*\n\nCheck out all Code Llama model checkpoints [here](https://huggingface.co/models?search=code_llama) and the officially released ones in the [Meta Llama org](https://huggingface.co/meta-llama).\n\nThis model was contributed by [ArthurZucker](https://huggingface.co/ArthurZ). The original code of the authors can be found [here](https://github.com/facebookresearch/llama).\n\n## Usage tips and examples\n\n<Tip warning={true}>\n\nThe `Llama2` family models, on which Code Llama is based, were trained using `bfloat16`, but the original inference uses `float16`. Let's look at the different precisions:\n\n* `float32`: PyTorch convention on model initialization is to load models in `float32`, no matter with which `dtype` the model weights were stored. `transformers` also follows this convention for consistency with PyTorch. This will be picked by default. If you want the `AutoModel` API to cast the load the checkpoints with the storage weights type, you must specify `torch_dtype=\"auto\"`, e.g. `model = AutoModelForCausalLM.from_pretrained(\"path\", torch_dtype = \"auto\")`.\n* `bfloat16`: Code Llama was trained with this precision, so we recommend using it for further training or fine-tuning.\n* `float16`: We recommend running inference using this precision, as it's usually faster than `bfloat16`, and evaluation metrics show no discernible degradation with respect to `bfloat16`. You can also run inference using `bfloat16`, and we recommend you check inference results with both `float16` and `bfloat16` after fine-tuning.\n\nAs mentioned above, the `dtype` of the storage weights is mostly irrelevant unless you are using `torch_dtype=\"auto\"` when initializing a model using. The reason is that the model will first be downloaded (using the `dtype` of the checkpoints online) and then will be casted to the default `dtype` of `torch` (becomes `torch.float32`). If there is a specified `torch_dtype`, it will be used instead.\n\n</Tip>\n\n\nTips:\n- The infilling task is supported out of the box. You should be using the `tokenizer.fill_token` where you want your input to be filled.\n- The model conversion script is the same as for the `Llama2` family:\n\nHere is a sample usage:\n\n```bash\npython src/transformers/models/llama/convert_llama_weights_to_hf.py \\\n    --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path\n```\n\nNote that executing the script requires enough CPU RAM to host the whole model in float16 precision (even if the biggest versions\ncome in several checkpoints they each contain a part of each weight of the model, so we need to load them all in RAM)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 27e50627-271f-4016-85d4-e628a016ac3a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/code_llama", "title": "CodeLlama", "tokens": 2064, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"27e50627-271f-4016-85d4-e628a016ac3a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/code_llama\", \"title\": \"CodeLlama\", \"tokens\": 2064, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"177b6cd1-07b3-4bce-882d-b7e0afc62f62\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/code_llama\", \"title\": \"CodeLlama\", \"tokens\": 2064, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"18affe0a10ce1571f11526a8082f64dd55465a1884ca565bcc3d19dd4e0db30d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"adbff70c-1b26-485e-9a13-e7dcb813fad5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/code_llama\", \"title\": \"CodeLlama\", \"tokens\": 2064, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7879a7a667e001ded38d968292d0944c7b5cefa03e5fcaa9d22087c3c75c1ca8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e7b84b54-35e3-48d0-a408-ceedb14167e3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"70b84ecc4635d909ea9f23b1152a872de5e2f65d3f7be9a81ae610f9e06dabf4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2762, \"end_char_idx\": 5313, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "177b6cd1-07b3-4bce-882d-b7e0afc62f62", "doc_id": "177b6cd1-07b3-4bce-882d-b7e0afc62f62", "ref_doc_id": "177b6cd1-07b3-4bce-882d-b7e0afc62f62", "chroma:document": "`model = AutoModelForCausalLM.from_pretrained(\"path\", torch_dtype = \"auto\")`.\n* `bfloat16`: Code Llama was trained with this precision, so we recommend using it for further training or fine-tuning.\n* `float16`: We recommend running inference using this precision, as it's usually faster than `bfloat16`, and evaluation metrics show no discernible degradation with respect to `bfloat16`. You can also run inference using `bfloat16`, and we recommend you check inference results with both `float16` and `bfloat16` after fine-tuning.\n\nAs mentioned above, the `dtype` of the storage weights is mostly irrelevant unless you are using `torch_dtype=\"auto\"` when initializing a model using. The reason is that the model will first be downloaded (using the `dtype` of the checkpoints online) and then will be casted to the default `dtype` of `torch` (becomes `torch.float32`). If there is a specified `torch_dtype`, it will be used instead.\n\n</Tip>\n\n\nTips:\n- The infilling task is supported out of the box. You should be using the `tokenizer.fill_token` where you want your input to be filled.\n- The model conversion script is the same as for the `Llama2` family:\n\nHere is a sample usage:\n\n```bash\npython src/transformers/models/llama/convert_llama_weights_to_hf.py \\\n    --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path\n```\n\nNote that executing the script requires enough CPU RAM to host the whole model in float16 precision (even if the biggest versions\ncome in several checkpoints they each contain a part of each weight of the model, so we need to load them all in RAM).\n\nAfter conversion, the model and tokenizer can be loaded via:\n\n```python\n>>> from transformers import LlamaForCausalLM, CodeLlamaTokenizer\n\n>>> tokenizer = CodeLlamaTokenizer.from_pretrained(\"meta-llama/CodeLlama-7b-hf\")\n>>> model = LlamaForCausalLM.from_pretrained(\"meta-llama/CodeLlama-7b-hf\")\n>>> PROMPT = '''def remove_non_ascii(s: str) -> str:\n...     \"\"\" <FILL_ME>\n...     return result\n... '''\n>>> input_ids = tokenizer(PROMPT, return_tensors=\"pt\")[\"input_ids\"]\n>>> generated_ids = model.generate(input_ids, max_new_tokens=128)\n\n>>> filling = tokenizer.batch_decode(generated_ids[:, input_ids.shape[1]:], skip_special_tokens = True)[0]\n>>> print(PROMPT.replace(\"<FILL_ME>\", filling))\ndef remove_non_ascii(s: str) -> str:\n    \"\"\" Remove non-ASCII characters from a string.\n<BLANKLINE>\n    Args:\n        s: The string to remove non-ASCII characters from.\n<BLANKLINE>\n    Returns:\n        The string with non-ASCII characters removed.\n    \"\"\""}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: e7b84b54-35e3-48d0-a408-ceedb14167e3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/code_llama", "title": "CodeLlama", "tokens": 2064, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e7b84b54-35e3-48d0-a408-ceedb14167e3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/code_llama\", \"title\": \"CodeLlama\", \"tokens\": 2064, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"177b6cd1-07b3-4bce-882d-b7e0afc62f62\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/code_llama\", \"title\": \"CodeLlama\", \"tokens\": 2064, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"18affe0a10ce1571f11526a8082f64dd55465a1884ca565bcc3d19dd4e0db30d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"27e50627-271f-4016-85d4-e628a016ac3a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/code_llama\", \"title\": \"CodeLlama\", \"tokens\": 2064, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5f3b11954fe9015fa8bf82004d64912aa4c46366e5285e171a0a1260b3cdd146\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"54227a1e-d836-4b2f-b1ff-74a2c6c5746d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"43b31fe80bd05c5d1117fee998a00415aa7ffac84947db0921e6be6a5d070dda\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3760, \"end_char_idx\": 6705, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "177b6cd1-07b3-4bce-882d-b7e0afc62f62", "doc_id": "177b6cd1-07b3-4bce-882d-b7e0afc62f62", "ref_doc_id": "177b6cd1-07b3-4bce-882d-b7e0afc62f62", "chroma:document": "You should be using the `tokenizer.fill_token` where you want your input to be filled.\n- The model conversion script is the same as for the `Llama2` family:\n\nHere is a sample usage:\n\n```bash\npython src/transformers/models/llama/convert_llama_weights_to_hf.py \\\n    --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path\n```\n\nNote that executing the script requires enough CPU RAM to host the whole model in float16 precision (even if the biggest versions\ncome in several checkpoints they each contain a part of each weight of the model, so we need to load them all in RAM).\n\nAfter conversion, the model and tokenizer can be loaded via:\n\n```python\n>>> from transformers import LlamaForCausalLM, CodeLlamaTokenizer\n\n>>> tokenizer = CodeLlamaTokenizer.from_pretrained(\"meta-llama/CodeLlama-7b-hf\")\n>>> model = LlamaForCausalLM.from_pretrained(\"meta-llama/CodeLlama-7b-hf\")\n>>> PROMPT = '''def remove_non_ascii(s: str) -> str:\n...     \"\"\" <FILL_ME>\n...     return result\n... '''\n>>> input_ids = tokenizer(PROMPT, return_tensors=\"pt\")[\"input_ids\"]\n>>> generated_ids = model.generate(input_ids, max_new_tokens=128)\n\n>>> filling = tokenizer.batch_decode(generated_ids[:, input_ids.shape[1]:], skip_special_tokens = True)[0]\n>>> print(PROMPT.replace(\"<FILL_ME>\", filling))\ndef remove_non_ascii(s: str) -> str:\n    \"\"\" Remove non-ASCII characters from a string.\n<BLANKLINE>\n    Args:\n        s: The string to remove non-ASCII characters from.\n<BLANKLINE>\n    Returns:\n        The string with non-ASCII characters removed.\n    \"\"\"\n    result = \"\"\n    for c in s:\n        if ord(c) < 128:\n            result += c\n    return result\n<BLANKLINE>\n```\n\nIf you only want the infilled part:\n```python\n>>> from transformers import pipeline\n>>> import torch\n\n>>> generator = pipeline(\"text-generation\",model=\"meta-llama/CodeLlama-7b-hf\",torch_dtype=torch.float16, device_map=\"auto\")\n>>> generator('def remove_non_ascii(s: str) -> str:\\n    \"\"\" <FILL_ME>\\n    return result', max_new_tokens = 128)\n[{'generated_text': 'def remove_non_ascii(s: str) -> str:\\n    \"\"\" <FILL_ME>\\n    return resultRemove non-ASCII characters from a string. \"\"\"\\n    result = \"\"\\n    for c in s:\\n        if ord(c) < 128:\\n            result += c'}]\n```\n\nUnder the hood, the tokenizer [automatically splits by `<FILL_ME>`](https://huggingface.co/docs/transformers/main/model_doc/code_llama#transformers.CodeLlamaTokenizer.fill_token) to create a formatted input string that follows [the original training pattern](https://github.com/facebookresearch/codellama/blob/cb51c14ec761370ba2e2bc351374a79265d0465e/llama/generation.py#L402). This is more robust than preparing the pattern yourself: it avoids pitfalls, such as token glueing, that are very hard to debug.  To see how much CPU and GPU memory you need for this model or others, try [this calculator](https://huggingface.co/spaces/hf-accelerate/model-memory-usage) which can help determine that value."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 54227a1e-d836-4b2f-b1ff-74a2c6c5746d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/code_llama", "title": "CodeLlama", "tokens": 2064, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"54227a1e-d836-4b2f-b1ff-74a2c6c5746d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/code_llama\", \"title\": \"CodeLlama\", \"tokens\": 2064, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"177b6cd1-07b3-4bce-882d-b7e0afc62f62\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/code_llama\", \"title\": \"CodeLlama\", \"tokens\": 2064, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"18affe0a10ce1571f11526a8082f64dd55465a1884ca565bcc3d19dd4e0db30d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e7b84b54-35e3-48d0-a408-ceedb14167e3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/code_llama\", \"title\": \"CodeLlama\", \"tokens\": 2064, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c89154f1cc60f4a2223be658921c0ff3abb012b07e802b4f3b4eddf14253c661\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5227, \"end_char_idx\": 7598, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "177b6cd1-07b3-4bce-882d-b7e0afc62f62", "doc_id": "177b6cd1-07b3-4bce-882d-b7e0afc62f62", "ref_doc_id": "177b6cd1-07b3-4bce-882d-b7e0afc62f62", "chroma:document": "<BLANKLINE>\n    Returns:\n        The string with non-ASCII characters removed.\n    \"\"\"\n    result = \"\"\n    for c in s:\n        if ord(c) < 128:\n            result += c\n    return result\n<BLANKLINE>\n```\n\nIf you only want the infilled part:\n```python\n>>> from transformers import pipeline\n>>> import torch\n\n>>> generator = pipeline(\"text-generation\",model=\"meta-llama/CodeLlama-7b-hf\",torch_dtype=torch.float16, device_map=\"auto\")\n>>> generator('def remove_non_ascii(s: str) -> str:\\n    \"\"\" <FILL_ME>\\n    return result', max_new_tokens = 128)\n[{'generated_text': 'def remove_non_ascii(s: str) -> str:\\n    \"\"\" <FILL_ME>\\n    return resultRemove non-ASCII characters from a string. \"\"\"\\n    result = \"\"\\n    for c in s:\\n        if ord(c) < 128:\\n            result += c'}]\n```\n\nUnder the hood, the tokenizer [automatically splits by `<FILL_ME>`](https://huggingface.co/docs/transformers/main/model_doc/code_llama#transformers.CodeLlamaTokenizer.fill_token) to create a formatted input string that follows [the original training pattern](https://github.com/facebookresearch/codellama/blob/cb51c14ec761370ba2e2bc351374a79265d0465e/llama/generation.py#L402). This is more robust than preparing the pattern yourself: it avoids pitfalls, such as token glueing, that are very hard to debug.  To see how much CPU and GPU memory you need for this model or others, try [this calculator](https://huggingface.co/spaces/hf-accelerate/model-memory-usage) which can help determine that value.\n\nThe LLaMA tokenizer is a BPE model based on [sentencepiece](https://github.com/google/sentencepiece). One quirk of sentencepiece is that when decoding a sequence, if the first token is the start of the word (e.g. \"Banana\"), the tokenizer does not prepend the prefix space to the string.\n\n<Tip>\n\nCode Llama has the same architecture as the `Llama2` models, refer to [Llama2's documentation page](llama2) for the API reference.\nFind Code Llama tokenizer reference below. \n</Tip>\n\n\n## CodeLlamaTokenizer\n\n[[autodoc]] CodeLlamaTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## CodeLlamaTokenizerFast\n\n[[autodoc]] CodeLlamaTokenizerFast\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - update_post_processor\n    - save_vocabulary"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 2b707b32-08ca-4d76-8157-62ab1835bf96
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llava_next", "title": "LLaVA-NeXT", "tokens": 3311, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2b707b32-08ca-4d76-8157-62ab1835bf96\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next\", \"title\": \"LLaVA-NeXT\", \"tokens\": 3311, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"128a4cf7-02bd-4aa2-a337-31cddab0f051\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next\", \"title\": \"LLaVA-NeXT\", \"tokens\": 3311, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"69c8e374fdbbc2accf670616a51b54836bbc28c029d7e57275f97890637e1a5b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"109e52d7-ad26-4597-9c24-6eb29814ae9b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8bcbcee9fda4e7ff004eda6e8109b1f9abbfae05e50e6207d170c135759082df\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3173, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "128a4cf7-02bd-4aa2-a337-31cddab0f051", "doc_id": "128a4cf7-02bd-4aa2-a337-31cddab0f051", "ref_doc_id": "128a4cf7-02bd-4aa2-a337-31cddab0f051", "chroma:document": "# LLaVA-NeXT\n\n## Overview\n\nThe LLaVA-NeXT model was proposed in [LLaVA-NeXT: Improved reasoning, OCR, and world knowledge](https://llava-vl.github.io/blog/2024-01-30-llava-next/) by Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, Yong Jae Lee. LLaVa-NeXT (also called LLaVa-1.6) improves upon [LLaVa](llava) by increasing the input image resolution and training on an improved visual instruction tuning dataset to improve OCR and common sense reasoning.\n\nThe introduction from the blog is the following:\n\n*In October 2023, we released LLaVA-1.5 with a simple and efficient design along with great performance on a benchmark suite of 12 datasets. It has since served as the foundation of many comprehensive studies of data, model, and capabilities of large multimodal models (LMM), and has enabled various new applications.\n\nToday, we are thrilled to present LLaVA-NeXT, with improved reasoning, OCR, and world knowledge. LLaVA-NeXT even exceeds Gemini Pro on several benchmarks.\n\nCompared with LLaVA-1.5, LLaVA-NeXT has several improvements:\n\nIncreasing the input image resolution to 4x more pixels. This allows it to grasp more visual details. It supports three aspect ratios, up to 672x672, 336x1344, 1344x336 resolution.\nBetter visual reasoning and OCR capability with an improved visual instruction tuning data mixture.\nBetter visual conversation for more scenarios, covering different applications. Better world knowledge and logical reasoning.\nEfficient deployment and inference with SGLang.\nAlong with performance improvements, LLaVA-NeXT maintains the minimalist design and data efficiency of LLaVA-1.5. It re-uses the pretrained connector of LLaVA-1.5, and still uses less than 1M visual instruction tuning samples. The largest 34B variant finishes training in ~1 day with 32 A100s.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/llava_next_overview.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> LLaVa-NeXT incorporates a higher input resolution by encoding various patches of the input image. Taken from the <a href=\"https://arxiv.org/abs/2310.03744\">original paper.</a> </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/haotian-liu/LLaVA/tree/main).\n\n## Usage tips\n\n- We advise users to use `padding_side=\"left\"` when computing batched generation as it leads to more accurate results. Simply make sure to call `processor.tokenizer.padding_side = \"left\"` before generating.\n\n<Tip warning={true}>\n\n- Llava-Next uses different number of patches for images and thus has to pad the inputs inside modeling code, aside from the padding done when processing the inputs. The default setting is \"left-padding\" if model is in `eval()` mode, otherwise \"right-padding\".\n\n</Tip>\n\n\n- Note that each checkpoint has been trained with a specific prompt format, depending on which large language model (LLM) was used. You can use the processor's `apply_chat_template` to format your prompts correctly. For that you have to construct a conversation history, passing a plain string will not format your prompt."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 109e52d7-ad26-4597-9c24-6eb29814ae9b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llava_next", "title": "LLaVA-NeXT", "tokens": 3311, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"109e52d7-ad26-4597-9c24-6eb29814ae9b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next\", \"title\": \"LLaVA-NeXT\", \"tokens\": 3311, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"128a4cf7-02bd-4aa2-a337-31cddab0f051\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next\", \"title\": \"LLaVA-NeXT\", \"tokens\": 3311, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"69c8e374fdbbc2accf670616a51b54836bbc28c029d7e57275f97890637e1a5b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2b707b32-08ca-4d76-8157-62ab1835bf96\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next\", \"title\": \"LLaVA-NeXT\", \"tokens\": 3311, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"efdc7ccdd4002692a8cf338e74e89c65d18ee996a941513a5f4dcf0135ed4301\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e8cf49c8-0a86-4725-b361-7a3cd6e2fab6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6bbc7058afad91efee48d0ea76ea00afef82cac144a896939aee01c1be48e142\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2851, \"end_char_idx\": 5340, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "128a4cf7-02bd-4aa2-a337-31cddab0f051", "doc_id": "128a4cf7-02bd-4aa2-a337-31cddab0f051", "ref_doc_id": "128a4cf7-02bd-4aa2-a337-31cddab0f051", "chroma:document": "- Note that each checkpoint has been trained with a specific prompt format, depending on which large language model (LLM) was used. You can use the processor's `apply_chat_template` to format your prompts correctly. For that you have to construct a conversation history, passing a plain string will not format your prompt. Each message in the conversation history for chat templates is a dictionary with keys \"role\" and \"content\". The \"content\" should be a list of dictionaries, for \"text\" and \"image\" modalities. Below is an example of how to do that and the list of formats accepted by each checkpoint.\n\nWe will use [llava-v1.6-mistral-7b-hf](https://huggingface.co/llava-hf/llava-hf/llava-v1.6-mistral-7b-hf) and a conversation history of text and image. Each content field has to be a list of dicts, as follows:\n\n```python\nfrom transformers import LlavaNextProcessor\n\nprocessor = LlavaNextProcessor.from_pretrained(\"llava-hf/llava-hf/llava-v1.6-mistral-7b-hf\")\n\nconversation = [\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image\"},\n            {\"type\": \"text\", \"text\": \"What\u2019s shown in this image?\"},\n        ],\n    },\n    {\n        \"role\": \"assistant\",\n        \"content\": [{\"type\": \"text\", \"text\": \"This image shows a red stop sign.\"},]\n    },\n    {\n\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"Describe the image in more details.\"},\n        ],\n    },\n]\n\ntext_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)\n\n# Note that the template simply formats your prompt, you still have to tokenize it and obtain pixel values for your images\nprint(text_prompt)\n>>> \"[INST] <image>\\nWhat's shown in this image? [/INST] This image shows a red stop sign. [INST] Describe the image in more details. [/INST]\"\n```\n\n- If you want to construct a chat prompt yourself, below is a list of possible formats\n.\n[llava-v1.6-mistral-7b-hf](https://huggingface.co/llava-hf/llava-v1.6-mistral-7b-hf) requires the following format:\n```bash\n\"[INST] <image>\\nWhat is shown in this image? [/INST]\"\n```\n\n[llava-v1.6-vicuna-7b-hf](https://huggingface.co/llava-hf/llava-v1.6-vicuna-7b-hf) and [llava-v1.6-vicuna-13b-hf](https://huggingface.co/llava-hf/llava-v1.6-vicuna-13b-hf) require the following format:\n```bash\n\"A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions. USER: <image>\\nWhat is shown in this image?"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: e8cf49c8-0a86-4725-b361-7a3cd6e2fab6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llava_next", "title": "LLaVA-NeXT", "tokens": 3311, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e8cf49c8-0a86-4725-b361-7a3cd6e2fab6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next\", \"title\": \"LLaVA-NeXT\", \"tokens\": 3311, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"128a4cf7-02bd-4aa2-a337-31cddab0f051\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next\", \"title\": \"LLaVA-NeXT\", \"tokens\": 3311, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"69c8e374fdbbc2accf670616a51b54836bbc28c029d7e57275f97890637e1a5b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"109e52d7-ad26-4597-9c24-6eb29814ae9b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next\", \"title\": \"LLaVA-NeXT\", \"tokens\": 3311, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a0d783ccac70bde87493ee3f8037dd5a1d32b0099ee7225ffb277032bd41e866\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e857792e-b7be-4aa1-9f84-71cc5679deaa\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2190ce1a5a0580b4142e1f1700041ba9209f48bc717868207d13d73e46269dd7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3981, \"end_char_idx\": 5826, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "128a4cf7-02bd-4aa2-a337-31cddab0f051", "doc_id": "128a4cf7-02bd-4aa2-a337-31cddab0f051", "ref_doc_id": "128a4cf7-02bd-4aa2-a337-31cddab0f051", "chroma:document": "\"},\n        ],\n    },\n    {\n        \"role\": \"assistant\",\n        \"content\": [{\"type\": \"text\", \"text\": \"This image shows a red stop sign.\"},]\n    },\n    {\n\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"Describe the image in more details.\"},\n        ],\n    },\n]\n\ntext_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)\n\n# Note that the template simply formats your prompt, you still have to tokenize it and obtain pixel values for your images\nprint(text_prompt)\n>>> \"[INST] <image>\\nWhat's shown in this image? [/INST] This image shows a red stop sign. [INST] Describe the image in more details. [/INST]\"\n```\n\n- If you want to construct a chat prompt yourself, below is a list of possible formats\n.\n[llava-v1.6-mistral-7b-hf](https://huggingface.co/llava-hf/llava-v1.6-mistral-7b-hf) requires the following format:\n```bash\n\"[INST] <image>\\nWhat is shown in this image? [/INST]\"\n```\n\n[llava-v1.6-vicuna-7b-hf](https://huggingface.co/llava-hf/llava-v1.6-vicuna-7b-hf) and [llava-v1.6-vicuna-13b-hf](https://huggingface.co/llava-hf/llava-v1.6-vicuna-13b-hf) require the following format:\n```bash\n\"A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions. USER: <image>\\nWhat is shown in this image? ASSISTANT:\"\n```\n\n[llava-v1.6-34b-hf](https://huggingface.co/llava-hf/llava-v1.6-34b-hf) requires the following format:\n```bash\n\"<|im_start|>system\\nAnswer the questions.<|im_end|><|im_start|>user\\n<image>\\nWhat is shown in this image?<|im_end|><|im_start|>assistant\\n\"\n```\n\n[llama3-llava-next-8b-hf](https://huggingface.co/llava-hf/llava-next-8b-hf) requires the following format:\n\n```bash\n\"<|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful language and vision assistant."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: e857792e-b7be-4aa1-9f84-71cc5679deaa
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llava_next", "title": "LLaVA-NeXT", "tokens": 3311, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e857792e-b7be-4aa1-9f84-71cc5679deaa\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next\", \"title\": \"LLaVA-NeXT\", \"tokens\": 3311, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"128a4cf7-02bd-4aa2-a337-31cddab0f051\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next\", \"title\": \"LLaVA-NeXT\", \"tokens\": 3311, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"69c8e374fdbbc2accf670616a51b54836bbc28c029d7e57275f97890637e1a5b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e8cf49c8-0a86-4725-b361-7a3cd6e2fab6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next\", \"title\": \"LLaVA-NeXT\", \"tokens\": 3311, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5a7e5722bc4db91794c1d55ba29298fdd7f47650c1d0d0cce6b1a220f5a671ee\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a942a5f7-2613-4517-9c58-bb55b8763ceb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"224153815aaa13977fed0a4b0237e8192d02fe2b7a912db04b49382f8c8cfbfe\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4745, \"end_char_idx\": 7433, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "128a4cf7-02bd-4aa2-a337-31cddab0f051", "doc_id": "128a4cf7-02bd-4aa2-a337-31cddab0f051", "ref_doc_id": "128a4cf7-02bd-4aa2-a337-31cddab0f051", "chroma:document": "[llava-v1.6-mistral-7b-hf](https://huggingface.co/llava-hf/llava-v1.6-mistral-7b-hf) requires the following format:\n```bash\n\"[INST] <image>\\nWhat is shown in this image? [/INST]\"\n```\n\n[llava-v1.6-vicuna-7b-hf](https://huggingface.co/llava-hf/llava-v1.6-vicuna-7b-hf) and [llava-v1.6-vicuna-13b-hf](https://huggingface.co/llava-hf/llava-v1.6-vicuna-13b-hf) require the following format:\n```bash\n\"A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions. USER: <image>\\nWhat is shown in this image? ASSISTANT:\"\n```\n\n[llava-v1.6-34b-hf](https://huggingface.co/llava-hf/llava-v1.6-34b-hf) requires the following format:\n```bash\n\"<|im_start|>system\\nAnswer the questions.<|im_end|><|im_start|>user\\n<image>\\nWhat is shown in this image?<|im_end|><|im_start|>assistant\\n\"\n```\n\n[llama3-llava-next-8b-hf](https://huggingface.co/llava-hf/llava-next-8b-hf) requires the following format:\n\n```bash\n\"<|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful language and vision assistant. You are able to understand the visual content that the user provides, and assist the user with a variety of tasks using natural language.<|eot_id|><|start_header_id|><|start_header_id|>user<|end_header_id|>\\n\\n<image>\\nWhat is shown in this image?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n\"\n```\n\n[llava-next-72b-hf](https://huggingface.co/llava-hf/llava-next-72b-hf) and [llava-next-110b-hf](https://huggingface.co/llava-hf/llava-next-110b-hf) require the following format:\n\n```bash\n\"<|im_start|>system\\nYou are a helpful assistant.<|im_end|>\\n<|im_start|>user\\n<image>\\nWhat is shown in this image?<|im_end|>\\n<|im_start|>assistant\\n\"\n```\n\n## Usage example\n\n### Single image inference\n\nHere's how to load the model and perform inference in half-precision (`torch.float16`):\n\n```python\nfrom transformers import LlavaNextProcessor, LlavaNextForConditionalGeneration\nimport torch\nfrom PIL import Image\nimport requests\n\nprocessor = LlavaNextProcessor.from_pretrained(\"llava-hf/llava-v1.6-mistral-7b-hf\")\n\nmodel = LlavaNextForConditionalGeneration.from_pretrained(\"llava-hf/llava-v1.6-mistral-7b-hf\", torch_dtype=torch.float16, low_cpu_mem_usage=True) \nmodel.to(\"cuda:0\")\n\n# prepare image and text prompt, using the appropriate prompt template\nurl = \"https://github.com/haotian-liu/LLaVA/blob/1a91fc274d7c35a9b50b3cb29c4247ae5837ce39/images/llava_v1_5_radar.jpg?raw=true\"\nimage = Image.open(requests.get(url, stream=True).raw)\n\nconversation = [\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image\"},\n            {\"type\": \"text\", \"text\": \"What is shown in this image?"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: a942a5f7-2613-4517-9c58-bb55b8763ceb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llava_next", "title": "LLaVA-NeXT", "tokens": 3311, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a942a5f7-2613-4517-9c58-bb55b8763ceb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next\", \"title\": \"LLaVA-NeXT\", \"tokens\": 3311, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"128a4cf7-02bd-4aa2-a337-31cddab0f051\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next\", \"title\": \"LLaVA-NeXT\", \"tokens\": 3311, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"69c8e374fdbbc2accf670616a51b54836bbc28c029d7e57275f97890637e1a5b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e857792e-b7be-4aa1-9f84-71cc5679deaa\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next\", \"title\": \"LLaVA-NeXT\", \"tokens\": 3311, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"328af22a2300952e2a7da2a6967b06f369c27419365b12677d29e9b7287718ec\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e77d4f39-487a-43af-8da3-680e04806146\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"be4ca95a0b44117d08a0aa9842f2c7e3c6ac292766840432bc7182507c2c2309\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7433, \"end_char_idx\": 10517, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "128a4cf7-02bd-4aa2-a337-31cddab0f051", "doc_id": "128a4cf7-02bd-4aa2-a337-31cddab0f051", "ref_doc_id": "128a4cf7-02bd-4aa2-a337-31cddab0f051", "chroma:document": "\"},\n        ],\n    },\n]\nprompt = processor.apply_chat_template(conversation, add_generation_prompt=True)\ninputs = processor(prompt, image, return_tensors=\"pt\").to(\"cuda:0\")\n\n# autoregressively complete prompt\noutput = model.generate(**inputs, max_new_tokens=100)\n\nprint(processor.decode(output[0], skip_special_tokens=True))\n```\n\n### Multi image inference\n\nLLaVa-Next can perform inference with multiple images as input, where images either belong to the same prompt or different prompts (in batched inference). Here is how you can do it:\n\n```python\nimport requests\nfrom PIL import Image\nimport torch\nfrom transformers import AutoProcessor, LlavaNextForConditionalGeneration\n\n# Load the model in half-precision\nmodel = LlavaNextForConditionalGeneration.from_pretrained(\"llava-hf/llava-v1.6-mistral-7b-hf\", torch_dtype=torch.float16, device_map=\"auto\")\nprocessor = AutoProcessor.from_pretrained(\"llava-hf/llava-v1.6-mistral-7b-hf\")\n\n# Get three different images\nurl = \"https://www.ilankelman.org/stopsigns/australia.jpg\"\nimage_stop = Image.open(requests.get(url, stream=True).raw)\n\nurl = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nimage_cats = Image.open(requests.get(url, stream=True).raw)\n\nurl = \"https://huggingface.co/microsoft/kosmos-2-patch14-224/resolve/main/snowman.jpg\"\nimage_snowman = Image.open(requests.get(url, stream=True).raw)\n\n# Prepare a batch of two prompts, where the first one is a multi-turn conversation and the second is not\nconversation_1 = [\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image\"},\n            {\"type\": \"text\", \"text\": \"What is shown in this image?\"},\n            ],\n    },\n    {\n        \"role\": \"assistant\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"There is a red stop sign in the image.\"},\n            ],\n    },\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image\"},\n            {\"type\": \"text\", \"text\": \"What about this image? How many cats do you see?\"},\n            ],\n    },\n]\n\nconversation_2 = [\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image\"},\n            {\"type\": \"text\", \"text\": \"What is shown in this image?\"},\n            ],\n    },\n]\n\nprompt_1 = processor.apply_chat_template(conversation_1, add_generation_prompt=True)\nprompt_2 = processor.apply_chat_template(conversation_2, add_generation_prompt=True)\nprompts = [prompt_1, prompt_2]\n\n# We can simply feed images in the order they have to be used in the text prompt\n# Each \"<image>\" token uses one image leaving the next for the subsequent \"<image>\" tokens\ninputs = processor(text=prompts, images=[image_stop, image_cats, image_snowman], padding=True, return_tensors=\"pt\").to(model.device)\n\n# Generate\ngenerate_ids = model.generate(**inputs, max_new_tokens=30)\nprocessor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)\n```\n\n## Model optimization\n\n### Quantization using Bitsandbytes\n\nThe model can be loaded in 8 or 4 bits, greatly reducing the memory requirements while maintaining the performance of the original model."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: e77d4f39-487a-43af-8da3-680e04806146
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llava_next", "title": "LLaVA-NeXT", "tokens": 3311, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e77d4f39-487a-43af-8da3-680e04806146\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next\", \"title\": \"LLaVA-NeXT\", \"tokens\": 3311, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"128a4cf7-02bd-4aa2-a337-31cddab0f051\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next\", \"title\": \"LLaVA-NeXT\", \"tokens\": 3311, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"69c8e374fdbbc2accf670616a51b54836bbc28c029d7e57275f97890637e1a5b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a942a5f7-2613-4517-9c58-bb55b8763ceb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next\", \"title\": \"LLaVA-NeXT\", \"tokens\": 3311, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3a2b56f0ad9618ff1882bcea8ac0c17d437c38ae14f5a1c2f4b83fc83e782b83\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9061, \"end_char_idx\": 11918, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "128a4cf7-02bd-4aa2-a337-31cddab0f051", "doc_id": "128a4cf7-02bd-4aa2-a337-31cddab0f051", "ref_doc_id": "128a4cf7-02bd-4aa2-a337-31cddab0f051", "chroma:document": "\"},\n            ],\n    },\n    {\n        \"role\": \"assistant\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"There is a red stop sign in the image.\"},\n            ],\n    },\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image\"},\n            {\"type\": \"text\", \"text\": \"What about this image? How many cats do you see?\"},\n            ],\n    },\n]\n\nconversation_2 = [\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image\"},\n            {\"type\": \"text\", \"text\": \"What is shown in this image?\"},\n            ],\n    },\n]\n\nprompt_1 = processor.apply_chat_template(conversation_1, add_generation_prompt=True)\nprompt_2 = processor.apply_chat_template(conversation_2, add_generation_prompt=True)\nprompts = [prompt_1, prompt_2]\n\n# We can simply feed images in the order they have to be used in the text prompt\n# Each \"<image>\" token uses one image leaving the next for the subsequent \"<image>\" tokens\ninputs = processor(text=prompts, images=[image_stop, image_cats, image_snowman], padding=True, return_tensors=\"pt\").to(model.device)\n\n# Generate\ngenerate_ids = model.generate(**inputs, max_new_tokens=30)\nprocessor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)\n```\n\n## Model optimization\n\n### Quantization using Bitsandbytes\n\nThe model can be loaded in 8 or 4 bits, greatly reducing the memory requirements while maintaining the performance of the original model. First make sure to install bitsandbytes, `pip install bitsandbytes` and make sure to have access to a CUDA compatible GPU device. Simply change the snippet above with:\n\n```python\nfrom transformers import LlavaNextForConditionalGeneration, BitsAndBytesConfig\n\n# specify how to quantize the model\nquantization_config = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_quant_type=\"nf4\",\n    bnb_4bit_compute_dtype=torch.float16,\n)\n\nmodel = LlavaNextForConditionalGeneration.from_pretrained(\"llava-hf/llava-v1.6-mistral-7b-hf\", quantization_config=quantization_config, device_map=\"auto\")\n```\n\n### Use Flash-Attention 2 to further speed-up generation\n\nFirst make sure to install flash-attn. Refer to the [original repository of Flash Attention](https://github.com/Dao-AILab/flash-attention) regarding that package installation. Simply change the snippet above with:\n\n```python\nfrom transformers import LlavaNextForConditionalGeneration\n\nmodel = LlavaNextForConditionalGeneration.from_pretrained(\n    model_id, \n    torch_dtype=torch.float16, \n    low_cpu_mem_usage=True,\n    use_flash_attention_2=True\n).to(0)\n```\n\n## LlavaNextConfig\n\n[[autodoc]] LlavaNextConfig\n\n## LlavaNextImageProcessor\n\n[[autodoc]] LlavaNextImageProcessor\n    - preprocess\n\n## LlavaNextProcessor\n\n[[autodoc]] LlavaNextProcessor\n\n## LlavaNextForConditionalGeneration\n\n[[autodoc]] LlavaNextForConditionalGeneration\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 684c23eb-af84-43a9-a308-097da846f807
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xglm", "title": "XGLM", "tokens": 999, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"684c23eb-af84-43a9-a308-097da846f807\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xglm\", \"title\": \"XGLM\", \"tokens\": 999, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"07a0cf35-4803-4523-9e16-51f4be809d57\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xglm\", \"title\": \"XGLM\", \"tokens\": 999, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1fafa44d10a88b34cb80261b380c21dac121df2bc2cb3a95fcb70e0b5f7bc99a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"645f5f17-951d-4213-b787-b7d871814486\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"cb48e1f95e492b18be7b03a8e5287b16e9de60040679cebea38817c967aa47a1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2118, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "07a0cf35-4803-4523-9e16-51f4be809d57", "doc_id": "07a0cf35-4803-4523-9e16-51f4be809d57", "ref_doc_id": "07a0cf35-4803-4523-9e16-51f4be809d57", "chroma:document": "# XGLM\n\n## Overview\n\nThe XGLM model was proposed in [Few-shot Learning with Multilingual Language Models](https://arxiv.org/abs/2112.10668)\nby Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, \nShruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O'Horo, \nJeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, Xian Li.\n\nThe abstract from the paper is the following:\n\n*Large-scale autoregressive language models such as GPT-3 are few-shot learners that can perform a wide range of language \ntasks without fine-tuning. While these models are known to be able to jointly represent many different languages, \ntheir training data is dominated by English, potentially limiting their cross-lingual generalization. \nIn this work, we train multilingual autoregressive language models on a balanced corpus covering a diverse set of languages, \nand study their few- and zero-shot learning capabilities in a wide range of tasks. Our largest model with 7.5 billion parameters \nsets new state of the art in few-shot learning in more than 20 representative languages, outperforming GPT-3 of comparable size \nin multilingual commonsense reasoning (with +7.4% absolute accuracy improvement in 0-shot settings and +9.4% in 4-shot settings) \nand natural language inference (+5.4% in each of 0-shot and 4-shot settings). On the FLORES-101 machine translation benchmark, \nour model outperforms GPT-3 on 171 out of 182 translation directions with 32 training examples, while surpassing the \nofficial supervised baseline in 45 directions. We present a detailed analysis of where the model succeeds and fails, \nshowing in particular that it enables cross-lingual in-context learning on some tasks, while there is still room for improvement \non surface form robustness and adaptation to tasks that do not have a natural cloze form. Finally, we evaluate our models \nin social value tasks such as hate speech detection in five languages and find it has limitations similar to comparable sized GPT-3 models.*"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 645f5f17-951d-4213-b787-b7d871814486
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xglm", "title": "XGLM", "tokens": 999, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"645f5f17-951d-4213-b787-b7d871814486\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xglm\", \"title\": \"XGLM\", \"tokens\": 999, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"07a0cf35-4803-4523-9e16-51f4be809d57\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xglm\", \"title\": \"XGLM\", \"tokens\": 999, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1fafa44d10a88b34cb80261b380c21dac121df2bc2cb3a95fcb70e0b5f7bc99a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"684c23eb-af84-43a9-a308-097da846f807\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xglm\", \"title\": \"XGLM\", \"tokens\": 999, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"51739c7628dc650501961767272cb7ab9044f6c65e6195b14ec6b73ed81f06db\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2121, \"end_char_idx\": 3085, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "07a0cf35-4803-4523-9e16-51f4be809d57", "doc_id": "07a0cf35-4803-4523-9e16-51f4be809d57", "ref_doc_id": "07a0cf35-4803-4523-9e16-51f4be809d57", "chroma:document": "This model was contributed by [Suraj](https://huggingface.co/valhalla). The original code can be found [here](https://github.com/pytorch/fairseq/tree/main/examples/xglm).\n\n## Resources\n\n- [Causal language modeling task guide](../tasks/language_modeling)\n\n## XGLMConfig\n\n[[autodoc]] XGLMConfig\n\n## XGLMTokenizer\n\n[[autodoc]] XGLMTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## XGLMTokenizerFast\n\n[[autodoc]] XGLMTokenizerFast\n\n<frameworkcontent>\n<pt>\n\n## XGLMModel\n\n[[autodoc]] XGLMModel\n    - forward\n\n## XGLMForCausalLM\n\n[[autodoc]] XGLMForCausalLM\n    - forward\n\n</pt>\n<tf>\n\n## TFXGLMModel\n\n[[autodoc]] TFXGLMModel\n    - call\n\n## TFXGLMForCausalLM\n\n[[autodoc]] TFXGLMForCausalLM\n    - call\n\n</tf>\n<jax>\n\n## FlaxXGLMModel\n\n[[autodoc]] FlaxXGLMModel\n    - __call__\n\n## FlaxXGLMForCausalLM\n\n[[autodoc]] FlaxXGLMForCausalLM\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: f6fce9f8-28dd-42c6-bb4b-63938dd723f2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f6fce9f8-28dd-42c6-bb4b-63938dd723f2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9f3577ae-d4e7-495b-ab78-e92dd8a8d367\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0101159c-45b2-444d-952f-4900e073a73b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"24ba341ce000d143f08391e307bb8f97cdc356cf1567f293166fd7b31df73d7c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2885, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "doc_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "ref_doc_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "chroma:document": "# LayoutLMV2\n\n## Overview\n\nThe LayoutLMV2 model was proposed in [LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding](https://arxiv.org/abs/2012.14740) by Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu,\nDinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, Lidong Zhou. LayoutLMV2 improves [LayoutLM](layoutlm) to obtain\nstate-of-the-art results across several document image understanding benchmarks:\n\n- information extraction from scanned documents: the [FUNSD](https://guillaumejaume.github.io/FUNSD/) dataset (a\n  collection of 199 annotated forms comprising more than 30,000 words), the [CORD](https://github.com/clovaai/cord)\n  dataset (a collection of 800 receipts for training, 100 for validation and 100 for testing), the [SROIE](https://rrc.cvc.uab.es/?ch=13) dataset (a collection of 626 receipts for training and 347 receipts for testing)\n  and the [Kleister-NDA](https://github.com/applicaai/kleister-nda) dataset (a collection of non-disclosure\n  agreements from the EDGAR database, including 254 documents for training, 83 documents for validation, and 203\n  documents for testing).\n- document image classification: the [RVL-CDIP](https://www.cs.cmu.edu/~aharley/rvl-cdip/) dataset (a collection of\n  400,000 images belonging to one of 16 classes).\n- document visual question answering: the [DocVQA](https://arxiv.org/abs/2007.00398) dataset (a collection of 50,000\n  questions defined on 12,000+ document images).\n\nThe abstract from the paper is the following:\n\n*Pre-training of text and layout has proved effective in a variety of visually-rich document understanding tasks due to\nits effective model architecture and the advantage of large-scale unlabeled scanned/digital-born documents. In this\npaper, we present LayoutLMv2 by pre-training text, layout and image in a multi-modal framework, where new model\narchitectures and pre-training tasks are leveraged. Specifically, LayoutLMv2 not only uses the existing masked\nvisual-language modeling task but also the new text-image alignment and text-image matching tasks in the pre-training\nstage, where cross-modality interaction is better learned. Meanwhile, it also integrates a spatial-aware self-attention\nmechanism into the Transformer architecture, so that the model can fully understand the relative positional\nrelationship among different text blocks. Experiment results show that LayoutLMv2 outperforms strong baselines and\nachieves new state-of-the-art results on a wide variety of downstream visually-rich document understanding tasks,\nincluding FUNSD (0.7895 -> 0.8420), CORD (0.9493 -> 0.9601), SROIE (0.9524 -> 0.9781), Kleister-NDA (0.834 -> 0.852),\nRVL-CDIP (0.9443 -> 0.9564), and DocVQA (0.7295 -> 0.8672). The pre-trained LayoutLMv2 model is publicly available at\nthis https URL.*\n\nLayoutLMv2 depends on `detectron2`, `torchvision` and `tesseract`."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 0101159c-45b2-444d-952f-4900e073a73b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0101159c-45b2-444d-952f-4900e073a73b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9f3577ae-d4e7-495b-ab78-e92dd8a8d367\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f6fce9f8-28dd-42c6-bb4b-63938dd723f2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"986397e737f6cc88422c2e044e9593faa1bb6d88f378ffb6765e9fa4dcb14f12\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9837ed20-0062-4893-9842-67aa1a942d12\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"451899999b7387093a3b39221144539623e1f6e72ead6d29bf01c3b0a6686131\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1317, \"end_char_idx\": 4294, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "doc_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "ref_doc_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "chroma:document": "- document visual question answering: the [DocVQA](https://arxiv.org/abs/2007.00398) dataset (a collection of 50,000\n  questions defined on 12,000+ document images).\n\nThe abstract from the paper is the following:\n\n*Pre-training of text and layout has proved effective in a variety of visually-rich document understanding tasks due to\nits effective model architecture and the advantage of large-scale unlabeled scanned/digital-born documents. In this\npaper, we present LayoutLMv2 by pre-training text, layout and image in a multi-modal framework, where new model\narchitectures and pre-training tasks are leveraged. Specifically, LayoutLMv2 not only uses the existing masked\nvisual-language modeling task but also the new text-image alignment and text-image matching tasks in the pre-training\nstage, where cross-modality interaction is better learned. Meanwhile, it also integrates a spatial-aware self-attention\nmechanism into the Transformer architecture, so that the model can fully understand the relative positional\nrelationship among different text blocks. Experiment results show that LayoutLMv2 outperforms strong baselines and\nachieves new state-of-the-art results on a wide variety of downstream visually-rich document understanding tasks,\nincluding FUNSD (0.7895 -> 0.8420), CORD (0.9493 -> 0.9601), SROIE (0.9524 -> 0.9781), Kleister-NDA (0.834 -> 0.852),\nRVL-CDIP (0.9443 -> 0.9564), and DocVQA (0.7295 -> 0.8672). The pre-trained LayoutLMv2 model is publicly available at\nthis https URL.*\n\nLayoutLMv2 depends on `detectron2`, `torchvision` and `tesseract`. Run the\nfollowing to install them:\n```bash\npython -m pip install 'git+https://github.com/facebookresearch/detectron2.git'\npython -m pip install torchvision tesseract\n```\n(If you are developing for LayoutLMv2, note that passing the doctests also requires the installation of these packages.)\n\n## Usage tips\n\n- The main difference between LayoutLMv1 and LayoutLMv2 is that the latter incorporates visual embeddings during\n  pre-training (while LayoutLMv1 only adds visual embeddings during fine-tuning).\n- LayoutLMv2 adds both a relative 1D attention bias as well as a spatial 2D attention bias to the attention scores in\n  the self-attention layers. Details can be found on page 5 of the [paper](https://arxiv.org/abs/2012.14740).\n- Demo notebooks on how to use the LayoutLMv2 model on RVL-CDIP, FUNSD, DocVQA, CORD can be found [here](https://github.com/NielsRogge/Transformers-Tutorials).\n- LayoutLMv2 uses Facebook AI's [Detectron2](https://github.com/facebookresearch/detectron2/) package for its visual\n  backbone. See [this link](https://detectron2.readthedocs.io/en/latest/tutorials/install.html) for installation\n  instructions.\n- In addition to `input_ids`, [`~LayoutLMv2Model.forward`] expects 2 additional inputs, namely\n  `image` and `bbox`. The `image` input corresponds to the original document image in which the text\n  tokens occur. The model expects each document image to be of size 224x224."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 9837ed20-0062-4893-9842-67aa1a942d12
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9837ed20-0062-4893-9842-67aa1a942d12\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9f3577ae-d4e7-495b-ab78-e92dd8a8d367\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0101159c-45b2-444d-952f-4900e073a73b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0e7b0c6a181e27aad01dcfb8db6cff9dc033898c2672c02024a421c15c02a834\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e8ee820b-a48d-4cf2-87b4-f94adc7eb32f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c6f8af7ec45894bf1b5fc87695831eea4dd059bf39fbfe266086db47b0d214cf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2743, \"end_char_idx\": 5351, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "doc_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "ref_doc_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "chroma:document": "The pre-trained LayoutLMv2 model is publicly available at\nthis https URL.*\n\nLayoutLMv2 depends on `detectron2`, `torchvision` and `tesseract`. Run the\nfollowing to install them:\n```bash\npython -m pip install 'git+https://github.com/facebookresearch/detectron2.git'\npython -m pip install torchvision tesseract\n```\n(If you are developing for LayoutLMv2, note that passing the doctests also requires the installation of these packages.)\n\n## Usage tips\n\n- The main difference between LayoutLMv1 and LayoutLMv2 is that the latter incorporates visual embeddings during\n  pre-training (while LayoutLMv1 only adds visual embeddings during fine-tuning).\n- LayoutLMv2 adds both a relative 1D attention bias as well as a spatial 2D attention bias to the attention scores in\n  the self-attention layers. Details can be found on page 5 of the [paper](https://arxiv.org/abs/2012.14740).\n- Demo notebooks on how to use the LayoutLMv2 model on RVL-CDIP, FUNSD, DocVQA, CORD can be found [here](https://github.com/NielsRogge/Transformers-Tutorials).\n- LayoutLMv2 uses Facebook AI's [Detectron2](https://github.com/facebookresearch/detectron2/) package for its visual\n  backbone. See [this link](https://detectron2.readthedocs.io/en/latest/tutorials/install.html) for installation\n  instructions.\n- In addition to `input_ids`, [`~LayoutLMv2Model.forward`] expects 2 additional inputs, namely\n  `image` and `bbox`. The `image` input corresponds to the original document image in which the text\n  tokens occur. The model expects each document image to be of size 224x224. This means that if you have a batch of\n  document images, `image` should be a tensor of shape (batch_size, 3, 224, 224). This can be either a\n  `torch.Tensor` or a `Detectron2.structures.ImageList`. You don't need to normalize the channels, as this is\n  done by the model. Important to note is that the visual backbone expects BGR channels instead of RGB, as all models\n  in Detectron2 are pre-trained using the BGR format. The `bbox` input are the bounding boxes (i.e. 2D-positions)\n  of the input text tokens. This is identical to [`LayoutLMModel`]. These can be obtained using an\n  external OCR engine such as Google's [Tesseract](https://github.com/tesseract-ocr/tesseract) (there's a [Python\n  wrapper](https://pypi.org/project/pytesseract/) available). Each bounding box should be in (x0, y0, x1, y1)\n  format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1)\n  represents the position of the lower right corner. Note that one first needs to normalize the bounding boxes to be on\n  a 0-1000 scale."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: e8ee820b-a48d-4cf2-87b4-f94adc7eb32f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e8ee820b-a48d-4cf2-87b4-f94adc7eb32f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9f3577ae-d4e7-495b-ab78-e92dd8a8d367\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9837ed20-0062-4893-9842-67aa1a942d12\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5a3b68a93396a806c340a6ab136f3a5bfbb4637d50028d658044368dadebfe19\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"77b78658-c871-4cdc-9347-c9f0d922bcc3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d531aa9498bf8ffc564af98eb5777ea26e6e269f5bf33527401c728f1a993fc6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3905, \"end_char_idx\": 6806, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "doc_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "ref_doc_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "chroma:document": "See [this link](https://detectron2.readthedocs.io/en/latest/tutorials/install.html) for installation\n  instructions.\n- In addition to `input_ids`, [`~LayoutLMv2Model.forward`] expects 2 additional inputs, namely\n  `image` and `bbox`. The `image` input corresponds to the original document image in which the text\n  tokens occur. The model expects each document image to be of size 224x224. This means that if you have a batch of\n  document images, `image` should be a tensor of shape (batch_size, 3, 224, 224). This can be either a\n  `torch.Tensor` or a `Detectron2.structures.ImageList`. You don't need to normalize the channels, as this is\n  done by the model. Important to note is that the visual backbone expects BGR channels instead of RGB, as all models\n  in Detectron2 are pre-trained using the BGR format. The `bbox` input are the bounding boxes (i.e. 2D-positions)\n  of the input text tokens. This is identical to [`LayoutLMModel`]. These can be obtained using an\n  external OCR engine such as Google's [Tesseract](https://github.com/tesseract-ocr/tesseract) (there's a [Python\n  wrapper](https://pypi.org/project/pytesseract/) available). Each bounding box should be in (x0, y0, x1, y1)\n  format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1)\n  represents the position of the lower right corner. Note that one first needs to normalize the bounding boxes to be on\n  a 0-1000 scale. To normalize, you can use the following function:\n\n```python\ndef normalize_bbox(bbox, width, height):\n    return [\n        int(1000 * (bbox[0] / width)),\n        int(1000 * (bbox[1] / height)),\n        int(1000 * (bbox[2] / width)),\n        int(1000 * (bbox[3] / height)),\n    ]\n```\n\nHere, `width` and `height` correspond to the width and height of the original document in which the token\noccurs (before resizing the image). Those can be obtained using the Python Image Library (PIL) library for example, as\nfollows:\n\n```python\nfrom PIL import Image\n\nimage = Image.open(\n    \"name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images).\"\n)\n\nwidth, height = image.size\n```\n\nHowever, this model includes a brand new [`~transformers.LayoutLMv2Processor`] which can be used to directly\nprepare data for the model (including applying OCR under the hood). More information can be found in the \"Usage\"\nsection below.\n\n- Internally, [`~transformers.LayoutLMv2Model`] will send the `image` input through its visual backbone to\n  obtain a lower-resolution feature map, whose shape is equal to the `image_feature_pool_shape` attribute of\n  [`~transformers.LayoutLMv2Config`]. This feature map is then flattened to obtain a sequence of image tokens. As\n  the size of the feature map is 7x7 by default, one obtains 49 image tokens. These are then concatenated with the text\n  tokens, and send through the Transformer encoder."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 77b78658-c871-4cdc-9347-c9f0d922bcc3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"77b78658-c871-4cdc-9347-c9f0d922bcc3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9f3577ae-d4e7-495b-ab78-e92dd8a8d367\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e8ee820b-a48d-4cf2-87b4-f94adc7eb32f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"caf80668ee15fa5716183094280addbfd8a45e1020911b811c3c7745bc6e4a0a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"465248cc-186c-40f2-9485-e547fe210575\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4ac22479df246c933397775b5a5d15f9422f5e3339f8c9e576915face30fbae1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5267, \"end_char_idx\": 8127, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "doc_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "ref_doc_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "chroma:document": "Note that one first needs to normalize the bounding boxes to be on\n  a 0-1000 scale. To normalize, you can use the following function:\n\n```python\ndef normalize_bbox(bbox, width, height):\n    return [\n        int(1000 * (bbox[0] / width)),\n        int(1000 * (bbox[1] / height)),\n        int(1000 * (bbox[2] / width)),\n        int(1000 * (bbox[3] / height)),\n    ]\n```\n\nHere, `width` and `height` correspond to the width and height of the original document in which the token\noccurs (before resizing the image). Those can be obtained using the Python Image Library (PIL) library for example, as\nfollows:\n\n```python\nfrom PIL import Image\n\nimage = Image.open(\n    \"name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images).\"\n)\n\nwidth, height = image.size\n```\n\nHowever, this model includes a brand new [`~transformers.LayoutLMv2Processor`] which can be used to directly\nprepare data for the model (including applying OCR under the hood). More information can be found in the \"Usage\"\nsection below.\n\n- Internally, [`~transformers.LayoutLMv2Model`] will send the `image` input through its visual backbone to\n  obtain a lower-resolution feature map, whose shape is equal to the `image_feature_pool_shape` attribute of\n  [`~transformers.LayoutLMv2Config`]. This feature map is then flattened to obtain a sequence of image tokens. As\n  the size of the feature map is 7x7 by default, one obtains 49 image tokens. These are then concatenated with the text\n  tokens, and send through the Transformer encoder. This means that the last hidden states of the model will have a\n  length of 512 + 49 = 561, if you pad the text tokens up to the max length. More generally, the last hidden states\n  will have a shape of `seq_length` + `image_feature_pool_shape[0]` *\n  `config.image_feature_pool_shape[1]`.\n- When calling [`~transformers.LayoutLMv2Model.from_pretrained`], a warning will be printed with a long list of\n  parameter names that are not initialized. This is not a problem, as these parameters are batch normalization\n  statistics, which are going to have values when fine-tuning on a custom dataset.\n- If you want to train the model in a distributed environment, make sure to call [`synchronize_batch_norm`] on the\n  model in order to properly synchronize the batch normalization layers of the visual backbone.\n\nIn addition, there's LayoutXLM, which is a multilingual version of LayoutLMv2. More information can be found on\n[LayoutXLM's documentation page](layoutxlm).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with LayoutLMv2. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 465248cc-186c-40f2-9485-e547fe210575
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"465248cc-186c-40f2-9485-e547fe210575\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9f3577ae-d4e7-495b-ab78-e92dd8a8d367\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"77b78658-c871-4cdc-9347-c9f0d922bcc3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"322fe81d5489bad6bd4e4f4f7f80622e561f49af479448f4454549e7fff133b7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d2ffece4-cbcf-4933-9663-19560fe94419\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"71bc6e21020fea58f988daed6ca08706339e1d1e6ca9b2a9c52a0100c8cae53e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6559, \"end_char_idx\": 9560, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "doc_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "ref_doc_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "chroma:document": "This feature map is then flattened to obtain a sequence of image tokens. As\n  the size of the feature map is 7x7 by default, one obtains 49 image tokens. These are then concatenated with the text\n  tokens, and send through the Transformer encoder. This means that the last hidden states of the model will have a\n  length of 512 + 49 = 561, if you pad the text tokens up to the max length. More generally, the last hidden states\n  will have a shape of `seq_length` + `image_feature_pool_shape[0]` *\n  `config.image_feature_pool_shape[1]`.\n- When calling [`~transformers.LayoutLMv2Model.from_pretrained`], a warning will be printed with a long list of\n  parameter names that are not initialized. This is not a problem, as these parameters are batch normalization\n  statistics, which are going to have values when fine-tuning on a custom dataset.\n- If you want to train the model in a distributed environment, make sure to call [`synchronize_batch_norm`] on the\n  model in order to properly synchronize the batch normalization layers of the visual backbone.\n\nIn addition, there's LayoutXLM, which is a multilingual version of LayoutLMv2. More information can be found on\n[LayoutXLM's documentation page](layoutxlm).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with LayoutLMv2. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- A notebook on how to [finetune LayoutLMv2 for text-classification on RVL-CDIP dataset](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/RVL-CDIP/Fine_tuning_LayoutLMv2ForSequenceClassification_on_RVL_CDIP.ipynb).\n- See also: [Text classification task guide](../tasks/sequence_classification)\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- A notebook on how to [finetune LayoutLMv2 for question-answering on DocVQA dataset](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/DocVQA/Fine_tuning_LayoutLMv2ForQuestionAnswering_on_DocVQA.ipynb).\n- See also: [Question answering task guide](../tasks/question_answering)\n- See also: [Document question answering task guide](../tasks/document_question_answering)\n\n\n<PipelineTag pipeline=\"token-classification\"/>\n\n- A notebook on how to [finetune LayoutLMv2 for token-classification on CORD dataset](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/CORD/Fine_tuning_LayoutLMv2ForTokenClassification_on_CORD.ipynb).\n- A notebook on how to [finetune LayoutLMv2 for token-classification on FUNSD dataset](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/FUNSD/Fine_tuning_LayoutLMv2ForTokenClassification_on_FUNSD_using_HuggingFace_Trainer.ipynb)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: d2ffece4-cbcf-4933-9663-19560fe94419
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d2ffece4-cbcf-4933-9663-19560fe94419\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9f3577ae-d4e7-495b-ab78-e92dd8a8d367\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"465248cc-186c-40f2-9485-e547fe210575\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fef0c9a3a87404af277ea75bf4a153d3e8b0944d4d7aea5231f1b172118d2515\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d3807c3d-530c-4737-982f-cda4f9288175\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"47215afe5810e740365e4e1b698be9b7c2cfee049a0a5bbad17937c92b0c6300\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8029, \"end_char_idx\": 11098, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "doc_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "ref_doc_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "chroma:document": "The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- A notebook on how to [finetune LayoutLMv2 for text-classification on RVL-CDIP dataset](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/RVL-CDIP/Fine_tuning_LayoutLMv2ForSequenceClassification_on_RVL_CDIP.ipynb).\n- See also: [Text classification task guide](../tasks/sequence_classification)\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- A notebook on how to [finetune LayoutLMv2 for question-answering on DocVQA dataset](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/DocVQA/Fine_tuning_LayoutLMv2ForQuestionAnswering_on_DocVQA.ipynb).\n- See also: [Question answering task guide](../tasks/question_answering)\n- See also: [Document question answering task guide](../tasks/document_question_answering)\n\n\n<PipelineTag pipeline=\"token-classification\"/>\n\n- A notebook on how to [finetune LayoutLMv2 for token-classification on CORD dataset](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/CORD/Fine_tuning_LayoutLMv2ForTokenClassification_on_CORD.ipynb).\n- A notebook on how to [finetune LayoutLMv2 for token-classification on FUNSD dataset](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/FUNSD/Fine_tuning_LayoutLMv2ForTokenClassification_on_FUNSD_using_HuggingFace_Trainer.ipynb).\n- See also: [Token classification task guide](../tasks/token_classification)\n\n## Usage: LayoutLMv2Processor\n\nThe easiest way to prepare data for the model is to use [`LayoutLMv2Processor`], which internally\ncombines a image processor ([`LayoutLMv2ImageProcessor`]) and a tokenizer\n([`LayoutLMv2Tokenizer`] or [`LayoutLMv2TokenizerFast`]). The image processor\nhandles the image modality, while the tokenizer handles the text modality. A processor combines both, which is ideal\nfor a multi-modal model like LayoutLMv2. Note that you can still use both separately, if you only want to handle one\nmodality.\n\n```python\nfrom transformers import LayoutLMv2ImageProcessor, LayoutLMv2TokenizerFast, LayoutLMv2Processor\n\nimage_processor = LayoutLMv2ImageProcessor()  # apply_ocr is set to True by default\ntokenizer = LayoutLMv2TokenizerFast.from_pretrained(\"microsoft/layoutlmv2-base-uncased\")\nprocessor = LayoutLMv2Processor(image_processor, tokenizer)\n```\n\nIn short, one can provide a document image (and possibly additional data) to [`LayoutLMv2Processor`],\nand it will create the inputs expected by the model. Internally, the processor first uses\n[`LayoutLMv2ImageProcessor`] to apply OCR on the image to get a list of words and normalized\nbounding boxes, as well to resize the image to a given size in order to get the `image` input. The words and\nnormalized bounding boxes are then provided to [`LayoutLMv2Tokenizer`] or\n[`LayoutLMv2TokenizerFast`], which converts them to token-level `input_ids`,\n`attention_mask`, `token_type_ids`, `bbox`."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: d3807c3d-530c-4737-982f-cda4f9288175
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d3807c3d-530c-4737-982f-cda4f9288175\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9f3577ae-d4e7-495b-ab78-e92dd8a8d367\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d2ffece4-cbcf-4933-9663-19560fe94419\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a8336422e5e4a2cf5763c5fb7d40f4b1b8a9b4e87f71d84addf6d6c5240abe2b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6cd4c025-11d5-4b3d-9444-b519a04a1e25\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"83d4b2cb5c4e4058b228cfa676b68d244f8ced8c4f34f64052425d42a0381d5b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9561, \"end_char_idx\": 12350, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "doc_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "ref_doc_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "chroma:document": "- See also: [Token classification task guide](../tasks/token_classification)\n\n## Usage: LayoutLMv2Processor\n\nThe easiest way to prepare data for the model is to use [`LayoutLMv2Processor`], which internally\ncombines a image processor ([`LayoutLMv2ImageProcessor`]) and a tokenizer\n([`LayoutLMv2Tokenizer`] or [`LayoutLMv2TokenizerFast`]). The image processor\nhandles the image modality, while the tokenizer handles the text modality. A processor combines both, which is ideal\nfor a multi-modal model like LayoutLMv2. Note that you can still use both separately, if you only want to handle one\nmodality.\n\n```python\nfrom transformers import LayoutLMv2ImageProcessor, LayoutLMv2TokenizerFast, LayoutLMv2Processor\n\nimage_processor = LayoutLMv2ImageProcessor()  # apply_ocr is set to True by default\ntokenizer = LayoutLMv2TokenizerFast.from_pretrained(\"microsoft/layoutlmv2-base-uncased\")\nprocessor = LayoutLMv2Processor(image_processor, tokenizer)\n```\n\nIn short, one can provide a document image (and possibly additional data) to [`LayoutLMv2Processor`],\nand it will create the inputs expected by the model. Internally, the processor first uses\n[`LayoutLMv2ImageProcessor`] to apply OCR on the image to get a list of words and normalized\nbounding boxes, as well to resize the image to a given size in order to get the `image` input. The words and\nnormalized bounding boxes are then provided to [`LayoutLMv2Tokenizer`] or\n[`LayoutLMv2TokenizerFast`], which converts them to token-level `input_ids`,\n`attention_mask`, `token_type_ids`, `bbox`. Optionally, one can provide word labels to the processor,\nwhich are turned into token-level `labels`.\n\n[`LayoutLMv2Processor`] uses [PyTesseract](https://pypi.org/project/pytesseract/), a Python\nwrapper around Google's Tesseract OCR engine, under the hood. Note that you can still use your own OCR engine of\nchoice, and provide the words and normalized boxes yourself. This requires initializing\n[`LayoutLMv2ImageProcessor`] with `apply_ocr` set to `False`.\n\nIn total, there are 5 use cases that are supported by the processor. Below, we list them all. Note that each of these\nuse cases work for both batched and non-batched inputs (we illustrate them for non-batched inputs).\n\n**Use case 1: document image classification (training, inference) + token classification (inference), apply_ocr =\nTrue**\n\nThis is the simplest case, in which the processor (actually the image processor) will perform OCR on the image to get\nthe words and normalized bounding boxes.\n\n```python\nfrom transformers import LayoutLMv2Processor\nfrom PIL import Image\n\nprocessor = LayoutLMv2Processor.from_pretrained(\"microsoft/layoutlmv2-base-uncased\")\n\nimage = Image.open(\n    \"name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images).\""}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 6cd4c025-11d5-4b3d-9444-b519a04a1e25
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6cd4c025-11d5-4b3d-9444-b519a04a1e25\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9f3577ae-d4e7-495b-ab78-e92dd8a8d367\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d3807c3d-530c-4737-982f-cda4f9288175\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b789baa80406ed679ad91e913a574e1b14661598ecedbcfe0f19bf85de6cd4c1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0dbc518a-b8f1-4882-af16-a279fd8d3d8a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9cc199a8fd3a99d1b946c4e20a2cae2f1bbe18e070ec31e62e4470b77e5eff0e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10890, \"end_char_idx\": 14026, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "doc_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "ref_doc_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "chroma:document": "The words and\nnormalized bounding boxes are then provided to [`LayoutLMv2Tokenizer`] or\n[`LayoutLMv2TokenizerFast`], which converts them to token-level `input_ids`,\n`attention_mask`, `token_type_ids`, `bbox`. Optionally, one can provide word labels to the processor,\nwhich are turned into token-level `labels`.\n\n[`LayoutLMv2Processor`] uses [PyTesseract](https://pypi.org/project/pytesseract/), a Python\nwrapper around Google's Tesseract OCR engine, under the hood. Note that you can still use your own OCR engine of\nchoice, and provide the words and normalized boxes yourself. This requires initializing\n[`LayoutLMv2ImageProcessor`] with `apply_ocr` set to `False`.\n\nIn total, there are 5 use cases that are supported by the processor. Below, we list them all. Note that each of these\nuse cases work for both batched and non-batched inputs (we illustrate them for non-batched inputs).\n\n**Use case 1: document image classification (training, inference) + token classification (inference), apply_ocr =\nTrue**\n\nThis is the simplest case, in which the processor (actually the image processor) will perform OCR on the image to get\nthe words and normalized bounding boxes.\n\n```python\nfrom transformers import LayoutLMv2Processor\nfrom PIL import Image\n\nprocessor = LayoutLMv2Processor.from_pretrained(\"microsoft/layoutlmv2-base-uncased\")\n\nimage = Image.open(\n    \"name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images).\"\n).convert(\"RGB\")\nencoding = processor(\n    image, return_tensors=\"pt\"\n)  # you can also add all tokenizer parameters here such as padding, truncation\nprint(encoding.keys())\n# dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'bbox', 'image'])\n```\n\n**Use case 2: document image classification (training, inference) + token classification (inference), apply_ocr=False**\n\nIn case one wants to do OCR themselves, one can initialize the image processor with `apply_ocr` set to\n`False`. In that case, one should provide the words and corresponding (normalized) bounding boxes themselves to\nthe processor.\n\n```python\nfrom transformers import LayoutLMv2Processor\nfrom PIL import Image\n\nprocessor = LayoutLMv2Processor.from_pretrained(\"microsoft/layoutlmv2-base-uncased\", revision=\"no_ocr\")\n\nimage = Image.open(\n    \"name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images).\"\n).convert(\"RGB\")\nwords = [\"hello\", \"world\"]\nboxes = [[1, 2, 3, 4], [5, 6, 7, 8]]  # make sure to normalize your bounding boxes\nencoding = processor(image, words, boxes=boxes, return_tensors=\"pt\")\nprint(encoding.keys())\n# dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'bbox', 'image'])\n```\n\n**Use case 3: token classification (training), apply_ocr=False**\n\nFor token classification tasks (such as FUNSD, CORD, SROIE, Kleister-NDA), one can also provide the corresponding word\nlabels in order to train a model. The processor will then convert these into token-level `labels`. By default, it\nwill only label the first wordpiece of a word, and label the remaining wordpieces with -100, which is the\n`ignore_index` of PyTorch's CrossEntropyLoss."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 0dbc518a-b8f1-4882-af16-a279fd8d3d8a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0dbc518a-b8f1-4882-af16-a279fd8d3d8a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9f3577ae-d4e7-495b-ab78-e92dd8a8d367\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6cd4c025-11d5-4b3d-9444-b519a04a1e25\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bbc052b043393f9c8d1fc74795cdf6be4921202006b19c7635b7b38e953d7dec\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3f71cdd6-527d-4760-b22b-082fc2a28185\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bffa87630189c671fd705e6073f4ea42c511f3deb80a825f0e5982ffae7547a1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12842, \"end_char_idx\": 15797, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "doc_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "ref_doc_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "chroma:document": "In that case, one should provide the words and corresponding (normalized) bounding boxes themselves to\nthe processor.\n\n```python\nfrom transformers import LayoutLMv2Processor\nfrom PIL import Image\n\nprocessor = LayoutLMv2Processor.from_pretrained(\"microsoft/layoutlmv2-base-uncased\", revision=\"no_ocr\")\n\nimage = Image.open(\n    \"name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images).\"\n).convert(\"RGB\")\nwords = [\"hello\", \"world\"]\nboxes = [[1, 2, 3, 4], [5, 6, 7, 8]]  # make sure to normalize your bounding boxes\nencoding = processor(image, words, boxes=boxes, return_tensors=\"pt\")\nprint(encoding.keys())\n# dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'bbox', 'image'])\n```\n\n**Use case 3: token classification (training), apply_ocr=False**\n\nFor token classification tasks (such as FUNSD, CORD, SROIE, Kleister-NDA), one can also provide the corresponding word\nlabels in order to train a model. The processor will then convert these into token-level `labels`. By default, it\nwill only label the first wordpiece of a word, and label the remaining wordpieces with -100, which is the\n`ignore_index` of PyTorch's CrossEntropyLoss. In case you want all wordpieces of a word to be labeled, you can\ninitialize the tokenizer with `only_label_first_subword` set to `False`.\n\n```python\nfrom transformers import LayoutLMv2Processor\nfrom PIL import Image\n\nprocessor = LayoutLMv2Processor.from_pretrained(\"microsoft/layoutlmv2-base-uncased\", revision=\"no_ocr\")\n\nimage = Image.open(\n    \"name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images).\"\n).convert(\"RGB\")\nwords = [\"hello\", \"world\"]\nboxes = [[1, 2, 3, 4], [5, 6, 7, 8]]  # make sure to normalize your bounding boxes\nword_labels = [1, 2]\nencoding = processor(image, words, boxes=boxes, word_labels=word_labels, return_tensors=\"pt\")\nprint(encoding.keys())\n# dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'bbox', 'labels', 'image'])\n```\n\n**Use case 4: visual question answering (inference), apply_ocr=True**\n\nFor visual question answering tasks (such as DocVQA), you can provide a question to the processor. By default, the\nprocessor will apply OCR on the image, and create [CLS] question tokens [SEP] word tokens [SEP].\n\n```python\nfrom transformers import LayoutLMv2Processor\nfrom PIL import Image\n\nprocessor = LayoutLMv2Processor.from_pretrained(\"microsoft/layoutlmv2-base-uncased\")\n\nimage = Image.open(\n    \"name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images).\"\n).convert(\"RGB\")\nquestion = \"What's his name?\"\nencoding = processor(image, question, return_tensors=\"pt\")\nprint(encoding.keys())\n# dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'bbox', 'image'])\n```\n\n**Use case 5: visual question answering (inference), apply_ocr=False**\n\nFor visual question answering tasks (such as DocVQA), you can provide a question to the processor."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 3f71cdd6-527d-4760-b22b-082fc2a28185
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3f71cdd6-527d-4760-b22b-082fc2a28185\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9f3577ae-d4e7-495b-ab78-e92dd8a8d367\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0dbc518a-b8f1-4882-af16-a279fd8d3d8a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ab6e96adc85940729724dacff944092812a585b1b972646ac100aa60637b12f0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bed224ac-4cbe-4dd3-8027-edff0ccda011\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"cbc8471dd3a1451e287a7687a0ea2e0e1546e1f9d982554a815907b38e30e044\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14422, \"end_char_idx\": 16274, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "doc_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "ref_doc_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "chroma:document": "of your documents (PDFs must be converted to images).\"\n).convert(\"RGB\")\nwords = [\"hello\", \"world\"]\nboxes = [[1, 2, 3, 4], [5, 6, 7, 8]]  # make sure to normalize your bounding boxes\nword_labels = [1, 2]\nencoding = processor(image, words, boxes=boxes, word_labels=word_labels, return_tensors=\"pt\")\nprint(encoding.keys())\n# dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'bbox', 'labels', 'image'])\n```\n\n**Use case 4: visual question answering (inference), apply_ocr=True**\n\nFor visual question answering tasks (such as DocVQA), you can provide a question to the processor. By default, the\nprocessor will apply OCR on the image, and create [CLS] question tokens [SEP] word tokens [SEP].\n\n```python\nfrom transformers import LayoutLMv2Processor\nfrom PIL import Image\n\nprocessor = LayoutLMv2Processor.from_pretrained(\"microsoft/layoutlmv2-base-uncased\")\n\nimage = Image.open(\n    \"name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images).\"\n).convert(\"RGB\")\nquestion = \"What's his name?\"\nencoding = processor(image, question, return_tensors=\"pt\")\nprint(encoding.keys())\n# dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'bbox', 'image'])\n```\n\n**Use case 5: visual question answering (inference), apply_ocr=False**\n\nFor visual question answering tasks (such as DocVQA), you can provide a question to the processor. If you want to\nperform OCR yourself, you can provide your own words and (normalized) bounding boxes to the processor.\n\n```python\nfrom transformers import LayoutLMv2Processor\nfrom PIL import Image\n\nprocessor = LayoutLMv2Processor.from_pretrained(\"microsoft/layoutlmv2-base-uncased\", revision=\"no_ocr\")\n\nimage = Image.open(\n    \"name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images).\"\n).convert(\"RGB\")\nquestion = \"What's his name?\""}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: bed224ac-4cbe-4dd3-8027-edff0ccda011
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"bed224ac-4cbe-4dd3-8027-edff0ccda011\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9f3577ae-d4e7-495b-ab78-e92dd8a8d367\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3f71cdd6-527d-4760-b22b-082fc2a28185\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f06ac39622f1608d01d0986dde382db55af518362654ba06f4e661af7e9e8bcd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15007, \"end_char_idx\": 17351, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "doc_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "ref_doc_id": "9f3577ae-d4e7-495b-ab78-e92dd8a8d367", "chroma:document": "By default, the\nprocessor will apply OCR on the image, and create [CLS] question tokens [SEP] word tokens [SEP].\n\n```python\nfrom transformers import LayoutLMv2Processor\nfrom PIL import Image\n\nprocessor = LayoutLMv2Processor.from_pretrained(\"microsoft/layoutlmv2-base-uncased\")\n\nimage = Image.open(\n    \"name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images).\"\n).convert(\"RGB\")\nquestion = \"What's his name?\"\nencoding = processor(image, question, return_tensors=\"pt\")\nprint(encoding.keys())\n# dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'bbox', 'image'])\n```\n\n**Use case 5: visual question answering (inference), apply_ocr=False**\n\nFor visual question answering tasks (such as DocVQA), you can provide a question to the processor. If you want to\nperform OCR yourself, you can provide your own words and (normalized) bounding boxes to the processor.\n\n```python\nfrom transformers import LayoutLMv2Processor\nfrom PIL import Image\n\nprocessor = LayoutLMv2Processor.from_pretrained(\"microsoft/layoutlmv2-base-uncased\", revision=\"no_ocr\")\n\nimage = Image.open(\n    \"name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images).\"\n).convert(\"RGB\")\nquestion = \"What's his name?\"\nwords = [\"hello\", \"world\"]\nboxes = [[1, 2, 3, 4], [5, 6, 7, 8]]  # make sure to normalize your bounding boxes\nencoding = processor(image, question, words, boxes=boxes, return_tensors=\"pt\")\nprint(encoding.keys())\n# dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'bbox', 'image'])\n```\n\n## LayoutLMv2Config\n\n[[autodoc]] LayoutLMv2Config\n\n## LayoutLMv2FeatureExtractor\n\n[[autodoc]] LayoutLMv2FeatureExtractor\n    - __call__\n\n## LayoutLMv2ImageProcessor\n\n[[autodoc]] LayoutLMv2ImageProcessor\n    - preprocess\n\n## LayoutLMv2Tokenizer\n\n[[autodoc]] LayoutLMv2Tokenizer\n    - __call__\n    - save_vocabulary\n\n## LayoutLMv2TokenizerFast\n\n[[autodoc]] LayoutLMv2TokenizerFast\n    - __call__\n\n## LayoutLMv2Processor\n\n[[autodoc]] LayoutLMv2Processor\n    - __call__\n\n## LayoutLMv2Model\n\n[[autodoc]] LayoutLMv2Model\n    - forward\n\n## LayoutLMv2ForSequenceClassification\n\n[[autodoc]] LayoutLMv2ForSequenceClassification\n\n## LayoutLMv2ForTokenClassification\n\n[[autodoc]] LayoutLMv2ForTokenClassification\n\n## LayoutLMv2ForQuestionAnswering\n\n[[autodoc]] LayoutLMv2ForQuestionAnswering"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 9f91ffc6-d54e-4eba-bfda-9ad82e01e766
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/retribert", "title": "RetriBERT", "tokens": 462, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9f91ffc6-d54e-4eba-bfda-9ad82e01e766\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/retribert\", \"title\": \"RetriBERT\", \"tokens\": 462, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2758c8d5-86f7-4f8a-9e73-66b1bbecfe3c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/retribert\", \"title\": \"RetriBERT\", \"tokens\": 462, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e22becdb82c278b17d0311781b3b1504689ce983bc1016c950ea6ac26e3db3e1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1127, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2758c8d5-86f7-4f8a-9e73-66b1bbecfe3c", "doc_id": "2758c8d5-86f7-4f8a-9e73-66b1bbecfe3c", "ref_doc_id": "2758c8d5-86f7-4f8a-9e73-66b1bbecfe3c", "chroma:document": "# RetriBERT\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, so we won't accept any new PRs changing its code.\n\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.30.0.\nYou can do so by running the following command: `pip install -U transformers==4.30.0`.\n\n</Tip>\n\n## Overview\n\nThe RetriBERT model was proposed in the blog post [Explain Anything Like I'm Five: A Model for Open Domain Long Form\nQuestion Answering](https://yjernite.github.io/lfqa.html). RetriBERT is a small model that uses either a single or\npair of BERT encoders with lower-dimension projection for dense semantic indexing of text.\n\nThis model was contributed by [yjernite](https://huggingface.co/yjernite). Code to train and use the model can be\nfound [here](https://github.com/huggingface/transformers/tree/main/examples/research-projects/distillation).\n\n\n## RetriBertConfig\n\n[[autodoc]] RetriBertConfig\n\n## RetriBertTokenizer\n\n[[autodoc]] RetriBertTokenizer\n\n## RetriBertTokenizerFast\n\n[[autodoc]] RetriBertTokenizerFast\n\n## RetriBertModel\n\n[[autodoc]] RetriBertModel\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 6bc289a7-48b5-4a5b-9ce9-0c12dce09ca7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bark", "title": "Bark", "tokens": 2279, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6bc289a7-48b5-4a5b-9ce9-0c12dce09ca7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bark\", \"title\": \"Bark\", \"tokens\": 2279, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"507d5465-a99b-44e2-96ae-bcb43867c7e9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bark\", \"title\": \"Bark\", \"tokens\": 2279, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e023131db50dd78a4993c78bf26a853014d36071eb90afe90b5db0b4ad24cf1d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d79c81ea-e870-40fa-92a0-ab44147bede6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fb46d642bf545c8182efb842fe820fc6c157f1937f480a046cbd2d6774292712\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3230, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "507d5465-a99b-44e2-96ae-bcb43867c7e9", "doc_id": "507d5465-a99b-44e2-96ae-bcb43867c7e9", "ref_doc_id": "507d5465-a99b-44e2-96ae-bcb43867c7e9", "chroma:document": "# Bark\n\n## Overview\n\nBark is a transformer-based text-to-speech model proposed by Suno AI in [suno-ai/bark](https://github.com/suno-ai/bark).\n\nBark is made of 4 main models:\n\n- [`BarkSemanticModel`] (also referred to as the 'text' model): a causal auto-regressive transformer model that takes as input tokenized text, and predicts semantic text tokens that capture the meaning of the text.\n- [`BarkCoarseModel`] (also referred to as the 'coarse acoustics' model): a causal autoregressive transformer, that takes as input the results of the [`BarkSemanticModel`] model. It aims at predicting the first two audio codebooks necessary for EnCodec.\n- [`BarkFineModel`] (the 'fine acoustics' model), this time a non-causal autoencoder transformer, which iteratively predicts the last codebooks based on the sum of the previous codebooks embeddings.\n- having predicted all the codebook channels from the [`EncodecModel`], Bark uses it to decode the output audio array.\n\nIt should be noted that each of the first three modules can support conditional speaker embeddings to condition the output sound according to specific predefined voice.\n\nThis model was contributed by [Yoach Lacombe (ylacombe)](https://huggingface.co/ylacombe) and [Sanchit Gandhi (sanchit-gandhi)](https://github.com/sanchit-gandhi).\nThe original code can be found [here](https://github.com/suno-ai/bark).\n\n### Optimizing Bark\n\nBark can be optimized with just a few extra lines of code, which **significantly reduces its memory footprint** and **accelerates inference**.\n\n#### Using half-precision\n\nYou can speed up inference and reduce memory footprint by 50% simply by loading the model in half-precision.\n\n```python\nfrom transformers import BarkModel\nimport torch\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nmodel = BarkModel.from_pretrained(\"suno/bark-small\", torch_dtype=torch.float16).to(device)\n```\n\n#### Using CPU offload\n\nAs mentioned above, Bark is made up of 4 sub-models, which are called up sequentially during audio generation. In other words, while one sub-model is in use, the other sub-models are idle.\n\nIf you're using a CUDA device, a simple solution to benefit from an 80% reduction in memory footprint is to offload the submodels from GPU to CPU when they're idle. This operation is called *CPU offloading*. You can use it with one line of code as follows:\n\n```python\nmodel.enable_cpu_offload()\n```\n\nNote that \ud83e\udd17 Accelerate must be installed before using this feature. [Here's how to install it.](https://huggingface.co/docs/accelerate/basic_tutorials/install)\n\n#### Using Better Transformer\n\nBetter Transformer is an \ud83e\udd17 Optimum feature that performs kernel fusion under the hood. You can gain 20% to 30% in speed with zero performance degradation. It only requires one line of code to export the model to \ud83e\udd17 Better Transformer:\n\n```python\nmodel =  model.to_bettertransformer()\n```\n\nNote that \ud83e\udd17 Optimum must be installed before using this feature. [Here's how to install it.](https://huggingface.co/docs/optimum/installation)\n\n#### Using Flash Attention 2\n\nFlash Attention 2 is an even faster, optimized version of the previous optimization.\n\n##### Installation \n\nFirst, check whether your hardware is compatible with Flash Attention 2."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: d79c81ea-e870-40fa-92a0-ab44147bede6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bark", "title": "Bark", "tokens": 2279, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d79c81ea-e870-40fa-92a0-ab44147bede6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bark\", \"title\": \"Bark\", \"tokens\": 2279, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"507d5465-a99b-44e2-96ae-bcb43867c7e9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bark\", \"title\": \"Bark\", \"tokens\": 2279, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e023131db50dd78a4993c78bf26a853014d36071eb90afe90b5db0b4ad24cf1d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6bc289a7-48b5-4a5b-9ce9-0c12dce09ca7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bark\", \"title\": \"Bark\", \"tokens\": 2279, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5c4e160636eb0fdaa9b1c6aea0b758e554e4672797e511c71a9c485f7759856f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8705fd33-7256-41c4-9d4e-b7a1110b6106\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4a2a8f083b7c71018ca31ca093bf9928cf9f957002a454970c96f09db8c09131\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1535, \"end_char_idx\": 4457, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "507d5465-a99b-44e2-96ae-bcb43867c7e9", "doc_id": "507d5465-a99b-44e2-96ae-bcb43867c7e9", "ref_doc_id": "507d5465-a99b-44e2-96ae-bcb43867c7e9", "chroma:document": "#### Using half-precision\n\nYou can speed up inference and reduce memory footprint by 50% simply by loading the model in half-precision.\n\n```python\nfrom transformers import BarkModel\nimport torch\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nmodel = BarkModel.from_pretrained(\"suno/bark-small\", torch_dtype=torch.float16).to(device)\n```\n\n#### Using CPU offload\n\nAs mentioned above, Bark is made up of 4 sub-models, which are called up sequentially during audio generation. In other words, while one sub-model is in use, the other sub-models are idle.\n\nIf you're using a CUDA device, a simple solution to benefit from an 80% reduction in memory footprint is to offload the submodels from GPU to CPU when they're idle. This operation is called *CPU offloading*. You can use it with one line of code as follows:\n\n```python\nmodel.enable_cpu_offload()\n```\n\nNote that \ud83e\udd17 Accelerate must be installed before using this feature. [Here's how to install it.](https://huggingface.co/docs/accelerate/basic_tutorials/install)\n\n#### Using Better Transformer\n\nBetter Transformer is an \ud83e\udd17 Optimum feature that performs kernel fusion under the hood. You can gain 20% to 30% in speed with zero performance degradation. It only requires one line of code to export the model to \ud83e\udd17 Better Transformer:\n\n```python\nmodel =  model.to_bettertransformer()\n```\n\nNote that \ud83e\udd17 Optimum must be installed before using this feature. [Here's how to install it.](https://huggingface.co/docs/optimum/installation)\n\n#### Using Flash Attention 2\n\nFlash Attention 2 is an even faster, optimized version of the previous optimization.\n\n##### Installation \n\nFirst, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features). If your hardware is not compatible with Flash Attention 2, you can still benefit from attention kernel optimisations through Better Transformer support covered [above](https://huggingface.co/docs/transformers/main/en/model_doc/bark#using-better-transformer).\n\nNext, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\n\n##### Usage\n\nTo load a model using Flash Attention 2, we can pass the `attn_implementation=\"flash_attention_2\"` flag to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). We'll also load the model in half-precision (e.g. `torch.float16`), since it results in almost no degradation to audio quality but significantly lower memory usage and faster inference:\n\n```python\nmodel = BarkModel.from_pretrained(\"suno/bark-small\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\").to(device)\n```\n\n##### Performance comparison"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 8705fd33-7256-41c4-9d4e-b7a1110b6106
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bark", "title": "Bark", "tokens": 2279, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8705fd33-7256-41c4-9d4e-b7a1110b6106\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bark\", \"title\": \"Bark\", \"tokens\": 2279, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"507d5465-a99b-44e2-96ae-bcb43867c7e9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bark\", \"title\": \"Bark\", \"tokens\": 2279, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e023131db50dd78a4993c78bf26a853014d36071eb90afe90b5db0b4ad24cf1d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d79c81ea-e870-40fa-92a0-ab44147bede6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bark\", \"title\": \"Bark\", \"tokens\": 2279, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9496910fd08e3947854fa3dd7c18cc72887c0a9ab34beada93f1307fd34f5f21\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"26ebf375-f3cd-4d89-aa46-3e8ca0ac37e9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e47fc7a09883719e5c019374ab407b628b1bcd859cddda56c88f20f673fccbd5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2744, \"end_char_idx\": 5495, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "507d5465-a99b-44e2-96ae-bcb43867c7e9", "doc_id": "507d5465-a99b-44e2-96ae-bcb43867c7e9", "ref_doc_id": "507d5465-a99b-44e2-96ae-bcb43867c7e9", "chroma:document": "It only requires one line of code to export the model to \ud83e\udd17 Better Transformer:\n\n```python\nmodel =  model.to_bettertransformer()\n```\n\nNote that \ud83e\udd17 Optimum must be installed before using this feature. [Here's how to install it.](https://huggingface.co/docs/optimum/installation)\n\n#### Using Flash Attention 2\n\nFlash Attention 2 is an even faster, optimized version of the previous optimization.\n\n##### Installation \n\nFirst, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features). If your hardware is not compatible with Flash Attention 2, you can still benefit from attention kernel optimisations through Better Transformer support covered [above](https://huggingface.co/docs/transformers/main/en/model_doc/bark#using-better-transformer).\n\nNext, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\n\n##### Usage\n\nTo load a model using Flash Attention 2, we can pass the `attn_implementation=\"flash_attention_2\"` flag to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). We'll also load the model in half-precision (e.g. `torch.float16`), since it results in almost no degradation to audio quality but significantly lower memory usage and faster inference:\n\n```python\nmodel = BarkModel.from_pretrained(\"suno/bark-small\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\").to(device)\n```\n\n##### Performance comparison\n\n\nThe following diagram shows the latency for the native attention implementation (no optimisation) against Better Transformer and Flash Attention 2. In all cases, we generate 400 semantic tokens on a 40GB A100 GPU with PyTorch 2.1. Flash Attention 2 is also consistently faster than Better Transformer, and its performance improves even more as batch sizes increase:\n\n<div style=\"text-align: center\">\n<img src=\"https://huggingface.co/datasets/ylacombe/benchmark-comparison/resolve/main/Bark%20Optimization%20Benchmark.png\">\n</div>\n\nTo put this into perspective, on an NVIDIA A100 and when generating 400 semantic tokens with a batch size of 16, you can get 17 times the [throughput](https://huggingface.co/blog/optimizing-bark#throughput) and still be 2 seconds faster than generating sentences one by one with the native model implementation. In other words, all the samples will be generated 17 times faster.\n\nAt batch size 8, on an NVIDIA A100, Flash Attention 2 is also 10% faster than Better Transformer, and at batch size 16, 25%."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 26ebf375-f3cd-4d89-aa46-3e8ca0ac37e9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bark", "title": "Bark", "tokens": 2279, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"26ebf375-f3cd-4d89-aa46-3e8ca0ac37e9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bark\", \"title\": \"Bark\", \"tokens\": 2279, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"507d5465-a99b-44e2-96ae-bcb43867c7e9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bark\", \"title\": \"Bark\", \"tokens\": 2279, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e023131db50dd78a4993c78bf26a853014d36071eb90afe90b5db0b4ad24cf1d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8705fd33-7256-41c4-9d4e-b7a1110b6106\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bark\", \"title\": \"Bark\", \"tokens\": 2279, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2cb92c43baff8cdd69ec61a16e32c25a6d1b7cbb4227a60e4f8e814b1895688d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ad725c88-5f42-49a1-96b5-1220e2228aaa\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3ec107b5b570974fe4a806e8de33ce2b70d0cea59aa5d31c6e08ca3cfcba2ef4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3839, \"end_char_idx\": 7658, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "507d5465-a99b-44e2-96ae-bcb43867c7e9", "doc_id": "507d5465-a99b-44e2-96ae-bcb43867c7e9", "ref_doc_id": "507d5465-a99b-44e2-96ae-bcb43867c7e9", "chroma:document": "##### Usage\n\nTo load a model using Flash Attention 2, we can pass the `attn_implementation=\"flash_attention_2\"` flag to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). We'll also load the model in half-precision (e.g. `torch.float16`), since it results in almost no degradation to audio quality but significantly lower memory usage and faster inference:\n\n```python\nmodel = BarkModel.from_pretrained(\"suno/bark-small\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\").to(device)\n```\n\n##### Performance comparison\n\n\nThe following diagram shows the latency for the native attention implementation (no optimisation) against Better Transformer and Flash Attention 2. In all cases, we generate 400 semantic tokens on a 40GB A100 GPU with PyTorch 2.1. Flash Attention 2 is also consistently faster than Better Transformer, and its performance improves even more as batch sizes increase:\n\n<div style=\"text-align: center\">\n<img src=\"https://huggingface.co/datasets/ylacombe/benchmark-comparison/resolve/main/Bark%20Optimization%20Benchmark.png\">\n</div>\n\nTo put this into perspective, on an NVIDIA A100 and when generating 400 semantic tokens with a batch size of 16, you can get 17 times the [throughput](https://huggingface.co/blog/optimizing-bark#throughput) and still be 2 seconds faster than generating sentences one by one with the native model implementation. In other words, all the samples will be generated 17 times faster.\n\nAt batch size 8, on an NVIDIA A100, Flash Attention 2 is also 10% faster than Better Transformer, and at batch size 16, 25%.\n\n\n#### Combining optimization techniques\n\nYou can combine optimization techniques, and use CPU offload, half-precision and Flash Attention 2 (or \ud83e\udd17 Better Transformer) all at once.\n\n```python\nfrom transformers import BarkModel\nimport torch\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n\n# load in fp16 and use Flash Attention 2\nmodel = BarkModel.from_pretrained(\"suno/bark-small\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\").to(device)\n\n# enable CPU offload\nmodel.enable_cpu_offload()\n```\n\nFind out more on inference optimization techniques [here](https://huggingface.co/docs/transformers/perf_infer_gpu_one).\n\n### Usage tips\n\nSuno offers a library of voice presets in a number of languages [here](https://suno-ai.notion.site/8b8e8749ed514b0cbf3f699013548683?v=bc67cff786b04b50b3ceb756fd05f68c).\nThese presets are also uploaded in the hub [here](https://huggingface.co/suno/bark-small/tree/main/speaker_embeddings) or [here](https://huggingface.co/suno/bark/tree/main/speaker_embeddings).\n\n```python\n>>> from transformers import AutoProcessor, BarkModel\n\n>>> processor = AutoProcessor.from_pretrained(\"suno/bark\")\n>>> model = BarkModel.from_pretrained(\"suno/bark\")\n\n>>> voice_preset = \"v2/en_speaker_6\"\n\n>>> inputs = processor(\"Hello, my dog is cute\", voice_preset=voice_preset)\n\n>>> audio_array = model.generate(**inputs)\n>>> audio_array = audio_array.cpu().numpy().squeeze()\n```\n\nBark can generate highly realistic, **multilingual** speech as well as other audio - including music, background noise and simple sound effects. \n\n```python\n>>> # Multilingual speech - simplified Chinese\n>>> inputs = processor(\"\u60ca\u4eba\u7684\uff01\u6211\u4f1a\u8bf4\u4e2d\u6587\")\n\n>>> # Multilingual speech - French - let's use a voice_preset as well\n>>> inputs = processor(\"Incroyable! Je peux g\u00e9n\u00e9rer du son.\", voice_preset=\"fr_speaker_5\")\n\n>>> # Bark can also generate music. You can help it out by adding music notes around your lyrics.\n>>> inputs = processor(\"\u266a Hello, my dog is cute \u266a\")\n\n>>> audio_array = model.generate(**inputs)\n>>> audio_array = audio_array.cpu().numpy().squeeze()\n```\n\nThe model can also produce **nonverbal communications** like laughing, sighing and crying."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: ad725c88-5f42-49a1-96b5-1220e2228aaa
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bark", "title": "Bark", "tokens": 2279, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ad725c88-5f42-49a1-96b5-1220e2228aaa\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bark\", \"title\": \"Bark\", \"tokens\": 2279, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"507d5465-a99b-44e2-96ae-bcb43867c7e9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bark\", \"title\": \"Bark\", \"tokens\": 2279, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e023131db50dd78a4993c78bf26a853014d36071eb90afe90b5db0b4ad24cf1d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"26ebf375-f3cd-4d89-aa46-3e8ca0ac37e9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bark\", \"title\": \"Bark\", \"tokens\": 2279, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"817762c7b856ebfc08f2649840eedecbe664779483f7169b62e348e54dd74a3e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7661, \"end_char_idx\": 8890, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "507d5465-a99b-44e2-96ae-bcb43867c7e9", "doc_id": "507d5465-a99b-44e2-96ae-bcb43867c7e9", "ref_doc_id": "507d5465-a99b-44e2-96ae-bcb43867c7e9", "chroma:document": "```python\n>>> # Adding non-speech cues to the input text\n>>> inputs = processor(\"Hello uh ... [clears throat], my dog is cute [laughter]\")\n\n>>> audio_array = model.generate(**inputs)\n>>> audio_array = audio_array.cpu().numpy().squeeze()\n```\n\nTo save the audio, simply take the sample rate from the model config and some scipy utility:\n\n```python\n>>> from scipy.io.wavfile import write as write_wav\n\n>>> # save audio to disk, but first take the sample rate from the model config\n>>> sample_rate = model.generation_config.sample_rate\n>>> write_wav(\"bark_generation.wav\", sample_rate, audio_array)\n```\n\n## BarkConfig\n\n[[autodoc]] BarkConfig\n    - all\n\n## BarkProcessor\n\n[[autodoc]] BarkProcessor\n    - all\n    - __call__\n\n## BarkModel\n\n[[autodoc]] BarkModel\n    - generate\n    - enable_cpu_offload\n\n## BarkSemanticModel\n\n[[autodoc]] BarkSemanticModel\n    - forward\n\n## BarkCoarseModel\n\n[[autodoc]] BarkCoarseModel\n    - forward\n\n## BarkFineModel\n\n[[autodoc]] BarkFineModel\n    - forward\n\n## BarkCausalModel\n\n[[autodoc]] BarkCausalModel\n    - forward\n\n## BarkCoarseConfig\n\n[[autodoc]] BarkCoarseConfig\n    - all\n\n## BarkFineConfig\n\n[[autodoc]] BarkFineConfig\n    - all\n\n## BarkSemanticConfig\n\n[[autodoc]] BarkSemanticConfig\n    - all"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 161636b8-0ce3-4d09-b0f7-6bd51ebd3f80
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/ernie_m", "title": "ErnieM", "tokens": 970, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"161636b8-0ce3-4d09-b0f7-6bd51ebd3f80\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/ernie_m\", \"title\": \"ErnieM\", \"tokens\": 970, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f2121659-7a86-4e10-baa5-11378a876c4f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/ernie_m\", \"title\": \"ErnieM\", \"tokens\": 970, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c132dc5980022b1c3c3eb0cccd19de9d7dd59841cb2e25ce19d9b0a3e0bce6e4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"47f6ed7e-bd13-428f-a5e9-d6f95e9893d1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"623b37112bb4f8c2164042414adf8ad56ea8be22598a1cabb170134a86e6ac2c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3267, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f2121659-7a86-4e10-baa5-11378a876c4f", "doc_id": "f2121659-7a86-4e10-baa5-11378a876c4f", "ref_doc_id": "f2121659-7a86-4e10-baa5-11378a876c4f", "chroma:document": "# ErnieM\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we don't accept any new PRs changing its code.\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.\nYou can do so by running the following command: `pip install -U transformers==4.40.2`.\n\n</Tip>\n\n## Overview\n\nThe ErnieM model was proposed in [ERNIE-M: Enhanced Multilingual Representation by Aligning\nCross-lingual Semantics with Monolingual Corpora](https://arxiv.org/abs/2012.15674)  by Xuan Ouyang, Shuohuan Wang, Chao Pang, Yu Sun,\nHao Tian, Hua Wu, Haifeng Wang.\n\nThe abstract from the paper is the following:\n\n*Recent studies have demonstrated that pre-trained cross-lingual models achieve impressive performance in downstream cross-lingual tasks. This improvement benefits from learning a large amount of monolingual and parallel corpora. Although it is generally acknowledged that parallel corpora are critical for improving the model performance, existing methods are often constrained by the size of parallel corpora, especially for lowresource languages. In this paper, we propose ERNIE-M, a new training method that encourages the model to align the representation of multiple languages with monolingual corpora, to overcome the constraint that the parallel corpus size places on the model performance. Our key insight is to integrate back-translation into the pre-training process. We generate pseudo-parallel sentence pairs on a monolingual corpus to enable the learning of semantic alignments between different languages, thereby enhancing the semantic modeling of cross-lingual models. Experimental results show that ERNIE-M outperforms existing cross-lingual models and delivers new state-of-the-art results in various cross-lingual downstream tasks.*\nThis model was contributed by [Susnato Dhar](https://huggingface.co/susnato). The original code can be found [here](https://github.com/PaddlePaddle/PaddleNLP/tree/develop/paddlenlp/transformers/ernie_m).\n\n\n## Usage tips\n\n- Ernie-M is a BERT-like model so it is a stacked Transformer Encoder.\n- Instead of using MaskedLM for pretraining (like BERT) the authors used two novel techniques: `Cross-attention Masked Language Modeling` and `Back-translation Masked Language Modeling`. For now these two LMHead objectives are not implemented here.\n- It is a multilingual language model.\n- Next Sentence Prediction was not used in pretraining process.\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## ErnieMConfig\n\n[[autodoc]] ErnieMConfig\n\n\n## ErnieMTokenizer\n\n[[autodoc]] ErnieMTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n\n## ErnieMModel\n\n[[autodoc]] ErnieMModel\n    - forward\n\n## ErnieMForSequenceClassification\n\n[[autodoc]] ErnieMForSequenceClassification\n    - forward\n\n\n## ErnieMForMultipleChoice\n\n[[autodoc]] ErnieMForMultipleChoice\n    - forward\n\n\n## ErnieMForTokenClassification\n\n[[autodoc]] ErnieMForTokenClassification\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 47f6ed7e-bd13-428f-a5e9-d6f95e9893d1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/ernie_m", "title": "ErnieM", "tokens": 970, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"47f6ed7e-bd13-428f-a5e9-d6f95e9893d1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/ernie_m\", \"title\": \"ErnieM\", \"tokens\": 970, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f2121659-7a86-4e10-baa5-11378a876c4f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/ernie_m\", \"title\": \"ErnieM\", \"tokens\": 970, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c132dc5980022b1c3c3eb0cccd19de9d7dd59841cb2e25ce19d9b0a3e0bce6e4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"161636b8-0ce3-4d09-b0f7-6bd51ebd3f80\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/ernie_m\", \"title\": \"ErnieM\", \"tokens\": 970, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4993d7f8dc72344bbba71ed3a904493d30cd2e7c1af5d98bd8997e45f73c3814\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2011, \"end_char_idx\": 3446, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f2121659-7a86-4e10-baa5-11378a876c4f", "doc_id": "f2121659-7a86-4e10-baa5-11378a876c4f", "ref_doc_id": "f2121659-7a86-4e10-baa5-11378a876c4f", "chroma:document": "## Usage tips\n\n- Ernie-M is a BERT-like model so it is a stacked Transformer Encoder.\n- Instead of using MaskedLM for pretraining (like BERT) the authors used two novel techniques: `Cross-attention Masked Language Modeling` and `Back-translation Masked Language Modeling`. For now these two LMHead objectives are not implemented here.\n- It is a multilingual language model.\n- Next Sentence Prediction was not used in pretraining process.\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## ErnieMConfig\n\n[[autodoc]] ErnieMConfig\n\n\n## ErnieMTokenizer\n\n[[autodoc]] ErnieMTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n\n## ErnieMModel\n\n[[autodoc]] ErnieMModel\n    - forward\n\n## ErnieMForSequenceClassification\n\n[[autodoc]] ErnieMForSequenceClassification\n    - forward\n\n\n## ErnieMForMultipleChoice\n\n[[autodoc]] ErnieMForMultipleChoice\n    - forward\n\n\n## ErnieMForTokenClassification\n\n[[autodoc]] ErnieMForTokenClassification\n    - forward\n\n\n## ErnieMForQuestionAnswering\n\n[[autodoc]] ErnieMForQuestionAnswering\n    - forward\n\n## ErnieMForInformationExtraction\n\n[[autodoc]] ErnieMForInformationExtraction\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 43fb70ef-956c-4718-a5f5-19f924ca2346
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/seggpt", "title": "SegGPT", "tokens": 1012, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"43fb70ef-956c-4718-a5f5-19f924ca2346\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seggpt\", \"title\": \"SegGPT\", \"tokens\": 1012, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1ba74670-246d-4ca8-85ab-3fea6acd1b2a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seggpt\", \"title\": \"SegGPT\", \"tokens\": 1012, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9710c0628b0c252fe9fb1c339e373efe622fa35a4dc4f0ef0decb79e1c84ea4a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4d1e3235-aa04-43ba-9e67-50e47b25833a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"050bdc5091aae5f3ec9e79e94af8e9a1e3d00366402b636df996b49b515483db\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3280, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1ba74670-246d-4ca8-85ab-3fea6acd1b2a", "doc_id": "1ba74670-246d-4ca8-85ab-3fea6acd1b2a", "ref_doc_id": "1ba74670-246d-4ca8-85ab-3fea6acd1b2a", "chroma:document": "# SegGPT\n\n## Overview\n\nThe SegGPT model was proposed in [SegGPT: Segmenting Everything In Context](https://arxiv.org/abs/2304.03284) by Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, Tiejun Huang. SegGPT employs a decoder-only Transformer that can generate a segmentation mask given an input image, a prompt image and its corresponding prompt mask. The model achieves remarkable one-shot results with 56.1 mIoU on COCO-20 and 85.6 mIoU on FSS-1000.\n\nThe abstract from the paper is the following:\n\n*We present SegGPT, a generalist model for segmenting everything in context. We unify various segmentation tasks into a generalist in-context learning framework that accommodates different kinds of segmentation data by transforming them into the same format of images. The training of SegGPT is formulated as an in-context coloring problem with random color mapping for each data sample. The objective is to accomplish diverse tasks according to the context, rather than relying on specific colors. After training, SegGPT can perform arbitrary segmentation tasks in images or videos via in-context inference, such as object instance, stuff, part, contour, and text. SegGPT is evaluated on a broad range of tasks, including few-shot semantic segmentation, video object segmentation, semantic segmentation, and panoptic segmentation. Our results show strong capabilities in segmenting in-domain and out-of*\n\nTips:\n- One can use [`SegGptImageProcessor`] to prepare image input, prompt and mask to the model.\n- One can either use segmentation maps or RGB images as prompt masks. If using the latter make sure to set `do_convert_rgb=False` in the `preprocess` method.\n- It's highly advisable to pass `num_labels` when using `segmetantion_maps` (not considering background) during preprocessing and postprocessing with [`SegGptImageProcessor`] for your use case.\n- When doing inference with [`SegGptForImageSegmentation`] if your `batch_size` is greater than 1 you can use feature ensemble across your images by passing `feature_ensemble=True` in the forward method.\n\nHere's how to use the model for one-shot semantic segmentation:\n\n```python\nimport torch\nfrom datasets import load_dataset\nfrom transformers import SegGptImageProcessor, SegGptForImageSegmentation\n\ncheckpoint = \"BAAI/seggpt-vit-large\"\nimage_processor = SegGptImageProcessor.from_pretrained(checkpoint)\nmodel = SegGptForImageSegmentation.from_pretrained(checkpoint)\n\ndataset_id = \"EduardoPacheco/FoodSeg103\"\nds = load_dataset(dataset_id, split=\"train\")\n# Number of labels in FoodSeg103 (not including background)\nnum_labels = 103\n\nimage_input = ds[4][\"image\"]\nground_truth = ds[4][\"label\"]\nimage_prompt = ds[29][\"image\"]\nmask_prompt = ds[29][\"label\"]\n\ninputs = image_processor(\n    images=image_input, \n    prompt_images=image_prompt,\n    segmentation_maps=mask_prompt, \n    num_labels=num_labels,\n    return_tensors=\"pt\"\n)\n\nwith torch.no_grad():\n    outputs = model(**inputs)\n\ntarget_sizes = [image_input.size[::-1]]\nmask = image_processor.post_process_semantic_segmentation(outputs, target_sizes, num_labels=num_labels)[0]\n```\n\nThis model was contributed by [EduardoPacheco](https://huggingface.co/EduardoPacheco).\nThe original code can be found [here]([(https://github.com/baaivision/Painter/tree/main))."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 4d1e3235-aa04-43ba-9e67-50e47b25833a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/seggpt", "title": "SegGPT", "tokens": 1012, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4d1e3235-aa04-43ba-9e67-50e47b25833a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seggpt\", \"title\": \"SegGPT\", \"tokens\": 1012, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1ba74670-246d-4ca8-85ab-3fea6acd1b2a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seggpt\", \"title\": \"SegGPT\", \"tokens\": 1012, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9710c0628b0c252fe9fb1c339e373efe622fa35a4dc4f0ef0decb79e1c84ea4a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"43fb70ef-956c-4718-a5f5-19f924ca2346\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seggpt\", \"title\": \"SegGPT\", \"tokens\": 1012, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"88329ac691b57817d5a4fe064260a73ce0f3e8b487c5e6907c4f2f2db099b812\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3283, \"end_char_idx\": 3581, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1ba74670-246d-4ca8-85ab-3fea6acd1b2a", "doc_id": "1ba74670-246d-4ca8-85ab-3fea6acd1b2a", "ref_doc_id": "1ba74670-246d-4ca8-85ab-3fea6acd1b2a", "chroma:document": "## SegGptConfig\n\n[[autodoc]] SegGptConfig\n\n## SegGptImageProcessor\n\n[[autodoc]] SegGptImageProcessor\n    - preprocess\n    - post_process_semantic_segmentation\n\n## SegGptModel\n\n[[autodoc]] SegGptModel\n    - forward\n\n## SegGptForImageSegmentation\n\n[[autodoc]] SegGptForImageSegmentation\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 3fd5cd17-89cb-4967-bde4-2456734991d7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/fastspeech2_conformer", "title": "FastSpeech2Conformer", "tokens": 1686, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3fd5cd17-89cb-4967-bde4-2456734991d7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fastspeech2_conformer\", \"title\": \"FastSpeech2Conformer\", \"tokens\": 1686, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"09df89d4-6263-44fe-a252-feaea1361212\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fastspeech2_conformer\", \"title\": \"FastSpeech2Conformer\", \"tokens\": 1686, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"def2b861944e0ca72b775d1986a210926bf59970ec9000f1fa994b750ad6d4aa\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"30ef40cc-825f-4c0e-8f5b-407dfa27a28d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1102cfabdfdb4a0bb54c0fe097f018c19a4477b104572e2c3f9197f8e9765d7e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2641, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "09df89d4-6263-44fe-a252-feaea1361212", "doc_id": "09df89d4-6263-44fe-a252-feaea1361212", "ref_doc_id": "09df89d4-6263-44fe-a252-feaea1361212", "chroma:document": "# FastSpeech2Conformer\n\n## Overview\n\nThe FastSpeech2Conformer model was proposed with the paper [Recent Developments On Espnet Toolkit Boosted By Conformer](https://arxiv.org/abs/2010.13956) by Pengcheng Guo, Florian Boyer, Xuankai Chang, Tomoki Hayashi, Yosuke Higuchi, Hirofumi Inaguma, Naoyuki Kamo, Chenda Li, Daniel Garcia-Romero, Jiatong Shi, Jing Shi, Shinji Watanabe, Kun Wei, Wangyou Zhang, and Yuekai Zhang.\n\nThe abstract from the original FastSpeech2 paper is the following:\n\n*Non-autoregressive text to speech (TTS) models such as FastSpeech (Ren et al., 2019) can synthesize speech significantly faster than previous autoregressive models with comparable quality. The training of FastSpeech model relies on an autoregressive teacher model for duration prediction (to provide more information as input) and knowledge distillation (to simplify the data distribution in output), which can ease the one-to-many mapping problem (i.e., multiple speech variations correspond to the same text) in TTS. However, FastSpeech has several disadvantages: 1) the teacher-student distillation pipeline is complicated and time-consuming, 2) the duration extracted from the teacher model is not accurate enough, and the target mel-spectrograms distilled from teacher model suffer from information loss due to data simplification, both of which limit the voice quality. In this paper, we propose FastSpeech 2, which addresses the issues in FastSpeech and better solves the one-to-many mapping problem in TTS by 1) directly training the model with ground-truth target instead of the simplified output from teacher, and 2) introducing more variation information of speech (e.g., pitch, energy and more accurate duration) as conditional inputs. Specifically, we extract duration, pitch and energy from speech waveform and directly take them as conditional inputs in training and use predicted values in inference. We further design FastSpeech 2s, which is the first attempt to directly generate speech waveform from text in parallel, enjoying the benefit of fully end-to-end inference. Experimental results show that 1) FastSpeech 2 achieves a 3x training speed-up over FastSpeech, and FastSpeech 2s enjoys even faster inference speed; 2) FastSpeech 2 and 2s outperform FastSpeech in voice quality, and FastSpeech 2 can even surpass autoregressive models. Audio samples are available at https://speechresearch.github.io/fastspeech2/.*\n\nThis model was contributed by [Connor Henderson](https://huggingface.co/connor-henderson). The original code can be found [here](https://github.com/espnet/espnet/blob/master/espnet2/tts/fastspeech2/fastspeech2.py)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 30ef40cc-825f-4c0e-8f5b-407dfa27a28d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/fastspeech2_conformer", "title": "FastSpeech2Conformer", "tokens": 1686, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"30ef40cc-825f-4c0e-8f5b-407dfa27a28d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fastspeech2_conformer\", \"title\": \"FastSpeech2Conformer\", \"tokens\": 1686, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"09df89d4-6263-44fe-a252-feaea1361212\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fastspeech2_conformer\", \"title\": \"FastSpeech2Conformer\", \"tokens\": 1686, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"def2b861944e0ca72b775d1986a210926bf59970ec9000f1fa994b750ad6d4aa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3fd5cd17-89cb-4967-bde4-2456734991d7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fastspeech2_conformer\", \"title\": \"FastSpeech2Conformer\", \"tokens\": 1686, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e13746a38df065a33fb502fdbe82b50b75495a593e1121047d344275b85e5680\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"43ef7647-e91b-482a-aa3b-5fbfe0113e7b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b54c25d46b697cf2fef25d9b174c94ae3029eff0ee79f865f65a64ece432c581\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2644, \"end_char_idx\": 5598, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "09df89d4-6263-44fe-a252-feaea1361212", "doc_id": "09df89d4-6263-44fe-a252-feaea1361212", "ref_doc_id": "09df89d4-6263-44fe-a252-feaea1361212", "chroma:document": "## \ud83e\udd17 Model Architecture\nFastSpeech2's general structure with a Mel-spectrogram decoder was implemented, and the traditional transformer blocks were replaced with conformer blocks as done in the ESPnet library.\n\n#### FastSpeech2 Model Architecture\n![FastSpeech2 Model Architecture](https://www.microsoft.com/en-us/research/uploads/prod/2021/04/fastspeech2-1.png)\n\n#### Conformer Blocks\n![Conformer Blocks](https://www.researchgate.net/profile/Hirofumi-Inaguma-2/publication/344911155/figure/fig2/AS:951455406108673@1603856054097/An-overview-of-Conformer-block.png)\n\n#### Convolution Module\n![Convolution Module](https://d3i71xaburhd42.cloudfront.net/8809d0732f6147d4ad9218c8f9b20227c837a746/2-Figure1-1.png)\n\n## \ud83e\udd17 Transformers Usage\n\nYou can run FastSpeech2Conformer locally with the \ud83e\udd17 Transformers library.\n\n1. First install the \ud83e\udd17 [Transformers library](https://github.com/huggingface/transformers), g2p-en:\n\n```bash\npip install --upgrade pip\npip install --upgrade transformers g2p-en\n```\n\n2. Run inference via the Transformers modelling code with the model and hifigan separately\n\n```python\n\nfrom transformers import FastSpeech2ConformerTokenizer, FastSpeech2ConformerModel, FastSpeech2ConformerHifiGan\nimport soundfile as sf\n\ntokenizer = FastSpeech2ConformerTokenizer.from_pretrained(\"espnet/fastspeech2_conformer\")\ninputs = tokenizer(\"Hello, my dog is cute.\", return_tensors=\"pt\")\ninput_ids = inputs[\"input_ids\"]\n\nmodel = FastSpeech2ConformerModel.from_pretrained(\"espnet/fastspeech2_conformer\")\noutput_dict = model(input_ids, return_dict=True)\nspectrogram = output_dict[\"spectrogram\"]\n\nhifigan = FastSpeech2ConformerHifiGan.from_pretrained(\"espnet/fastspeech2_conformer_hifigan\")\nwaveform = hifigan(spectrogram)\n\nsf.write(\"speech.wav\", waveform.squeeze().detach().numpy(), samplerate=22050)\n```\n\n3. Run inference via the Transformers modelling code with the model and hifigan combined\n\n```python\nfrom transformers import FastSpeech2ConformerTokenizer, FastSpeech2ConformerWithHifiGan\nimport soundfile as sf\n\ntokenizer = FastSpeech2ConformerTokenizer.from_pretrained(\"espnet/fastspeech2_conformer\")\ninputs = tokenizer(\"Hello, my dog is cute.\", return_tensors=\"pt\")\ninput_ids = inputs[\"input_ids\"]\n\nmodel = FastSpeech2ConformerWithHifiGan.from_pretrained(\"espnet/fastspeech2_conformer_with_hifigan\")\noutput_dict = model(input_ids, return_dict=True)\nwaveform = output_dict[\"waveform\"]\n\nsf.write(\"speech.wav\", waveform.squeeze().detach().numpy(), samplerate=22050)\n```\n\n4. Run inference with a pipeline and specify which vocoder to use\n```python\nfrom transformers import pipeline, FastSpeech2ConformerHifiGan\nimport soundfile as sf\n\nvocoder = FastSpeech2ConformerHifiGan.from_pretrained(\"espnet/fastspeech2_conformer_hifigan\")\nsynthesiser = pipeline(model=\"espnet/fastspeech2_conformer\", vocoder=vocoder)\n\nspeech = synthesiser(\"Hello, my dog is cooler than you!\")\n\nsf.write(\"speech.wav\", speech[\"audio\"].squeeze(), samplerate=speech[\"sampling_rate\"])\n```"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 43ef7647-e91b-482a-aa3b-5fbfe0113e7b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/fastspeech2_conformer", "title": "FastSpeech2Conformer", "tokens": 1686, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"43ef7647-e91b-482a-aa3b-5fbfe0113e7b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fastspeech2_conformer\", \"title\": \"FastSpeech2Conformer\", \"tokens\": 1686, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"09df89d4-6263-44fe-a252-feaea1361212\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fastspeech2_conformer\", \"title\": \"FastSpeech2Conformer\", \"tokens\": 1686, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"def2b861944e0ca72b775d1986a210926bf59970ec9000f1fa994b750ad6d4aa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"30ef40cc-825f-4c0e-8f5b-407dfa27a28d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fastspeech2_conformer\", \"title\": \"FastSpeech2Conformer\", \"tokens\": 1686, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0cac8dbac6f52a98e686d55399f44368669e071cba661a3dd9c10cbe5e66f344\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5601, \"end_char_idx\": 6259, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "09df89d4-6263-44fe-a252-feaea1361212", "doc_id": "09df89d4-6263-44fe-a252-feaea1361212", "ref_doc_id": "09df89d4-6263-44fe-a252-feaea1361212", "chroma:document": "## FastSpeech2ConformerConfig\n\n[[autodoc]] FastSpeech2ConformerConfig\n\n## FastSpeech2ConformerHifiGanConfig\n\n[[autodoc]] FastSpeech2ConformerHifiGanConfig\n\n## FastSpeech2ConformerWithHifiGanConfig\n\n[[autodoc]] FastSpeech2ConformerWithHifiGanConfig\n\n## FastSpeech2ConformerTokenizer\n\n[[autodoc]] FastSpeech2ConformerTokenizer\n    - __call__\n    - save_vocabulary\n    - decode\n    - batch_decode\n\n## FastSpeech2ConformerModel\n\n[[autodoc]] FastSpeech2ConformerModel\n    - forward\n\n## FastSpeech2ConformerHifiGan\n\n[[autodoc]] FastSpeech2ConformerHifiGan\n    - forward\n\n## FastSpeech2ConformerWithHifiGan\n\n[[autodoc]] FastSpeech2ConformerWithHifiGan\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 9975e969-2f5d-4e70-84b3-6288d648e16b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xclip", "title": "X-CLIP", "tokens": 1005, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9975e969-2f5d-4e70-84b3-6288d648e16b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xclip\", \"title\": \"X-CLIP\", \"tokens\": 1005, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d7778733-7d4f-404f-a53b-bcb9607f96a1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xclip\", \"title\": \"X-CLIP\", \"tokens\": 1005, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"40a004e7c40d19e666beb2724c740fe98f61a3bdcd676171eff04e4a0d65d538\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c1057d5c-8b9b-46c1-aee4-07cafd53d2c7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a9ca49fb8a63ba71ce65d371bfd3ec931e89e75c573411baa4ba7bc3d15088ef\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3076, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d7778733-7d4f-404f-a53b-bcb9607f96a1", "doc_id": "d7778733-7d4f-404f-a53b-bcb9607f96a1", "ref_doc_id": "d7778733-7d4f-404f-a53b-bcb9607f96a1", "chroma:document": "# X-CLIP\n\n## Overview\n\nThe X-CLIP model was proposed in [Expanding Language-Image Pretrained Models for General Video Recognition](https://arxiv.org/abs/2208.02816) by Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, Haibin Ling.\nX-CLIP is a minimal extension of [CLIP](clip) for video. The model consists of a text encoder, a cross-frame vision encoder, a multi-frame integration Transformer, and a video-specific prompt generator.\n\nThe abstract from the paper is the following:\n\n*Contrastive language-image pretraining has shown great success in learning visual-textual joint representation from web-scale data, demonstrating remarkable \"zero-shot\" generalization ability for various image tasks. However, how to effectively expand such new language-image pretraining methods to video domains is still an open problem. In this work, we present a simple yet effective approach that adapts the pretrained language-image models to video recognition directly, instead of pretraining a new model from scratch. More concretely, to capture the long-range dependencies of frames along the temporal dimension, we propose a cross-frame attention mechanism that explicitly exchanges information across frames. Such module is lightweight and can be plugged into pretrained language-image models seamlessly. Moreover, we propose a video-specific prompting scheme, which leverages video content information for generating discriminative textual prompts. Extensive experiments demonstrate that our approach is effective and can be generalized to different video recognition scenarios. In particular, under fully-supervised settings, our approach achieves a top-1 accuracy of 87.1% on Kinectics-400, while using 12 times fewer FLOPs compared with Swin-L and ViViT-H. In zero-shot experiments, our approach surpasses the current state-of-the-art methods by +7.6% and +14.9% in terms of top-1 accuracy under two popular protocols. In few-shot scenarios, our approach outperforms previous best methods by +32.1% and +23.1% when the labeled data is extremely limited.*\n\nTips:\n\n- Usage of X-CLIP is identical to [CLIP](clip).\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/xclip_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> X-CLIP architecture. Taken from the <a href=\"https://arxiv.org/abs/2208.02816\">original paper.</a> </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/microsoft/VideoX/tree/master/X-CLIP).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with X-CLIP.\n\n- Demo notebooks for X-CLIP can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/X-CLIP).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: c1057d5c-8b9b-46c1-aee4-07cafd53d2c7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xclip", "title": "X-CLIP", "tokens": 1005, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c1057d5c-8b9b-46c1-aee4-07cafd53d2c7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xclip\", \"title\": \"X-CLIP\", \"tokens\": 1005, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d7778733-7d4f-404f-a53b-bcb9607f96a1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xclip\", \"title\": \"X-CLIP\", \"tokens\": 1005, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"40a004e7c40d19e666beb2724c740fe98f61a3bdcd676171eff04e4a0d65d538\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9975e969-2f5d-4e70-84b3-6288d648e16b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xclip\", \"title\": \"X-CLIP\", \"tokens\": 1005, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fc17b184a9f7f5dccb3f49c342e4d3265c05df0faa5a5bad7142aa4822097b80\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1488, \"end_char_idx\": 3525, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d7778733-7d4f-404f-a53b-bcb9607f96a1", "doc_id": "d7778733-7d4f-404f-a53b-bcb9607f96a1", "ref_doc_id": "d7778733-7d4f-404f-a53b-bcb9607f96a1", "chroma:document": "Extensive experiments demonstrate that our approach is effective and can be generalized to different video recognition scenarios. In particular, under fully-supervised settings, our approach achieves a top-1 accuracy of 87.1% on Kinectics-400, while using 12 times fewer FLOPs compared with Swin-L and ViViT-H. In zero-shot experiments, our approach surpasses the current state-of-the-art methods by +7.6% and +14.9% in terms of top-1 accuracy under two popular protocols. In few-shot scenarios, our approach outperforms previous best methods by +32.1% and +23.1% when the labeled data is extremely limited.*\n\nTips:\n\n- Usage of X-CLIP is identical to [CLIP](clip).\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/xclip_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> X-CLIP architecture. Taken from the <a href=\"https://arxiv.org/abs/2208.02816\">original paper.</a> </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/microsoft/VideoX/tree/master/X-CLIP).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with X-CLIP.\n\n- Demo notebooks for X-CLIP can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/X-CLIP).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## XCLIPProcessor\n\n[[autodoc]] XCLIPProcessor\n\n## XCLIPConfig\n\n[[autodoc]] XCLIPConfig\n    - from_text_vision_configs\n\n## XCLIPTextConfig\n\n[[autodoc]] XCLIPTextConfig\n\n## XCLIPVisionConfig\n\n[[autodoc]] XCLIPVisionConfig\n\n## XCLIPModel\n\n[[autodoc]] XCLIPModel\n    - forward\n    - get_text_features\n    - get_video_features\n\n## XCLIPTextModel\n\n[[autodoc]] XCLIPTextModel\n    - forward\n\n## XCLIPVisionModel\n\n[[autodoc]] XCLIPVisionModel\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 05871cf7-1d76-4a96-a96b-a41c587e5a9e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/videomae", "title": "VideoMAE", "tokens": 1478, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"05871cf7-1d76-4a96-a96b-a41c587e5a9e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/videomae\", \"title\": \"VideoMAE\", \"tokens\": 1478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"781ec81c-baea-4877-b7cd-42a22c2f57c2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/videomae\", \"title\": \"VideoMAE\", \"tokens\": 1478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"79b40e8e575b51be54a8686e025cdc2ff0c421c9fd139169d212f24d71bb1ca0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1d4ea763-ae73-4624-b84b-fce48cfcc754\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"74dbb02ca2c248bbe5608882fd2c92217fb1857e165ac137047494e4e94ade01\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3001, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "781ec81c-baea-4877-b7cd-42a22c2f57c2", "doc_id": "781ec81c-baea-4877-b7cd-42a22c2f57c2", "ref_doc_id": "781ec81c-baea-4877-b7cd-42a22c2f57c2", "chroma:document": "# VideoMAE\n\n## Overview\n\nThe VideoMAE model was proposed in [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://arxiv.org/abs/2203.12602) by Zhan Tong, Yibing Song, Jue Wang, Limin Wang.\nVideoMAE extends masked auto encoders ([MAE](vit_mae)) to video, claiming state-of-the-art performance on several video classification benchmarks.\n\nThe abstract from the paper is the following:\n\n*Pre-training video transformers on extra large-scale datasets is generally required to achieve premier performance on relatively small datasets. In this paper, we show that video masked autoencoders (VideoMAE) are data-efficient learners for self-supervised video pre-training (SSVP). We are inspired by the recent ImageMAE and propose customized video tube masking and reconstruction. These simple designs turn out to be effective for overcoming information leakage caused by the temporal correlation during video reconstruction. We obtain three important findings on SSVP: (1) An extremely high proportion of masking ratio (i.e., 90% to 95%) still yields favorable performance of VideoMAE. The temporally redundant video content enables higher masking ratio than that of images. (2) VideoMAE achieves impressive results on very small datasets (i.e., around 3k-4k videos) without using any extra data. This is partially ascribed to the challenging task of video reconstruction to enforce high-level structure learning. (3) VideoMAE shows that data quality is more important than data quantity for SSVP. Domain shift between pre-training and target datasets are important issues in SSVP. Notably, our VideoMAE with the vanilla ViT backbone can achieve 83.9% on Kinects-400, 75.3% on Something-Something V2, 90.8% on UCF101, and 61.1% on HMDB51 without using any extra data.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/videomae_architecture.jpeg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> VideoMAE pre-training. Taken from the <a href=\"https://arxiv.org/abs/2203.12602\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/MCG-NJU/VideoMAE).\n\n## Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 1d4ea763-ae73-4624-b84b-fce48cfcc754
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/videomae", "title": "VideoMAE", "tokens": 1478, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1d4ea763-ae73-4624-b84b-fce48cfcc754\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/videomae\", \"title\": \"VideoMAE\", \"tokens\": 1478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"781ec81c-baea-4877-b7cd-42a22c2f57c2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/videomae\", \"title\": \"VideoMAE\", \"tokens\": 1478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"79b40e8e575b51be54a8686e025cdc2ff0c421c9fd139169d212f24d71bb1ca0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"05871cf7-1d76-4a96-a96b-a41c587e5a9e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/videomae\", \"title\": \"VideoMAE\", \"tokens\": 1478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c8c73508872c436c77f109629f644f2dee085f32fe4b5c9769548c78c65b936b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f861fd47-9057-49b6-bf6a-11ace8f610e7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"26daa989fc42d04257acb5afb63fce97a5306762c8d57c0b8926d6290f19a639\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1542, \"end_char_idx\": 4683, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "781ec81c-baea-4877-b7cd-42a22c2f57c2", "doc_id": "781ec81c-baea-4877-b7cd-42a22c2f57c2", "ref_doc_id": "781ec81c-baea-4877-b7cd-42a22c2f57c2", "chroma:document": "Domain shift between pre-training and target datasets are important issues in SSVP. Notably, our VideoMAE with the vanilla ViT backbone can achieve 83.9% on Kinects-400, 75.3% on Something-Something V2, 90.8% on UCF101, and 61.1% on HMDB51 without using any extra data.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/videomae_architecture.jpeg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> VideoMAE pre-training. Taken from the <a href=\"https://arxiv.org/abs/2203.12602\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/MCG-NJU/VideoMAE).\n\n## Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```\nfrom transformers import VideoMAEForVideoClassification\nmodel = VideoMAEForVideoClassification.from_pretrained(\"MCG-NJU/videomae-base-finetuned-kinetics\", attn_implementation=\"sdpa\", torch_dtype=torch.float16)\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `MCG-NJU/videomae-base-finetuned-kinetics` model, we saw the following speedups during inference.\n\n|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |\n|--------------|-------------------------------------------|-------------------------------------------|------------------------------|\n|            1 |                                        37 |                                        10 |                      3.7  |\n|            2 |                                        24 |                                        18 |                      1.33 |\n|            4 |                                        43 |                                        32 |                      1.34 |\n|            8 |                                        84 |                                        60 |                      1.4  |\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with VideoMAE. If\nyou're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll\nreview it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: f861fd47-9057-49b6-bf6a-11ace8f610e7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/videomae", "title": "VideoMAE", "tokens": 1478, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f861fd47-9057-49b6-bf6a-11ace8f610e7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/videomae\", \"title\": \"VideoMAE\", \"tokens\": 1478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"781ec81c-baea-4877-b7cd-42a22c2f57c2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/videomae\", \"title\": \"VideoMAE\", \"tokens\": 1478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"79b40e8e575b51be54a8686e025cdc2ff0c421c9fd139169d212f24d71bb1ca0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1d4ea763-ae73-4624-b84b-fce48cfcc754\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/videomae\", \"title\": \"VideoMAE\", \"tokens\": 1478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0197d4fba3265d40096d73ab77e5df56b831a3af7a01451f8bf95be4761dfbac\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3003, \"end_char_idx\": 5655, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "781ec81c-baea-4877-b7cd-42a22c2f57c2", "doc_id": "781ec81c-baea-4877-b7cd-42a22c2f57c2", "ref_doc_id": "781ec81c-baea-4877-b7cd-42a22c2f57c2", "chroma:document": "```\nfrom transformers import VideoMAEForVideoClassification\nmodel = VideoMAEForVideoClassification.from_pretrained(\"MCG-NJU/videomae-base-finetuned-kinetics\", attn_implementation=\"sdpa\", torch_dtype=torch.float16)\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `MCG-NJU/videomae-base-finetuned-kinetics` model, we saw the following speedups during inference.\n\n|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |\n|--------------|-------------------------------------------|-------------------------------------------|------------------------------|\n|            1 |                                        37 |                                        10 |                      3.7  |\n|            2 |                                        24 |                                        18 |                      1.33 |\n|            4 |                                        43 |                                        32 |                      1.34 |\n|            8 |                                        84 |                                        60 |                      1.4  |\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with VideoMAE. If\nyou're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll\nreview it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n**Video classification**\n- [A notebook](https://github.com/huggingface/notebooks/blob/main/examples/video_classification.ipynb) that shows how\nto fine-tune a VideoMAE model on a custom dataset.\n- [Video classification task guide](../tasks/video_classification)\n- [A \ud83e\udd17 Space](https://huggingface.co/spaces/sayakpaul/video-classification-ucf101-subset) showing how to perform inference with a video classification model.\n\n## VideoMAEConfig\n\n[[autodoc]] VideoMAEConfig\n\n## VideoMAEFeatureExtractor\n\n[[autodoc]] VideoMAEFeatureExtractor\n    - __call__\n\n## VideoMAEImageProcessor\n\n[[autodoc]] VideoMAEImageProcessor\n    - preprocess\n\n## VideoMAEModel\n\n[[autodoc]] VideoMAEModel\n    - forward\n\n## VideoMAEForPreTraining\n\n`VideoMAEForPreTraining` includes the decoder on top for self-supervised pre-training.\n\n[[autodoc]] transformers.VideoMAEForPreTraining\n    - forward\n\n## VideoMAEForVideoClassification\n\n[[autodoc]] transformers.VideoMAEForVideoClassification\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 9cc671ad-c923-4994-a477-e055f1a65029
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vit", "title": "Vision Transformer (ViT)", "tokens": 2874, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9cc671ad-c923-4994-a477-e055f1a65029\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit\", \"title\": \"Vision Transformer (ViT)\", \"tokens\": 2874, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6189bf0d-ccf5-4b84-bfb6-261bf69e4598\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit\", \"title\": \"Vision Transformer (ViT)\", \"tokens\": 2874, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5279b9299cef45c6785f117e8d54b4a7583f5b9fcd9d1b7cf5dc7bc4d9831006\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e56e8ea0-bb4a-412c-b777-20cd9ae270c8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f208690bb0a8145af4cd177ceefb3138f224890000facd968d3c493ff40d5ea9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3328, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6189bf0d-ccf5-4b84-bfb6-261bf69e4598", "doc_id": "6189bf0d-ccf5-4b84-bfb6-261bf69e4598", "ref_doc_id": "6189bf0d-ccf5-4b84-bfb6-261bf69e4598", "chroma:document": "# Vision Transformer (ViT)\n\n## Overview\n\nThe Vision Transformer (ViT) model was proposed in [An Image is Worth 16x16 Words: Transformers for Image Recognition\nat Scale](https://arxiv.org/abs/2010.11929) by Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk\nWeissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob\nUszkoreit, Neil Houlsby. It's the first paper that successfully trains a Transformer encoder on ImageNet, attaining\nvery good results compared to familiar convolutional architectures.\n\nThe abstract from the paper is the following:\n\n*While the Transformer architecture has become the de-facto standard for natural language processing tasks, its\napplications to computer vision remain limited. In vision, attention is either applied in conjunction with\nconvolutional networks, or used to replace certain components of convolutional networks while keeping their overall\nstructure in place. We show that this reliance on CNNs is not necessary and a pure transformer applied directly to\nsequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of\ndata and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.),\nVision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring\nsubstantially fewer computational resources to train.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/vit_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> ViT architecture. Taken from the <a href=\"https://arxiv.org/abs/2010.11929\">original paper.</a> </small>\n\nFollowing the original Vision Transformer, some follow-up works have been made:\n\n- [DeiT](deit) (Data-efficient Image Transformers) by Facebook AI. DeiT models are distilled vision transformers.\n  The authors of DeiT also released more efficiently trained ViT models, which you can directly plug into [`ViTModel`] or\n  [`ViTForImageClassification`]. There are 4 variants available (in 3 different sizes): *facebook/deit-tiny-patch16-224*,\n  *facebook/deit-small-patch16-224*, *facebook/deit-base-patch16-224* and *facebook/deit-base-patch16-384*. Note that one should\n  use [`DeiTImageProcessor`] in order to prepare images for the model.\n\n- [BEiT](beit) (BERT pre-training of Image Transformers) by Microsoft Research. BEiT models outperform supervised pre-trained\n  vision transformers using a self-supervised method inspired by BERT (masked image modeling) and based on a VQ-VAE.\n\n- DINO (a method for self-supervised training of Vision Transformers) by Facebook AI. Vision Transformers trained using\n  the DINO method show very interesting properties not seen with convolutional models. They are capable of segmenting\n  objects, without having ever been trained to do so. DINO checkpoints can be found on the [hub](https://huggingface.co/models?other=dino).\n\n- [MAE](vit_mae) (Masked Autoencoders) by Facebook AI. By pre-training Vision Transformers to reconstruct pixel values for a high portion\n  (75%) of masked patches (using an asymmetric encoder-decoder architecture), the authors show that this simple method outperforms\n  supervised pre-training after fine-tuning."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: e56e8ea0-bb4a-412c-b777-20cd9ae270c8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vit", "title": "Vision Transformer (ViT)", "tokens": 2874, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e56e8ea0-bb4a-412c-b777-20cd9ae270c8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit\", \"title\": \"Vision Transformer (ViT)\", \"tokens\": 2874, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6189bf0d-ccf5-4b84-bfb6-261bf69e4598\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit\", \"title\": \"Vision Transformer (ViT)\", \"tokens\": 2874, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5279b9299cef45c6785f117e8d54b4a7583f5b9fcd9d1b7cf5dc7bc4d9831006\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9cc671ad-c923-4994-a477-e055f1a65029\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit\", \"title\": \"Vision Transformer (ViT)\", \"tokens\": 2874, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"23a8450c56f3d3c4fbaa0dca78fccb707024934bb661b690692bc71780c076fc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3cd6a2a1-ab26-436e-9f8a-40cb12bbaeb4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0f3c80007d7fa609bd8b70a9b5d01f567f2249d9ee639c32bfa5762877ae9c14\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1900, \"end_char_idx\": 5083, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6189bf0d-ccf5-4b84-bfb6-261bf69e4598", "doc_id": "6189bf0d-ccf5-4b84-bfb6-261bf69e4598", "ref_doc_id": "6189bf0d-ccf5-4b84-bfb6-261bf69e4598", "chroma:document": "DeiT models are distilled vision transformers.\n  The authors of DeiT also released more efficiently trained ViT models, which you can directly plug into [`ViTModel`] or\n  [`ViTForImageClassification`]. There are 4 variants available (in 3 different sizes): *facebook/deit-tiny-patch16-224*,\n  *facebook/deit-small-patch16-224*, *facebook/deit-base-patch16-224* and *facebook/deit-base-patch16-384*. Note that one should\n  use [`DeiTImageProcessor`] in order to prepare images for the model.\n\n- [BEiT](beit) (BERT pre-training of Image Transformers) by Microsoft Research. BEiT models outperform supervised pre-trained\n  vision transformers using a self-supervised method inspired by BERT (masked image modeling) and based on a VQ-VAE.\n\n- DINO (a method for self-supervised training of Vision Transformers) by Facebook AI. Vision Transformers trained using\n  the DINO method show very interesting properties not seen with convolutional models. They are capable of segmenting\n  objects, without having ever been trained to do so. DINO checkpoints can be found on the [hub](https://huggingface.co/models?other=dino).\n\n- [MAE](vit_mae) (Masked Autoencoders) by Facebook AI. By pre-training Vision Transformers to reconstruct pixel values for a high portion\n  (75%) of masked patches (using an asymmetric encoder-decoder architecture), the authors show that this simple method outperforms\n  supervised pre-training after fine-tuning.\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code (written in JAX) can be\nfound [here](https://github.com/google-research/vision_transformer).\n\nNote that we converted the weights from Ross Wightman's [timm library](https://github.com/rwightman/pytorch-image-models),\nwho already converted the weights from JAX to PyTorch. Credits go to him!\n\n## Usage tips\n\n- To feed images to the Transformer encoder, each image is split into a sequence of fixed-size non-overlapping patches,\n  which are then linearly embedded. A [CLS] token is added to serve as representation of an entire image, which can be\n  used for classification. The authors also add absolute position embeddings, and feed the resulting sequence of\n  vectors to a standard Transformer encoder.\n- As the Vision Transformer expects each image to be of the same size (resolution), one can use\n  [`ViTImageProcessor`] to resize (or rescale) and normalize images for the model.\n- Both the patch resolution and image resolution used during pre-training or fine-tuning are reflected in the name of\n  each checkpoint. For example, `google/vit-base-patch16-224` refers to a base-sized architecture with patch\n  resolution of 16x16 and fine-tuning resolution of 224x224. All checkpoints can be found on the [hub](https://huggingface.co/models?search=vit).\n- The available checkpoints are either (1) pre-trained on [ImageNet-21k](http://www.image-net.org/) (a collection of\n  14 million images and 21k classes) only, or (2) also fine-tuned on [ImageNet](http://www.image-net.org/challenges/LSVRC/2012/) (also referred to as ILSVRC 2012, a collection of 1.3 million\n  images and 1,000 classes).\n- The Vision Transformer was pre-trained using a resolution of 224x224."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 3cd6a2a1-ab26-436e-9f8a-40cb12bbaeb4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vit", "title": "Vision Transformer (ViT)", "tokens": 2874, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3cd6a2a1-ab26-436e-9f8a-40cb12bbaeb4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit\", \"title\": \"Vision Transformer (ViT)\", \"tokens\": 2874, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6189bf0d-ccf5-4b84-bfb6-261bf69e4598\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit\", \"title\": \"Vision Transformer (ViT)\", \"tokens\": 2874, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5279b9299cef45c6785f117e8d54b4a7583f5b9fcd9d1b7cf5dc7bc4d9831006\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e56e8ea0-bb4a-412c-b777-20cd9ae270c8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit\", \"title\": \"Vision Transformer (ViT)\", \"tokens\": 2874, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b7965715fc65dfd16e15c002fd79ecf5931458c52426f6aeb6734d5ac832f20b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c7add624-9805-4ca8-9d9d-b1227c44e75c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f000ddf0efdd7ec24f787f5f3e31fecfb6fd76251f7e04e948cc7652721115a8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3513, \"end_char_idx\": 6673, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6189bf0d-ccf5-4b84-bfb6-261bf69e4598", "doc_id": "6189bf0d-ccf5-4b84-bfb6-261bf69e4598", "ref_doc_id": "6189bf0d-ccf5-4b84-bfb6-261bf69e4598", "chroma:document": "Note that we converted the weights from Ross Wightman's [timm library](https://github.com/rwightman/pytorch-image-models),\nwho already converted the weights from JAX to PyTorch. Credits go to him!\n\n## Usage tips\n\n- To feed images to the Transformer encoder, each image is split into a sequence of fixed-size non-overlapping patches,\n  which are then linearly embedded. A [CLS] token is added to serve as representation of an entire image, which can be\n  used for classification. The authors also add absolute position embeddings, and feed the resulting sequence of\n  vectors to a standard Transformer encoder.\n- As the Vision Transformer expects each image to be of the same size (resolution), one can use\n  [`ViTImageProcessor`] to resize (or rescale) and normalize images for the model.\n- Both the patch resolution and image resolution used during pre-training or fine-tuning are reflected in the name of\n  each checkpoint. For example, `google/vit-base-patch16-224` refers to a base-sized architecture with patch\n  resolution of 16x16 and fine-tuning resolution of 224x224. All checkpoints can be found on the [hub](https://huggingface.co/models?search=vit).\n- The available checkpoints are either (1) pre-trained on [ImageNet-21k](http://www.image-net.org/) (a collection of\n  14 million images and 21k classes) only, or (2) also fine-tuned on [ImageNet](http://www.image-net.org/challenges/LSVRC/2012/) (also referred to as ILSVRC 2012, a collection of 1.3 million\n  images and 1,000 classes).\n- The Vision Transformer was pre-trained using a resolution of 224x224. During fine-tuning, it is often beneficial to\n  use a higher resolution than pre-training [(Touvron et al., 2019)](https://arxiv.org/abs/1906.06423), [(Kolesnikov\n  et al., 2020)](https://arxiv.org/abs/1912.11370). In order to fine-tune at higher resolution, the authors perform\n  2D interpolation of the pre-trained position embeddings, according to their location in the original image.\n- The best results are obtained with supervised pre-training, which is not the case in NLP. The authors also performed\n  an experiment with a self-supervised pre-training objective, namely masked patched prediction (inspired by masked\n  language modeling). With this approach, the smaller ViT-B/16 model achieves 79.9% accuracy on ImageNet, a significant\n  improvement of 2% to training from scratch, but still 4% behind supervised pre-training.\n\n### Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: c7add624-9805-4ca8-9d9d-b1227c44e75c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vit", "title": "Vision Transformer (ViT)", "tokens": 2874, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c7add624-9805-4ca8-9d9d-b1227c44e75c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit\", \"title\": \"Vision Transformer (ViT)\", \"tokens\": 2874, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6189bf0d-ccf5-4b84-bfb6-261bf69e4598\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit\", \"title\": \"Vision Transformer (ViT)\", \"tokens\": 2874, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5279b9299cef45c6785f117e8d54b4a7583f5b9fcd9d1b7cf5dc7bc4d9831006\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3cd6a2a1-ab26-436e-9f8a-40cb12bbaeb4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit\", \"title\": \"Vision Transformer (ViT)\", \"tokens\": 2874, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e20ca46aa6aa681bf39b5c915a2a9069f90ab05bb8c01b964cdf9b40826f3987\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d9704690-c0cd-4356-b56e-43a710d5733a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c7d150dd84c9eb6b0fa496bbcd10457e8a7789d2ee87cbd5fb641d2e832843eb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5084, \"end_char_idx\": 8496, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6189bf0d-ccf5-4b84-bfb6-261bf69e4598", "doc_id": "6189bf0d-ccf5-4b84-bfb6-261bf69e4598", "ref_doc_id": "6189bf0d-ccf5-4b84-bfb6-261bf69e4598", "chroma:document": "During fine-tuning, it is often beneficial to\n  use a higher resolution than pre-training [(Touvron et al., 2019)](https://arxiv.org/abs/1906.06423), [(Kolesnikov\n  et al., 2020)](https://arxiv.org/abs/1912.11370). In order to fine-tune at higher resolution, the authors perform\n  2D interpolation of the pre-trained position embeddings, according to their location in the original image.\n- The best results are obtained with supervised pre-training, which is not the case in NLP. The authors also performed\n  an experiment with a self-supervised pre-training objective, namely masked patched prediction (inspired by masked\n  language modeling). With this approach, the smaller ViT-B/16 model achieves 79.9% accuracy on ImageNet, a significant\n  improvement of 2% to training from scratch, but still 4% behind supervised pre-training.\n\n### Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```\nfrom transformers import ViTForImageClassification\nmodel = ViTForImageClassification.from_pretrained(\"google/vit-base-patch16-224\", attn_implementation=\"sdpa\", torch_dtype=torch.float16)\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `google/vit-base-patch16-224` model, we saw the following speedups during inference.\n\n|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |\n|--------------|-------------------------------------------|-------------------------------------------|------------------------------|\n|            1 |                                         7 |                                         6 |                      1.17 |\n|            2 |                                         8 |                                         6 |                      1.33 |\n|            4 |                                         8 |                                         6 |                      1.33 |\n|            8 |                                         8 |                                         6 |                      1.33 |\n\n## Resources\n\nDemo notebooks regarding inference as well as fine-tuning ViT on custom data can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/VisionTransformer).\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with ViT. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: d9704690-c0cd-4356-b56e-43a710d5733a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vit", "title": "Vision Transformer (ViT)", "tokens": 2874, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d9704690-c0cd-4356-b56e-43a710d5733a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit\", \"title\": \"Vision Transformer (ViT)\", \"tokens\": 2874, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6189bf0d-ccf5-4b84-bfb6-261bf69e4598\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit\", \"title\": \"Vision Transformer (ViT)\", \"tokens\": 2874, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5279b9299cef45c6785f117e8d54b4a7583f5b9fcd9d1b7cf5dc7bc4d9831006\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c7add624-9805-4ca8-9d9d-b1227c44e75c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit\", \"title\": \"Vision Transformer (ViT)\", \"tokens\": 2874, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9b091568d32bc12de1944061c5fce4f5410ea18229b90ee83142fad3634591a9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6675, \"end_char_idx\": 11212, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6189bf0d-ccf5-4b84-bfb6-261bf69e4598", "doc_id": "6189bf0d-ccf5-4b84-bfb6-261bf69e4598", "ref_doc_id": "6189bf0d-ccf5-4b84-bfb6-261bf69e4598", "chroma:document": "```\nfrom transformers import ViTForImageClassification\nmodel = ViTForImageClassification.from_pretrained(\"google/vit-base-patch16-224\", attn_implementation=\"sdpa\", torch_dtype=torch.float16)\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `google/vit-base-patch16-224` model, we saw the following speedups during inference.\n\n|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |\n|--------------|-------------------------------------------|-------------------------------------------|------------------------------|\n|            1 |                                         7 |                                         6 |                      1.17 |\n|            2 |                                         8 |                                         6 |                      1.33 |\n|            4 |                                         8 |                                         6 |                      1.33 |\n|            8 |                                         8 |                                         6 |                      1.33 |\n\n## Resources\n\nDemo notebooks regarding inference as well as fine-tuning ViT on custom data can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/VisionTransformer).\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with ViT. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n`ViTForImageClassification` is supported by:\n<PipelineTag pipeline=\"image-classification\"/>\n\n- A blog post on how to [Fine-Tune ViT for Image Classification with Hugging Face Transformers](https://huggingface.co/blog/fine-tune-vit)\n- A blog post on [Image Classification with Hugging Face Transformers and `Keras`](https://www.philschmid.de/image-classification-huggingface-transformers-keras)\n- A notebook on [Fine-tuning for Image Classification with Hugging Face Transformers](https://github.com/huggingface/notebooks/blob/main/examples/image_classification.ipynb)\n- A notebook on how to [Fine-tune the Vision Transformer on CIFAR-10 with the Hugging Face Trainer](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/VisionTransformer/Fine_tuning_the_Vision_Transformer_on_CIFAR_10_with_the_%F0%9F%A4%97_Trainer.ipynb)\n- A notebook on how to [Fine-tune the Vision Transformer on CIFAR-10 with PyTorch Lightning](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/VisionTransformer/Fine_tuning_the_Vision_Transformer_on_CIFAR_10_with_PyTorch_Lightning.ipynb)\n\n\u2697\ufe0f Optimization\n\n- A blog post on how to [Accelerate Vision Transformer (ViT) with Quantization using Optimum](https://www.philschmid.de/optimizing-vision-transformer)\n\n\u26a1\ufe0f Inference\n\n- A notebook on [Quick demo: Vision Transformer (ViT) by Google Brain](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/VisionTransformer/Quick_demo_of_HuggingFace_version_of_Vision_Transformer_inference.ipynb)\n\n\ud83d\ude80 Deploy\n\n- A blog post on [Deploying Tensorflow Vision Models in Hugging Face with TF Serving](https://huggingface.co/blog/tf-serving-vision)\n- A blog post on [Deploying Hugging Face ViT on Vertex AI](https://huggingface.co/blog/deploy-vertex-ai)\n- A blog post on [Deploying Hugging Face ViT on Kubernetes with TF Serving](https://huggingface.co/blog/deploy-tfserving-kubernetes)\n\n## ViTConfig\n\n[[autodoc]] ViTConfig\n\n## ViTFeatureExtractor\n\n[[autodoc]] ViTFeatureExtractor\n    - __call__\n\n## ViTImageProcessor\n\n[[autodoc]] ViTImageProcessor\n    - preprocess\n\n## ViTImageProcessorFast\n\n[[autodoc]] ViTImageProcessorFast\n    - preprocess\n\n<frameworkcontent>\n<pt>\n\n## ViTModel\n\n[[autodoc]] ViTModel\n    - forward\n\n## ViTForMaskedImageModeling\n\n[[autodoc]] ViTForMaskedImageModeling\n    - forward\n\n## ViTForImageClassification\n\n[[autodoc]] ViTForImageClassification\n    - forward\n\n</pt>\n<tf>\n\n## TFViTModel\n\n[[autodoc]] TFViTModel\n    - call\n\n## TFViTForImageClassification\n\n[[autodoc]] TFViTForImageClassification\n    - call\n\n</tf>\n<jax>\n\n## FlaxVitModel\n\n[[autodoc]] FlaxViTModel\n    - __call__\n\n## FlaxViTForImageClassification\n\n[[autodoc]] FlaxViTForImageClassification\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 691bc900-13b9-433e-ab75-7e9f7d2205af
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/jamba", "title": "Jamba", "tokens": 1474, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"691bc900-13b9-433e-ab75-7e9f7d2205af\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jamba\", \"title\": \"Jamba\", \"tokens\": 1474, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f4c806f2-9320-469d-ac22-2ebcc21a15db\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jamba\", \"title\": \"Jamba\", \"tokens\": 1474, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e72895cf58c8f553e9b16e19ef52164eaef78505e21890ab1cb01a5ce3b8900c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"71f82b90-586d-417d-83ea-3d5dfad653fe\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5f6d9d44d2a0a2dc8f7dff3d4017a86ae4e29dfd3a5d2a5baba17cb5056c4053\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3224, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f4c806f2-9320-469d-ac22-2ebcc21a15db", "doc_id": "f4c806f2-9320-469d-ac22-2ebcc21a15db", "ref_doc_id": "f4c806f2-9320-469d-ac22-2ebcc21a15db", "chroma:document": "# Jamba\n\n## Overview\n\nJamba is a state-of-the-art, hybrid SSM-Transformer LLM. It is the first production-scale Mamba implementation, which opens up interesting research and application opportunities. While this initial experimentation shows encouraging gains, we expect these to be further enhanced with future optimizations and explorations.\n\nFor full details of this model please read the [release blog post](https://www.ai21.com/blog/announcing-jamba).\n\n### Model Details\n\nJamba is a pretrained, mixture-of-experts (MoE) generative text model, with 12B active parameters and an overall of 52B parameters across all experts. It supports a 256K context length, and can fit up to 140K tokens on a single 80GB GPU.\n\nAs depicted in the diagram below, Jamba's architecture features a blocks-and-layers approach that allows Jamba to successfully integrate Transformer and Mamba architectures altogether. Each Jamba block contains either an attention or a Mamba layer, followed by a multi-layer perceptron (MLP), producing an overall ratio of one Transformer layer out of every eight total layers.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/jamba_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n## Usage\n\n### Presequities\n\nJamba requires you use `transformers` version 4.39.0 or higher:\n```bash\npip install transformers>=4.39.0\n```\n\nIn order to run optimized Mamba implementations, you first need to install `mamba-ssm` and `causal-conv1d`:\n```bash\npip install mamba-ssm causal-conv1d>=1.2.0\n```\nYou also have to have the model on a CUDA device.\n\nYou can run the model not using the optimized Mamba kernels, but it is **not** recommended as it will result in significantly lower latencies. In order to do that, you'll need to specify `use_mamba_kernels=False` when loading the model.\n\n### Run the model\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel = AutoModelForCausalLM.from_pretrained(\"ai21labs/Jamba-v0.1\")\ntokenizer = AutoTokenizer.from_pretrained(\"ai21labs/Jamba-v0.1\")\n\ninput_ids = tokenizer(\"In the recent Super Bowl LVIII,\", return_tensors='pt').to(model.device)[\"input_ids\"]\n\noutputs = model.generate(input_ids, max_new_tokens=216)\n\nprint(tokenizer.batch_decode(outputs))\n# [\"<|startoftext|>In the recent Super Bowl LVIII, the Kansas City Chiefs emerged victorious, defeating the San Francisco 49ers in a thrilling overtime showdown. The game was a nail-biter, with both teams showcasing their skills and determination.\\n\\nThe Chiefs, led by their star quarterback Patrick Mahomes, displayed their offensive prowess, while the 49ers, led by their strong defense, put up a tough fight. The game went into overtime, with the Chiefs ultimately securing the win with a touchdown.\\n\\nThe victory marked the Chiefs' second Super Bowl win in four years, solidifying their status as one of the top teams in the NFL. The game was a testament to the skill and talent of both teams, and a thrilling end to the NFL season.\\n\\nThe Super Bowl is not just about the game itself, but also about the halftime show and the commercials. This year's halftime show featured a star-studded lineup, including Usher, Alicia Keys, and Lil Jon."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 71f82b90-586d-417d-83ea-3d5dfad653fe
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/jamba", "title": "Jamba", "tokens": 1474, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"71f82b90-586d-417d-83ea-3d5dfad653fe\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jamba\", \"title\": \"Jamba\", \"tokens\": 1474, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f4c806f2-9320-469d-ac22-2ebcc21a15db\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jamba\", \"title\": \"Jamba\", \"tokens\": 1474, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e72895cf58c8f553e9b16e19ef52164eaef78505e21890ab1cb01a5ce3b8900c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"691bc900-13b9-433e-ab75-7e9f7d2205af\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jamba\", \"title\": \"Jamba\", \"tokens\": 1474, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"33156d6de15c70296dce5b178196027bfee83783965ab70228b683788d48ecde\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"30e84b9c-a4d3-4955-82c4-e5edd2a97e57\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"adbba3a1979e614f15bbb5d39c2e92546abab509692eecddb39203c978a41ee5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1611, \"end_char_idx\": 4816, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f4c806f2-9320-469d-ac22-2ebcc21a15db", "doc_id": "f4c806f2-9320-469d-ac22-2ebcc21a15db", "ref_doc_id": "f4c806f2-9320-469d-ac22-2ebcc21a15db", "chroma:document": "You can run the model not using the optimized Mamba kernels, but it is **not** recommended as it will result in significantly lower latencies. In order to do that, you'll need to specify `use_mamba_kernels=False` when loading the model.\n\n### Run the model\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel = AutoModelForCausalLM.from_pretrained(\"ai21labs/Jamba-v0.1\")\ntokenizer = AutoTokenizer.from_pretrained(\"ai21labs/Jamba-v0.1\")\n\ninput_ids = tokenizer(\"In the recent Super Bowl LVIII,\", return_tensors='pt').to(model.device)[\"input_ids\"]\n\noutputs = model.generate(input_ids, max_new_tokens=216)\n\nprint(tokenizer.batch_decode(outputs))\n# [\"<|startoftext|>In the recent Super Bowl LVIII, the Kansas City Chiefs emerged victorious, defeating the San Francisco 49ers in a thrilling overtime showdown. The game was a nail-biter, with both teams showcasing their skills and determination.\\n\\nThe Chiefs, led by their star quarterback Patrick Mahomes, displayed their offensive prowess, while the 49ers, led by their strong defense, put up a tough fight. The game went into overtime, with the Chiefs ultimately securing the win with a touchdown.\\n\\nThe victory marked the Chiefs' second Super Bowl win in four years, solidifying their status as one of the top teams in the NFL. The game was a testament to the skill and talent of both teams, and a thrilling end to the NFL season.\\n\\nThe Super Bowl is not just about the game itself, but also about the halftime show and the commercials. This year's halftime show featured a star-studded lineup, including Usher, Alicia Keys, and Lil Jon. The show was a spectacle of music and dance, with the performers delivering an energetic and entertaining performance.\\n\"]\n```\n\n<details>\n<summary><strong>Loading the model in half precision</strong></summary>\n\nThe published checkpoint is saved in BF16. In order to load it into RAM in BF16/FP16, you need to specify `torch_dtype`:\n\n```python\nfrom transformers import AutoModelForCausalLM\nimport torch\nmodel = AutoModelForCausalLM.from_pretrained(\"ai21labs/Jamba-v0.1\", torch_dtype=torch.bfloat16)\n# you can also use torch_dtype=torch.float16\n```\n\nWhen using half precision, you can enable the [FlashAttention2](https://github.com/Dao-AILab/flash-attention) implementation of the Attention blocks. In order to use it, you also need the model on a CUDA device. Since in this precision the model is to big to fit on a single 80GB GPU, you'll also need to parallelize it using [accelerate](https://huggingface.co/docs/accelerate/index):\n```python\nfrom transformers import AutoModelForCausalLM\nimport torch\nmodel = AutoModelForCausalLM.from_pretrained(\"ai21labs/Jamba-v0.1\",\n                                             torch_dtype=torch.bfloat16,\n                                             attn_implementation=\"flash_attention_2\",\n                                             device_map=\"auto\")\n```\n\n</details>\n<details><summary><strong>Load the model in 8-bit</strong></summary>\n\n**Using 8-bit precision, it is possible to fit up to 140K sequence lengths on a single 80GB GPU.** You can easily quantize the model to 8-bit using [bitsandbytes](https://huggingface.co/docs/bitsandbytes/index)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 30e84b9c-a4d3-4955-82c4-e5edd2a97e57
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/jamba", "title": "Jamba", "tokens": 1474, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"30e84b9c-a4d3-4955-82c4-e5edd2a97e57\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jamba\", \"title\": \"Jamba\", \"tokens\": 1474, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f4c806f2-9320-469d-ac22-2ebcc21a15db\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jamba\", \"title\": \"Jamba\", \"tokens\": 1474, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e72895cf58c8f553e9b16e19ef52164eaef78505e21890ab1cb01a5ce3b8900c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"71f82b90-586d-417d-83ea-3d5dfad653fe\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jamba\", \"title\": \"Jamba\", \"tokens\": 1474, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"102366a28ad619aa9e26c486b8e83e3d779cdc33a285b9b82b6fe92c618df6e7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3124, \"end_char_idx\": 5556, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f4c806f2-9320-469d-ac22-2ebcc21a15db", "doc_id": "f4c806f2-9320-469d-ac22-2ebcc21a15db", "ref_doc_id": "f4c806f2-9320-469d-ac22-2ebcc21a15db", "chroma:document": "This year's halftime show featured a star-studded lineup, including Usher, Alicia Keys, and Lil Jon. The show was a spectacle of music and dance, with the performers delivering an energetic and entertaining performance.\\n\"]\n```\n\n<details>\n<summary><strong>Loading the model in half precision</strong></summary>\n\nThe published checkpoint is saved in BF16. In order to load it into RAM in BF16/FP16, you need to specify `torch_dtype`:\n\n```python\nfrom transformers import AutoModelForCausalLM\nimport torch\nmodel = AutoModelForCausalLM.from_pretrained(\"ai21labs/Jamba-v0.1\", torch_dtype=torch.bfloat16)\n# you can also use torch_dtype=torch.float16\n```\n\nWhen using half precision, you can enable the [FlashAttention2](https://github.com/Dao-AILab/flash-attention) implementation of the Attention blocks. In order to use it, you also need the model on a CUDA device. Since in this precision the model is to big to fit on a single 80GB GPU, you'll also need to parallelize it using [accelerate](https://huggingface.co/docs/accelerate/index):\n```python\nfrom transformers import AutoModelForCausalLM\nimport torch\nmodel = AutoModelForCausalLM.from_pretrained(\"ai21labs/Jamba-v0.1\",\n                                             torch_dtype=torch.bfloat16,\n                                             attn_implementation=\"flash_attention_2\",\n                                             device_map=\"auto\")\n```\n\n</details>\n<details><summary><strong>Load the model in 8-bit</strong></summary>\n\n**Using 8-bit precision, it is possible to fit up to 140K sequence lengths on a single 80GB GPU.** You can easily quantize the model to 8-bit using [bitsandbytes](https://huggingface.co/docs/bitsandbytes/index). In order to not degrade model quality, we recommend to exclude the Mamba blocks from the quantization:\n\n```python\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\nquantization_config = BitsAndBytesConfig(load_in_8bit=True, llm_int8_skip_modules=[\"mamba\"])\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"ai21labs/Jamba-v0.1\", torch_dtype=torch.bfloat16, attn_implementation=\"flash_attention_2\", quantization_config=quantization_config\n)\n```\n</details>\n\n## JambaConfig\n\n[[autodoc]] JambaConfig\n\n\n## JambaModel\n\n[[autodoc]] JambaModel\n    - forward\n\n\n## JambaForCausalLM\n\n[[autodoc]] JambaForCausalLM\n    - forward\n\n\n## JambaForSequenceClassification\n\n[[autodoc]] transformers.JambaForSequenceClassification\n    - forward"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: b0a099ea-eb8c-4850-b44f-8f5e1d1168a0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llama", "title": "LLaMA", "tokens": 2025, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b0a099ea-eb8c-4850-b44f-8f5e1d1168a0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama\", \"title\": \"LLaMA\", \"tokens\": 2025, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8294a444-300c-40fc-8f2f-75851d9fb3ec\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama\", \"title\": \"LLaMA\", \"tokens\": 2025, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"30013dab60426b0c6f75250a5fa1216ef98523c9cedcadb2f63b8e0cb3a74204\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"936eeea2-5b36-4555-a6e4-079b8058fc82\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2ba5d0c0df263d545be96b6e21a428d756dcc06322811b8f2408ab72258c6b33\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2927, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8294a444-300c-40fc-8f2f-75851d9fb3ec", "doc_id": "8294a444-300c-40fc-8f2f-75851d9fb3ec", "ref_doc_id": "8294a444-300c-40fc-8f2f-75851d9fb3ec", "chroma:document": "# LLaMA\n\n## Overview\n\nThe LLaMA model was proposed in [LLaMA: Open and Efficient Foundation Language Models](https://arxiv.org/abs/2302.13971) by Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth\u00e9e Lacroix, Baptiste Rozi\u00e8re, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, Guillaume Lample. It is a collection of foundation language models ranging from 7B to 65B parameters.\n\nThe abstract from the paper is the following:\n\n*We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B. We release all our models to the research community. *\n\nThis model was contributed by [zphang](https://huggingface.co/zphang) with contributions from [BlackSamorez](https://huggingface.co/BlackSamorez). The code of the implementation in Hugging Face is based on GPT-NeoX [here](https://github.com/EleutherAI/gpt-neox). The original code of the authors can be found [here](https://github.com/facebookresearch/llama).\n\n## Usage tips\n\n- Weights for the LLaMA models can be obtained from by filling out [this form](https://docs.google.com/forms/d/e/1FAIpQLSfqNECQnMkycAp2jP4Z9TFX0cGR4uf7b_fBxjY_OjhJILlKGA/viewform?usp=send_form)\n- After downloading the weights, they will need to be converted to the Hugging Face Transformers format using the [conversion script](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/convert_llama_weights_to_hf.py). The script can be called with the following (example) command:\n\n```bash\npython src/transformers/models/llama/convert_llama_weights_to_hf.py \\\n    --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path\n```\n\n- After conversion, the model and tokenizer can be loaded via:\n\n```python\nfrom transformers import LlamaForCausalLM, LlamaTokenizer\n\ntokenizer = LlamaTokenizer.from_pretrained(\"/output/path\")\nmodel = LlamaForCausalLM.from_pretrained(\"/output/path\")\n```\n\nNote that executing the script requires enough CPU RAM to host the whole model in float16 precision (even if the biggest versions\ncome in several checkpoints they each contain a part of each weight of the model, so we need to load them all in RAM). For the 65B model, it's thus 130GB of RAM needed.\n\n- The LLaMA tokenizer is a BPE model based on [sentencepiece](https://github.com/google/sentencepiece). One quirk of sentencepiece is that when decoding a sequence, if the first token is the start of the word (e.g. \"Banana\"), the tokenizer does not prepend the prefix space to the string."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 936eeea2-5b36-4555-a6e4-079b8058fc82
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llama", "title": "LLaMA", "tokens": 2025, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"936eeea2-5b36-4555-a6e4-079b8058fc82\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama\", \"title\": \"LLaMA\", \"tokens\": 2025, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8294a444-300c-40fc-8f2f-75851d9fb3ec\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama\", \"title\": \"LLaMA\", \"tokens\": 2025, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"30013dab60426b0c6f75250a5fa1216ef98523c9cedcadb2f63b8e0cb3a74204\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b0a099ea-eb8c-4850-b44f-8f5e1d1168a0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama\", \"title\": \"LLaMA\", \"tokens\": 2025, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"af06f8fb49fc311bd236a7d6d21388fa38dfe9d6785b5995f626d16c53f68882\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ab537017-3991-41e8-aa3f-d2fb7e8ddc71\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ea5b533443d7a5392adef42f8c712c0e6b5e43149fd5ac37142c44f2f4b5d94a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1846, \"end_char_idx\": 4707, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8294a444-300c-40fc-8f2f-75851d9fb3ec", "doc_id": "8294a444-300c-40fc-8f2f-75851d9fb3ec", "ref_doc_id": "8294a444-300c-40fc-8f2f-75851d9fb3ec", "chroma:document": "The script can be called with the following (example) command:\n\n```bash\npython src/transformers/models/llama/convert_llama_weights_to_hf.py \\\n    --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path\n```\n\n- After conversion, the model and tokenizer can be loaded via:\n\n```python\nfrom transformers import LlamaForCausalLM, LlamaTokenizer\n\ntokenizer = LlamaTokenizer.from_pretrained(\"/output/path\")\nmodel = LlamaForCausalLM.from_pretrained(\"/output/path\")\n```\n\nNote that executing the script requires enough CPU RAM to host the whole model in float16 precision (even if the biggest versions\ncome in several checkpoints they each contain a part of each weight of the model, so we need to load them all in RAM). For the 65B model, it's thus 130GB of RAM needed.\n\n- The LLaMA tokenizer is a BPE model based on [sentencepiece](https://github.com/google/sentencepiece). One quirk of sentencepiece is that when decoding a sequence, if the first token is the start of the word (e.g. \"Banana\"), the tokenizer does not prepend the prefix space to the string.\n\nThis model was contributed by [zphang](https://huggingface.co/zphang) with contributions from [BlackSamorez](https://huggingface.co/BlackSamorez). The code of the implementation in Hugging Face is based on GPT-NeoX [here](https://github.com/EleutherAI/gpt-neox). The original code of the authors can be found [here](https://github.com/facebookresearch/llama). The Flax version of the implementation was contributed by [afmck](https://huggingface.co/afmck) with the code in the implementation based on Hugging Face's Flax GPT-Neo.\n\n\nBased on the original LLaMA model, Meta AI has released some follow-up works:\n\n- **Llama2**: Llama2 is an improved version of Llama with some architectural tweaks (Grouped Query Attention), and is pre-trained on 2Trillion tokens. Refer to the documentation of Llama2 which can be found [here](llama2).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with LLaMA. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- A [notebook](https://colab.research.google.com/github/bigscience-workshop/petals/blob/main/examples/prompt-tuning-sst2.ipynb#scrollTo=f04ba4d2) on how to use prompt tuning to adapt the LLaMA model for text classification task. \ud83c\udf0e\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- [StackLLaMA: A hands-on guide to train LLaMA with RLHF](https://huggingface.co/blog/stackllama#stackllama-a-hands-on-guide-to-train-llama-with-rlhf), a blog post about how to train LLaMA to answer questions on [Stack Exchange](https://stackexchange.com/) with RLHF."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: ab537017-3991-41e8-aa3f-d2fb7e8ddc71
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llama", "title": "LLaMA", "tokens": 2025, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ab537017-3991-41e8-aa3f-d2fb7e8ddc71\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama\", \"title\": \"LLaMA\", \"tokens\": 2025, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8294a444-300c-40fc-8f2f-75851d9fb3ec\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama\", \"title\": \"LLaMA\", \"tokens\": 2025, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"30013dab60426b0c6f75250a5fa1216ef98523c9cedcadb2f63b8e0cb3a74204\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"936eeea2-5b36-4555-a6e4-079b8058fc82\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama\", \"title\": \"LLaMA\", \"tokens\": 2025, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cb0ca60ff9043520b826d8a931f202e4e38b06d4fdfed2431366ce7814160ffd\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"88618174-86ec-4ba4-a436-74ec8a9808d9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f5e831bddeb404184352ad69569fcb49951a541f4cc776139a81b7fb9d6feeec\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3192, \"end_char_idx\": 5795, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8294a444-300c-40fc-8f2f-75851d9fb3ec", "doc_id": "8294a444-300c-40fc-8f2f-75851d9fb3ec", "ref_doc_id": "8294a444-300c-40fc-8f2f-75851d9fb3ec", "chroma:document": "The original code of the authors can be found [here](https://github.com/facebookresearch/llama). The Flax version of the implementation was contributed by [afmck](https://huggingface.co/afmck) with the code in the implementation based on Hugging Face's Flax GPT-Neo.\n\n\nBased on the original LLaMA model, Meta AI has released some follow-up works:\n\n- **Llama2**: Llama2 is an improved version of Llama with some architectural tweaks (Grouped Query Attention), and is pre-trained on 2Trillion tokens. Refer to the documentation of Llama2 which can be found [here](llama2).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with LLaMA. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- A [notebook](https://colab.research.google.com/github/bigscience-workshop/petals/blob/main/examples/prompt-tuning-sst2.ipynb#scrollTo=f04ba4d2) on how to use prompt tuning to adapt the LLaMA model for text classification task. \ud83c\udf0e\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- [StackLLaMA: A hands-on guide to train LLaMA with RLHF](https://huggingface.co/blog/stackllama#stackllama-a-hands-on-guide-to-train-llama-with-rlhf), a blog post about how to train LLaMA to answer questions on [Stack Exchange](https://stackexchange.com/) with RLHF.\n\n\u2697\ufe0f Optimization\n- A [notebook](https://colab.research.google.com/drive/1SQUXq1AMZPSLD4mk3A3swUIc6Y2dclme?usp=sharing) on how to fine-tune LLaMA model using xturing library on GPU which has limited memory. \ud83c\udf0e \n\n\u26a1\ufe0f Inference\n- A [notebook](https://colab.research.google.com/github/DominguesM/alpaca-lora-ptbr-7b/blob/main/notebooks/02%20-%20Evaluate.ipynb) on how to run the LLaMA Model using PeftModel from the \ud83e\udd17 PEFT library. \ud83c\udf0e \n- A [notebook](https://colab.research.google.com/drive/1l2GiSSPbajVyp2Nk3CFT4t3uH6-5TiBe?usp=sharing) on how to load a PEFT adapter LLaMA model with LangChain. \ud83c\udf0e\n\n\ud83d\ude80 Deploy\n- A [notebook](https://colab.research.google.com/github/lxe/simple-llama-finetuner/blob/master/Simple_LLaMA_FineTuner.ipynb#scrollTo=3PM_DilAZD8T) on how to fine-tune LLaMA model using LoRA method via the \ud83e\udd17 PEFT library with intuitive UI. \ud83c\udf0e \n- A [notebook](https://github.com/aws/amazon-sagemaker-examples/blob/main/introduction_to_amazon_algorithms/jumpstart-foundation-models/text-generation-open-llama.ipynb) on how to deploy Open-LLaMA model for text generation on Amazon SageMaker."}


