# embeddings_queue

created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: dbfd220e-e4f0-4327-8759-1679e6d108b2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"dbfd220e-e4f0-4327-8759-1679e6d108b2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1de2f80f-f53c-43e8-b5f3-49b3b915c9f7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"87094aa86af139e913445f44b4ef5b8586c1ce3c46f8f7ba6b5f094bd133cd43\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6b3b6ea1-5d47-47b7-9f6d-fb4f09bc2f42\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1beaf4910b255e2a1e20a0c80758fd8dfc22ed9395830a61a40f7c5d9b025dae\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f6db85fc-923e-4f8c-98a4-a3f66ae6de95\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"abda1ecd7f6a252d38b6c6d758b288b1206f071d8922386db280b0f1ba6ace4b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11881, \"end_char_idx\": 15023, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "doc_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "ref_doc_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "chroma:document": "... Question: What modern tool is used to make gazpacho?\n... Answer:\n... \"\"\"\n\n>>> sequences = pipe(\n...     prompt,\n...     max_new_tokens=10,\n...     do_sample=True,\n...     top_k=10,\n...     return_full_text = False,\n... )\n\n>>> for seq in sequences:\n...     print(f\"Result: {seq['generated_text']}\")\nResult: Modern tools often used to make gazpacho include\n```\n\n#### Reasoning\n\nReasoning is one of the most difficult tasks for LLMs, and achieving good results often requires applying advanced prompting techniques, like \n[Chain-of-though](#chain-of-thought).\n\nLet's try if we can make a model reason about a simple arithmetics task with a basic prompt: \n\n```python\n>>> torch.manual_seed(5) # doctest: +IGNORE_RESULT\n>>> prompt = \"\"\"There are 5 groups of students in the class. Each group has 4 students. How many students are there in the class?\"\"\"\n\n>>> sequences = pipe(\n...     prompt,\n...     max_new_tokens=30,\n...     do_sample=True,\n...     top_k=10,\n...     return_full_text = False,\n... )\n\n>>> for seq in sequences:\n...     print(f\"Result: {seq['generated_text']}\")\nResult: \nThere are a total of 5 groups, so there are 5 x 4=20 students in the class.\n```\n\nCorrect! Let's increase the complexity a little and see if we can still get away with a basic prompt:\n\n```python\n>>> torch.manual_seed(6) # doctest: +IGNORE_RESULT\n>>> prompt = \"\"\"I baked 15 muffins. I ate 2 muffins and gave 5 muffins to a neighbor. My partner then bought 6 more muffins and ate 2. How many muffins do we now have?\"\"\"\n\n>>> sequences = pipe(\n...     prompt,\n...     max_new_tokens=10,\n...     do_sample=True,\n...     top_k=10,\n...     return_full_text = False,\n... )\n\n>>> for seq in sequences:\n...     print(f\"Result: {seq['generated_text']}\")\nResult: \nThe total number of muffins now is 21\n```\n\nThis is a wrong answer, it should be 12. In this case, this can be due to the prompt being too basic, or due to the choice \nof model, after all we've picked the smallest version of Falcon. Reasoning is difficult for models of all sizes, but larger \nmodels are likely to perform better. \n\n## Best practices of LLM prompting\n\nIn this section of the guide we have compiled a list of best practices that tend to improve the prompt results:\n\n* When choosing the model to work with, the latest and most capable models are likely to perform better. \n* Start with a simple and short prompt, and iterate from there.\n* Put the instructions at the beginning of the prompt, or at the very end. When working with large context, models apply various optimizations to prevent Attention complexity from scaling quadratically. This may make a model more attentive to the beginning or end of a prompt than the middle.\n* Clearly separate instructions from the text they apply to - more on this in the next section. \n* Be specific and descriptive about the task and the desired outcome - its format, length, style, language, etc.\n* Avoid ambiguous descriptions and instructions.\n* Favor instructions that say \"what to do\" instead of those that say \"what not to do\".\n* \"Lead\" the output in the right direction by writing the first word (or even begin the first sentence for the model)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f6db85fc-923e-4f8c-98a4-a3f66ae6de95
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f6db85fc-923e-4f8c-98a4-a3f66ae6de95\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1de2f80f-f53c-43e8-b5f3-49b3b915c9f7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"87094aa86af139e913445f44b4ef5b8586c1ce3c46f8f7ba6b5f094bd133cd43\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dbfd220e-e4f0-4327-8759-1679e6d108b2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9b7c6f45e6da5eaddd485c8beee12b3a2b2447f2ccd9906c27126869cbf96908\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"61c21fa6-b00f-475a-87e1-1c6f9e8dac0e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8b3a606d274766b2746a230d1a90339abbdd921988772a75fd31113ad58271be\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13297, \"end_char_idx\": 16446, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "doc_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "ref_doc_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "chroma:document": "My partner then bought 6 more muffins and ate 2. How many muffins do we now have?\"\"\"\n\n>>> sequences = pipe(\n...     prompt,\n...     max_new_tokens=10,\n...     do_sample=True,\n...     top_k=10,\n...     return_full_text = False,\n... )\n\n>>> for seq in sequences:\n...     print(f\"Result: {seq['generated_text']}\")\nResult: \nThe total number of muffins now is 21\n```\n\nThis is a wrong answer, it should be 12. In this case, this can be due to the prompt being too basic, or due to the choice \nof model, after all we've picked the smallest version of Falcon. Reasoning is difficult for models of all sizes, but larger \nmodels are likely to perform better. \n\n## Best practices of LLM prompting\n\nIn this section of the guide we have compiled a list of best practices that tend to improve the prompt results:\n\n* When choosing the model to work with, the latest and most capable models are likely to perform better. \n* Start with a simple and short prompt, and iterate from there.\n* Put the instructions at the beginning of the prompt, or at the very end. When working with large context, models apply various optimizations to prevent Attention complexity from scaling quadratically. This may make a model more attentive to the beginning or end of a prompt than the middle.\n* Clearly separate instructions from the text they apply to - more on this in the next section. \n* Be specific and descriptive about the task and the desired outcome - its format, length, style, language, etc.\n* Avoid ambiguous descriptions and instructions.\n* Favor instructions that say \"what to do\" instead of those that say \"what not to do\".\n* \"Lead\" the output in the right direction by writing the first word (or even begin the first sentence for the model).\n* Use advanced techniques like [Few-shot prompting](#few-shot-prompting) and [Chain-of-thought](#chain-of-thought)\n* Test your prompts with different models to assess their robustness. \n* Version and track the performance of your prompts. \n\n## Advanced prompting techniques\n\n### Few-shot prompting\n\nThe basic prompts in the sections above are the examples of \"zero-shot\" prompts, meaning, the model has been given \ninstructions and context, but no examples with solutions. LLMs that have been fine-tuned on instruction datasets, generally \nperform well on such \"zero-shot\" tasks. However, you may find that your task has more complexity or nuance, and, perhaps, \nyou have some requirements for the output that the model doesn't catch on just from the instructions. In this case, you can \ntry the technique called few-shot prompting. \n\nIn few-shot prompting, we provide examples in the prompt giving the model more context to improve the performance. \nThe examples condition the model to generate the output following the patterns in the examples.\n\nHere's an example: \n\n```python\n>>> torch.manual_seed(0) # doctest: +IGNORE_RESULT\n>>> prompt = \"\"\"Text: The first human went into space and orbited the Earth on April 12, 1961.\n... Date: 04/12/1961\n... Text: The first-ever televised presidential debate in the United States took place on September 28, 1960, between presidential candidates John F. Kennedy and Richard Nixon."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 61c21fa6-b00f-475a-87e1-1c6f9e8dac0e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"61c21fa6-b00f-475a-87e1-1c6f9e8dac0e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1de2f80f-f53c-43e8-b5f3-49b3b915c9f7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"87094aa86af139e913445f44b4ef5b8586c1ce3c46f8f7ba6b5f094bd133cd43\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f6db85fc-923e-4f8c-98a4-a3f66ae6de95\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"53fe599b117017c912e3fa9aaa1a95cab1aef9c2ee2ca3228b6a3d31f8988c08\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a51ea7db-4125-47af-93d1-bae91071acc2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c5a4439e54f05064446fc6b74544e1e4fcb7a049bdb97044326c2ee526b1998e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14656, \"end_char_idx\": 18090, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "doc_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "ref_doc_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "chroma:document": "* Be specific and descriptive about the task and the desired outcome - its format, length, style, language, etc.\n* Avoid ambiguous descriptions and instructions.\n* Favor instructions that say \"what to do\" instead of those that say \"what not to do\".\n* \"Lead\" the output in the right direction by writing the first word (or even begin the first sentence for the model).\n* Use advanced techniques like [Few-shot prompting](#few-shot-prompting) and [Chain-of-thought](#chain-of-thought)\n* Test your prompts with different models to assess their robustness. \n* Version and track the performance of your prompts. \n\n## Advanced prompting techniques\n\n### Few-shot prompting\n\nThe basic prompts in the sections above are the examples of \"zero-shot\" prompts, meaning, the model has been given \ninstructions and context, but no examples with solutions. LLMs that have been fine-tuned on instruction datasets, generally \nperform well on such \"zero-shot\" tasks. However, you may find that your task has more complexity or nuance, and, perhaps, \nyou have some requirements for the output that the model doesn't catch on just from the instructions. In this case, you can \ntry the technique called few-shot prompting. \n\nIn few-shot prompting, we provide examples in the prompt giving the model more context to improve the performance. \nThe examples condition the model to generate the output following the patterns in the examples.\n\nHere's an example: \n\n```python\n>>> torch.manual_seed(0) # doctest: +IGNORE_RESULT\n>>> prompt = \"\"\"Text: The first human went into space and orbited the Earth on April 12, 1961.\n... Date: 04/12/1961\n... Text: The first-ever televised presidential debate in the United States took place on September 28, 1960, between presidential candidates John F. Kennedy and Richard Nixon. \n... Date:\"\"\"\n\n>>> sequences = pipe(\n...     prompt,\n...     max_new_tokens=8,\n...     do_sample=True,\n...     top_k=10,\n... )\n\n>>> for seq in sequences:\n...     print(f\"Result: {seq['generated_text']}\")\nResult: Text: The first human went into space and orbited the Earth on April 12, 1961.\nDate: 04/12/1961\nText: The first-ever televised presidential debate in the United States took place on September 28, 1960, between presidential candidates John F. Kennedy and Richard Nixon. \nDate: 09/28/1960\n```\n\nIn the above code snippet we used a single example to demonstrate the desired output to the model, so this can be called a \n\"one-shot\" prompting. However, depending on the task complexity you may need to use more than one example. \n\nLimitations of the few-shot prompting technique: \n- While LLMs can pick up on the patterns in the examples, these technique doesn't work well on complex reasoning tasks\n- Few-shot prompting requires creating lengthy prompts. Prompts with large number of tokens can increase computation and latency. There's also a limit to the length of the prompts.  \n- Sometimes when given a number of examples, models can learn patterns that you didn't intend them to learn, e.g. that the third movie review is always negative.\n\n### Chain-of-thought\n\nChain-of-thought (CoT) prompting is a technique that nudges a model to produce intermediate reasoning steps thus improving \nthe results on complex reasoning tasks. \n\nThere are two ways of steering a model to producing the reasoning steps:\n- few-shot prompting by illustrating examples with detailed answers to questions, showing the model how to work through a problem."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a51ea7db-4125-47af-93d1-bae91071acc2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a51ea7db-4125-47af-93d1-bae91071acc2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1de2f80f-f53c-43e8-b5f3-49b3b915c9f7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"87094aa86af139e913445f44b4ef5b8586c1ce3c46f8f7ba6b5f094bd133cd43\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"61c21fa6-b00f-475a-87e1-1c6f9e8dac0e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0a7054e4652aeaf5821ccb87c14193d8e602117dbf19efd1fb3bd0014415f7de\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0df8790f-8999-4dac-b2a0-4384472d7e73\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c96370dedb093cf98777c9e0a2f9c993a97a7eceb89f76d041faa082788e1e0f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 16448, \"end_char_idx\": 19751, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "doc_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "ref_doc_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "chroma:document": "... Date:\"\"\"\n\n>>> sequences = pipe(\n...     prompt,\n...     max_new_tokens=8,\n...     do_sample=True,\n...     top_k=10,\n... )\n\n>>> for seq in sequences:\n...     print(f\"Result: {seq['generated_text']}\")\nResult: Text: The first human went into space and orbited the Earth on April 12, 1961.\nDate: 04/12/1961\nText: The first-ever televised presidential debate in the United States took place on September 28, 1960, between presidential candidates John F. Kennedy and Richard Nixon. \nDate: 09/28/1960\n```\n\nIn the above code snippet we used a single example to demonstrate the desired output to the model, so this can be called a \n\"one-shot\" prompting. However, depending on the task complexity you may need to use more than one example. \n\nLimitations of the few-shot prompting technique: \n- While LLMs can pick up on the patterns in the examples, these technique doesn't work well on complex reasoning tasks\n- Few-shot prompting requires creating lengthy prompts. Prompts with large number of tokens can increase computation and latency. There's also a limit to the length of the prompts.  \n- Sometimes when given a number of examples, models can learn patterns that you didn't intend them to learn, e.g. that the third movie review is always negative.\n\n### Chain-of-thought\n\nChain-of-thought (CoT) prompting is a technique that nudges a model to produce intermediate reasoning steps thus improving \nthe results on complex reasoning tasks. \n\nThere are two ways of steering a model to producing the reasoning steps:\n- few-shot prompting by illustrating examples with detailed answers to questions, showing the model how to work through a problem.\n- by instructing the model to reason by adding phrases like \"Let's think step by step\" or \"Take a deep breath and work through the problem step by step.\"\n\nIf we apply the CoT technique to the muffins example from the [reasoning section](#reasoning) and use a larger model, \nsuch as (`tiiuae/falcon-180B-chat`) which you can play with in the [HuggingChat](https://huggingface.co/chat/), \nwe'll get a significant improvement on the reasoning result:\n\n```text\nLet's go through this step-by-step:\n1. You start with 15 muffins.\n2. You eat 2 muffins, leaving you with 13 muffins.\n3. You give 5 muffins to your neighbor, leaving you with 8 muffins.\n4. Your partner buys 6 more muffins, bringing the total number of muffins to 14.\n5. Your partner eats 2 muffins, leaving you with 12 muffins.\nTherefore, you now have 12 muffins.\n```\n\n## Prompting vs fine-tuning\n\nYou can achieve great results by optimizing your prompts, however, you may still ponder whether fine-tuning a model \nwould work better for your case. Here are some scenarios when fine-tuning a smaller model may be a preferred option:\n\n- Your domain is wildly different from what LLMs were pre-trained on and extensive prompt optimization did not yield sufficient results. \n- You need your model to work well in a low-resource language.\n- You need the model to be trained on sensitive data that is under strict regulations. \n- You have to use a small model due to cost, privacy, infrastructure or other limitations. \n\nIn all of the above examples, you will need to make sure that you either already have or can easily obtain a large enough \ndomain-specific dataset at a reasonable cost to fine-tune a model."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0df8790f-8999-4dac-b2a0-4384472d7e73
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0df8790f-8999-4dac-b2a0-4384472d7e73\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1de2f80f-f53c-43e8-b5f3-49b3b915c9f7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"87094aa86af139e913445f44b4ef5b8586c1ce3c46f8f7ba6b5f094bd133cd43\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a51ea7db-4125-47af-93d1-bae91071acc2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5945652a544f9922ead69b9aa9b6069f1a29b979865e6104bc159b0e7daa48bc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 18246, \"end_char_idx\": 19928, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "doc_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "ref_doc_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "chroma:document": "If we apply the CoT technique to the muffins example from the [reasoning section](#reasoning) and use a larger model, \nsuch as (`tiiuae/falcon-180B-chat`) which you can play with in the [HuggingChat](https://huggingface.co/chat/), \nwe'll get a significant improvement on the reasoning result:\n\n```text\nLet's go through this step-by-step:\n1. You start with 15 muffins.\n2. You eat 2 muffins, leaving you with 13 muffins.\n3. You give 5 muffins to your neighbor, leaving you with 8 muffins.\n4. Your partner buys 6 more muffins, bringing the total number of muffins to 14.\n5. Your partner eats 2 muffins, leaving you with 12 muffins.\nTherefore, you now have 12 muffins.\n```\n\n## Prompting vs fine-tuning\n\nYou can achieve great results by optimizing your prompts, however, you may still ponder whether fine-tuning a model \nwould work better for your case. Here are some scenarios when fine-tuning a smaller model may be a preferred option:\n\n- Your domain is wildly different from what LLMs were pre-trained on and extensive prompt optimization did not yield sufficient results. \n- You need your model to work well in a low-resource language.\n- You need the model to be trained on sensitive data that is under strict regulations. \n- You have to use a small model due to cost, privacy, infrastructure or other limitations. \n\nIn all of the above examples, you will need to make sure that you either already have or can easily obtain a large enough \ndomain-specific dataset at a reasonable cost to fine-tune a model. You will also need to have enough time and resources \nto fine-tune a model.\n\nIf the above examples are not the case for you, optimizing prompts can prove to be more beneficial."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6edf77b6-42cf-4c62-9046-09a7c8058dbc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/image_feature_extraction", "title": "Image Feature Extraction", "tokens": 1193, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6edf77b6-42cf-4c62-9046-09a7c8058dbc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_feature_extraction\", \"title\": \"Image Feature Extraction\", \"tokens\": 1193, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ea7b80cd-6ab7-4a17-8572-9839fa0fd05c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_feature_extraction\", \"title\": \"Image Feature Extraction\", \"tokens\": 1193, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f3c15d4d2bf0ad340d04363c6ded3b1efccdb513f6513786ce9db9b215c5c609\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bba040d3-860b-45be-9a4d-6efd8480cbf5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"36a127afa0ec2632ec488a1ebb400bd26a02d56908eb287d635de86b6cfdbd84\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3190, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ea7b80cd-6ab7-4a17-8572-9839fa0fd05c", "doc_id": "ea7b80cd-6ab7-4a17-8572-9839fa0fd05c", "ref_doc_id": "ea7b80cd-6ab7-4a17-8572-9839fa0fd05c", "chroma:document": "# Image Feature Extraction\n\n[[open-in-colab]]\n\nImage feature extraction is the task of extracting semantically meaningful features given an image. This has many use cases, including image similarity and image retrieval. Moreover, most computer vision models can be used for image feature extraction, where one can remove the task-specific head (image classification, object detection etc) and get the features. These features are very useful on a higher level: edge detection, corner detection and so on. They may also contain information about the real world (e.g. what a cat looks like) depending on how deep the model is. Therefore, these outputs can be used to train new classifiers on a specific dataset.\n\nIn this guide, you will:\n\n- Learn to build a simple image similarity system on top of the `image-feature-extraction` pipeline.\n- Accomplish the same task with bare model inference.\n\n## Image Similarity using `image-feature-extraction` Pipeline\n\nWe have two images of cats sitting on top of fish nets, one of them is generated. \n\n```python\nfrom PIL import Image\nimport requests\n\nimg_urls = [\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/cats.png\", \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/cats.jpeg\"]\nimage_real = Image.open(requests.get(img_urls[0], stream=True).raw).convert(\"RGB\")\nimage_gen = Image.open(requests.get(img_urls[1], stream=True).raw).convert(\"RGB\")\n```\n\nLet's see the pipeline in action. First, initialize the pipeline. If you don't pass any model to it, the pipeline will be automatically initialized with [google/vit-base-patch16-224](google/vit-base-patch16-224). If you'd like to calculate similarity, set `pool` to True.\n\n```python\nimport torch\nfrom transformers import pipeline\n\nDEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\npipe = pipeline(task=\"image-feature-extraction\", model_name=\"google/vit-base-patch16-384\", device=DEVICE, pool=True)\n```\n\nTo infer with `pipe` pass both images to it.\n\n```python\noutputs = pipe([image_real, image_gen])\n```\n\nThe output contains pooled embeddings of those two images.\n\n```python\n# get the length of a single output\nprint(len(outputs[0][0]))\n# show outputs\nprint(outputs)\n\n# 768\n# [[[-0.03909236937761307, 0.43381670117378235, -0.06913255900144577,\n```\n\nTo get the similarity score, we need to pass them to a similarity function. \n\n```python\nfrom torch.nn.functional import cosine_similarity\n\nsimilarity_score = cosine_similarity(torch.Tensor(outputs[0]),\n                                     torch.Tensor(outputs[1]), dim=1)\n\nprint(similarity_score)\n\n# tensor([0.6043])\n```\n\nIf you want to get the last hidden states before pooling, avoid passing any value for the `pool` parameter, as it is set to `False`\u00a0by default. These hidden states are useful for training new classifiers or models based on the features from the model.\n\n```python\npipe = pipeline(task=\"image-feature-extraction\", model_name=\"google/vit-base-patch16-224\", device=DEVICE)\noutput = pipe(image_real)\n```\n\nSince the outputs are unpooled, we get the last hidden states where the first dimension is the batch size, and the last two are the embedding shape."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: bba040d3-860b-45be-9a4d-6efd8480cbf5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/image_feature_extraction", "title": "Image Feature Extraction", "tokens": 1193, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"bba040d3-860b-45be-9a4d-6efd8480cbf5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_feature_extraction\", \"title\": \"Image Feature Extraction\", \"tokens\": 1193, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ea7b80cd-6ab7-4a17-8572-9839fa0fd05c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_feature_extraction\", \"title\": \"Image Feature Extraction\", \"tokens\": 1193, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f3c15d4d2bf0ad340d04363c6ded3b1efccdb513f6513786ce9db9b215c5c609\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6edf77b6-42cf-4c62-9046-09a7c8058dbc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_feature_extraction\", \"title\": \"Image Feature Extraction\", \"tokens\": 1193, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a5de3d07742147b3cb9f23467455361cafea61d21d1b5dc9e9ee31cddd8deba3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1665, \"end_char_idx\": 4439, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ea7b80cd-6ab7-4a17-8572-9839fa0fd05c", "doc_id": "ea7b80cd-6ab7-4a17-8572-9839fa0fd05c", "ref_doc_id": "ea7b80cd-6ab7-4a17-8572-9839fa0fd05c", "chroma:document": "If you'd like to calculate similarity, set `pool` to True.\n\n```python\nimport torch\nfrom transformers import pipeline\n\nDEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\npipe = pipeline(task=\"image-feature-extraction\", model_name=\"google/vit-base-patch16-384\", device=DEVICE, pool=True)\n```\n\nTo infer with `pipe` pass both images to it.\n\n```python\noutputs = pipe([image_real, image_gen])\n```\n\nThe output contains pooled embeddings of those two images.\n\n```python\n# get the length of a single output\nprint(len(outputs[0][0]))\n# show outputs\nprint(outputs)\n\n# 768\n# [[[-0.03909236937761307, 0.43381670117378235, -0.06913255900144577,\n```\n\nTo get the similarity score, we need to pass them to a similarity function. \n\n```python\nfrom torch.nn.functional import cosine_similarity\n\nsimilarity_score = cosine_similarity(torch.Tensor(outputs[0]),\n                                     torch.Tensor(outputs[1]), dim=1)\n\nprint(similarity_score)\n\n# tensor([0.6043])\n```\n\nIf you want to get the last hidden states before pooling, avoid passing any value for the `pool` parameter, as it is set to `False`\u00a0by default. These hidden states are useful for training new classifiers or models based on the features from the model.\n\n```python\npipe = pipeline(task=\"image-feature-extraction\", model_name=\"google/vit-base-patch16-224\", device=DEVICE)\noutput = pipe(image_real)\n```\n\nSince the outputs are unpooled, we get the last hidden states where the first dimension is the batch size, and the last two are the embedding shape.\n\n```python\nimport numpy as np\nprint(np.array(outputs).shape)\n# (1, 197, 768)\n```\n\n## Getting Features and Similarities using `AutoModel`\n\nWe can also use `AutoModel`\u00a0class of transformers to get the features. `AutoModel` loads any transformers model with no task-specific head, and we can use this to get the features.\n\n```python\nfrom transformers import AutoImageProcessor, AutoModel\n\nprocessor = AutoImageProcessor.from_pretrained(\"google/vit-base-patch16-224\")\nmodel = AutoModel.from_pretrained(\"google/vit-base-patch16-224\").to(DEVICE)\n```\n\nLet's write a simple function for inference. We will pass the inputs to the `processor` first and pass its outputs to the `model`.\n\n```python\ndef infer(image):\n  inputs = processor(image, return_tensors=\"pt\").to(DEVICE)\n  outputs = model(**inputs)\n  return outputs.pooler_output\n```\n\nWe can pass the images directly to this function and get the embeddings.\n\n```python\nembed_real = infer(image_real)\nembed_gen = infer(image_gen)\n```\n\nWe can get the similarity again over the embeddings.\n\n```python\nfrom torch.nn.functional import cosine_similarity\n\nsimilarity_score = cosine_similarity(embed_real, embed_gen, dim=1)\nprint(similarity_score)\n\n# tensor([0.6061], device='cuda:0', grad_fn=<SumBackward1>)\n```"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: ec26ebd8-0874-431b-83ac-2174cd73007d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/timesformer", "title": "TimeSformer", "tokens": 618, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ec26ebd8-0874-431b-83ac-2174cd73007d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/timesformer\", \"title\": \"TimeSformer\", \"tokens\": 618, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8a98df82-0cd5-45d8-ac75-c95fce631549\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/timesformer\", \"title\": \"TimeSformer\", \"tokens\": 618, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8a6d62b2b634dd0d8525b1f44755a2784b055d22d16f36afd0c670958b8c63dc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2264, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8a98df82-0cd5-45d8-ac75-c95fce631549", "doc_id": "8a98df82-0cd5-45d8-ac75-c95fce631549", "ref_doc_id": "8a98df82-0cd5-45d8-ac75-c95fce631549", "chroma:document": "# TimeSformer\n\n## Overview\n\nThe TimeSformer model was proposed in [TimeSformer: Is Space-Time Attention All You Need for Video Understanding?](https://arxiv.org/abs/2102.05095) by Facebook Research.\nThis work is a milestone in action-recognition field being the first video transformer. It inspired many transformer based video understanding and classification papers.\n\nThe abstract from the paper is the following:\n\n*We present a convolution-free approach to video classification built exclusively on self-attention over space and time. Our method, named \"TimeSformer,\" adapts the standard Transformer architecture to video by enabling spatiotemporal feature learning directly from a sequence of frame-level patches. Our experimental study compares different self-attention schemes and suggests that \"divided attention,\" where temporal attention and spatial attention are separately applied within each block, leads to the best video classification accuracy among the design choices considered. Despite the radically new design, TimeSformer achieves state-of-the-art results on several action recognition benchmarks, including the best reported accuracy on Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks, our model is faster to train, it can achieve dramatically higher test efficiency (at a small drop in accuracy), and it can also be applied to much longer video clips (over one minute long). Code and models are available at: [this https URL](https://github.com/facebookresearch/TimeSformer).*\n\nThis model was contributed by [fcakyon](https://huggingface.co/fcakyon).\nThe original code can be found [here](https://github.com/facebookresearch/TimeSformer).\n\n## Usage tips\n\nThere are many pretrained variants. Select your pretrained model based on the dataset it is trained on. Moreover,\nthe number of input frames per clip changes based on the model size so you should consider this parameter while selecting your pretrained model.\n\n## Resources\n\n- [Video classification task guide](../tasks/video_classification)\n\n## TimesformerConfig\n\n[[autodoc]] TimesformerConfig\n\n## TimesformerModel\n\n[[autodoc]] TimesformerModel\n    - forward\n\n## TimesformerForVideoClassification\n\n[[autodoc]] TimesformerForVideoClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f9a8ba69-b972-4790-9ac6-c84bde842107
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/deberta-v2", "title": "DeBERTa-v2", "tokens": 1600, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f9a8ba69-b972-4790-9ac6-c84bde842107\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deberta-v2\", \"title\": \"DeBERTa-v2\", \"tokens\": 1600, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"48d7be7f-faba-4d81-a3f6-9657d9866e78\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deberta-v2\", \"title\": \"DeBERTa-v2\", \"tokens\": 1600, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a2b0a3f7da958b59525f998fad765610dcae6d310d2e5b34c37ee1d9dd99673f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"308c0ac3-6452-458c-881c-c6aa986427f2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5c0b5410f0f79766d3f04ee7b3176639182f584280148e40fb995f1e79f65b53\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3233, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "48d7be7f-faba-4d81-a3f6-9657d9866e78", "doc_id": "48d7be7f-faba-4d81-a3f6-9657d9866e78", "ref_doc_id": "48d7be7f-faba-4d81-a3f6-9657d9866e78", "chroma:document": "# DeBERTa-v2\n\n## Overview\n\nThe DeBERTa model was proposed in [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://arxiv.org/abs/2006.03654) by Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen It is based on Google's\nBERT model released in 2018 and Facebook's RoBERTa model released in 2019.\n\nIt builds on RoBERTa with disentangled attention and enhanced mask decoder training with half of the data used in\nRoBERTa.\n\nThe abstract from the paper is the following:\n\n*Recent progress in pre-trained neural language models has significantly improved the performance of many natural\nlanguage processing (NLP) tasks. In this paper we propose a new model architecture DeBERTa (Decoding-enhanced BERT with\ndisentangled attention) that improves the BERT and RoBERTa models using two novel techniques. The first is the\ndisentangled attention mechanism, where each word is represented using two vectors that encode its content and\nposition, respectively, and the attention weights among words are computed using disentangled matrices on their\ncontents and relative positions. Second, an enhanced mask decoder is used to replace the output softmax layer to\npredict the masked tokens for model pretraining. We show that these two techniques significantly improve the efficiency\nof model pretraining and performance of downstream tasks. Compared to RoBERTa-Large, a DeBERTa model trained on half of\nthe training data performs consistently better on a wide range of NLP tasks, achieving improvements on MNLI by +0.9%\n(90.2% vs. 91.1%), on SQuAD v2.0 by +2.3% (88.4% vs. 90.7%) and RACE by +3.6% (83.2% vs. 86.8%). The DeBERTa code and\npre-trained models will be made publicly available at https://github.com/microsoft/DeBERTa.*\n\n\nThe following information is visible directly on the [original implementation\nrepository](https://github.com/microsoft/DeBERTa). DeBERTa v2 is the second version of the DeBERTa model. It includes\nthe 1.5B model used for the SuperGLUE single-model submission and achieving 89.9, versus human baseline 89.8. You can\nfind more details about this submission in the authors'\n[blog](https://www.microsoft.com/en-us/research/blog/microsoft-deberta-surpasses-human-performance-on-the-superglue-benchmark/)\n\nNew in v2:\n\n- **Vocabulary** In v2 the tokenizer is changed to use a new vocabulary of size 128K built from the training data.\n  Instead of a GPT2-based tokenizer, the tokenizer is now\n  [sentencepiece-based](https://github.com/google/sentencepiece) tokenizer.\n- **nGiE(nGram Induced Input Encoding)** The DeBERTa-v2 model uses an additional convolution layer aside with the first\n  transformer layer to better learn the local dependency of input tokens.\n- **Sharing position projection matrix with content projection matrix in attention layer** Based on previous\n  experiments, this can save parameters without affecting the performance.\n- **Apply bucket to encode relative positions** The DeBERTa-v2 model uses log bucket to encode relative positions\n  similar to T5.\n- **900M model & 1.5B model** Two additional model sizes are available: 900M and 1.5B, which significantly improves the\n  performance of downstream tasks.\n\nThis model was contributed by [DeBERTa](https://huggingface.co/DeBERTa)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 308c0ac3-6452-458c-881c-c6aa986427f2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/deberta-v2", "title": "DeBERTa-v2", "tokens": 1600, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"308c0ac3-6452-458c-881c-c6aa986427f2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deberta-v2\", \"title\": \"DeBERTa-v2\", \"tokens\": 1600, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"48d7be7f-faba-4d81-a3f6-9657d9866e78\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deberta-v2\", \"title\": \"DeBERTa-v2\", \"tokens\": 1600, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a2b0a3f7da958b59525f998fad765610dcae6d310d2e5b34c37ee1d9dd99673f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f9a8ba69-b972-4790-9ac6-c84bde842107\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deberta-v2\", \"title\": \"DeBERTa-v2\", \"tokens\": 1600, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9dcf18ab7d28fa214fd4eb62252653e9e79c6043ff3affb1e9f902b33e09b1e3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2436728d-7440-46c3-a5e2-56c0177fb27c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"dd5f0f9b6e3dc3970365ecc33f1ef9975b636bd5c84316f667c8b3ce5db19402\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1736, \"end_char_idx\": 3409, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "48d7be7f-faba-4d81-a3f6-9657d9866e78", "doc_id": "48d7be7f-faba-4d81-a3f6-9657d9866e78", "ref_doc_id": "48d7be7f-faba-4d81-a3f6-9657d9866e78", "chroma:document": "The following information is visible directly on the [original implementation\nrepository](https://github.com/microsoft/DeBERTa). DeBERTa v2 is the second version of the DeBERTa model. It includes\nthe 1.5B model used for the SuperGLUE single-model submission and achieving 89.9, versus human baseline 89.8. You can\nfind more details about this submission in the authors'\n[blog](https://www.microsoft.com/en-us/research/blog/microsoft-deberta-surpasses-human-performance-on-the-superglue-benchmark/)\n\nNew in v2:\n\n- **Vocabulary** In v2 the tokenizer is changed to use a new vocabulary of size 128K built from the training data.\n  Instead of a GPT2-based tokenizer, the tokenizer is now\n  [sentencepiece-based](https://github.com/google/sentencepiece) tokenizer.\n- **nGiE(nGram Induced Input Encoding)** The DeBERTa-v2 model uses an additional convolution layer aside with the first\n  transformer layer to better learn the local dependency of input tokens.\n- **Sharing position projection matrix with content projection matrix in attention layer** Based on previous\n  experiments, this can save parameters without affecting the performance.\n- **Apply bucket to encode relative positions** The DeBERTa-v2 model uses log bucket to encode relative positions\n  similar to T5.\n- **900M model & 1.5B model** Two additional model sizes are available: 900M and 1.5B, which significantly improves the\n  performance of downstream tasks.\n\nThis model was contributed by [DeBERTa](https://huggingface.co/DeBERTa). This model TF 2.0 implementation was\ncontributed by [kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/microsoft/DeBERTa)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 2436728d-7440-46c3-a5e2-56c0177fb27c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/deberta-v2", "title": "DeBERTa-v2", "tokens": 1600, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2436728d-7440-46c3-a5e2-56c0177fb27c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deberta-v2\", \"title\": \"DeBERTa-v2\", \"tokens\": 1600, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"48d7be7f-faba-4d81-a3f6-9657d9866e78\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deberta-v2\", \"title\": \"DeBERTa-v2\", \"tokens\": 1600, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a2b0a3f7da958b59525f998fad765610dcae6d310d2e5b34c37ee1d9dd99673f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"308c0ac3-6452-458c-881c-c6aa986427f2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deberta-v2\", \"title\": \"DeBERTa-v2\", \"tokens\": 1600, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8199d0506116bfa5c5c9eb7073030c49fcb48bd014b8ad8524ac907b26abdaa5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1865, \"end_char_idx\": 5383, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "48d7be7f-faba-4d81-a3f6-9657d9866e78", "doc_id": "48d7be7f-faba-4d81-a3f6-9657d9866e78", "ref_doc_id": "48d7be7f-faba-4d81-a3f6-9657d9866e78", "chroma:document": "DeBERTa v2 is the second version of the DeBERTa model. It includes\nthe 1.5B model used for the SuperGLUE single-model submission and achieving 89.9, versus human baseline 89.8. You can\nfind more details about this submission in the authors'\n[blog](https://www.microsoft.com/en-us/research/blog/microsoft-deberta-surpasses-human-performance-on-the-superglue-benchmark/)\n\nNew in v2:\n\n- **Vocabulary** In v2 the tokenizer is changed to use a new vocabulary of size 128K built from the training data.\n  Instead of a GPT2-based tokenizer, the tokenizer is now\n  [sentencepiece-based](https://github.com/google/sentencepiece) tokenizer.\n- **nGiE(nGram Induced Input Encoding)** The DeBERTa-v2 model uses an additional convolution layer aside with the first\n  transformer layer to better learn the local dependency of input tokens.\n- **Sharing position projection matrix with content projection matrix in attention layer** Based on previous\n  experiments, this can save parameters without affecting the performance.\n- **Apply bucket to encode relative positions** The DeBERTa-v2 model uses log bucket to encode relative positions\n  similar to T5.\n- **900M model & 1.5B model** Two additional model sizes are available: 900M and 1.5B, which significantly improves the\n  performance of downstream tasks.\n\nThis model was contributed by [DeBERTa](https://huggingface.co/DeBERTa). This model TF 2.0 implementation was\ncontributed by [kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/microsoft/DeBERTa).\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## DebertaV2Config\n\n[[autodoc]] DebertaV2Config\n\n## DebertaV2Tokenizer\n\n[[autodoc]] DebertaV2Tokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## DebertaV2TokenizerFast\n\n[[autodoc]] DebertaV2TokenizerFast\n    - build_inputs_with_special_tokens\n    - create_token_type_ids_from_sequences\n\n<frameworkcontent>\n<pt>\n\n## DebertaV2Model\n\n[[autodoc]] DebertaV2Model\n    - forward\n\n## DebertaV2PreTrainedModel\n\n[[autodoc]] DebertaV2PreTrainedModel\n    - forward\n\n## DebertaV2ForMaskedLM\n\n[[autodoc]] DebertaV2ForMaskedLM\n    - forward\n\n## DebertaV2ForSequenceClassification\n\n[[autodoc]] DebertaV2ForSequenceClassification\n    - forward\n\n## DebertaV2ForTokenClassification\n\n[[autodoc]] DebertaV2ForTokenClassification\n    - forward\n\n## DebertaV2ForQuestionAnswering\n\n[[autodoc]] DebertaV2ForQuestionAnswering\n    - forward\n\n## DebertaV2ForMultipleChoice\n\n[[autodoc]] DebertaV2ForMultipleChoice\n    - forward\n\n</pt>\n<tf>\n\n## TFDebertaV2Model\n\n[[autodoc]] TFDebertaV2Model\n    - call\n\n## TFDebertaV2PreTrainedModel\n\n[[autodoc]] TFDebertaV2PreTrainedModel\n    - call\n\n## TFDebertaV2ForMaskedLM\n\n[[autodoc]] TFDebertaV2ForMaskedLM\n    - call\n\n## TFDebertaV2ForSequenceClassification\n\n[[autodoc]] TFDebertaV2ForSequenceClassification\n    - call\n\n## TFDebertaV2ForTokenClassification\n\n[[autodoc]] TFDebertaV2ForTokenClassification\n    - call\n\n## TFDebertaV2ForQuestionAnswering\n\n[[autodoc]] TFDebertaV2ForQuestionAnswering\n    - call\n\n## TFDebertaV2ForMultipleChoice\n\n[[autodoc]] TFDebertaV2ForMultipleChoice\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: b8bedfe5-feb1-4e0b-bcc0-c3fe71775f06
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/chinese_clip", "title": "Chinese-CLIP", "tokens": 1360, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b8bedfe5-feb1-4e0b-bcc0-c3fe71775f06\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/chinese_clip\", \"title\": \"Chinese-CLIP\", \"tokens\": 1360, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cf55ed01-ba10-4c53-a99f-402b16d63c90\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/chinese_clip\", \"title\": \"Chinese-CLIP\", \"tokens\": 1360, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7be88278ac429548bc0d58c1994b3bb78b3b6e719e977a7a3b2651371ee5e6c1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"97d81c10-e8a3-422a-8708-47bdbbd91c99\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"17f2bd829d8c2ac5492787b81b2441a879e5e6c1cbde4e13e95adfa66ac5bb1f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1854, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cf55ed01-ba10-4c53-a99f-402b16d63c90", "doc_id": "cf55ed01-ba10-4c53-a99f-402b16d63c90", "ref_doc_id": "cf55ed01-ba10-4c53-a99f-402b16d63c90", "chroma:document": "# Chinese-CLIP\n\n## Overview\n\nThe Chinese-CLIP model was proposed in [Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese](https://arxiv.org/abs/2211.01335) by An Yang, Junshu Pan, Junyang Lin, Rui Men, Yichang Zhang, Jingren Zhou, Chang Zhou.\nChinese-CLIP is an implementation of CLIP (Radford et al., 2021) on a large-scale dataset of Chinese image-text pairs. It is capable of performing cross-modal retrieval and also playing as a vision backbone for vision tasks like zero-shot image classification, open-domain object detection, etc. The original Chinese-CLIP code is released [at this link](https://github.com/OFA-Sys/Chinese-CLIP).\n\nThe abstract from the paper is the following:\n\n*The tremendous success of CLIP (Radford et al., 2021) has promoted the research and application of contrastive learning for vision-language pretraining. In this work, we construct a large-scale dataset of image-text pairs in Chinese, where most data are retrieved from publicly available datasets, and we pretrain Chinese CLIP models on the new dataset. We develop 5 Chinese CLIP models of multiple sizes, spanning from 77 to 958 million parameters. Furthermore, we propose a two-stage pretraining method, where the model is first trained with the image encoder frozen and then trained with all parameters being optimized, to achieve enhanced model performance. Our comprehensive experiments demonstrate that Chinese CLIP can achieve the state-of-the-art performance on MUGE, Flickr30K-CN, and COCO-CN in the setups of zero-shot learning and finetuning, and it is able to achieve competitive performance in zero-shot image classification based on the evaluation on the ELEVATER benchmark (Li et al., 2022). Our codes, pretrained models, and demos have been released.*\n\nThe Chinese-CLIP model was contributed by [OFA-Sys](https://huggingface.co/OFA-Sys)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 97d81c10-e8a3-422a-8708-47bdbbd91c99
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/chinese_clip", "title": "Chinese-CLIP", "tokens": 1360, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"97d81c10-e8a3-422a-8708-47bdbbd91c99\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/chinese_clip\", \"title\": \"Chinese-CLIP\", \"tokens\": 1360, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cf55ed01-ba10-4c53-a99f-402b16d63c90\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/chinese_clip\", \"title\": \"Chinese-CLIP\", \"tokens\": 1360, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7be88278ac429548bc0d58c1994b3bb78b3b6e719e977a7a3b2651371ee5e6c1\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b8bedfe5-feb1-4e0b-bcc0-c3fe71775f06\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/chinese_clip\", \"title\": \"Chinese-CLIP\", \"tokens\": 1360, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"241507b6ecfee96fb26dd0b1c9eb6f233998d68861ed27921da9e12ae5633bd1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 256, \"end_char_idx\": 4559, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cf55ed01-ba10-4c53-a99f-402b16d63c90", "doc_id": "cf55ed01-ba10-4c53-a99f-402b16d63c90", "ref_doc_id": "cf55ed01-ba10-4c53-a99f-402b16d63c90", "chroma:document": "Chinese-CLIP is an implementation of CLIP (Radford et al., 2021) on a large-scale dataset of Chinese image-text pairs. It is capable of performing cross-modal retrieval and also playing as a vision backbone for vision tasks like zero-shot image classification, open-domain object detection, etc. The original Chinese-CLIP code is released [at this link](https://github.com/OFA-Sys/Chinese-CLIP).\n\nThe abstract from the paper is the following:\n\n*The tremendous success of CLIP (Radford et al., 2021) has promoted the research and application of contrastive learning for vision-language pretraining. In this work, we construct a large-scale dataset of image-text pairs in Chinese, where most data are retrieved from publicly available datasets, and we pretrain Chinese CLIP models on the new dataset. We develop 5 Chinese CLIP models of multiple sizes, spanning from 77 to 958 million parameters. Furthermore, we propose a two-stage pretraining method, where the model is first trained with the image encoder frozen and then trained with all parameters being optimized, to achieve enhanced model performance. Our comprehensive experiments demonstrate that Chinese CLIP can achieve the state-of-the-art performance on MUGE, Flickr30K-CN, and COCO-CN in the setups of zero-shot learning and finetuning, and it is able to achieve competitive performance in zero-shot image classification based on the evaluation on the ELEVATER benchmark (Li et al., 2022). Our codes, pretrained models, and demos have been released.*\n\nThe Chinese-CLIP model was contributed by [OFA-Sys](https://huggingface.co/OFA-Sys).\n\n## Usage example\n\nThe code snippet below shows how to compute image & text features and similarities:\n\n```python\n>>> from PIL import Image\n>>> import requests\n>>> from transformers import ChineseCLIPProcessor, ChineseCLIPModel\n\n>>> model = ChineseCLIPModel.from_pretrained(\"OFA-Sys/chinese-clip-vit-base-patch16\")\n>>> processor = ChineseCLIPProcessor.from_pretrained(\"OFA-Sys/chinese-clip-vit-base-patch16\")\n\n>>> url = \"https://clip-cn-beijing.oss-cn-beijing.aliyuncs.com/pokemon.jpeg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n>>> # Squirtle, Bulbasaur, Charmander, Pikachu in English\n>>> texts = [\"\u6770\u5c3c\u9f9f\", \"\u5999\u86d9\u79cd\u5b50\", \"\u5c0f\u706b\u9f99\", \"\u76ae\u5361\u4e18\"]\n\n>>> # compute image feature\n>>> inputs = processor(images=image, return_tensors=\"pt\")\n>>> image_features = model.get_image_features(**inputs)\n>>> image_features = image_features / image_features.norm(p=2, dim=-1, keepdim=True)  # normalize\n\n>>> # compute text features\n>>> inputs = processor(text=texts, padding=True, return_tensors=\"pt\")\n>>> text_features = model.get_text_features(**inputs)\n>>> text_features = text_features / text_features.norm(p=2, dim=-1, keepdim=True)  # normalize\n\n>>> # compute image-text similarity scores\n>>> inputs = processor(text=texts, images=image, return_tensors=\"pt\", padding=True)\n>>> outputs = model(**inputs)\n>>> logits_per_image = outputs.logits_per_image  # this is the image-text similarity score\n>>> probs = logits_per_image.softmax(dim=1)  # probs: [[1.2686e-03, 5.4499e-02, 6.7968e-04, 9.4355e-01]]\n```\n\nCurrently, following scales of pretrained Chinese-CLIP models are available on \ud83e\udd17 Hub:\n\n- [OFA-Sys/chinese-clip-vit-base-patch16](https://huggingface.co/OFA-Sys/chinese-clip-vit-base-patch16)\n- [OFA-Sys/chinese-clip-vit-large-patch14](https://huggingface.co/OFA-Sys/chinese-clip-vit-large-patch14)\n- [OFA-Sys/chinese-clip-vit-large-patch14-336px](https://huggingface.co/OFA-Sys/chinese-clip-vit-large-patch14-336px)\n- [OFA-Sys/chinese-clip-vit-huge-patch14](https://huggingface.co/OFA-Sys/chinese-clip-vit-huge-patch14)\n\n## ChineseCLIPConfig\n\n[[autodoc]] ChineseCLIPConfig\n    - from_text_vision_configs\n\n## ChineseCLIPTextConfig\n\n[[autodoc]] ChineseCLIPTextConfig\n\n## ChineseCLIPVisionConfig\n\n[[autodoc]] ChineseCLIPVisionConfig\n\n## ChineseCLIPImageProcessor\n\n[[autodoc]] ChineseCLIPImageProcessor\n    - preprocess\n\n## ChineseCLIPFeatureExtractor\n\n[[autodoc]] ChineseCLIPFeatureExtractor\n\n## ChineseCLIPProcessor\n\n[[autodoc]] ChineseCLIPProcessor\n\n## ChineseCLIPModel\n\n[[autodoc]] ChineseCLIPModel\n    - forward\n    - get_text_features\n    - get_image_features\n\n## ChineseCLIPTextModel\n\n[[autodoc]] ChineseCLIPTextModel\n    - forward\n\n## ChineseCLIPVisionModel\n\n[[autodoc]] ChineseCLIPVisionModel\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 50215eac-4d5e-4ebe-b4fe-5c9cd7201c47
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/phobert", "title": "PhoBERT", "tokens": 619, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"50215eac-4d5e-4ebe-b4fe-5c9cd7201c47\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phobert\", \"title\": \"PhoBERT\", \"tokens\": 619, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"49b99832-036d-48ce-b095-a41b4e92e377\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phobert\", \"title\": \"PhoBERT\", \"tokens\": 619, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2ecd4a04b3b65c3cbb24f9a6baf9ba35c728b34ef27c378d76ee1e69a7d6ebf2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1819, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "49b99832-036d-48ce-b095-a41b4e92e377", "doc_id": "49b99832-036d-48ce-b095-a41b4e92e377", "ref_doc_id": "49b99832-036d-48ce-b095-a41b4e92e377", "chroma:document": "# PhoBERT\n\n## Overview\n\nThe PhoBERT model was proposed in [PhoBERT: Pre-trained language models for Vietnamese](https://www.aclweb.org/anthology/2020.findings-emnlp.92.pdf) by Dat Quoc Nguyen, Anh Tuan Nguyen.\n\nThe abstract from the paper is the following:\n\n*We present PhoBERT with two versions, PhoBERT-base and PhoBERT-large, the first public large-scale monolingual\nlanguage models pre-trained for Vietnamese. Experimental results show that PhoBERT consistently outperforms the recent\nbest pre-trained multilingual model XLM-R (Conneau et al., 2020) and improves the state-of-the-art in multiple\nVietnamese-specific NLP tasks including Part-of-speech tagging, Dependency parsing, Named-entity recognition and\nNatural language inference.*\n\nThis model was contributed by [dqnguyen](https://huggingface.co/dqnguyen). The original code can be found [here](https://github.com/VinAIResearch/PhoBERT).\n\n## Usage example\n\n```python\n>>> import torch\n>>> from transformers import AutoModel, AutoTokenizer\n\n>>> phobert = AutoModel.from_pretrained(\"vinai/phobert-base\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"vinai/phobert-base\")\n\n>>> # INPUT TEXT MUST BE ALREADY WORD-SEGMENTED!\n>>> line = \"T\u00f4i l\u00e0 sinh_vi\u00ean tr\u01b0\u1eddng \u0111\u1ea1i_h\u1ecdc C\u00f4ng_ngh\u1ec7 .\"\n\n>>> input_ids = torch.tensor([tokenizer.encode(line)])\n\n>>> with torch.no_grad():\n...     features = phobert(input_ids)  # Models outputs are now tuples\n\n>>> # With TensorFlow 2.0+:\n>>> # from transformers import TFAutoModel\n>>> # phobert = TFAutoModel.from_pretrained(\"vinai/phobert-base\")\n```\n\n<Tip> \n\nPhoBERT implementation is the same as BERT, except for tokenization. Refer to [EART documentation](bert) for information on \nconfiguration classes and their parameters. PhoBERT-specific tokenizer is documented below.  \n\n</Tip>\n\n## PhobertTokenizer\n\n[[autodoc]] PhobertTokenizer"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 701fc691-e684-477d-9484-d7e70a03c357
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/fuyu", "title": "Fuyu", "tokens": 1342, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"701fc691-e684-477d-9484-d7e70a03c357\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fuyu\", \"title\": \"Fuyu\", \"tokens\": 1342, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a4cd5900-b582-42bb-ab60-af2099564227\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fuyu\", \"title\": \"Fuyu\", \"tokens\": 1342, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9f69fb8ba7f94fc215cb82cfaf869c3e94d5935701b4afce105e805f35f7140c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c0d2b0dd-fbc2-4abc-b147-225727d934f1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"df19df25a4103fd5170846116d2543e7497198f30e0a85086934b9c093a04e62\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1802, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a4cd5900-b582-42bb-ab60-af2099564227", "doc_id": "a4cd5900-b582-42bb-ab60-af2099564227", "ref_doc_id": "a4cd5900-b582-42bb-ab60-af2099564227", "chroma:document": "# Fuyu\n\n## Overview\n\nThe Fuyu model was created by [ADEPT](https://www.adept.ai/blog/fuyu-8b), and authored by Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, Sa\u011fnak Ta\u015f\u0131rlar. \n\nThe authors introduced Fuyu-8B, a decoder-only multimodal model based on the classic transformers architecture, with query and key normalization. A linear encoder is added to create multimodal embeddings from image inputs. \n\nBy treating image tokens like text tokens and using a special image-newline character, the model knows when an image line ends. Image positional embeddings are removed. This avoids the need for different training phases for various image resolutions. With 8 billion parameters and licensed under CC-BY-NC, Fuyu-8B is notable for its ability to handle both text and images, its impressive context size of 16K, and its overall performance.\n\n<Tip warning={true}>\n\nThe `Fuyu` models were trained using `bfloat16`, but the original inference uses `float16` The checkpoints uploaded on the hub use `torch_dtype = 'float16'` which will be\nused by the `AutoModel` API to cast the checkpoints from `torch.float32` to `torch.float16`. \n\nThe `dtype` of the online weights is mostly irrelevant, unless you are using `torch_dtype=\"auto\"` when initializing a model using `model = AutoModelForCausalLM.from_pretrained(\"path\", torch_dtype = \"auto\")`. The reason is that the model will first be downloaded ( using the `dtype` of the checkpoints online) then it will be cast to the default `dtype` of `torch` (becomes `torch.float32`). Users should specify the `torch_dtype` they want, and if they don't it will be `torch.float32`.\n\nFinetuning the model in `float16` is not recommended and known to produce `nan`, as such the model should be fine-tuned in `bfloat16`.\n\n</Tip>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: c0d2b0dd-fbc2-4abc-b147-225727d934f1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/fuyu", "title": "Fuyu", "tokens": 1342, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c0d2b0dd-fbc2-4abc-b147-225727d934f1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fuyu\", \"title\": \"Fuyu\", \"tokens\": 1342, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a4cd5900-b582-42bb-ab60-af2099564227\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fuyu\", \"title\": \"Fuyu\", \"tokens\": 1342, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9f69fb8ba7f94fc215cb82cfaf869c3e94d5935701b4afce105e805f35f7140c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"701fc691-e684-477d-9484-d7e70a03c357\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fuyu\", \"title\": \"Fuyu\", \"tokens\": 1342, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"08d439a23bae5a23b4c95547af01066a1222851ac9681ddb64fb35e72215b56c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1805, \"end_char_idx\": 4448, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a4cd5900-b582-42bb-ab60-af2099564227", "doc_id": "a4cd5900-b582-42bb-ab60-af2099564227", "ref_doc_id": "a4cd5900-b582-42bb-ab60-af2099564227", "chroma:document": "Tips:\n\n- To convert the model, you need to clone the original repository using `git clone https://github.com/persimmon-ai-labs/adept-inference`, then get the checkpoints:\n\n```bash\ngit clone https://github.com/persimmon-ai-labs/adept-inference\nwget path/to/fuyu-8b-model-weights.tar\ntar -xvf fuyu-8b-model-weights.tar\npython src/transformers/models/fuyu/convert_fuyu_weights_to_hf.py  --input_dir /path/to/downloaded/fuyu/weights/ --output_dir /output/path \\\n    --pt_model_path /path/to/fuyu_8b_release/iter_0001251/mp_rank_00/model_optim_rng.pt\n    --ada_lib_path /path/to/adept-inference\n```\n\nFor the chat model:\n```bash\nwget https://axtkn4xl5cip.objectstorage.us-phoenix-1.oci.customer-oci.com/n/axtkn4xl5cip/b/adept-public-data/o/8b_chat_model_release.tar\ntar -xvf 8b_base_model_release.tar\n```\nThen, model can be loaded via:\n\n```py \nfrom transformers import FuyuConfig, FuyuForCausalLM\nmodel_config = FuyuConfig()\nmodel = FuyuForCausalLM(model_config).from_pretrained('/output/path')\n```\n\nInputs need to be passed through a specific Processor to have the correct formats.\nA processor requires an image_processor and a tokenizer. Hence, inputs can be loaded via:\n\n```py\nfrom PIL import Image\nfrom transformers import AutoTokenizer\nfrom transformers.models.fuyu.processing_fuyu import FuyuProcessor\nfrom transformers.models.fuyu.image_processing_fuyu import FuyuImageProcessor\n\n\ntokenizer = AutoTokenizer.from_pretrained('adept-hf-collab/fuyu-8b')\nimage_processor = FuyuImageProcessor()\n\n\nprocessor = FuyuProcessor(image_processor=image_processor, tokenizer=tokenizer)\ntext_prompt = \"Generate a coco-style caption.\\\\n\"\n\nbus_image_url = \"https://huggingface.co/datasets/hf-internal-testing/fixtures-captioning/resolve/main/bus.png\"\nbus_image_pil = Image.open(io.BytesIO(requests.get(bus_image_url).content))\ninputs_to_model = processor(text=text_prompt, images=bus_image_pil)\n\n\n```\n\nThis model was contributed by [Molbap](https://huggingface.co/Molbap).\nThe original code can be found [here](https://github.com/persimmon-ai-labs/adept-inference).\n\n- Fuyu uses a `sentencepiece` based tokenizer, with a `Unigram` model. It supports bytefallback, which is only available in `tokenizers==0.14.0` for the fast tokenizer.\nThe `LlamaTokenizer` is used as it is a standard wrapper around sentencepiece. \n\n- The authors suggest to use the following prompt for image captioning: `f\"Generate a coco-style caption.\\\\n\"`\n\n\n## FuyuConfig\n\n[[autodoc]] FuyuConfig\n\n## FuyuForCausalLM\n\n[[autodoc]] FuyuForCausalLM\n    - forward\n\n## FuyuImageProcessor\n\n[[autodoc]] FuyuImageProcessor\n    - __call__\n\n## FuyuProcessor\n\n[[autodoc]] FuyuProcessor\n    - __call__"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 8df1f6bb-6f05-403c-8ba9-5f42a587bd50
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vision-encoder-decoder", "title": "Vision Encoder Decoder Models", "tokens": 1947, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8df1f6bb-6f05-403c-8ba9-5f42a587bd50\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vision-encoder-decoder\", \"title\": \"Vision Encoder Decoder Models\", \"tokens\": 1947, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"86adba1a-edb0-487f-8d4e-64df592443ac\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vision-encoder-decoder\", \"title\": \"Vision Encoder Decoder Models\", \"tokens\": 1947, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"10a79f49aa80ba46749be3a4aa3a3c92ca07fef4f06196be9609dc10569a1691\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3b08b938-9444-4e5a-924c-b7bdd2e05222\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f8b4d53639728d9d0b0b5af771050b8218b32ab0696b56d5d4cfb8a5b0d7f134\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3285, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "86adba1a-edb0-487f-8d4e-64df592443ac", "doc_id": "86adba1a-edb0-487f-8d4e-64df592443ac", "ref_doc_id": "86adba1a-edb0-487f-8d4e-64df592443ac", "chroma:document": "# Vision Encoder Decoder Models\n\n## Overview\n\nThe [`VisionEncoderDecoderModel`] can be used to initialize an image-to-text model with any\npretrained Transformer-based vision model as the encoder (*e.g.* [ViT](vit), [BEiT](beit), [DeiT](deit), [Swin](swin))\nand any pretrained language model as the decoder (*e.g.* [RoBERTa](roberta), [GPT2](gpt2), [BERT](bert), [DistilBERT](distilbert)).\n\nThe effectiveness of initializing image-to-text-sequence models with pretrained checkpoints has been shown in (for\nexample) [TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models](https://arxiv.org/abs/2109.10282) by Minghao Li, Tengchao Lv, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang,\nZhoujun Li, Furu Wei.\n\nAfter such a [`VisionEncoderDecoderModel`] has been trained/fine-tuned, it can be saved/loaded just like any other models (see the examples below\nfor more information).\n\nAn example application is image captioning, in which the encoder is used to encode the image, after which an autoregressive language model generates\nthe caption. Another example is optical character recognition. Refer to [TrOCR](trocr), which is an instance of [`VisionEncoderDecoderModel`].\n\n## Randomly initializing `VisionEncoderDecoderModel` from model configurations.\n\n[`VisionEncoderDecoderModel`] can be randomly initialized from an encoder and a decoder config. In the following example, we show how to do this using the default [`ViTModel`] configuration for the encoder\nand the default [`BertForCausalLM`] configuration for the decoder.\n\n```python\n>>> from transformers import BertConfig, ViTConfig, VisionEncoderDecoderConfig, VisionEncoderDecoderModel\n\n>>> config_encoder = ViTConfig()\n>>> config_decoder = BertConfig()\n\n>>> config = VisionEncoderDecoderConfig.from_encoder_decoder_configs(config_encoder, config_decoder)\n>>> model = VisionEncoderDecoderModel(config=config)\n```\n\n## Initialising `VisionEncoderDecoderModel` from a pretrained encoder and a pretrained decoder.\n\n[`VisionEncoderDecoderModel`] can be initialized from a pretrained encoder checkpoint and a pretrained decoder checkpoint. Note that any pretrained Transformer-based vision model, *e.g.* [Swin](swin), can serve as the encoder and both pretrained auto-encoding models, *e.g.* BERT, pretrained causal language models, *e.g.* GPT2, as well as the pretrained decoder part of sequence-to-sequence models, *e.g.* decoder of BART, can be used as the decoder.\nDepending on which architecture you choose as the decoder, the cross-attention layers might be randomly initialized.\nInitializing [`VisionEncoderDecoderModel`] from a pretrained encoder and decoder checkpoint requires the model to be fine-tuned on a downstream task, as has been shown in [the *Warm-starting-encoder-decoder blog post*](https://huggingface.co/blog/warm-starting-encoder-decoder).\nTo do so, the `VisionEncoderDecoderModel` class provides a [`VisionEncoderDecoderModel.from_encoder_decoder_pretrained`] method.\n\n```python\n>>> from transformers import VisionEncoderDecoderModel\n\n>>> model = VisionEncoderDecoderModel.from_encoder_decoder_pretrained(\n...     \"microsoft/swin-base-patch4-window7-224-in22k\", \"google-bert/bert-base-uncased\"\n... )\n```\n\n## Loading an existing `VisionEncoderDecoderModel` checkpoint and perform inference."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 3b08b938-9444-4e5a-924c-b7bdd2e05222
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vision-encoder-decoder", "title": "Vision Encoder Decoder Models", "tokens": 1947, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3b08b938-9444-4e5a-924c-b7bdd2e05222\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vision-encoder-decoder\", \"title\": \"Vision Encoder Decoder Models\", \"tokens\": 1947, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"86adba1a-edb0-487f-8d4e-64df592443ac\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vision-encoder-decoder\", \"title\": \"Vision Encoder Decoder Models\", \"tokens\": 1947, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"10a79f49aa80ba46749be3a4aa3a3c92ca07fef4f06196be9609dc10569a1691\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8df1f6bb-6f05-403c-8ba9-5f42a587bd50\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vision-encoder-decoder\", \"title\": \"Vision Encoder Decoder Models\", \"tokens\": 1947, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5e5c4a2f6789647c09e90e846df3a33a64648af144c2ef8f3c57ff409aa02fd0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a1813132-d2ed-4628-a93c-178519631d23\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ae11813f277c9263e608bb4adf35038e84abe053db2eb15028784c20896a79dc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1546, \"end_char_idx\": 5022, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "86adba1a-edb0-487f-8d4e-64df592443ac", "doc_id": "86adba1a-edb0-487f-8d4e-64df592443ac", "ref_doc_id": "86adba1a-edb0-487f-8d4e-64df592443ac", "chroma:document": "```python\n>>> from transformers import BertConfig, ViTConfig, VisionEncoderDecoderConfig, VisionEncoderDecoderModel\n\n>>> config_encoder = ViTConfig()\n>>> config_decoder = BertConfig()\n\n>>> config = VisionEncoderDecoderConfig.from_encoder_decoder_configs(config_encoder, config_decoder)\n>>> model = VisionEncoderDecoderModel(config=config)\n```\n\n## Initialising `VisionEncoderDecoderModel` from a pretrained encoder and a pretrained decoder.\n\n[`VisionEncoderDecoderModel`] can be initialized from a pretrained encoder checkpoint and a pretrained decoder checkpoint. Note that any pretrained Transformer-based vision model, *e.g.* [Swin](swin), can serve as the encoder and both pretrained auto-encoding models, *e.g.* BERT, pretrained causal language models, *e.g.* GPT2, as well as the pretrained decoder part of sequence-to-sequence models, *e.g.* decoder of BART, can be used as the decoder.\nDepending on which architecture you choose as the decoder, the cross-attention layers might be randomly initialized.\nInitializing [`VisionEncoderDecoderModel`] from a pretrained encoder and decoder checkpoint requires the model to be fine-tuned on a downstream task, as has been shown in [the *Warm-starting-encoder-decoder blog post*](https://huggingface.co/blog/warm-starting-encoder-decoder).\nTo do so, the `VisionEncoderDecoderModel` class provides a [`VisionEncoderDecoderModel.from_encoder_decoder_pretrained`] method.\n\n```python\n>>> from transformers import VisionEncoderDecoderModel\n\n>>> model = VisionEncoderDecoderModel.from_encoder_decoder_pretrained(\n...     \"microsoft/swin-base-patch4-window7-224-in22k\", \"google-bert/bert-base-uncased\"\n... )\n```\n\n## Loading an existing `VisionEncoderDecoderModel` checkpoint and perform inference.\n\nTo load fine-tuned checkpoints of the `VisionEncoderDecoderModel` class, [`VisionEncoderDecoderModel`] provides the `from_pretrained(...)` method just like any other model architecture in Transformers.\n\nTo perform inference, one uses the [`generate`] method, which allows to autoregressively generate text. This method supports various forms of decoding, such as greedy, beam search and multinomial sampling.\n\n```python\n>>> import requests\n>>> from PIL import Image\n\n>>> from transformers import GPT2TokenizerFast, ViTImageProcessor, VisionEncoderDecoderModel\n\n>>> # load a fine-tuned image captioning model and corresponding tokenizer and image processor\n>>> model = VisionEncoderDecoderModel.from_pretrained(\"nlpconnect/vit-gpt2-image-captioning\")\n>>> tokenizer = GPT2TokenizerFast.from_pretrained(\"nlpconnect/vit-gpt2-image-captioning\")\n>>> image_processor = ViTImageProcessor.from_pretrained(\"nlpconnect/vit-gpt2-image-captioning\")\n\n>>> # let's perform inference on an image\n>>> url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n>>> pixel_values = image_processor(image, return_tensors=\"pt\").pixel_values\n\n>>> # autoregressively generate caption (uses greedy decoding by default)\n>>> generated_ids = model.generate(pixel_values)\n>>> generated_text = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]\n>>> print(generated_text)\na cat laying on a blanket next to a cat laying on a bed\n```\n\n## Loading a PyTorch checkpoint into `TFVisionEncoderDecoderModel`.\n\n[`TFVisionEncoderDecoderModel.from_pretrained`] currently doesn't support initializing the model from a\nPyTorch checkpoint. Passing `from_pt=True` to this method will throw an exception."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a1813132-d2ed-4628-a93c-178519631d23
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vision-encoder-decoder", "title": "Vision Encoder Decoder Models", "tokens": 1947, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a1813132-d2ed-4628-a93c-178519631d23\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vision-encoder-decoder\", \"title\": \"Vision Encoder Decoder Models\", \"tokens\": 1947, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"86adba1a-edb0-487f-8d4e-64df592443ac\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vision-encoder-decoder\", \"title\": \"Vision Encoder Decoder Models\", \"tokens\": 1947, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"10a79f49aa80ba46749be3a4aa3a3c92ca07fef4f06196be9609dc10569a1691\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3b08b938-9444-4e5a-924c-b7bdd2e05222\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vision-encoder-decoder\", \"title\": \"Vision Encoder Decoder Models\", \"tokens\": 1947, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f42acd7c56a246d61d9e6cba3b14c8d230269b97cf3f0d42f7eb096816c56e1f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1ec962fb-6f3b-4320-835d-805269e488ba\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f9c69b04f585b9f72c6c09ebeeba2531c7e37a4b5ac1766d4cddc3c8de2fd2d2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3287, \"end_char_idx\": 6038, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "86adba1a-edb0-487f-8d4e-64df592443ac", "doc_id": "86adba1a-edb0-487f-8d4e-64df592443ac", "ref_doc_id": "86adba1a-edb0-487f-8d4e-64df592443ac", "chroma:document": "To load fine-tuned checkpoints of the `VisionEncoderDecoderModel` class, [`VisionEncoderDecoderModel`] provides the `from_pretrained(...)` method just like any other model architecture in Transformers.\n\nTo perform inference, one uses the [`generate`] method, which allows to autoregressively generate text. This method supports various forms of decoding, such as greedy, beam search and multinomial sampling.\n\n```python\n>>> import requests\n>>> from PIL import Image\n\n>>> from transformers import GPT2TokenizerFast, ViTImageProcessor, VisionEncoderDecoderModel\n\n>>> # load a fine-tuned image captioning model and corresponding tokenizer and image processor\n>>> model = VisionEncoderDecoderModel.from_pretrained(\"nlpconnect/vit-gpt2-image-captioning\")\n>>> tokenizer = GPT2TokenizerFast.from_pretrained(\"nlpconnect/vit-gpt2-image-captioning\")\n>>> image_processor = ViTImageProcessor.from_pretrained(\"nlpconnect/vit-gpt2-image-captioning\")\n\n>>> # let's perform inference on an image\n>>> url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n>>> pixel_values = image_processor(image, return_tensors=\"pt\").pixel_values\n\n>>> # autoregressively generate caption (uses greedy decoding by default)\n>>> generated_ids = model.generate(pixel_values)\n>>> generated_text = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]\n>>> print(generated_text)\na cat laying on a blanket next to a cat laying on a bed\n```\n\n## Loading a PyTorch checkpoint into `TFVisionEncoderDecoderModel`.\n\n[`TFVisionEncoderDecoderModel.from_pretrained`] currently doesn't support initializing the model from a\nPyTorch checkpoint. Passing `from_pt=True` to this method will throw an exception. If there are only PyTorch\ncheckpoints for a particular vision encoder-decoder model, a workaround is:\n\n```python\n>>> from transformers import VisionEncoderDecoderModel, TFVisionEncoderDecoderModel\n\n>>> _model = VisionEncoderDecoderModel.from_pretrained(\"nlpconnect/vit-gpt2-image-captioning\")\n\n>>> _model.encoder.save_pretrained(\"./encoder\")\n>>> _model.decoder.save_pretrained(\"./decoder\")\n\n>>> model = TFVisionEncoderDecoderModel.from_encoder_decoder_pretrained(\n...     \"./encoder\", \"./decoder\", encoder_from_pt=True, decoder_from_pt=True\n... )\n>>> # This is only for copying some specific attributes of this particular model.\n>>> model.config = _model.config\n```\n\n## Training\n\nOnce the model is created, it can be fine-tuned similar to BART, T5 or any other encoder-decoder model on a dataset of (image, text) pairs.\nAs you can see, only 2 inputs are required for the model in order to compute a loss: `pixel_values` (which are the\nimages) and `labels` (which are the `input_ids` of the encoded target sequence)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 1ec962fb-6f3b-4320-835d-805269e488ba
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vision-encoder-decoder", "title": "Vision Encoder Decoder Models", "tokens": 1947, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1ec962fb-6f3b-4320-835d-805269e488ba\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vision-encoder-decoder\", \"title\": \"Vision Encoder Decoder Models\", \"tokens\": 1947, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"86adba1a-edb0-487f-8d4e-64df592443ac\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vision-encoder-decoder\", \"title\": \"Vision Encoder Decoder Models\", \"tokens\": 1947, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"10a79f49aa80ba46749be3a4aa3a3c92ca07fef4f06196be9609dc10569a1691\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a1813132-d2ed-4628-a93c-178519631d23\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vision-encoder-decoder\", \"title\": \"Vision Encoder Decoder Models\", \"tokens\": 1947, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b70e2e65288c4fbf1ea87d67d6c6fcb20c6d037a9a9a6dfacc1524756b976c1b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4836, \"end_char_idx\": 7759, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "86adba1a-edb0-487f-8d4e-64df592443ac", "doc_id": "86adba1a-edb0-487f-8d4e-64df592443ac", "ref_doc_id": "86adba1a-edb0-487f-8d4e-64df592443ac", "chroma:document": "[`TFVisionEncoderDecoderModel.from_pretrained`] currently doesn't support initializing the model from a\nPyTorch checkpoint. Passing `from_pt=True` to this method will throw an exception. If there are only PyTorch\ncheckpoints for a particular vision encoder-decoder model, a workaround is:\n\n```python\n>>> from transformers import VisionEncoderDecoderModel, TFVisionEncoderDecoderModel\n\n>>> _model = VisionEncoderDecoderModel.from_pretrained(\"nlpconnect/vit-gpt2-image-captioning\")\n\n>>> _model.encoder.save_pretrained(\"./encoder\")\n>>> _model.decoder.save_pretrained(\"./decoder\")\n\n>>> model = TFVisionEncoderDecoderModel.from_encoder_decoder_pretrained(\n...     \"./encoder\", \"./decoder\", encoder_from_pt=True, decoder_from_pt=True\n... )\n>>> # This is only for copying some specific attributes of this particular model.\n>>> model.config = _model.config\n```\n\n## Training\n\nOnce the model is created, it can be fine-tuned similar to BART, T5 or any other encoder-decoder model on a dataset of (image, text) pairs.\nAs you can see, only 2 inputs are required for the model in order to compute a loss: `pixel_values` (which are the\nimages) and `labels` (which are the `input_ids` of the encoded target sequence).\n\n```python\n>>> from transformers import ViTImageProcessor, BertTokenizer, VisionEncoderDecoderModel\n>>> from datasets import load_dataset\n\n>>> image_processor = ViTImageProcessor.from_pretrained(\"google/vit-base-patch16-224-in21k\")\n>>> tokenizer = BertTokenizer.from_pretrained(\"google-bert/bert-base-uncased\")\n>>> model = VisionEncoderDecoderModel.from_encoder_decoder_pretrained(\n...     \"google/vit-base-patch16-224-in21k\", \"google-bert/bert-base-uncased\"\n... )\n\n>>> model.config.decoder_start_token_id = tokenizer.cls_token_id\n>>> model.config.pad_token_id = tokenizer.pad_token_id\n\n>>> dataset = load_dataset(\"huggingface/cats-image\")\n>>> image = dataset[\"test\"][\"image\"][0]\n>>> pixel_values = image_processor(image, return_tensors=\"pt\").pixel_values\n\n>>> labels = tokenizer(\n...     \"an image of two cats chilling on a couch\",\n...     return_tensors=\"pt\",\n... ).input_ids\n\n>>> # the forward function automatically creates the correct decoder_input_ids\n>>> loss = model(pixel_values=pixel_values, labels=labels).loss\n```\n\nThis model was contributed by [nielsr](https://github.com/nielsrogge). This model's TensorFlow and Flax versions\nwere contributed by [ydshieh](https://github.com/ydshieh).\n\n## VisionEncoderDecoderConfig\n\n[[autodoc]] VisionEncoderDecoderConfig\n\n<frameworkcontent>\n<pt>\n\n## VisionEncoderDecoderModel\n\n[[autodoc]] VisionEncoderDecoderModel\n    - forward\n    - from_encoder_decoder_pretrained\n\n</pt>\n<tf>\n\n## TFVisionEncoderDecoderModel\n\n[[autodoc]] TFVisionEncoderDecoderModel\n    - call\n    - from_encoder_decoder_pretrained\n\n</tf>\n<jax>\n\n## FlaxVisionEncoderDecoderModel\n\n[[autodoc]] FlaxVisionEncoderDecoderModel\n    - __call__\n    - from_encoder_decoder_pretrained\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 7535e774-a3f3-4b8b-897b-52cd75ca4160
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/matcha", "title": "MatCha", "tokens": 1174, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7535e774-a3f3-4b8b-897b-52cd75ca4160\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/matcha\", \"title\": \"MatCha\", \"tokens\": 1174, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"28c876dc-4d86-4b38-b9d6-0d71ef9a6300\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/matcha\", \"title\": \"MatCha\", \"tokens\": 1174, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5cd6600845e500561381e36e302e0fefa62bd424dc7fd719db21ee6e1601b1ff\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e218cf9f-13d8-4752-a2f0-14544d92e1d1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7ed83f2b8d269d3d272ee86167eb0f689b68a6c19bd30c9f985f9637e6483d36\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3243, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "28c876dc-4d86-4b38-b9d6-0d71ef9a6300", "doc_id": "28c876dc-4d86-4b38-b9d6-0d71ef9a6300", "ref_doc_id": "28c876dc-4d86-4b38-b9d6-0d71ef9a6300", "chroma:document": "# MatCha\n\n## Overview\n\nMatCha has been proposed in the paper [MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering](https://arxiv.org/abs/2212.09662), from Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Collier, Julian Martin Eisenschlos.\n\nThe abstract of the paper states the following:\n\n*Visual language data such as plots, charts, and infographics are ubiquitous in the human world. However, state-of-the-art vision-language models do not perform well on these data. We propose MatCha (Math reasoning and Chart derendering pretraining) to enhance visual language models' capabilities in jointly modeling charts/plots and language data. Specifically, we propose several pretraining tasks that cover plot deconstruction and numerical reasoning which are the key capabilities in visual language modeling. We perform the MatCha pretraining starting from Pix2Struct, a recently proposed image-to-text visual language model. On standard benchmarks such as PlotQA and ChartQA, the MatCha model outperforms state-of-the-art methods by as much as nearly 20%. We also examine how well MatCha pretraining transfers to domains such as screenshots, textbook diagrams, and document figures and observe overall improvement, verifying the usefulness of MatCha pretraining on broader visual language tasks.*\n\n## Model description\n\nMatCha is a model that is trained using `Pix2Struct` architecture. You can find more information about `Pix2Struct` in the [Pix2Struct documentation](https://huggingface.co/docs/transformers/main/en/model_doc/pix2struct).\nMatCha is a Visual Question Answering subset of `Pix2Struct` architecture. It renders the input question on the image and predicts the answer.\n\n## Usage\n\nCurrently 6 checkpoints are available for MatCha:\n\n- `google/matcha`: the base MatCha model, used to fine-tune MatCha on downstream tasks\n- `google/matcha-chartqa`: MatCha model fine-tuned on ChartQA dataset. It can be used to answer questions about charts.\n- `google/matcha-plotqa-v1`: MatCha model fine-tuned on PlotQA dataset. It can be used to answer questions about plots.\n- `google/matcha-plotqa-v2`: MatCha model fine-tuned on PlotQA dataset. It can be used to answer questions about plots.\n- `google/matcha-chart2text-statista`: MatCha model fine-tuned on Statista dataset. \n- `google/matcha-chart2text-pew`: MatCha model fine-tuned on Pew dataset.\n\nThe models finetuned on `chart2text-pew` and `chart2text-statista` are more suited for summarization, whereas the models finetuned on `plotqa` and `chartqa` are more suited for question answering.\n\nYou can use these models as follows (example on a ChatQA dataset):\n\n```python\nfrom transformers import AutoProcessor, Pix2StructForConditionalGeneration\nimport requests\nfrom PIL import Image\n\nmodel = Pix2StructForConditionalGeneration.from_pretrained(\"google/matcha-chartqa\").to(0)\nprocessor = AutoProcessor.from_pretrained(\"google/matcha-chartqa\")\nurl = \"https://raw.githubusercontent.com/vis-nlp/ChartQA/main/ChartQA%20Dataset/val/png/20294671002019.png\"\nimage = Image.open(requests.get(url, stream=True).raw)\n\ninputs = processor(images=image, text=\"Is the sum of all 4 places greater than Laos?"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e218cf9f-13d8-4752-a2f0-14544d92e1d1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/matcha", "title": "MatCha", "tokens": 1174, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e218cf9f-13d8-4752-a2f0-14544d92e1d1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/matcha\", \"title\": \"MatCha\", \"tokens\": 1174, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"28c876dc-4d86-4b38-b9d6-0d71ef9a6300\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/matcha\", \"title\": \"MatCha\", \"tokens\": 1174, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5cd6600845e500561381e36e302e0fefa62bd424dc7fd719db21ee6e1601b1ff\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7535e774-a3f3-4b8b-897b-52cd75ca4160\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/matcha\", \"title\": \"MatCha\", \"tokens\": 1174, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0397b63917c7aba7dc89b92dbddf37113e28355906cd0925690e57fa6f4929a8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1711, \"end_char_idx\": 4282, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "28c876dc-4d86-4b38-b9d6-0d71ef9a6300", "doc_id": "28c876dc-4d86-4b38-b9d6-0d71ef9a6300", "ref_doc_id": "28c876dc-4d86-4b38-b9d6-0d71ef9a6300", "chroma:document": "It renders the input question on the image and predicts the answer.\n\n## Usage\n\nCurrently 6 checkpoints are available for MatCha:\n\n- `google/matcha`: the base MatCha model, used to fine-tune MatCha on downstream tasks\n- `google/matcha-chartqa`: MatCha model fine-tuned on ChartQA dataset. It can be used to answer questions about charts.\n- `google/matcha-plotqa-v1`: MatCha model fine-tuned on PlotQA dataset. It can be used to answer questions about plots.\n- `google/matcha-plotqa-v2`: MatCha model fine-tuned on PlotQA dataset. It can be used to answer questions about plots.\n- `google/matcha-chart2text-statista`: MatCha model fine-tuned on Statista dataset. \n- `google/matcha-chart2text-pew`: MatCha model fine-tuned on Pew dataset.\n\nThe models finetuned on `chart2text-pew` and `chart2text-statista` are more suited for summarization, whereas the models finetuned on `plotqa` and `chartqa` are more suited for question answering.\n\nYou can use these models as follows (example on a ChatQA dataset):\n\n```python\nfrom transformers import AutoProcessor, Pix2StructForConditionalGeneration\nimport requests\nfrom PIL import Image\n\nmodel = Pix2StructForConditionalGeneration.from_pretrained(\"google/matcha-chartqa\").to(0)\nprocessor = AutoProcessor.from_pretrained(\"google/matcha-chartqa\")\nurl = \"https://raw.githubusercontent.com/vis-nlp/ChartQA/main/ChartQA%20Dataset/val/png/20294671002019.png\"\nimage = Image.open(requests.get(url, stream=True).raw)\n\ninputs = processor(images=image, text=\"Is the sum of all 4 places greater than Laos?\", return_tensors=\"pt\").to(0)\npredictions = model.generate(**inputs, max_new_tokens=512)\nprint(processor.decode(predictions[0], skip_special_tokens=True))\n```\n\n## Fine-tuning\n\nTo fine-tune MatCha, refer to the pix2struct [fine-tuning notebook](https://github.com/huggingface/notebooks/blob/main/examples/image_captioning_pix2struct.ipynb). For `Pix2Struct` models, we have found out that fine-tuning the model with Adafactor and cosine learning rate scheduler leads to faste convergence:\n```python\nfrom transformers.optimization import Adafactor, get_cosine_schedule_with_warmup\n\noptimizer = Adafactor(self.parameters(), scale_parameter=False, relative_step=False, lr=0.01, weight_decay=1e-05)\nscheduler = get_cosine_schedule_with_warmup(optimizer, num_warmup_steps=1000, num_training_steps=40000)\n```\n\n<Tip>\n\nMatCha is a model that is trained using `Pix2Struct` architecture. You can find more information about `Pix2Struct` in the [Pix2Struct documentation](https://huggingface.co/docs/transformers/main/en/model_doc/pix2struct).\n\n</Tip>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 275f1d91-e8b1-4565-acbd-756d76343ce9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/switch_transformers", "title": "SwitchTransformers", "tokens": 918, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"275f1d91-e8b1-4565-acbd-756d76343ce9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/switch_transformers\", \"title\": \"SwitchTransformers\", \"tokens\": 918, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"37d53b56-2eee-4f39-9a56-f910fc14c4ca\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/switch_transformers\", \"title\": \"SwitchTransformers\", \"tokens\": 918, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"42acf944716d237fecd5ac44ed291b6908dd33a616bd78436a9bf3847d13fde7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3455, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "37d53b56-2eee-4f39-9a56-f910fc14c4ca", "doc_id": "37d53b56-2eee-4f39-9a56-f910fc14c4ca", "ref_doc_id": "37d53b56-2eee-4f39-9a56-f910fc14c4ca", "chroma:document": "# SwitchTransformers\n\n## Overview\n\nThe SwitchTransformers model was proposed in [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://arxiv.org/abs/2101.03961) by William Fedus, Barret Zoph, Noam Shazeer.\n\nThe Switch Transformer model uses a sparse T5 encoder-decoder architecture, where the MLP are replaced by a Mixture of Experts (MoE). A routing mechanism (top 1 in this case) associates each token to one of the expert, where each expert is a dense MLP. While switch transformers have a lot more weights than their equivalent dense models, the sparsity allows better scaling and better finetuning performance at scale.\nDuring a forward pass, only a fraction of the weights are used. The routing mechanism allows the model to select relevant weights on the fly which increases the model capacity without increasing the number of operations.\n\nThe abstract from the paper is the following:\n\n*In deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) defies this and instead selects different parameters for each incoming example. The result is a sparsely-activated model -- with outrageous numbers of parameters -- but a constant computational cost. However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication costs and training instability -- we address these with the Switch Transformer. We simplify the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs. Our proposed training techniques help wrangle the instabilities and we show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats. We design models based off T5-Base and T5-Large to obtain up to 7x increases in pre-training speed with the same computational resources. These improvements extend into multilingual settings where we measure gains over the mT5-Base version across all 101 languages. Finally, we advance the current scale of language models by pre-training up to trillion parameter models on the \"Colossal Clean Crawled Corpus\" and achieve a 4x speedup over the T5-XXL model.*\n\nThis model was contributed by [Younes Belkada](https://huggingface.co/ybelkada) and [Arthur Zucker](https://huggingface.co/ArthurZ).\nThe original code can be found [here](https://github.com/google/flaxformer/tree/main/flaxformer/architectures/moe).\n\n## Usage tips\n\n- SwitchTransformers uses the [`T5Tokenizer`], which can be loaded directly from each model's repository.\n- The released weights are pretrained on English [Masked Language Modeling](https://moon-ci-docs.huggingface.co/docs/transformers/pr_19323/en/glossary#general-terms) task, and should be finetuned.\n\n## Resources\n\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n## SwitchTransformersConfig\n\n[[autodoc]] SwitchTransformersConfig\n\n## SwitchTransformersTop1Router\n\n[[autodoc]] SwitchTransformersTop1Router\n    - _compute_router_probabilities\n    - forward\n\n## SwitchTransformersSparseMLP\n\n[[autodoc]] SwitchTransformersSparseMLP\n    - forward\n\n## SwitchTransformersModel\n\n[[autodoc]] SwitchTransformersModel\n    - forward\n\n## SwitchTransformersForConditionalGeneration\n\n[[autodoc]] SwitchTransformersForConditionalGeneration\n    - forward\n\n## SwitchTransformersEncoderModel\n\n[[autodoc]] SwitchTransformersEncoderModel\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 40325ce9-c0a2-40bc-b885-3277b2e0f444
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vits", "title": "VITS", "tokens": 1929, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"40325ce9-c0a2-40bc-b885-3277b2e0f444\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vits\", \"title\": \"VITS\", \"tokens\": 1929, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5a66721e-a9fd-4a58-9340-0a951ce67890\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vits\", \"title\": \"VITS\", \"tokens\": 1929, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fb1e1f5c47b860ac03acf8942d6c42928d6862d23da65e755d6c29cceea6d92d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"553e4d45-33df-455b-8184-ac89a6fa486a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bcd05965cef9eeee86eb9d277262d3d4f84e1373e2d404e59e035f89057cbf80\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3450, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5a66721e-a9fd-4a58-9340-0a951ce67890", "doc_id": "5a66721e-a9fd-4a58-9340-0a951ce67890", "ref_doc_id": "5a66721e-a9fd-4a58-9340-0a951ce67890", "chroma:document": "# VITS\n\n## Overview\n\nThe VITS model was proposed in [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://arxiv.org/abs/2106.06103) by Jaehyeon Kim, Jungil Kong, Juhee Son.\n\nVITS (**V**ariational **I**nference with adversarial learning for end-to-end **T**ext-to-**S**peech) is an end-to-end \nspeech synthesis model that predicts a speech waveform conditional on an input text sequence. It is a conditional variational \nautoencoder (VAE) comprised of a posterior encoder, decoder, and conditional prior.\n\nA set of spectrogram-based acoustic features are predicted by the flow-based module, which is formed of a Transformer-based\ntext encoder and multiple coupling layers. The spectrogram is decoded using a stack of transposed convolutional layers,\nmuch in the same style as the HiFi-GAN vocoder. Motivated by the one-to-many nature of the TTS problem, where the same text \ninput can be spoken in multiple ways, the model also includes a stochastic duration predictor, which allows the model to \nsynthesise speech with different rhythms from the same input text. \n\nThe model is trained end-to-end with a combination of losses derived from variational lower bound and adversarial training. \nTo improve the expressiveness of the model, normalizing flows are applied to the conditional prior distribution. During \ninference, the text encodings are up-sampled based on the duration prediction module, and then mapped into the \nwaveform using a cascade of the flow module and HiFi-GAN decoder. Due to the stochastic nature of the duration predictor,\nthe model is non-deterministic, and thus requires a fixed seed to generate the same speech waveform.\n\nThe abstract from the paper is the following:\n\n*Several recent end-to-end text-to-speech (TTS) models enabling single-stage training and parallel sampling have been proposed, but their sample quality does not match that of two-stage TTS systems. In this work, we present a parallel end-to-end TTS method that generates more natural sounding audio than current two-stage models. Our method adopts variational inference augmented with normalizing flows and an adversarial training process, which improves the expressive power of generative modeling. We also propose a stochastic duration predictor to synthesize speech with diverse rhythms from input text. With the uncertainty modeling over latent variables and the stochastic duration predictor, our method expresses the natural one-to-many relationship in which a text input can be spoken in multiple ways with different pitches and rhythms. A subjective human evaluation (mean opinion score, or MOS) on the LJ Speech, a single speaker dataset, shows that our method outperforms the best publicly available TTS systems and achieves a MOS comparable to ground truth.*\n\nThis model can also be used with TTS checkpoints from [Massively Multilingual Speech (MMS)](https://arxiv.org/abs/2305.13516) \nas these checkpoints use the same architecture and a slightly modified tokenizer.\n\nThis model was contributed by [Matthijs](https://huggingface.co/Matthijs) and [sanchit-gandhi](https://huggingface.co/sanchit-gandhi). The original code can be found [here](https://github.com/jaywalnut310/vits).\n\n## Usage examples\n\nBoth the VITS and MMS-TTS checkpoints can be used with the same API. Since the flow-based model is non-deterministic, it \nis good practice to set a seed to ensure reproducibility of the outputs."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 553e4d45-33df-455b-8184-ac89a6fa486a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vits", "title": "VITS", "tokens": 1929, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"553e4d45-33df-455b-8184-ac89a6fa486a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vits\", \"title\": \"VITS\", \"tokens\": 1929, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5a66721e-a9fd-4a58-9340-0a951ce67890\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vits\", \"title\": \"VITS\", \"tokens\": 1929, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fb1e1f5c47b860ac03acf8942d6c42928d6862d23da65e755d6c29cceea6d92d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"40325ce9-c0a2-40bc-b885-3277b2e0f444\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vits\", \"title\": \"VITS\", \"tokens\": 1929, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cfe07cf9a305c264cf0edca895f79bff9a07fbc270789a83182a87501aadc99c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0a0a8694-df45-4027-94a8-306191644aa4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b951b1690f54bc9eff057db02ffd2edfff11702be6c65f7ef7e2578a98d35702\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1695, \"end_char_idx\": 4609, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5a66721e-a9fd-4a58-9340-0a951ce67890", "doc_id": "5a66721e-a9fd-4a58-9340-0a951ce67890", "ref_doc_id": "5a66721e-a9fd-4a58-9340-0a951ce67890", "chroma:document": "The abstract from the paper is the following:\n\n*Several recent end-to-end text-to-speech (TTS) models enabling single-stage training and parallel sampling have been proposed, but their sample quality does not match that of two-stage TTS systems. In this work, we present a parallel end-to-end TTS method that generates more natural sounding audio than current two-stage models. Our method adopts variational inference augmented with normalizing flows and an adversarial training process, which improves the expressive power of generative modeling. We also propose a stochastic duration predictor to synthesize speech with diverse rhythms from input text. With the uncertainty modeling over latent variables and the stochastic duration predictor, our method expresses the natural one-to-many relationship in which a text input can be spoken in multiple ways with different pitches and rhythms. A subjective human evaluation (mean opinion score, or MOS) on the LJ Speech, a single speaker dataset, shows that our method outperforms the best publicly available TTS systems and achieves a MOS comparable to ground truth.*\n\nThis model can also be used with TTS checkpoints from [Massively Multilingual Speech (MMS)](https://arxiv.org/abs/2305.13516) \nas these checkpoints use the same architecture and a slightly modified tokenizer.\n\nThis model was contributed by [Matthijs](https://huggingface.co/Matthijs) and [sanchit-gandhi](https://huggingface.co/sanchit-gandhi). The original code can be found [here](https://github.com/jaywalnut310/vits).\n\n## Usage examples\n\nBoth the VITS and MMS-TTS checkpoints can be used with the same API. Since the flow-based model is non-deterministic, it \nis good practice to set a seed to ensure reproducibility of the outputs. For languages with a Roman alphabet, \nsuch as English or French, the tokenizer can be used directly to pre-process the text inputs. The following code example \nruns a forward pass using the MMS-TTS English checkpoint:\n\n```python\nimport torch\nfrom transformers import VitsTokenizer, VitsModel, set_seed\n\ntokenizer = VitsTokenizer.from_pretrained(\"facebook/mms-tts-eng\")\nmodel = VitsModel.from_pretrained(\"facebook/mms-tts-eng\")\n\ninputs = tokenizer(text=\"Hello - my dog is cute\", return_tensors=\"pt\")\n\nset_seed(555)  # make deterministic\n\nwith torch.no_grad():\n   outputs = model(**inputs)\n\nwaveform = outputs.waveform[0]\n```\n\nThe resulting waveform can be saved as a `.wav` file:\n\n```python\nimport scipy\n\nscipy.io.wavfile.write(\"techno.wav\", rate=model.config.sampling_rate, data=waveform)\n```\n\nOr displayed in a Jupyter Notebook / Google Colab:\n\n```python\nfrom IPython.display import Audio\n\nAudio(waveform, rate=model.config.sampling_rate)\n```\n\nFor certain languages with a non-Roman alphabet, such as Arabic, Mandarin or Hindi, the [`uroman`](https://github.com/isi-nlp/uroman) \nperl package is required to pre-process the text inputs to the Roman alphabet."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0a0a8694-df45-4027-94a8-306191644aa4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vits", "title": "VITS", "tokens": 1929, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0a0a8694-df45-4027-94a8-306191644aa4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vits\", \"title\": \"VITS\", \"tokens\": 1929, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5a66721e-a9fd-4a58-9340-0a951ce67890\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vits\", \"title\": \"VITS\", \"tokens\": 1929, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fb1e1f5c47b860ac03acf8942d6c42928d6862d23da65e755d6c29cceea6d92d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"553e4d45-33df-455b-8184-ac89a6fa486a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vits\", \"title\": \"VITS\", \"tokens\": 1929, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fbd3597fdf801e06827197e3bc78941f571fe96f5a295edb8e1b4f47ee660f28\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8a883442-ec7d-4ec6-851b-13976eb2aee0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9de18149105f5d90d9ac4978bbcb04883328a392a0a49efc35ea4ab4a5dc0b46\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3024, \"end_char_idx\": 6000, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5a66721e-a9fd-4a58-9340-0a951ce67890", "doc_id": "5a66721e-a9fd-4a58-9340-0a951ce67890", "ref_doc_id": "5a66721e-a9fd-4a58-9340-0a951ce67890", "chroma:document": "This model was contributed by [Matthijs](https://huggingface.co/Matthijs) and [sanchit-gandhi](https://huggingface.co/sanchit-gandhi). The original code can be found [here](https://github.com/jaywalnut310/vits).\n\n## Usage examples\n\nBoth the VITS and MMS-TTS checkpoints can be used with the same API. Since the flow-based model is non-deterministic, it \nis good practice to set a seed to ensure reproducibility of the outputs. For languages with a Roman alphabet, \nsuch as English or French, the tokenizer can be used directly to pre-process the text inputs. The following code example \nruns a forward pass using the MMS-TTS English checkpoint:\n\n```python\nimport torch\nfrom transformers import VitsTokenizer, VitsModel, set_seed\n\ntokenizer = VitsTokenizer.from_pretrained(\"facebook/mms-tts-eng\")\nmodel = VitsModel.from_pretrained(\"facebook/mms-tts-eng\")\n\ninputs = tokenizer(text=\"Hello - my dog is cute\", return_tensors=\"pt\")\n\nset_seed(555)  # make deterministic\n\nwith torch.no_grad():\n   outputs = model(**inputs)\n\nwaveform = outputs.waveform[0]\n```\n\nThe resulting waveform can be saved as a `.wav` file:\n\n```python\nimport scipy\n\nscipy.io.wavfile.write(\"techno.wav\", rate=model.config.sampling_rate, data=waveform)\n```\n\nOr displayed in a Jupyter Notebook / Google Colab:\n\n```python\nfrom IPython.display import Audio\n\nAudio(waveform, rate=model.config.sampling_rate)\n```\n\nFor certain languages with a non-Roman alphabet, such as Arabic, Mandarin or Hindi, the [`uroman`](https://github.com/isi-nlp/uroman) \nperl package is required to pre-process the text inputs to the Roman alphabet.\n\nYou can check whether you require the `uroman` package for your language by inspecting the `is_uroman` attribute of \nthe pre-trained `tokenizer`:\n\n```python\nfrom transformers import VitsTokenizer\n\ntokenizer = VitsTokenizer.from_pretrained(\"facebook/mms-tts-eng\")\nprint(tokenizer.is_uroman)\n```\nIf the is_uroman attribute is `True`, the tokenizer will automatically apply the `uroman` package to your text inputs, but you need to install uroman if not already installed using:  \n```\npip install --upgrade uroman\n```\nNote: Python version required to use `uroman` as python package should be >= `3.10`. \nYou can use the tokenizer as usual without any additional preprocessing steps:\n```python\nimport torch\nfrom transformers import VitsTokenizer, VitsModel, set_seed\nimport os\nimport subprocess\n\ntokenizer = VitsTokenizer.from_pretrained(\"facebook/mms-tts-kor\")\nmodel = VitsModel.from_pretrained(\"facebook/mms-tts-kor\")\ntext = \"\uc774\ubd10 \ubb34\uc2a8 \uc77c\uc774\uc57c\"\ninputs = tokenizer(text=text, return_tensors=\"pt\")\n\nset_seed(555)  # make deterministic\nwith torch.no_grad():\n   outputs = model(inputs[\"input_ids\"])\n\nwaveform = outputs.waveform[0]\n```\nIf you don't want to upgrade to python >= `3.10`, then you can use the `uroman` perl package to pre-process the text inputs to the Roman alphabet.\nTo do this, first clone the uroman repository to your local machine and set the bash variable `UROMAN` to the local path:"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 8a883442-ec7d-4ec6-851b-13976eb2aee0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vits", "title": "VITS", "tokens": 1929, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8a883442-ec7d-4ec6-851b-13976eb2aee0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vits\", \"title\": \"VITS\", \"tokens\": 1929, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5a66721e-a9fd-4a58-9340-0a951ce67890\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vits\", \"title\": \"VITS\", \"tokens\": 1929, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fb1e1f5c47b860ac03acf8942d6c42928d6862d23da65e755d6c29cceea6d92d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0a0a8694-df45-4027-94a8-306191644aa4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vits\", \"title\": \"VITS\", \"tokens\": 1929, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a02ead86d1fae59615b1763b1347ef0e976c88bb4865be1c84ebe2a9553f5bb0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4611, \"end_char_idx\": 7688, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5a66721e-a9fd-4a58-9340-0a951ce67890", "doc_id": "5a66721e-a9fd-4a58-9340-0a951ce67890", "ref_doc_id": "5a66721e-a9fd-4a58-9340-0a951ce67890", "chroma:document": "You can check whether you require the `uroman` package for your language by inspecting the `is_uroman` attribute of \nthe pre-trained `tokenizer`:\n\n```python\nfrom transformers import VitsTokenizer\n\ntokenizer = VitsTokenizer.from_pretrained(\"facebook/mms-tts-eng\")\nprint(tokenizer.is_uroman)\n```\nIf the is_uroman attribute is `True`, the tokenizer will automatically apply the `uroman` package to your text inputs, but you need to install uroman if not already installed using:  \n```\npip install --upgrade uroman\n```\nNote: Python version required to use `uroman` as python package should be >= `3.10`. \nYou can use the tokenizer as usual without any additional preprocessing steps:\n```python\nimport torch\nfrom transformers import VitsTokenizer, VitsModel, set_seed\nimport os\nimport subprocess\n\ntokenizer = VitsTokenizer.from_pretrained(\"facebook/mms-tts-kor\")\nmodel = VitsModel.from_pretrained(\"facebook/mms-tts-kor\")\ntext = \"\uc774\ubd10 \ubb34\uc2a8 \uc77c\uc774\uc57c\"\ninputs = tokenizer(text=text, return_tensors=\"pt\")\n\nset_seed(555)  # make deterministic\nwith torch.no_grad():\n   outputs = model(inputs[\"input_ids\"])\n\nwaveform = outputs.waveform[0]\n```\nIf you don't want to upgrade to python >= `3.10`, then you can use the `uroman` perl package to pre-process the text inputs to the Roman alphabet.\nTo do this, first clone the uroman repository to your local machine and set the bash variable `UROMAN` to the local path:\n\n\n```bash\ngit clone https://github.com/isi-nlp/uroman.git\ncd uroman\nexport UROMAN=$(pwd)\n```\n\nYou can then pre-process the text input using the following code snippet. You can either rely on using the bash variable \n`UROMAN` to point to the uroman repository, or you can pass the uroman directory as an argument to the `uromaize` function:\n\n```python\nimport torch\nfrom transformers import VitsTokenizer, VitsModel, set_seed\nimport os\nimport subprocess\n\ntokenizer = VitsTokenizer.from_pretrained(\"facebook/mms-tts-kor\")\nmodel = VitsModel.from_pretrained(\"facebook/mms-tts-kor\")\n\ndef uromanize(input_string, uroman_path):\n    \"\"\"Convert non-Roman strings to Roman using the `uroman` perl package.\"\"\"\n    script_path = os.path.join(uroman_path, \"bin\", \"uroman.pl\")\n\n    command = [\"perl\", script_path]\n\n    process = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)\n    # Execute the perl command\n    stdout, stderr = process.communicate(input=input_string.encode())\n\n    if process.returncode != 0:\n        raise ValueError(f\"Error {process.returncode}: {stderr.decode()}\")\n\n    # Return the output as a string and skip the new-line character at the end\n    return stdout.decode()[:-1]\n\ntext = \"\uc774\ubd10 \ubb34\uc2a8 \uc77c\uc774\uc57c\"\nuromaized_text = uromanize(text, uroman_path=os.environ[\"UROMAN\"])\n\ninputs = tokenizer(text=uromaized_text, return_tensors=\"pt\")\n\nset_seed(555)  # make deterministic\nwith torch.no_grad():\n   outputs = model(inputs[\"input_ids\"])\n\nwaveform = outputs.waveform[0]\n```\n\n## VitsConfig\n\n[[autodoc]] VitsConfig\n\n## VitsTokenizer\n\n[[autodoc]] VitsTokenizer\n    - __call__\n    - save_vocabulary\n\n## VitsModel\n\n[[autodoc]] VitsModel\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a4ace232-6b2d-4170-b71a-0a73fba78aa0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/rag", "title": "RAG", "tokens": 1095, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a4ace232-6b2d-4170-b71a-0a73fba78aa0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rag\", \"title\": \"RAG\", \"tokens\": 1095, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"de0da16a-d32a-4b15-82d2-c76c347c4c7c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rag\", \"title\": \"RAG\", \"tokens\": 1095, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"66f0f489a31b9270f6d92a744e267e2988c3f5e8881ccd9ca441948a02a4ba7b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8fd33163-53ed-4823-a5e7-180f54d12e16\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8ee0b0f41e322443a262654be367d4e29f922eeefaefd2a25fa267005dd11929\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3075, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "de0da16a-d32a-4b15-82d2-c76c347c4c7c", "doc_id": "de0da16a-d32a-4b15-82d2-c76c347c4c7c", "ref_doc_id": "de0da16a-d32a-4b15-82d2-c76c347c4c7c", "chroma:document": "# RAG\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=rag\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-rag-blueviolet\">\n</a>\n</div>\n\n## Overview\n\nRetrieval-augmented generation (\"RAG\") models combine the powers of pretrained dense retrieval (DPR) and\nsequence-to-sequence models. RAG models retrieve documents, pass them to a seq2seq model, then marginalize to generate\noutputs. The retriever and seq2seq modules are initialized from pretrained models, and fine-tuned jointly, allowing\nboth retrieval and generation to adapt to downstream tasks.\n\nIt is based on the paper [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/abs/2005.11401) by Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir\nKarpukhin, Naman Goyal, Heinrich K\u00fcttler, Mike Lewis, Wen-tau Yih, Tim Rockt\u00e4schel, Sebastian Riedel, Douwe Kiela.\n\nThe abstract from the paper is the following:\n\n*Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve\nstate-of-the-art results when fine-tuned on downstream NLP tasks. However, their ability to access and precisely\nmanipulate knowledge is still limited, and hence on knowledge-intensive tasks, their performance lags behind\ntask-specific architectures. Additionally, providing provenance for their decisions and updating their world knowledge\nremain open research problems. Pre-trained models with a differentiable access mechanism to explicit nonparametric\nmemory can overcome this issue, but have so far been only investigated for extractive downstream tasks. We explore a\ngeneral-purpose fine-tuning recipe for retrieval-augmented generation (RAG) \u2014 models which combine pre-trained\nparametric and non-parametric memory for language generation. We introduce RAG models where the parametric memory is a\npre-trained seq2seq model and the non-parametric memory is a dense vector index of Wikipedia, accessed with a\npre-trained neural retriever. We compare two RAG formulations, one which conditions on the same retrieved passages\nacross the whole generated sequence, the other can use different passages per token. We fine-tune and evaluate our\nmodels on a wide range of knowledge-intensive NLP tasks and set the state-of-the-art on three open domain QA tasks,\noutperforming parametric seq2seq models and task-specific retrieve-and-extract architectures. For language generation\ntasks, we find that RAG models generate more specific, diverse and factual language than a state-of-the-art\nparametric-only seq2seq baseline.*\n\nThis model was contributed by [ola13](https://huggingface.co/ola13).\n\n## Usage tips\n\nRetrieval-augmented generation (\"RAG\") models combine the powers of pretrained dense retrieval (DPR) and Seq2Seq models. \nRAG models retrieve docs, pass them to a seq2seq model, then marginalize to generate outputs. The retriever and seq2seq \nmodules are initialized from pretrained models, and fine-tuned jointly, allowing both retrieval and generation to adapt \nto downstream tasks."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 8fd33163-53ed-4823-a5e7-180f54d12e16
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/rag", "title": "RAG", "tokens": 1095, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8fd33163-53ed-4823-a5e7-180f54d12e16\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rag\", \"title\": \"RAG\", \"tokens\": 1095, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"de0da16a-d32a-4b15-82d2-c76c347c4c7c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rag\", \"title\": \"RAG\", \"tokens\": 1095, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"66f0f489a31b9270f6d92a744e267e2988c3f5e8881ccd9ca441948a02a4ba7b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a4ace232-6b2d-4170-b71a-0a73fba78aa0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rag\", \"title\": \"RAG\", \"tokens\": 1095, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"534d5ba6ad357a6229b61dafca3abb32685881a6464d92272e2c887ca3f63e55\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1336, \"end_char_idx\": 3878, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "de0da16a-d32a-4b15-82d2-c76c347c4c7c", "doc_id": "de0da16a-d32a-4b15-82d2-c76c347c4c7c", "ref_doc_id": "de0da16a-d32a-4b15-82d2-c76c347c4c7c", "chroma:document": "Additionally, providing provenance for their decisions and updating their world knowledge\nremain open research problems. Pre-trained models with a differentiable access mechanism to explicit nonparametric\nmemory can overcome this issue, but have so far been only investigated for extractive downstream tasks. We explore a\ngeneral-purpose fine-tuning recipe for retrieval-augmented generation (RAG) \u2014 models which combine pre-trained\nparametric and non-parametric memory for language generation. We introduce RAG models where the parametric memory is a\npre-trained seq2seq model and the non-parametric memory is a dense vector index of Wikipedia, accessed with a\npre-trained neural retriever. We compare two RAG formulations, one which conditions on the same retrieved passages\nacross the whole generated sequence, the other can use different passages per token. We fine-tune and evaluate our\nmodels on a wide range of knowledge-intensive NLP tasks and set the state-of-the-art on three open domain QA tasks,\noutperforming parametric seq2seq models and task-specific retrieve-and-extract architectures. For language generation\ntasks, we find that RAG models generate more specific, diverse and factual language than a state-of-the-art\nparametric-only seq2seq baseline.*\n\nThis model was contributed by [ola13](https://huggingface.co/ola13).\n\n## Usage tips\n\nRetrieval-augmented generation (\"RAG\") models combine the powers of pretrained dense retrieval (DPR) and Seq2Seq models. \nRAG models retrieve docs, pass them to a seq2seq model, then marginalize to generate outputs. The retriever and seq2seq \nmodules are initialized from pretrained models, and fine-tuned jointly, allowing both retrieval and generation to adapt \nto downstream tasks.\n\n## RagConfig\n\n[[autodoc]] RagConfig\n\n## RagTokenizer\n\n[[autodoc]] RagTokenizer\n\n## Rag specific outputs\n\n[[autodoc]] models.rag.modeling_rag.RetrievAugLMMarginOutput\n\n[[autodoc]] models.rag.modeling_rag.RetrievAugLMOutput\n\n## RagRetriever\n\n[[autodoc]] RagRetriever\n\n<frameworkcontent>\n<pt>\n\n## RagModel\n\n[[autodoc]] RagModel\n    - forward\n\n## RagSequenceForGeneration\n\n[[autodoc]] RagSequenceForGeneration\n    - forward\n    - generate\n\n## RagTokenForGeneration\n\n[[autodoc]] RagTokenForGeneration\n    - forward\n    - generate\n\n</pt>\n<tf>\n\n## TFRagModel\n\n[[autodoc]] TFRagModel\n    - call\n\n## TFRagSequenceForGeneration\n\n[[autodoc]] TFRagSequenceForGeneration\n    - call\n    - generate\n\n## TFRagTokenForGeneration\n\n[[autodoc]] TFRagTokenForGeneration\n    - call\n    - generate\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 5fcf9e32-a53a-4d40-88d2-67dc4f3d3ce4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mobilebert", "title": "MobileBERT", "tokens": 1329, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5fcf9e32-a53a-4d40-88d2-67dc4f3d3ce4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilebert\", \"title\": \"MobileBERT\", \"tokens\": 1329, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"892b8804-de82-4145-9038-00aed791ff74\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilebert\", \"title\": \"MobileBERT\", \"tokens\": 1329, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d30515b3b1df27173a8f460e680315b37a9574b5299a9d3662909a89f6dceb62\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"541552d9-b8fb-46a9-9dbd-85f79792c51d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e1fd1110ad3fe4e69c25bebca773101e3267a7868d3f71c12ac046126fb0a88c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2410, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "892b8804-de82-4145-9038-00aed791ff74", "doc_id": "892b8804-de82-4145-9038-00aed791ff74", "ref_doc_id": "892b8804-de82-4145-9038-00aed791ff74", "chroma:document": "# MobileBERT\n\n## Overview\n\nThe MobileBERT model was proposed in [MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://arxiv.org/abs/2004.02984) by Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny\nZhou. It's a bidirectional transformer based on the BERT model, which is compressed and accelerated using several\napproaches.\n\nThe abstract from the paper is the following:\n\n*Natural Language Processing (NLP) has recently achieved great success by using huge pre-trained models with hundreds\nof millions of parameters. However, these models suffer from heavy model sizes and high latency such that they cannot\nbe deployed to resource-limited mobile devices. In this paper, we propose MobileBERT for compressing and accelerating\nthe popular BERT model. Like the original BERT, MobileBERT is task-agnostic, that is, it can be generically applied to\nvarious downstream NLP tasks via simple fine-tuning. Basically, MobileBERT is a thin version of BERT_LARGE, while\nequipped with bottleneck structures and a carefully designed balance between self-attentions and feed-forward networks.\nTo train MobileBERT, we first train a specially designed teacher model, an inverted-bottleneck incorporated BERT_LARGE\nmodel. Then, we conduct knowledge transfer from this teacher to MobileBERT. Empirical studies show that MobileBERT is\n4.3x smaller and 5.5x faster than BERT_BASE while achieving competitive results on well-known benchmarks. On the\nnatural language inference tasks of GLUE, MobileBERT achieves a GLUEscore o 77.7 (0.6 lower than BERT_BASE), and 62 ms\nlatency on a Pixel 4 phone. On the SQuAD v1.1/v2.0 question answering task, MobileBERT achieves a dev F1 score of\n90.0/79.2 (1.5/2.1 higher than BERT_BASE).*\n\nThis model was contributed by [vshampor](https://huggingface.co/vshampor). The original code can be found [here](https://github.com/google-research/google-research/tree/master/mobilebert).\n\n## Usage tips\n\n- MobileBERT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather\n  than the left.\n- MobileBERT is similar to BERT and therefore relies on the masked language modeling (MLM) objective. It is therefore\n  efficient at predicting masked tokens and at NLU in general, but is not optimal for text generation. Models trained\n  with a causal language modeling (CLM) objective are better in that regard."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 541552d9-b8fb-46a9-9dbd-85f79792c51d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mobilebert", "title": "MobileBERT", "tokens": 1329, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"541552d9-b8fb-46a9-9dbd-85f79792c51d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilebert\", \"title\": \"MobileBERT\", \"tokens\": 1329, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"892b8804-de82-4145-9038-00aed791ff74\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilebert\", \"title\": \"MobileBERT\", \"tokens\": 1329, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d30515b3b1df27173a8f460e680315b37a9574b5299a9d3662909a89f6dceb62\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5fcf9e32-a53a-4d40-88d2-67dc4f3d3ce4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilebert\", \"title\": \"MobileBERT\", \"tokens\": 1329, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a063e637a21df6ec785bac6c22eac6c70d75cf2235bf6c0a241f7a169adc310a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2413, \"end_char_idx\": 4607, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "892b8804-de82-4145-9038-00aed791ff74", "doc_id": "892b8804-de82-4145-9038-00aed791ff74", "ref_doc_id": "892b8804-de82-4145-9038-00aed791ff74", "chroma:document": "## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## MobileBertConfig\n\n[[autodoc]] MobileBertConfig\n\n## MobileBertTokenizer\n\n[[autodoc]] MobileBertTokenizer\n\n## MobileBertTokenizerFast\n\n[[autodoc]] MobileBertTokenizerFast\n\n## MobileBert specific outputs\n\n[[autodoc]] models.mobilebert.modeling_mobilebert.MobileBertForPreTrainingOutput\n\n[[autodoc]] models.mobilebert.modeling_tf_mobilebert.TFMobileBertForPreTrainingOutput\n\n<frameworkcontent>\n<pt>\n\n## MobileBertModel\n\n[[autodoc]] MobileBertModel\n    - forward\n\n## MobileBertForPreTraining\n\n[[autodoc]] MobileBertForPreTraining\n    - forward\n\n## MobileBertForMaskedLM\n\n[[autodoc]] MobileBertForMaskedLM\n    - forward\n\n## MobileBertForNextSentencePrediction\n\n[[autodoc]] MobileBertForNextSentencePrediction\n    - forward\n\n## MobileBertForSequenceClassification\n\n[[autodoc]] MobileBertForSequenceClassification\n    - forward\n\n## MobileBertForMultipleChoice\n\n[[autodoc]] MobileBertForMultipleChoice\n    - forward\n\n## MobileBertForTokenClassification\n\n[[autodoc]] MobileBertForTokenClassification\n    - forward\n\n## MobileBertForQuestionAnswering\n\n[[autodoc]] MobileBertForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFMobileBertModel\n\n[[autodoc]] TFMobileBertModel\n    - call\n\n## TFMobileBertForPreTraining\n\n[[autodoc]] TFMobileBertForPreTraining\n    - call\n\n## TFMobileBertForMaskedLM\n\n[[autodoc]] TFMobileBertForMaskedLM\n    - call\n\n## TFMobileBertForNextSentencePrediction\n\n[[autodoc]] TFMobileBertForNextSentencePrediction\n    - call\n\n## TFMobileBertForSequenceClassification\n\n[[autodoc]] TFMobileBertForSequenceClassification\n    - call\n\n## TFMobileBertForMultipleChoice\n\n[[autodoc]] TFMobileBertForMultipleChoice\n    - call\n\n## TFMobileBertForTokenClassification\n\n[[autodoc]] TFMobileBertForTokenClassification\n    - call\n\n## TFMobileBertForQuestionAnswering\n\n[[autodoc]] TFMobileBertForQuestionAnswering\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6f1c4b8a-77c1-4c72-b15c-176e95e003d6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/pvt_v2", "title": "Pyramid Vision Transformer V2 (PVTv2)", "tokens": 2114, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6f1c4b8a-77c1-4c72-b15c-176e95e003d6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pvt_v2\", \"title\": \"Pyramid Vision Transformer V2 (PVTv2)\", \"tokens\": 2114, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"728fdd47-4068-4135-8b1d-c196ab97e3f1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pvt_v2\", \"title\": \"Pyramid Vision Transformer V2 (PVTv2)\", \"tokens\": 2114, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"07cb9bb28b2efb5f14eeda98fb8e5660f2894f9e2bef7ead8f771c91e753ad88\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6bab012c-d81b-46fb-98a8-f57b69730be2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"26f5fa39308f72010911f4168a92ec9b3b44fb5a444268cb288b26b62c28445d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3435, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "728fdd47-4068-4135-8b1d-c196ab97e3f1", "doc_id": "728fdd47-4068-4135-8b1d-c196ab97e3f1", "ref_doc_id": "728fdd47-4068-4135-8b1d-c196ab97e3f1", "chroma:document": "# Pyramid Vision Transformer V2 (PVTv2)\n\n## Overview\n\nThe PVTv2 model was proposed in\n[PVT v2: Improved Baselines with Pyramid Vision Transformer](https://arxiv.org/abs/2106.13797) by Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. As an improved variant of PVT, it eschews position embeddings, relying instead on positional information encoded through zero-padding and overlapping patch embeddings. This lack of reliance on position embeddings simplifies the architecture, and enables running inference at any resolution without needing to interpolate them.\n\nThe PVTv2 encoder structure has been successfully deployed to achieve state-of-the-art scores in [Segformer](https://arxiv.org/abs/2105.15203) for semantic segmentation, [GLPN](https://arxiv.org/abs/2201.07436) for monocular depth, and [Panoptic Segformer](https://arxiv.org/abs/2109.03814) for panoptic segmentation.\n\nPVTv2 belongs to a family of models called [hierarchical transformers](https://natecibik.medium.com/the-rise-of-vision-transformers-f623c980419f) , which make adaptations to transformer layers in order to generate multi-scale feature maps. Unlike the columnal structure of Vision Transformer ([ViT](https://arxiv.org/abs/2010.11929)) which loses fine-grained detail, multi-scale feature maps are known preserve this detail and aid performance in dense prediction tasks. In the case of PVTv2, this is achieved by generating image patch tokens using 2D convolution with overlapping kernels in each encoder layer.\n\nThe multi-scale features of hierarchical transformers allow them to be easily swapped in for traditional workhorse computer vision backbone models like ResNet in larger architectures. Both Segformer and Panoptic Segformer demonstrated that configurations using PVTv2 for a backbone consistently outperformed those with similarly sized ResNet backbones. \n\nAnother powerful feature of the PVTv2 is the complexity reduction in the self-attention layers called Spatial Reduction Attention (SRA), which uses 2D convolution layers to project hidden states to a smaller resolution before attending to them with the queries, improving the $O(n^2)$ complexity of self-attention to $O(n^2/R)$, with $R$ being the spatial reduction ratio (`sr_ratio`, aka kernel size and stride in the 2D convolution).\n\nSRA was introduced in PVT, and is the default attention complexity reduction method used in PVTv2. However, PVTv2 also introduced the option of using a self-attention mechanism with linear complexity related to image size, which they called \"Linear SRA\". This method uses average pooling to reduce the hidden states to a fixed size that is invariant to their original resolution (although this is inherently more lossy than regular SRA). This option can be enabled by setting `linear_attention` to `True` in the PVTv2Config.\n\n### Abstract from the paper:\n\n*Transformer recently has presented encouraging progress in computer vision. In this work, we present new baselines by improving the original Pyramid Vision Transformer (PVT v1) by adding three designs, including (1) linear complexity attention layer, (2) overlapping patch embedding, and (3) convolutional feed-forward network. With these modifications, PVT v2 reduces the computational complexity of PVT v1 to linear and achieves significant improvements on fundamental vision tasks such as classification, detection, and segmentation."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6bab012c-d81b-46fb-98a8-f57b69730be2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/pvt_v2", "title": "Pyramid Vision Transformer V2 (PVTv2)", "tokens": 2114, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6bab012c-d81b-46fb-98a8-f57b69730be2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pvt_v2\", \"title\": \"Pyramid Vision Transformer V2 (PVTv2)\", \"tokens\": 2114, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"728fdd47-4068-4135-8b1d-c196ab97e3f1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pvt_v2\", \"title\": \"Pyramid Vision Transformer V2 (PVTv2)\", \"tokens\": 2114, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"07cb9bb28b2efb5f14eeda98fb8e5660f2894f9e2bef7ead8f771c91e753ad88\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6f1c4b8a-77c1-4c72-b15c-176e95e003d6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pvt_v2\", \"title\": \"Pyramid Vision Transformer V2 (PVTv2)\", \"tokens\": 2114, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c16956542091149944d0183cc460212f3f9228083f54f59418e79e3f1435dbff\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"37d04b24-1029-4c2a-b557-fd5ac196ae78\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2ad32c3bfe2e7b8e141b1e79a96f963d361ad02e739937f2a85577344cd21f04\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1550, \"end_char_idx\": 5124, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "728fdd47-4068-4135-8b1d-c196ab97e3f1", "doc_id": "728fdd47-4068-4135-8b1d-c196ab97e3f1", "ref_doc_id": "728fdd47-4068-4135-8b1d-c196ab97e3f1", "chroma:document": "The multi-scale features of hierarchical transformers allow them to be easily swapped in for traditional workhorse computer vision backbone models like ResNet in larger architectures. Both Segformer and Panoptic Segformer demonstrated that configurations using PVTv2 for a backbone consistently outperformed those with similarly sized ResNet backbones. \n\nAnother powerful feature of the PVTv2 is the complexity reduction in the self-attention layers called Spatial Reduction Attention (SRA), which uses 2D convolution layers to project hidden states to a smaller resolution before attending to them with the queries, improving the $O(n^2)$ complexity of self-attention to $O(n^2/R)$, with $R$ being the spatial reduction ratio (`sr_ratio`, aka kernel size and stride in the 2D convolution).\n\nSRA was introduced in PVT, and is the default attention complexity reduction method used in PVTv2. However, PVTv2 also introduced the option of using a self-attention mechanism with linear complexity related to image size, which they called \"Linear SRA\". This method uses average pooling to reduce the hidden states to a fixed size that is invariant to their original resolution (although this is inherently more lossy than regular SRA). This option can be enabled by setting `linear_attention` to `True` in the PVTv2Config.\n\n### Abstract from the paper:\n\n*Transformer recently has presented encouraging progress in computer vision. In this work, we present new baselines by improving the original Pyramid Vision Transformer (PVT v1) by adding three designs, including (1) linear complexity attention layer, (2) overlapping patch embedding, and (3) convolutional feed-forward network. With these modifications, PVT v2 reduces the computational complexity of PVT v1 to linear and achieves significant improvements on fundamental vision tasks such as classification, detection, and segmentation. Notably, the proposed PVT v2 achieves comparable or better performances than recent works such as Swin Transformer. We hope this work will facilitate state-of-the-art Transformer researches in computer vision. Code is available at https://github.com/whai362/PVT.*\n\nThis model was contributed by [FoamoftheSea](https://huggingface.co/FoamoftheSea). The original code can be found [here](https://github.com/whai362/PVT).\n\n## Usage tips\n\n- [PVTv2](https://arxiv.org/abs/2106.13797) is a hierarchical transformer model which has demonstrated powerful performance in image classification and multiple other tasks, used as a backbone for semantic segmentation in [Segformer](https://arxiv.org/abs/2105.15203), monocular depth estimation in [GLPN](https://arxiv.org/abs/2201.07436), and panoptic segmentation in [Panoptic Segformer](https://arxiv.org/abs/2109.03814), consistently showing higher performance than similar ResNet configurations.\n- Hierarchical transformers like PVTv2 achieve superior data and parameter efficiency on image data compared with pure transformer architectures by incorporating design elements of convolutional neural networks (CNNs) into their encoders. This creates a best-of-both-worlds architecture that infuses the useful inductive biases of CNNs like translation equivariance and locality into the network while still enjoying the benefits of dynamic data response and global relationship modeling provided by the self-attention mechanism of [transformers](https://arxiv.org/abs/1706.03762).\n- PVTv2 uses overlapping patch embeddings to create multi-scale feature maps, which are infused with location information using zero-padding and depth-wise convolutions."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 37d04b24-1029-4c2a-b557-fd5ac196ae78
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/pvt_v2", "title": "Pyramid Vision Transformer V2 (PVTv2)", "tokens": 2114, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"37d04b24-1029-4c2a-b557-fd5ac196ae78\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pvt_v2\", \"title\": \"Pyramid Vision Transformer V2 (PVTv2)\", \"tokens\": 2114, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"728fdd47-4068-4135-8b1d-c196ab97e3f1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pvt_v2\", \"title\": \"Pyramid Vision Transformer V2 (PVTv2)\", \"tokens\": 2114, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"07cb9bb28b2efb5f14eeda98fb8e5660f2894f9e2bef7ead8f771c91e753ad88\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6bab012c-d81b-46fb-98a8-f57b69730be2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pvt_v2\", \"title\": \"Pyramid Vision Transformer V2 (PVTv2)\", \"tokens\": 2114, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"10e45edd72eab713eeeb4cab3a26f1fa1f089c2977e93e934b8811f854bc6e08\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cf133075-f8be-4266-bd73-714fb88870eb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5e500083fc5272d9381c2e888bbee8a0d9d14a1caf55e12e02a27747269df45f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3436, \"end_char_idx\": 6012, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "728fdd47-4068-4135-8b1d-c196ab97e3f1", "doc_id": "728fdd47-4068-4135-8b1d-c196ab97e3f1", "ref_doc_id": "728fdd47-4068-4135-8b1d-c196ab97e3f1", "chroma:document": "Notably, the proposed PVT v2 achieves comparable or better performances than recent works such as Swin Transformer. We hope this work will facilitate state-of-the-art Transformer researches in computer vision. Code is available at https://github.com/whai362/PVT.*\n\nThis model was contributed by [FoamoftheSea](https://huggingface.co/FoamoftheSea). The original code can be found [here](https://github.com/whai362/PVT).\n\n## Usage tips\n\n- [PVTv2](https://arxiv.org/abs/2106.13797) is a hierarchical transformer model which has demonstrated powerful performance in image classification and multiple other tasks, used as a backbone for semantic segmentation in [Segformer](https://arxiv.org/abs/2105.15203), monocular depth estimation in [GLPN](https://arxiv.org/abs/2201.07436), and panoptic segmentation in [Panoptic Segformer](https://arxiv.org/abs/2109.03814), consistently showing higher performance than similar ResNet configurations.\n- Hierarchical transformers like PVTv2 achieve superior data and parameter efficiency on image data compared with pure transformer architectures by incorporating design elements of convolutional neural networks (CNNs) into their encoders. This creates a best-of-both-worlds architecture that infuses the useful inductive biases of CNNs like translation equivariance and locality into the network while still enjoying the benefits of dynamic data response and global relationship modeling provided by the self-attention mechanism of [transformers](https://arxiv.org/abs/1706.03762).\n- PVTv2 uses overlapping patch embeddings to create multi-scale feature maps, which are infused with location information using zero-padding and depth-wise convolutions.\n- To reduce the complexity in the attention layers, PVTv2 performs a spatial reduction on the hidden states using either strided 2D convolution (SRA) or fixed-size average pooling (Linear SRA). Although inherently more lossy, Linear SRA provides impressive performance with a linear complexity with respect to image size. To use Linear SRA in the self-attention layers, set `linear_attention=True` in the `PvtV2Config`.\n- [`PvtV2Model`] is the hierarchical transformer encoder (which is also often referred to as Mix Transformer or MiT in the literature). [`PvtV2ForImageClassification`] adds a simple classifier head on top to perform Image Classification. [`PvtV2Backbone`] can be used with the [`AutoBackbone`] system in larger architectures like Deformable DETR.\n- ImageNet pretrained weights for all model sizes can be found on the [hub](https://huggingface.co/models?other=pvt_v2)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: cf133075-f8be-4266-bd73-714fb88870eb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/pvt_v2", "title": "Pyramid Vision Transformer V2 (PVTv2)", "tokens": 2114, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"cf133075-f8be-4266-bd73-714fb88870eb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pvt_v2\", \"title\": \"Pyramid Vision Transformer V2 (PVTv2)\", \"tokens\": 2114, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"728fdd47-4068-4135-8b1d-c196ab97e3f1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pvt_v2\", \"title\": \"Pyramid Vision Transformer V2 (PVTv2)\", \"tokens\": 2114, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"07cb9bb28b2efb5f14eeda98fb8e5660f2894f9e2bef7ead8f771c91e753ad88\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"37d04b24-1029-4c2a-b557-fd5ac196ae78\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pvt_v2\", \"title\": \"Pyramid Vision Transformer V2 (PVTv2)\", \"tokens\": 2114, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b99dfc8b3773d4937b2d931a773f61bc0f20075bdb8195151e99634ddb8e6378\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ab79e1c1-914b-4b20-8bac-007f8e76621e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8a43301674d73bce30f4473587d937e21ad82c330b447e4e58f2f62dd70aedef\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4373, \"end_char_idx\": 8115, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "728fdd47-4068-4135-8b1d-c196ab97e3f1", "doc_id": "728fdd47-4068-4135-8b1d-c196ab97e3f1", "ref_doc_id": "728fdd47-4068-4135-8b1d-c196ab97e3f1", "chroma:document": "- Hierarchical transformers like PVTv2 achieve superior data and parameter efficiency on image data compared with pure transformer architectures by incorporating design elements of convolutional neural networks (CNNs) into their encoders. This creates a best-of-both-worlds architecture that infuses the useful inductive biases of CNNs like translation equivariance and locality into the network while still enjoying the benefits of dynamic data response and global relationship modeling provided by the self-attention mechanism of [transformers](https://arxiv.org/abs/1706.03762).\n- PVTv2 uses overlapping patch embeddings to create multi-scale feature maps, which are infused with location information using zero-padding and depth-wise convolutions.\n- To reduce the complexity in the attention layers, PVTv2 performs a spatial reduction on the hidden states using either strided 2D convolution (SRA) or fixed-size average pooling (Linear SRA). Although inherently more lossy, Linear SRA provides impressive performance with a linear complexity with respect to image size. To use Linear SRA in the self-attention layers, set `linear_attention=True` in the `PvtV2Config`.\n- [`PvtV2Model`] is the hierarchical transformer encoder (which is also often referred to as Mix Transformer or MiT in the literature). [`PvtV2ForImageClassification`] adds a simple classifier head on top to perform Image Classification. [`PvtV2Backbone`] can be used with the [`AutoBackbone`] system in larger architectures like Deformable DETR.\n- ImageNet pretrained weights for all model sizes can be found on the [hub](https://huggingface.co/models?other=pvt_v2).\n\n The best way to get started with the PVTv2 is to load the pretrained checkpoint with the size of your choosing using `AutoModelForImageClassification`:\n```python\nimport requests\nimport torch\n\nfrom transformers import AutoModelForImageClassification, AutoImageProcessor\nfrom PIL import Image\n\nmodel = AutoModelForImageClassification.from_pretrained(\"OpenGVLab/pvt_v2_b0\")\nimage_processor = AutoImageProcessor.from_pretrained(\"OpenGVLab/pvt_v2_b0\")\nurl = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nimage = Image.open(requests.get(url, stream=True).raw)\nprocessed = image_processor(image)\noutputs = model(torch.tensor(processed[\"pixel_values\"]))\n```\n\nTo use the PVTv2 as a backbone for more complex architectures like DeformableDETR, you can use AutoBackbone (this model would need fine-tuning as you're replacing the backbone in the pretrained model):\n\n```python\nimport requests\nimport torch\n\nfrom transformers import AutoConfig, AutoModelForObjectDetection, AutoImageProcessor\nfrom PIL import Image\n\nmodel = AutoModelForObjectDetection.from_config(\n    config=AutoConfig.from_pretrained(\n        \"SenseTime/deformable-detr\",\n        backbone_config=AutoConfig.from_pretrained(\"OpenGVLab/pvt_v2_b5\"),\n        use_timm_backbone=False\n    ),\n)\n\nimage_processor = AutoImageProcessor.from_pretrained(\"SenseTime/deformable-detr\")\nurl = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nimage = Image.open(requests.get(url, stream=True).raw)\nprocessed = image_processor(image)\noutputs = model(torch.tensor(processed[\"pixel_values\"]))\n```\n\n[PVTv2](https://github.com/whai362/PVT/tree/v2) performance on ImageNet-1K by model size (B0-B5):\n\n| Method           | Size | Acc@1 | #Params (M) |\n|------------------|:----:|:-----:|:-----------:|\n| PVT-V2-B0        |  224 |  70.5 |     3.7     |\n| PVT-V2-B1        |  224 |  78.7 |     14.0    |\n| PVT-V2-B2-Linear |  224 |  82.1 |     22.6    |\n| PVT-V2-B2        |  224 |  82.0 |     25.4    |\n| PVT-V2-B3        |  224 |  83.1 |     45.2    |\n| PVT-V2-B4        |  224 |  83.6 |     62.6    |\n| PVT-V2-B5        |  224 |  83.8 |     82.0    |"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: ab79e1c1-914b-4b20-8bac-007f8e76621e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/pvt_v2", "title": "Pyramid Vision Transformer V2 (PVTv2)", "tokens": 2114, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ab79e1c1-914b-4b20-8bac-007f8e76621e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pvt_v2\", \"title\": \"Pyramid Vision Transformer V2 (PVTv2)\", \"tokens\": 2114, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"728fdd47-4068-4135-8b1d-c196ab97e3f1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pvt_v2\", \"title\": \"Pyramid Vision Transformer V2 (PVTv2)\", \"tokens\": 2114, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"07cb9bb28b2efb5f14eeda98fb8e5660f2894f9e2bef7ead8f771c91e753ad88\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cf133075-f8be-4266-bd73-714fb88870eb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/pvt_v2\", \"title\": \"Pyramid Vision Transformer V2 (PVTv2)\", \"tokens\": 2114, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9b99656f03df57fa4e38d125a226e553f3f93c30ffd0464a16bdcea889d7b301\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8118, \"end_char_idx\": 8293, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "728fdd47-4068-4135-8b1d-c196ab97e3f1", "doc_id": "728fdd47-4068-4135-8b1d-c196ab97e3f1", "ref_doc_id": "728fdd47-4068-4135-8b1d-c196ab97e3f1", "chroma:document": "## PvtV2Config\n\n[[autodoc]] PvtV2Config\n\n## PvtForImageClassification\n\n[[autodoc]] PvtV2ForImageClassification\n    - forward\n\n## PvtModel\n\n[[autodoc]] PvtV2Model\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f4fe8ca5-5ea6-4a4b-ae22-01fdba996700
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/trocr", "title": "TrOCR", "tokens": 1704, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f4fe8ca5-5ea6-4a4b-ae22-01fdba996700\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/trocr\", \"title\": \"TrOCR\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"427fb293-6de9-476b-87c6-d5f1169e3090\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/trocr\", \"title\": \"TrOCR\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7b32a4b8847a1905f6b356bb3801af7070f92f55e4c9dd38e20fe57baf8cfd09\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ae6551ad-b71f-4f4e-9211-af26186407c5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3083a32de0faa1e89f9510ce3035b357ea3c230a6c4a61f64b3d42ffc880c1c6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3221, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "427fb293-6de9-476b-87c6-d5f1169e3090", "doc_id": "427fb293-6de9-476b-87c6-d5f1169e3090", "ref_doc_id": "427fb293-6de9-476b-87c6-d5f1169e3090", "chroma:document": "# TrOCR\n\n## Overview\n\nThe TrOCR model was proposed in [TrOCR: Transformer-based Optical Character Recognition with Pre-trained\nModels](https://arxiv.org/abs/2109.10282) by Minghao Li, Tengchao Lv, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang,\nZhoujun Li, Furu Wei. TrOCR consists of an image Transformer encoder and an autoregressive text Transformer decoder to\nperform [optical character recognition (OCR)](https://en.wikipedia.org/wiki/Optical_character_recognition).\n\nThe abstract from the paper is the following:\n\n*Text recognition is a long-standing research problem for document digitalization. Existing approaches for text recognition\nare usually built based on CNN for image understanding and RNN for char-level text generation. In addition, another language\nmodel is usually needed to improve the overall accuracy as a post-processing step. In this paper, we propose an end-to-end\ntext recognition approach with pre-trained image Transformer and text Transformer models, namely TrOCR, which leverages the\nTransformer architecture for both image understanding and wordpiece-level text generation. The TrOCR model is simple but\neffective, and can be pre-trained with large-scale synthetic data and fine-tuned with human-labeled datasets. Experiments\nshow that the TrOCR model outperforms the current state-of-the-art models on both printed and handwritten text recognition\ntasks.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/trocr_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> TrOCR architecture. Taken from the <a href=\"https://arxiv.org/abs/2109.10282\">original paper</a>. </small>\n\nPlease refer to the [`VisionEncoderDecoder`] class on how to use this model.\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found\n[here](https://github.com/microsoft/unilm/tree/6f60612e7cc86a2a1ae85c47231507a587ab4e01/trocr).\n\n## Usage tips\n\n- The quickest way to get started with TrOCR is by checking the [tutorial\n  notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/TrOCR), which show how to use the model\n  at inference time as well as fine-tuning on custom data.\n- TrOCR is pre-trained in 2 stages before being fine-tuned on downstream datasets. It achieves state-of-the-art results\n  on both printed (e.g. the [SROIE dataset](https://paperswithcode.com/dataset/sroie) and handwritten (e.g. the [IAM\n  Handwriting dataset](https://fki.tic.heia-fr.ch/databases/iam-handwriting-database>) text recognition tasks. For more\n  information, see the [official models](https://huggingface.co/models?other=trocr>).\n- TrOCR is always used within the [VisionEncoderDecoder](vision-encoder-decoder) framework.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with TrOCR. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- A blog post on [Accelerating Document AI](https://huggingface.co/blog/document-ai) with TrOCR."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: ae6551ad-b71f-4f4e-9211-af26186407c5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/trocr", "title": "TrOCR", "tokens": 1704, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ae6551ad-b71f-4f4e-9211-af26186407c5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/trocr\", \"title\": \"TrOCR\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"427fb293-6de9-476b-87c6-d5f1169e3090\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/trocr\", \"title\": \"TrOCR\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7b32a4b8847a1905f6b356bb3801af7070f92f55e4c9dd38e20fe57baf8cfd09\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f4fe8ca5-5ea6-4a4b-ae22-01fdba996700\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/trocr\", \"title\": \"TrOCR\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7fdf91138c5ded548f5e9a1247f45872352d7a8dce81823cd31dae612e2e57e7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f0312663-e793-492d-92ed-e73e4947e9df\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3aa8757042d4916f50e8470731435e2ba83e3964fec1c8ac69118c98cc40f045\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1726, \"end_char_idx\": 4527, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "427fb293-6de9-476b-87c6-d5f1169e3090", "doc_id": "427fb293-6de9-476b-87c6-d5f1169e3090", "ref_doc_id": "427fb293-6de9-476b-87c6-d5f1169e3090", "chroma:document": "This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found\n[here](https://github.com/microsoft/unilm/tree/6f60612e7cc86a2a1ae85c47231507a587ab4e01/trocr).\n\n## Usage tips\n\n- The quickest way to get started with TrOCR is by checking the [tutorial\n  notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/TrOCR), which show how to use the model\n  at inference time as well as fine-tuning on custom data.\n- TrOCR is pre-trained in 2 stages before being fine-tuned on downstream datasets. It achieves state-of-the-art results\n  on both printed (e.g. the [SROIE dataset](https://paperswithcode.com/dataset/sroie) and handwritten (e.g. the [IAM\n  Handwriting dataset](https://fki.tic.heia-fr.ch/databases/iam-handwriting-database>) text recognition tasks. For more\n  information, see the [official models](https://huggingface.co/models?other=trocr>).\n- TrOCR is always used within the [VisionEncoderDecoder](vision-encoder-decoder) framework.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with TrOCR. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- A blog post on [Accelerating Document AI](https://huggingface.co/blog/document-ai) with TrOCR.\n- A blog post on how to [Document AI](https://github.com/philschmid/document-ai-transformers) with TrOCR.\n- A notebook on how to [finetune TrOCR on IAM Handwriting Database using Seq2SeqTrainer](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/TrOCR/Fine_tune_TrOCR_on_IAM_Handwriting_Database_using_Seq2SeqTrainer.ipynb).\n- A notebook on [inference with TrOCR](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/TrOCR/Inference_with_TrOCR_%2B_Gradio_demo.ipynb) and Gradio demo.\n- A notebook on [finetune TrOCR on the IAM Handwriting Database](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/TrOCR/Fine_tune_TrOCR_on_IAM_Handwriting_Database_using_native_PyTorch.ipynb) using native PyTorch.\n- A notebook on [evaluating TrOCR on the IAM test set](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/TrOCR/Evaluating_TrOCR_base_handwritten_on_the_IAM_test_set.ipynb).\n\n<PipelineTag pipeline=\"text-generation\"/>\n\n- [Casual language modeling](https://huggingface.co/docs/transformers/tasks/language_modeling) task guide.\n\n\u26a1\ufe0f Inference\n\n- An interactive-demo on [TrOCR handwritten character recognition](https://huggingface.co/spaces/nielsr/TrOCR-handwritten)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f0312663-e793-492d-92ed-e73e4947e9df
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/trocr", "title": "TrOCR", "tokens": 1704, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f0312663-e793-492d-92ed-e73e4947e9df\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/trocr\", \"title\": \"TrOCR\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"427fb293-6de9-476b-87c6-d5f1169e3090\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/trocr\", \"title\": \"TrOCR\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7b32a4b8847a1905f6b356bb3801af7070f92f55e4c9dd38e20fe57baf8cfd09\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ae6551ad-b71f-4f4e-9211-af26186407c5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/trocr\", \"title\": \"TrOCR\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bfeabe17678bff387bfe52e3d23e80453157c3043c206e45cb46f287b196d183\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"88e54cc4-b21d-4592-9d43-1ec78e571242\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"56b4c8050daf5f434ed000a8e2dfa18bd1e8a134ad0053ddcad8c5908ecc1e21\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3078, \"end_char_idx\": 5979, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "427fb293-6de9-476b-87c6-d5f1169e3090", "doc_id": "427fb293-6de9-476b-87c6-d5f1169e3090", "ref_doc_id": "427fb293-6de9-476b-87c6-d5f1169e3090", "chroma:document": "<PipelineTag pipeline=\"text-classification\"/>\n\n- A blog post on [Accelerating Document AI](https://huggingface.co/blog/document-ai) with TrOCR.\n- A blog post on how to [Document AI](https://github.com/philschmid/document-ai-transformers) with TrOCR.\n- A notebook on how to [finetune TrOCR on IAM Handwriting Database using Seq2SeqTrainer](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/TrOCR/Fine_tune_TrOCR_on_IAM_Handwriting_Database_using_Seq2SeqTrainer.ipynb).\n- A notebook on [inference with TrOCR](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/TrOCR/Inference_with_TrOCR_%2B_Gradio_demo.ipynb) and Gradio demo.\n- A notebook on [finetune TrOCR on the IAM Handwriting Database](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/TrOCR/Fine_tune_TrOCR_on_IAM_Handwriting_Database_using_native_PyTorch.ipynb) using native PyTorch.\n- A notebook on [evaluating TrOCR on the IAM test set](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/TrOCR/Evaluating_TrOCR_base_handwritten_on_the_IAM_test_set.ipynb).\n\n<PipelineTag pipeline=\"text-generation\"/>\n\n- [Casual language modeling](https://huggingface.co/docs/transformers/tasks/language_modeling) task guide.\n\n\u26a1\ufe0f Inference\n\n- An interactive-demo on [TrOCR handwritten character recognition](https://huggingface.co/spaces/nielsr/TrOCR-handwritten).\n\n## Inference\n\nTrOCR's [`VisionEncoderDecoder`] model accepts images as input and makes use of\n[`~generation.GenerationMixin.generate`] to autoregressively generate text given the input image.\n\nThe [`ViTImageProcessor`/`DeiTImageProcessor`] class is responsible for preprocessing the input image and\n[`RobertaTokenizer`/`XLMRobertaTokenizer`] decodes the generated target tokens to the target string. The\n[`TrOCRProcessor`] wraps [`ViTImageProcessor`/`DeiTImageProcessor`] and [`RobertaTokenizer`/`XLMRobertaTokenizer`]\ninto a single instance to both extract the input features and decode the predicted token ids.\n\n- Step-by-step Optical Character Recognition (OCR)\n\n``` py\n>>> from transformers import TrOCRProcessor, VisionEncoderDecoderModel\n>>> import requests\n>>> from PIL import Image\n\n>>> processor = TrOCRProcessor.from_pretrained(\"microsoft/trocr-base-handwritten\")\n>>> model = VisionEncoderDecoderModel.from_pretrained(\"microsoft/trocr-base-handwritten\")\n\n>>> # load image from the IAM dataset\n>>> url = \"https://fki.tic.heia-fr.ch/static/img/a01-122-02.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw).convert(\"RGB\")\n\n>>> pixel_values = processor(image, return_tensors=\"pt\").pixel_values\n>>> generated_ids = model.generate(pixel_values)\n\n>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]\n```\n\nSee the [model hub](https://huggingface.co/models?filter=trocr) to look for TrOCR checkpoints."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 88e54cc4-b21d-4592-9d43-1ec78e571242
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/trocr", "title": "TrOCR", "tokens": 1704, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"88e54cc4-b21d-4592-9d43-1ec78e571242\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/trocr\", \"title\": \"TrOCR\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"427fb293-6de9-476b-87c6-d5f1169e3090\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/trocr\", \"title\": \"TrOCR\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7b32a4b8847a1905f6b356bb3801af7070f92f55e4c9dd38e20fe57baf8cfd09\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f0312663-e793-492d-92ed-e73e4947e9df\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/trocr\", \"title\": \"TrOCR\", \"tokens\": 1704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"020469f650433c83c362977e0d992af0b84457414f1593189c6f3c6d9416fd2d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4390, \"end_char_idx\": 6224, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "427fb293-6de9-476b-87c6-d5f1169e3090", "doc_id": "427fb293-6de9-476b-87c6-d5f1169e3090", "ref_doc_id": "427fb293-6de9-476b-87c6-d5f1169e3090", "chroma:document": "\u26a1\ufe0f Inference\n\n- An interactive-demo on [TrOCR handwritten character recognition](https://huggingface.co/spaces/nielsr/TrOCR-handwritten).\n\n## Inference\n\nTrOCR's [`VisionEncoderDecoder`] model accepts images as input and makes use of\n[`~generation.GenerationMixin.generate`] to autoregressively generate text given the input image.\n\nThe [`ViTImageProcessor`/`DeiTImageProcessor`] class is responsible for preprocessing the input image and\n[`RobertaTokenizer`/`XLMRobertaTokenizer`] decodes the generated target tokens to the target string. The\n[`TrOCRProcessor`] wraps [`ViTImageProcessor`/`DeiTImageProcessor`] and [`RobertaTokenizer`/`XLMRobertaTokenizer`]\ninto a single instance to both extract the input features and decode the predicted token ids.\n\n- Step-by-step Optical Character Recognition (OCR)\n\n``` py\n>>> from transformers import TrOCRProcessor, VisionEncoderDecoderModel\n>>> import requests\n>>> from PIL import Image\n\n>>> processor = TrOCRProcessor.from_pretrained(\"microsoft/trocr-base-handwritten\")\n>>> model = VisionEncoderDecoderModel.from_pretrained(\"microsoft/trocr-base-handwritten\")\n\n>>> # load image from the IAM dataset\n>>> url = \"https://fki.tic.heia-fr.ch/static/img/a01-122-02.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw).convert(\"RGB\")\n\n>>> pixel_values = processor(image, return_tensors=\"pt\").pixel_values\n>>> generated_ids = model.generate(pixel_values)\n\n>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]\n```\n\nSee the [model hub](https://huggingface.co/models?filter=trocr) to look for TrOCR checkpoints.\n\n## TrOCRConfig\n\n[[autodoc]] TrOCRConfig\n\n## TrOCRProcessor\n\n[[autodoc]] TrOCRProcessor\n    - __call__\n    - from_pretrained\n    - save_pretrained\n    - batch_decode\n    - decode\n\n## TrOCRForCausalLM\n\n[[autodoc]] TrOCRForCausalLM\n     - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: c3070402-29a8-46f4-bfff-ac14b2ecd6b5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bartpho", "title": "BARTpho", "tokens": 959, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c3070402-29a8-46f4-bfff-ac14b2ecd6b5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bartpho\", \"title\": \"BARTpho\", \"tokens\": 959, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b3978b25-b1ec-4d9d-aab2-23195938b8e3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bartpho\", \"title\": \"BARTpho\", \"tokens\": 959, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8c44a25c732b5d6ef1588fb60f7c6dbadf4be3d9a29198625ce69077015d75ff\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"12062b97-3455-4c2b-a7ba-00f9856945bb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"659ef2213e8b23c6080b70df8143e0c7bbcd6c6fc793770b0453b383b7db4997\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2891, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b3978b25-b1ec-4d9d-aab2-23195938b8e3", "doc_id": "b3978b25-b1ec-4d9d-aab2-23195938b8e3", "ref_doc_id": "b3978b25-b1ec-4d9d-aab2-23195938b8e3", "chroma:document": "# BARTpho\n\n## Overview\n\nThe BARTpho model was proposed in [BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamese](https://arxiv.org/abs/2109.09701) by Nguyen Luong Tran, Duong Minh Le and Dat Quoc Nguyen.\n\nThe abstract from the paper is the following:\n\n*We present BARTpho with two versions -- BARTpho_word and BARTpho_syllable -- the first public large-scale monolingual\nsequence-to-sequence models pre-trained for Vietnamese. Our BARTpho uses the \"large\" architecture and pre-training\nscheme of the sequence-to-sequence denoising model BART, thus especially suitable for generative NLP tasks. Experiments\non a downstream task of Vietnamese text summarization show that in both automatic and human evaluations, our BARTpho\noutperforms the strong baseline mBART and improves the state-of-the-art. We release BARTpho to facilitate future\nresearch and applications of generative Vietnamese NLP tasks.*\n\nThis model was contributed by [dqnguyen](https://huggingface.co/dqnguyen). The original code can be found [here](https://github.com/VinAIResearch/BARTpho).\n\n## Usage example\n\n```python\n>>> import torch\n>>> from transformers import AutoModel, AutoTokenizer\n\n>>> bartpho = AutoModel.from_pretrained(\"vinai/bartpho-syllable\")\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"vinai/bartpho-syllable\")\n\n>>> line = \"Ch\u00fang t\u00f4i l\u00e0 nh\u1eefng nghi\u00ean c\u1ee9u vi\u00ean.\"\n\n>>> input_ids = tokenizer(line, return_tensors=\"pt\")\n\n>>> with torch.no_grad():\n...     features = bartpho(**input_ids)  # Models outputs are now tuples\n\n>>> # With TensorFlow 2.0+:\n>>> from transformers import TFAutoModel\n\n>>> bartpho = TFAutoModel.from_pretrained(\"vinai/bartpho-syllable\")\n>>> input_ids = tokenizer(line, return_tensors=\"tf\")\n>>> features = bartpho(**input_ids)\n```\n\n## Usage tips\n\n- Following mBART, BARTpho uses the \"large\" architecture of BART with an additional layer-normalization layer on top of\n  both the encoder and decoder. Thus, usage examples in the [documentation of BART](bart), when adapting to use\n  with BARTpho, should be adjusted by replacing the BART-specialized classes with the mBART-specialized counterparts.\n  For example:\n\n```python\n>>> from transformers import MBartForConditionalGeneration\n\n>>> bartpho = MBartForConditionalGeneration.from_pretrained(\"vinai/bartpho-syllable\")\n>>> TXT = \"Ch\u00fang t\u00f4i l\u00e0 <mask> nghi\u00ean c\u1ee9u vi\u00ean.\"\n>>> input_ids = tokenizer([TXT], return_tensors=\"pt\")[\"input_ids\"]\n>>> logits = bartpho(input_ids).logits\n>>> masked_index = (input_ids[0] == tokenizer.mask_token_id).nonzero().item()\n>>> probs = logits[0, masked_index].softmax(dim=0)\n>>> values, predictions = probs.topk(5)\n>>> print(tokenizer.decode(predictions).split())\n```\n\n- This implementation is only for tokenization: \"monolingual_vocab_file\" consists of Vietnamese-specialized types\n  extracted from the pre-trained SentencePiece model \"vocab_file\" that is available from the multilingual XLM-RoBERTa."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 12062b97-3455-4c2b-a7ba-00f9856945bb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bartpho", "title": "BARTpho", "tokens": 959, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"12062b97-3455-4c2b-a7ba-00f9856945bb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bartpho\", \"title\": \"BARTpho\", \"tokens\": 959, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b3978b25-b1ec-4d9d-aab2-23195938b8e3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bartpho\", \"title\": \"BARTpho\", \"tokens\": 959, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8c44a25c732b5d6ef1588fb60f7c6dbadf4be3d9a29198625ce69077015d75ff\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c3070402-29a8-46f4-bfff-ac14b2ecd6b5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bartpho\", \"title\": \"BARTpho\", \"tokens\": 959, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1373c9e34caffc244c05b548585112bd12ce3e81c8f70cbc2849ecd3aa1bc36f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1358, \"end_char_idx\": 3154, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b3978b25-b1ec-4d9d-aab2-23195938b8e3", "doc_id": "b3978b25-b1ec-4d9d-aab2-23195938b8e3", "ref_doc_id": "b3978b25-b1ec-4d9d-aab2-23195938b8e3", "chroma:document": ">>> input_ids = tokenizer(line, return_tensors=\"pt\")\n\n>>> with torch.no_grad():\n...     features = bartpho(**input_ids)  # Models outputs are now tuples\n\n>>> # With TensorFlow 2.0+:\n>>> from transformers import TFAutoModel\n\n>>> bartpho = TFAutoModel.from_pretrained(\"vinai/bartpho-syllable\")\n>>> input_ids = tokenizer(line, return_tensors=\"tf\")\n>>> features = bartpho(**input_ids)\n```\n\n## Usage tips\n\n- Following mBART, BARTpho uses the \"large\" architecture of BART with an additional layer-normalization layer on top of\n  both the encoder and decoder. Thus, usage examples in the [documentation of BART](bart), when adapting to use\n  with BARTpho, should be adjusted by replacing the BART-specialized classes with the mBART-specialized counterparts.\n  For example:\n\n```python\n>>> from transformers import MBartForConditionalGeneration\n\n>>> bartpho = MBartForConditionalGeneration.from_pretrained(\"vinai/bartpho-syllable\")\n>>> TXT = \"Ch\u00fang t\u00f4i l\u00e0 <mask> nghi\u00ean c\u1ee9u vi\u00ean.\"\n>>> input_ids = tokenizer([TXT], return_tensors=\"pt\")[\"input_ids\"]\n>>> logits = bartpho(input_ids).logits\n>>> masked_index = (input_ids[0] == tokenizer.mask_token_id).nonzero().item()\n>>> probs = logits[0, masked_index].softmax(dim=0)\n>>> values, predictions = probs.topk(5)\n>>> print(tokenizer.decode(predictions).split())\n```\n\n- This implementation is only for tokenization: \"monolingual_vocab_file\" consists of Vietnamese-specialized types\n  extracted from the pre-trained SentencePiece model \"vocab_file\" that is available from the multilingual XLM-RoBERTa.\n  Other languages, if employing this pre-trained multilingual SentencePiece model \"vocab_file\" for subword\n  segmentation, can reuse BartphoTokenizer with their own language-specialized \"monolingual_vocab_file\".\n\n## BartphoTokenizer\n\n[[autodoc]] BartphoTokenizer"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e82f5414-3d5e-4e1f-b683-160d44af8842
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/biogpt", "title": "BioGPT", "tokens": 949, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e82f5414-3d5e-4e1f-b683-160d44af8842\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/biogpt\", \"title\": \"BioGPT\", \"tokens\": 949, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"55a266c2-ced3-4c87-97a2-941a1a2582f8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/biogpt\", \"title\": \"BioGPT\", \"tokens\": 949, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1f0e9175b8bf15f63a0372f9f8c8b5083f311f6022d15799137b2d8d03dbad48\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"302801b5-1d2b-4b0c-b7d2-de1fa9b086cd\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"222cf8c3c392d7965a3fc46ada93147f1de257bfc3f673339839c8b4d0b998a0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3231, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "55a266c2-ced3-4c87-97a2-941a1a2582f8", "doc_id": "55a266c2-ced3-4c87-97a2-941a1a2582f8", "ref_doc_id": "55a266c2-ced3-4c87-97a2-941a1a2582f8", "chroma:document": "# BioGPT\n\n## Overview\n\nThe BioGPT model was proposed in [BioGPT: generative pre-trained transformer for biomedical text generation and mining](https://academic.oup.com/bib/advance-article/doi/10.1093/bib/bbac409/6713511?guestAccessKey=a66d9b5d-4f83-4017-bb52-405815c907b9) by Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon and Tie-Yan Liu. BioGPT is a domain-specific generative pre-trained Transformer language model for biomedical text generation and mining. BioGPT follows the Transformer language model backbone, and is pre-trained on 15M PubMed abstracts from scratch.\n\nThe abstract from the paper is the following:\n\n*Pre-trained language models have attracted increasing attention in the biomedical domain, inspired by their great success in the general natural language domain. Among the two main branches of pre-trained language models in the general language domain, i.e. BERT (and its variants) and GPT (and its variants), the first one has been extensively studied in the biomedical domain, such as BioBERT and PubMedBERT. While they have achieved great success on a variety of discriminative downstream biomedical tasks, the lack of generation ability constrains their application scope. In this paper, we propose BioGPT, a domain-specific generative Transformer language model pre-trained on large-scale biomedical literature. We evaluate BioGPT on six biomedical natural language processing tasks and demonstrate that our model outperforms previous models on most tasks. Especially, we get 44.98%, 38.42% and 40.76% F1 score on BC5CDR, KD-DTI and DDI end-to-end relation extraction tasks, respectively, and 78.2% accuracy on PubMedQA, creating a new record. Our case study on text generation further demonstrates the advantage of BioGPT on biomedical literature to generate fluent descriptions for biomedical terms.*\n\nThis model was contributed by [kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/microsoft/BioGPT).\n\n## Usage tips\n\n- BioGPT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than the left.\n- BioGPT was trained with a causal language modeling (CLM) objective and is therefore powerful at predicting the next token in a sequence. Leveraging this feature allows BioGPT to generate syntactically coherent text as it can be observed in the run_generation.py example script.\n- The model can take the `past_key_values` (for PyTorch) as input, which is the previously computed key/value attention pairs. Using this (past_key_values or past) value prevents the model from re-computing pre-computed values in the context of text generation. For PyTorch, see past_key_values argument of the BioGptForCausalLM.forward() method for more information on its usage.\n\n## Resources\n\n- [Causal language modeling task guide](../tasks/language_modeling)\n\n## BioGptConfig\n\n[[autodoc]] BioGptConfig\n\n\n## BioGptTokenizer\n\n[[autodoc]] BioGptTokenizer\n    - save_vocabulary\n\n\n## BioGptModel\n\n[[autodoc]] BioGptModel\n    - forward\n\n\n## BioGptForCausalLM\n\n[[autodoc]] BioGptForCausalLM\n    - forward\n\n    \n## BioGptForTokenClassification\n\n[[autodoc]] BioGptForTokenClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 302801b5-1d2b-4b0c-b7d2-de1fa9b086cd
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/biogpt", "title": "BioGPT", "tokens": 949, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"302801b5-1d2b-4b0c-b7d2-de1fa9b086cd\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/biogpt\", \"title\": \"BioGPT\", \"tokens\": 949, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"55a266c2-ced3-4c87-97a2-941a1a2582f8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/biogpt\", \"title\": \"BioGPT\", \"tokens\": 949, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1f0e9175b8bf15f63a0372f9f8c8b5083f311f6022d15799137b2d8d03dbad48\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e82f5414-3d5e-4e1f-b683-160d44af8842\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/biogpt\", \"title\": \"BioGPT\", \"tokens\": 949, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5310debde05321e72c201fe5a229804509bd9e5be3e9c3053951708c553b0b9f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2944, \"end_char_idx\": 3327, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "55a266c2-ced3-4c87-97a2-941a1a2582f8", "doc_id": "55a266c2-ced3-4c87-97a2-941a1a2582f8", "ref_doc_id": "55a266c2-ced3-4c87-97a2-941a1a2582f8", "chroma:document": "## BioGptTokenizer\n\n[[autodoc]] BioGptTokenizer\n    - save_vocabulary\n\n\n## BioGptModel\n\n[[autodoc]] BioGptModel\n    - forward\n\n\n## BioGptForCausalLM\n\n[[autodoc]] BioGptForCausalLM\n    - forward\n\n    \n## BioGptForTokenClassification\n\n[[autodoc]] BioGptForTokenClassification\n    - forward\n\n\n## BioGptForSequenceClassification\n\n[[autodoc]] BioGptForSequenceClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 1c04c8d1-21cb-4604-877c-7b81c1e644c2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/speecht5", "title": "SpeechT5", "tokens": 845, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1c04c8d1-21cb-4604-877c-7b81c1e644c2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/speecht5\", \"title\": \"SpeechT5\", \"tokens\": 845, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d873360f-586a-4f20-8f68-2d238dfc9dbd\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/speecht5\", \"title\": \"SpeechT5\", \"tokens\": 845, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5ab9b289ae152568516619e94dd97d860fa184aed6ffcf730e5f1035a65dffce\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2738, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d873360f-586a-4f20-8f68-2d238dfc9dbd", "doc_id": "d873360f-586a-4f20-8f68-2d238dfc9dbd", "ref_doc_id": "d873360f-586a-4f20-8f68-2d238dfc9dbd", "chroma:document": "# SpeechT5\n\n## Overview\n\nThe SpeechT5 model was proposed in [SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing](https://arxiv.org/abs/2110.07205) by Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, Furu Wei.\n\nThe abstract from the paper is the following:\n\n*Motivated by the success of T5 (Text-To-Text Transfer Transformer) in pre-trained natural language processing models, we propose a unified-modal SpeechT5 framework that explores the encoder-decoder pre-training for self-supervised speech/text representation learning. The SpeechT5 framework consists of a shared encoder-decoder network and six modal-specific (speech/text) pre/post-nets. After preprocessing the input speech/text through the pre-nets, the shared encoder-decoder network models the sequence-to-sequence transformation, and then the post-nets generate the output in the speech/text modality based on the output of the decoder. Leveraging large-scale unlabeled speech and text data, we pre-train SpeechT5 to learn a unified-modal representation, hoping to improve the modeling capability for both speech and text. To align the textual and speech information into this unified semantic space, we propose a cross-modal vector quantization approach that randomly mixes up speech/text states with latent units as the interface between encoder and decoder. Extensive evaluations show the superiority of the proposed SpeechT5 framework on a wide variety of spoken language processing tasks, including automatic speech recognition, speech synthesis, speech translation, voice conversion, speech enhancement, and speaker identification.*\n\nThis model was contributed by [Matthijs](https://huggingface.co/Matthijs). The original code can be found [here](https://github.com/microsoft/SpeechT5).\n\n## SpeechT5Config\n\n[[autodoc]] SpeechT5Config\n\n## SpeechT5HifiGanConfig\n\n[[autodoc]] SpeechT5HifiGanConfig\n\n## SpeechT5Tokenizer\n\n[[autodoc]] SpeechT5Tokenizer\n    - __call__\n    - save_vocabulary\n    - decode\n    - batch_decode\n\n## SpeechT5FeatureExtractor\n\n[[autodoc]] SpeechT5FeatureExtractor\n    - __call__\n\n## SpeechT5Processor\n\n[[autodoc]] SpeechT5Processor\n    - __call__\n    - pad\n    - from_pretrained\n    - save_pretrained\n    - batch_decode\n    - decode\n\n## SpeechT5Model\n\n[[autodoc]] SpeechT5Model\n    - forward\n\n## SpeechT5ForSpeechToText\n\n[[autodoc]] SpeechT5ForSpeechToText\n    - forward\n\n## SpeechT5ForTextToSpeech\n\n[[autodoc]] SpeechT5ForTextToSpeech\n    - forward\n    - generate\n\n## SpeechT5ForSpeechToSpeech\n\n[[autodoc]] SpeechT5ForSpeechToSpeech\n    - forward\n    - generate_speech\n\n## SpeechT5HifiGan\n\n[[autodoc]] SpeechT5HifiGan\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 9c20daa7-ce47-4383-b90c-dc1246b2ea0b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mobilenet_v1", "title": "MobileNet V1", "tokens": 1239, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9c20daa7-ce47-4383-b90c-dc1246b2ea0b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilenet_v1\", \"title\": \"MobileNet V1\", \"tokens\": 1239, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"67ea26be-073f-493f-af1f-cc539db15387\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilenet_v1\", \"title\": \"MobileNet V1\", \"tokens\": 1239, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cc756dcef81023f96529c34925be84d96b335510f0e706d4887bbf897364c5f5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d1c709ed-90c6-4597-998b-5e1c9b3fc48e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e723db98fb83ed78eabc5540a51f436096ac48e25bc0f039e2c90155d197212c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3373, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "67ea26be-073f-493f-af1f-cc539db15387", "doc_id": "67ea26be-073f-493f-af1f-cc539db15387", "ref_doc_id": "67ea26be-073f-493f-af1f-cc539db15387", "chroma:document": "# MobileNet V1\n\n## Overview\n\nThe MobileNet model was proposed in [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://arxiv.org/abs/1704.04861) by Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, Hartwig Adam.\n\nThe abstract from the paper is the following:\n\n*We present a class of efficient models called MobileNets for mobile and embedded vision applications. MobileNets are based on a streamlined architecture that uses depth-wise separable convolutions to build light weight deep neural networks. We introduce two simple global hyper-parameters that efficiently trade off between latency and accuracy. These hyper-parameters allow the model builder to choose the right sized model for their application based on the constraints of the problem. We present extensive experiments on resource and accuracy tradeoffs and show strong performance compared to other popular models on ImageNet classification. We then demonstrate the effectiveness of MobileNets across a wide range of applications and use cases including object detection, finegrain classification, face attributes and large scale geo-localization.*\n\nThis model was contributed by [matthijs](https://huggingface.co/Matthijs). The original code and weights can be found [here](https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet_v1.md).\n\n## Usage tips\n\n- The checkpoints are named **mobilenet\\_v1\\_*depth*\\_*size***, for example **mobilenet\\_v1\\_1.0\\_224**, where **1.0** is the depth multiplier (sometimes also referred to as \"alpha\" or the width multiplier) and **224** is the resolution of the input images the model was trained on.\n\n- Even though the checkpoint is trained on images of specific size, the model will work on images of any size. The smallest supported image size is 32x32.\n\n- One can use [`MobileNetV1ImageProcessor`] to prepare images for the model.\n\n- The available image classification checkpoints are pre-trained on [ImageNet-1k](https://huggingface.co/datasets/imagenet-1k) (also referred to as ILSVRC 2012, a collection of 1.3 million images and 1,000 classes). However, the model predicts 1001 classes: the 1000 classes from ImageNet plus an extra \u201cbackground\u201d class (index 0).\n\n- The original TensorFlow checkpoints use different padding rules than PyTorch, requiring the model to determine the padding amount at inference time, since this depends on the input image size. To use native PyTorch padding behavior, create a [`MobileNetV1Config`] with `tf_padding = False`.\n\nUnsupported features:\n\n- The [`MobileNetV1Model`] outputs a globally pooled version of the last hidden state. In the original model it is possible to use a 7x7 average pooling layer with stride 2 instead of global pooling. For larger inputs, this gives a pooled output that is larger than 1x1 pixel. The HuggingFace implementation does not support this.\n\n- It is currently not possible to specify an `output_stride`. For smaller output strides, the original model invokes dilated convolution to prevent the spatial resolution from being reduced further. The output stride of the HuggingFace model is always 32.\n\n- The original TensorFlow checkpoints include quantized models. We do not support these models as they include additional \"FakeQuantization\" operations to unquantize the weights."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: d1c709ed-90c6-4597-998b-5e1c9b3fc48e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mobilenet_v1", "title": "MobileNet V1", "tokens": 1239, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d1c709ed-90c6-4597-998b-5e1c9b3fc48e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilenet_v1\", \"title\": \"MobileNet V1\", \"tokens\": 1239, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"67ea26be-073f-493f-af1f-cc539db15387\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilenet_v1\", \"title\": \"MobileNet V1\", \"tokens\": 1239, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cc756dcef81023f96529c34925be84d96b335510f0e706d4887bbf897364c5f5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9c20daa7-ce47-4383-b90c-dc1246b2ea0b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mobilenet_v1\", \"title\": \"MobileNet V1\", \"tokens\": 1239, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cefc6fa4e0b6711b01ab6a5a93328d435f2e1a89adb533173b5215a8a54ea80d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1721, \"end_char_idx\": 4803, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "67ea26be-073f-493f-af1f-cc539db15387", "doc_id": "67ea26be-073f-493f-af1f-cc539db15387", "ref_doc_id": "67ea26be-073f-493f-af1f-cc539db15387", "chroma:document": "- Even though the checkpoint is trained on images of specific size, the model will work on images of any size. The smallest supported image size is 32x32.\n\n- One can use [`MobileNetV1ImageProcessor`] to prepare images for the model.\n\n- The available image classification checkpoints are pre-trained on [ImageNet-1k](https://huggingface.co/datasets/imagenet-1k) (also referred to as ILSVRC 2012, a collection of 1.3 million images and 1,000 classes). However, the model predicts 1001 classes: the 1000 classes from ImageNet plus an extra \u201cbackground\u201d class (index 0).\n\n- The original TensorFlow checkpoints use different padding rules than PyTorch, requiring the model to determine the padding amount at inference time, since this depends on the input image size. To use native PyTorch padding behavior, create a [`MobileNetV1Config`] with `tf_padding = False`.\n\nUnsupported features:\n\n- The [`MobileNetV1Model`] outputs a globally pooled version of the last hidden state. In the original model it is possible to use a 7x7 average pooling layer with stride 2 instead of global pooling. For larger inputs, this gives a pooled output that is larger than 1x1 pixel. The HuggingFace implementation does not support this.\n\n- It is currently not possible to specify an `output_stride`. For smaller output strides, the original model invokes dilated convolution to prevent the spatial resolution from being reduced further. The output stride of the HuggingFace model is always 32.\n\n- The original TensorFlow checkpoints include quantized models. We do not support these models as they include additional \"FakeQuantization\" operations to unquantize the weights.\n\n- It's common to extract the output from the pointwise layers at indices 5, 11, 12, 13 for downstream purposes. Using `output_hidden_states=True` returns the output from all intermediate layers. There is currently no way to limit this to specific layers.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with MobileNetV1.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`MobileNetV1ForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## MobileNetV1Config\n\n[[autodoc]] MobileNetV1Config\n\n## MobileNetV1FeatureExtractor\n\n[[autodoc]] MobileNetV1FeatureExtractor\n    - preprocess\n\n## MobileNetV1ImageProcessor\n\n[[autodoc]] MobileNetV1ImageProcessor\n    - preprocess\n\n## MobileNetV1Model\n\n[[autodoc]] MobileNetV1Model\n    - forward\n\n## MobileNetV1ForImageClassification\n\n[[autodoc]] MobileNetV1ForImageClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: c23570ec-19dc-45ef-9c7d-21eac25a5056
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/jetmoe", "title": "JetMoe", "tokens": 506, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c23570ec-19dc-45ef-9c7d-21eac25a5056\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jetmoe\", \"title\": \"JetMoe\", \"tokens\": 506, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e663ecc6-63c8-43da-8b5b-7343951fe91b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jetmoe\", \"title\": \"JetMoe\", \"tokens\": 506, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8684ce766325fedc198ac2435f441b62b98fbbe99672150e91a13eadec9131e7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1247, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e663ecc6-63c8-43da-8b5b-7343951fe91b", "doc_id": "e663ecc6-63c8-43da-8b5b-7343951fe91b", "ref_doc_id": "e663ecc6-63c8-43da-8b5b-7343951fe91b", "chroma:document": "# JetMoe\n\n## Overview\n\n**JetMoe-8B** is an 8B Mixture-of-Experts (MoE) language model developed by [Yikang Shen](https://scholar.google.com.hk/citations?user=qff5rRYAAAAJ) and [MyShell](https://myshell.ai/).\nJetMoe project aims to provide a LLaMA2-level performance and efficient language model with a limited budget.\nTo achieve this goal, JetMoe uses a sparsely activated architecture inspired by the [ModuleFormer](https://arxiv.org/abs/2306.04640). \nEach JetMoe block consists of two MoE layers: Mixture of Attention Heads and Mixture of MLP Experts.\nGiven the input tokens, it activates a subset of its experts to process them.\nThis sparse activation schema enables JetMoe to achieve much better training throughput than similar size dense models. \nThe training throughput of JetMoe-8B is around 100B tokens per day on a cluster of 96 H100 GPUs with a straightforward 3-way pipeline parallelism strategy.\n\nThis model was contributed by [Yikang Shen](https://huggingface.co/YikangS).\n\n\n## JetMoeConfig\n\n[[autodoc]] JetMoeConfig\n\n## JetMoeModel\n\n[[autodoc]] JetMoeModel\n    - forward\n\n## JetMoeForCausalLM\n\n[[autodoc]] JetMoeForCausalLM\n    - forward\n\n## JetMoeForSequenceClassification\n\n[[autodoc]] JetMoeForSequenceClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 130be32a-a792-445a-a8a3-341259e943d9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta-xl", "title": "XLM-RoBERTa-XL", "tokens": 849, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"130be32a-a792-445a-a8a3-341259e943d9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-roberta-xl\", \"title\": \"XLM-RoBERTa-XL\", \"tokens\": 849, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"21260625-3704-4554-8734-a32316a5e0bc\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-roberta-xl\", \"title\": \"XLM-RoBERTa-XL\", \"tokens\": 849, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"75d481267b5776122b84e5856634c7c7bf373394d2081b29254048efaa23d3fe\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2614, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "21260625-3704-4554-8734-a32316a5e0bc", "doc_id": "21260625-3704-4554-8734-a32316a5e0bc", "ref_doc_id": "21260625-3704-4554-8734-a32316a5e0bc", "chroma:document": "# XLM-RoBERTa-XL\n\n## Overview\n\nThe XLM-RoBERTa-XL model was proposed in [Larger-Scale Transformers for Multilingual Masked Language Modeling](https://arxiv.org/abs/2105.00572) by Naman Goyal, Jingfei Du, Myle Ott, Giri Anantharaman, Alexis Conneau. \n\nThe abstract from the paper is the following:\n\n*Recent work has demonstrated the effectiveness of cross-lingual language model pretraining for cross-lingual understanding. In this study, we present the results of two larger multilingual masked language models, with 3.5B and 10.7B parameters. Our two new models dubbed XLM-R XL and XLM-R XXL outperform XLM-R by 1.8% and 2.4% average accuracy on XNLI. Our model also outperforms the RoBERTa-Large model on several English tasks of the GLUE benchmark by 0.3% on average while handling 99 more languages. This suggests pretrained models with larger capacity may obtain both strong performance on high-resource languages while greatly improving low-resource languages. We make our code and models publicly available.*\n\nThis model was contributed by [Soonhwan-Kwon](https://github.com/Soonhwan-Kwon) and [stefan-it](https://huggingface.co/stefan-it). The original code can be found [here](https://github.com/pytorch/fairseq/tree/master/examples/xlmr).\n\n## Usage tips\n\nXLM-RoBERTa-XL is a multilingual model trained on 100 different languages. Unlike some XLM multilingual models, it does \nnot require `lang` tensors to understand which language is used, and should be able to determine the correct \nlanguage from the input ids.\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## XLMRobertaXLConfig\n\n[[autodoc]] XLMRobertaXLConfig\n\n## XLMRobertaXLModel\n\n[[autodoc]] XLMRobertaXLModel\n    - forward\n\n## XLMRobertaXLForCausalLM\n\n[[autodoc]] XLMRobertaXLForCausalLM\n    - forward\n\n## XLMRobertaXLForMaskedLM\n\n[[autodoc]] XLMRobertaXLForMaskedLM\n    - forward\n\n## XLMRobertaXLForSequenceClassification\n\n[[autodoc]] XLMRobertaXLForSequenceClassification\n    - forward\n\n## XLMRobertaXLForMultipleChoice\n\n[[autodoc]] XLMRobertaXLForMultipleChoice\n    - forward\n\n## XLMRobertaXLForTokenClassification\n\n[[autodoc]] XLMRobertaXLForTokenClassification\n    - forward\n\n## XLMRobertaXLForQuestionAnswering\n\n[[autodoc]] XLMRobertaXLForQuestionAnswering\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 13a05da0-1552-4b78-aaf9-8cd923c32ecb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bit", "title": "Big Transfer (BiT)", "tokens": 890, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"13a05da0-1552-4b78-aaf9-8cd923c32ecb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bit\", \"title\": \"Big Transfer (BiT)\", \"tokens\": 890, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f99d93cc-3e91-479b-aa2a-9fd1f1181d8d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bit\", \"title\": \"Big Transfer (BiT)\", \"tokens\": 890, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"528ac4260d29d9fc92230c49b6f5fa694d0a5405a764fb143540ad9638c0a137\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3082, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f99d93cc-3e91-479b-aa2a-9fd1f1181d8d", "doc_id": "f99d93cc-3e91-479b-aa2a-9fd1f1181d8d", "ref_doc_id": "f99d93cc-3e91-479b-aa2a-9fd1f1181d8d", "chroma:document": "# Big Transfer (BiT)\n\n## Overview\n\nThe BiT model was proposed in [Big Transfer (BiT): General Visual Representation Learning](https://arxiv.org/abs/1912.11370) by Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, Neil Houlsby.\nBiT is a simple recipe for scaling up pre-training of [ResNet](resnet)-like architectures (specifically, ResNetv2). The method results in significant improvements for transfer learning.\n\nThe abstract from the paper is the following:\n\n*Transfer of pre-trained representations improves sample efficiency and simplifies hyperparameter tuning when training deep neural networks for vision. We revisit the paradigm of pre-training on large supervised datasets and fine-tuning the model on a target task. We scale up pre-training, and propose a simple recipe that we call Big Transfer (BiT). By combining a few carefully selected components, and transferring using a simple heuristic, we achieve strong performance on over 20 datasets. BiT performs well across a surprisingly wide range of data regimes -- from 1 example per class to 1M total examples. BiT achieves 87.5% top-1 accuracy on ILSVRC-2012, 99.4% on CIFAR-10, and 76.3% on the 19 task Visual Task Adaptation Benchmark (VTAB). On small datasets, BiT attains 76.8% on ILSVRC-2012 with 10 examples per class, and 97.0% on CIFAR-10 with 10 examples per class. We conduct detailed analysis of the main components that lead to high transfer performance.*\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/google-research/big_transfer).\n\n## Usage tips\n\n- BiT models are equivalent to ResNetv2 in terms of architecture, except that: 1) all batch normalization layers are replaced by [group normalization](https://arxiv.org/abs/1803.08494),\n2) [weight standardization](https://arxiv.org/abs/1903.10520) is used for convolutional layers. The authors show that the combination of both is useful for training with large batch sizes, and has a significant\nimpact on transfer learning.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with BiT.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`BitForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## BitConfig\n\n[[autodoc]] BitConfig\n\n## BitImageProcessor\n\n[[autodoc]] BitImageProcessor\n    - preprocess\n\n## BitModel\n\n[[autodoc]] BitModel\n    - forward\n\n## BitForImageClassification\n\n[[autodoc]] BitForImageClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 54a533e4-b846-49d9-8b84-6199f3ffbaa8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/idefics", "title": "IDEFICS", "tokens": 743, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"54a533e4-b846-49d9-8b84-6199f3ffbaa8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics\", \"title\": \"IDEFICS\", \"tokens\": 743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f2115022-d44b-41db-bae9-35d11628efd3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics\", \"title\": \"IDEFICS\", \"tokens\": 743, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b941ca0f6bc20ee29516c8e0810f9f8fe26d387aeec4c12f95043ab212cf8a90\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2319, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f2115022-d44b-41db-bae9-35d11628efd3", "doc_id": "f2115022-d44b-41db-bae9-35d11628efd3", "ref_doc_id": "f2115022-d44b-41db-bae9-35d11628efd3", "chroma:document": "# IDEFICS\n\n## Overview\n\nThe IDEFICS model was proposed in [OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents\n](https://huggingface.co/papers/2306.16527\n) by Hugo Lauren\u00e7on, Lucile Saulnier, L\u00e9o Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, Victor Sanh\n\nThe abstract from the paper is the following:\n\n*Large multimodal models trained on natural documents, which interleave images and text, outperform models trained on image-text pairs on various multimodal benchmarks that require reasoning over one or multiple images to generate a text. However, the datasets used to train these models have not been released, and the collection process has not been fully specified. We introduce the OBELICS dataset, an open web-scale filtered dataset of interleaved image-text documents comprising 141 million web pages extracted from Common Crawl, 353 million associated images, and 115 billion text tokens. We describe the dataset creation process, present comprehensive filtering rules, and provide an analysis of the dataset's content. To show the viability of OBELISC, we train an 80 billion parameters vision and language model on the dataset and obtain competitive performance on various multimodal benchmarks. We release the code to reproduce the dataset along with the dataset itself.*\n\nThis model was contributed by [HuggingFaceM4](https://huggingface.co/HuggingFaceM4). The original code can be found [here](<INSERT LINK TO GITHUB REPO HERE>). (TODO: don't have a public link yet).\n\n\n<Tip warning={true}>\n\nIDEFICS modeling code in Transformers is for finetuning and inferencing the pre-trained IDEFICS models.\n\nTo train a new IDEFICS model from scratch use the m4 codebase (a link will be provided once it's made public)\n\n</Tip>\n\n\n## IdeficsConfig\n\n[[autodoc]] IdeficsConfig\n\n## IdeficsModel\n\n[[autodoc]] IdeficsModel\n    - forward\n\n## IdeficsForVisionText2Text\n\n[[autodoc]] IdeficsForVisionText2Text\n    - forward\n\n## TFIdeficsModel\n\n[[autodoc]] TFIdeficsModel\n    - call\n\n## TFIdeficsForVisionText2Text\n\n[[autodoc]] TFIdeficsForVisionText2Text\n    - call\n\n## IdeficsImageProcessor\n\n[[autodoc]] IdeficsImageProcessor\n    - preprocess\n\n## IdeficsProcessor\n\n[[autodoc]] IdeficsProcessor\n    - __call__"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 7b5916b6-3faf-4c90-a5cf-22c4eec3d77d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vilt", "title": "ViLT", "tokens": 1055, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7b5916b6-3faf-4c90-a5cf-22c4eec3d77d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vilt\", \"title\": \"ViLT\", \"tokens\": 1055, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ab6afca4-f9ff-4791-a08c-bef6083949d9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vilt\", \"title\": \"ViLT\", \"tokens\": 1055, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"40866bcab7c8c567834d5846571dc84f5d1ba53fb55cb6326439e5d5d4c628b8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0d6e4f7e-b04b-49f9-a160-a3e15511804e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e1bdb4b6cbed24efffabdc6e056d3eadf630777174a9729feb5d5d62dbead28d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2977, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ab6afca4-f9ff-4791-a08c-bef6083949d9", "doc_id": "ab6afca4-f9ff-4791-a08c-bef6083949d9", "ref_doc_id": "ab6afca4-f9ff-4791-a08c-bef6083949d9", "chroma:document": "# ViLT\n\n## Overview\n\nThe ViLT model was proposed in [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://arxiv.org/abs/2102.03334)\nby Wonjae Kim, Bokyung Son, Ildoo Kim. ViLT incorporates text embeddings into a Vision Transformer (ViT), allowing it to have a minimal design\nfor Vision-and-Language Pre-training (VLP).\n\nThe abstract from the paper is the following:\n\n*Vision-and-Language Pre-training (VLP) has improved performance on various joint vision-and-language downstream tasks.\nCurrent approaches to VLP heavily rely on image feature extraction processes, most of which involve region supervision\n(e.g., object detection) and the convolutional architecture (e.g., ResNet). Although disregarded in the literature, we\nfind it problematic in terms of both (1) efficiency/speed, that simply extracting input features requires much more\ncomputation than the multimodal interaction steps; and (2) expressive power, as it is upper bounded to the expressive\npower of the visual embedder and its predefined visual vocabulary. In this paper, we present a minimal VLP model,\nVision-and-Language Transformer (ViLT), monolithic in the sense that the processing of visual inputs is drastically\nsimplified to just the same convolution-free manner that we process textual inputs. We show that ViLT is up to tens of\ntimes faster than previous VLP models, yet with competitive or better downstream task performance.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vilt_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> ViLT architecture. Taken from the <a href=\"https://arxiv.org/abs/2102.03334\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/dandelin/ViLT).\n\n## Usage tips\n\n- The quickest way to get started with ViLT is by checking the [example notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/ViLT)\n  (which showcase both inference and fine-tuning on custom data).\n- ViLT is a model that takes both `pixel_values` and `input_ids` as input. One can use [`ViltProcessor`] to prepare data for the model.\n  This processor wraps a image processor (for the image modality) and a tokenizer (for the language modality) into one.\n- ViLT is trained with images of various sizes: the authors resize the shorter edge of input images to 384 and limit the longer edge to\n  under 640 while preserving the aspect ratio. To make batching of images possible, the authors use a `pixel_mask` that indicates\n  which pixel values are real and which are padding. [`ViltProcessor`] automatically creates this for you.\n- The design of ViLT is very similar to that of a standard Vision Transformer (ViT). The only difference is that the model includes\n  additional embedding layers for the language modality.\n- The PyTorch version of this model is only available in torch 1.10 and higher."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0d6e4f7e-b04b-49f9-a160-a3e15511804e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vilt", "title": "ViLT", "tokens": 1055, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0d6e4f7e-b04b-49f9-a160-a3e15511804e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vilt\", \"title\": \"ViLT\", \"tokens\": 1055, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ab6afca4-f9ff-4791-a08c-bef6083949d9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vilt\", \"title\": \"ViLT\", \"tokens\": 1055, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"40866bcab7c8c567834d5846571dc84f5d1ba53fb55cb6326439e5d5d4c628b8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7b5916b6-3faf-4c90-a5cf-22c4eec3d77d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vilt\", \"title\": \"ViLT\", \"tokens\": 1055, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f0412cf16d1609611eada350244e6b16fcb969917e09179cde87e6c2b2535e18\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1442, \"end_char_idx\": 3692, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ab6afca4-f9ff-4791-a08c-bef6083949d9", "doc_id": "ab6afca4-f9ff-4791-a08c-bef6083949d9", "ref_doc_id": "ab6afca4-f9ff-4791-a08c-bef6083949d9", "chroma:document": "*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vilt_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> ViLT architecture. Taken from the <a href=\"https://arxiv.org/abs/2102.03334\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/dandelin/ViLT).\n\n## Usage tips\n\n- The quickest way to get started with ViLT is by checking the [example notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/ViLT)\n  (which showcase both inference and fine-tuning on custom data).\n- ViLT is a model that takes both `pixel_values` and `input_ids` as input. One can use [`ViltProcessor`] to prepare data for the model.\n  This processor wraps a image processor (for the image modality) and a tokenizer (for the language modality) into one.\n- ViLT is trained with images of various sizes: the authors resize the shorter edge of input images to 384 and limit the longer edge to\n  under 640 while preserving the aspect ratio. To make batching of images possible, the authors use a `pixel_mask` that indicates\n  which pixel values are real and which are padding. [`ViltProcessor`] automatically creates this for you.\n- The design of ViLT is very similar to that of a standard Vision Transformer (ViT). The only difference is that the model includes\n  additional embedding layers for the language modality.\n- The PyTorch version of this model is only available in torch 1.10 and higher.\n\n## ViltConfig\n\n[[autodoc]] ViltConfig\n\n## ViltFeatureExtractor\n\n[[autodoc]] ViltFeatureExtractor\n    - __call__\n\n## ViltImageProcessor\n\n[[autodoc]] ViltImageProcessor\n    - preprocess\n\n## ViltProcessor\n\n[[autodoc]] ViltProcessor\n    - __call__\n\n## ViltModel\n\n[[autodoc]] ViltModel\n    - forward\n\n## ViltForMaskedLM\n\n[[autodoc]] ViltForMaskedLM\n    - forward\n\n## ViltForQuestionAnswering\n\n[[autodoc]] ViltForQuestionAnswering\n    - forward\n\n## ViltForImagesAndTextClassification\n\n[[autodoc]] ViltForImagesAndTextClassification\n    - forward\n\n## ViltForImageAndTextRetrieval\n\n[[autodoc]] ViltForImageAndTextRetrieval\n    - forward\n\n## ViltForTokenClassification\n\n[[autodoc]] ViltForTokenClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0b3ac0d3-f761-437c-acec-e1a82fc26c92
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mpt", "title": "MPT", "tokens": 704, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0b3ac0d3-f761-437c-acec-e1a82fc26c92\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mpt\", \"title\": \"MPT\", \"tokens\": 704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"354ef785-1919-4196-87f1-00426df9d9a5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mpt\", \"title\": \"MPT\", \"tokens\": 704, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"09a8105580481f1d98ee1bebfc0a5025d8e3363e4fc5f2d2ed794ccf4693b168\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2103, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "354ef785-1919-4196-87f1-00426df9d9a5", "doc_id": "354ef785-1919-4196-87f1-00426df9d9a5", "ref_doc_id": "354ef785-1919-4196-87f1-00426df9d9a5", "chroma:document": "# MPT\n\n## Overview\n\nThe MPT model was proposed by the [MosaicML](https://www.mosaicml.com/) team and released with multiple sizes and finetuned variants. The MPT models is a series of open source and commercially usable LLMs pre-trained on 1T tokens. \n\nMPT models are GPT-style decoder-only transformers with several improvements: performance-optimized layer implementations, architecture changes that provide greater training stability, and the elimination of context length limits by replacing positional embeddings with ALiBi. \n\n- MPT base: MPT base pre-trained models on next token prediction \n- MPT instruct: MPT base models fine-tuned on instruction based tasks\n- MPT storywriter: MPT base models fine-tuned for 2500 steps on 65k-token excerpts of fiction books contained in the books3 corpus, this enables the model to handle very long sequences\n\nThe original code is available at the  [`llm-foundry`](https://github.com/mosaicml/llm-foundry/tree/main) repository.\n\nRead more about it [in the release blogpost](https://www.mosaicml.com/blog/mpt-7b)\n\n## Usage tips\n\n- Learn more about some techniques behind training of the model [in this section of llm-foundry repository](https://github.com/mosaicml/llm-foundry/blob/main/TUTORIAL.md#faqs)\n- If you want to use the advanced version of the model (triton kernels, direct flash attention integration), you can still use the original model implementation by adding `trust_remote_code=True` when calling `from_pretrained`.\n\n## Resources\n\n- [Fine-tuning Notebook](https://colab.research.google.com/drive/1HCpQkLL7UXW8xJUJJ29X7QAeNJKO0frZ?usp=sharing) on how to fine-tune MPT-7B on a free Google Colab instance to turn the model into a Chatbot.\n\n## MptConfig\n\n[[autodoc]] MptConfig\n    - all\n\n## MptModel\n\n[[autodoc]] MptModel\n    - forward\n\n## MptForCausalLM\n\n[[autodoc]] MptForCausalLM\n    - forward\n\n## MptForSequenceClassification\n\n[[autodoc]] MptForSequenceClassification\n    - forward\n\n## MptForTokenClassification\n\n[[autodoc]] MptForTokenClassification\n    - forward\n\n## MptForQuestionAnswering\n\n[[autodoc]] MptForQuestionAnswering\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 10181ac3-6fb5-489f-b8fe-a254b4bb386b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/code_llama", "title": "CodeLlama", "tokens": 2064, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"10181ac3-6fb5-489f-b8fe-a254b4bb386b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/code_llama\", \"title\": \"CodeLlama\", \"tokens\": 2064, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb41233b-83da-45a6-8f44-ed52193202be\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/code_llama\", \"title\": \"CodeLlama\", \"tokens\": 2064, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"18affe0a10ce1571f11526a8082f64dd55465a1884ca565bcc3d19dd4e0db30d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4b0b77bd-506c-41c3-9bfd-1c876520e34a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"452619de1cc79c67d6bba72b39f7e7e5b71310d035d82bcd183bb01345f026ed\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2959, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb41233b-83da-45a6-8f44-ed52193202be", "doc_id": "cb41233b-83da-45a6-8f44-ed52193202be", "ref_doc_id": "cb41233b-83da-45a6-8f44-ed52193202be", "chroma:document": "# CodeLlama\n\n## Overview\n\nThe Code Llama model was proposed in [Code Llama: Open Foundation Models for Code](https://ai.meta.com/research/publications/code-llama-open-foundation-models-for-code/) by Baptiste Rozi\u00e8re, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, J\u00e9r\u00e9my Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre D\u00e9fossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, Gabriel Synnaeve.\n\nThe abstract from the paper is the following:\n\n*We release Code Llama, a family of large language models for code based on Llama 2 providing state-of-the-art performance among open models, infilling capabilities, support for large input contexts, and zero-shot instruction following ability for programming tasks. We provide multiple flavors to cover a wide range of applications: foundation models (Code Llama), Python specializations (Code Llama - Python), and instruction-following models (Code Llama - Instruct) with 7B, 13B and 34B parameters each. All models are trained on sequences of 16k tokens and show improvements on inputs with up to 100k tokens. 7B and 13B Code Llama and Code Llama - Instruct variants support infilling based on surrounding content. Code Llama reaches state-of-the-art performance among open models on several code benchmarks, with scores of up to 53% and 55% on HumanEval and MBPP, respectively. Notably, Code Llama - Python 7B outperforms Llama 2 70B on HumanEval and MBPP, and all our models outperform every other publicly available model on MultiPL-E. We release Code Llama under a permissive license that allows for both research and commercial use.*\n\nCheck out all Code Llama model checkpoints [here](https://huggingface.co/models?search=code_llama) and the officially released ones in the [Meta Llama org](https://huggingface.co/meta-llama).\n\nThis model was contributed by [ArthurZucker](https://huggingface.co/ArthurZ). The original code of the authors can be found [here](https://github.com/facebookresearch/llama).\n\n## Usage tips and examples\n\n<Tip warning={true}>\n\nThe `Llama2` family models, on which Code Llama is based, were trained using `bfloat16`, but the original inference uses `float16`. Let's look at the different precisions:\n\n* `float32`: PyTorch convention on model initialization is to load models in `float32`, no matter with which `dtype` the model weights were stored. `transformers` also follows this convention for consistency with PyTorch. This will be picked by default. If you want the `AutoModel` API to cast the load the checkpoints with the storage weights type, you must specify `torch_dtype=\"auto\"`, e.g. `model = AutoModelForCausalLM.from_pretrained(\"path\", torch_dtype = \"auto\")`.\n* `bfloat16`: Code Llama was trained with this precision, so we recommend using it for further training or fine-tuning."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 4b0b77bd-506c-41c3-9bfd-1c876520e34a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/code_llama", "title": "CodeLlama", "tokens": 2064, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4b0b77bd-506c-41c3-9bfd-1c876520e34a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/code_llama\", \"title\": \"CodeLlama\", \"tokens\": 2064, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb41233b-83da-45a6-8f44-ed52193202be\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/code_llama\", \"title\": \"CodeLlama\", \"tokens\": 2064, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"18affe0a10ce1571f11526a8082f64dd55465a1884ca565bcc3d19dd4e0db30d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"10181ac3-6fb5-489f-b8fe-a254b4bb386b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/code_llama\", \"title\": \"CodeLlama\", \"tokens\": 2064, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"05dea4d9a0290fa0a20acc15d36be3089515b889f97f151e7aabcb4848741c7c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7e13f30e-981b-4b8a-a69f-ba9f87fecf90\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2b1cbc272f1079ada86ccd7aac3e504c05ce46c243096b3f18662ae8dedf4bea\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1351, \"end_char_idx\": 4366, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb41233b-83da-45a6-8f44-ed52193202be", "doc_id": "cb41233b-83da-45a6-8f44-ed52193202be", "ref_doc_id": "cb41233b-83da-45a6-8f44-ed52193202be", "chroma:document": "Code Llama reaches state-of-the-art performance among open models on several code benchmarks, with scores of up to 53% and 55% on HumanEval and MBPP, respectively. Notably, Code Llama - Python 7B outperforms Llama 2 70B on HumanEval and MBPP, and all our models outperform every other publicly available model on MultiPL-E. We release Code Llama under a permissive license that allows for both research and commercial use.*\n\nCheck out all Code Llama model checkpoints [here](https://huggingface.co/models?search=code_llama) and the officially released ones in the [Meta Llama org](https://huggingface.co/meta-llama).\n\nThis model was contributed by [ArthurZucker](https://huggingface.co/ArthurZ). The original code of the authors can be found [here](https://github.com/facebookresearch/llama).\n\n## Usage tips and examples\n\n<Tip warning={true}>\n\nThe `Llama2` family models, on which Code Llama is based, were trained using `bfloat16`, but the original inference uses `float16`. Let's look at the different precisions:\n\n* `float32`: PyTorch convention on model initialization is to load models in `float32`, no matter with which `dtype` the model weights were stored. `transformers` also follows this convention for consistency with PyTorch. This will be picked by default. If you want the `AutoModel` API to cast the load the checkpoints with the storage weights type, you must specify `torch_dtype=\"auto\"`, e.g. `model = AutoModelForCausalLM.from_pretrained(\"path\", torch_dtype = \"auto\")`.\n* `bfloat16`: Code Llama was trained with this precision, so we recommend using it for further training or fine-tuning.\n* `float16`: We recommend running inference using this precision, as it's usually faster than `bfloat16`, and evaluation metrics show no discernible degradation with respect to `bfloat16`. You can also run inference using `bfloat16`, and we recommend you check inference results with both `float16` and `bfloat16` after fine-tuning.\n\nAs mentioned above, the `dtype` of the storage weights is mostly irrelevant unless you are using `torch_dtype=\"auto\"` when initializing a model using. The reason is that the model will first be downloaded (using the `dtype` of the checkpoints online) and then will be casted to the default `dtype` of `torch` (becomes `torch.float32`). If there is a specified `torch_dtype`, it will be used instead.\n\n</Tip>\n\n\nTips:\n- The infilling task is supported out of the box. You should be using the `tokenizer.fill_token` where you want your input to be filled.\n- The model conversion script is the same as for the `Llama2` family:\n\nHere is a sample usage:\n\n```bash\npython src/transformers/models/llama/convert_llama_weights_to_hf.py \\\n    --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path\n```\n\nNote that executing the script requires enough CPU RAM to host the whole model in float16 precision (even if the biggest versions\ncome in several checkpoints they each contain a part of each weight of the model, so we need to load them all in RAM)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 7e13f30e-981b-4b8a-a69f-ba9f87fecf90
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/code_llama", "title": "CodeLlama", "tokens": 2064, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7e13f30e-981b-4b8a-a69f-ba9f87fecf90\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/code_llama\", \"title\": \"CodeLlama\", \"tokens\": 2064, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb41233b-83da-45a6-8f44-ed52193202be\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/code_llama\", \"title\": \"CodeLlama\", \"tokens\": 2064, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"18affe0a10ce1571f11526a8082f64dd55465a1884ca565bcc3d19dd4e0db30d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4b0b77bd-506c-41c3-9bfd-1c876520e34a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/code_llama\", \"title\": \"CodeLlama\", \"tokens\": 2064, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7879a7a667e001ded38d968292d0944c7b5cefa03e5fcaa9d22087c3c75c1ca8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a0069c3d-91b0-495b-8aa7-09586f940bfe\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"70b84ecc4635d909ea9f23b1152a872de5e2f65d3f7be9a81ae610f9e06dabf4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2762, \"end_char_idx\": 5313, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb41233b-83da-45a6-8f44-ed52193202be", "doc_id": "cb41233b-83da-45a6-8f44-ed52193202be", "ref_doc_id": "cb41233b-83da-45a6-8f44-ed52193202be", "chroma:document": "`model = AutoModelForCausalLM.from_pretrained(\"path\", torch_dtype = \"auto\")`.\n* `bfloat16`: Code Llama was trained with this precision, so we recommend using it for further training or fine-tuning.\n* `float16`: We recommend running inference using this precision, as it's usually faster than `bfloat16`, and evaluation metrics show no discernible degradation with respect to `bfloat16`. You can also run inference using `bfloat16`, and we recommend you check inference results with both `float16` and `bfloat16` after fine-tuning.\n\nAs mentioned above, the `dtype` of the storage weights is mostly irrelevant unless you are using `torch_dtype=\"auto\"` when initializing a model using. The reason is that the model will first be downloaded (using the `dtype` of the checkpoints online) and then will be casted to the default `dtype` of `torch` (becomes `torch.float32`). If there is a specified `torch_dtype`, it will be used instead.\n\n</Tip>\n\n\nTips:\n- The infilling task is supported out of the box. You should be using the `tokenizer.fill_token` where you want your input to be filled.\n- The model conversion script is the same as for the `Llama2` family:\n\nHere is a sample usage:\n\n```bash\npython src/transformers/models/llama/convert_llama_weights_to_hf.py \\\n    --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path\n```\n\nNote that executing the script requires enough CPU RAM to host the whole model in float16 precision (even if the biggest versions\ncome in several checkpoints they each contain a part of each weight of the model, so we need to load them all in RAM).\n\nAfter conversion, the model and tokenizer can be loaded via:\n\n```python\n>>> from transformers import LlamaForCausalLM, CodeLlamaTokenizer\n\n>>> tokenizer = CodeLlamaTokenizer.from_pretrained(\"meta-llama/CodeLlama-7b-hf\")\n>>> model = LlamaForCausalLM.from_pretrained(\"meta-llama/CodeLlama-7b-hf\")\n>>> PROMPT = '''def remove_non_ascii(s: str) -> str:\n...     \"\"\" <FILL_ME>\n...     return result\n... '''\n>>> input_ids = tokenizer(PROMPT, return_tensors=\"pt\")[\"input_ids\"]\n>>> generated_ids = model.generate(input_ids, max_new_tokens=128)\n\n>>> filling = tokenizer.batch_decode(generated_ids[:, input_ids.shape[1]:], skip_special_tokens = True)[0]\n>>> print(PROMPT.replace(\"<FILL_ME>\", filling))\ndef remove_non_ascii(s: str) -> str:\n    \"\"\" Remove non-ASCII characters from a string.\n<BLANKLINE>\n    Args:\n        s: The string to remove non-ASCII characters from.\n<BLANKLINE>\n    Returns:\n        The string with non-ASCII characters removed.\n    \"\"\""}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a0069c3d-91b0-495b-8aa7-09586f940bfe
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/code_llama", "title": "CodeLlama", "tokens": 2064, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a0069c3d-91b0-495b-8aa7-09586f940bfe\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/code_llama\", \"title\": \"CodeLlama\", \"tokens\": 2064, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb41233b-83da-45a6-8f44-ed52193202be\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/code_llama\", \"title\": \"CodeLlama\", \"tokens\": 2064, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"18affe0a10ce1571f11526a8082f64dd55465a1884ca565bcc3d19dd4e0db30d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7e13f30e-981b-4b8a-a69f-ba9f87fecf90\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/code_llama\", \"title\": \"CodeLlama\", \"tokens\": 2064, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5f3b11954fe9015fa8bf82004d64912aa4c46366e5285e171a0a1260b3cdd146\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"26c554f5-a3c6-43c4-9cd7-2cbb71839f29\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"43b31fe80bd05c5d1117fee998a00415aa7ffac84947db0921e6be6a5d070dda\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3760, \"end_char_idx\": 6705, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb41233b-83da-45a6-8f44-ed52193202be", "doc_id": "cb41233b-83da-45a6-8f44-ed52193202be", "ref_doc_id": "cb41233b-83da-45a6-8f44-ed52193202be", "chroma:document": "You should be using the `tokenizer.fill_token` where you want your input to be filled.\n- The model conversion script is the same as for the `Llama2` family:\n\nHere is a sample usage:\n\n```bash\npython src/transformers/models/llama/convert_llama_weights_to_hf.py \\\n    --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path\n```\n\nNote that executing the script requires enough CPU RAM to host the whole model in float16 precision (even if the biggest versions\ncome in several checkpoints they each contain a part of each weight of the model, so we need to load them all in RAM).\n\nAfter conversion, the model and tokenizer can be loaded via:\n\n```python\n>>> from transformers import LlamaForCausalLM, CodeLlamaTokenizer\n\n>>> tokenizer = CodeLlamaTokenizer.from_pretrained(\"meta-llama/CodeLlama-7b-hf\")\n>>> model = LlamaForCausalLM.from_pretrained(\"meta-llama/CodeLlama-7b-hf\")\n>>> PROMPT = '''def remove_non_ascii(s: str) -> str:\n...     \"\"\" <FILL_ME>\n...     return result\n... '''\n>>> input_ids = tokenizer(PROMPT, return_tensors=\"pt\")[\"input_ids\"]\n>>> generated_ids = model.generate(input_ids, max_new_tokens=128)\n\n>>> filling = tokenizer.batch_decode(generated_ids[:, input_ids.shape[1]:], skip_special_tokens = True)[0]\n>>> print(PROMPT.replace(\"<FILL_ME>\", filling))\ndef remove_non_ascii(s: str) -> str:\n    \"\"\" Remove non-ASCII characters from a string.\n<BLANKLINE>\n    Args:\n        s: The string to remove non-ASCII characters from.\n<BLANKLINE>\n    Returns:\n        The string with non-ASCII characters removed.\n    \"\"\"\n    result = \"\"\n    for c in s:\n        if ord(c) < 128:\n            result += c\n    return result\n<BLANKLINE>\n```\n\nIf you only want the infilled part:\n```python\n>>> from transformers import pipeline\n>>> import torch\n\n>>> generator = pipeline(\"text-generation\",model=\"meta-llama/CodeLlama-7b-hf\",torch_dtype=torch.float16, device_map=\"auto\")\n>>> generator('def remove_non_ascii(s: str) -> str:\\n    \"\"\" <FILL_ME>\\n    return result', max_new_tokens = 128)\n[{'generated_text': 'def remove_non_ascii(s: str) -> str:\\n    \"\"\" <FILL_ME>\\n    return resultRemove non-ASCII characters from a string. \"\"\"\\n    result = \"\"\\n    for c in s:\\n        if ord(c) < 128:\\n            result += c'}]\n```\n\nUnder the hood, the tokenizer [automatically splits by `<FILL_ME>`](https://huggingface.co/docs/transformers/main/model_doc/code_llama#transformers.CodeLlamaTokenizer.fill_token) to create a formatted input string that follows [the original training pattern](https://github.com/facebookresearch/codellama/blob/cb51c14ec761370ba2e2bc351374a79265d0465e/llama/generation.py#L402). This is more robust than preparing the pattern yourself: it avoids pitfalls, such as token glueing, that are very hard to debug.  To see how much CPU and GPU memory you need for this model or others, try [this calculator](https://huggingface.co/spaces/hf-accelerate/model-memory-usage) which can help determine that value."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 26c554f5-a3c6-43c4-9cd7-2cbb71839f29
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/code_llama", "title": "CodeLlama", "tokens": 2064, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"26c554f5-a3c6-43c4-9cd7-2cbb71839f29\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/code_llama\", \"title\": \"CodeLlama\", \"tokens\": 2064, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb41233b-83da-45a6-8f44-ed52193202be\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/code_llama\", \"title\": \"CodeLlama\", \"tokens\": 2064, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"18affe0a10ce1571f11526a8082f64dd55465a1884ca565bcc3d19dd4e0db30d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a0069c3d-91b0-495b-8aa7-09586f940bfe\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/code_llama\", \"title\": \"CodeLlama\", \"tokens\": 2064, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c89154f1cc60f4a2223be658921c0ff3abb012b07e802b4f3b4eddf14253c661\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5227, \"end_char_idx\": 7598, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb41233b-83da-45a6-8f44-ed52193202be", "doc_id": "cb41233b-83da-45a6-8f44-ed52193202be", "ref_doc_id": "cb41233b-83da-45a6-8f44-ed52193202be", "chroma:document": "<BLANKLINE>\n    Returns:\n        The string with non-ASCII characters removed.\n    \"\"\"\n    result = \"\"\n    for c in s:\n        if ord(c) < 128:\n            result += c\n    return result\n<BLANKLINE>\n```\n\nIf you only want the infilled part:\n```python\n>>> from transformers import pipeline\n>>> import torch\n\n>>> generator = pipeline(\"text-generation\",model=\"meta-llama/CodeLlama-7b-hf\",torch_dtype=torch.float16, device_map=\"auto\")\n>>> generator('def remove_non_ascii(s: str) -> str:\\n    \"\"\" <FILL_ME>\\n    return result', max_new_tokens = 128)\n[{'generated_text': 'def remove_non_ascii(s: str) -> str:\\n    \"\"\" <FILL_ME>\\n    return resultRemove non-ASCII characters from a string. \"\"\"\\n    result = \"\"\\n    for c in s:\\n        if ord(c) < 128:\\n            result += c'}]\n```\n\nUnder the hood, the tokenizer [automatically splits by `<FILL_ME>`](https://huggingface.co/docs/transformers/main/model_doc/code_llama#transformers.CodeLlamaTokenizer.fill_token) to create a formatted input string that follows [the original training pattern](https://github.com/facebookresearch/codellama/blob/cb51c14ec761370ba2e2bc351374a79265d0465e/llama/generation.py#L402). This is more robust than preparing the pattern yourself: it avoids pitfalls, such as token glueing, that are very hard to debug.  To see how much CPU and GPU memory you need for this model or others, try [this calculator](https://huggingface.co/spaces/hf-accelerate/model-memory-usage) which can help determine that value.\n\nThe LLaMA tokenizer is a BPE model based on [sentencepiece](https://github.com/google/sentencepiece). One quirk of sentencepiece is that when decoding a sequence, if the first token is the start of the word (e.g. \"Banana\"), the tokenizer does not prepend the prefix space to the string.\n\n<Tip>\n\nCode Llama has the same architecture as the `Llama2` models, refer to [Llama2's documentation page](llama2) for the API reference.\nFind Code Llama tokenizer reference below. \n</Tip>\n\n\n## CodeLlamaTokenizer\n\n[[autodoc]] CodeLlamaTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## CodeLlamaTokenizerFast\n\n[[autodoc]] CodeLlamaTokenizerFast\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - update_post_processor\n    - save_vocabulary"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 91ef34b8-de71-4e63-8ca1-3efb20e120c1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llava_next", "title": "LLaVA-NeXT", "tokens": 3301, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"91ef34b8-de71-4e63-8ca1-3efb20e120c1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next\", \"title\": \"LLaVA-NeXT\", \"tokens\": 3301, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9990cdb1-3b28-4fad-aa10-e0ddc6620d23\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next\", \"title\": \"LLaVA-NeXT\", \"tokens\": 3301, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"08f02de732c0fc26ec8f972528e3fc9ac13140f50fbf752aaad6c0c5bc4c2150\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3d0e32f4-8d79-4b00-a03e-be6deb08eb05\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"cc5ae288ea2785bdc6fb7d714bab44b9ea38ed6094d3600f61067eb252957c31\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3173, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9990cdb1-3b28-4fad-aa10-e0ddc6620d23", "doc_id": "9990cdb1-3b28-4fad-aa10-e0ddc6620d23", "ref_doc_id": "9990cdb1-3b28-4fad-aa10-e0ddc6620d23", "chroma:document": "# LLaVA-NeXT\n\n## Overview\n\nThe LLaVA-NeXT model was proposed in [LLaVA-NeXT: Improved reasoning, OCR, and world knowledge](https://llava-vl.github.io/blog/2024-01-30-llava-next/) by Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, Yong Jae Lee. LLaVa-NeXT (also called LLaVa-1.6) improves upon [LLaVa](llava) by increasing the input image resolution and training on an improved visual instruction tuning dataset to improve OCR and common sense reasoning.\n\nThe introduction from the blog is the following:\n\n*In October 2023, we released LLaVA-1.5 with a simple and efficient design along with great performance on a benchmark suite of 12 datasets. It has since served as the foundation of many comprehensive studies of data, model, and capabilities of large multimodal models (LMM), and has enabled various new applications.\n\nToday, we are thrilled to present LLaVA-NeXT, with improved reasoning, OCR, and world knowledge. LLaVA-NeXT even exceeds Gemini Pro on several benchmarks.\n\nCompared with LLaVA-1.5, LLaVA-NeXT has several improvements:\n\nIncreasing the input image resolution to 4x more pixels. This allows it to grasp more visual details. It supports three aspect ratios, up to 672x672, 336x1344, 1344x336 resolution.\nBetter visual reasoning and OCR capability with an improved visual instruction tuning data mixture.\nBetter visual conversation for more scenarios, covering different applications. Better world knowledge and logical reasoning.\nEfficient deployment and inference with SGLang.\nAlong with performance improvements, LLaVA-NeXT maintains the minimalist design and data efficiency of LLaVA-1.5. It re-uses the pretrained connector of LLaVA-1.5, and still uses less than 1M visual instruction tuning samples. The largest 34B variant finishes training in ~1 day with 32 A100s.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/llava_next_overview.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> LLaVa-NeXT incorporates a higher input resolution by encoding various patches of the input image. Taken from the <a href=\"https://arxiv.org/abs/2310.03744\">original paper.</a> </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/haotian-liu/LLaVA/tree/main).\n\n## Usage tips\n\n- We advise users to use `padding_side=\"left\"` when computing batched generation as it leads to more accurate results. Simply make sure to call `processor.tokenizer.padding_side = \"left\"` before generating.\n\n<Tip warning={true}>\n\n- Llava-Next uses different number of patches for images and thus has to pad the inputs inside modeling code, aside from the padding done when processing the inputs. The default setting is \"left-padding\" if model is in `eval()` mode, otherwise \"right-padding\".\n\n</Tip>\n\n\n- Note that each checkpoint has been trained with a specific prompt format, depending on which large language model (LLM) was used. You can use the processor's `apply_chat_template` to format your prompts correctly. For that you have to construct a conversation history, passing a plain string will not format your prompt."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 3d0e32f4-8d79-4b00-a03e-be6deb08eb05
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llava_next", "title": "LLaVA-NeXT", "tokens": 3301, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3d0e32f4-8d79-4b00-a03e-be6deb08eb05\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next\", \"title\": \"LLaVA-NeXT\", \"tokens\": 3301, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9990cdb1-3b28-4fad-aa10-e0ddc6620d23\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next\", \"title\": \"LLaVA-NeXT\", \"tokens\": 3301, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"08f02de732c0fc26ec8f972528e3fc9ac13140f50fbf752aaad6c0c5bc4c2150\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"91ef34b8-de71-4e63-8ca1-3efb20e120c1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next\", \"title\": \"LLaVA-NeXT\", \"tokens\": 3301, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c282bd1e257146a4279ba27ebb40eaf5ad2cdcb92aef98525c3b1c6e3a6f103b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"89bb0620-e28b-4c22-92e6-abff7fc57902\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6bbc7058afad91efee48d0ea76ea00afef82cac144a896939aee01c1be48e142\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2851, \"end_char_idx\": 5322, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9990cdb1-3b28-4fad-aa10-e0ddc6620d23", "doc_id": "9990cdb1-3b28-4fad-aa10-e0ddc6620d23", "ref_doc_id": "9990cdb1-3b28-4fad-aa10-e0ddc6620d23", "chroma:document": "- Note that each checkpoint has been trained with a specific prompt format, depending on which large language model (LLM) was used. You can use the processor's `apply_chat_template` to format your prompts correctly. For that you have to construct a conversation history, passing a plain string will not format your prompt. Each message in the conversation history for chat templates is a dictionary with keys \"role\" and \"content\". The \"content\" should be a list of dictionaries, for \"text\" and \"image\" modalities. Below is an example of how to do that and the list of formats accepted by each checkpoint.\n\nWe will use [llava-v1.6-mistral-7b-hf](https://huggingface.co/llava-hf/llava-v1.6-mistral-7b-hf) and a conversation history of text and image. Each content field has to be a list of dicts, as follows:\n\n```python\nfrom transformers import LlavaNextProcessor\n\nprocessor = LlavaNextProcessor.from_pretrained(\"llava-hf/llava-v1.6-mistral-7b-hf\")\n\nconversation = [\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image\"},\n            {\"type\": \"text\", \"text\": \"What\u2019s shown in this image?\"},\n        ],\n    },\n    {\n        \"role\": \"assistant\",\n        \"content\": [{\"type\": \"text\", \"text\": \"This image shows a red stop sign.\"},]\n    },\n    {\n\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"Describe the image in more details.\"},\n        ],\n    },\n]\n\ntext_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)\n\n# Note that the template simply formats your prompt, you still have to tokenize it and obtain pixel values for your images\nprint(text_prompt)\n>>> \"[INST] <image>\\nWhat's shown in this image? [/INST] This image shows a red stop sign. [INST] Describe the image in more details. [/INST]\"\n```\n\n- If you want to construct a chat prompt yourself, below is a list of possible formats\n.\n[llava-v1.6-mistral-7b-hf](https://huggingface.co/llava-hf/llava-v1.6-mistral-7b-hf) requires the following format:\n```bash\n\"[INST] <image>\\nWhat is shown in this image? [/INST]\"\n```\n\n[llava-v1.6-vicuna-7b-hf](https://huggingface.co/llava-hf/llava-v1.6-vicuna-7b-hf) and [llava-v1.6-vicuna-13b-hf](https://huggingface.co/llava-hf/llava-v1.6-vicuna-13b-hf) require the following format:\n```bash\n\"A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions. USER: <image>\\nWhat is shown in this image?"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 89bb0620-e28b-4c22-92e6-abff7fc57902
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llava_next", "title": "LLaVA-NeXT", "tokens": 3301, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"89bb0620-e28b-4c22-92e6-abff7fc57902\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next\", \"title\": \"LLaVA-NeXT\", \"tokens\": 3301, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9990cdb1-3b28-4fad-aa10-e0ddc6620d23\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next\", \"title\": \"LLaVA-NeXT\", \"tokens\": 3301, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"08f02de732c0fc26ec8f972528e3fc9ac13140f50fbf752aaad6c0c5bc4c2150\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3d0e32f4-8d79-4b00-a03e-be6deb08eb05\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next\", \"title\": \"LLaVA-NeXT\", \"tokens\": 3301, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"254ea0b82db6462380d7a8b45223e081eca9fb2941c0feacbf2969c20c889db4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0625fcdf-fa51-4d39-8048-d99850beacbf\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2190ce1a5a0580b4142e1f1700041ba9209f48bc717868207d13d73e46269dd7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3963, \"end_char_idx\": 5808, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9990cdb1-3b28-4fad-aa10-e0ddc6620d23", "doc_id": "9990cdb1-3b28-4fad-aa10-e0ddc6620d23", "ref_doc_id": "9990cdb1-3b28-4fad-aa10-e0ddc6620d23", "chroma:document": "\"},\n        ],\n    },\n    {\n        \"role\": \"assistant\",\n        \"content\": [{\"type\": \"text\", \"text\": \"This image shows a red stop sign.\"},]\n    },\n    {\n\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"Describe the image in more details.\"},\n        ],\n    },\n]\n\ntext_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)\n\n# Note that the template simply formats your prompt, you still have to tokenize it and obtain pixel values for your images\nprint(text_prompt)\n>>> \"[INST] <image>\\nWhat's shown in this image? [/INST] This image shows a red stop sign. [INST] Describe the image in more details. [/INST]\"\n```\n\n- If you want to construct a chat prompt yourself, below is a list of possible formats\n.\n[llava-v1.6-mistral-7b-hf](https://huggingface.co/llava-hf/llava-v1.6-mistral-7b-hf) requires the following format:\n```bash\n\"[INST] <image>\\nWhat is shown in this image? [/INST]\"\n```\n\n[llava-v1.6-vicuna-7b-hf](https://huggingface.co/llava-hf/llava-v1.6-vicuna-7b-hf) and [llava-v1.6-vicuna-13b-hf](https://huggingface.co/llava-hf/llava-v1.6-vicuna-13b-hf) require the following format:\n```bash\n\"A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions. USER: <image>\\nWhat is shown in this image? ASSISTANT:\"\n```\n\n[llava-v1.6-34b-hf](https://huggingface.co/llava-hf/llava-v1.6-34b-hf) requires the following format:\n```bash\n\"<|im_start|>system\\nAnswer the questions.<|im_end|><|im_start|>user\\n<image>\\nWhat is shown in this image?<|im_end|><|im_start|>assistant\\n\"\n```\n\n[llama3-llava-next-8b-hf](https://huggingface.co/llava-hf/llava-next-8b-hf) requires the following format:\n\n```bash\n\"<|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful language and vision assistant."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0625fcdf-fa51-4d39-8048-d99850beacbf
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llava_next", "title": "LLaVA-NeXT", "tokens": 3301, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0625fcdf-fa51-4d39-8048-d99850beacbf\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next\", \"title\": \"LLaVA-NeXT\", \"tokens\": 3301, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9990cdb1-3b28-4fad-aa10-e0ddc6620d23\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next\", \"title\": \"LLaVA-NeXT\", \"tokens\": 3301, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"08f02de732c0fc26ec8f972528e3fc9ac13140f50fbf752aaad6c0c5bc4c2150\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"89bb0620-e28b-4c22-92e6-abff7fc57902\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next\", \"title\": \"LLaVA-NeXT\", \"tokens\": 3301, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6696e29c53ed9a5fab95b30e368d05bcfe827191ab05a0e43a1ac0c234b8005f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9eb4208a-0f98-423f-98f4-5295448ed47e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"224153815aaa13977fed0a4b0237e8192d02fe2b7a912db04b49382f8c8cfbfe\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4727, \"end_char_idx\": 7415, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9990cdb1-3b28-4fad-aa10-e0ddc6620d23", "doc_id": "9990cdb1-3b28-4fad-aa10-e0ddc6620d23", "ref_doc_id": "9990cdb1-3b28-4fad-aa10-e0ddc6620d23", "chroma:document": "[llava-v1.6-mistral-7b-hf](https://huggingface.co/llava-hf/llava-v1.6-mistral-7b-hf) requires the following format:\n```bash\n\"[INST] <image>\\nWhat is shown in this image? [/INST]\"\n```\n\n[llava-v1.6-vicuna-7b-hf](https://huggingface.co/llava-hf/llava-v1.6-vicuna-7b-hf) and [llava-v1.6-vicuna-13b-hf](https://huggingface.co/llava-hf/llava-v1.6-vicuna-13b-hf) require the following format:\n```bash\n\"A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions. USER: <image>\\nWhat is shown in this image? ASSISTANT:\"\n```\n\n[llava-v1.6-34b-hf](https://huggingface.co/llava-hf/llava-v1.6-34b-hf) requires the following format:\n```bash\n\"<|im_start|>system\\nAnswer the questions.<|im_end|><|im_start|>user\\n<image>\\nWhat is shown in this image?<|im_end|><|im_start|>assistant\\n\"\n```\n\n[llama3-llava-next-8b-hf](https://huggingface.co/llava-hf/llava-next-8b-hf) requires the following format:\n\n```bash\n\"<|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful language and vision assistant. You are able to understand the visual content that the user provides, and assist the user with a variety of tasks using natural language.<|eot_id|><|start_header_id|><|start_header_id|>user<|end_header_id|>\\n\\n<image>\\nWhat is shown in this image?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n\"\n```\n\n[llava-next-72b-hf](https://huggingface.co/llava-hf/llava-next-72b-hf) and [llava-next-110b-hf](https://huggingface.co/llava-hf/llava-next-110b-hf) require the following format:\n\n```bash\n\"<|im_start|>system\\nYou are a helpful assistant.<|im_end|>\\n<|im_start|>user\\n<image>\\nWhat is shown in this image?<|im_end|>\\n<|im_start|>assistant\\n\"\n```\n\n## Usage example\n\n### Single image inference\n\nHere's how to load the model and perform inference in half-precision (`torch.float16`):\n\n```python\nfrom transformers import LlavaNextProcessor, LlavaNextForConditionalGeneration\nimport torch\nfrom PIL import Image\nimport requests\n\nprocessor = LlavaNextProcessor.from_pretrained(\"llava-hf/llava-v1.6-mistral-7b-hf\")\n\nmodel = LlavaNextForConditionalGeneration.from_pretrained(\"llava-hf/llava-v1.6-mistral-7b-hf\", torch_dtype=torch.float16, low_cpu_mem_usage=True) \nmodel.to(\"cuda:0\")\n\n# prepare image and text prompt, using the appropriate prompt template\nurl = \"https://github.com/haotian-liu/LLaVA/blob/1a91fc274d7c35a9b50b3cb29c4247ae5837ce39/images/llava_v1_5_radar.jpg?raw=true\"\nimage = Image.open(requests.get(url, stream=True).raw)\n\nconversation = [\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image\"},\n            {\"type\": \"text\", \"text\": \"What is shown in this image?"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 9eb4208a-0f98-423f-98f4-5295448ed47e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llava_next", "title": "LLaVA-NeXT", "tokens": 3301, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9eb4208a-0f98-423f-98f4-5295448ed47e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next\", \"title\": \"LLaVA-NeXT\", \"tokens\": 3301, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9990cdb1-3b28-4fad-aa10-e0ddc6620d23\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next\", \"title\": \"LLaVA-NeXT\", \"tokens\": 3301, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"08f02de732c0fc26ec8f972528e3fc9ac13140f50fbf752aaad6c0c5bc4c2150\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0625fcdf-fa51-4d39-8048-d99850beacbf\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next\", \"title\": \"LLaVA-NeXT\", \"tokens\": 3301, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f26031ae50fe8b3764819ba9b0deb5c034e4bfe6c81e7ccbbe41fb55c2ba2769\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d2869a39-935c-4b1b-8f1b-5273ae25200c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"be4ca95a0b44117d08a0aa9842f2c7e3c6ac292766840432bc7182507c2c2309\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7415, \"end_char_idx\": 10499, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9990cdb1-3b28-4fad-aa10-e0ddc6620d23", "doc_id": "9990cdb1-3b28-4fad-aa10-e0ddc6620d23", "ref_doc_id": "9990cdb1-3b28-4fad-aa10-e0ddc6620d23", "chroma:document": "\"},\n        ],\n    },\n]\nprompt = processor.apply_chat_template(conversation, add_generation_prompt=True)\ninputs = processor(prompt, image, return_tensors=\"pt\").to(\"cuda:0\")\n\n# autoregressively complete prompt\noutput = model.generate(**inputs, max_new_tokens=100)\n\nprint(processor.decode(output[0], skip_special_tokens=True))\n```\n\n### Multi image inference\n\nLLaVa-Next can perform inference with multiple images as input, where images either belong to the same prompt or different prompts (in batched inference). Here is how you can do it:\n\n```python\nimport requests\nfrom PIL import Image\nimport torch\nfrom transformers import AutoProcessor, LlavaNextForConditionalGeneration\n\n# Load the model in half-precision\nmodel = LlavaNextForConditionalGeneration.from_pretrained(\"llava-hf/llava-v1.6-mistral-7b-hf\", torch_dtype=torch.float16, device_map=\"auto\")\nprocessor = AutoProcessor.from_pretrained(\"llava-hf/llava-v1.6-mistral-7b-hf\")\n\n# Get three different images\nurl = \"https://www.ilankelman.org/stopsigns/australia.jpg\"\nimage_stop = Image.open(requests.get(url, stream=True).raw)\n\nurl = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nimage_cats = Image.open(requests.get(url, stream=True).raw)\n\nurl = \"https://huggingface.co/microsoft/kosmos-2-patch14-224/resolve/main/snowman.jpg\"\nimage_snowman = Image.open(requests.get(url, stream=True).raw)\n\n# Prepare a batch of two prompts, where the first one is a multi-turn conversation and the second is not\nconversation_1 = [\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image\"},\n            {\"type\": \"text\", \"text\": \"What is shown in this image?\"},\n            ],\n    },\n    {\n        \"role\": \"assistant\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"There is a red stop sign in the image.\"},\n            ],\n    },\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image\"},\n            {\"type\": \"text\", \"text\": \"What about this image? How many cats do you see?\"},\n            ],\n    },\n]\n\nconversation_2 = [\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image\"},\n            {\"type\": \"text\", \"text\": \"What is shown in this image?\"},\n            ],\n    },\n]\n\nprompt_1 = processor.apply_chat_template(conversation_1, add_generation_prompt=True)\nprompt_2 = processor.apply_chat_template(conversation_2, add_generation_prompt=True)\nprompts = [prompt_1, prompt_2]\n\n# We can simply feed images in the order they have to be used in the text prompt\n# Each \"<image>\" token uses one image leaving the next for the subsequent \"<image>\" tokens\ninputs = processor(text=prompts, images=[image_stop, image_cats, image_snowman], padding=True, return_tensors=\"pt\").to(model.device)\n\n# Generate\ngenerate_ids = model.generate(**inputs, max_new_tokens=30)\nprocessor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)\n```\n\n## Model optimization\n\n### Quantization using Bitsandbytes\n\nThe model can be loaded in 8 or 4 bits, greatly reducing the memory requirements while maintaining the performance of the original model."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: d2869a39-935c-4b1b-8f1b-5273ae25200c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llava_next", "title": "LLaVA-NeXT", "tokens": 3301, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d2869a39-935c-4b1b-8f1b-5273ae25200c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next\", \"title\": \"LLaVA-NeXT\", \"tokens\": 3301, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9990cdb1-3b28-4fad-aa10-e0ddc6620d23\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next\", \"title\": \"LLaVA-NeXT\", \"tokens\": 3301, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"08f02de732c0fc26ec8f972528e3fc9ac13140f50fbf752aaad6c0c5bc4c2150\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9eb4208a-0f98-423f-98f4-5295448ed47e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next\", \"title\": \"LLaVA-NeXT\", \"tokens\": 3301, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c4f43b74cb54062f9412bae37a6ef34259a7cb1912cf12e82745f8a203de9447\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9043, \"end_char_idx\": 11900, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9990cdb1-3b28-4fad-aa10-e0ddc6620d23", "doc_id": "9990cdb1-3b28-4fad-aa10-e0ddc6620d23", "ref_doc_id": "9990cdb1-3b28-4fad-aa10-e0ddc6620d23", "chroma:document": "\"},\n            ],\n    },\n    {\n        \"role\": \"assistant\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"There is a red stop sign in the image.\"},\n            ],\n    },\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image\"},\n            {\"type\": \"text\", \"text\": \"What about this image? How many cats do you see?\"},\n            ],\n    },\n]\n\nconversation_2 = [\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image\"},\n            {\"type\": \"text\", \"text\": \"What is shown in this image?\"},\n            ],\n    },\n]\n\nprompt_1 = processor.apply_chat_template(conversation_1, add_generation_prompt=True)\nprompt_2 = processor.apply_chat_template(conversation_2, add_generation_prompt=True)\nprompts = [prompt_1, prompt_2]\n\n# We can simply feed images in the order they have to be used in the text prompt\n# Each \"<image>\" token uses one image leaving the next for the subsequent \"<image>\" tokens\ninputs = processor(text=prompts, images=[image_stop, image_cats, image_snowman], padding=True, return_tensors=\"pt\").to(model.device)\n\n# Generate\ngenerate_ids = model.generate(**inputs, max_new_tokens=30)\nprocessor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)\n```\n\n## Model optimization\n\n### Quantization using Bitsandbytes\n\nThe model can be loaded in 8 or 4 bits, greatly reducing the memory requirements while maintaining the performance of the original model. First make sure to install bitsandbytes, `pip install bitsandbytes` and make sure to have access to a CUDA compatible GPU device. Simply change the snippet above with:\n\n```python\nfrom transformers import LlavaNextForConditionalGeneration, BitsAndBytesConfig\n\n# specify how to quantize the model\nquantization_config = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_quant_type=\"nf4\",\n    bnb_4bit_compute_dtype=torch.float16,\n)\n\nmodel = LlavaNextForConditionalGeneration.from_pretrained(\"llava-hf/llava-v1.6-mistral-7b-hf\", quantization_config=quantization_config, device_map=\"auto\")\n```\n\n### Use Flash-Attention 2 to further speed-up generation\n\nFirst make sure to install flash-attn. Refer to the [original repository of Flash Attention](https://github.com/Dao-AILab/flash-attention) regarding that package installation. Simply change the snippet above with:\n\n```python\nfrom transformers import LlavaNextForConditionalGeneration\n\nmodel = LlavaNextForConditionalGeneration.from_pretrained(\n    model_id, \n    torch_dtype=torch.float16, \n    low_cpu_mem_usage=True,\n    use_flash_attention_2=True\n).to(0)\n```\n\n## LlavaNextConfig\n\n[[autodoc]] LlavaNextConfig\n\n## LlavaNextImageProcessor\n\n[[autodoc]] LlavaNextImageProcessor\n    - preprocess\n\n## LlavaNextProcessor\n\n[[autodoc]] LlavaNextProcessor\n\n## LlavaNextForConditionalGeneration\n\n[[autodoc]] LlavaNextForConditionalGeneration\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a66ca2d4-5805-445e-9bb1-be5fe3d208b3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xglm", "title": "XGLM", "tokens": 999, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a66ca2d4-5805-445e-9bb1-be5fe3d208b3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xglm\", \"title\": \"XGLM\", \"tokens\": 999, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f13959af-548a-463d-bc41-1e99ebf7d10a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xglm\", \"title\": \"XGLM\", \"tokens\": 999, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1fafa44d10a88b34cb80261b380c21dac121df2bc2cb3a95fcb70e0b5f7bc99a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dad91187-8fa2-47ec-80b7-2d0f775ebdc7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"cb48e1f95e492b18be7b03a8e5287b16e9de60040679cebea38817c967aa47a1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2118, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f13959af-548a-463d-bc41-1e99ebf7d10a", "doc_id": "f13959af-548a-463d-bc41-1e99ebf7d10a", "ref_doc_id": "f13959af-548a-463d-bc41-1e99ebf7d10a", "chroma:document": "# XGLM\n\n## Overview\n\nThe XGLM model was proposed in [Few-shot Learning with Multilingual Language Models](https://arxiv.org/abs/2112.10668)\nby Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, \nShruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O'Horo, \nJeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, Xian Li.\n\nThe abstract from the paper is the following:\n\n*Large-scale autoregressive language models such as GPT-3 are few-shot learners that can perform a wide range of language \ntasks without fine-tuning. While these models are known to be able to jointly represent many different languages, \ntheir training data is dominated by English, potentially limiting their cross-lingual generalization. \nIn this work, we train multilingual autoregressive language models on a balanced corpus covering a diverse set of languages, \nand study their few- and zero-shot learning capabilities in a wide range of tasks. Our largest model with 7.5 billion parameters \nsets new state of the art in few-shot learning in more than 20 representative languages, outperforming GPT-3 of comparable size \nin multilingual commonsense reasoning (with +7.4% absolute accuracy improvement in 0-shot settings and +9.4% in 4-shot settings) \nand natural language inference (+5.4% in each of 0-shot and 4-shot settings). On the FLORES-101 machine translation benchmark, \nour model outperforms GPT-3 on 171 out of 182 translation directions with 32 training examples, while surpassing the \nofficial supervised baseline in 45 directions. We present a detailed analysis of where the model succeeds and fails, \nshowing in particular that it enables cross-lingual in-context learning on some tasks, while there is still room for improvement \non surface form robustness and adaptation to tasks that do not have a natural cloze form. Finally, we evaluate our models \nin social value tasks such as hate speech detection in five languages and find it has limitations similar to comparable sized GPT-3 models.*"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: dad91187-8fa2-47ec-80b7-2d0f775ebdc7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xglm", "title": "XGLM", "tokens": 999, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"dad91187-8fa2-47ec-80b7-2d0f775ebdc7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xglm\", \"title\": \"XGLM\", \"tokens\": 999, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f13959af-548a-463d-bc41-1e99ebf7d10a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xglm\", \"title\": \"XGLM\", \"tokens\": 999, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1fafa44d10a88b34cb80261b380c21dac121df2bc2cb3a95fcb70e0b5f7bc99a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a66ca2d4-5805-445e-9bb1-be5fe3d208b3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xglm\", \"title\": \"XGLM\", \"tokens\": 999, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"51739c7628dc650501961767272cb7ab9044f6c65e6195b14ec6b73ed81f06db\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2121, \"end_char_idx\": 3085, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f13959af-548a-463d-bc41-1e99ebf7d10a", "doc_id": "f13959af-548a-463d-bc41-1e99ebf7d10a", "ref_doc_id": "f13959af-548a-463d-bc41-1e99ebf7d10a", "chroma:document": "This model was contributed by [Suraj](https://huggingface.co/valhalla). The original code can be found [here](https://github.com/pytorch/fairseq/tree/main/examples/xglm).\n\n## Resources\n\n- [Causal language modeling task guide](../tasks/language_modeling)\n\n## XGLMConfig\n\n[[autodoc]] XGLMConfig\n\n## XGLMTokenizer\n\n[[autodoc]] XGLMTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## XGLMTokenizerFast\n\n[[autodoc]] XGLMTokenizerFast\n\n<frameworkcontent>\n<pt>\n\n## XGLMModel\n\n[[autodoc]] XGLMModel\n    - forward\n\n## XGLMForCausalLM\n\n[[autodoc]] XGLMForCausalLM\n    - forward\n\n</pt>\n<tf>\n\n## TFXGLMModel\n\n[[autodoc]] TFXGLMModel\n    - call\n\n## TFXGLMForCausalLM\n\n[[autodoc]] TFXGLMForCausalLM\n    - call\n\n</tf>\n<jax>\n\n## FlaxXGLMModel\n\n[[autodoc]] FlaxXGLMModel\n    - __call__\n\n## FlaxXGLMForCausalLM\n\n[[autodoc]] FlaxXGLMForCausalLM\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 3215fb88-c1fa-4b61-8670-42817324616f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3215fb88-c1fa-4b61-8670-42817324616f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c3999cb4-27da-4160-9f57-8e2fbb75c610\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f119c223-3dfd-4f5c-b63c-c2a5eef0ad99\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"24ba341ce000d143f08391e307bb8f97cdc356cf1567f293166fd7b31df73d7c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2885, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "doc_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "ref_doc_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "chroma:document": "# LayoutLMV2\n\n## Overview\n\nThe LayoutLMV2 model was proposed in [LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding](https://arxiv.org/abs/2012.14740) by Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu,\nDinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, Lidong Zhou. LayoutLMV2 improves [LayoutLM](layoutlm) to obtain\nstate-of-the-art results across several document image understanding benchmarks:\n\n- information extraction from scanned documents: the [FUNSD](https://guillaumejaume.github.io/FUNSD/) dataset (a\n  collection of 199 annotated forms comprising more than 30,000 words), the [CORD](https://github.com/clovaai/cord)\n  dataset (a collection of 800 receipts for training, 100 for validation and 100 for testing), the [SROIE](https://rrc.cvc.uab.es/?ch=13) dataset (a collection of 626 receipts for training and 347 receipts for testing)\n  and the [Kleister-NDA](https://github.com/applicaai/kleister-nda) dataset (a collection of non-disclosure\n  agreements from the EDGAR database, including 254 documents for training, 83 documents for validation, and 203\n  documents for testing).\n- document image classification: the [RVL-CDIP](https://www.cs.cmu.edu/~aharley/rvl-cdip/) dataset (a collection of\n  400,000 images belonging to one of 16 classes).\n- document visual question answering: the [DocVQA](https://arxiv.org/abs/2007.00398) dataset (a collection of 50,000\n  questions defined on 12,000+ document images).\n\nThe abstract from the paper is the following:\n\n*Pre-training of text and layout has proved effective in a variety of visually-rich document understanding tasks due to\nits effective model architecture and the advantage of large-scale unlabeled scanned/digital-born documents. In this\npaper, we present LayoutLMv2 by pre-training text, layout and image in a multi-modal framework, where new model\narchitectures and pre-training tasks are leveraged. Specifically, LayoutLMv2 not only uses the existing masked\nvisual-language modeling task but also the new text-image alignment and text-image matching tasks in the pre-training\nstage, where cross-modality interaction is better learned. Meanwhile, it also integrates a spatial-aware self-attention\nmechanism into the Transformer architecture, so that the model can fully understand the relative positional\nrelationship among different text blocks. Experiment results show that LayoutLMv2 outperforms strong baselines and\nachieves new state-of-the-art results on a wide variety of downstream visually-rich document understanding tasks,\nincluding FUNSD (0.7895 -> 0.8420), CORD (0.9493 -> 0.9601), SROIE (0.9524 -> 0.9781), Kleister-NDA (0.834 -> 0.852),\nRVL-CDIP (0.9443 -> 0.9564), and DocVQA (0.7295 -> 0.8672). The pre-trained LayoutLMv2 model is publicly available at\nthis https URL.*\n\nLayoutLMv2 depends on `detectron2`, `torchvision` and `tesseract`."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f119c223-3dfd-4f5c-b63c-c2a5eef0ad99
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f119c223-3dfd-4f5c-b63c-c2a5eef0ad99\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c3999cb4-27da-4160-9f57-8e2fbb75c610\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3215fb88-c1fa-4b61-8670-42817324616f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"986397e737f6cc88422c2e044e9593faa1bb6d88f378ffb6765e9fa4dcb14f12\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"55c1bee6-bce8-40c5-8dd5-b1f75ce60363\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"451899999b7387093a3b39221144539623e1f6e72ead6d29bf01c3b0a6686131\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1317, \"end_char_idx\": 4294, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "doc_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "ref_doc_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "chroma:document": "- document visual question answering: the [DocVQA](https://arxiv.org/abs/2007.00398) dataset (a collection of 50,000\n  questions defined on 12,000+ document images).\n\nThe abstract from the paper is the following:\n\n*Pre-training of text and layout has proved effective in a variety of visually-rich document understanding tasks due to\nits effective model architecture and the advantage of large-scale unlabeled scanned/digital-born documents. In this\npaper, we present LayoutLMv2 by pre-training text, layout and image in a multi-modal framework, where new model\narchitectures and pre-training tasks are leveraged. Specifically, LayoutLMv2 not only uses the existing masked\nvisual-language modeling task but also the new text-image alignment and text-image matching tasks in the pre-training\nstage, where cross-modality interaction is better learned. Meanwhile, it also integrates a spatial-aware self-attention\nmechanism into the Transformer architecture, so that the model can fully understand the relative positional\nrelationship among different text blocks. Experiment results show that LayoutLMv2 outperforms strong baselines and\nachieves new state-of-the-art results on a wide variety of downstream visually-rich document understanding tasks,\nincluding FUNSD (0.7895 -> 0.8420), CORD (0.9493 -> 0.9601), SROIE (0.9524 -> 0.9781), Kleister-NDA (0.834 -> 0.852),\nRVL-CDIP (0.9443 -> 0.9564), and DocVQA (0.7295 -> 0.8672). The pre-trained LayoutLMv2 model is publicly available at\nthis https URL.*\n\nLayoutLMv2 depends on `detectron2`, `torchvision` and `tesseract`. Run the\nfollowing to install them:\n```bash\npython -m pip install 'git+https://github.com/facebookresearch/detectron2.git'\npython -m pip install torchvision tesseract\n```\n(If you are developing for LayoutLMv2, note that passing the doctests also requires the installation of these packages.)\n\n## Usage tips\n\n- The main difference between LayoutLMv1 and LayoutLMv2 is that the latter incorporates visual embeddings during\n  pre-training (while LayoutLMv1 only adds visual embeddings during fine-tuning).\n- LayoutLMv2 adds both a relative 1D attention bias as well as a spatial 2D attention bias to the attention scores in\n  the self-attention layers. Details can be found on page 5 of the [paper](https://arxiv.org/abs/2012.14740).\n- Demo notebooks on how to use the LayoutLMv2 model on RVL-CDIP, FUNSD, DocVQA, CORD can be found [here](https://github.com/NielsRogge/Transformers-Tutorials).\n- LayoutLMv2 uses Facebook AI's [Detectron2](https://github.com/facebookresearch/detectron2/) package for its visual\n  backbone. See [this link](https://detectron2.readthedocs.io/en/latest/tutorials/install.html) for installation\n  instructions.\n- In addition to `input_ids`, [`~LayoutLMv2Model.forward`] expects 2 additional inputs, namely\n  `image` and `bbox`. The `image` input corresponds to the original document image in which the text\n  tokens occur. The model expects each document image to be of size 224x224."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 55c1bee6-bce8-40c5-8dd5-b1f75ce60363
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"55c1bee6-bce8-40c5-8dd5-b1f75ce60363\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c3999cb4-27da-4160-9f57-8e2fbb75c610\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f119c223-3dfd-4f5c-b63c-c2a5eef0ad99\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0e7b0c6a181e27aad01dcfb8db6cff9dc033898c2672c02024a421c15c02a834\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9c5cbd70-eb19-4513-b02e-d0db42fc357c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c6f8af7ec45894bf1b5fc87695831eea4dd059bf39fbfe266086db47b0d214cf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2743, \"end_char_idx\": 5351, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "doc_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "ref_doc_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "chroma:document": "The pre-trained LayoutLMv2 model is publicly available at\nthis https URL.*\n\nLayoutLMv2 depends on `detectron2`, `torchvision` and `tesseract`. Run the\nfollowing to install them:\n```bash\npython -m pip install 'git+https://github.com/facebookresearch/detectron2.git'\npython -m pip install torchvision tesseract\n```\n(If you are developing for LayoutLMv2, note that passing the doctests also requires the installation of these packages.)\n\n## Usage tips\n\n- The main difference between LayoutLMv1 and LayoutLMv2 is that the latter incorporates visual embeddings during\n  pre-training (while LayoutLMv1 only adds visual embeddings during fine-tuning).\n- LayoutLMv2 adds both a relative 1D attention bias as well as a spatial 2D attention bias to the attention scores in\n  the self-attention layers. Details can be found on page 5 of the [paper](https://arxiv.org/abs/2012.14740).\n- Demo notebooks on how to use the LayoutLMv2 model on RVL-CDIP, FUNSD, DocVQA, CORD can be found [here](https://github.com/NielsRogge/Transformers-Tutorials).\n- LayoutLMv2 uses Facebook AI's [Detectron2](https://github.com/facebookresearch/detectron2/) package for its visual\n  backbone. See [this link](https://detectron2.readthedocs.io/en/latest/tutorials/install.html) for installation\n  instructions.\n- In addition to `input_ids`, [`~LayoutLMv2Model.forward`] expects 2 additional inputs, namely\n  `image` and `bbox`. The `image` input corresponds to the original document image in which the text\n  tokens occur. The model expects each document image to be of size 224x224. This means that if you have a batch of\n  document images, `image` should be a tensor of shape (batch_size, 3, 224, 224). This can be either a\n  `torch.Tensor` or a `Detectron2.structures.ImageList`. You don't need to normalize the channels, as this is\n  done by the model. Important to note is that the visual backbone expects BGR channels instead of RGB, as all models\n  in Detectron2 are pre-trained using the BGR format. The `bbox` input are the bounding boxes (i.e. 2D-positions)\n  of the input text tokens. This is identical to [`LayoutLMModel`]. These can be obtained using an\n  external OCR engine such as Google's [Tesseract](https://github.com/tesseract-ocr/tesseract) (there's a [Python\n  wrapper](https://pypi.org/project/pytesseract/) available). Each bounding box should be in (x0, y0, x1, y1)\n  format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1)\n  represents the position of the lower right corner. Note that one first needs to normalize the bounding boxes to be on\n  a 0-1000 scale."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 9c5cbd70-eb19-4513-b02e-d0db42fc357c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9c5cbd70-eb19-4513-b02e-d0db42fc357c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c3999cb4-27da-4160-9f57-8e2fbb75c610\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"55c1bee6-bce8-40c5-8dd5-b1f75ce60363\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5a3b68a93396a806c340a6ab136f3a5bfbb4637d50028d658044368dadebfe19\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c4c432a9-8c7f-4215-a862-2bd250b324d5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d531aa9498bf8ffc564af98eb5777ea26e6e269f5bf33527401c728f1a993fc6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3905, \"end_char_idx\": 6806, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "doc_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "ref_doc_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "chroma:document": "See [this link](https://detectron2.readthedocs.io/en/latest/tutorials/install.html) for installation\n  instructions.\n- In addition to `input_ids`, [`~LayoutLMv2Model.forward`] expects 2 additional inputs, namely\n  `image` and `bbox`. The `image` input corresponds to the original document image in which the text\n  tokens occur. The model expects each document image to be of size 224x224. This means that if you have a batch of\n  document images, `image` should be a tensor of shape (batch_size, 3, 224, 224). This can be either a\n  `torch.Tensor` or a `Detectron2.structures.ImageList`. You don't need to normalize the channels, as this is\n  done by the model. Important to note is that the visual backbone expects BGR channels instead of RGB, as all models\n  in Detectron2 are pre-trained using the BGR format. The `bbox` input are the bounding boxes (i.e. 2D-positions)\n  of the input text tokens. This is identical to [`LayoutLMModel`]. These can be obtained using an\n  external OCR engine such as Google's [Tesseract](https://github.com/tesseract-ocr/tesseract) (there's a [Python\n  wrapper](https://pypi.org/project/pytesseract/) available). Each bounding box should be in (x0, y0, x1, y1)\n  format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1)\n  represents the position of the lower right corner. Note that one first needs to normalize the bounding boxes to be on\n  a 0-1000 scale. To normalize, you can use the following function:\n\n```python\ndef normalize_bbox(bbox, width, height):\n    return [\n        int(1000 * (bbox[0] / width)),\n        int(1000 * (bbox[1] / height)),\n        int(1000 * (bbox[2] / width)),\n        int(1000 * (bbox[3] / height)),\n    ]\n```\n\nHere, `width` and `height` correspond to the width and height of the original document in which the token\noccurs (before resizing the image). Those can be obtained using the Python Image Library (PIL) library for example, as\nfollows:\n\n```python\nfrom PIL import Image\n\nimage = Image.open(\n    \"name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images).\"\n)\n\nwidth, height = image.size\n```\n\nHowever, this model includes a brand new [`~transformers.LayoutLMv2Processor`] which can be used to directly\nprepare data for the model (including applying OCR under the hood). More information can be found in the \"Usage\"\nsection below.\n\n- Internally, [`~transformers.LayoutLMv2Model`] will send the `image` input through its visual backbone to\n  obtain a lower-resolution feature map, whose shape is equal to the `image_feature_pool_shape` attribute of\n  [`~transformers.LayoutLMv2Config`]. This feature map is then flattened to obtain a sequence of image tokens. As\n  the size of the feature map is 7x7 by default, one obtains 49 image tokens. These are then concatenated with the text\n  tokens, and send through the Transformer encoder."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: c4c432a9-8c7f-4215-a862-2bd250b324d5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c4c432a9-8c7f-4215-a862-2bd250b324d5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c3999cb4-27da-4160-9f57-8e2fbb75c610\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9c5cbd70-eb19-4513-b02e-d0db42fc357c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"caf80668ee15fa5716183094280addbfd8a45e1020911b811c3c7745bc6e4a0a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4babf1e6-475e-4e25-978f-e2dcde4c021c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4ac22479df246c933397775b5a5d15f9422f5e3339f8c9e576915face30fbae1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5267, \"end_char_idx\": 8127, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "doc_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "ref_doc_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "chroma:document": "Note that one first needs to normalize the bounding boxes to be on\n  a 0-1000 scale. To normalize, you can use the following function:\n\n```python\ndef normalize_bbox(bbox, width, height):\n    return [\n        int(1000 * (bbox[0] / width)),\n        int(1000 * (bbox[1] / height)),\n        int(1000 * (bbox[2] / width)),\n        int(1000 * (bbox[3] / height)),\n    ]\n```\n\nHere, `width` and `height` correspond to the width and height of the original document in which the token\noccurs (before resizing the image). Those can be obtained using the Python Image Library (PIL) library for example, as\nfollows:\n\n```python\nfrom PIL import Image\n\nimage = Image.open(\n    \"name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images).\"\n)\n\nwidth, height = image.size\n```\n\nHowever, this model includes a brand new [`~transformers.LayoutLMv2Processor`] which can be used to directly\nprepare data for the model (including applying OCR under the hood). More information can be found in the \"Usage\"\nsection below.\n\n- Internally, [`~transformers.LayoutLMv2Model`] will send the `image` input through its visual backbone to\n  obtain a lower-resolution feature map, whose shape is equal to the `image_feature_pool_shape` attribute of\n  [`~transformers.LayoutLMv2Config`]. This feature map is then flattened to obtain a sequence of image tokens. As\n  the size of the feature map is 7x7 by default, one obtains 49 image tokens. These are then concatenated with the text\n  tokens, and send through the Transformer encoder. This means that the last hidden states of the model will have a\n  length of 512 + 49 = 561, if you pad the text tokens up to the max length. More generally, the last hidden states\n  will have a shape of `seq_length` + `image_feature_pool_shape[0]` *\n  `config.image_feature_pool_shape[1]`.\n- When calling [`~transformers.LayoutLMv2Model.from_pretrained`], a warning will be printed with a long list of\n  parameter names that are not initialized. This is not a problem, as these parameters are batch normalization\n  statistics, which are going to have values when fine-tuning on a custom dataset.\n- If you want to train the model in a distributed environment, make sure to call [`synchronize_batch_norm`] on the\n  model in order to properly synchronize the batch normalization layers of the visual backbone.\n\nIn addition, there's LayoutXLM, which is a multilingual version of LayoutLMv2. More information can be found on\n[LayoutXLM's documentation page](layoutxlm).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with LayoutLMv2. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 4babf1e6-475e-4e25-978f-e2dcde4c021c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4babf1e6-475e-4e25-978f-e2dcde4c021c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c3999cb4-27da-4160-9f57-8e2fbb75c610\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c4c432a9-8c7f-4215-a862-2bd250b324d5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"322fe81d5489bad6bd4e4f4f7f80622e561f49af479448f4454549e7fff133b7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bbcd785a-3965-4075-98ea-4175b1499a7c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"71bc6e21020fea58f988daed6ca08706339e1d1e6ca9b2a9c52a0100c8cae53e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6559, \"end_char_idx\": 9560, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "doc_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "ref_doc_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "chroma:document": "This feature map is then flattened to obtain a sequence of image tokens. As\n  the size of the feature map is 7x7 by default, one obtains 49 image tokens. These are then concatenated with the text\n  tokens, and send through the Transformer encoder. This means that the last hidden states of the model will have a\n  length of 512 + 49 = 561, if you pad the text tokens up to the max length. More generally, the last hidden states\n  will have a shape of `seq_length` + `image_feature_pool_shape[0]` *\n  `config.image_feature_pool_shape[1]`.\n- When calling [`~transformers.LayoutLMv2Model.from_pretrained`], a warning will be printed with a long list of\n  parameter names that are not initialized. This is not a problem, as these parameters are batch normalization\n  statistics, which are going to have values when fine-tuning on a custom dataset.\n- If you want to train the model in a distributed environment, make sure to call [`synchronize_batch_norm`] on the\n  model in order to properly synchronize the batch normalization layers of the visual backbone.\n\nIn addition, there's LayoutXLM, which is a multilingual version of LayoutLMv2. More information can be found on\n[LayoutXLM's documentation page](layoutxlm).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with LayoutLMv2. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- A notebook on how to [finetune LayoutLMv2 for text-classification on RVL-CDIP dataset](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/RVL-CDIP/Fine_tuning_LayoutLMv2ForSequenceClassification_on_RVL_CDIP.ipynb).\n- See also: [Text classification task guide](../tasks/sequence_classification)\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- A notebook on how to [finetune LayoutLMv2 for question-answering on DocVQA dataset](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/DocVQA/Fine_tuning_LayoutLMv2ForQuestionAnswering_on_DocVQA.ipynb).\n- See also: [Question answering task guide](../tasks/question_answering)\n- See also: [Document question answering task guide](../tasks/document_question_answering)\n\n\n<PipelineTag pipeline=\"token-classification\"/>\n\n- A notebook on how to [finetune LayoutLMv2 for token-classification on CORD dataset](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/CORD/Fine_tuning_LayoutLMv2ForTokenClassification_on_CORD.ipynb).\n- A notebook on how to [finetune LayoutLMv2 for token-classification on FUNSD dataset](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/FUNSD/Fine_tuning_LayoutLMv2ForTokenClassification_on_FUNSD_using_HuggingFace_Trainer.ipynb)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: bbcd785a-3965-4075-98ea-4175b1499a7c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"bbcd785a-3965-4075-98ea-4175b1499a7c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c3999cb4-27da-4160-9f57-8e2fbb75c610\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4babf1e6-475e-4e25-978f-e2dcde4c021c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fef0c9a3a87404af277ea75bf4a153d3e8b0944d4d7aea5231f1b172118d2515\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5f6e964a-a9e6-4902-9362-c9ad9b19f9dc\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"47215afe5810e740365e4e1b698be9b7c2cfee049a0a5bbad17937c92b0c6300\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8029, \"end_char_idx\": 11098, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "doc_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "ref_doc_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "chroma:document": "The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- A notebook on how to [finetune LayoutLMv2 for text-classification on RVL-CDIP dataset](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/RVL-CDIP/Fine_tuning_LayoutLMv2ForSequenceClassification_on_RVL_CDIP.ipynb).\n- See also: [Text classification task guide](../tasks/sequence_classification)\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- A notebook on how to [finetune LayoutLMv2 for question-answering on DocVQA dataset](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/DocVQA/Fine_tuning_LayoutLMv2ForQuestionAnswering_on_DocVQA.ipynb).\n- See also: [Question answering task guide](../tasks/question_answering)\n- See also: [Document question answering task guide](../tasks/document_question_answering)\n\n\n<PipelineTag pipeline=\"token-classification\"/>\n\n- A notebook on how to [finetune LayoutLMv2 for token-classification on CORD dataset](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/CORD/Fine_tuning_LayoutLMv2ForTokenClassification_on_CORD.ipynb).\n- A notebook on how to [finetune LayoutLMv2 for token-classification on FUNSD dataset](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/FUNSD/Fine_tuning_LayoutLMv2ForTokenClassification_on_FUNSD_using_HuggingFace_Trainer.ipynb).\n- See also: [Token classification task guide](../tasks/token_classification)\n\n## Usage: LayoutLMv2Processor\n\nThe easiest way to prepare data for the model is to use [`LayoutLMv2Processor`], which internally\ncombines a image processor ([`LayoutLMv2ImageProcessor`]) and a tokenizer\n([`LayoutLMv2Tokenizer`] or [`LayoutLMv2TokenizerFast`]). The image processor\nhandles the image modality, while the tokenizer handles the text modality. A processor combines both, which is ideal\nfor a multi-modal model like LayoutLMv2. Note that you can still use both separately, if you only want to handle one\nmodality.\n\n```python\nfrom transformers import LayoutLMv2ImageProcessor, LayoutLMv2TokenizerFast, LayoutLMv2Processor\n\nimage_processor = LayoutLMv2ImageProcessor()  # apply_ocr is set to True by default\ntokenizer = LayoutLMv2TokenizerFast.from_pretrained(\"microsoft/layoutlmv2-base-uncased\")\nprocessor = LayoutLMv2Processor(image_processor, tokenizer)\n```\n\nIn short, one can provide a document image (and possibly additional data) to [`LayoutLMv2Processor`],\nand it will create the inputs expected by the model. Internally, the processor first uses\n[`LayoutLMv2ImageProcessor`] to apply OCR on the image to get a list of words and normalized\nbounding boxes, as well to resize the image to a given size in order to get the `image` input. The words and\nnormalized bounding boxes are then provided to [`LayoutLMv2Tokenizer`] or\n[`LayoutLMv2TokenizerFast`], which converts them to token-level `input_ids`,\n`attention_mask`, `token_type_ids`, `bbox`."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 5f6e964a-a9e6-4902-9362-c9ad9b19f9dc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5f6e964a-a9e6-4902-9362-c9ad9b19f9dc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c3999cb4-27da-4160-9f57-8e2fbb75c610\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bbcd785a-3965-4075-98ea-4175b1499a7c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a8336422e5e4a2cf5763c5fb7d40f4b1b8a9b4e87f71d84addf6d6c5240abe2b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dfe0a283-7582-4ac8-9451-0758aae7fd3c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"83d4b2cb5c4e4058b228cfa676b68d244f8ced8c4f34f64052425d42a0381d5b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9561, \"end_char_idx\": 12350, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "doc_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "ref_doc_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "chroma:document": "- See also: [Token classification task guide](../tasks/token_classification)\n\n## Usage: LayoutLMv2Processor\n\nThe easiest way to prepare data for the model is to use [`LayoutLMv2Processor`], which internally\ncombines a image processor ([`LayoutLMv2ImageProcessor`]) and a tokenizer\n([`LayoutLMv2Tokenizer`] or [`LayoutLMv2TokenizerFast`]). The image processor\nhandles the image modality, while the tokenizer handles the text modality. A processor combines both, which is ideal\nfor a multi-modal model like LayoutLMv2. Note that you can still use both separately, if you only want to handle one\nmodality.\n\n```python\nfrom transformers import LayoutLMv2ImageProcessor, LayoutLMv2TokenizerFast, LayoutLMv2Processor\n\nimage_processor = LayoutLMv2ImageProcessor()  # apply_ocr is set to True by default\ntokenizer = LayoutLMv2TokenizerFast.from_pretrained(\"microsoft/layoutlmv2-base-uncased\")\nprocessor = LayoutLMv2Processor(image_processor, tokenizer)\n```\n\nIn short, one can provide a document image (and possibly additional data) to [`LayoutLMv2Processor`],\nand it will create the inputs expected by the model. Internally, the processor first uses\n[`LayoutLMv2ImageProcessor`] to apply OCR on the image to get a list of words and normalized\nbounding boxes, as well to resize the image to a given size in order to get the `image` input. The words and\nnormalized bounding boxes are then provided to [`LayoutLMv2Tokenizer`] or\n[`LayoutLMv2TokenizerFast`], which converts them to token-level `input_ids`,\n`attention_mask`, `token_type_ids`, `bbox`. Optionally, one can provide word labels to the processor,\nwhich are turned into token-level `labels`.\n\n[`LayoutLMv2Processor`] uses [PyTesseract](https://pypi.org/project/pytesseract/), a Python\nwrapper around Google's Tesseract OCR engine, under the hood. Note that you can still use your own OCR engine of\nchoice, and provide the words and normalized boxes yourself. This requires initializing\n[`LayoutLMv2ImageProcessor`] with `apply_ocr` set to `False`.\n\nIn total, there are 5 use cases that are supported by the processor. Below, we list them all. Note that each of these\nuse cases work for both batched and non-batched inputs (we illustrate them for non-batched inputs).\n\n**Use case 1: document image classification (training, inference) + token classification (inference), apply_ocr =\nTrue**\n\nThis is the simplest case, in which the processor (actually the image processor) will perform OCR on the image to get\nthe words and normalized bounding boxes.\n\n```python\nfrom transformers import LayoutLMv2Processor\nfrom PIL import Image\n\nprocessor = LayoutLMv2Processor.from_pretrained(\"microsoft/layoutlmv2-base-uncased\")\n\nimage = Image.open(\n    \"name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images).\""}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: dfe0a283-7582-4ac8-9451-0758aae7fd3c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"dfe0a283-7582-4ac8-9451-0758aae7fd3c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c3999cb4-27da-4160-9f57-8e2fbb75c610\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5f6e964a-a9e6-4902-9362-c9ad9b19f9dc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b789baa80406ed679ad91e913a574e1b14661598ecedbcfe0f19bf85de6cd4c1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e4db6da7-b3e2-40ec-a84b-144b178c764b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9cc199a8fd3a99d1b946c4e20a2cae2f1bbe18e070ec31e62e4470b77e5eff0e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10890, \"end_char_idx\": 14026, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "doc_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "ref_doc_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "chroma:document": "The words and\nnormalized bounding boxes are then provided to [`LayoutLMv2Tokenizer`] or\n[`LayoutLMv2TokenizerFast`], which converts them to token-level `input_ids`,\n`attention_mask`, `token_type_ids`, `bbox`. Optionally, one can provide word labels to the processor,\nwhich are turned into token-level `labels`.\n\n[`LayoutLMv2Processor`] uses [PyTesseract](https://pypi.org/project/pytesseract/), a Python\nwrapper around Google's Tesseract OCR engine, under the hood. Note that you can still use your own OCR engine of\nchoice, and provide the words and normalized boxes yourself. This requires initializing\n[`LayoutLMv2ImageProcessor`] with `apply_ocr` set to `False`.\n\nIn total, there are 5 use cases that are supported by the processor. Below, we list them all. Note that each of these\nuse cases work for both batched and non-batched inputs (we illustrate them for non-batched inputs).\n\n**Use case 1: document image classification (training, inference) + token classification (inference), apply_ocr =\nTrue**\n\nThis is the simplest case, in which the processor (actually the image processor) will perform OCR on the image to get\nthe words and normalized bounding boxes.\n\n```python\nfrom transformers import LayoutLMv2Processor\nfrom PIL import Image\n\nprocessor = LayoutLMv2Processor.from_pretrained(\"microsoft/layoutlmv2-base-uncased\")\n\nimage = Image.open(\n    \"name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images).\"\n).convert(\"RGB\")\nencoding = processor(\n    image, return_tensors=\"pt\"\n)  # you can also add all tokenizer parameters here such as padding, truncation\nprint(encoding.keys())\n# dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'bbox', 'image'])\n```\n\n**Use case 2: document image classification (training, inference) + token classification (inference), apply_ocr=False**\n\nIn case one wants to do OCR themselves, one can initialize the image processor with `apply_ocr` set to\n`False`. In that case, one should provide the words and corresponding (normalized) bounding boxes themselves to\nthe processor.\n\n```python\nfrom transformers import LayoutLMv2Processor\nfrom PIL import Image\n\nprocessor = LayoutLMv2Processor.from_pretrained(\"microsoft/layoutlmv2-base-uncased\", revision=\"no_ocr\")\n\nimage = Image.open(\n    \"name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images).\"\n).convert(\"RGB\")\nwords = [\"hello\", \"world\"]\nboxes = [[1, 2, 3, 4], [5, 6, 7, 8]]  # make sure to normalize your bounding boxes\nencoding = processor(image, words, boxes=boxes, return_tensors=\"pt\")\nprint(encoding.keys())\n# dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'bbox', 'image'])\n```\n\n**Use case 3: token classification (training), apply_ocr=False**\n\nFor token classification tasks (such as FUNSD, CORD, SROIE, Kleister-NDA), one can also provide the corresponding word\nlabels in order to train a model. The processor will then convert these into token-level `labels`. By default, it\nwill only label the first wordpiece of a word, and label the remaining wordpieces with -100, which is the\n`ignore_index` of PyTorch's CrossEntropyLoss."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e4db6da7-b3e2-40ec-a84b-144b178c764b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e4db6da7-b3e2-40ec-a84b-144b178c764b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c3999cb4-27da-4160-9f57-8e2fbb75c610\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dfe0a283-7582-4ac8-9451-0758aae7fd3c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bbc052b043393f9c8d1fc74795cdf6be4921202006b19c7635b7b38e953d7dec\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"966d5fc8-2a46-436f-9ba5-550715dd1666\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bffa87630189c671fd705e6073f4ea42c511f3deb80a825f0e5982ffae7547a1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12842, \"end_char_idx\": 15797, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "doc_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "ref_doc_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "chroma:document": "In that case, one should provide the words and corresponding (normalized) bounding boxes themselves to\nthe processor.\n\n```python\nfrom transformers import LayoutLMv2Processor\nfrom PIL import Image\n\nprocessor = LayoutLMv2Processor.from_pretrained(\"microsoft/layoutlmv2-base-uncased\", revision=\"no_ocr\")\n\nimage = Image.open(\n    \"name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images).\"\n).convert(\"RGB\")\nwords = [\"hello\", \"world\"]\nboxes = [[1, 2, 3, 4], [5, 6, 7, 8]]  # make sure to normalize your bounding boxes\nencoding = processor(image, words, boxes=boxes, return_tensors=\"pt\")\nprint(encoding.keys())\n# dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'bbox', 'image'])\n```\n\n**Use case 3: token classification (training), apply_ocr=False**\n\nFor token classification tasks (such as FUNSD, CORD, SROIE, Kleister-NDA), one can also provide the corresponding word\nlabels in order to train a model. The processor will then convert these into token-level `labels`. By default, it\nwill only label the first wordpiece of a word, and label the remaining wordpieces with -100, which is the\n`ignore_index` of PyTorch's CrossEntropyLoss. In case you want all wordpieces of a word to be labeled, you can\ninitialize the tokenizer with `only_label_first_subword` set to `False`.\n\n```python\nfrom transformers import LayoutLMv2Processor\nfrom PIL import Image\n\nprocessor = LayoutLMv2Processor.from_pretrained(\"microsoft/layoutlmv2-base-uncased\", revision=\"no_ocr\")\n\nimage = Image.open(\n    \"name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images).\"\n).convert(\"RGB\")\nwords = [\"hello\", \"world\"]\nboxes = [[1, 2, 3, 4], [5, 6, 7, 8]]  # make sure to normalize your bounding boxes\nword_labels = [1, 2]\nencoding = processor(image, words, boxes=boxes, word_labels=word_labels, return_tensors=\"pt\")\nprint(encoding.keys())\n# dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'bbox', 'labels', 'image'])\n```\n\n**Use case 4: visual question answering (inference), apply_ocr=True**\n\nFor visual question answering tasks (such as DocVQA), you can provide a question to the processor. By default, the\nprocessor will apply OCR on the image, and create [CLS] question tokens [SEP] word tokens [SEP].\n\n```python\nfrom transformers import LayoutLMv2Processor\nfrom PIL import Image\n\nprocessor = LayoutLMv2Processor.from_pretrained(\"microsoft/layoutlmv2-base-uncased\")\n\nimage = Image.open(\n    \"name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images).\"\n).convert(\"RGB\")\nquestion = \"What's his name?\"\nencoding = processor(image, question, return_tensors=\"pt\")\nprint(encoding.keys())\n# dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'bbox', 'image'])\n```\n\n**Use case 5: visual question answering (inference), apply_ocr=False**\n\nFor visual question answering tasks (such as DocVQA), you can provide a question to the processor."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 966d5fc8-2a46-436f-9ba5-550715dd1666
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"966d5fc8-2a46-436f-9ba5-550715dd1666\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c3999cb4-27da-4160-9f57-8e2fbb75c610\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e4db6da7-b3e2-40ec-a84b-144b178c764b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ab6e96adc85940729724dacff944092812a585b1b972646ac100aa60637b12f0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"30ec7155-1a43-45a2-8b38-378ed6c848a6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"cbc8471dd3a1451e287a7687a0ea2e0e1546e1f9d982554a815907b38e30e044\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14422, \"end_char_idx\": 16274, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "doc_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "ref_doc_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "chroma:document": "of your documents (PDFs must be converted to images).\"\n).convert(\"RGB\")\nwords = [\"hello\", \"world\"]\nboxes = [[1, 2, 3, 4], [5, 6, 7, 8]]  # make sure to normalize your bounding boxes\nword_labels = [1, 2]\nencoding = processor(image, words, boxes=boxes, word_labels=word_labels, return_tensors=\"pt\")\nprint(encoding.keys())\n# dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'bbox', 'labels', 'image'])\n```\n\n**Use case 4: visual question answering (inference), apply_ocr=True**\n\nFor visual question answering tasks (such as DocVQA), you can provide a question to the processor. By default, the\nprocessor will apply OCR on the image, and create [CLS] question tokens [SEP] word tokens [SEP].\n\n```python\nfrom transformers import LayoutLMv2Processor\nfrom PIL import Image\n\nprocessor = LayoutLMv2Processor.from_pretrained(\"microsoft/layoutlmv2-base-uncased\")\n\nimage = Image.open(\n    \"name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images).\"\n).convert(\"RGB\")\nquestion = \"What's his name?\"\nencoding = processor(image, question, return_tensors=\"pt\")\nprint(encoding.keys())\n# dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'bbox', 'image'])\n```\n\n**Use case 5: visual question answering (inference), apply_ocr=False**\n\nFor visual question answering tasks (such as DocVQA), you can provide a question to the processor. If you want to\nperform OCR yourself, you can provide your own words and (normalized) bounding boxes to the processor.\n\n```python\nfrom transformers import LayoutLMv2Processor\nfrom PIL import Image\n\nprocessor = LayoutLMv2Processor.from_pretrained(\"microsoft/layoutlmv2-base-uncased\", revision=\"no_ocr\")\n\nimage = Image.open(\n    \"name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images).\"\n).convert(\"RGB\")\nquestion = \"What's his name?\""}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 30ec7155-1a43-45a2-8b38-378ed6c848a6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"30ec7155-1a43-45a2-8b38-378ed6c848a6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c3999cb4-27da-4160-9f57-8e2fbb75c610\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"966d5fc8-2a46-436f-9ba5-550715dd1666\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv2\", \"title\": \"LayoutLMV2\", \"tokens\": 4571, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f06ac39622f1608d01d0986dde382db55af518362654ba06f4e661af7e9e8bcd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15007, \"end_char_idx\": 17351, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "doc_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "ref_doc_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "chroma:document": "By default, the\nprocessor will apply OCR on the image, and create [CLS] question tokens [SEP] word tokens [SEP].\n\n```python\nfrom transformers import LayoutLMv2Processor\nfrom PIL import Image\n\nprocessor = LayoutLMv2Processor.from_pretrained(\"microsoft/layoutlmv2-base-uncased\")\n\nimage = Image.open(\n    \"name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images).\"\n).convert(\"RGB\")\nquestion = \"What's his name?\"\nencoding = processor(image, question, return_tensors=\"pt\")\nprint(encoding.keys())\n# dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'bbox', 'image'])\n```\n\n**Use case 5: visual question answering (inference), apply_ocr=False**\n\nFor visual question answering tasks (such as DocVQA), you can provide a question to the processor. If you want to\nperform OCR yourself, you can provide your own words and (normalized) bounding boxes to the processor.\n\n```python\nfrom transformers import LayoutLMv2Processor\nfrom PIL import Image\n\nprocessor = LayoutLMv2Processor.from_pretrained(\"microsoft/layoutlmv2-base-uncased\", revision=\"no_ocr\")\n\nimage = Image.open(\n    \"name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images).\"\n).convert(\"RGB\")\nquestion = \"What's his name?\"\nwords = [\"hello\", \"world\"]\nboxes = [[1, 2, 3, 4], [5, 6, 7, 8]]  # make sure to normalize your bounding boxes\nencoding = processor(image, question, words, boxes=boxes, return_tensors=\"pt\")\nprint(encoding.keys())\n# dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'bbox', 'image'])\n```\n\n## LayoutLMv2Config\n\n[[autodoc]] LayoutLMv2Config\n\n## LayoutLMv2FeatureExtractor\n\n[[autodoc]] LayoutLMv2FeatureExtractor\n    - __call__\n\n## LayoutLMv2ImageProcessor\n\n[[autodoc]] LayoutLMv2ImageProcessor\n    - preprocess\n\n## LayoutLMv2Tokenizer\n\n[[autodoc]] LayoutLMv2Tokenizer\n    - __call__\n    - save_vocabulary\n\n## LayoutLMv2TokenizerFast\n\n[[autodoc]] LayoutLMv2TokenizerFast\n    - __call__\n\n## LayoutLMv2Processor\n\n[[autodoc]] LayoutLMv2Processor\n    - __call__\n\n## LayoutLMv2Model\n\n[[autodoc]] LayoutLMv2Model\n    - forward\n\n## LayoutLMv2ForSequenceClassification\n\n[[autodoc]] LayoutLMv2ForSequenceClassification\n\n## LayoutLMv2ForTokenClassification\n\n[[autodoc]] LayoutLMv2ForTokenClassification\n\n## LayoutLMv2ForQuestionAnswering\n\n[[autodoc]] LayoutLMv2ForQuestionAnswering"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 69bb0690-7085-47a1-9154-1113abeea30e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/retribert", "title": "RetriBERT", "tokens": 462, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"69bb0690-7085-47a1-9154-1113abeea30e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/retribert\", \"title\": \"RetriBERT\", \"tokens\": 462, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"79060bf7-6d83-45d0-90b4-035b22c8d9f1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/retribert\", \"title\": \"RetriBERT\", \"tokens\": 462, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e22becdb82c278b17d0311781b3b1504689ce983bc1016c950ea6ac26e3db3e1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1127, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "79060bf7-6d83-45d0-90b4-035b22c8d9f1", "doc_id": "79060bf7-6d83-45d0-90b4-035b22c8d9f1", "ref_doc_id": "79060bf7-6d83-45d0-90b4-035b22c8d9f1", "chroma:document": "# RetriBERT\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, so we won't accept any new PRs changing its code.\n\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.30.0.\nYou can do so by running the following command: `pip install -U transformers==4.30.0`.\n\n</Tip>\n\n## Overview\n\nThe RetriBERT model was proposed in the blog post [Explain Anything Like I'm Five: A Model for Open Domain Long Form\nQuestion Answering](https://yjernite.github.io/lfqa.html). RetriBERT is a small model that uses either a single or\npair of BERT encoders with lower-dimension projection for dense semantic indexing of text.\n\nThis model was contributed by [yjernite](https://huggingface.co/yjernite). Code to train and use the model can be\nfound [here](https://github.com/huggingface/transformers/tree/main/examples/research-projects/distillation).\n\n\n## RetriBertConfig\n\n[[autodoc]] RetriBertConfig\n\n## RetriBertTokenizer\n\n[[autodoc]] RetriBertTokenizer\n\n## RetriBertTokenizerFast\n\n[[autodoc]] RetriBertTokenizerFast\n\n## RetriBertModel\n\n[[autodoc]] RetriBertModel\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 4edc5c68-1c06-4b72-8b15-abf5f3717006
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bark", "title": "Bark", "tokens": 2279, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4edc5c68-1c06-4b72-8b15-abf5f3717006\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bark\", \"title\": \"Bark\", \"tokens\": 2279, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1561756b-cc39-435e-9962-9cf71dd1ec38\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bark\", \"title\": \"Bark\", \"tokens\": 2279, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e023131db50dd78a4993c78bf26a853014d36071eb90afe90b5db0b4ad24cf1d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6f87aa01-19f5-4dfd-aabd-79170a2855a4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fb46d642bf545c8182efb842fe820fc6c157f1937f480a046cbd2d6774292712\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3230, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1561756b-cc39-435e-9962-9cf71dd1ec38", "doc_id": "1561756b-cc39-435e-9962-9cf71dd1ec38", "ref_doc_id": "1561756b-cc39-435e-9962-9cf71dd1ec38", "chroma:document": "# Bark\n\n## Overview\n\nBark is a transformer-based text-to-speech model proposed by Suno AI in [suno-ai/bark](https://github.com/suno-ai/bark).\n\nBark is made of 4 main models:\n\n- [`BarkSemanticModel`] (also referred to as the 'text' model): a causal auto-regressive transformer model that takes as input tokenized text, and predicts semantic text tokens that capture the meaning of the text.\n- [`BarkCoarseModel`] (also referred to as the 'coarse acoustics' model): a causal autoregressive transformer, that takes as input the results of the [`BarkSemanticModel`] model. It aims at predicting the first two audio codebooks necessary for EnCodec.\n- [`BarkFineModel`] (the 'fine acoustics' model), this time a non-causal autoencoder transformer, which iteratively predicts the last codebooks based on the sum of the previous codebooks embeddings.\n- having predicted all the codebook channels from the [`EncodecModel`], Bark uses it to decode the output audio array.\n\nIt should be noted that each of the first three modules can support conditional speaker embeddings to condition the output sound according to specific predefined voice.\n\nThis model was contributed by [Yoach Lacombe (ylacombe)](https://huggingface.co/ylacombe) and [Sanchit Gandhi (sanchit-gandhi)](https://github.com/sanchit-gandhi).\nThe original code can be found [here](https://github.com/suno-ai/bark).\n\n### Optimizing Bark\n\nBark can be optimized with just a few extra lines of code, which **significantly reduces its memory footprint** and **accelerates inference**.\n\n#### Using half-precision\n\nYou can speed up inference and reduce memory footprint by 50% simply by loading the model in half-precision.\n\n```python\nfrom transformers import BarkModel\nimport torch\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nmodel = BarkModel.from_pretrained(\"suno/bark-small\", torch_dtype=torch.float16).to(device)\n```\n\n#### Using CPU offload\n\nAs mentioned above, Bark is made up of 4 sub-models, which are called up sequentially during audio generation. In other words, while one sub-model is in use, the other sub-models are idle.\n\nIf you're using a CUDA device, a simple solution to benefit from an 80% reduction in memory footprint is to offload the submodels from GPU to CPU when they're idle. This operation is called *CPU offloading*. You can use it with one line of code as follows:\n\n```python\nmodel.enable_cpu_offload()\n```\n\nNote that \ud83e\udd17 Accelerate must be installed before using this feature. [Here's how to install it.](https://huggingface.co/docs/accelerate/basic_tutorials/install)\n\n#### Using Better Transformer\n\nBetter Transformer is an \ud83e\udd17 Optimum feature that performs kernel fusion under the hood. You can gain 20% to 30% in speed with zero performance degradation. It only requires one line of code to export the model to \ud83e\udd17 Better Transformer:\n\n```python\nmodel =  model.to_bettertransformer()\n```\n\nNote that \ud83e\udd17 Optimum must be installed before using this feature. [Here's how to install it.](https://huggingface.co/docs/optimum/installation)\n\n#### Using Flash Attention 2\n\nFlash Attention 2 is an even faster, optimized version of the previous optimization.\n\n##### Installation \n\nFirst, check whether your hardware is compatible with Flash Attention 2."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6f87aa01-19f5-4dfd-aabd-79170a2855a4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bark", "title": "Bark", "tokens": 2279, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6f87aa01-19f5-4dfd-aabd-79170a2855a4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bark\", \"title\": \"Bark\", \"tokens\": 2279, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1561756b-cc39-435e-9962-9cf71dd1ec38\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bark\", \"title\": \"Bark\", \"tokens\": 2279, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e023131db50dd78a4993c78bf26a853014d36071eb90afe90b5db0b4ad24cf1d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4edc5c68-1c06-4b72-8b15-abf5f3717006\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bark\", \"title\": \"Bark\", \"tokens\": 2279, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5c4e160636eb0fdaa9b1c6aea0b758e554e4672797e511c71a9c485f7759856f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5526658e-858c-4b78-8c51-9cd9f2787469\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4a2a8f083b7c71018ca31ca093bf9928cf9f957002a454970c96f09db8c09131\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1535, \"end_char_idx\": 4457, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1561756b-cc39-435e-9962-9cf71dd1ec38", "doc_id": "1561756b-cc39-435e-9962-9cf71dd1ec38", "ref_doc_id": "1561756b-cc39-435e-9962-9cf71dd1ec38", "chroma:document": "#### Using half-precision\n\nYou can speed up inference and reduce memory footprint by 50% simply by loading the model in half-precision.\n\n```python\nfrom transformers import BarkModel\nimport torch\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nmodel = BarkModel.from_pretrained(\"suno/bark-small\", torch_dtype=torch.float16).to(device)\n```\n\n#### Using CPU offload\n\nAs mentioned above, Bark is made up of 4 sub-models, which are called up sequentially during audio generation. In other words, while one sub-model is in use, the other sub-models are idle.\n\nIf you're using a CUDA device, a simple solution to benefit from an 80% reduction in memory footprint is to offload the submodels from GPU to CPU when they're idle. This operation is called *CPU offloading*. You can use it with one line of code as follows:\n\n```python\nmodel.enable_cpu_offload()\n```\n\nNote that \ud83e\udd17 Accelerate must be installed before using this feature. [Here's how to install it.](https://huggingface.co/docs/accelerate/basic_tutorials/install)\n\n#### Using Better Transformer\n\nBetter Transformer is an \ud83e\udd17 Optimum feature that performs kernel fusion under the hood. You can gain 20% to 30% in speed with zero performance degradation. It only requires one line of code to export the model to \ud83e\udd17 Better Transformer:\n\n```python\nmodel =  model.to_bettertransformer()\n```\n\nNote that \ud83e\udd17 Optimum must be installed before using this feature. [Here's how to install it.](https://huggingface.co/docs/optimum/installation)\n\n#### Using Flash Attention 2\n\nFlash Attention 2 is an even faster, optimized version of the previous optimization.\n\n##### Installation \n\nFirst, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features). If your hardware is not compatible with Flash Attention 2, you can still benefit from attention kernel optimisations through Better Transformer support covered [above](https://huggingface.co/docs/transformers/main/en/model_doc/bark#using-better-transformer).\n\nNext, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\n\n##### Usage\n\nTo load a model using Flash Attention 2, we can pass the `attn_implementation=\"flash_attention_2\"` flag to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). We'll also load the model in half-precision (e.g. `torch.float16`), since it results in almost no degradation to audio quality but significantly lower memory usage and faster inference:\n\n```python\nmodel = BarkModel.from_pretrained(\"suno/bark-small\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\").to(device)\n```\n\n##### Performance comparison"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 5526658e-858c-4b78-8c51-9cd9f2787469
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bark", "title": "Bark", "tokens": 2279, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5526658e-858c-4b78-8c51-9cd9f2787469\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bark\", \"title\": \"Bark\", \"tokens\": 2279, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1561756b-cc39-435e-9962-9cf71dd1ec38\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bark\", \"title\": \"Bark\", \"tokens\": 2279, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e023131db50dd78a4993c78bf26a853014d36071eb90afe90b5db0b4ad24cf1d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6f87aa01-19f5-4dfd-aabd-79170a2855a4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bark\", \"title\": \"Bark\", \"tokens\": 2279, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9496910fd08e3947854fa3dd7c18cc72887c0a9ab34beada93f1307fd34f5f21\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"05404693-f199-4e83-a337-c0092096dbe5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e47fc7a09883719e5c019374ab407b628b1bcd859cddda56c88f20f673fccbd5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2744, \"end_char_idx\": 5495, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1561756b-cc39-435e-9962-9cf71dd1ec38", "doc_id": "1561756b-cc39-435e-9962-9cf71dd1ec38", "ref_doc_id": "1561756b-cc39-435e-9962-9cf71dd1ec38", "chroma:document": "It only requires one line of code to export the model to \ud83e\udd17 Better Transformer:\n\n```python\nmodel =  model.to_bettertransformer()\n```\n\nNote that \ud83e\udd17 Optimum must be installed before using this feature. [Here's how to install it.](https://huggingface.co/docs/optimum/installation)\n\n#### Using Flash Attention 2\n\nFlash Attention 2 is an even faster, optimized version of the previous optimization.\n\n##### Installation \n\nFirst, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features). If your hardware is not compatible with Flash Attention 2, you can still benefit from attention kernel optimisations through Better Transformer support covered [above](https://huggingface.co/docs/transformers/main/en/model_doc/bark#using-better-transformer).\n\nNext, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\n\n##### Usage\n\nTo load a model using Flash Attention 2, we can pass the `attn_implementation=\"flash_attention_2\"` flag to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). We'll also load the model in half-precision (e.g. `torch.float16`), since it results in almost no degradation to audio quality but significantly lower memory usage and faster inference:\n\n```python\nmodel = BarkModel.from_pretrained(\"suno/bark-small\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\").to(device)\n```\n\n##### Performance comparison\n\n\nThe following diagram shows the latency for the native attention implementation (no optimisation) against Better Transformer and Flash Attention 2. In all cases, we generate 400 semantic tokens on a 40GB A100 GPU with PyTorch 2.1. Flash Attention 2 is also consistently faster than Better Transformer, and its performance improves even more as batch sizes increase:\n\n<div style=\"text-align: center\">\n<img src=\"https://huggingface.co/datasets/ylacombe/benchmark-comparison/resolve/main/Bark%20Optimization%20Benchmark.png\">\n</div>\n\nTo put this into perspective, on an NVIDIA A100 and when generating 400 semantic tokens with a batch size of 16, you can get 17 times the [throughput](https://huggingface.co/blog/optimizing-bark#throughput) and still be 2 seconds faster than generating sentences one by one with the native model implementation. In other words, all the samples will be generated 17 times faster.\n\nAt batch size 8, on an NVIDIA A100, Flash Attention 2 is also 10% faster than Better Transformer, and at batch size 16, 25%."}


