# embeddings_queue

created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 5a966c6f-cf79-4c3c-9613-52f40dc5a9d5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/cohere", "title": "Cohere", "tokens": 1441, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5a966c6f-cf79-4c3c-9613-52f40dc5a9d5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cohere\", \"title\": \"Cohere\", \"tokens\": 1441, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"df90e0e8-1bf7-4ec6-a734-0126b7db9dc9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cohere\", \"title\": \"Cohere\", \"tokens\": 1441, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"da67d8ba0be098800d4628f91e861c5e781efc06f9055313167a33636937a092\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"95bfcca4-2431-496d-8d08-b23f7bf08e62\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"06a49eae299cbe977a8db5498e8706b78b041dfa1ca26991b1aa6c6b1be911e0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2844, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "df90e0e8-1bf7-4ec6-a734-0126b7db9dc9", "doc_id": "df90e0e8-1bf7-4ec6-a734-0126b7db9dc9", "ref_doc_id": "df90e0e8-1bf7-4ec6-a734-0126b7db9dc9", "chroma:document": "# Cohere\n\n## Overview\n\nThe Cohere Command-R model was proposed in the blogpost [Command-R: Retrieval Augmented Generation at Production Scale](https://txt.cohere.com/command-r/) by the Cohere Team.\n\nThe abstract from the paper is the following:\n\n*Command-R is a scalable generative model targeting RAG and Tool Use to enable production-scale AI for enterprise. Today, we are introducing Command-R, a new LLM aimed at large-scale production workloads. Command-R targets the emerging \u201cscalable\u201d category of models that balance high efficiency with strong accuracy, enabling companies to move beyond proof of concept, and into production.*\n\n*Command-R is a generative model optimized for long context tasks such as retrieval augmented generation (RAG) and using external APIs and tools. It is designed to work in concert with our industry-leading Embed and Rerank models to provide best-in-class integration for RAG applications and excel at enterprise use cases. As a model built for companies to implement at scale, Command-R boasts:\n- Strong accuracy on RAG and Tool Use\n- Low latency, and high throughput\n- Longer 128k context and lower pricing\n- Strong capabilities across 10 key languages\n- Model weights available on HuggingFace for research and evaluation\n\nCheckout model checkpoints [here](https://huggingface.co/CohereForAI/c4ai-command-r-v01).\nThis model was contributed by [Saurabh Dash](https://huggingface.co/saurabhdash) and [Ahmet \u00dcst\u00fcn](https://huggingface.co/ahmetustun). The code of the implementation in Hugging Face is based on GPT-NeoX [here](https://github.com/EleutherAI/gpt-neox).\n\n## Usage tips\n\n<Tip warning={true}>\n\nThe checkpoints uploaded on the Hub use `torch_dtype = 'float16'`, which will be\nused by the `AutoModel` API to cast the checkpoints from `torch.float32` to `torch.float16`. \n\nThe `dtype` of the online weights is mostly irrelevant unless you are using `torch_dtype=\"auto\"` when initializing a model using `model = AutoModelForCausalLM.from_pretrained(\"path\", torch_dtype = \"auto\")`. The reason is that the model will first be downloaded ( using the `dtype` of the checkpoints online), then it will be casted to the default `dtype` of `torch` (becomes `torch.float32`), and finally, if there is a `torch_dtype` provided in the config, it will be used. \n\nTraining the model in `float16` is not recommended and is known to produce `nan`; as such, the model should be trained in `bfloat16`.\n\n</Tip>\nThe model and tokenizer can be loaded via:\n\n```python\n# pip install transformers\nfrom transformers import AutoTokenizer, AutoModelForCausalLM\n\nmodel_id = \"CohereForAI/c4ai-command-r-v01\"\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nmodel = AutoModelForCausalLM.from_pretrained(model_id)\n\n# Format message with the command-r chat template\nmessages = [{\"role\": \"user\", \"content\": \"Hello, how are you?\"}]"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 95bfcca4-2431-496d-8d08-b23f7bf08e62
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/cohere", "title": "Cohere", "tokens": 1441, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"95bfcca4-2431-496d-8d08-b23f7bf08e62\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cohere\", \"title\": \"Cohere\", \"tokens\": 1441, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"df90e0e8-1bf7-4ec6-a734-0126b7db9dc9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cohere\", \"title\": \"Cohere\", \"tokens\": 1441, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"da67d8ba0be098800d4628f91e861c5e781efc06f9055313167a33636937a092\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5a966c6f-cf79-4c3c-9613-52f40dc5a9d5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cohere\", \"title\": \"Cohere\", \"tokens\": 1441, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"29dc6601e2f09f3994b29f5db54ffb4488558d5abf8cd7b919c1715b198d8c03\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bfedeb38-c993-425b-b479-b306073511fd\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"98896c33887a60da0521ff2a10d42500790e0aa73423af8aeeb260cdaef48621\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1352, \"end_char_idx\": 4038, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "df90e0e8-1bf7-4ec6-a734-0126b7db9dc9", "doc_id": "df90e0e8-1bf7-4ec6-a734-0126b7db9dc9", "ref_doc_id": "df90e0e8-1bf7-4ec6-a734-0126b7db9dc9", "chroma:document": "This model was contributed by [Saurabh Dash](https://huggingface.co/saurabhdash) and [Ahmet \u00dcst\u00fcn](https://huggingface.co/ahmetustun). The code of the implementation in Hugging Face is based on GPT-NeoX [here](https://github.com/EleutherAI/gpt-neox).\n\n## Usage tips\n\n<Tip warning={true}>\n\nThe checkpoints uploaded on the Hub use `torch_dtype = 'float16'`, which will be\nused by the `AutoModel` API to cast the checkpoints from `torch.float32` to `torch.float16`. \n\nThe `dtype` of the online weights is mostly irrelevant unless you are using `torch_dtype=\"auto\"` when initializing a model using `model = AutoModelForCausalLM.from_pretrained(\"path\", torch_dtype = \"auto\")`. The reason is that the model will first be downloaded ( using the `dtype` of the checkpoints online), then it will be casted to the default `dtype` of `torch` (becomes `torch.float32`), and finally, if there is a `torch_dtype` provided in the config, it will be used. \n\nTraining the model in `float16` is not recommended and is known to produce `nan`; as such, the model should be trained in `bfloat16`.\n\n</Tip>\nThe model and tokenizer can be loaded via:\n\n```python\n# pip install transformers\nfrom transformers import AutoTokenizer, AutoModelForCausalLM\n\nmodel_id = \"CohereForAI/c4ai-command-r-v01\"\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nmodel = AutoModelForCausalLM.from_pretrained(model_id)\n\n# Format message with the command-r chat template\nmessages = [{\"role\": \"user\", \"content\": \"Hello, how are you?\"}]\ninput_ids = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors=\"pt\")\n## <BOS_TOKEN><|START_OF_TURN_TOKEN|><|USER_TOKEN|>Hello, how are you?<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>\n\ngen_tokens = model.generate(\n    input_ids, \n    max_new_tokens=100, \n    do_sample=True, \n    temperature=0.3,\n    )\n\ngen_text = tokenizer.decode(gen_tokens[0])\nprint(gen_text)\n```\n\n- When using Flash Attention 2 via `attn_implementation=\"flash_attention_2\"`, don't pass `torch_dtype` to the `from_pretrained` class method and use Automatic Mixed-Precision training. When using `Trainer`, it is simply specifying either `fp16` or `bf16` to `True`. Otherwise, make sure you are using `torch.autocast`. This is required because the Flash Attention only support `fp16` and `bf16` data type.\n\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Command-R. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: bfedeb38-c993-425b-b479-b306073511fd
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/cohere", "title": "Cohere", "tokens": 1441, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"bfedeb38-c993-425b-b479-b306073511fd\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cohere\", \"title\": \"Cohere\", \"tokens\": 1441, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"df90e0e8-1bf7-4ec6-a734-0126b7db9dc9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cohere\", \"title\": \"Cohere\", \"tokens\": 1441, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"da67d8ba0be098800d4628f91e861c5e781efc06f9055313167a33636937a092\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"95bfcca4-2431-496d-8d08-b23f7bf08e62\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cohere\", \"title\": \"Cohere\", \"tokens\": 1441, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"eb535fb3aed75d6d56e0cccdcf6f77fd81f03e51a749ad55695f82a333f5945b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3c525b07-2ac2-4ff9-abbc-c14f06d3c76c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"15e6216aa2d9b487337d0d692fd8cf2bad2093eadcd639f533de92c2e2180395\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2429, \"end_char_idx\": 5500, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "df90e0e8-1bf7-4ec6-a734-0126b7db9dc9", "doc_id": "df90e0e8-1bf7-4ec6-a734-0126b7db9dc9", "ref_doc_id": "df90e0e8-1bf7-4ec6-a734-0126b7db9dc9", "chroma:document": "</Tip>\nThe model and tokenizer can be loaded via:\n\n```python\n# pip install transformers\nfrom transformers import AutoTokenizer, AutoModelForCausalLM\n\nmodel_id = \"CohereForAI/c4ai-command-r-v01\"\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nmodel = AutoModelForCausalLM.from_pretrained(model_id)\n\n# Format message with the command-r chat template\nmessages = [{\"role\": \"user\", \"content\": \"Hello, how are you?\"}]\ninput_ids = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors=\"pt\")\n## <BOS_TOKEN><|START_OF_TURN_TOKEN|><|USER_TOKEN|>Hello, how are you?<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>\n\ngen_tokens = model.generate(\n    input_ids, \n    max_new_tokens=100, \n    do_sample=True, \n    temperature=0.3,\n    )\n\ngen_text = tokenizer.decode(gen_tokens[0])\nprint(gen_text)\n```\n\n- When using Flash Attention 2 via `attn_implementation=\"flash_attention_2\"`, don't pass `torch_dtype` to the `from_pretrained` class method and use Automatic Mixed-Precision training. When using `Trainer`, it is simply specifying either `fp16` or `bf16` to `True`. Otherwise, make sure you are using `torch.autocast`. This is required because the Flash Attention only support `fp16` and `bf16` data type.\n\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Command-R. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n\n<PipelineTag pipeline=\"text-generation\"/>\n\nLoading FP16 model\n```python\n# pip install transformers\nfrom transformers import AutoTokenizer, AutoModelForCausalLM\n\nmodel_id = \"CohereForAI/c4ai-command-r-v01\"\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nmodel = AutoModelForCausalLM.from_pretrained(model_id)\n\n# Format message with the command-r chat template\nmessages = [{\"role\": \"user\", \"content\": \"Hello, how are you?\"}]\ninput_ids = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors=\"pt\")\n## <BOS_TOKEN><|START_OF_TURN_TOKEN|><|USER_TOKEN|>Hello, how are you?<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>\n\ngen_tokens = model.generate(\n    input_ids, \n    max_new_tokens=100, \n    do_sample=True, \n    temperature=0.3,\n    )\n\ngen_text = tokenizer.decode(gen_tokens[0])\nprint(gen_text)\n```\n\nLoading bitsnbytes 4bit quantized model\n```python\n# pip install transformers bitsandbytes accelerate\nfrom transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig\n\nbnb_config = BitsAndBytesConfig(load_in_4bit=True)\n\nmodel_id = \"CohereForAI/c4ai-command-r-v01\"\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nmodel = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config)\n\ngen_tokens = model.generate(\n    input_ids, \n    max_new_tokens=100, \n    do_sample=True, \n    temperature=0.3,\n    )\n\ngen_text = tokenizer.decode(gen_tokens[0])\nprint(gen_text)\n```"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 3c525b07-2ac2-4ff9-abbc-c14f06d3c76c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/cohere", "title": "Cohere", "tokens": 1441, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3c525b07-2ac2-4ff9-abbc-c14f06d3c76c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cohere\", \"title\": \"Cohere\", \"tokens\": 1441, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"df90e0e8-1bf7-4ec6-a734-0126b7db9dc9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cohere\", \"title\": \"Cohere\", \"tokens\": 1441, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"da67d8ba0be098800d4628f91e861c5e781efc06f9055313167a33636937a092\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bfedeb38-c993-425b-b479-b306073511fd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cohere\", \"title\": \"Cohere\", \"tokens\": 1441, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"498cda6a084f69e1da1638061828fc8a4e1b9a4e55348fa088eac23de38e0661\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4041, \"end_char_idx\": 5886, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "df90e0e8-1bf7-4ec6-a734-0126b7db9dc9", "doc_id": "df90e0e8-1bf7-4ec6-a734-0126b7db9dc9", "ref_doc_id": "df90e0e8-1bf7-4ec6-a734-0126b7db9dc9", "chroma:document": "<PipelineTag pipeline=\"text-generation\"/>\n\nLoading FP16 model\n```python\n# pip install transformers\nfrom transformers import AutoTokenizer, AutoModelForCausalLM\n\nmodel_id = \"CohereForAI/c4ai-command-r-v01\"\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nmodel = AutoModelForCausalLM.from_pretrained(model_id)\n\n# Format message with the command-r chat template\nmessages = [{\"role\": \"user\", \"content\": \"Hello, how are you?\"}]\ninput_ids = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors=\"pt\")\n## <BOS_TOKEN><|START_OF_TURN_TOKEN|><|USER_TOKEN|>Hello, how are you?<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>\n\ngen_tokens = model.generate(\n    input_ids, \n    max_new_tokens=100, \n    do_sample=True, \n    temperature=0.3,\n    )\n\ngen_text = tokenizer.decode(gen_tokens[0])\nprint(gen_text)\n```\n\nLoading bitsnbytes 4bit quantized model\n```python\n# pip install transformers bitsandbytes accelerate\nfrom transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig\n\nbnb_config = BitsAndBytesConfig(load_in_4bit=True)\n\nmodel_id = \"CohereForAI/c4ai-command-r-v01\"\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nmodel = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config)\n\ngen_tokens = model.generate(\n    input_ids, \n    max_new_tokens=100, \n    do_sample=True, \n    temperature=0.3,\n    )\n\ngen_text = tokenizer.decode(gen_tokens[0])\nprint(gen_text)\n```\n\n\n## CohereConfig\n\n[[autodoc]] CohereConfig\n\n## CohereTokenizerFast\n\n[[autodoc]] CohereTokenizerFast\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - update_post_processor\n    - save_vocabulary\n\n## CohereModel\n\n[[autodoc]] CohereModel\n    - forward\n\n\n## CohereForCausalLM\n\n[[autodoc]] CohereForCausalLM\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 8554c9ac-afb4-4827-ac13-30e7c4783b69
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/prophetnet", "title": "ProphetNet", "tokens": 993, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8554c9ac-afb4-4827-ac13-30e7c4783b69\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/prophetnet\", \"title\": \"ProphetNet\", \"tokens\": 993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cd653b22-1eb3-49e4-bd2a-f53428d8b0b8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/prophetnet\", \"title\": \"ProphetNet\", \"tokens\": 993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4118920ee1620615aa9e579b9f3556d654e737594180f6fab4d25eedbdef5f24\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"52c92ed7-0bb2-4d94-80cb-e495a6a6617e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d3d9af46590fc64f156313f524aa4e75ad480d6763b22a922cfd7b10ba3e2589\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2353, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cd653b22-1eb3-49e4-bd2a-f53428d8b0b8", "doc_id": "cd653b22-1eb3-49e4-bd2a-f53428d8b0b8", "ref_doc_id": "cd653b22-1eb3-49e4-bd2a-f53428d8b0b8", "chroma:document": "# ProphetNet\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=prophetnet\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-prophetnet-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/prophetnet-large-uncased\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nThe ProphetNet model was proposed in [ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training,](https://arxiv.org/abs/2001.04063) by Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei\nZhang, Ming Zhou on 13 Jan, 2020.\n\nProphetNet is an encoder-decoder model and can predict n-future tokens for \"ngram\" language modeling instead of just\nthe next token.\n\nThe abstract from the paper is the following:\n\n*In this paper, we present a new sequence-to-sequence pretraining model called ProphetNet, which introduces a novel\nself-supervised objective named future n-gram prediction and the proposed n-stream self-attention mechanism. Instead of\nthe optimization of one-step ahead prediction in traditional sequence-to-sequence model, the ProphetNet is optimized by\nn-step ahead prediction which predicts the next n tokens simultaneously based on previous context tokens at each time\nstep. The future n-gram prediction explicitly encourages the model to plan for the future tokens and prevent\noverfitting on strong local correlations. We pre-train ProphetNet using a base scale dataset (16GB) and a large scale\ndataset (160GB) respectively. Then we conduct experiments on CNN/DailyMail, Gigaword, and SQuAD 1.1 benchmarks for\nabstractive summarization and question generation tasks. Experimental results show that ProphetNet achieves new\nstate-of-the-art results on all these datasets compared to the models using the same scale pretraining corpus.*\n\nThe Authors' code can be found [here](https://github.com/microsoft/ProphetNet).\n\n## Usage tips\n\n- ProphetNet is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than\n  the left.\n- The model architecture is based on the original Transformer, but replaces the \u201cstandard\u201d self-attention mechanism in the decoder by a main self-attention mechanism and a self and n-stream (predict) self-attention mechanism."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 52c92ed7-0bb2-4d94-80cb-e495a6a6617e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/prophetnet", "title": "ProphetNet", "tokens": 993, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"52c92ed7-0bb2-4d94-80cb-e495a6a6617e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/prophetnet\", \"title\": \"ProphetNet\", \"tokens\": 993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cd653b22-1eb3-49e4-bd2a-f53428d8b0b8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/prophetnet\", \"title\": \"ProphetNet\", \"tokens\": 993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4118920ee1620615aa9e579b9f3556d654e737594180f6fab4d25eedbdef5f24\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8554c9ac-afb4-4827-ac13-30e7c4783b69\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/prophetnet\", \"title\": \"ProphetNet\", \"tokens\": 993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8735d371389e8541bc23aa2d7d0270007426ea0456a63706c83635e09ab110f8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 673, \"end_char_idx\": 3365, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cd653b22-1eb3-49e4-bd2a-f53428d8b0b8", "doc_id": "cd653b22-1eb3-49e4-bd2a-f53428d8b0b8", "ref_doc_id": "cd653b22-1eb3-49e4-bd2a-f53428d8b0b8", "chroma:document": "ProphetNet is an encoder-decoder model and can predict n-future tokens for \"ngram\" language modeling instead of just\nthe next token.\n\nThe abstract from the paper is the following:\n\n*In this paper, we present a new sequence-to-sequence pretraining model called ProphetNet, which introduces a novel\nself-supervised objective named future n-gram prediction and the proposed n-stream self-attention mechanism. Instead of\nthe optimization of one-step ahead prediction in traditional sequence-to-sequence model, the ProphetNet is optimized by\nn-step ahead prediction which predicts the next n tokens simultaneously based on previous context tokens at each time\nstep. The future n-gram prediction explicitly encourages the model to plan for the future tokens and prevent\noverfitting on strong local correlations. We pre-train ProphetNet using a base scale dataset (16GB) and a large scale\ndataset (160GB) respectively. Then we conduct experiments on CNN/DailyMail, Gigaword, and SQuAD 1.1 benchmarks for\nabstractive summarization and question generation tasks. Experimental results show that ProphetNet achieves new\nstate-of-the-art results on all these datasets compared to the models using the same scale pretraining corpus.*\n\nThe Authors' code can be found [here](https://github.com/microsoft/ProphetNet).\n\n## Usage tips\n\n- ProphetNet is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than\n  the left.\n- The model architecture is based on the original Transformer, but replaces the \u201cstandard\u201d self-attention mechanism in the decoder by a main self-attention mechanism and a self and n-stream (predict) self-attention mechanism.\n\n## Resources\n\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n## ProphetNetConfig\n\n[[autodoc]] ProphetNetConfig\n\n## ProphetNetTokenizer\n\n[[autodoc]] ProphetNetTokenizer\n\n## ProphetNet specific outputs\n\n[[autodoc]] models.prophetnet.modeling_prophetnet.ProphetNetSeq2SeqLMOutput\n\n[[autodoc]] models.prophetnet.modeling_prophetnet.ProphetNetSeq2SeqModelOutput\n\n[[autodoc]] models.prophetnet.modeling_prophetnet.ProphetNetDecoderModelOutput\n\n[[autodoc]] models.prophetnet.modeling_prophetnet.ProphetNetDecoderLMOutput\n\n## ProphetNetModel\n\n[[autodoc]] ProphetNetModel\n    - forward\n\n## ProphetNetEncoder\n\n[[autodoc]] ProphetNetEncoder\n    - forward\n\n## ProphetNetDecoder\n\n[[autodoc]] ProphetNetDecoder\n    - forward\n\n## ProphetNetForConditionalGeneration\n\n[[autodoc]] ProphetNetForConditionalGeneration\n    - forward\n\n## ProphetNetForCausalLM\n\n[[autodoc]] ProphetNetForCausalLM\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 39123431-aa9c-452f-bbd4-f35eb4e62bc7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"39123431-aa9c-452f-bbd4-f35eb4e62bc7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d030793b-21c8-4359-9a34-bfecb53a094c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ef4f7470b3740873a190873eabc8e70c334e15e388e086cf29e882cb993a6710\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b9a07ddf-4cec-41d4-8eef-0c808fcb7da8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4ea3143e15481c06ef3dbe0ea85abe948696006a18018d72b934560e364f4f3e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2773, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d030793b-21c8-4359-9a34-bfecb53a094c", "doc_id": "d030793b-21c8-4359-9a34-bfecb53a094c", "ref_doc_id": "d030793b-21c8-4359-9a34-bfecb53a094c", "chroma:document": "# GPT-NeoX\n\n## Overview\n\nWe introduce GPT-NeoX-20B, a 20 billion parameter autoregressive language model trained on the Pile, whose weights will\nbe made freely and openly available to the public through a permissive license. It is, to the best of our knowledge,\nthe largest dense autoregressive model that has publicly available weights at the time of submission. In this work,\nwe describe GPT-NeoX-20B's architecture and training and evaluate its performance on a range of language-understanding,\nmathematics, and knowledge-based tasks. We find that GPT-NeoX-20B is a particularly powerful few-shot reasoner and\ngains far more in performance when evaluated five-shot than similarly sized GPT-3 and FairSeq models. We open-source\nthe training and evaluation code, as well as the model weights, at [https://github.com/EleutherAI/gpt-neox](https://github.com/EleutherAI/gpt-neox).\n\nDevelopment of the model was led by Sid Black, Stella Biderman and Eric Hallahan, and the model was trained with\ngenerous the support of [CoreWeave](https://www.coreweave.com/).\n\nGPT-NeoX-20B was trained with fp16, thus it is recommended to initialize the model as follows:\n\n```python\nmodel = GPTNeoXForCausalLM.from_pretrained(\"EleutherAI/gpt-neox-20b\").half().cuda()\n```\n\nGPT-NeoX-20B also has a different tokenizer from the one used in GPT-J-6B and GPT-Neo. The new tokenizer allocates\nadditional tokens to whitespace characters, making the model more suitable for certain tasks like code generation.\n\n## Usage example\n\nThe `generate()` method can be used to generate text using GPT Neo model.\n\n```python\n>>> from transformers import GPTNeoXForCausalLM, GPTNeoXTokenizerFast\n\n>>> model = GPTNeoXForCausalLM.from_pretrained(\"EleutherAI/gpt-neox-20b\")\n>>> tokenizer = GPTNeoXTokenizerFast.from_pretrained(\"EleutherAI/gpt-neox-20b\")\n\n>>> prompt = \"GPTNeoX20B is a 20B-parameter autoregressive Transformer model developed by EleutherAI.\"\n\n>>> input_ids = tokenizer(prompt, return_tensors=\"pt\").input_ids\n\n>>> gen_tokens = model.generate(\n...     input_ids,\n...     do_sample=True,\n...     temperature=0.9,\n...     max_length=100,\n... )\n>>> gen_text = tokenizer.batch_decode(gen_tokens)[0]\n```\n\n## Using Flash Attention 2\n\nFlash Attention 2 is an faster, optimized version of the model.\n\n### Installation \n\nFirst, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features). If your hardware is not compatible with Flash Attention 2, you can still benefit from attention kernel optimisations through Better Transformer support covered [above](https://huggingface.co/docs/transformers/main/en/model_doc/bark#using-better-transformer)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: b9a07ddf-4cec-41d4-8eef-0c808fcb7da8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b9a07ddf-4cec-41d4-8eef-0c808fcb7da8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d030793b-21c8-4359-9a34-bfecb53a094c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ef4f7470b3740873a190873eabc8e70c334e15e388e086cf29e882cb993a6710\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"39123431-aa9c-452f-bbd4-f35eb4e62bc7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b90ead5f109e5c7785387d1cb5fe29b2463fb94c68bf115544fe5192c8cc1749\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2dc1540b-bcdf-44b1-a552-83a40b3ea6cb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"66d0a5e20132381cda993924da98df5706c9cfe258c1253a834c9698cb749e98\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1341, \"end_char_idx\": 4350, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d030793b-21c8-4359-9a34-bfecb53a094c", "doc_id": "d030793b-21c8-4359-9a34-bfecb53a094c", "ref_doc_id": "d030793b-21c8-4359-9a34-bfecb53a094c", "chroma:document": "The new tokenizer allocates\nadditional tokens to whitespace characters, making the model more suitable for certain tasks like code generation.\n\n## Usage example\n\nThe `generate()` method can be used to generate text using GPT Neo model.\n\n```python\n>>> from transformers import GPTNeoXForCausalLM, GPTNeoXTokenizerFast\n\n>>> model = GPTNeoXForCausalLM.from_pretrained(\"EleutherAI/gpt-neox-20b\")\n>>> tokenizer = GPTNeoXTokenizerFast.from_pretrained(\"EleutherAI/gpt-neox-20b\")\n\n>>> prompt = \"GPTNeoX20B is a 20B-parameter autoregressive Transformer model developed by EleutherAI.\"\n\n>>> input_ids = tokenizer(prompt, return_tensors=\"pt\").input_ids\n\n>>> gen_tokens = model.generate(\n...     input_ids,\n...     do_sample=True,\n...     temperature=0.9,\n...     max_length=100,\n... )\n>>> gen_text = tokenizer.batch_decode(gen_tokens)[0]\n```\n\n## Using Flash Attention 2\n\nFlash Attention 2 is an faster, optimized version of the model.\n\n### Installation \n\nFirst, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features). If your hardware is not compatible with Flash Attention 2, you can still benefit from attention kernel optimisations through Better Transformer support covered [above](https://huggingface.co/docs/transformers/main/en/model_doc/bark#using-better-transformer).\n\nNext, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\n### Usage\n\nTo load a model using Flash Attention 2, we can pass the argument `attn_implementation=\"flash_attention_2\"` to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). We'll also load the model in half-precision (e.g. `torch.float16`), since it results in almost no degradation to audio quality but significantly lower memory usage and faster inference:\n\n```python\n>>> from transformers import GPTNeoXForCausalLM, GPTNeoXTokenizerFast\n\nmodel = GPTNeoXForCausalLM.from_pretrained(\"EleutherAI/gpt-neox-20b\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\").to(device)\n...\n```\n\n\n### Expected speedups\n\nBelow is an expected speedup diagram that compares pure inference time between the native implementation in transformers using `stockmark/gpt-neox-japanese-1.4b` checkpoint and the Flash Attention 2 version of the model using a sequence length of 2048.\n\n<div style=\"text-align: center\">\n<img src=\"https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/gpt-neox-1.8b-speedup.jpg\">\n</div>\n\n\n## Using Scaled Dot Product Attention (SDPA)\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function\nencompasses several implementations that can be applied depending on the inputs and the hardware in use."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 2dc1540b-bcdf-44b1-a552-83a40b3ea6cb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2dc1540b-bcdf-44b1-a552-83a40b3ea6cb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d030793b-21c8-4359-9a34-bfecb53a094c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ef4f7470b3740873a190873eabc8e70c334e15e388e086cf29e882cb993a6710\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b9a07ddf-4cec-41d4-8eef-0c808fcb7da8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b155bbb65ff0c78e13eb7faa7264a8694494824b31b2a6aff81d530dbb16eb0b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8bd939b3-fb69-4a4c-8a3c-e0e858540b63\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"aed132cddf7d762190785d510ade4241d0f314cf5e4b6edfeac5f95304ace185\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2775, \"end_char_idx\": 5792, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d030793b-21c8-4359-9a34-bfecb53a094c", "doc_id": "d030793b-21c8-4359-9a34-bfecb53a094c", "ref_doc_id": "d030793b-21c8-4359-9a34-bfecb53a094c", "chroma:document": "Next, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\n### Usage\n\nTo load a model using Flash Attention 2, we can pass the argument `attn_implementation=\"flash_attention_2\"` to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). We'll also load the model in half-precision (e.g. `torch.float16`), since it results in almost no degradation to audio quality but significantly lower memory usage and faster inference:\n\n```python\n>>> from transformers import GPTNeoXForCausalLM, GPTNeoXTokenizerFast\n\nmodel = GPTNeoXForCausalLM.from_pretrained(\"EleutherAI/gpt-neox-20b\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\").to(device)\n...\n```\n\n\n### Expected speedups\n\nBelow is an expected speedup diagram that compares pure inference time between the native implementation in transformers using `stockmark/gpt-neox-japanese-1.4b` checkpoint and the Flash Attention 2 version of the model using a sequence length of 2048.\n\n<div style=\"text-align: center\">\n<img src=\"https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/gpt-neox-1.8b-speedup.jpg\">\n</div>\n\n\n## Using Scaled Dot Product Attention (SDPA)\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function\nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the\n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html)\nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set\n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```python\nfrom transformers import GPTNeoXForCausalLM\nmodel = GPTNeoXForCausalLM.from_pretrained(\"EleutherAI/gpt-neox-20b\", torch_dtype=torch.float16, attn_implementation=\"sdpa\")\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (rtx3080ti-16GB, PyTorch 2.2.1, OS Ubuntu 22.04) using `float16` with\n[pythia-410m-deduped](https://huggingface.co/EleutherAI/pythia-410m-deduped), we saw the\nfollowing speedups during training and inference.\n\n### Training\n| Batch size |    Seq len | Time per batch (Eager - s) |    Time per batch (SDPA - s) | Speedup (%) | Eager peak mem (MB) | SDPA peak mem (MB) |    Mem saving (%) |\n|-----------:|-----------:|---------------------------:|-----------------------------:|------------:|--------------------:|-------------------:|------------------:|\n|          1 |        128 |                      0.024 |                        0."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 8bd939b3-fb69-4a4c-8a3c-e0e858540b63
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8bd939b3-fb69-4a4c-8a3c-e0e858540b63\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d030793b-21c8-4359-9a34-bfecb53a094c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ef4f7470b3740873a190873eabc8e70c334e15e388e086cf29e882cb993a6710\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2dc1540b-bcdf-44b1-a552-83a40b3ea6cb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9e8b65282f7ae19de09b99959d5cb91e1b065b18df33905c5223620ba28fac90\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"916869d3-0949-4a34-a6f2-342b17324132\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d5aa62e8f963b5cf99315654a67f2a32fef652371576c01c0b9b70b0644cdd45\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4351, \"end_char_idx\": 7126, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d030793b-21c8-4359-9a34-bfecb53a094c", "doc_id": "d030793b-21c8-4359-9a34-bfecb53a094c", "ref_doc_id": "d030793b-21c8-4359-9a34-bfecb53a094c", "chroma:document": "See the\n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html)\nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set\n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```python\nfrom transformers import GPTNeoXForCausalLM\nmodel = GPTNeoXForCausalLM.from_pretrained(\"EleutherAI/gpt-neox-20b\", torch_dtype=torch.float16, attn_implementation=\"sdpa\")\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (rtx3080ti-16GB, PyTorch 2.2.1, OS Ubuntu 22.04) using `float16` with\n[pythia-410m-deduped](https://huggingface.co/EleutherAI/pythia-410m-deduped), we saw the\nfollowing speedups during training and inference.\n\n### Training\n| Batch size |    Seq len | Time per batch (Eager - s) |    Time per batch (SDPA - s) | Speedup (%) | Eager peak mem (MB) | SDPA peak mem (MB) |    Mem saving (%) |\n|-----------:|-----------:|---------------------------:|-----------------------------:|------------:|--------------------:|-------------------:|------------------:|\n|          1 |        128 |                      0.024 |                        0.019 |      28.945 |             1789.95 |            1789.95 |                 0 |\n|          1 |        256 |                      0.039 |                        0.031 |       23.18 |             1845.83 |            1844.84 |             0.053 |\n|          1 |        512 |                       0.08 |                        0.055 |      45.524 |             2278.38 |            1953.76 |            16.615 |\n|          1 |       1024 |                       0.19 |                        0.102 |      86.777 |             4772.36 |            2408.35 |            98.159 |\n|          1 |       2048 |                      0.565 |                        0.204 |     177.098 |             13484.1 |            3882.01 |           247.348 |\n|          2 |        128 |                      0.037 |                        0.032 |      15.121 |             1843.86 |            1844.78 |             -0.05 |\n|          2 |        256 |                      0.067 |                        0.055 |      21.706 |             1999.72 |            1951.67 |             2.462 |\n|          2 |        512 |                      0.144 |                        0.096 |      50.046 |             3613.16 |            2406.77 |            50.125 |\n|          2 |       1024 |                      0.366 |                        0.193 |      89."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 916869d3-0949-4a34-a6f2-342b17324132
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"916869d3-0949-4a34-a6f2-342b17324132\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d030793b-21c8-4359-9a34-bfecb53a094c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ef4f7470b3740873a190873eabc8e70c334e15e388e086cf29e882cb993a6710\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8bd939b3-fb69-4a4c-8a3c-e0e858540b63\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ab2883033bd8c81645edb53a934121abf9146296bdcda21ca9123b45ea9725f8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9394e4d2-4bef-4b01-9a1f-de75a6ada381\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d4a7402e1aa1033db1de0865a8b3e08b68165389a06bd26544496bc1eead35b6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5761, \"end_char_idx\": 8642, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d030793b-21c8-4359-9a34-bfecb53a094c", "doc_id": "d030793b-21c8-4359-9a34-bfecb53a094c", "ref_doc_id": "d030793b-21c8-4359-9a34-bfecb53a094c", "chroma:document": "024 |                        0.019 |      28.945 |             1789.95 |            1789.95 |                 0 |\n|          1 |        256 |                      0.039 |                        0.031 |       23.18 |             1845.83 |            1844.84 |             0.053 |\n|          1 |        512 |                       0.08 |                        0.055 |      45.524 |             2278.38 |            1953.76 |            16.615 |\n|          1 |       1024 |                       0.19 |                        0.102 |      86.777 |             4772.36 |            2408.35 |            98.159 |\n|          1 |       2048 |                      0.565 |                        0.204 |     177.098 |             13484.1 |            3882.01 |           247.348 |\n|          2 |        128 |                      0.037 |                        0.032 |      15.121 |             1843.86 |            1844.78 |             -0.05 |\n|          2 |        256 |                      0.067 |                        0.055 |      21.706 |             1999.72 |            1951.67 |             2.462 |\n|          2 |        512 |                      0.144 |                        0.096 |      50.046 |             3613.16 |            2406.77 |            50.125 |\n|          2 |       1024 |                      0.366 |                        0.193 |      89.666 |             8707.55 |            3878.86 |           124.487 |\n|          2 |       2048 |                        OOM |                        0.379 |           / |                 OOM |            6825.13 | SDPA does not OOM |\n|          4 |        128 |                       0.06 |                        0.054 |      11.539 |              1947.6 |            1952.06 |            -0.228 |\n|          4 |        256 |                      0.119 |                        0.093 |      28.072 |             3008.39 |            2405.99 |            25.038 |\n|          4 |        512 |                      0.275 |                        0.187 |      47.145 |             6290.58 |            3877.29 |            62.242 |\n|          4 |       1024 |                        OOM |                         0.36 |           / |                 OOM |            6821.98 | SDPA does not OOM |\n|          4 |       2048 |                        OOM |                        0.731 |           / |                 OOM |            12705.1 | SDPA does not OOM |\n\n### Inference\n|    Batch size |      Seq len |    Per token latency Eager (ms) |    Per token latency SDPA (ms) |    Speedup (%) |    Mem Eager (MB) |   Mem SDPA (MB) |    Mem saved (%) |\n|--------------:|-------------:|--------------------------------:|-------------------------------:|---------------:|------------------:|----------------:|-----------------:|\n|             1 |          128 |                           6.569 |                          5."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 9394e4d2-4bef-4b01-9a1f-de75a6ada381
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9394e4d2-4bef-4b01-9a1f-de75a6ada381\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d030793b-21c8-4359-9a34-bfecb53a094c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ef4f7470b3740873a190873eabc8e70c334e15e388e086cf29e882cb993a6710\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"916869d3-0949-4a34-a6f2-342b17324132\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fdad865537297dff7afd35d5f7f9a6795ba18024f4c89d91d839de2f41bd795d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7d4353bb-9c4a-4815-9558-de02807c526d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"25c80eb907589a49695a45bc851b5bb9e2419288568874be07af4329f1542df3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7112, \"end_char_idx\": 10051, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d030793b-21c8-4359-9a34-bfecb53a094c", "doc_id": "d030793b-21c8-4359-9a34-bfecb53a094c", "ref_doc_id": "d030793b-21c8-4359-9a34-bfecb53a094c", "chroma:document": "193 |      89.666 |             8707.55 |            3878.86 |           124.487 |\n|          2 |       2048 |                        OOM |                        0.379 |           / |                 OOM |            6825.13 | SDPA does not OOM |\n|          4 |        128 |                       0.06 |                        0.054 |      11.539 |              1947.6 |            1952.06 |            -0.228 |\n|          4 |        256 |                      0.119 |                        0.093 |      28.072 |             3008.39 |            2405.99 |            25.038 |\n|          4 |        512 |                      0.275 |                        0.187 |      47.145 |             6290.58 |            3877.29 |            62.242 |\n|          4 |       1024 |                        OOM |                         0.36 |           / |                 OOM |            6821.98 | SDPA does not OOM |\n|          4 |       2048 |                        OOM |                        0.731 |           / |                 OOM |            12705.1 | SDPA does not OOM |\n\n### Inference\n|    Batch size |      Seq len |    Per token latency Eager (ms) |    Per token latency SDPA (ms) |    Speedup (%) |    Mem Eager (MB) |   Mem SDPA (MB) |    Mem saved (%) |\n|--------------:|-------------:|--------------------------------:|-------------------------------:|---------------:|------------------:|----------------:|-----------------:|\n|             1 |          128 |                           6.569 |                          5.858 |          12.14 |           974.831 |         974.826 |                0 |\n|             1 |          256 |                           7.009 |                          5.863 |         19.542 |           1029.01 |         1028.08 |             0.09 |\n|             1 |          512 |                           7.157 |                          5.965 |         19.983 |           1137.54 |         1137.52 |            0.001 |\n|             1 |         1024 |                           7.523 |                          6.506 |         15.637 |            1329.3 |         1329.26 |            0.003 |\n|             1 |         2048 |                           9.271 |                          9.205 |          0.713 |           1752.47 |         1734.51 |            1.036 |\n|             2 |          128 |                           7.239 |                          5.959 |         21.493 |            1044.8 |         1028.37 |            1.597 |\n|             2 |          256 |                           7.228 |                          6.036 |         19.757 |           1167.32 |         1137.73 |            2.601 |\n|             2 |          512 |                           7.538 |                          6.693 |         12.628 |           1352.93 |         1329.55 |            1.758 |\n|             2 |         1024 |                           8.916 |                          8.632 |          3."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 7d4353bb-9c4a-4815-9558-de02807c526d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7d4353bb-9c4a-4815-9558-de02807c526d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d030793b-21c8-4359-9a34-bfecb53a094c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ef4f7470b3740873a190873eabc8e70c334e15e388e086cf29e882cb993a6710\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9394e4d2-4bef-4b01-9a1f-de75a6ada381\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"eefc4673723e82d8d23e09c6652196e07cb4db2642c556610a9536ce6531b9b7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4b08997d-e6a7-4162-994b-4adff308cc8b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"70c6c0b621f7905838cfa95f82bb2e260928033966bce8fe6c72fcfac3e9a074\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8609, \"end_char_idx\": 11157, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d030793b-21c8-4359-9a34-bfecb53a094c", "doc_id": "d030793b-21c8-4359-9a34-bfecb53a094c", "ref_doc_id": "d030793b-21c8-4359-9a34-bfecb53a094c", "chroma:document": "569 |                          5.858 |          12.14 |           974.831 |         974.826 |                0 |\n|             1 |          256 |                           7.009 |                          5.863 |         19.542 |           1029.01 |         1028.08 |             0.09 |\n|             1 |          512 |                           7.157 |                          5.965 |         19.983 |           1137.54 |         1137.52 |            0.001 |\n|             1 |         1024 |                           7.523 |                          6.506 |         15.637 |            1329.3 |         1329.26 |            0.003 |\n|             1 |         2048 |                           9.271 |                          9.205 |          0.713 |           1752.47 |         1734.51 |            1.036 |\n|             2 |          128 |                           7.239 |                          5.959 |         21.493 |            1044.8 |         1028.37 |            1.597 |\n|             2 |          256 |                           7.228 |                          6.036 |         19.757 |           1167.32 |         1137.73 |            2.601 |\n|             2 |          512 |                           7.538 |                          6.693 |         12.628 |           1352.93 |         1329.55 |            1.758 |\n|             2 |         1024 |                           8.916 |                          8.632 |          3.291 |           1752.56 |         1734.62 |            1.034 |\n|             2 |         2048 |                          12.628 |                         12.606 |          0.181 |           2558.72 |          2545.8 |            0.508 |\n|             4 |          128 |                           7.278 |                          6.046 |         20.373 |           1168.41 |         1137.79 |            2.691 |\n|             4 |          256 |                           7.614 |                          6.588 |         15.574 |            1353.1 |         1329.79 |            1.753 |\n|             4 |          512 |                           8.798 |                          8.144 |          8.028 |           1752.76 |         1734.85 |            1.032 |\n|             4 |         1024 |                          11.765 |                         11.303 |           4.09 |           2558.96 |         2546.04 |            0.508 |\n|             4 |         2048 |                          19.568 |                         17.735 |          10.33 |            4175.5 |         4165.26 |            0.246 |"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 4b08997d-e6a7-4162-994b-4adff308cc8b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4b08997d-e6a7-4162-994b-4adff308cc8b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d030793b-21c8-4359-9a34-bfecb53a094c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ef4f7470b3740873a190873eabc8e70c334e15e388e086cf29e882cb993a6710\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7d4353bb-9c4a-4815-9558-de02807c526d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2efa2a2f0356e24c7cc25866979bf30ef4d84774ada7456a29311ce5f7a053c5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9724, \"end_char_idx\": 11746, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d030793b-21c8-4359-9a34-bfecb53a094c", "doc_id": "d030793b-21c8-4359-9a34-bfecb53a094c", "ref_doc_id": "d030793b-21c8-4359-9a34-bfecb53a094c", "chroma:document": "32 |         1137.73 |            2.601 |\n|             2 |          512 |                           7.538 |                          6.693 |         12.628 |           1352.93 |         1329.55 |            1.758 |\n|             2 |         1024 |                           8.916 |                          8.632 |          3.291 |           1752.56 |         1734.62 |            1.034 |\n|             2 |         2048 |                          12.628 |                         12.606 |          0.181 |           2558.72 |          2545.8 |            0.508 |\n|             4 |          128 |                           7.278 |                          6.046 |         20.373 |           1168.41 |         1137.79 |            2.691 |\n|             4 |          256 |                           7.614 |                          6.588 |         15.574 |            1353.1 |         1329.79 |            1.753 |\n|             4 |          512 |                           8.798 |                          8.144 |          8.028 |           1752.76 |         1734.85 |            1.032 |\n|             4 |         1024 |                          11.765 |                         11.303 |           4.09 |           2558.96 |         2546.04 |            0.508 |\n|             4 |         2048 |                          19.568 |                         17.735 |          10.33 |            4175.5 |         4165.26 |            0.246 |\n\n\n## Resources\n\n- [Causal language modeling task guide](../tasks/language_modeling)\n\n## GPTNeoXConfig\n\n[[autodoc]] GPTNeoXConfig\n\n## GPTNeoXTokenizerFast\n\n[[autodoc]] GPTNeoXTokenizerFast\n\n## GPTNeoXModel\n\n[[autodoc]] GPTNeoXModel\n    - forward\n\n## GPTNeoXForCausalLM\n\n[[autodoc]] GPTNeoXForCausalLM\n    - forward\n\n## GPTNeoXForQuestionAnswering\n\n[[autodoc]] GPTNeoXForQuestionAnswering\n    - forward\n\n## GPTNeoXForSequenceClassification\n\n[[autodoc]] GPTNeoXForSequenceClassification\n    - forward\n\n## GPTNeoXForTokenClassification\n\n[[autodoc]] GPTNeoXForTokenClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0d755725-c76b-4170-8609-c4ec210f71dc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/fsmt", "title": "FSMT", "tokens": 619, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0d755725-c76b-4170-8609-c4ec210f71dc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fsmt\", \"title\": \"FSMT\", \"tokens\": 619, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"290ee3ec-ffdb-4b18-bab7-e342d6b9d809\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fsmt\", \"title\": \"FSMT\", \"tokens\": 619, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9f948210ce433a04081d7ce17f42bd09cfbf40fd8cff98ed1a103daf93737882\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1964, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "290ee3ec-ffdb-4b18-bab7-e342d6b9d809", "doc_id": "290ee3ec-ffdb-4b18-bab7-e342d6b9d809", "ref_doc_id": "290ee3ec-ffdb-4b18-bab7-e342d6b9d809", "chroma:document": "# FSMT\n\n## Overview\n\nFSMT (FairSeq MachineTranslation) models were introduced in [Facebook FAIR's WMT19 News Translation Task Submission](https://arxiv.org/abs/1907.06616) by Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, Sergey Edunov.\n\nThe abstract of the paper is the following:\n\n*This paper describes Facebook FAIR's submission to the WMT19 shared news translation task. We participate in two\nlanguage pairs and four language directions, English <-> German and English <-> Russian. Following our submission from\nlast year, our baseline systems are large BPE-based transformer models trained with the Fairseq sequence modeling\ntoolkit which rely on sampled back-translations. This year we experiment with different bitext data filtering schemes,\nas well as with adding filtered back-translated data. We also ensemble and fine-tune our models on domain-specific\ndata, then decode using noisy channel model reranking. Our submissions are ranked first in all four directions of the\nhuman evaluation campaign. On En->De, our system significantly outperforms other systems as well as human translations.\nThis system improves upon our WMT'18 submission by 4.5 BLEU points.*\n\nThis model was contributed by [stas](https://huggingface.co/stas). The original code can be found\n[here](https://github.com/pytorch/fairseq/tree/master/examples/wmt19).\n\n## Implementation Notes\n\n- FSMT uses source and target vocabulary pairs that aren't combined into one. It doesn't share embeddings tokens\n  either. Its tokenizer is very similar to [`XLMTokenizer`] and the main model is derived from\n  [`BartModel`].\n\n\n## FSMTConfig\n\n[[autodoc]] FSMTConfig\n\n## FSMTTokenizer\n\n[[autodoc]] FSMTTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## FSMTModel\n\n[[autodoc]] FSMTModel\n    - forward\n\n## FSMTForConditionalGeneration\n\n[[autodoc]] FSMTForConditionalGeneration\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 25cf8630-3dbf-4b2d-93cb-50b1bc6b45b4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mt5", "title": "mT5", "tokens": 1157, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"25cf8630-3dbf-4b2d-93cb-50b1bc6b45b4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mt5\", \"title\": \"mT5\", \"tokens\": 1157, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f0af61ce-d6fd-4c78-b274-093af50b3e65\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mt5\", \"title\": \"mT5\", \"tokens\": 1157, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"05bf87ab05b63a1f8c65c13b66237cef6268559b98eba2c06ccf6c0716ee0921\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f9a29c2d-c74b-41ff-8d1e-ca57edc3bccc\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"40cdcd1a879c8d17fb25945fd2d6ea4018dd8551dbf02b230be59f85def6497e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2635, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f0af61ce-d6fd-4c78-b274-093af50b3e65", "doc_id": "f0af61ce-d6fd-4c78-b274-093af50b3e65", "ref_doc_id": "f0af61ce-d6fd-4c78-b274-093af50b3e65", "chroma:document": "# mT5\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=mt5\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-mt5-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/mt5-small-finetuned-arxiv-cs-finetuned-arxiv-cs-full\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nThe mT5 model was presented in [mT5: A massively multilingual pre-trained text-to-text transformer](https://arxiv.org/abs/2010.11934) by Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya\nSiddhant, Aditya Barua, Colin Raffel.\n\nThe abstract from the paper is the following:\n\n*The recent \"Text-to-Text Transfer Transformer\" (T5) leveraged a unified text-to-text format and scale to attain\nstate-of-the-art results on a wide variety of English-language NLP tasks. In this paper, we introduce mT5, a\nmultilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages. We detail\nthe design and modified training of mT5 and demonstrate its state-of-the-art performance on many multilingual\nbenchmarks. We also describe a simple technique to prevent \"accidental translation\" in the zero-shot setting, where a\ngenerative model chooses to (partially) translate its prediction into the wrong language. All of the code and model\ncheckpoints used in this work are publicly available.*\n\nNote: mT5 was only pre-trained on [mC4](https://huggingface.co/datasets/mc4) excluding any supervised training.\nTherefore, this model has to be fine-tuned before it is usable on a downstream task, unlike the original T5 model.\nSince mT5 was pre-trained unsupervisedly, there's no real advantage to using a task prefix during single-task\nfine-tuning. If you are doing multi-task fine-tuning, you should use a prefix.\n\nGoogle has released the following variants:\n\n- [google/mt5-small](https://huggingface.co/google/mt5-small)\n\n- [google/mt5-base](https://huggingface.co/google/mt5-base)\n\n- [google/mt5-large](https://huggingface.co/google/mt5-large)\n\n- [google/mt5-xl](https://huggingface.co/google/mt5-xl)\n\n- [google/mt5-xxl](https://huggingface.co/google/mt5-xxl).\n\nThis model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The original code can be\nfound [here](https://github.com/google-research/multilingual-t5).\n\n## Resources\n\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n## MT5Config\n\n[[autodoc]] MT5Config\n\n## MT5Tokenizer\n\n[[autodoc]] MT5Tokenizer\n\nSee [`T5Tokenizer`] for all details."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f9a29c2d-c74b-41ff-8d1e-ca57edc3bccc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mt5", "title": "mT5", "tokens": 1157, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f9a29c2d-c74b-41ff-8d1e-ca57edc3bccc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mt5\", \"title\": \"mT5\", \"tokens\": 1157, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f0af61ce-d6fd-4c78-b274-093af50b3e65\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mt5\", \"title\": \"mT5\", \"tokens\": 1157, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"05bf87ab05b63a1f8c65c13b66237cef6268559b98eba2c06ccf6c0716ee0921\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"25cf8630-3dbf-4b2d-93cb-50b1bc6b45b4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mt5\", \"title\": \"mT5\", \"tokens\": 1157, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"74d179f070725e71ff09ffba3f0c06ec3d5a398f69f24accb2449e57e6c2d8ab\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2638, \"end_char_idx\": 3523, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f0af61ce-d6fd-4c78-b274-093af50b3e65", "doc_id": "f0af61ce-d6fd-4c78-b274-093af50b3e65", "ref_doc_id": "f0af61ce-d6fd-4c78-b274-093af50b3e65", "chroma:document": "## MT5TokenizerFast\n\n[[autodoc]] MT5TokenizerFast\n\nSee [`T5TokenizerFast`] for all details.\n\n<frameworkcontent>\n<pt>\n\n## MT5Model\n\n[[autodoc]] MT5Model\n\n## MT5ForConditionalGeneration\n\n[[autodoc]] MT5ForConditionalGeneration\n\n## MT5EncoderModel\n\n[[autodoc]] MT5EncoderModel\n\n## MT5ForSequenceClassification\n\n[[autodoc]] MT5ForSequenceClassification\n\n## MT5ForTokenClassification\n\n[[autodoc]] MT5ForTokenClassification\n\n## MT5ForQuestionAnswering\n\n[[autodoc]] MT5ForQuestionAnswering\n\n</pt>\n<tf>\n\n## TFMT5Model\n\n[[autodoc]] TFMT5Model\n\n## TFMT5ForConditionalGeneration\n\n[[autodoc]] TFMT5ForConditionalGeneration\n\n## TFMT5EncoderModel\n\n[[autodoc]] TFMT5EncoderModel\n\n</tf>\n<jax>\n\n## FlaxMT5Model\n\n[[autodoc]] FlaxMT5Model\n\n## FlaxMT5ForConditionalGeneration\n\n[[autodoc]] FlaxMT5ForConditionalGeneration\n\n## FlaxMT5EncoderModel\n\n[[autodoc]] FlaxMT5EncoderModel\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 361fa246-b666-4875-9bcb-613772ced974
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/hiera", "title": "Hiera", "tokens": 962, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"361fa246-b666-4875-9bcb-613772ced974\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/hiera\", \"title\": \"Hiera\", \"tokens\": 962, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a05d9220-9b90-425a-9dc1-f5bee382dc5f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/hiera\", \"title\": \"Hiera\", \"tokens\": 962, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a8ed85747a4819352a057d005f3ca0143122eab9f6bc78b716587fcf9b875b5e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d4d979b9-7249-444c-9bad-232994b034f5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"61b5e04bc01bc1c45f0e07a9e1b382e9366f6559a8cf6587865cdd5a40b13285\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3385, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a05d9220-9b90-425a-9dc1-f5bee382dc5f", "doc_id": "a05d9220-9b90-425a-9dc1-f5bee382dc5f", "ref_doc_id": "a05d9220-9b90-425a-9dc1-f5bee382dc5f", "chroma:document": "# Hiera\n\n## Overview\n\nHiera was proposed in [Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles](https://arxiv.org/abs/2306.00989) by Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, Jitendra Malik, Yanghao Li, Christoph Feichtenhofer\n\nThe paper introduces \"Hiera,\" a hierarchical Vision Transformer that simplifies the architecture of modern hierarchical vision transformers by removing unnecessary components without compromising on accuracy or efficiency. Unlike traditional transformers that add complex vision-specific components to improve supervised classification performance, Hiera demonstrates that such additions, often termed \"bells-and-whistles,\" are not essential for high accuracy. By leveraging a strong visual pretext task (MAE) for pretraining, Hiera retains simplicity and achieves superior accuracy and speed both in inference and training across various image and video recognition tasks. The approach suggests that spatial biases required for vision tasks can be effectively learned through proper pretraining, eliminating the need for added architectural complexity. \n\nThe abstract from the paper is the following:\n\n*Modern hierarchical vision transformers have added several vision-specific components in the pursuit of supervised classification performance. While these components lead to effective accuracies and attractive FLOP counts, the added complexity actually makes these transformers slower than their vanilla ViT counterparts. In this paper, we argue that this additional bulk is unnecessary. By pretraining with a strong visual pretext task (MAE), we can strip out all the bells-and-whistles from a state-of-the-art multi-stage vision transformer without losing accuracy. In the process, we create Hiera, an extremely simple hierarchical vision transformer that is more accurate than previous models while being significantly faster both at inference and during training. We evaluate Hiera on a variety of tasks for image and video recognition. Our code and models are available at https://github.com/facebookresearch/hiera.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/hiera_overview.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Hiera architecture. Taken from the <a href=\"https://arxiv.org/abs/2306.00989\">original paper.</a> </small>\n\nThis model was a joint contibution by [EduardoPacheco](https://huggingface.co/EduardoPacheco) and [namangarg110](https://huggingface.co/namangarg110). The original code can be found [here] (https://github.com/facebookresearch/hiera).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Hiera. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`HieraForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: d4d979b9-7249-444c-9bad-232994b034f5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/hiera", "title": "Hiera", "tokens": 962, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d4d979b9-7249-444c-9bad-232994b034f5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/hiera\", \"title\": \"Hiera\", \"tokens\": 962, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a05d9220-9b90-425a-9dc1-f5bee382dc5f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/hiera\", \"title\": \"Hiera\", \"tokens\": 962, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a8ed85747a4819352a057d005f3ca0143122eab9f6bc78b716587fcf9b875b5e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"361fa246-b666-4875-9bcb-613772ced974\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/hiera\", \"title\": \"Hiera\", \"tokens\": 962, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cc659b3aa7876e94a35d6b77a30a1ddce99d8bf3ea6631c4726b3182925ebebe\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1650, \"end_char_idx\": 3716, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a05d9220-9b90-425a-9dc1-f5bee382dc5f", "doc_id": "a05d9220-9b90-425a-9dc1-f5bee382dc5f", "ref_doc_id": "a05d9220-9b90-425a-9dc1-f5bee382dc5f", "chroma:document": "By pretraining with a strong visual pretext task (MAE), we can strip out all the bells-and-whistles from a state-of-the-art multi-stage vision transformer without losing accuracy. In the process, we create Hiera, an extremely simple hierarchical vision transformer that is more accurate than previous models while being significantly faster both at inference and during training. We evaluate Hiera on a variety of tasks for image and video recognition. Our code and models are available at https://github.com/facebookresearch/hiera.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/hiera_overview.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Hiera architecture. Taken from the <a href=\"https://arxiv.org/abs/2306.00989\">original paper.</a> </small>\n\nThis model was a joint contibution by [EduardoPacheco](https://huggingface.co/EduardoPacheco) and [namangarg110](https://huggingface.co/namangarg110). The original code can be found [here] (https://github.com/facebookresearch/hiera).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Hiera. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`HieraForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\n## HieraConfig\n\n[[autodoc]] HieraConfig\n\n## HieraModel\n\n[[autodoc]] HieraModel\n    - forward\n\n## HieraForPreTraining\n\n[[autodoc]] HieraForPreTraining\n    - forward\n  \n## HieraForImageClassification\n\n[[autodoc]] HieraForImageClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 50b42bfc-6b86-4077-a3dc-324a308ebb46
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/convbert", "title": "ConvBERT", "tokens": 1149, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"50b42bfc-6b86-4077-a3dc-324a308ebb46\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/convbert\", \"title\": \"ConvBERT\", \"tokens\": 1149, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e1777ded-4baa-49d1-9818-66a20012da22\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/convbert\", \"title\": \"ConvBERT\", \"tokens\": 1149, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1afc2a94bb644cf4f77007249fc7fb3ab6edb7314c78aa2d27df1890f3f80ebe\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d34387f2-3187-4c80-891a-4c17c072aa3e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3540d043a01b0486307106a78e2201e6ac349b060dd8d39f363f229f14d35781\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2185, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e1777ded-4baa-49d1-9818-66a20012da22", "doc_id": "e1777ded-4baa-49d1-9818-66a20012da22", "ref_doc_id": "e1777ded-4baa-49d1-9818-66a20012da22", "chroma:document": "# ConvBERT\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=convbert\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-convbert-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/conv-bert-base\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nThe ConvBERT model was proposed in [ConvBERT: Improving BERT with Span-based Dynamic Convolution](https://arxiv.org/abs/2008.02496) by Zihang Jiang, Weihao Yu, Daquan Zhou, Yunpeng Chen, Jiashi Feng, Shuicheng\nYan.\n\nThe abstract from the paper is the following:\n\n*Pre-trained language models like BERT and its variants have recently achieved impressive performance in various\nnatural language understanding tasks. However, BERT heavily relies on the global self-attention block and thus suffers\nlarge memory footprint and computation cost. Although all its attention heads query on the whole input sequence for\ngenerating the attention map from a global perspective, we observe some heads only need to learn local dependencies,\nwhich means the existence of computation redundancy. We therefore propose a novel span-based dynamic convolution to\nreplace these self-attention heads to directly model local dependencies. The novel convolution heads, together with the\nrest self-attention heads, form a new mixed attention block that is more efficient at both global and local context\nlearning. We equip BERT with this mixed attention design and build a ConvBERT model. Experiments have shown that\nConvBERT significantly outperforms BERT and its variants in various downstream tasks, with lower training cost and\nfewer model parameters. Remarkably, ConvBERTbase model achieves 86.4 GLUE score, 0.7 higher than ELECTRAbase, while\nusing less than 1/4 training cost. Code and pre-trained models will be released.*\n\nThis model was contributed by [abhishek](https://huggingface.co/abhishek). The original implementation can be found\nhere: https://github.com/yitu-opensource/ConvBert\n\n## Usage tips\n\nConvBERT training tips are similar to those of BERT. For usage tips refer to [BERT documentation](bert)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: d34387f2-3187-4c80-891a-4c17c072aa3e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/convbert", "title": "ConvBERT", "tokens": 1149, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d34387f2-3187-4c80-891a-4c17c072aa3e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/convbert\", \"title\": \"ConvBERT\", \"tokens\": 1149, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e1777ded-4baa-49d1-9818-66a20012da22\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/convbert\", \"title\": \"ConvBERT\", \"tokens\": 1149, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1afc2a94bb644cf4f77007249fc7fb3ab6edb7314c78aa2d27df1890f3f80ebe\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"50b42bfc-6b86-4077-a3dc-324a308ebb46\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/convbert\", \"title\": \"ConvBERT\", \"tokens\": 1149, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2bc8ed791dd1165fcac273802869da8ca0fcd2e9d09037a6a4e509119d3ddcc0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 608, \"end_char_idx\": 3884, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e1777ded-4baa-49d1-9818-66a20012da22", "doc_id": "e1777ded-4baa-49d1-9818-66a20012da22", "ref_doc_id": "e1777ded-4baa-49d1-9818-66a20012da22", "chroma:document": "The abstract from the paper is the following:\n\n*Pre-trained language models like BERT and its variants have recently achieved impressive performance in various\nnatural language understanding tasks. However, BERT heavily relies on the global self-attention block and thus suffers\nlarge memory footprint and computation cost. Although all its attention heads query on the whole input sequence for\ngenerating the attention map from a global perspective, we observe some heads only need to learn local dependencies,\nwhich means the existence of computation redundancy. We therefore propose a novel span-based dynamic convolution to\nreplace these self-attention heads to directly model local dependencies. The novel convolution heads, together with the\nrest self-attention heads, form a new mixed attention block that is more efficient at both global and local context\nlearning. We equip BERT with this mixed attention design and build a ConvBERT model. Experiments have shown that\nConvBERT significantly outperforms BERT and its variants in various downstream tasks, with lower training cost and\nfewer model parameters. Remarkably, ConvBERTbase model achieves 86.4 GLUE score, 0.7 higher than ELECTRAbase, while\nusing less than 1/4 training cost. Code and pre-trained models will be released.*\n\nThis model was contributed by [abhishek](https://huggingface.co/abhishek). The original implementation can be found\nhere: https://github.com/yitu-opensource/ConvBert\n\n## Usage tips\n\nConvBERT training tips are similar to those of BERT. For usage tips refer to [BERT documentation](bert).\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## ConvBertConfig\n\n[[autodoc]] ConvBertConfig\n\n## ConvBertTokenizer\n\n[[autodoc]] ConvBertTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## ConvBertTokenizerFast\n\n[[autodoc]] ConvBertTokenizerFast\n\n<frameworkcontent>\n<pt>\n\n## ConvBertModel\n\n[[autodoc]] ConvBertModel\n    - forward\n\n## ConvBertForMaskedLM\n\n[[autodoc]] ConvBertForMaskedLM\n    - forward\n\n## ConvBertForSequenceClassification\n\n[[autodoc]] ConvBertForSequenceClassification\n    - forward\n\n## ConvBertForMultipleChoice\n\n[[autodoc]] ConvBertForMultipleChoice\n    - forward\n\n## ConvBertForTokenClassification\n\n[[autodoc]] ConvBertForTokenClassification\n    - forward\n\n## ConvBertForQuestionAnswering\n\n[[autodoc]] ConvBertForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFConvBertModel\n\n[[autodoc]] TFConvBertModel\n    - call\n\n## TFConvBertForMaskedLM\n\n[[autodoc]] TFConvBertForMaskedLM\n    - call\n\n## TFConvBertForSequenceClassification\n\n[[autodoc]] TFConvBertForSequenceClassification\n    - call\n\n## TFConvBertForMultipleChoice\n\n[[autodoc]] TFConvBertForMultipleChoice\n    - call\n\n## TFConvBertForTokenClassification\n\n[[autodoc]] TFConvBertForTokenClassification\n    - call\n\n## TFConvBertForQuestionAnswering\n\n[[autodoc]] TFConvBertForQuestionAnswering\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 50aa0523-44ac-48ea-9899-900211a72682
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/sam", "title": "SAM", "tokens": 1817, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"50aa0523-44ac-48ea-9899-900211a72682\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/sam\", \"title\": \"SAM\", \"tokens\": 1817, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9468ead4-28c0-45e4-bdf7-1408fbde2549\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/sam\", \"title\": \"SAM\", \"tokens\": 1817, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0e97cbd12622112d426aa57dafbe97772cbbb5b69d5da2b2ec416ce26d0e5a3a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ba500c85-c0d5-4025-bc43-52d4182b749f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5949f0fc9a6193534d047dde4720e0b6d9ec4a115216cc5a020a4b8f47b0b3d6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3287, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9468ead4-28c0-45e4-bdf7-1408fbde2549", "doc_id": "9468ead4-28c0-45e4-bdf7-1408fbde2549", "ref_doc_id": "9468ead4-28c0-45e4-bdf7-1408fbde2549", "chroma:document": "# SAM\n\n## Overview\n\nSAM (Segment Anything Model) was proposed in [Segment Anything](https://arxiv.org/pdf/2304.02643v1.pdf) by Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alex Berg, Wan-Yen Lo, Piotr Dollar, Ross Girshick.\n\nThe model can be used to predict segmentation masks of any object of interest given an input image. \n\n![example image](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/sam-output.png)\n\nThe abstract from the paper is the following:\n\n*We introduce the Segment Anything (SA) project: a new task, model, and dataset for image segmentation. Using our efficient model in a data collection loop, we built the largest segmentation dataset to date (by far), with over 1 billion masks on 11M licensed and privacy respecting images. The model is designed and trained to be promptable, so it can transfer zero-shot to new image distributions and tasks. We evaluate its capabilities on numerous tasks and find that its zero-shot performance is impressive -- often competitive with or even superior to prior fully supervised results. We are releasing the Segment Anything Model (SAM) and corresponding dataset (SA-1B) of 1B masks and 11M images at [https://segment-anything.com](https://segment-anything.com) to foster research into foundation models for computer vision.*\n\nTips:\n\n- The model predicts binary masks that states the presence or not of the object of interest given an image.\n- The model predicts much better results if input 2D points and/or input bounding boxes are provided\n- You can prompt multiple points for the same image, and predict a single mask. \n- Fine-tuning the model is not supported yet\n- According to the paper, textual input should be also supported. However, at this time of writing this seems not to be supported according to [the official repository](https://github.com/facebookresearch/segment-anything/issues/4#issuecomment-1497626844). \n\n\nThis model was contributed by [ybelkada](https://huggingface.co/ybelkada) and [ArthurZ](https://huggingface.co/ArthurZ).\nThe original code can be found [here](https://github.com/facebookresearch/segment-anything).\n\nBelow is an example on how to run mask generation given an image and a 2D point:\n\n```python\nimport torch\nfrom PIL import Image\nimport requests\nfrom transformers import SamModel, SamProcessor\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nmodel = SamModel.from_pretrained(\"facebook/sam-vit-huge\").to(device)\nprocessor = SamProcessor.from_pretrained(\"facebook/sam-vit-huge\")\n\nimg_url = \"https://huggingface.co/ybelkada/segment-anything/resolve/main/assets/car.png\"\nraw_image = Image.open(requests.get(img_url, stream=True).raw).convert(\"RGB\")\ninput_points = [[[450, 600]]]  # 2D location of a window in the image\n\ninputs = processor(raw_image, input_points=input_points, return_tensors=\"pt\").to(device)\nwith torch.no_grad():\n    outputs = model(**inputs)\n\nmasks = processor.image_processor.post_process_masks(\n    outputs.pred_masks.cpu(), inputs[\"original_sizes\"].cpu(), inputs[\"reshaped_input_sizes\"].cpu()\n)\nscores = outputs.iou_scores\n```\n\nYou can also process your own masks alongside the input images in the processor to be passed to the model."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: ba500c85-c0d5-4025-bc43-52d4182b749f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/sam", "title": "SAM", "tokens": 1817, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ba500c85-c0d5-4025-bc43-52d4182b749f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/sam\", \"title\": \"SAM\", \"tokens\": 1817, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9468ead4-28c0-45e4-bdf7-1408fbde2549\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/sam\", \"title\": \"SAM\", \"tokens\": 1817, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0e97cbd12622112d426aa57dafbe97772cbbb5b69d5da2b2ec416ce26d0e5a3a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"50aa0523-44ac-48ea-9899-900211a72682\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/sam\", \"title\": \"SAM\", \"tokens\": 1817, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fd8d3a60dce5c92f726b6394c2399f51d3191e7d64d5036d72add1c6523474a7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6f7365c9-9e94-434a-bbdb-adb40f690ea4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"02f618440a7997e331ddc8f45c1cc1198aafa966f5f0a606ed8e6ebea7021633\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2014, \"end_char_idx\": 5003, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9468ead4-28c0-45e4-bdf7-1408fbde2549", "doc_id": "9468ead4-28c0-45e4-bdf7-1408fbde2549", "ref_doc_id": "9468ead4-28c0-45e4-bdf7-1408fbde2549", "chroma:document": "This model was contributed by [ybelkada](https://huggingface.co/ybelkada) and [ArthurZ](https://huggingface.co/ArthurZ).\nThe original code can be found [here](https://github.com/facebookresearch/segment-anything).\n\nBelow is an example on how to run mask generation given an image and a 2D point:\n\n```python\nimport torch\nfrom PIL import Image\nimport requests\nfrom transformers import SamModel, SamProcessor\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nmodel = SamModel.from_pretrained(\"facebook/sam-vit-huge\").to(device)\nprocessor = SamProcessor.from_pretrained(\"facebook/sam-vit-huge\")\n\nimg_url = \"https://huggingface.co/ybelkada/segment-anything/resolve/main/assets/car.png\"\nraw_image = Image.open(requests.get(img_url, stream=True).raw).convert(\"RGB\")\ninput_points = [[[450, 600]]]  # 2D location of a window in the image\n\ninputs = processor(raw_image, input_points=input_points, return_tensors=\"pt\").to(device)\nwith torch.no_grad():\n    outputs = model(**inputs)\n\nmasks = processor.image_processor.post_process_masks(\n    outputs.pred_masks.cpu(), inputs[\"original_sizes\"].cpu(), inputs[\"reshaped_input_sizes\"].cpu()\n)\nscores = outputs.iou_scores\n```\n\nYou can also process your own masks alongside the input images in the processor to be passed to the model.\n\n```python\nimport torch\nfrom PIL import Image\nimport requests\nfrom transformers import SamModel, SamProcessor\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nmodel = SamModel.from_pretrained(\"facebook/sam-vit-huge\").to(device)\nprocessor = SamProcessor.from_pretrained(\"facebook/sam-vit-huge\")\n\nimg_url = \"https://huggingface.co/ybelkada/segment-anything/resolve/main/assets/car.png\"\nraw_image = Image.open(requests.get(img_url, stream=True).raw).convert(\"RGB\")\nmask_url = \"https://huggingface.co/ybelkada/segment-anything/resolve/main/assets/car.png\"\nsegmentation_map = Image.open(requests.get(mask_url, stream=True).raw).convert(\"1\")\ninput_points = [[[450, 600]]]  # 2D location of a window in the image\n\ninputs = processor(raw_image, input_points=input_points, segmentation_maps=segmentation_map, return_tensors=\"pt\").to(device)\nwith torch.no_grad():\n    outputs = model(**inputs)\n\nmasks = processor.image_processor.post_process_masks(\n    outputs.pred_masks.cpu(), inputs[\"original_sizes\"].cpu(), inputs[\"reshaped_input_sizes\"].cpu()\n)\nscores = outputs.iou_scores\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with SAM.\n\n- [Demo notebook](https://github.com/huggingface/notebooks/blob/main/examples/segment_anything.ipynb) for using the model.\n- [Demo notebook](https://github.com/huggingface/notebooks/blob/main/examples/automatic_mask_generation.ipynb) for using the automatic mask generation pipeline.\n- [Demo notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/SAM/Run_inference_with_MedSAM_using_HuggingFace_Transformers.ipynb) for inference with MedSAM, a fine-tuned version of SAM on the medical domain."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6f7365c9-9e94-434a-bbdb-adb40f690ea4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/sam", "title": "SAM", "tokens": 1817, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6f7365c9-9e94-434a-bbdb-adb40f690ea4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/sam\", \"title\": \"SAM\", \"tokens\": 1817, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9468ead4-28c0-45e4-bdf7-1408fbde2549\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/sam\", \"title\": \"SAM\", \"tokens\": 1817, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0e97cbd12622112d426aa57dafbe97772cbbb5b69d5da2b2ec416ce26d0e5a3a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ba500c85-c0d5-4025-bc43-52d4182b749f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/sam\", \"title\": \"SAM\", \"tokens\": 1817, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cb8048a6816cd958e2d8ada5f333f1542b1cef71d0f0188d17b853a24c488f41\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4491, \"end_char_idx\": 6712, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9468ead4-28c0-45e4-bdf7-1408fbde2549", "doc_id": "9468ead4-28c0-45e4-bdf7-1408fbde2549", "ref_doc_id": "9468ead4-28c0-45e4-bdf7-1408fbde2549", "chroma:document": "- [Demo notebook](https://github.com/huggingface/notebooks/blob/main/examples/segment_anything.ipynb) for using the model.\n- [Demo notebook](https://github.com/huggingface/notebooks/blob/main/examples/automatic_mask_generation.ipynb) for using the automatic mask generation pipeline.\n- [Demo notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/SAM/Run_inference_with_MedSAM_using_HuggingFace_Transformers.ipynb) for inference with MedSAM, a fine-tuned version of SAM on the medical domain. \ud83c\udf0e\n- [Demo notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/SAM/Fine_tune_SAM_(segment_anything)_on_a_custom_dataset.ipynb) for fine-tuning the model on custom data. \ud83c\udf0e\n\n## SlimSAM\n\nSlimSAM, a pruned version of SAM, was proposed in [0.1% Data Makes Segment Anything Slim](https://arxiv.org/abs/2312.05284) by Zigeng Chen et al. SlimSAM reduces the size of the SAM models considerably while maintaining the same performance.\n\nCheckpoints can be found on the [hub](https://huggingface.co/models?other=slimsam), and they can be used as a drop-in replacement of SAM.\n\n## Grounded SAM\n\nOne can combine [Grounding DINO](grounding-dino) with SAM for text-based mask generation as introduced in [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://arxiv.org/abs/2401.14159). You can refer to this [demo notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/Grounding%20DINO/GroundingDINO_with_Segment_Anything.ipynb) \ud83c\udf0d for details.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/grounded_sam.png\"\nalt=\"drawing\" width=\"900\"/>\n\n<small> Grounded SAM overview. Taken from the <a href=\"https://github.com/IDEA-Research/Grounded-Segment-Anything\">original repository</a>. </small>\n\n## SamConfig\n\n[[autodoc]] SamConfig\n\n## SamVisionConfig\n\n[[autodoc]] SamVisionConfig\n\n## SamMaskDecoderConfig\n\n[[autodoc]] SamMaskDecoderConfig\n\n## SamPromptEncoderConfig\n\n[[autodoc]] SamPromptEncoderConfig\n\n\n## SamProcessor\n\n[[autodoc]] SamProcessor\n\n\n## SamImageProcessor\n\n[[autodoc]] SamImageProcessor\n\n\n## SamModel\n\n[[autodoc]] SamModel\n    - forward\n\n\n## TFSamModel\n\n[[autodoc]] TFSamModel\n    - call"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 7c13cc3f-7f07-4d65-a5d0-f501e5b1e9a7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/falcon", "title": "Falcon", "tokens": 708, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7c13cc3f-7f07-4d65-a5d0-f501e5b1e9a7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/falcon\", \"title\": \"Falcon\", \"tokens\": 708, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"005624c4-aa23-4941-9c15-abe32b64f9b5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/falcon\", \"title\": \"Falcon\", \"tokens\": 708, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ab8b34dcf9e528fda0bede17055940b50840eef09d68a690f2478dc387b5d31d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2444, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "005624c4-aa23-4941-9c15-abe32b64f9b5", "doc_id": "005624c4-aa23-4941-9c15-abe32b64f9b5", "ref_doc_id": "005624c4-aa23-4941-9c15-abe32b64f9b5", "chroma:document": "# Falcon\n\n## Overview\n\nFalcon is a class of causal decoder-only models built by [TII](https://www.tii.ae/). The largest Falcon checkpoints\nhave been trained on >=1T tokens of text, with a particular emphasis on the [RefinedWeb](https://arxiv.org/abs/2306.01116)\ncorpus. They are made available under the Apache 2.0 license.\n\n\nFalcon's architecture is modern and optimized for inference, with multi-query attention and support for efficient\nattention variants like `FlashAttention`. Both 'base' models trained only as causal language models as well as\n'instruct' models that have received further fine-tuning are available.\n\n\nFalcon models are (as of 2023) some of the largest and most powerful open-source language models,\nand consistently rank highly in the [OpenLLM leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard).\n\n## Converting custom checkpoints \n\n<Tip>\n\nFalcon models were initially added to the Hugging Face Hub as custom code checkpoints. However, Falcon is now fully\nsupported in the Transformers library. If you fine-tuned a model from a custom code checkpoint, we recommend converting\nyour checkpoint to the new in-library format, as this should give significant improvements to stability and\nperformance, especially for generation, as well as removing the need to use `trust_remote_code=True`!\n\n</Tip>\n\nYou can convert custom code checkpoints to full Transformers checkpoints using the `convert_custom_code_checkpoint.py` \nscript located in the\n[Falcon model directory](https://github.com/huggingface/transformers/tree/main/src/transformers/models/falcon)\nof the Transformers library. To use this script, simply call it with \n`python convert_custom_code_checkpoint.py --checkpoint_dir my_model`. This will convert your checkpoint in-place, and\nyou can immediately load it from the directory afterwards with e.g. `from_pretrained()`. If your model hasn't been\nuploaded to the Hub, we recommend making a backup before attempting the conversion, just in case!\n\n\n## FalconConfig\n\n[[autodoc]] FalconConfig\n    - all\n\n## FalconModel\n\n[[autodoc]] FalconModel\n    - forward\n\n## FalconForCausalLM\n\n[[autodoc]] FalconForCausalLM\n    - forward\n\n## FalconForSequenceClassification\n\n[[autodoc]] FalconForSequenceClassification\n    - forward\n\n## FalconForTokenClassification\n\n[[autodoc]] FalconForTokenClassification\n    - forward\n\n## FalconForQuestionAnswering\n\n[[autodoc]] FalconForQuestionAnswering\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 931fc181-c629-4989-8e9b-968ef2356769
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/barthez", "title": "BARThez", "tokens": 728, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"931fc181-c629-4989-8e9b-968ef2356769\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/barthez\", \"title\": \"BARThez\", \"tokens\": 728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c3dd0ec0-3eea-4a0f-a434-21238fa8cb84\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/barthez\", \"title\": \"BARThez\", \"tokens\": 728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"50867ebb6b133c0b8d10a773f31f5b4043b6eb5b5f7dd20b08b3cc3026ca00cb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2286, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c3dd0ec0-3eea-4a0f-a434-21238fa8cb84", "doc_id": "c3dd0ec0-3eea-4a0f-a434-21238fa8cb84", "ref_doc_id": "c3dd0ec0-3eea-4a0f-a434-21238fa8cb84", "chroma:document": "# BARThez\n\n## Overview\n\nThe BARThez model was proposed in [BARThez: a Skilled Pretrained French Sequence-to-Sequence Model](https://arxiv.org/abs/2010.12321) by Moussa Kamal Eddine, Antoine J.-P. Tixier, Michalis Vazirgiannis on 23 Oct,\n2020.\n\nThe abstract of the paper:\n\n\n*Inductive transfer learning, enabled by self-supervised learning, have taken the entire Natural Language Processing\n(NLP) field by storm, with models such as BERT and BART setting new state of the art on countless natural language\nunderstanding tasks. While there are some notable exceptions, most of the available models and research have been\nconducted for the English language. In this work, we introduce BARThez, the first BART model for the French language\n(to the best of our knowledge). BARThez was pretrained on a very large monolingual French corpus from past research\nthat we adapted to suit BART's perturbation schemes. Unlike already existing BERT-based French language models such as\nCamemBERT and FlauBERT, BARThez is particularly well-suited for generative tasks, since not only its encoder but also\nits decoder is pretrained. In addition to discriminative tasks from the FLUE benchmark, we evaluate BARThez on a novel\nsummarization dataset, OrangeSum, that we release with this paper. We also continue the pretraining of an already\npretrained multilingual BART on BARThez's corpus, and we show that the resulting model, which we call mBARTHez,\nprovides a significant boost over vanilla BARThez, and is on par with or outperforms CamemBERT and FlauBERT.*\n\nThis model was contributed by [moussakam](https://huggingface.co/moussakam). The Authors' code can be found [here](https://github.com/moussaKam/BARThez).\n\n<Tip> \n\nBARThez implementation is the same as BART, except for tokenization. Refer to [BART documentation](bart) for information on \nconfiguration classes and their parameters. BARThez-specific tokenizers are documented below.  \n\n</Tip>\n\n## Resources\n\n- BARThez can be fine-tuned on sequence-to-sequence tasks in a similar way as BART, check:\n  [examples/pytorch/summarization/](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization/README.md).\n\n\n## BarthezTokenizer\n\n[[autodoc]] BarthezTokenizer\n\n## BarthezTokenizerFast\n\n[[autodoc]] BarthezTokenizerFast"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 295ec289-da1a-4edb-81fb-81628a8f90fb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/luke", "title": "LUKE", "tokens": 2104, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"295ec289-da1a-4edb-81fb-81628a8f90fb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/luke\", \"title\": \"LUKE\", \"tokens\": 2104, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0f906553-f1a5-452f-b572-89d5a4f81d7b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/luke\", \"title\": \"LUKE\", \"tokens\": 2104, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4cdf3055da16fa352e473117cdd96b1119edf5beba766b91d195c7db6e5edcaa\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0c6dfd23-674e-4035-adca-a15e3a2c802e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"71230b91dfc3590aa82329aa8db5865bdc3b280da566a97ddf023e647cce60ac\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3455, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0f906553-f1a5-452f-b572-89d5a4f81d7b", "doc_id": "0f906553-f1a5-452f-b572-89d5a4f81d7b", "ref_doc_id": "0f906553-f1a5-452f-b572-89d5a4f81d7b", "chroma:document": "# LUKE\n\n## Overview\n\nThe LUKE model was proposed in [LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](https://arxiv.org/abs/2010.01057) by Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda and Yuji Matsumoto.\nIt is based on RoBERTa and adds entity embeddings as well as an entity-aware self-attention mechanism, which helps\nimprove performance on various downstream tasks involving reasoning about entities such as named entity recognition,\nextractive and cloze-style question answering, entity typing, and relation classification.\n\nThe abstract from the paper is the following:\n\n*Entity representations are useful in natural language tasks involving entities. In this paper, we propose new\npretrained contextualized representations of words and entities based on the bidirectional transformer. The proposed\nmodel treats words and entities in a given text as independent tokens, and outputs contextualized representations of\nthem. Our model is trained using a new pretraining task based on the masked language model of BERT. The task involves\npredicting randomly masked words and entities in a large entity-annotated corpus retrieved from Wikipedia. We also\npropose an entity-aware self-attention mechanism that is an extension of the self-attention mechanism of the\ntransformer, and considers the types of tokens (words or entities) when computing attention scores. The proposed model\nachieves impressive empirical performance on a wide range of entity-related tasks. In particular, it obtains\nstate-of-the-art results on five well-known datasets: Open Entity (entity typing), TACRED (relation classification),\nCoNLL-2003 (named entity recognition), ReCoRD (cloze-style question answering), and SQuAD 1.1 (extractive question\nanswering).*\n\nThis model was contributed by [ikuyamada](https://huggingface.co/ikuyamada) and [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/studio-ousia/luke).\n\n## Usage tips\n\n- This implementation is the same as [`RobertaModel`] with the addition of entity embeddings as well\n  as an entity-aware self-attention mechanism, which improves performance on tasks involving reasoning about entities.\n- LUKE treats entities as input tokens; therefore, it takes `entity_ids`, `entity_attention_mask`,\n  `entity_token_type_ids` and `entity_position_ids` as extra input. You can obtain those using\n  [`LukeTokenizer`].\n- [`LukeTokenizer`] takes `entities` and `entity_spans` (character-based start and end\n  positions of the entities in the input text) as extra input. `entities` typically consist of [MASK] entities or\n  Wikipedia entities. The brief description when inputting these entities are as follows:\n\n  - *Inputting [MASK] entities to compute entity representations*: The [MASK] entity is used to mask entities to be\n    predicted during pretraining. When LUKE receives the [MASK] entity, it tries to predict the original entity by\n    gathering the information about the entity from the input text. Therefore, the [MASK] entity can be used to address\n    downstream tasks requiring the information of entities in text such as entity typing, relation classification, and\n    named entity recognition.\n  - *Inputting Wikipedia entities to compute knowledge-enhanced token representations*: LUKE learns rich information\n    (or knowledge) about Wikipedia entities during pretraining and stores the information in its entity embedding."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0c6dfd23-674e-4035-adca-a15e3a2c802e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/luke", "title": "LUKE", "tokens": 2104, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0c6dfd23-674e-4035-adca-a15e3a2c802e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/luke\", \"title\": \"LUKE\", \"tokens\": 2104, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0f906553-f1a5-452f-b572-89d5a4f81d7b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/luke\", \"title\": \"LUKE\", \"tokens\": 2104, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4cdf3055da16fa352e473117cdd96b1119edf5beba766b91d195c7db6e5edcaa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"295ec289-da1a-4edb-81fb-81628a8f90fb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/luke\", \"title\": \"LUKE\", \"tokens\": 2104, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"793a45a6920d1204f7d6d3a34bb34d45ecdc7d83f9f258054693ee355999a8cb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"099b354e-8588-4b68-81ff-ced8244f59fd\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5eefc557c728e86ebab5a622ad313d9972541c66d04c8d7d929cac4455b09d29\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1780, \"end_char_idx\": 5005, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0f906553-f1a5-452f-b572-89d5a4f81d7b", "doc_id": "0f906553-f1a5-452f-b572-89d5a4f81d7b", "ref_doc_id": "0f906553-f1a5-452f-b572-89d5a4f81d7b", "chroma:document": "*\n\nThis model was contributed by [ikuyamada](https://huggingface.co/ikuyamada) and [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/studio-ousia/luke).\n\n## Usage tips\n\n- This implementation is the same as [`RobertaModel`] with the addition of entity embeddings as well\n  as an entity-aware self-attention mechanism, which improves performance on tasks involving reasoning about entities.\n- LUKE treats entities as input tokens; therefore, it takes `entity_ids`, `entity_attention_mask`,\n  `entity_token_type_ids` and `entity_position_ids` as extra input. You can obtain those using\n  [`LukeTokenizer`].\n- [`LukeTokenizer`] takes `entities` and `entity_spans` (character-based start and end\n  positions of the entities in the input text) as extra input. `entities` typically consist of [MASK] entities or\n  Wikipedia entities. The brief description when inputting these entities are as follows:\n\n  - *Inputting [MASK] entities to compute entity representations*: The [MASK] entity is used to mask entities to be\n    predicted during pretraining. When LUKE receives the [MASK] entity, it tries to predict the original entity by\n    gathering the information about the entity from the input text. Therefore, the [MASK] entity can be used to address\n    downstream tasks requiring the information of entities in text such as entity typing, relation classification, and\n    named entity recognition.\n  - *Inputting Wikipedia entities to compute knowledge-enhanced token representations*: LUKE learns rich information\n    (or knowledge) about Wikipedia entities during pretraining and stores the information in its entity embedding. By\n    using Wikipedia entities as input tokens, LUKE outputs token representations enriched by the information stored in\n    the embeddings of these entities. This is particularly effective for tasks requiring real-world knowledge, such as\n    question answering.\n\n- There are three head models for the former use case:\n\n  - [`LukeForEntityClassification`], for tasks to classify a single entity in an input text such as\n    entity typing, e.g. the [Open Entity dataset](https://www.cs.utexas.edu/~eunsol/html_pages/open_entity.html).\n    This model places a linear head on top of the output entity representation.\n  - [`LukeForEntityPairClassification`], for tasks to classify the relationship between two entities\n    such as relation classification, e.g. the [TACRED dataset](https://nlp.stanford.edu/projects/tacred/). This\n    model places a linear head on top of the concatenated output representation of the pair of given entities.\n  - [`LukeForEntitySpanClassification`], for tasks to classify the sequence of entity spans, such as\n    named entity recognition (NER). This model places a linear head on top of the output entity representations. You\n    can address NER using this model by inputting all possible entity spans in the text to the model.\n\n  [`LukeTokenizer`] has a `task` argument, which enables you to easily create an input to these\n  head models by specifying `task=\"entity_classification\"`, `task=\"entity_pair_classification\"`, or\n  `task=\"entity_span_classification\"`. Please refer to the example code of each head models."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 099b354e-8588-4b68-81ff-ced8244f59fd
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/luke", "title": "LUKE", "tokens": 2104, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"099b354e-8588-4b68-81ff-ced8244f59fd\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/luke\", \"title\": \"LUKE\", \"tokens\": 2104, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0f906553-f1a5-452f-b572-89d5a4f81d7b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/luke\", \"title\": \"LUKE\", \"tokens\": 2104, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4cdf3055da16fa352e473117cdd96b1119edf5beba766b91d195c7db6e5edcaa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0c6dfd23-674e-4035-adca-a15e3a2c802e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/luke\", \"title\": \"LUKE\", \"tokens\": 2104, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2b8c56359dba0917e784d36712834e85e44defe09f6bf33ce30aa03677e73d7e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7488acf1-d35b-4e75-a03e-cbf35d030dbf\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8c707e36fe102771bbe4e5cc0da4f9dd23ad91bc0fb9d9e7a8fe31bdb21c8be1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3226, \"end_char_idx\": 5410, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0f906553-f1a5-452f-b572-89d5a4f81d7b", "doc_id": "0f906553-f1a5-452f-b572-89d5a4f81d7b", "ref_doc_id": "0f906553-f1a5-452f-b572-89d5a4f81d7b", "chroma:document": "- *Inputting Wikipedia entities to compute knowledge-enhanced token representations*: LUKE learns rich information\n    (or knowledge) about Wikipedia entities during pretraining and stores the information in its entity embedding. By\n    using Wikipedia entities as input tokens, LUKE outputs token representations enriched by the information stored in\n    the embeddings of these entities. This is particularly effective for tasks requiring real-world knowledge, such as\n    question answering.\n\n- There are three head models for the former use case:\n\n  - [`LukeForEntityClassification`], for tasks to classify a single entity in an input text such as\n    entity typing, e.g. the [Open Entity dataset](https://www.cs.utexas.edu/~eunsol/html_pages/open_entity.html).\n    This model places a linear head on top of the output entity representation.\n  - [`LukeForEntityPairClassification`], for tasks to classify the relationship between two entities\n    such as relation classification, e.g. the [TACRED dataset](https://nlp.stanford.edu/projects/tacred/). This\n    model places a linear head on top of the concatenated output representation of the pair of given entities.\n  - [`LukeForEntitySpanClassification`], for tasks to classify the sequence of entity spans, such as\n    named entity recognition (NER). This model places a linear head on top of the output entity representations. You\n    can address NER using this model by inputting all possible entity spans in the text to the model.\n\n  [`LukeTokenizer`] has a `task` argument, which enables you to easily create an input to these\n  head models by specifying `task=\"entity_classification\"`, `task=\"entity_pair_classification\"`, or\n  `task=\"entity_span_classification\"`. Please refer to the example code of each head models.\n\nUsage example:\n\n```python\n>>> from transformers import LukeTokenizer, LukeModel, LukeForEntityPairClassification\n\n>>> model = LukeModel.from_pretrained(\"studio-ousia/luke-base\")\n>>> tokenizer = LukeTokenizer.from_pretrained(\"studio-ousia/luke-base\")\n# Example 1: Computing the contextualized entity representation corresponding to the entity mention \"Beyonc\u00e9\"\n\n>>> text = \"Beyonc\u00e9 lives in Los Angeles.\""}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 7488acf1-d35b-4e75-a03e-cbf35d030dbf
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/luke", "title": "LUKE", "tokens": 2104, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7488acf1-d35b-4e75-a03e-cbf35d030dbf\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/luke\", \"title\": \"LUKE\", \"tokens\": 2104, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0f906553-f1a5-452f-b572-89d5a4f81d7b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/luke\", \"title\": \"LUKE\", \"tokens\": 2104, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4cdf3055da16fa352e473117cdd96b1119edf5beba766b91d195c7db6e5edcaa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"099b354e-8588-4b68-81ff-ced8244f59fd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/luke\", \"title\": \"LUKE\", \"tokens\": 2104, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"01453de1a5c8b3f519ec4ef36239455d965644d3e1566add8026b4c4d4e5b10c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3616, \"end_char_idx\": 8615, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0f906553-f1a5-452f-b572-89d5a4f81d7b", "doc_id": "0f906553-f1a5-452f-b572-89d5a4f81d7b", "ref_doc_id": "0f906553-f1a5-452f-b572-89d5a4f81d7b", "chroma:document": "This is particularly effective for tasks requiring real-world knowledge, such as\n    question answering.\n\n- There are three head models for the former use case:\n\n  - [`LukeForEntityClassification`], for tasks to classify a single entity in an input text such as\n    entity typing, e.g. the [Open Entity dataset](https://www.cs.utexas.edu/~eunsol/html_pages/open_entity.html).\n    This model places a linear head on top of the output entity representation.\n  - [`LukeForEntityPairClassification`], for tasks to classify the relationship between two entities\n    such as relation classification, e.g. the [TACRED dataset](https://nlp.stanford.edu/projects/tacred/). This\n    model places a linear head on top of the concatenated output representation of the pair of given entities.\n  - [`LukeForEntitySpanClassification`], for tasks to classify the sequence of entity spans, such as\n    named entity recognition (NER). This model places a linear head on top of the output entity representations. You\n    can address NER using this model by inputting all possible entity spans in the text to the model.\n\n  [`LukeTokenizer`] has a `task` argument, which enables you to easily create an input to these\n  head models by specifying `task=\"entity_classification\"`, `task=\"entity_pair_classification\"`, or\n  `task=\"entity_span_classification\"`. Please refer to the example code of each head models.\n\nUsage example:\n\n```python\n>>> from transformers import LukeTokenizer, LukeModel, LukeForEntityPairClassification\n\n>>> model = LukeModel.from_pretrained(\"studio-ousia/luke-base\")\n>>> tokenizer = LukeTokenizer.from_pretrained(\"studio-ousia/luke-base\")\n# Example 1: Computing the contextualized entity representation corresponding to the entity mention \"Beyonc\u00e9\"\n\n>>> text = \"Beyonc\u00e9 lives in Los Angeles.\"\n>>> entity_spans = [(0, 7)]  # character-based entity span corresponding to \"Beyonc\u00e9\"\n>>> inputs = tokenizer(text, entity_spans=entity_spans, add_prefix_space=True, return_tensors=\"pt\")\n>>> outputs = model(**inputs)\n>>> word_last_hidden_state = outputs.last_hidden_state\n>>> entity_last_hidden_state = outputs.entity_last_hidden_state\n# Example 2: Inputting Wikipedia entities to obtain enriched contextualized representations\n\n>>> entities = [\n...     \"Beyonc\u00e9\",\n...     \"Los Angeles\",\n... ]  # Wikipedia entity titles corresponding to the entity mentions \"Beyonc\u00e9\" and \"Los Angeles\"\n>>> entity_spans = [(0, 7), (17, 28)]  # character-based entity spans corresponding to \"Beyonc\u00e9\" and \"Los Angeles\"\n>>> inputs = tokenizer(text, entities=entities, entity_spans=entity_spans, add_prefix_space=True, return_tensors=\"pt\")\n>>> outputs = model(**inputs)\n>>> word_last_hidden_state = outputs.last_hidden_state\n>>> entity_last_hidden_state = outputs.entity_last_hidden_state\n# Example 3: Classifying the relationship between two entities using LukeForEntityPairClassification head model\n\n>>> model = LukeForEntityPairClassification.from_pretrained(\"studio-ousia/luke-large-finetuned-tacred\")\n>>> tokenizer = LukeTokenizer.from_pretrained(\"studio-ousia/luke-large-finetuned-tacred\")\n>>> entity_spans = [(0, 7), (17, 28)]  # character-based entity spans corresponding to \"Beyonc\u00e9\" and \"Los Angeles\"\n>>> inputs = tokenizer(text, entity_spans=entity_spans, return_tensors=\"pt\")\n>>> outputs = model(**inputs)\n>>> logits = outputs.logits\n>>> predicted_class_idx = int(logits[0].argmax())\n>>> print(\"Predicted class:\", model.config.id2label[predicted_class_idx])\n```\n\n## Resources\n\n- [A demo notebook on how to fine-tune [`LukeForEntityPairClassification`] for relation classification](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/LUKE)\n- [Notebooks showcasing how you to reproduce the results as reported in the paper with the HuggingFace implementation of LUKE](https://github.com/studio-ousia/luke/tree/master/notebooks)\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## LukeConfig\n\n[[autodoc]] LukeConfig\n\n## LukeTokenizer\n\n[[autodoc]] LukeTokenizer\n    - __call__\n    - save_vocabulary\n\n## LukeModel\n\n[[autodoc]] LukeModel\n    - forward\n\n## LukeForMaskedLM\n\n[[autodoc]] LukeForMaskedLM\n    - forward\n\n## LukeForEntityClassification\n\n[[autodoc]] LukeForEntityClassification\n    - forward\n\n## LukeForEntityPairClassification\n\n[[autodoc]] LukeForEntityPairClassification\n    - forward\n\n## LukeForEntitySpanClassification\n\n[[autodoc]] LukeForEntitySpanClassification\n    - forward\n\n## LukeForSequenceClassification\n\n[[autodoc]] LukeForSequenceClassification\n    - forward\n\n## LukeForMultipleChoice\n\n[[autodoc]] LukeForMultipleChoice\n    - forward\n\n## LukeForTokenClassification\n\n[[autodoc]] LukeForTokenClassification\n    - forward\n\n## LukeForQuestionAnswering\n\n[[autodoc]] LukeForQuestionAnswering\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f797e72e-5853-4781-9fb6-b84c1c8ac9e8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/focalnet", "title": "FocalNet", "tokens": 878, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f797e72e-5853-4781-9fb6-b84c1c8ac9e8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/focalnet\", \"title\": \"FocalNet\", \"tokens\": 878, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7292e9f4-2e71-44a6-96f5-cf0ea8425164\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/focalnet\", \"title\": \"FocalNet\", \"tokens\": 878, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ea6f618c6a2070cdd7ab8b84fdd13b13b33aa9b12ff8174eca6009ddee5ac22a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2840, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7292e9f4-2e71-44a6-96f5-cf0ea8425164", "doc_id": "7292e9f4-2e71-44a6-96f5-cf0ea8425164", "ref_doc_id": "7292e9f4-2e71-44a6-96f5-cf0ea8425164", "chroma:document": "# FocalNet\n\n## Overview\n\nThe FocalNet model was proposed in [Focal Modulation Networks](https://arxiv.org/abs/2203.11926) by Jianwei Yang, Chunyuan Li, Xiyang Dai, Lu Yuan, Jianfeng Gao.\nFocalNets completely replace self-attention (used in models like [ViT](vit) and [Swin](swin)) by a focal modulation mechanism for modeling token interactions in vision.\nThe authors claim that FocalNets outperform self-attention based models with similar computational costs on the tasks of image classification, object detection, and segmentation.\n\nThe abstract from the paper is the following:\n\n*We propose focal modulation networks (FocalNets in short), where self-attention (SA) is completely replaced by a focal modulation mechanism for modeling token interactions in vision. Focal modulation comprises three components: (i) hierarchical contextualization, implemented using a stack of depth-wise convolutional layers, to encode visual contexts from short to long ranges, (ii) gated aggregation to selectively gather contexts for each query token based on its\ncontent, and (iii) element-wise modulation or affine transformation to inject the aggregated context into the query. Extensive experiments show FocalNets outperform the state-of-the-art SA counterparts (e.g., Swin and Focal Transformers) with similar computational costs on the tasks of image classification, object detection, and segmentation. Specifically, FocalNets with tiny and base size achieve 82.3% and 83.9% top-1 accuracy on ImageNet-1K. After pretrained on ImageNet-22K in 224 resolution, it attains 86.5% and 87.3% top-1 accuracy when finetuned with resolution 224 and 384, respectively. When transferred to downstream tasks, FocalNets exhibit clear superiority. For object detection with Mask R-CNN, FocalNet base trained with 1\\times outperforms the Swin counterpart by 2.1 points and already surpasses Swin trained with 3\\times schedule (49.0 v.s. 48.5). For semantic segmentation with UPerNet, FocalNet base at single-scale outperforms Swin by 2.4, and beats Swin at multi-scale (50.5 v.s. 49.7). Using large FocalNet and Mask2former, we achieve 58.5 mIoU for ADE20K semantic segmentation, and 57.9 PQ for COCO Panoptic Segmentation. Using huge FocalNet and DINO, we achieved 64.3 and 64.4 mAP on COCO minival and test-dev, respectively, establishing new SoTA on top of much larger attention-based models like Swinv2-G and BEIT-3.*\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/microsoft/FocalNet).\n\n## FocalNetConfig\n\n[[autodoc]] FocalNetConfig\n\n## FocalNetModel\n\n[[autodoc]] FocalNetModel\n    - forward\n\n## FocalNetForMaskedImageModeling\n\n[[autodoc]] FocalNetForMaskedImageModeling\n    - forward\n\n## FocalNetForImageClassification\n\n[[autodoc]] FocalNetForImageClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: d752a68a-3e66-44b1-b0e8-78d9bf1d3e50
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/ernie", "title": "ERNIE", "tokens": 1181, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d752a68a-3e66-44b1-b0e8-78d9bf1d3e50\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/ernie\", \"title\": \"ERNIE\", \"tokens\": 1181, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ed58bf15-b439-4775-bc1a-20c63798f827\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/ernie\", \"title\": \"ERNIE\", \"tokens\": 1181, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9e9847047bba8267fc9a8c14b435c9fc91ff4e105b13d24e031e6dd12887bd18\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5aa3a0ea-7b49-4ca9-89ed-1b5a479b5052\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"623aa9ce8663614044bb6a1d57fd891e2a89fa4a98813feb1753d1e2a7f76554\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2016, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ed58bf15-b439-4775-bc1a-20c63798f827", "doc_id": "ed58bf15-b439-4775-bc1a-20c63798f827", "ref_doc_id": "ed58bf15-b439-4775-bc1a-20c63798f827", "chroma:document": "# ERNIE\n\n## Overview\nERNIE is a series of powerful models proposed by baidu, especially in Chinese tasks,\nincluding [ERNIE1.0](https://arxiv.org/abs/1904.09223), [ERNIE2.0](https://ojs.aaai.org/index.php/AAAI/article/view/6428),\n[ERNIE3.0](https://arxiv.org/abs/2107.02137), [ERNIE-Gram](https://arxiv.org/abs/2010.12148), [ERNIE-health](https://arxiv.org/abs/2110.07244), etc.\n\nThese models are contributed by [nghuyong](https://huggingface.co/nghuyong) and the official code can be found in [PaddleNLP](https://github.com/PaddlePaddle/PaddleNLP) (in PaddlePaddle).\n\n### Usage example\nTake `ernie-1.0-base-zh` as an example:\n\n```Python\nfrom transformers import AutoTokenizer, AutoModel\ntokenizer = AutoTokenizer.from_pretrained(\"nghuyong/ernie-1.0-base-zh\")\nmodel = AutoModel.from_pretrained(\"nghuyong/ernie-1.0-base-zh\")\n```\n\n### Model checkpoints\n\n|     Model Name      | Language |           Description           |\n|:-------------------:|:--------:|:-------------------------------:|\n|  ernie-1.0-base-zh  | Chinese  | Layer:12, Heads:12, Hidden:768  |\n|  ernie-2.0-base-en  | English  | Layer:12, Heads:12, Hidden:768  |\n| ernie-2.0-large-en  | English  | Layer:24, Heads:16, Hidden:1024 |\n|  ernie-3.0-base-zh  | Chinese  | Layer:12, Heads:12, Hidden:768  |\n| ernie-3.0-medium-zh | Chinese  |  Layer:6, Heads:12, Hidden:768  |\n|  ernie-3.0-mini-zh  | Chinese  |  Layer:6, Heads:12, Hidden:384  |\n| ernie-3.0-micro-zh  | Chinese  |  Layer:4, Heads:12, Hidden:384  |\n|  ernie-3.0-nano-zh  | Chinese  |  Layer:4, Heads:12, Hidden:312  |\n|   ernie-health-zh   | Chinese  | Layer:12, Heads:12, Hidden:768  |\n|    ernie-gram-zh    | Chinese  | Layer:12, Heads:12, Hidden:768  |\n\nYou can find all the supported models from huggingface's model hub: [huggingface.co/nghuyong](https://huggingface.co/nghuyong), and model details from paddle's official\nrepo: [PaddleNLP](https://paddlenlp.readthedocs.io/zh/latest/model_zoo/transformers/ERNIE/contents.html)\nand [ERNIE](https://github.com/PaddlePaddle/ERNIE/blob/repro)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 5aa3a0ea-7b49-4ca9-89ed-1b5a479b5052
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/ernie", "title": "ERNIE", "tokens": 1181, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5aa3a0ea-7b49-4ca9-89ed-1b5a479b5052\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/ernie\", \"title\": \"ERNIE\", \"tokens\": 1181, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ed58bf15-b439-4775-bc1a-20c63798f827\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/ernie\", \"title\": \"ERNIE\", \"tokens\": 1181, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9e9847047bba8267fc9a8c14b435c9fc91ff4e105b13d24e031e6dd12887bd18\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d752a68a-3e66-44b1-b0e8-78d9bf1d3e50\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/ernie\", \"title\": \"ERNIE\", \"tokens\": 1181, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3ec88ada192c55a5e8e02ffcc4c1f63a8589c8d011fa8365652b778f24b242dc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2018, \"end_char_idx\": 3262, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ed58bf15-b439-4775-bc1a-20c63798f827", "doc_id": "ed58bf15-b439-4775-bc1a-20c63798f827", "ref_doc_id": "ed58bf15-b439-4775-bc1a-20c63798f827", "chroma:document": "## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## ErnieConfig\n\n[[autodoc]] ErnieConfig\n    - all\n\n## Ernie specific outputs\n\n[[autodoc]] models.ernie.modeling_ernie.ErnieForPreTrainingOutput\n\n## ErnieModel\n\n[[autodoc]] ErnieModel\n    - forward\n\n## ErnieForPreTraining\n\n[[autodoc]] ErnieForPreTraining\n    - forward\n\n## ErnieForCausalLM\n\n[[autodoc]] ErnieForCausalLM\n    - forward\n\n## ErnieForMaskedLM\n\n[[autodoc]] ErnieForMaskedLM\n    - forward\n\n## ErnieForNextSentencePrediction\n\n[[autodoc]] ErnieForNextSentencePrediction\n    - forward\n\n## ErnieForSequenceClassification\n\n[[autodoc]] ErnieForSequenceClassification\n    - forward\n\n## ErnieForMultipleChoice\n\n[[autodoc]] ErnieForMultipleChoice\n    - forward\n\n## ErnieForTokenClassification\n\n[[autodoc]] ErnieForTokenClassification\n    - forward\n\n## ErnieForQuestionAnswering\n\n[[autodoc]] ErnieForQuestionAnswering\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 2d90a89c-ee78-463c-83bb-e92660b761e7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/clip", "title": "CLIP", "tokens": 4185, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2d90a89c-ee78-463c-83bb-e92660b761e7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8f44dcc4-6d1f-49e7-b97c-8476b34e727c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"291c95f788715055df474b490aef232932985fb6bf7910bb91bf4c07a1afa296\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"46fe9f12-afe9-45a6-81b2-ca8676d10a3a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d3318ccb681b9d82e824d6673cd36977ebfc8fc580b91791bef9370aa4a7e2a6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3522, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8f44dcc4-6d1f-49e7-b97c-8476b34e727c", "doc_id": "8f44dcc4-6d1f-49e7-b97c-8476b34e727c", "ref_doc_id": "8f44dcc4-6d1f-49e7-b97c-8476b34e727c", "chroma:document": "# CLIP\n\n## Overview\n\nThe CLIP model was proposed in [Learning Transferable Visual Models From Natural Language Supervision](https://arxiv.org/abs/2103.00020) by Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh,\nSandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever. CLIP\n(Contrastive Language-Image Pre-Training) is a neural network trained on a variety of (image, text) pairs. It can be\ninstructed in natural language to predict the most relevant text snippet, given an image, without directly optimizing\nfor the task, similarly to the zero-shot capabilities of GPT-2 and 3.\n\nThe abstract from the paper is the following:\n\n*State-of-the-art computer vision systems are trained to predict a fixed set of predetermined object categories. This\nrestricted form of supervision limits their generality and usability since additional labeled data is needed to specify\nany other visual concept. Learning directly from raw text about images is a promising alternative which leverages a\nmuch broader source of supervision. We demonstrate that the simple pre-training task of predicting which caption goes\nwith which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of 400\nmillion (image, text) pairs collected from the internet. After pre-training, natural language is used to reference\nlearned visual concepts (or describe new ones) enabling zero-shot transfer of the model to downstream tasks. We study\nthe performance of this approach by benchmarking on over 30 different existing computer vision datasets, spanning tasks\nsuch as OCR, action recognition in videos, geo-localization, and many types of fine-grained object classification. The\nmodel transfers non-trivially to most tasks and is often competitive with a fully supervised baseline without the need\nfor any dataset specific training. For instance, we match the accuracy of the original ResNet-50 on ImageNet zero-shot\nwithout needing to use any of the 1.28 million training examples it was trained on. We release our code and pre-trained\nmodel weights at this https URL.*\n\nThis model was contributed by [valhalla](https://huggingface.co/valhalla). The original code can be found [here](https://github.com/openai/CLIP).\n\n## Usage tips and example\n\nCLIP is a multi-modal vision and language model. It can be used for image-text similarity and for zero-shot image\nclassification. CLIP uses a ViT like transformer to get visual features and a causal language model to get the text\nfeatures. Both the text and visual features are then projected to a latent space with identical dimension. The dot\nproduct between the projected image and text features is then used as a similar score.\n\nTo feed images to the Transformer encoder, each image is split into a sequence of fixed-size non-overlapping patches,\nwhich are then linearly embedded. A [CLS] token is added to serve as representation of an entire image. The authors\nalso add absolute position embeddings, and feed the resulting sequence of vectors to a standard Transformer encoder.\nThe [`CLIPImageProcessor`] can be used to resize (or rescale) and normalize images for the model.\n\nThe [`CLIPTokenizer`] is used to encode the text. The [`CLIPProcessor`] wraps\n[`CLIPImageProcessor`] and [`CLIPTokenizer`] into a single instance to both\nencode the text and prepare the images. The following example shows how to get the image-text similarity scores using\n[`CLIPProcessor`] and [`CLIPModel`]."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 46fe9f12-afe9-45a6-81b2-ca8676d10a3a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/clip", "title": "CLIP", "tokens": 4185, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"46fe9f12-afe9-45a6-81b2-ca8676d10a3a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8f44dcc4-6d1f-49e7-b97c-8476b34e727c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"291c95f788715055df474b490aef232932985fb6bf7910bb91bf4c07a1afa296\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2d90a89c-ee78-463c-83bb-e92660b761e7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"94e263657d40bad369389e7e2eae36785ad87fcc6599bb5c6ab5eb6ae6e2be8a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0fa1360f-9030-468b-9a40-c2fa4ab385bf\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8c9b609ffaa33c00e18c148884ad24c43adb5f9aa490f301120f904ce5aceecf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3525, \"end_char_idx\": 6715, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8f44dcc4-6d1f-49e7-b97c-8476b34e727c", "doc_id": "8f44dcc4-6d1f-49e7-b97c-8476b34e727c", "ref_doc_id": "8f44dcc4-6d1f-49e7-b97c-8476b34e727c", "chroma:document": "```python\n>>> from PIL import Image\n>>> import requests\n\n>>> from transformers import CLIPProcessor, CLIPModel\n\n>>> model = CLIPModel.from_pretrained(\"openai/clip-vit-base-patch32\")\n>>> processor = CLIPProcessor.from_pretrained(\"openai/clip-vit-base-patch32\")\n\n>>> url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n\n>>> inputs = processor(text=[\"a photo of a cat\", \"a photo of a dog\"], images=image, return_tensors=\"pt\", padding=True)\n\n>>> outputs = model(**inputs)\n>>> logits_per_image = outputs.logits_per_image  # this is the image-text similarity score\n>>> probs = logits_per_image.softmax(dim=1)  # we can take the softmax to get the label probabilities\n```\n\n\n### Combining CLIP and Flash Attention 2\n\nFirst, make sure to install the latest version of Flash Attention 2.\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nMake also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of flash-attn repository. Make also sure to load your model in half-precision (e.g. `torch.float16`)\n\n<Tip warning={true}>\n\nFor small batch sizes, you might notice a slowdown in your model when using flash attention. Refer to the section [Expected speedups with Flash Attention and SDPA](#Expected-speedups-with-Flash-Attention-and-SDPA) below and select an appropriate attention implementation.\n\n</Tip>\n\nTo load and run a model using Flash Attention 2, refer to the snippet below:\n\n```python\n>>> import torch\n>>> import requests\n>>> from PIL import Image\n\n>>> from transformers import CLIPProcessor, CLIPModel\n\n>>> device = \"cuda\"\n>>> torch_dtype = torch.float16\n\n>>> model = CLIPModel.from_pretrained(\n...     \"openai/clip-vit-base-patch32\",\n...     attn_implementation=\"flash_attention_2\",\n...     device_map=device,\n...     torch_dtype=torch_dtype,\n... )\n>>> processor = CLIPProcessor.from_pretrained(\"openai/clip-vit-base-patch32\")\n\n>>> url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n\n>>> inputs = processor(text=[\"a photo of a cat\", \"a photo of a dog\"], images=image, return_tensors=\"pt\", padding=True)\n>>> inputs.to(device)\n\n>>> with torch.no_grad():\n...     with torch.autocast(device):\n...         outputs = model(**inputs)\n\n>>> logits_per_image = outputs.logits_per_image  # this is the image-text similarity score\n>>> probs = logits_per_image.softmax(dim=1)  # we can take the softmax to get the label probabilities\n>>> print(probs)\ntensor([[0.9946, 0.0052]], device='cuda:0', dtype=torch.float16)\n```\n\n\n### Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0fa1360f-9030-468b-9a40-c2fa4ab385bf
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/clip", "title": "CLIP", "tokens": 4185, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0fa1360f-9030-468b-9a40-c2fa4ab385bf\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8f44dcc4-6d1f-49e7-b97c-8476b34e727c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"291c95f788715055df474b490aef232932985fb6bf7910bb91bf4c07a1afa296\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"46fe9f12-afe9-45a6-81b2-ca8676d10a3a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d053d7668e2b8edfe7357b56dc60742838e5d4762e6f5b4aaf30eb70e90d4785\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7d1cdb45-878d-4ec3-9556-e8fae0b0aedb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"52da01a60c34814bc5363d1e8e3352c2f893e8e81bcac1d0ac62ec43a933a6f7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6156, \"end_char_idx\": 8282, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8f44dcc4-6d1f-49e7-b97c-8476b34e727c", "doc_id": "8f44dcc4-6d1f-49e7-b97c-8476b34e727c", "ref_doc_id": "8f44dcc4-6d1f-49e7-b97c-8476b34e727c", "chroma:document": "### Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```python\nfrom transformers import CLIPModel\n\nmodel = CLIPModel.from_pretrained(\"openai/clip-vit-base-patch32\", torch_dtype=torch.float16, attn_implementation=\"sdpa\")\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\n### Expected speedups with Flash Attention and SDPA\n\nOn a local benchmark (NVIDIA A10G, PyTorch 2.3.1+cu121) with `float16`, we saw the following speedups during inference for `\"openai/clip-vit-large-patch14\"` checkpoint ([code](https://gist.github.com/qubvel/ac691a54e54f9fae8144275f866a7ff8)):\n\n#### CLIPTextModel\n\n|   Num text labels |   Eager (s/iter) |   FA2 (s/iter) |   FA2 speedup |   SDPA (s/iter) |   SDPA speedup |\n|------------------:|-----------------:|---------------:|--------------:|----------------:|---------------:|\n|                 4 |            0.009 |          0.012 |         0.737 |           0.007 |          1.269 |\n|                16 |            0.009 |          0.014 |         0.659 |           0.008 |          1.187 |\n|                32 |            0.018 |          0.021 |         0.862 |           0.016 |          1.142 |\n|                64 |            0.034 |          0.034 |         1.001 |           0.03  |          1.163 |\n|               128 |            0.063 |          0.058 |         1.09  |           0.054 |          1.174 |\n\n!"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 7d1cdb45-878d-4ec3-9556-e8fae0b0aedb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/clip", "title": "CLIP", "tokens": 4185, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7d1cdb45-878d-4ec3-9556-e8fae0b0aedb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8f44dcc4-6d1f-49e7-b97c-8476b34e727c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"291c95f788715055df474b490aef232932985fb6bf7910bb91bf4c07a1afa296\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0fa1360f-9030-468b-9a40-c2fa4ab385bf\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"81397ee65a5facb4c0e79eda0373bd9e270fbc64523048e141e81b47c6904dfc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9a33a102-a0e3-4892-a7ad-34c0784570a5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3e258336f1bf3c9df50ab201ba1d8fa3c945905a818c294594b7f92f29f4d26b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7161, \"end_char_idx\": 9072, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8f44dcc4-6d1f-49e7-b97c-8476b34e727c", "doc_id": "8f44dcc4-6d1f-49e7-b97c-8476b34e727c", "ref_doc_id": "8f44dcc4-6d1f-49e7-b97c-8476b34e727c", "chroma:document": "`torch.float16` or `torch.bfloat16`).\n\n### Expected speedups with Flash Attention and SDPA\n\nOn a local benchmark (NVIDIA A10G, PyTorch 2.3.1+cu121) with `float16`, we saw the following speedups during inference for `\"openai/clip-vit-large-patch14\"` checkpoint ([code](https://gist.github.com/qubvel/ac691a54e54f9fae8144275f866a7ff8)):\n\n#### CLIPTextModel\n\n|   Num text labels |   Eager (s/iter) |   FA2 (s/iter) |   FA2 speedup |   SDPA (s/iter) |   SDPA speedup |\n|------------------:|-----------------:|---------------:|--------------:|----------------:|---------------:|\n|                 4 |            0.009 |          0.012 |         0.737 |           0.007 |          1.269 |\n|                16 |            0.009 |          0.014 |         0.659 |           0.008 |          1.187 |\n|                32 |            0.018 |          0.021 |         0.862 |           0.016 |          1.142 |\n|                64 |            0.034 |          0.034 |         1.001 |           0.03  |          1.163 |\n|               128 |            0.063 |          0.058 |         1.09  |           0.054 |          1.174 |\n\n![clip_text_model_viz_3](https://github.com/user-attachments/assets/e9826b43-4e66-4f4c-952b-af4d90bd38eb)\n\n#### CLIPVisionModel\n\n|   Image batch size |   Eager (s/iter) |   FA2 (s/iter) |   FA2 speedup |   SDPA (s/iter) |   SDPA speedup |\n|-------------------:|-----------------:|---------------:|--------------:|----------------:|---------------:|\n|                  1 |            0.016 |          0.013 |         1.247 |           0.012 |          1.318 |\n|                  4 |            0.025 |          0.021 |         1.198 |           0.021 |          1.202 |\n|                 16 |            0.093 |          0.075 |         1.234 |           0.075 |          1.24  |\n|                 32 |            0.181 |          0.147 |         1.237 |           0.146 |          1.241 |\n\n!"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 9a33a102-a0e3-4892-a7ad-34c0784570a5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/clip", "title": "CLIP", "tokens": 4185, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9a33a102-a0e3-4892-a7ad-34c0784570a5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8f44dcc4-6d1f-49e7-b97c-8476b34e727c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"291c95f788715055df474b490aef232932985fb6bf7910bb91bf4c07a1afa296\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7d1cdb45-878d-4ec3-9556-e8fae0b0aedb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a3616684fe5bb8f096bda617d6142ec1a406bf5dc9001070e52255a3fba4e7a9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c5f3c51e-9f20-4c2d-92a5-0de890291449\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"eb72e2bfcb67fdec0622a02fda7064d4fcff0228fc7028797017f946b95a1a04\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8282, \"end_char_idx\": 11137, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8f44dcc4-6d1f-49e7-b97c-8476b34e727c", "doc_id": "8f44dcc4-6d1f-49e7-b97c-8476b34e727c", "ref_doc_id": "8f44dcc4-6d1f-49e7-b97c-8476b34e727c", "chroma:document": "[clip_text_model_viz_3](https://github.com/user-attachments/assets/e9826b43-4e66-4f4c-952b-af4d90bd38eb)\n\n#### CLIPVisionModel\n\n|   Image batch size |   Eager (s/iter) |   FA2 (s/iter) |   FA2 speedup |   SDPA (s/iter) |   SDPA speedup |\n|-------------------:|-----------------:|---------------:|--------------:|----------------:|---------------:|\n|                  1 |            0.016 |          0.013 |         1.247 |           0.012 |          1.318 |\n|                  4 |            0.025 |          0.021 |         1.198 |           0.021 |          1.202 |\n|                 16 |            0.093 |          0.075 |         1.234 |           0.075 |          1.24  |\n|                 32 |            0.181 |          0.147 |         1.237 |           0.146 |          1.241 |\n\n![clip_image_model_viz_3](https://github.com/user-attachments/assets/50a36206-e3b9-4adc-ac8e-926b8b071d63)\n\n#### CLIPModel\n\n|   Image batch size |   Num text labels |   Eager (s/iter) |   FA2 (s/iter) |   FA2 speedup |   SDPA (s/iter) |   SDPA speedup |\n|-------------------:|------------------:|-----------------:|---------------:|--------------:|----------------:|---------------:|\n|                  1 |                 4 |            0.025 |          0.026 |         0.954 |           0.02  |          1.217 |\n|                  1 |                16 |            0.026 |          0.028 |         0.918 |           0.02  |          1.287 |\n|                  1 |                64 |            0.042 |          0.046 |         0.906 |           0.036 |          1.167 |\n|                  4 |                 4 |            0.028 |          0.033 |         0.849 |           0.024 |          1.189 |\n|                  4 |                16 |            0.034 |          0.035 |         0.955 |           0.029 |          1.169 |\n|                  4 |                64 |            0.059 |          0.055 |         1.072 |           0.05  |          1.179 |\n|                 16 |                 4 |            0.096 |          0.088 |         1.091 |           0.078 |          1.234 |\n|                 16 |                16 |            0.102 |          0.09  |         1.129 |           0.083 |          1.224 |\n|                 16 |                64 |            0.127 |          0.11  |         1.157 |           0.105 |          1.218 |\n|                 32 |                 4 |            0.185 |          0.159 |         1.157 |           0.149 |          1.238 |\n|                 32 |                16 |            0.19  |          0.162 |         1.177 |           0.154 |          1.233 |\n|                 32 |                64 |            0.216 |          0.181 |         1.19  |           0.176 |          1.228 |\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with CLIP."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: c5f3c51e-9f20-4c2d-92a5-0de890291449
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/clip", "title": "CLIP", "tokens": 4185, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c5f3c51e-9f20-4c2d-92a5-0de890291449\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8f44dcc4-6d1f-49e7-b97c-8476b34e727c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"291c95f788715055df474b490aef232932985fb6bf7910bb91bf4c07a1afa296\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9a33a102-a0e3-4892-a7ad-34c0784570a5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"791b7f8bfddfa2ac95576f50d51180ee7425a77a9cc99ee445f1a281a22025d1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"60778d5b-0dd4-4328-8c8a-845618720254\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"596220c249636460c7095edfb37084f21d7f4482dfe4c66836377ac426b2cec0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11139, \"end_char_idx\": 13263, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8f44dcc4-6d1f-49e7-b97c-8476b34e727c", "doc_id": "8f44dcc4-6d1f-49e7-b97c-8476b34e727c", "ref_doc_id": "8f44dcc4-6d1f-49e7-b97c-8476b34e727c", "chroma:document": "- [Fine tuning CLIP with Remote Sensing (Satellite) images and captions](https://huggingface.co/blog/fine-tune-clip-rsicd), a blog post about how to fine-tune CLIP with [RSICD dataset](https://github.com/201528014227051/RSICD_optimal) and comparison of performance changes due to data augmentation.\n- This [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/contrastive-image-text) shows how to train a CLIP-like vision-text dual encoder model using a pre-trained vision and text encoder using [COCO dataset](https://cocodataset.org/#home).\n\n<PipelineTag pipeline=\"image-to-text\"/>\n\n- A [notebook](https://colab.research.google.com/drive/1tuoAC5F4sC7qid56Z0ap-stR3rwdk0ZV?usp=sharing) on how to use a pretrained CLIP for inference with beam search for image captioning. \ud83c\udf0e\n\n**Image retrieval**\n\n- A [notebook](https://colab.research.google.com/drive/1bLVwVKpAndpEDHqjzxVPr_9nGrSbuOQd?usp=sharing) on image retrieval using pretrained CLIP and computing MRR(Mean Reciprocal Rank) score. \ud83c\udf0e\n- A [notebook](https://colab.research.google.com/github/deep-diver/image_search_with_natural_language/blob/main/notebooks/Image_Search_CLIP.ipynb) on image retrieval and showing the similarity score. \ud83c\udf0e\n- A [notebook](https://colab.research.google.com/drive/1xO-wC_m_GNzgjIBQ4a4znvQkvDoZJvH4?usp=sharing) on how to map images and texts to the same vector space using Multilingual CLIP. \ud83c\udf0e \n- A [notebook](https://colab.research.google.com/github/vivien000/clip-demo/blob/master/clip.ipynb#scrollTo=uzdFhRGqiWkR) on how to run CLIP on semantic image search using [Unsplash](https://unsplash.com) and [TMDB](https://www.themoviedb.org/) datasets. \ud83c\udf0e\n\n**Explainability**\n\n- A [notebook](https://colab.research.google.com/github/hila-chefer/Transformer-MM-Explainability/blob/main/CLIP_explainability.ipynb) on how to visualize similarity between input token and image segment. \ud83c\udf0e\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it.\nThe resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 60778d5b-0dd4-4328-8c8a-845618720254
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/clip", "title": "CLIP", "tokens": 4185, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"60778d5b-0dd4-4328-8c8a-845618720254\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8f44dcc4-6d1f-49e7-b97c-8476b34e727c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"291c95f788715055df474b490aef232932985fb6bf7910bb91bf4c07a1afa296\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c5f3c51e-9f20-4c2d-92a5-0de890291449\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/clip\", \"title\": \"CLIP\", \"tokens\": 4185, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8d26bbed2db0a418fddd584e0fcdbbf5c27ba88d4f831f6c0ac417b42adc8c9f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11945, \"end_char_idx\": 14972, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8f44dcc4-6d1f-49e7-b97c-8476b34e727c", "doc_id": "8f44dcc4-6d1f-49e7-b97c-8476b34e727c", "ref_doc_id": "8f44dcc4-6d1f-49e7-b97c-8476b34e727c", "chroma:document": "\ud83c\udf0e\n\n**Image retrieval**\n\n- A [notebook](https://colab.research.google.com/drive/1bLVwVKpAndpEDHqjzxVPr_9nGrSbuOQd?usp=sharing) on image retrieval using pretrained CLIP and computing MRR(Mean Reciprocal Rank) score. \ud83c\udf0e\n- A [notebook](https://colab.research.google.com/github/deep-diver/image_search_with_natural_language/blob/main/notebooks/Image_Search_CLIP.ipynb) on image retrieval and showing the similarity score. \ud83c\udf0e\n- A [notebook](https://colab.research.google.com/drive/1xO-wC_m_GNzgjIBQ4a4znvQkvDoZJvH4?usp=sharing) on how to map images and texts to the same vector space using Multilingual CLIP. \ud83c\udf0e \n- A [notebook](https://colab.research.google.com/github/vivien000/clip-demo/blob/master/clip.ipynb#scrollTo=uzdFhRGqiWkR) on how to run CLIP on semantic image search using [Unsplash](https://unsplash.com) and [TMDB](https://www.themoviedb.org/) datasets. \ud83c\udf0e\n\n**Explainability**\n\n- A [notebook](https://colab.research.google.com/github/hila-chefer/Transformer-MM-Explainability/blob/main/CLIP_explainability.ipynb) on how to visualize similarity between input token and image segment. \ud83c\udf0e\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it.\nThe resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## CLIPConfig\n\n[[autodoc]] CLIPConfig\n    - from_text_vision_configs\n\n## CLIPTextConfig\n\n[[autodoc]] CLIPTextConfig\n\n## CLIPVisionConfig\n\n[[autodoc]] CLIPVisionConfig\n\n## CLIPTokenizer\n\n[[autodoc]] CLIPTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## CLIPTokenizerFast\n\n[[autodoc]] CLIPTokenizerFast\n\n## CLIPImageProcessor\n\n[[autodoc]] CLIPImageProcessor\n    - preprocess\n\n## CLIPFeatureExtractor\n\n[[autodoc]] CLIPFeatureExtractor\n\n## CLIPProcessor\n\n[[autodoc]] CLIPProcessor\n\n<frameworkcontent>\n<pt>\n\n## CLIPModel\n\n[[autodoc]] CLIPModel\n    - forward\n    - get_text_features\n    - get_image_features\n\n## CLIPTextModel\n\n[[autodoc]] CLIPTextModel\n    - forward\n\n## CLIPTextModelWithProjection\n\n[[autodoc]] CLIPTextModelWithProjection\n    - forward\n\n## CLIPVisionModelWithProjection\n\n[[autodoc]] CLIPVisionModelWithProjection\n    - forward\n\n## CLIPVisionModel\n\n[[autodoc]] CLIPVisionModel\n    - forward\n\n## CLIPForImageClassification\n\n[[autodoc]] CLIPForImageClassification\n    - forward\n\n</pt>\n<tf>\n\n## TFCLIPModel\n\n[[autodoc]] TFCLIPModel\n    - call\n    - get_text_features\n    - get_image_features\n\n## TFCLIPTextModel\n\n[[autodoc]] TFCLIPTextModel\n    - call\n\n## TFCLIPVisionModel\n\n[[autodoc]] TFCLIPVisionModel\n    - call\n\n</tf>\n<jax>\n\n## FlaxCLIPModel\n\n[[autodoc]] FlaxCLIPModel\n    - __call__\n    - get_text_features\n    - get_image_features\n\n## FlaxCLIPTextModel\n\n[[autodoc]] FlaxCLIPTextModel\n    - __call__\n\n## FlaxCLIPTextModelWithProjection\n\n[[autodoc]] FlaxCLIPTextModelWithProjection\n    - __call__\n\n## FlaxCLIPVisionModel\n\n[[autodoc]] FlaxCLIPVisionModel\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 61343643-790f-490f-a482-24c238485e33
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/rt_detr", "title": "RT-DETR", "tokens": 1617, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"61343643-790f-490f-a482-24c238485e33\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rt_detr\", \"title\": \"RT-DETR\", \"tokens\": 1617, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1c58242c-0298-4aa6-8653-d11be6767b2c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rt_detr\", \"title\": \"RT-DETR\", \"tokens\": 1617, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4566661e98d9827c0da46dcec5837c948b86667c5cb5005f1b768fe04adc594d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1407fcc8-94df-437c-9a03-5a01fae5fe1d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"20a1b526ff550aca1c48e43c3a93e72c0b853ce8b92f07de6a3a3e1f2bad3486\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3395, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1c58242c-0298-4aa6-8653-d11be6767b2c", "doc_id": "1c58242c-0298-4aa6-8653-d11be6767b2c", "ref_doc_id": "1c58242c-0298-4aa6-8653-d11be6767b2c", "chroma:document": "# RT-DETR\n\n## Overview\n\n\nThe RT-DETR model was proposed in [DETRs Beat YOLOs on Real-time Object Detection](https://arxiv.org/abs/2304.08069) by Wenyu Lv, Yian Zhao, Shangliang Xu, Jinman Wei, Guanzhong Wang, Cheng Cui, Yuning Du, Qingqing Dang, Yi Liu.\n\nRT-DETR is an object detection model that stands for \"Real-Time DEtection Transformer.\" This model is designed to perform object detection tasks with a focus on achieving real-time performance while maintaining high accuracy. Leveraging the transformer architecture, which has gained significant popularity in various fields of deep learning, RT-DETR processes images to identify and locate multiple objects within them.\n\nThe abstract from the paper is the following:\n\n*Recently, end-to-end transformer-based detectors (DETRs) have achieved remarkable performance. However, the issue of the high computational cost of DETRs has not been effectively addressed, limiting their practical application and preventing them from fully exploiting the benefits of no post-processing, such as non-maximum suppression (NMS). In this paper, we first analyze the influence of NMS in modern real-time object detectors on inference speed, and establish an end-to-end speed benchmark. To avoid the inference delay caused by NMS, we propose a Real-Time DEtection TRansformer (RT-DETR), the first real-time end-to-end object detector to our best knowledge. Specifically, we design an efficient hybrid encoder to efficiently process multi-scale features by decoupling the intra-scale interaction and cross-scale fusion, and propose IoU-aware query selection to improve the initialization of object queries. In addition, our proposed detector supports flexibly adjustment of the inference speed by using different decoder layers without the need for retraining, which facilitates the practical application of real-time object detectors. Our RT-DETR-L achieves 53.0% AP on COCO val2017 and 114 FPS on T4 GPU, while RT-DETR-X achieves 54.8% AP and 74 FPS, outperforming all YOLO detectors of the same scale in both speed and accuracy. Furthermore, our RT-DETR-R50 achieves 53.1% AP and 108 FPS, outperforming DINO-Deformable-DETR-R50 by 2.2% AP in accuracy and by about 21 times in FPS.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/rt_detr_overview.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> RT-DETR performance relative to YOLO models. Taken from the <a href=\"https://arxiv.org/abs/2304.08069\">original paper.</a> </small>\n\nThe model version was contributed by [rafaelpadilla](https://huggingface.co/rafaelpadilla) and [sangbumchoi](https://github.com/SangbumChoi). The original code can be found [here](https://github.com/lyuwenyu/RT-DETR/).\n\n\n## Usage tips\n\nInitially, an image is processed using a pre-trained convolutional neural network, specifically a Resnet-D variant as referenced in the original code. This network extracts features from the final three layers of the architecture. Following this, a hybrid encoder is employed to convert the multi-scale features into a sequential array of image features. Then, a decoder, equipped with auxiliary prediction heads is used to refine the object queries. This process facilitates the direct generation of bounding boxes, eliminating the need for any additional post-processing to acquire the logits and coordinates for the bounding boxes."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 1407fcc8-94df-437c-9a03-5a01fae5fe1d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/rt_detr", "title": "RT-DETR", "tokens": 1617, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1407fcc8-94df-437c-9a03-5a01fae5fe1d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rt_detr\", \"title\": \"RT-DETR\", \"tokens\": 1617, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1c58242c-0298-4aa6-8653-d11be6767b2c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rt_detr\", \"title\": \"RT-DETR\", \"tokens\": 1617, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4566661e98d9827c0da46dcec5837c948b86667c5cb5005f1b768fe04adc594d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"61343643-790f-490f-a482-24c238485e33\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rt_detr\", \"title\": \"RT-DETR\", \"tokens\": 1617, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"96486a55d6008d5a1a1f6d7f5c39cebd379801f071e5c4066738824246b12b9b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b682bb91-40dd-443e-b493-f4c3ae32b226\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"63d76cb9bd536d7c10a59dcb217146198c9293fd5cfc6d8daa9a51a5588c24e0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2746, \"end_char_idx\": 5264, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1c58242c-0298-4aa6-8653-d11be6767b2c", "doc_id": "1c58242c-0298-4aa6-8653-d11be6767b2c", "ref_doc_id": "1c58242c-0298-4aa6-8653-d11be6767b2c", "chroma:document": "## Usage tips\n\nInitially, an image is processed using a pre-trained convolutional neural network, specifically a Resnet-D variant as referenced in the original code. This network extracts features from the final three layers of the architecture. Following this, a hybrid encoder is employed to convert the multi-scale features into a sequential array of image features. Then, a decoder, equipped with auxiliary prediction heads is used to refine the object queries. This process facilitates the direct generation of bounding boxes, eliminating the need for any additional post-processing to acquire the logits and coordinates for the bounding boxes.\n\n```py\n>>> import torch\n>>> import requests\n\n>>> from PIL import Image\n>>> from transformers import RTDetrForObjectDetection, RTDetrImageProcessor\n\n>>> url = 'http://images.cocodataset.org/val2017/000000039769.jpg' \n>>> image = Image.open(requests.get(url, stream=True).raw)\n\n>>> image_processor = RTDetrImageProcessor.from_pretrained(\"PekingU/rtdetr_r50vd\")\n>>> model = RTDetrForObjectDetection.from_pretrained(\"PekingU/rtdetr_r50vd\")\n\n>>> inputs = image_processor(images=image, return_tensors=\"pt\")\n\n>>> with torch.no_grad():\n...     outputs = model(**inputs)\n\n>>> results = image_processor.post_process_object_detection(outputs, target_sizes=torch.tensor([image.size[::-1]]), threshold=0.3)\n\n>>> for result in results:\n...     for score, label_id, box in zip(result[\"scores\"], result[\"labels\"], result[\"boxes\"]):\n...         score, label = score.item(), label_id.item()\n...         box = [round(i, 2) for i in box.tolist()]\n...         print(f\"{model.config.id2label[label]}: {score:.2f} {box}\")\nsofa: 0.97 [0.14, 0.38, 640.13, 476.21]\ncat: 0.96 [343.38, 24.28, 640.14, 371.5]\ncat: 0.96 [13.23, 54.18, 318.98, 472.22]\nremote: 0.95 [40.11, 73.44, 175.96, 118.48]\nremote: 0.92 [333.73, 76.58, 369.97, 186.99]\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with RT-DETR.\n\n<PipelineTag pipeline=\"object-detection\"/>\n\n- Scripts for finetuning [`RTDetrForObjectDetection`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/object-detection).\n- See also: [Object detection task guide](../tasks/object_detection).\n- Notebooks regarding inference and fine-tuning RT-DETR on a custom dataset can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/RT-DETR)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: b682bb91-40dd-443e-b493-f4c3ae32b226
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/rt_detr", "title": "RT-DETR", "tokens": 1617, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b682bb91-40dd-443e-b493-f4c3ae32b226\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rt_detr\", \"title\": \"RT-DETR\", \"tokens\": 1617, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1c58242c-0298-4aa6-8653-d11be6767b2c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rt_detr\", \"title\": \"RT-DETR\", \"tokens\": 1617, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4566661e98d9827c0da46dcec5837c948b86667c5cb5005f1b768fe04adc594d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1407fcc8-94df-437c-9a03-5a01fae5fe1d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rt_detr\", \"title\": \"RT-DETR\", \"tokens\": 1617, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fcc231773a5887d7cd076c79cc99f779dd2972ca401939b8d0efcdb21b772a96\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4737, \"end_char_idx\": 5685, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1c58242c-0298-4aa6-8653-d11be6767b2c", "doc_id": "1c58242c-0298-4aa6-8653-d11be6767b2c", "ref_doc_id": "1c58242c-0298-4aa6-8653-d11be6767b2c", "chroma:document": "<PipelineTag pipeline=\"object-detection\"/>\n\n- Scripts for finetuning [`RTDetrForObjectDetection`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/object-detection).\n- See also: [Object detection task guide](../tasks/object_detection).\n- Notebooks regarding inference and fine-tuning RT-DETR on a custom dataset can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/RT-DETR). \ud83c\udf0e\n\n## RTDetrConfig\n\n[[autodoc]] RTDetrConfig\n\n## RTDetrResNetConfig\n\n[[autodoc]] RTDetrResNetConfig\n\n## RTDetrImageProcessor\n\n[[autodoc]] RTDetrImageProcessor\n    - preprocess\n    - post_process_object_detection\n\n## RTDetrModel\n\n[[autodoc]] RTDetrModel\n    - forward\n\n## RTDetrForObjectDetection\n\n[[autodoc]] RTDetrForObjectDetection\n    - forward\n\n## RTDetrResNetBackbone\n\n[[autodoc]] RTDetrResNetBackbone\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 652efb6b-928f-4646-aacf-f4e1bf0b8062
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/idefics2", "title": "Idefics2", "tokens": 2631, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"652efb6b-928f-4646-aacf-f4e1bf0b8062\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a5d000ad-7c35-4da1-b3c7-bafbb176df11\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fcc5f689170f580aa6e99beb8b9b83ce1ecb683ea2f11aa04efc3cd919ea891e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"29bf7c35-e5e2-443e-abb4-d7247c798398\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2e9c2619905eedbbd224c82c2e3dc0934d19acbf355c347fa2f378f811d987e6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3150, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a5d000ad-7c35-4da1-b3c7-bafbb176df11", "doc_id": "a5d000ad-7c35-4da1-b3c7-bafbb176df11", "ref_doc_id": "a5d000ad-7c35-4da1-b3c7-bafbb176df11", "chroma:document": "# Idefics2\n\n## Overview\n\nThe Idefics2 model was proposed in [What matters when building vision-language models?](https://arxiv.org/abs/2405.02246) by L\u00e9o Tronchon, Hugo Laurencon, Victor Sanh. The accompanying blog post can be found [here](https://huggingface.co/blog/idefics2).\n\nIdefics2 is an open multimodal model that accepts arbitrary sequences of image and text inputs and produces text\noutputs. The model can answer questions about images, describe visual content, create stories grounded on multiple\nimages, or simply behave as a pure language model without visual inputs. It improves upon IDEFICS-1, notably on\ndocument understanding, OCR, or visual reasoning. Idefics2 is lightweight (8 billion parameters) and treats\nimages in their native aspect ratio and resolution, which allows for varying inference efficiency.\n\nThe abstract from the paper is the following:\n\n*The growing interest in vision-language models (VLMs) has been driven by improvements in large language models and vision transformers. Despite the abundance of literature on this subject, we observe that critical decisions regarding the design of VLMs are often not justified. We argue that these unsupported decisions impede progress in the field by making it difficult to identify which choices improve model performance. To address this issue, we conduct extensive experiments around pre-trained models, architecture choice, data, and training methods. Our consolidation of findings includes the development of Idefics2, an efficient foundational VLM of 8 billion parameters. Idefics2 achieves state-of-the-art performance within its size category across various multimodal benchmarks, and is often on par with models four times its size. We release the model (base, instructed, and chat) along with the datasets created for its training.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/idefics2_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Idefics2 architecture. Taken from the <a href=\"https://arxiv.org/abs/2405.02246\">original paper.</a> </small>\n\nThis model was contributed by [amyeroberts](https://huggingface.co/amyeroberts).\nThe original code can be found [here](https://huggingface.co/HuggingFaceM4/idefics2).\n\n## Usage tips\n\n- Each sample can contain multiple images, and the number of images can vary between samples. The processor will pad the inputs to the maximum number of images in a batch for input to the model.\n- The processor has a `do_image_splitting` option. If `True`, each input image will be split into 4 sub-images, and concatenated with the original to form 5 images. This is useful for increasing model performance. Make sure `processor.image_processor.do_image_splitting` is set to `False` if the model was not trained with this option.\n- `text` passed to the processor should have the `<image>` tokens where the images should be inserted. And `<end_of_utterance>` at the end of each utterance if the text is a chat message.\n- The processor has its own `apply_chat_template` method to convert chat messages to text that can then be passed as `text` to the processor."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 29bf7c35-e5e2-443e-abb4-d7247c798398
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/idefics2", "title": "Idefics2", "tokens": 2631, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"29bf7c35-e5e2-443e-abb4-d7247c798398\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a5d000ad-7c35-4da1-b3c7-bafbb176df11\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fcc5f689170f580aa6e99beb8b9b83ce1ecb683ea2f11aa04efc3cd919ea891e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"652efb6b-928f-4646-aacf-f4e1bf0b8062\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bff7a27590632959e0b83494b7c36d62c8b61ffa38cdc935b4ef15ccd3d98654\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"adcee4fe-68c7-411a-b5ce-43b9f784c01f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"73f6cfa1b153a27e065e883bbd18df2da0e1c696748e6169faa538c2ce171e27\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1556, \"end_char_idx\": 3813, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a5d000ad-7c35-4da1-b3c7-bafbb176df11", "doc_id": "a5d000ad-7c35-4da1-b3c7-bafbb176df11", "ref_doc_id": "a5d000ad-7c35-4da1-b3c7-bafbb176df11", "chroma:document": "Idefics2 achieves state-of-the-art performance within its size category across various multimodal benchmarks, and is often on par with models four times its size. We release the model (base, instructed, and chat) along with the datasets created for its training.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/idefics2_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Idefics2 architecture. Taken from the <a href=\"https://arxiv.org/abs/2405.02246\">original paper.</a> </small>\n\nThis model was contributed by [amyeroberts](https://huggingface.co/amyeroberts).\nThe original code can be found [here](https://huggingface.co/HuggingFaceM4/idefics2).\n\n## Usage tips\n\n- Each sample can contain multiple images, and the number of images can vary between samples. The processor will pad the inputs to the maximum number of images in a batch for input to the model.\n- The processor has a `do_image_splitting` option. If `True`, each input image will be split into 4 sub-images, and concatenated with the original to form 5 images. This is useful for increasing model performance. Make sure `processor.image_processor.do_image_splitting` is set to `False` if the model was not trained with this option.\n- `text` passed to the processor should have the `<image>` tokens where the images should be inserted. And `<end_of_utterance>` at the end of each utterance if the text is a chat message.\n- The processor has its own `apply_chat_template` method to convert chat messages to text that can then be passed as `text` to the processor.\n\nExample of how to use the processor on chat messages:\n\n```python\nimport requests\nfrom PIL import Image\nfrom transformers import Idefics2Processor, Idefics2ForConditionalGeneration\nimport torch\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n\nurl_1 = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nurl_2 = \"http://images.cocodataset.org/val2017/000000219578.jpg\"\n\nimage_1 = Image.open(requests.get(url_1, stream=True).raw)\nimage_2 = Image.open(requests.get(url_2, stream=True).raw)\nimages = [image_1, image_2]\n\nmessages = [{\n    \"role\": \"user\",\n    \"content\": [\n        {\"type\": \"text\", \"text\": \"What\u2019s the difference between these two images?"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: adcee4fe-68c7-411a-b5ce-43b9f784c01f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/idefics2", "title": "Idefics2", "tokens": 2631, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"adcee4fe-68c7-411a-b5ce-43b9f784c01f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a5d000ad-7c35-4da1-b3c7-bafbb176df11\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fcc5f689170f580aa6e99beb8b9b83ce1ecb683ea2f11aa04efc3cd919ea891e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"29bf7c35-e5e2-443e-abb4-d7247c798398\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0af494fee99b82f9019e53214f58bceacad7a7b1787948798d9cf31232df95ad\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"70264e9f-b04d-438f-a5ed-fbf66995f79e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b3466f9230102948534028cf5a078bdf96b5e6d0f73e2695f6dd430cbc1e5387\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2275, \"end_char_idx\": 4803, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a5d000ad-7c35-4da1-b3c7-bafbb176df11", "doc_id": "a5d000ad-7c35-4da1-b3c7-bafbb176df11", "ref_doc_id": "a5d000ad-7c35-4da1-b3c7-bafbb176df11", "chroma:document": "## Usage tips\n\n- Each sample can contain multiple images, and the number of images can vary between samples. The processor will pad the inputs to the maximum number of images in a batch for input to the model.\n- The processor has a `do_image_splitting` option. If `True`, each input image will be split into 4 sub-images, and concatenated with the original to form 5 images. This is useful for increasing model performance. Make sure `processor.image_processor.do_image_splitting` is set to `False` if the model was not trained with this option.\n- `text` passed to the processor should have the `<image>` tokens where the images should be inserted. And `<end_of_utterance>` at the end of each utterance if the text is a chat message.\n- The processor has its own `apply_chat_template` method to convert chat messages to text that can then be passed as `text` to the processor.\n\nExample of how to use the processor on chat messages:\n\n```python\nimport requests\nfrom PIL import Image\nfrom transformers import Idefics2Processor, Idefics2ForConditionalGeneration\nimport torch\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n\nurl_1 = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nurl_2 = \"http://images.cocodataset.org/val2017/000000219578.jpg\"\n\nimage_1 = Image.open(requests.get(url_1, stream=True).raw)\nimage_2 = Image.open(requests.get(url_2, stream=True).raw)\nimages = [image_1, image_2]\n\nmessages = [{\n    \"role\": \"user\",\n    \"content\": [\n        {\"type\": \"text\", \"text\": \"What\u2019s the difference between these two images?\"},\n        {\"type\": \"image\"},\n        {\"type\": \"image\"},\n    ],\n}]\n\nprocessor = Idefics2Processor.from_pretrained(\"HuggingFaceM4/idefics2-8b\")\nmodel = Idefics2ForConditionalGeneration.from_pretrained(\"HuggingFaceM4/idefics2-8b\")\nmodel.to(device)\n\n# at inference time, one needs to pass `add_generation_prompt=True` in order to make sure the model completes the prompt\ntext = processor.apply_chat_template(messages, add_generation_prompt=True)\nprint(text)\n# 'User: What\u2019s the difference between these two images?<image><image><end_of_utterance>\\nAssistant:'\n\ninputs = processor(images=images, text=text, return_tensors=\"pt\").to(device)\n\ngenerated_text = model.generate(**inputs, max_new_tokens=500)\ngenerated_text = processor.batch_decode(generated_text, skip_special_tokens=True)[0]\nprint(\"Generated text:\", generated_text)\n```\n\n- During training, it's important to determine which tokens the model should not learn. For Idefics2, this typically comes down to the image and padding tokens."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 70264e9f-b04d-438f-a5ed-fbf66995f79e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/idefics2", "title": "Idefics2", "tokens": 2631, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"70264e9f-b04d-438f-a5ed-fbf66995f79e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a5d000ad-7c35-4da1-b3c7-bafbb176df11\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fcc5f689170f580aa6e99beb8b9b83ce1ecb683ea2f11aa04efc3cd919ea891e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"adcee4fe-68c7-411a-b5ce-43b9f784c01f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e006e2c3a11515a51dc9ca5b47b6a396372159ad50f52f19892e282ef9befb51\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f97ce71d-6cb2-4b64-9cb5-66020ffd0763\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"dd497a6a2c212fd436b639beda714e2a78ebf1d80259b40e5c353778051f3a4e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3813, \"end_char_idx\": 6898, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a5d000ad-7c35-4da1-b3c7-bafbb176df11", "doc_id": "a5d000ad-7c35-4da1-b3c7-bafbb176df11", "ref_doc_id": "a5d000ad-7c35-4da1-b3c7-bafbb176df11", "chroma:document": "\"},\n        {\"type\": \"image\"},\n        {\"type\": \"image\"},\n    ],\n}]\n\nprocessor = Idefics2Processor.from_pretrained(\"HuggingFaceM4/idefics2-8b\")\nmodel = Idefics2ForConditionalGeneration.from_pretrained(\"HuggingFaceM4/idefics2-8b\")\nmodel.to(device)\n\n# at inference time, one needs to pass `add_generation_prompt=True` in order to make sure the model completes the prompt\ntext = processor.apply_chat_template(messages, add_generation_prompt=True)\nprint(text)\n# 'User: What\u2019s the difference between these two images?<image><image><end_of_utterance>\\nAssistant:'\n\ninputs = processor(images=images, text=text, return_tensors=\"pt\").to(device)\n\ngenerated_text = model.generate(**inputs, max_new_tokens=500)\ngenerated_text = processor.batch_decode(generated_text, skip_special_tokens=True)[0]\nprint(\"Generated text:\", generated_text)\n```\n\n- During training, it's important to determine which tokens the model should not learn. For Idefics2, this typically comes down to the image and padding tokens. This means that one can create the labels as follows:\n\n```python\nimport requests\nfrom PIL import Image\nfrom transformers import Idefics2Processor, Idefics2ForConditionalGeneration\nimport torch\n\nurl_1 = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nurl_2 = \"http://images.cocodataset.org/val2017/000000219578.jpg\"\n\nimage_1 = Image.open(requests.get(url_1, stream=True).raw)\nimage_2 = Image.open(requests.get(url_2, stream=True).raw)\nimages = [image_1, image_2]\n\nmessages = [{\n    \"role\": \"user\",\n    \"content\": [\n        {\"type\": \"text\", \"text\": \"What\u2019s the difference between these two images?\"},\n        {\"type\": \"image\"},\n        {\"type\": \"image\"},\n    ],\n},\n{\n    \"role\": \"assistant\",\n    \"content\": [\n        {\"type\": \"text\", \"text\": \"The difference is that one image is about dogs and the other one about cats.\"},\n    ],\n}]\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n\nprocessor = Idefics2Processor.from_pretrained(\"HuggingFaceM4/idefics2-8b\")\nmodel = Idefics2ForConditionalGeneration.from_pretrained(\"HuggingFaceM4/idefics2-8b\")\nmodel.to(device)\n\ntext = processor.apply_chat_template(messages, add_generation_prompt=False)\ninputs = processor(images=images, text=text, return_tensors=\"pt\").to(device)\n\nlabels = inputs.input_ids.clone()\nlabels[labels == processor.tokenizer.pad_token_id] = -100\nlabels[labels == model.config.image_token_id] = -100\n\ninputs[\"labels\"] = labels\n\noutputs = model(**inputs)\nloss = outputs.loss\nloss.backward()\n```\n\nDo note that when training Idefics2 on multi-turn conversations between a user and an assistant, one typically also sets all the tokens corresponding to the user messages to -100.\n\n## Model optimizations: Flash Attention\n\nThe code snippets above showcase inference without any optimization tricks. However, one can drastically speed up the model by leveraging [Flash Attention](../perf_train_gpu_one.md#flash-attention-2), which is a faster implementation of the attention mechanism used inside the model.\n\nFirst, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f97ce71d-6cb2-4b64-9cb5-66020ffd0763
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/idefics2", "title": "Idefics2", "tokens": 2631, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f97ce71d-6cb2-4b64-9cb5-66020ffd0763\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a5d000ad-7c35-4da1-b3c7-bafbb176df11\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fcc5f689170f580aa6e99beb8b9b83ce1ecb683ea2f11aa04efc3cd919ea891e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"70264e9f-b04d-438f-a5ed-fbf66995f79e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"641cb4a9674eb91a0d6d44638d710738be447306dff22ee609c0df298e5953b7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"78ef2fab-9081-48c4-a46a-5ba7f00b552c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"68725f6522432aad0a3c718574d3f68662557193856c8a429b8f6703178490a3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5408, \"end_char_idx\": 8115, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a5d000ad-7c35-4da1-b3c7-bafbb176df11", "doc_id": "a5d000ad-7c35-4da1-b3c7-bafbb176df11", "ref_doc_id": "a5d000ad-7c35-4da1-b3c7-bafbb176df11", "chroma:document": "\"},\n        {\"type\": \"image\"},\n        {\"type\": \"image\"},\n    ],\n},\n{\n    \"role\": \"assistant\",\n    \"content\": [\n        {\"type\": \"text\", \"text\": \"The difference is that one image is about dogs and the other one about cats.\"},\n    ],\n}]\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n\nprocessor = Idefics2Processor.from_pretrained(\"HuggingFaceM4/idefics2-8b\")\nmodel = Idefics2ForConditionalGeneration.from_pretrained(\"HuggingFaceM4/idefics2-8b\")\nmodel.to(device)\n\ntext = processor.apply_chat_template(messages, add_generation_prompt=False)\ninputs = processor(images=images, text=text, return_tensors=\"pt\").to(device)\n\nlabels = inputs.input_ids.clone()\nlabels[labels == processor.tokenizer.pad_token_id] = -100\nlabels[labels == model.config.image_token_id] = -100\n\ninputs[\"labels\"] = labels\n\noutputs = model(**inputs)\nloss = outputs.loss\nloss.backward()\n```\n\nDo note that when training Idefics2 on multi-turn conversations between a user and an assistant, one typically also sets all the tokens corresponding to the user messages to -100.\n\n## Model optimizations: Flash Attention\n\nThe code snippets above showcase inference without any optimization tricks. However, one can drastically speed up the model by leveraging [Flash Attention](../perf_train_gpu_one.md#flash-attention-2), which is a faster implementation of the attention mechanism used inside the model.\n\nFirst, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature.\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nMake also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of the [flash attention repository](https://github.com/Dao-AILab/flash-attention). Make also sure to load your model in half-precision (e.g. `torch.float16`)\n\nTo load and run a model using Flash Attention-2, simply change the code snippet above with the following change:\n\n```diff\nmodel = Idefics2ForConditionalGeneration.from_pretrained(\n    \"HuggingFaceM4/idefics2-8b\",\n+    torch_dtype=torch.float16,    \n+    attn_implementation=\"flash_attention_2\",\n).to(device)\n```\n\n## Shrinking down Idefics2 using quantization\n\nAs the Idefics2 model has 8 billion parameters, that would require about 16GB of GPU RAM in half precision (float16), since each parameter is stored in 2 bytes. However, one can shrink down the size of the model using [quantization](../quantization.md). If the model is quantized to 4 bits (or half a byte per parameter), that requires only about 3.5GB of RAM.\n\nQuantizing a model is as simple as passing a `quantization_config` to the model. One can change the code snippet above with the changes below."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 78ef2fab-9081-48c4-a46a-5ba7f00b552c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/idefics2", "title": "Idefics2", "tokens": 2631, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"78ef2fab-9081-48c4-a46a-5ba7f00b552c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a5d000ad-7c35-4da1-b3c7-bafbb176df11\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fcc5f689170f580aa6e99beb8b9b83ce1ecb683ea2f11aa04efc3cd919ea891e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f97ce71d-6cb2-4b64-9cb5-66020ffd0763\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9d7796fca3eb9d524e4d6d88368c242ee07dd20d4978e6978fd82cdb8bd14456\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"09c64844-cceb-451c-bb4b-5097bfe4eb76\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"401b4134d593065e28611712b0303a7db7d356b3171682ca513cddf3b2048875\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6456, \"end_char_idx\": 9464, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a5d000ad-7c35-4da1-b3c7-bafbb176df11", "doc_id": "a5d000ad-7c35-4da1-b3c7-bafbb176df11", "ref_doc_id": "a5d000ad-7c35-4da1-b3c7-bafbb176df11", "chroma:document": "## Model optimizations: Flash Attention\n\nThe code snippets above showcase inference without any optimization tricks. However, one can drastically speed up the model by leveraging [Flash Attention](../perf_train_gpu_one.md#flash-attention-2), which is a faster implementation of the attention mechanism used inside the model.\n\nFirst, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature.\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nMake also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of the [flash attention repository](https://github.com/Dao-AILab/flash-attention). Make also sure to load your model in half-precision (e.g. `torch.float16`)\n\nTo load and run a model using Flash Attention-2, simply change the code snippet above with the following change:\n\n```diff\nmodel = Idefics2ForConditionalGeneration.from_pretrained(\n    \"HuggingFaceM4/idefics2-8b\",\n+    torch_dtype=torch.float16,    \n+    attn_implementation=\"flash_attention_2\",\n).to(device)\n```\n\n## Shrinking down Idefics2 using quantization\n\nAs the Idefics2 model has 8 billion parameters, that would require about 16GB of GPU RAM in half precision (float16), since each parameter is stored in 2 bytes. However, one can shrink down the size of the model using [quantization](../quantization.md). If the model is quantized to 4 bits (or half a byte per parameter), that requires only about 3.5GB of RAM.\n\nQuantizing a model is as simple as passing a `quantization_config` to the model. One can change the code snippet above with the changes below. We'll leverage the BitsAndyBytes quantization (but refer to [this page](../quantization.md) for other quantization methods):\n\n```diff\n+ from transformers import BitsAndBytesConfig\n\n+ quantization_config = BitsAndBytesConfig(\n+    load_in_4bit=True,\n+    bnb_4bit_quant_type=\"nf4\",\n+    bnb_4bit_use_double_quant=True,\n+    bnb_4bit_compute_dtype=torch.float16\n+ )\nmodel = Idefics2ForConditionalGeneration.from_pretrained(\n    \"HuggingFaceM4/idefics2-8b\",\n+    torch_dtype=torch.float16,    \n+    quantization_config=quantization_config,\n).to(device)\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Idefics2. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n- A notebook on how to fine-tune Idefics2 on a custom dataset using the [Trainer](../main_classes/trainer.md) can be found [here](https://colab.research.google.com/drive/1NtcTgRbSBKN7pYD3Vdx1j9m8pt3fhFDB?usp=sharing). It supports both full fine-tuning as well as (quantized) LoRa.\n- A script regarding how to fine-tune Idefics2 using the TRL library can be found [here](https://gist.github.com/edbeeching/228652fc6c2b29a1641be5a5778223cb)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 09c64844-cceb-451c-bb4b-5097bfe4eb76
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/idefics2", "title": "Idefics2", "tokens": 2631, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"09c64844-cceb-451c-bb4b-5097bfe4eb76\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a5d000ad-7c35-4da1-b3c7-bafbb176df11\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fcc5f689170f580aa6e99beb8b9b83ce1ecb683ea2f11aa04efc3cd919ea891e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"78ef2fab-9081-48c4-a46a-5ba7f00b552c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/idefics2\", \"title\": \"Idefics2\", \"tokens\": 2631, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"71ef05b16e9ec2e8a9c83a278ea4bac3cbd772a2c68b27e8aed1fb401e1331ab\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8054, \"end_char_idx\": 9992, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a5d000ad-7c35-4da1-b3c7-bafbb176df11", "doc_id": "a5d000ad-7c35-4da1-b3c7-bafbb176df11", "ref_doc_id": "a5d000ad-7c35-4da1-b3c7-bafbb176df11", "chroma:document": "One can change the code snippet above with the changes below. We'll leverage the BitsAndyBytes quantization (but refer to [this page](../quantization.md) for other quantization methods):\n\n```diff\n+ from transformers import BitsAndBytesConfig\n\n+ quantization_config = BitsAndBytesConfig(\n+    load_in_4bit=True,\n+    bnb_4bit_quant_type=\"nf4\",\n+    bnb_4bit_use_double_quant=True,\n+    bnb_4bit_compute_dtype=torch.float16\n+ )\nmodel = Idefics2ForConditionalGeneration.from_pretrained(\n    \"HuggingFaceM4/idefics2-8b\",\n+    torch_dtype=torch.float16,    \n+    quantization_config=quantization_config,\n).to(device)\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Idefics2. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n- A notebook on how to fine-tune Idefics2 on a custom dataset using the [Trainer](../main_classes/trainer.md) can be found [here](https://colab.research.google.com/drive/1NtcTgRbSBKN7pYD3Vdx1j9m8pt3fhFDB?usp=sharing). It supports both full fine-tuning as well as (quantized) LoRa.\n- A script regarding how to fine-tune Idefics2 using the TRL library can be found [here](https://gist.github.com/edbeeching/228652fc6c2b29a1641be5a5778223cb).\n- Demo notebook regarding fine-tuning Idefics2 for JSON extraction use cases can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Idefics2). \ud83c\udf0e\n\n## Idefics2Config\n\n[[autodoc]] Idefics2Config\n\n\n## Idefics2Model\n\n[[autodoc]] Idefics2Model\n    - forward\n\n\n## Idefics2ForConditionalGeneration\n\n[[autodoc]] Idefics2ForConditionalGeneration\n    - forward\n\n\n## Idefics2ImageProcessor\n[[autodoc]] Idefics2ImageProcessor\n    - preprocess\n\n\n## Idefics2Processor\n[[autodoc]] Idefics2Processor\n    - __call__"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6ab5c8db-606c-4bb3-9cb0-176028dccaa9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/informer", "title": "Informer", "tokens": 782, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6ab5c8db-606c-4bb3-9cb0-176028dccaa9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/informer\", \"title\": \"Informer\", \"tokens\": 782, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"57d36b17-4acc-4330-ba7b-7cbcb51a9e78\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/informer\", \"title\": \"Informer\", \"tokens\": 782, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"419433c4d6cda15872e3e0ae618b41016541dd044f257b8d02cb46167168c180\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2966, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "57d36b17-4acc-4330-ba7b-7cbcb51a9e78", "doc_id": "57d36b17-4acc-4330-ba7b-7cbcb51a9e78", "ref_doc_id": "57d36b17-4acc-4330-ba7b-7cbcb51a9e78", "chroma:document": "# Informer\n\n## Overview\n\nThe Informer model was proposed in [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://arxiv.org/abs/2012.07436) by Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang.\n\nThis method introduces a Probabilistic Attention mechanism to select the \"active\" queries rather than the \"lazy\" queries and provides a sparse Transformer thus mitigating the quadratic compute and memory requirements of vanilla attention.\n\nThe abstract from the paper is the following:\n\n*Many real-world applications require the prediction of long sequence time-series, such as electricity consumption planning. Long sequence time-series forecasting (LSTF) demands a high prediction capacity of the model, which is the ability to capture precise long-range dependency coupling between output and input efficiently. Recent studies have shown the potential of Transformer to increase the prediction capacity. However, there are several severe issues with Transformer that prevent it from being directly applicable to LSTF, including quadratic time complexity, high memory usage, and inherent limitation of the encoder-decoder architecture. To address these issues, we design an efficient transformer-based model for LSTF, named Informer, with three distinctive characteristics: (i) a ProbSparse self-attention mechanism, which achieves O(L logL) in time complexity and memory usage, and has comparable performance on sequences' dependency alignment. (ii) the self-attention distilling highlights dominating attention by halving cascading layer input, and efficiently handles extreme long input sequences. (iii) the generative style decoder, while conceptually simple, predicts the long time-series sequences at one forward operation rather than a step-by-step way, which drastically improves the inference speed of long-sequence predictions. Extensive experiments on four large-scale datasets demonstrate that Informer significantly outperforms existing methods and provides a new solution to the LSTF problem.*\n\nThis model was contributed by [elisim](https://huggingface.co/elisim) and [kashif](https://huggingface.co/kashif).\nThe original code can be found [here](https://github.com/zhouhaoyi/Informer2020).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n- Check out the Informer blog-post in HuggingFace blog: [Multivariate Probabilistic Time Series Forecasting with Informer](https://huggingface.co/blog/informer)\n\n## InformerConfig\n\n[[autodoc]] InformerConfig\n\n## InformerModel\n\n[[autodoc]] InformerModel\n    - forward\n\n## InformerForPrediction\n\n[[autodoc]] InformerForPrediction\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 932d8f4d-f7a1-4ffc-bbf7-febc7b27610c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/herbert", "title": "HerBERT", "tokens": 804, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"932d8f4d-f7a1-4ffc-bbf7-febc7b27610c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/herbert\", \"title\": \"HerBERT\", \"tokens\": 804, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"94789ccc-db65-4685-921d-b9184e97107d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/herbert\", \"title\": \"HerBERT\", \"tokens\": 804, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c32ff69b7c80d4f1b0968c4afa3e0693e6a59c2f73ac35a314e05eb40bc7f1b4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2839, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "94789ccc-db65-4685-921d-b9184e97107d", "doc_id": "94789ccc-db65-4685-921d-b9184e97107d", "ref_doc_id": "94789ccc-db65-4685-921d-b9184e97107d", "chroma:document": "# HerBERT\n\n## Overview\n\nThe HerBERT model was proposed in [KLEJ: Comprehensive Benchmark for Polish Language Understanding](https://www.aclweb.org/anthology/2020.acl-main.111.pdf) by Piotr Rybak, Robert Mroczkowski, Janusz Tracz, and\nIreneusz Gawlik. It is a BERT-based Language Model trained on Polish Corpora using only MLM objective with dynamic\nmasking of whole words.\n\nThe abstract from the paper is the following:\n\n*In recent years, a series of Transformer-based models unlocked major improvements in general natural language\nunderstanding (NLU) tasks. Such a fast pace of research would not be possible without general NLU benchmarks, which\nallow for a fair comparison of the proposed methods. However, such benchmarks are available only for a handful of\nlanguages. To alleviate this issue, we introduce a comprehensive multi-task benchmark for the Polish language\nunderstanding, accompanied by an online leaderboard. It consists of a diverse set of tasks, adopted from existing\ndatasets for named entity recognition, question-answering, textual entailment, and others. We also introduce a new\nsentiment analysis task for the e-commerce domain, named Allegro Reviews (AR). To ensure a common evaluation scheme and\npromote models that generalize to different NLU tasks, the benchmark includes datasets from varying domains and\napplications. Additionally, we release HerBERT, a Transformer-based model trained specifically for the Polish language,\nwhich has the best average performance and obtains the best results for three out of nine tasks. Finally, we provide an\nextensive evaluation, including several standard baselines and recently proposed, multilingual Transformer-based\nmodels.*\n\nThis model was contributed by [rmroczkowski](https://huggingface.co/rmroczkowski). The original code can be found\n[here](https://github.com/allegro/HerBERT).\n\n\n## Usage example\n\n```python\n>>> from transformers import HerbertTokenizer, RobertaModel\n\n>>> tokenizer = HerbertTokenizer.from_pretrained(\"allegro/herbert-klej-cased-tokenizer-v1\")\n>>> model = RobertaModel.from_pretrained(\"allegro/herbert-klej-cased-v1\")\n\n>>> encoded_input = tokenizer.encode(\"Kto ma lepsz\u0105 sztuk\u0119, ma lepszy rz\u0105d \u2013 to jasne.\", return_tensors=\"pt\")\n>>> outputs = model(encoded_input)\n\n>>> # HerBERT can also be loaded using AutoTokenizer and AutoModel:\n>>> import torch\n>>> from transformers import AutoModel, AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"allegro/herbert-klej-cased-tokenizer-v1\")\n>>> model = AutoModel.from_pretrained(\"allegro/herbert-klej-cased-v1\")\n```\n\n<Tip>\n\nHerbert implementation is the same as `BERT` except for the tokenization method. Refer to [BERT documentation](bert) \nfor API reference and examples.  \n\n</Tip>\n\n## HerbertTokenizer\n\n[[autodoc]] HerbertTokenizer\n\n## HerbertTokenizerFast\n\n[[autodoc]] HerbertTokenizerFast"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 9821fea9-e91b-4e46-8446-abf6c22d567a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/encodec", "title": "EnCodec", "tokens": 859, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9821fea9-e91b-4e46-8446-abf6c22d567a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/encodec\", \"title\": \"EnCodec\", \"tokens\": 859, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d7091078-ba25-4179-8411-8097e1438969\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/encodec\", \"title\": \"EnCodec\", \"tokens\": 859, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a22ef4613ac308a3e7d00b0c01fd91f05bbc7d2ab96a8e1326505474d543759d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3127, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d7091078-ba25-4179-8411-8097e1438969", "doc_id": "d7091078-ba25-4179-8411-8097e1438969", "ref_doc_id": "d7091078-ba25-4179-8411-8097e1438969", "chroma:document": "# EnCodec\n\n## Overview\n\nThe EnCodec neural codec model was proposed in [High Fidelity Neural Audio Compression](https://arxiv.org/abs/2210.13438) by Alexandre D\u00e9fossez, Jade Copet, Gabriel Synnaeve, Yossi Adi.\n\nThe abstract from the paper is the following:\n\n*We introduce a state-of-the-art real-time, high-fidelity, audio codec leveraging neural networks. It consists in a streaming encoder-decoder architecture with quantized latent space trained in an end-to-end fashion. We simplify and speed-up the training by using a single multiscale spectrogram adversary that efficiently reduces artifacts and produce high-quality samples. We introduce a novel loss balancer mechanism to stabilize training: the weight of a loss now defines the fraction of the overall gradient it should represent, thus decoupling the choice of this hyper-parameter from the typical scale of the loss. Finally, we study how lightweight Transformer models can be used to further compress the obtained representation by up to 40%, while staying faster than real time. We provide a detailed description of the key design choices of the proposed model including: training objective, architectural changes and a study of various perceptual loss functions. We present an extensive subjective evaluation (MUSHRA tests) together with an ablation study for a range of bandwidths and audio domains, including speech, noisy-reverberant speech, and music. Our approach is superior to the baselines methods across all evaluated settings, considering both 24 kHz monophonic and 48 kHz stereophonic audio.*\n\nThis model was contributed by [Matthijs](https://huggingface.co/Matthijs), [Patrick Von Platen](https://huggingface.co/patrickvonplaten) and [Arthur Zucker](https://huggingface.co/ArthurZ). \nThe original code can be found [here](https://github.com/facebookresearch/encodec).\n\n## Usage example \n\nHere is a quick example of how to encode and decode an audio using this model:\n\n```python \n>>> from datasets import load_dataset, Audio\n>>> from transformers import EncodecModel, AutoProcessor\n>>> librispeech_dummy = load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\n\n>>> model = EncodecModel.from_pretrained(\"facebook/encodec_24khz\")\n>>> processor = AutoProcessor.from_pretrained(\"facebook/encodec_24khz\")\n>>> librispeech_dummy = librispeech_dummy.cast_column(\"audio\", Audio(sampling_rate=processor.sampling_rate))\n>>> audio_sample = librispeech_dummy[-1][\"audio\"][\"array\"]\n>>> inputs = processor(raw_audio=audio_sample, sampling_rate=processor.sampling_rate, return_tensors=\"pt\")\n\n>>> encoder_outputs = model.encode(inputs[\"input_values\"], inputs[\"padding_mask\"])\n>>> audio_values = model.decode(encoder_outputs.audio_codes, encoder_outputs.audio_scales, inputs[\"padding_mask\"])[0]\n>>> # or the equivalent with a forward pass\n>>> audio_values = model(inputs[\"input_values\"], inputs[\"padding_mask\"]).audio_values\n```\n\n## EncodecConfig\n\n[[autodoc]] EncodecConfig\n\n## EncodecFeatureExtractor\n\n[[autodoc]] EncodecFeatureExtractor\n    - __call__\n\n## EncodecModel\n\n[[autodoc]] EncodecModel\n    - decode\n    - encode\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0d73fdee-55e4-44e1-8643-2aa47f8113f2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/blip-2", "title": "BLIP-2", "tokens": 1084, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0d73fdee-55e4-44e1-8643-2aa47f8113f2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blip-2\", \"title\": \"BLIP-2\", \"tokens\": 1084, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"143cbdd4-8715-4510-a538-d0ae07665640\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blip-2\", \"title\": \"BLIP-2\", \"tokens\": 1084, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4bc38ebc73af47e0b749ecfeff29cfedbd374b999759ea0a908c3ef2d061dabe\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c7141e4b-e611-4b79-bd7b-a20dfdd7f39e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4ccfce4e31e27ed8ce348277caa67d75365ca27819f56425c13f094d5110ccba\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3084, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "143cbdd4-8715-4510-a538-d0ae07665640", "doc_id": "143cbdd4-8715-4510-a538-d0ae07665640", "ref_doc_id": "143cbdd4-8715-4510-a538-d0ae07665640", "chroma:document": "# BLIP-2\n\n## Overview\n\nThe BLIP-2 model was proposed in [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://arxiv.org/abs/2301.12597) by\nJunnan Li, Dongxu Li, Silvio Savarese, Steven Hoi. BLIP-2 leverages frozen pre-trained image encoders and large language models (LLMs) by training a lightweight, 12-layer Transformer\nencoder in between them, achieving state-of-the-art performance on various vision-language tasks. Most notably, BLIP-2 improves upon [Flamingo](https://arxiv.org/abs/2204.14198), an 80 billion parameter model, by 8.7%\non zero-shot VQAv2 with 54x fewer trainable parameters. \n\nThe abstract from the paper is the following:\n\n*The cost of vision-and-language pre-training has become increasingly prohibitive due to end-to-end training of large-scale models. This paper proposes BLIP-2, a generic and efficient pre-training strategy that bootstraps vision-language pre-training from off-the-shelf frozen pre-trained image encoders and frozen large language models. BLIP-2 bridges the modality gap with a lightweight Querying Transformer, which is pre-trained in two stages. The first stage bootstraps vision-language representation learning from a frozen image encoder. The second stage bootstraps vision-to-language generative learning from a frozen language model. BLIP-2 achieves state-of-the-art performance on various vision-language tasks, despite having significantly fewer trainable parameters than existing methods. For example, our model outperforms Flamingo80B by 8.7% on zero-shot VQAv2 with 54x fewer trainable parameters. We also demonstrate the model's emerging capabilities of zero-shot image-to-text generation that can follow natural language instructions.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/blip2_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/> \n\n<small> BLIP-2 architecture. Taken from the <a href=\"https://arxiv.org/abs/2301.12597\">original paper.</a> </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/salesforce/LAVIS/tree/5ee63d688ba4cebff63acee04adaef2dee9af207).\n\n## Usage tips\n\n- BLIP-2 can be used for conditional text generation given an image and an optional text prompt. At inference time, it's recommended to use the [`generate`] method.\n- One can use [`Blip2Processor`] to prepare images for the model, and decode the predicted tokens ID's back to text.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with BLIP-2.\n\n- Demo notebooks for BLIP-2 for image captioning, visual question answering (VQA) and chat-like conversations can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/BLIP-2).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: c7141e4b-e611-4b79-bd7b-a20dfdd7f39e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/blip-2", "title": "BLIP-2", "tokens": 1084, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c7141e4b-e611-4b79-bd7b-a20dfdd7f39e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blip-2\", \"title\": \"BLIP-2\", \"tokens\": 1084, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"143cbdd4-8715-4510-a538-d0ae07665640\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blip-2\", \"title\": \"BLIP-2\", \"tokens\": 1084, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4bc38ebc73af47e0b749ecfeff29cfedbd374b999759ea0a908c3ef2d061dabe\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0d73fdee-55e4-44e1-8643-2aa47f8113f2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blip-2\", \"title\": \"BLIP-2\", \"tokens\": 1084, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b04e5cd283e4bf147f11735d54224805a17fcc866aab881b4b8a7dbecf15c456\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1502, \"end_char_idx\": 3686, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "143cbdd4-8715-4510-a538-d0ae07665640", "doc_id": "143cbdd4-8715-4510-a538-d0ae07665640", "ref_doc_id": "143cbdd4-8715-4510-a538-d0ae07665640", "chroma:document": "For example, our model outperforms Flamingo80B by 8.7% on zero-shot VQAv2 with 54x fewer trainable parameters. We also demonstrate the model's emerging capabilities of zero-shot image-to-text generation that can follow natural language instructions.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/blip2_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/> \n\n<small> BLIP-2 architecture. Taken from the <a href=\"https://arxiv.org/abs/2301.12597\">original paper.</a> </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/salesforce/LAVIS/tree/5ee63d688ba4cebff63acee04adaef2dee9af207).\n\n## Usage tips\n\n- BLIP-2 can be used for conditional text generation given an image and an optional text prompt. At inference time, it's recommended to use the [`generate`] method.\n- One can use [`Blip2Processor`] to prepare images for the model, and decode the predicted tokens ID's back to text.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with BLIP-2.\n\n- Demo notebooks for BLIP-2 for image captioning, visual question answering (VQA) and chat-like conversations can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/BLIP-2).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## Blip2Config\n\n[[autodoc]] Blip2Config\n    - from_vision_qformer_text_configs\n\n## Blip2VisionConfig\n\n[[autodoc]] Blip2VisionConfig\n\n## Blip2QFormerConfig\n\n[[autodoc]] Blip2QFormerConfig\n\n## Blip2Processor\n\n[[autodoc]] Blip2Processor\n\n## Blip2VisionModel\n\n[[autodoc]] Blip2VisionModel\n    - forward\n\n## Blip2QFormerModel\n\n[[autodoc]] Blip2QFormerModel\n    - forward\n\n## Blip2Model\n\n[[autodoc]] Blip2Model\n    - forward\n    - get_text_features\n    - get_image_features\n    - get_qformer_features\n\n## Blip2ForConditionalGeneration\n\n[[autodoc]] Blip2ForConditionalGeneration\n    - forward\n    - generate"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 99bb0cc2-85db-406c-abfd-7af66f055507
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bert", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"99bb0cc2-85db-406c-abfd-7af66f055507\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1db66e4c-3218-4d44-800d-be5f5bd8a224\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f7df4a4420758e63e9207b04521b07644e0a8e525d7769037dc18cb45af2b1f8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fb80596f-1d35-4382-bc57-5427407dc9e5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"88eaddb12c295fb028a6679b546f5d8db9e7856402de575dc51d1a2bc812b315\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3295, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1db66e4c-3218-4d44-800d-be5f5bd8a224", "doc_id": "1db66e4c-3218-4d44-800d-be5f5bd8a224", "ref_doc_id": "1db66e4c-3218-4d44-800d-be5f5bd8a224", "chroma:document": "# BERT\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=bert\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-bert-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/bert-base-uncased\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nThe BERT model was proposed in [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805) by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova. It's a\nbidirectional transformer pretrained using a combination of masked language modeling objective and next sentence\nprediction on a large corpus comprising the Toronto Book Corpus and Wikipedia.\n\nThe abstract from the paper is the following:\n\n*We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations\nfrom Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional\nrepresentations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result,\nthe pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models\nfor a wide range of tasks, such as question answering and language inference, without substantial task-specific\narchitecture modifications.*\n\n*BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural\nlanguage processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI\naccuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute\nimprovement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).*\n\nThis model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/google-research/bert).\n\n## Usage tips\n\n- BERT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than\n  the left.\n- BERT was trained with the masked language modeling (MLM) and next sentence prediction (NSP) objectives. It is\n  efficient at predicting masked tokens and at NLU in general, but is not optimal for text generation.\n- Corrupts the inputs by using random masking, more precisely, during pretraining, a given percentage of tokens (usually 15%) is masked by:\n\n    * a special mask token with probability 0.8\n    * a random token different from the one masked with probability 0.1\n    * the same token with probability 0.1\n    \n- The model must predict the original sentence, but has a second objective: inputs are two sentences A and B (with a separation token in between). With probability 50%, the sentences are consecutive in the corpus, in the remaining 50% they are not related. The model has to predict if the sentences are consecutive or not.\n\n### Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: fb80596f-1d35-4382-bc57-5427407dc9e5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bert", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"fb80596f-1d35-4382-bc57-5427407dc9e5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1db66e4c-3218-4d44-800d-be5f5bd8a224\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f7df4a4420758e63e9207b04521b07644e0a8e525d7769037dc18cb45af2b1f8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"99bb0cc2-85db-406c-abfd-7af66f055507\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"355590af18a9342600aae8a8e1c141a8cbe5045324a37fb38a590ba1ff98bccb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"480af265-997c-404e-b007-421dae2be090\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"56260d25965577b9debcd773320033151e7c2d60124058c3f914b3dc667d34f8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1872, \"end_char_idx\": 4217, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1db66e4c-3218-4d44-800d-be5f5bd8a224", "doc_id": "1db66e4c-3218-4d44-800d-be5f5bd8a224", "ref_doc_id": "1db66e4c-3218-4d44-800d-be5f5bd8a224", "chroma:document": "*\n\nThis model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/google-research/bert).\n\n## Usage tips\n\n- BERT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than\n  the left.\n- BERT was trained with the masked language modeling (MLM) and next sentence prediction (NSP) objectives. It is\n  efficient at predicting masked tokens and at NLU in general, but is not optimal for text generation.\n- Corrupts the inputs by using random masking, more precisely, during pretraining, a given percentage of tokens (usually 15%) is masked by:\n\n    * a special mask token with probability 0.8\n    * a random token different from the one masked with probability 0.1\n    * the same token with probability 0.1\n    \n- The model must predict the original sentence, but has a second objective: inputs are two sentences A and B (with a separation token in between). With probability 50%, the sentences are consecutive in the corpus, in the remaining 50% they are not related. The model has to predict if the sentences are consecutive or not.\n\n### Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```\nfrom transformers import BertModel\n\nmodel = BertModel.from_pretrained(\"bert-base-uncased\", torch_dtype=torch.float16, attn_implementation=\"sdpa\")\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-80GB, CPUx12, RAM 96.6GB, PyTorch 2.2.0, OS Ubuntu 22.04) with `float16`, we saw the \nfollowing speedups during training and inference."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 480af265-997c-404e-b007-421dae2be090
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bert", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"480af265-997c-404e-b007-421dae2be090\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1db66e4c-3218-4d44-800d-be5f5bd8a224\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f7df4a4420758e63e9207b04521b07644e0a8e525d7769037dc18cb45af2b1f8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"fb80596f-1d35-4382-bc57-5427407dc9e5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"dd16fc2dff1e5010339f48a6aa11d6ff17a730c28d9bfa157d5023848c30abeb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bc851189-ae80-441e-b6e9-2596a49d8414\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1c69cf33c2f8a695fe3d2039c85992cb4141057bb08c52c9070e5ed3d540d4e4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2847, \"end_char_idx\": 6418, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1db66e4c-3218-4d44-800d-be5f5bd8a224", "doc_id": "1db66e4c-3218-4d44-800d-be5f5bd8a224", "ref_doc_id": "1db66e4c-3218-4d44-800d-be5f5bd8a224", "chroma:document": "With probability 50%, the sentences are consecutive in the corpus, in the remaining 50% they are not related. The model has to predict if the sentences are consecutive or not.\n\n### Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```\nfrom transformers import BertModel\n\nmodel = BertModel.from_pretrained(\"bert-base-uncased\", torch_dtype=torch.float16, attn_implementation=\"sdpa\")\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-80GB, CPUx12, RAM 96.6GB, PyTorch 2.2.0, OS Ubuntu 22.04) with `float16`, we saw the \nfollowing speedups during training and inference.\n\n#### Training\n\n|batch_size|seq_len|Time per batch (eager - s)|Time per batch (sdpa - s)|Speedup (%)|Eager peak mem (MB)|sdpa peak mem (MB)|Mem saving (%)|\n|----------|-------|--------------------------|-------------------------|-----------|-------------------|------------------|--------------|\n|4         |256    |0.023                     |0.017                    |35.472     |939.213            |764.834           |22.800        |\n|4         |512    |0.023                     |0.018                    |23.687     |1970.447           |1227.162          |60.569        |\n|8         |256    |0.023                     |0.018                    |23.491     |1594.295           |1226.114          |30.028        |\n|8         |512    |0.035                     |0.025                    |43.058     |3629.401           |2134.262          |70.054        |\n|16        |256    |0.030                     |0.024                    |25.583     |2874.426           |2134.262          |34.680        |\n|16        |512    |0.064                     |0.044                    |46.223     |6964.659           |3961.013          |75.830        |\n\n#### Inference\n\n|batch_size|seq_len|Per token latency eager (ms)|Per token latency SDPA (ms)|Speedup (%)|Mem eager (MB)|Mem BT (MB)|Mem saved (%)|\n|----------|-------|----------------------------|---------------------------|-----------|--------------|-----------|-------------|\n|1         |128    |5.736                       |4.987                      |15.022     |282.661       |282.924    |-0.093       |\n|1         |256    |5.689                       |4.945                      |15.055     |298.686       |298.948    |-0.088       |\n|2         |128    |6.154                       |4.982                      |23.521     |314.523       |314.785    |-0.083       |\n|2         |256    |6.201                       |4.949                      |25.303     |347.546       |347.033    |0.148        |\n|4         |128    |6.049                       |4.987                      |21.305     |378.895       |379.301    |-0.107       |\n|4         |256    |6.285                       |5.364                      |17.166     |443.209       |444.382    |-0.264       |"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: bc851189-ae80-441e-b6e9-2596a49d8414
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bert", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"bc851189-ae80-441e-b6e9-2596a49d8414\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1db66e4c-3218-4d44-800d-be5f5bd8a224\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f7df4a4420758e63e9207b04521b07644e0a8e525d7769037dc18cb45af2b1f8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"480af265-997c-404e-b007-421dae2be090\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"64ee383e73ceaa65fbf9084bb0674354aad45b1bd4e359166d5d9ece600d75b0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b3196f44-911f-451d-8ef6-44310eff33ac\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"def6929d6cb1a9aa7ee8e3c4dd1570112e269ee633a6aeea9ca0681cd47bc9f3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6422, \"end_char_idx\": 9651, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1db66e4c-3218-4d44-800d-be5f5bd8a224", "doc_id": "1db66e4c-3218-4d44-800d-be5f5bd8a224", "ref_doc_id": "1db66e4c-3218-4d44-800d-be5f5bd8a224", "chroma:document": "## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with BERT. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- A blog post on [BERT Text Classification in a different language](https://www.philschmid.de/bert-text-classification-in-a-different-language).\n- A notebook for [Finetuning BERT (and friends) for multi-label text classification](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/BERT/Fine_tuning_BERT_(and_friends)_for_multi_label_text_classification.ipynb).\n- A notebook on how to [Finetune BERT for multi-label classification using PyTorch](https://colab.research.google.com/github/abhimishra91/transformers-tutorials/blob/master/transformers_multi_label_classification.ipynb). \ud83c\udf0e\n- A notebook on how to [warm-start an EncoderDecoder model with BERT for summarization](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/BERT2BERT_for_CNN_Dailymail.ipynb).\n- [`BertForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification.ipynb).\n- [`TFBertForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification-tf.ipynb).\n- [`FlaxBertForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification_flax.ipynb).\n- [Text classification task guide](../tasks/sequence_classification)\n\n<PipelineTag pipeline=\"token-classification\"/>\n\n- A blog post on how to use [Hugging Face Transformers with Keras: Fine-tune a non-English BERT for Named Entity Recognition](https://www.philschmid.de/huggingface-transformers-keras-tf).\n- A notebook for [Finetuning BERT for named-entity recognition](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/BERT/Custom_Named_Entity_Recognition_with_BERT_only_first_wordpiece.ipynb) using only the first wordpiece of each word in the word label during tokenization. To propagate the label of the word to all wordpieces, see this [version](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/BERT/Custom_Named_Entity_Recognition_with_BERT.ipynb) of the notebook instead.\n- [`BertForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification.ipynb)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: b3196f44-911f-451d-8ef6-44310eff33ac
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bert", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b3196f44-911f-451d-8ef6-44310eff33ac\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1db66e4c-3218-4d44-800d-be5f5bd8a224\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f7df4a4420758e63e9207b04521b07644e0a8e525d7769037dc18cb45af2b1f8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bc851189-ae80-441e-b6e9-2596a49d8414\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c49ecad549e83593a46b37e86e77ffc09dd1e0162eada8b984c52010676b33d2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"002ccc80-89e2-4767-971f-fb83f6194ad9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e299b0ad534f982b3b59f1e84cbb864c29c47248df5ffc5095462e6c0bc0f70a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8233, \"end_char_idx\": 11372, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1db66e4c-3218-4d44-800d-be5f5bd8a224", "doc_id": "1db66e4c-3218-4d44-800d-be5f5bd8a224", "ref_doc_id": "1db66e4c-3218-4d44-800d-be5f5bd8a224", "chroma:document": "- [`FlaxBertForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification_flax.ipynb).\n- [Text classification task guide](../tasks/sequence_classification)\n\n<PipelineTag pipeline=\"token-classification\"/>\n\n- A blog post on how to use [Hugging Face Transformers with Keras: Fine-tune a non-English BERT for Named Entity Recognition](https://www.philschmid.de/huggingface-transformers-keras-tf).\n- A notebook for [Finetuning BERT for named-entity recognition](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/BERT/Custom_Named_Entity_Recognition_with_BERT_only_first_wordpiece.ipynb) using only the first wordpiece of each word in the word label during tokenization. To propagate the label of the word to all wordpieces, see this [version](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/BERT/Custom_Named_Entity_Recognition_with_BERT.ipynb) of the notebook instead.\n- [`BertForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification.ipynb).\n- [`TFBertForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification-tf.ipynb).\n- [`FlaxBertForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/token-classification).\n- [Token classification](https://huggingface.co/course/chapter7/2?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Token classification task guide](../tasks/token_classification)\n\n<PipelineTag pipeline=\"fill-mask\"/>\n\n- [`BertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#robertabertdistilbert-and-masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n- [`TFBertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_mlmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- [`FlaxBertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).\n- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 002ccc80-89e2-4767-971f-fb83f6194ad9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bert", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"002ccc80-89e2-4767-971f-fb83f6194ad9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1db66e4c-3218-4d44-800d-be5f5bd8a224\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f7df4a4420758e63e9207b04521b07644e0a8e525d7769037dc18cb45af2b1f8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b3196f44-911f-451d-8ef6-44310eff33ac\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"865458efce7d4e9bf8fa295d231d7a5aa4f27a3ffea7d962726d81646b445bb3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5fe2573e-3cfe-4956-bd0e-8866d2efa10c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"256a523c841795735baf57a8e6e33e7916b89edf7ca973b48a589892b04ddd3a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9950, \"end_char_idx\": 12983, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1db66e4c-3218-4d44-800d-be5f5bd8a224", "doc_id": "1db66e4c-3218-4d44-800d-be5f5bd8a224", "ref_doc_id": "1db66e4c-3218-4d44-800d-be5f5bd8a224", "chroma:document": "- [`FlaxBertForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/token-classification).\n- [Token classification](https://huggingface.co/course/chapter7/2?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Token classification task guide](../tasks/token_classification)\n\n<PipelineTag pipeline=\"fill-mask\"/>\n\n- [`BertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#robertabertdistilbert-and-masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n- [`TFBertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_mlmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n- [`FlaxBertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).\n- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- [`BertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).\n- [`TFBertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).\n- [`FlaxBertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering).\n- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Question answering task guide](../tasks/question_answering)\n\n**Multiple choice**\n- [`BertForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb).\n- [`TFBertForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice-tf.ipynb)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 5fe2573e-3cfe-4956-bd0e-8866d2efa10c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bert", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5fe2573e-3cfe-4956-bd0e-8866d2efa10c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1db66e4c-3218-4d44-800d-be5f5bd8a224\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f7df4a4420758e63e9207b04521b07644e0a8e525d7769037dc18cb45af2b1f8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"002ccc80-89e2-4767-971f-fb83f6194ad9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"260306f9fb9c0b1d32b6c5aa8a2b72bae5c1adb775c80d30c5598ad060b26e4b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c13ebf1e-d0a1-41a8-af4c-4d264798231d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f2a867e9151f9196b3812d1ed74925fbe0495840bdfaee09ab74281f1e07f7b7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11373, \"end_char_idx\": 14364, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1db66e4c-3218-4d44-800d-be5f5bd8a224", "doc_id": "1db66e4c-3218-4d44-800d-be5f5bd8a224", "ref_doc_id": "1db66e4c-3218-4d44-800d-be5f5bd8a224", "chroma:document": "- [Masked language modeling task guide](../tasks/masked_language_modeling)\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- [`BertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).\n- [`TFBertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).\n- [`FlaxBertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering).\n- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the \ud83e\udd17 Hugging Face Course.\n- [Question answering task guide](../tasks/question_answering)\n\n**Multiple choice**\n- [`BertForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb).\n- [`TFBertForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice-tf.ipynb).\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n\u26a1\ufe0f **Inference**\n- A blog post on how to [Accelerate BERT inference with Hugging Face Transformers and AWS Inferentia](https://huggingface.co/blog/bert-inferentia-sagemaker).\n- A blog post on how to [Accelerate BERT inference with DeepSpeed-Inference on GPUs](https://www.philschmid.de/bert-deepspeed-inference).\n\n\u2699\ufe0f **Pretraining**\n- A blog post on [Pre-Training BERT with Hugging Face Transformers and Habana Gaudi](https://www.philschmid.de/pre-training-bert-habana).\n\n\ud83d\ude80 **Deploy**\n- A blog post on how to [Convert Transformers to ONNX with Hugging Face Optimum](https://www.philschmid.de/convert-transformers-to-onnx).\n- A blog post on how to [Setup Deep Learning environment for Hugging Face Transformers with Habana Gaudi on AWS](https://www.philschmid.de/getting-started-habana-gaudi#conclusion).\n- A blog post on [Autoscaling BERT with Hugging Face Transformers, Amazon SageMaker and Terraform module](https://www.philschmid.de/terraform-huggingface-amazon-sagemaker-advanced).\n- A blog post on [Serverless BERT with HuggingFace, AWS Lambda, and Docker](https://www.philschmid.de/serverless-bert-with-huggingface-aws-lambda-docker).\n- A blog post on [Hugging Face Transformers BERT fine-tuning using Amazon SageMaker and Training Compiler](https://www.philschmid.de/huggingface-amazon-sagemaker-training-compiler)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: c13ebf1e-d0a1-41a8-af4c-4d264798231d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bert", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c13ebf1e-d0a1-41a8-af4c-4d264798231d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1db66e4c-3218-4d44-800d-be5f5bd8a224\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f7df4a4420758e63e9207b04521b07644e0a8e525d7769037dc18cb45af2b1f8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5fe2573e-3cfe-4956-bd0e-8866d2efa10c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"81fe5ef6e13de34f6b0c2379bd66542d875f418cc02cd0cafc43d35ffc2e3d94\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8d79c284-77c7-4a37-b286-8ddedf2fcdc7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"63e76fbef7d093557edd0f166ffcd4634c9960737c7883277aaf86bb345b81ee\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12984, \"end_char_idx\": 15162, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1db66e4c-3218-4d44-800d-be5f5bd8a224", "doc_id": "1db66e4c-3218-4d44-800d-be5f5bd8a224", "ref_doc_id": "1db66e4c-3218-4d44-800d-be5f5bd8a224", "chroma:document": "- [Multiple choice task guide](../tasks/multiple_choice)\n\n\u26a1\ufe0f **Inference**\n- A blog post on how to [Accelerate BERT inference with Hugging Face Transformers and AWS Inferentia](https://huggingface.co/blog/bert-inferentia-sagemaker).\n- A blog post on how to [Accelerate BERT inference with DeepSpeed-Inference on GPUs](https://www.philschmid.de/bert-deepspeed-inference).\n\n\u2699\ufe0f **Pretraining**\n- A blog post on [Pre-Training BERT with Hugging Face Transformers and Habana Gaudi](https://www.philschmid.de/pre-training-bert-habana).\n\n\ud83d\ude80 **Deploy**\n- A blog post on how to [Convert Transformers to ONNX with Hugging Face Optimum](https://www.philschmid.de/convert-transformers-to-onnx).\n- A blog post on how to [Setup Deep Learning environment for Hugging Face Transformers with Habana Gaudi on AWS](https://www.philschmid.de/getting-started-habana-gaudi#conclusion).\n- A blog post on [Autoscaling BERT with Hugging Face Transformers, Amazon SageMaker and Terraform module](https://www.philschmid.de/terraform-huggingface-amazon-sagemaker-advanced).\n- A blog post on [Serverless BERT with HuggingFace, AWS Lambda, and Docker](https://www.philschmid.de/serverless-bert-with-huggingface-aws-lambda-docker).\n- A blog post on [Hugging Face Transformers BERT fine-tuning using Amazon SageMaker and Training Compiler](https://www.philschmid.de/huggingface-amazon-sagemaker-training-compiler).\n- A blog post on [Task-specific knowledge distillation for BERT using Transformers & Amazon SageMaker](https://www.philschmid.de/knowledge-distillation-bert-transformers).\n\n## BertConfig\n\n[[autodoc]] BertConfig\n    - all\n\n## BertTokenizer\n\n[[autodoc]] BertTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n<frameworkcontent>\n<pt>\n\n## BertTokenizerFast\n\n[[autodoc]] BertTokenizerFast\n\n</pt>\n<tf>\n\n## TFBertTokenizer\n\n[[autodoc]] TFBertTokenizer\n\n</tf>\n</frameworkcontent>\n\n## Bert specific outputs\n\n[[autodoc]] models.bert.modeling_bert.BertForPreTrainingOutput\n\n[[autodoc]] models.bert.modeling_tf_bert.TFBertForPreTrainingOutput\n\n[[autodoc]] models.bert.modeling_flax_bert.FlaxBertForPreTrainingOutput"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 8d79c284-77c7-4a37-b286-8ddedf2fcdc7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bert", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8d79c284-77c7-4a37-b286-8ddedf2fcdc7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1db66e4c-3218-4d44-800d-be5f5bd8a224\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f7df4a4420758e63e9207b04521b07644e0a8e525d7769037dc18cb45af2b1f8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c13ebf1e-d0a1-41a8-af4c-4d264798231d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert\", \"title\": \"BERT\", \"tokens\": 4596, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c46e81f3fe5a65c7d062d659bea0f067e31a0b19931402a2d06aeeab73d8aa9f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13665, \"end_char_idx\": 17343, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1db66e4c-3218-4d44-800d-be5f5bd8a224", "doc_id": "1db66e4c-3218-4d44-800d-be5f5bd8a224", "ref_doc_id": "1db66e4c-3218-4d44-800d-be5f5bd8a224", "chroma:document": "- A blog post on how to [Setup Deep Learning environment for Hugging Face Transformers with Habana Gaudi on AWS](https://www.philschmid.de/getting-started-habana-gaudi#conclusion).\n- A blog post on [Autoscaling BERT with Hugging Face Transformers, Amazon SageMaker and Terraform module](https://www.philschmid.de/terraform-huggingface-amazon-sagemaker-advanced).\n- A blog post on [Serverless BERT with HuggingFace, AWS Lambda, and Docker](https://www.philschmid.de/serverless-bert-with-huggingface-aws-lambda-docker).\n- A blog post on [Hugging Face Transformers BERT fine-tuning using Amazon SageMaker and Training Compiler](https://www.philschmid.de/huggingface-amazon-sagemaker-training-compiler).\n- A blog post on [Task-specific knowledge distillation for BERT using Transformers & Amazon SageMaker](https://www.philschmid.de/knowledge-distillation-bert-transformers).\n\n## BertConfig\n\n[[autodoc]] BertConfig\n    - all\n\n## BertTokenizer\n\n[[autodoc]] BertTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n<frameworkcontent>\n<pt>\n\n## BertTokenizerFast\n\n[[autodoc]] BertTokenizerFast\n\n</pt>\n<tf>\n\n## TFBertTokenizer\n\n[[autodoc]] TFBertTokenizer\n\n</tf>\n</frameworkcontent>\n\n## Bert specific outputs\n\n[[autodoc]] models.bert.modeling_bert.BertForPreTrainingOutput\n\n[[autodoc]] models.bert.modeling_tf_bert.TFBertForPreTrainingOutput\n\n[[autodoc]] models.bert.modeling_flax_bert.FlaxBertForPreTrainingOutput\n\n\n<frameworkcontent>\n<pt>\n\n## BertModel\n\n[[autodoc]] BertModel\n    - forward\n\n## BertForPreTraining\n\n[[autodoc]] BertForPreTraining\n    - forward\n\n## BertLMHeadModel\n\n[[autodoc]] BertLMHeadModel\n    - forward\n\n## BertForMaskedLM\n\n[[autodoc]] BertForMaskedLM\n    - forward\n\n## BertForNextSentencePrediction\n\n[[autodoc]] BertForNextSentencePrediction\n    - forward\n\n## BertForSequenceClassification\n\n[[autodoc]] BertForSequenceClassification\n    - forward\n\n## BertForMultipleChoice\n\n[[autodoc]] BertForMultipleChoice\n    - forward\n\n## BertForTokenClassification\n\n[[autodoc]] BertForTokenClassification\n    - forward\n\n## BertForQuestionAnswering\n\n[[autodoc]] BertForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFBertModel\n\n[[autodoc]] TFBertModel\n    - call\n\n## TFBertForPreTraining\n\n[[autodoc]] TFBertForPreTraining\n    - call\n\n## TFBertModelLMHeadModel\n\n[[autodoc]] TFBertLMHeadModel\n    - call\n\n## TFBertForMaskedLM\n\n[[autodoc]] TFBertForMaskedLM\n    - call\n\n## TFBertForNextSentencePrediction\n\n[[autodoc]] TFBertForNextSentencePrediction\n    - call\n\n## TFBertForSequenceClassification\n\n[[autodoc]] TFBertForSequenceClassification\n    - call\n\n## TFBertForMultipleChoice\n\n[[autodoc]] TFBertForMultipleChoice\n    - call\n\n## TFBertForTokenClassification\n\n[[autodoc]] TFBertForTokenClassification\n    - call\n\n## TFBertForQuestionAnswering\n\n[[autodoc]] TFBertForQuestionAnswering\n    - call\n\n</tf>\n<jax>\n\n## FlaxBertModel\n\n[[autodoc]] FlaxBertModel\n    - __call__\n\n## FlaxBertForPreTraining\n\n[[autodoc]] FlaxBertForPreTraining\n    - __call__\n\n## FlaxBertForCausalLM\n\n[[autodoc]] FlaxBertForCausalLM\n    - __call__\n\n## FlaxBertForMaskedLM\n\n[[autodoc]] FlaxBertForMaskedLM\n    - __call__\n\n## FlaxBertForNextSentencePrediction\n\n[[autodoc]] FlaxBertForNextSentencePrediction\n    - __call__\n\n## FlaxBertForSequenceClassification\n\n[[autodoc]] FlaxBertForSequenceClassification\n    - __call__\n\n## FlaxBertForMultipleChoice\n\n[[autodoc]] FlaxBertForMultipleChoice\n    - __call__\n\n## FlaxBertForTokenClassification\n\n[[autodoc]] FlaxBertForTokenClassification\n    - __call__\n\n## FlaxBertForQuestionAnswering\n\n[[autodoc]] FlaxBertForQuestionAnswering\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0819cc02-9422-44a6-9228-51c922e113bc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/umt5", "title": "UMT5", "tokens": 1296, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0819cc02-9422-44a6-9228-51c922e113bc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/umt5\", \"title\": \"UMT5\", \"tokens\": 1296, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"719ec75b-9ac8-4c6f-b32f-1e57c6e1b810\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/umt5\", \"title\": \"UMT5\", \"tokens\": 1296, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ac3c31740feb43025018d8163b7f83e604ea0dd81ff24a63df0c4226c1315de8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"41e561d6-02e6-487f-b8f6-2df5dad3ccc9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"27f919211001c19c4b92aef3cf267e3129b2129c077ff4e93e23a15d7107711a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2993, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "719ec75b-9ac8-4c6f-b32f-1e57c6e1b810", "doc_id": "719ec75b-9ac8-4c6f-b32f-1e57c6e1b810", "ref_doc_id": "719ec75b-9ac8-4c6f-b32f-1e57c6e1b810", "chroma:document": "# UMT5\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=umt5\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-mt5-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/mt5-small-finetuned-arxiv-cs-finetuned-arxiv-cs-full\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nThe UMT5 model was proposed in [UniMax: Fairer and More Effective Language Sampling for Large-Scale Multilingual Pretraining](https://openreview.net/forum?id=kXwdL1cWOAi) by Hyung Won Chung, Xavier Garcia, Adam Roberts, Yi Tay, Orhan Firat, Sharan Narang, Noah Constant.\n\nThe abstract from the paper is the following:\n\n*Pretrained multilingual large language models have typically used heuristic temperature-based sampling to balance between different languages. However previous work has not systematically evaluated the efficacy of different pretraining language distributions across model scales. In this paper, we propose a new sampling method, UniMax, that delivers more uniform coverage of head languages while mitigating overfitting on tail languages by explicitly capping the number of repeats over each language's corpus. We perform an extensive series of ablations testing a range of sampling strategies on a suite of multilingual benchmarks, while varying model scale. We find that UniMax outperforms standard temperature-based sampling, and the benefits persist as scale increases. As part of our contribution, we release: (i) an improved and refreshed mC4 multilingual corpus consisting of 29 trillion characters across 107 languages, and (ii) a suite of pretrained umT5 model checkpoints trained with UniMax sampling.*\n\nGoogle has released the following variants:\n\n- [google/umt5-small](https://huggingface.co/google/umt5-small)\n- [google/umt5-base](https://huggingface.co/google/umt5-base)\n- [google/umt5-xl](https://huggingface.co/google/umt5-xl)\n- [google/umt5-xxl](https://huggingface.co/google/umt5-xxl).\n\nThis model was contributed by [agemagician](https://huggingface.co/agemagician) and [stefan-it](https://huggingface.co/stefan-it). The original code can be\nfound [here](https://github.com/google-research/t5x).\n\n## Usage tips \n\n- UMT5 was only pre-trained on [mC4](https://huggingface.co/datasets/mc4) excluding any supervised training.\nTherefore, this model has to be fine-tuned before it is usable on a downstream task, unlike the original T5 model.\n- Since umT5 was pre-trained in an unsupervised manner, there's no real advantage to using a task prefix during single-task\nfine-tuning. If you are doing multi-task fine-tuning, you should use a prefix.\n\n## Differences with mT5?\n`UmT5` is based on mT5, with a non-shared relative positional bias that is computed for each layer. This means that the model set `has_relative_bias` for each layer.\nThe conversion script is also different because the model was saved in t5x's latest checkpointing format."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 41e561d6-02e6-487f-b8f6-2df5dad3ccc9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/umt5", "title": "UMT5", "tokens": 1296, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"41e561d6-02e6-487f-b8f6-2df5dad3ccc9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/umt5\", \"title\": \"UMT5\", \"tokens\": 1296, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"719ec75b-9ac8-4c6f-b32f-1e57c6e1b810\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/umt5\", \"title\": \"UMT5\", \"tokens\": 1296, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ac3c31740feb43025018d8163b7f83e604ea0dd81ff24a63df0c4226c1315de8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0819cc02-9422-44a6-9228-51c922e113bc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/umt5\", \"title\": \"UMT5\", \"tokens\": 1296, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"24232d1743b484c035eac6f9707e030b5b6afe3bbe831c27e337993672846348\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1511, \"end_char_idx\": 4254, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "719ec75b-9ac8-4c6f-b32f-1e57c6e1b810", "doc_id": "719ec75b-9ac8-4c6f-b32f-1e57c6e1b810", "ref_doc_id": "719ec75b-9ac8-4c6f-b32f-1e57c6e1b810", "chroma:document": "As part of our contribution, we release: (i) an improved and refreshed mC4 multilingual corpus consisting of 29 trillion characters across 107 languages, and (ii) a suite of pretrained umT5 model checkpoints trained with UniMax sampling.*\n\nGoogle has released the following variants:\n\n- [google/umt5-small](https://huggingface.co/google/umt5-small)\n- [google/umt5-base](https://huggingface.co/google/umt5-base)\n- [google/umt5-xl](https://huggingface.co/google/umt5-xl)\n- [google/umt5-xxl](https://huggingface.co/google/umt5-xxl).\n\nThis model was contributed by [agemagician](https://huggingface.co/agemagician) and [stefan-it](https://huggingface.co/stefan-it). The original code can be\nfound [here](https://github.com/google-research/t5x).\n\n## Usage tips \n\n- UMT5 was only pre-trained on [mC4](https://huggingface.co/datasets/mc4) excluding any supervised training.\nTherefore, this model has to be fine-tuned before it is usable on a downstream task, unlike the original T5 model.\n- Since umT5 was pre-trained in an unsupervised manner, there's no real advantage to using a task prefix during single-task\nfine-tuning. If you are doing multi-task fine-tuning, you should use a prefix.\n\n## Differences with mT5?\n`UmT5` is based on mT5, with a non-shared relative positional bias that is computed for each layer. This means that the model set `has_relative_bias` for each layer.\nThe conversion script is also different because the model was saved in t5x's latest checkpointing format.\n\n# Sample usage\n\n```python\n>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"google/umt5-small\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"google/umt5-small\")\n\n>>> inputs = tokenizer(\n...     \"A <extra_id_0> walks into a bar and orders a <extra_id_1> with <extra_id_2> pinch of <extra_id_3>.\",\n...     return_tensors=\"pt\",\n... )\n>>> outputs = model.generate(**inputs)\n>>> print(tokenizer.batch_decode(outputs))\n['<pad><extra_id_0>nyone who<extra_id_1> drink<extra_id_2> a<extra_id_3> alcohol<extra_id_4> A<extra_id_5> A. This<extra_id_6> I<extra_id_7><extra_id_52><extra_id_53></s>']\n```\n\n<Tip> \n\nRefer to [T5's documentation page](t5) for more tips, code examples and notebooks.\n</Tip>\n\n## UMT5Config\n\n[[autodoc]] UMT5Config\n\n## UMT5Model\n\n[[autodoc]] UMT5Model\n    - forward\n\n## UMT5ForConditionalGeneration\n\n[[autodoc]] UMT5ForConditionalGeneration\n    - forward\n\n## UMT5EncoderModel\n\n[[autodoc]] UMT5EncoderModel\n    - forward\n\n## UMT5ForSequenceClassification\n\n[[autodoc]] UMT5ForSequenceClassification\n    - forward\n\n## UMT5ForTokenClassification\n\n[[autodoc]] UMT5ForTokenClassification\n    - forward\n\n## UMT5ForQuestionAnswering\n\n[[autodoc]] UMT5ForQuestionAnswering\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 11dc7955-0f57-44cf-adfc-fcc289dd1eca
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/udop", "title": "UDOP", "tokens": 1659, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"11dc7955-0f57-44cf-adfc-fcc289dd1eca\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/udop\", \"title\": \"UDOP\", \"tokens\": 1659, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3f2e41d6-9cc9-480a-ac9c-5e750cd18cc9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/udop\", \"title\": \"UDOP\", \"tokens\": 1659, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"048971be410d1ad307cfdf125f112d6f67c3032221de1cbcc5e82cfd46a97328\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d0a91d4a-a7df-46cb-b609-cd2e9ae4c7d8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2007babad6136620a3d34fc0dd0138a221b8b0743365908e84e9eccf6ef7441c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3279, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3f2e41d6-9cc9-480a-ac9c-5e750cd18cc9", "doc_id": "3f2e41d6-9cc9-480a-ac9c-5e750cd18cc9", "ref_doc_id": "3f2e41d6-9cc9-480a-ac9c-5e750cd18cc9", "chroma:document": "# UDOP\n\n## Overview\n\nThe UDOP model was proposed in [Unifying Vision, Text, and Layout for Universal Document Processing](https://arxiv.org/abs/2212.02623) by Zineng Tang, Ziyi Yang, Guoxin Wang, Yuwei Fang, Yang Liu, Chenguang Zhu, Michael Zeng, Cha Zhang, Mohit Bansal.\nUDOP adopts an encoder-decoder Transformer architecture based on [T5](t5) for document AI tasks like document image classification, document parsing and document visual question answering.\n\nThe abstract from the paper is the following:\n\nWe propose Universal Document Processing (UDOP), a foundation Document AI model which unifies text, image, and layout modalities together with varied task formats, including document understanding and generation. UDOP leverages the spatial correlation between textual content and document image to model image, text, and layout modalities with one uniform representation. With a novel Vision-Text-Layout Transformer, UDOP unifies pretraining and multi-domain downstream tasks into a prompt-based sequence generation scheme. UDOP is pretrained on both large-scale unlabeled document corpora using innovative self-supervised objectives and diverse labeled data. UDOP also learns to generate document images from text and layout modalities via masked image reconstruction. To the best of our knowledge, this is the first time in the field of document AI that one model simultaneously achieves high-quality neural document editing and content customization. Our method sets the state-of-the-art on 9 Document AI tasks, e.g., document understanding and QA, across diverse data domains like finance reports, academic papers, and websites. UDOP ranks first on the leaderboard of the Document Understanding Benchmark (DUE).*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/udop_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> UDOP architecture. Taken from the <a href=\"https://arxiv.org/abs/2212.02623\">original paper.</a> </small>\n\n## Usage tips\n\n- In addition to *input_ids*, [`UdopForConditionalGeneration`] also expects the input `bbox`, which are\n  the bounding boxes (i.e. 2D-positions) of the input tokens. These can be obtained using an external OCR engine such\n  as Google's [Tesseract](https://github.com/tesseract-ocr/tesseract) (there's a [Python wrapper](https://pypi.org/project/pytesseract/) available). Each bounding box should be in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the\n  position of the lower right corner. Note that one first needs to normalize the bounding boxes to be on a 0-1000\n  scale. To normalize, you can use the following function:\n\n```python\ndef normalize_bbox(bbox, width, height):\n    return [\n        int(1000 * (bbox[0] / width)),\n        int(1000 * (bbox[1] / height)),\n        int(1000 * (bbox[2] / width)),\n        int(1000 * (bbox[3] / height)),\n    ]\n```\n\nHere, `width` and `height` correspond to the width and height of the original document in which the token\noccurs. Those can be obtained using the Python Image Library (PIL) library for example, as follows:\n\n```python\nfrom PIL import Image\n\n# Document can be a png, jpg, etc. PDFs must be converted to images."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: d0a91d4a-a7df-46cb-b609-cd2e9ae4c7d8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/udop", "title": "UDOP", "tokens": 1659, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d0a91d4a-a7df-46cb-b609-cd2e9ae4c7d8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/udop\", \"title\": \"UDOP\", \"tokens\": 1659, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3f2e41d6-9cc9-480a-ac9c-5e750cd18cc9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/udop\", \"title\": \"UDOP\", \"tokens\": 1659, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"048971be410d1ad307cfdf125f112d6f67c3032221de1cbcc5e82cfd46a97328\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"11dc7955-0f57-44cf-adfc-fcc289dd1eca\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/udop\", \"title\": \"UDOP\", \"tokens\": 1659, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b37c7afc4827e1d4ec11156ff2016a00df1dcec395c8409ea9e8ef32b669f0db\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a57979a2-c6de-4664-96c1-8840c8e85cb2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3b3a1db6741f805ed01694f9c80a4cfdc5cdbc1fbb12fbda879ef6c652b827f5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1917, \"end_char_idx\": 4870, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3f2e41d6-9cc9-480a-ac9c-5e750cd18cc9", "doc_id": "3f2e41d6-9cc9-480a-ac9c-5e750cd18cc9", "ref_doc_id": "3f2e41d6-9cc9-480a-ac9c-5e750cd18cc9", "chroma:document": "Taken from the <a href=\"https://arxiv.org/abs/2212.02623\">original paper.</a> </small>\n\n## Usage tips\n\n- In addition to *input_ids*, [`UdopForConditionalGeneration`] also expects the input `bbox`, which are\n  the bounding boxes (i.e. 2D-positions) of the input tokens. These can be obtained using an external OCR engine such\n  as Google's [Tesseract](https://github.com/tesseract-ocr/tesseract) (there's a [Python wrapper](https://pypi.org/project/pytesseract/) available). Each bounding box should be in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the\n  position of the lower right corner. Note that one first needs to normalize the bounding boxes to be on a 0-1000\n  scale. To normalize, you can use the following function:\n\n```python\ndef normalize_bbox(bbox, width, height):\n    return [\n        int(1000 * (bbox[0] / width)),\n        int(1000 * (bbox[1] / height)),\n        int(1000 * (bbox[2] / width)),\n        int(1000 * (bbox[3] / height)),\n    ]\n```\n\nHere, `width` and `height` correspond to the width and height of the original document in which the token\noccurs. Those can be obtained using the Python Image Library (PIL) library for example, as follows:\n\n```python\nfrom PIL import Image\n\n# Document can be a png, jpg, etc. PDFs must be converted to images.\nimage = Image.open(name_of_your_document).convert(\"RGB\")\n\nwidth, height = image.size\n```\n\nOne can use [`UdopProcessor`] to prepare images and text for the model, which takes care of all of this. By default, this class uses the Tesseract engine to extract a list of words and boxes (coordinates) from a given document. Its functionality is equivalent to that of [`LayoutLMv3Processor`], hence it supports passing either `apply_ocr=False` in case you prefer to use your own OCR engine or `apply_ocr=True` in case you want the default OCR engine to be used. Refer to the [usage guide of LayoutLMv2](layoutlmv2#usage-layoutlmv2processor) regarding all possible use cases (the functionality of `UdopProcessor` is identical).\n\n- If using an own OCR engine of choice, one recommendation is Azure's [Read API](https://learn.microsoft.com/en-us/azure/ai-services/computer-vision/how-to/call-read-api), which supports so-called line segments. Use of segment position embeddings typically results in better performance.\n- At inference time, it's recommended to use the `generate` method to autoregressively generate text given a document image.\n- The model has been pre-trained on both self-supervised and supervised objectives. One can use the various task prefixes (prompts) used during pre-training to test out the out-of-the-box capabilities. For instance, the model can be prompted with \"Question answering. What is the date?\", as \"Question answering.\" is the task prefix used during pre-training for DocVQA. Refer to the [paper](https://arxiv.org/abs/2212.02623) (table 1) for all task prefixes."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a57979a2-c6de-4664-96c1-8840c8e85cb2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/udop", "title": "UDOP", "tokens": 1659, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a57979a2-c6de-4664-96c1-8840c8e85cb2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/udop\", \"title\": \"UDOP\", \"tokens\": 1659, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3f2e41d6-9cc9-480a-ac9c-5e750cd18cc9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/udop\", \"title\": \"UDOP\", \"tokens\": 1659, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"048971be410d1ad307cfdf125f112d6f67c3032221de1cbcc5e82cfd46a97328\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d0a91d4a-a7df-46cb-b609-cd2e9ae4c7d8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/udop\", \"title\": \"UDOP\", \"tokens\": 1659, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9c7f0a43fe3957870772511a2df9a8e0020179c7cb9d1535423ba7e8ccf5d930\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d0a0e4e5-abe0-426e-b934-bc1c7176e56e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"05432d645f7a0cb96cde631a96879bcb4fd49942cb29691b93c96f6ce1968d96\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3246, \"end_char_idx\": 5826, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3f2e41d6-9cc9-480a-ac9c-5e750cd18cc9", "doc_id": "3f2e41d6-9cc9-480a-ac9c-5e750cd18cc9", "ref_doc_id": "3f2e41d6-9cc9-480a-ac9c-5e750cd18cc9", "chroma:document": "PDFs must be converted to images.\nimage = Image.open(name_of_your_document).convert(\"RGB\")\n\nwidth, height = image.size\n```\n\nOne can use [`UdopProcessor`] to prepare images and text for the model, which takes care of all of this. By default, this class uses the Tesseract engine to extract a list of words and boxes (coordinates) from a given document. Its functionality is equivalent to that of [`LayoutLMv3Processor`], hence it supports passing either `apply_ocr=False` in case you prefer to use your own OCR engine or `apply_ocr=True` in case you want the default OCR engine to be used. Refer to the [usage guide of LayoutLMv2](layoutlmv2#usage-layoutlmv2processor) regarding all possible use cases (the functionality of `UdopProcessor` is identical).\n\n- If using an own OCR engine of choice, one recommendation is Azure's [Read API](https://learn.microsoft.com/en-us/azure/ai-services/computer-vision/how-to/call-read-api), which supports so-called line segments. Use of segment position embeddings typically results in better performance.\n- At inference time, it's recommended to use the `generate` method to autoregressively generate text given a document image.\n- The model has been pre-trained on both self-supervised and supervised objectives. One can use the various task prefixes (prompts) used during pre-training to test out the out-of-the-box capabilities. For instance, the model can be prompted with \"Question answering. What is the date?\", as \"Question answering.\" is the task prefix used during pre-training for DocVQA. Refer to the [paper](https://arxiv.org/abs/2212.02623) (table 1) for all task prefixes.\n- One can also fine-tune [`UdopEncoderModel`], which is the encoder-only part of UDOP, which can be seen as a LayoutLMv3-like Transformer encoder. For discriminative tasks, one can just add a linear classifier on top of it and fine-tune it on a labeled dataset.\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/microsoft/UDOP).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with UDOP. If\nyou're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll\nreview it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n- Demo notebooks regarding UDOP can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/UDOP) that show how\nto fine-tune UDOP on a custom dataset as well as inference."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: d0a0e4e5-abe0-426e-b934-bc1c7176e56e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/udop", "title": "UDOP", "tokens": 1659, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d0a0e4e5-abe0-426e-b934-bc1c7176e56e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/udop\", \"title\": \"UDOP\", \"tokens\": 1659, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3f2e41d6-9cc9-480a-ac9c-5e750cd18cc9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/udop\", \"title\": \"UDOP\", \"tokens\": 1659, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"048971be410d1ad307cfdf125f112d6f67c3032221de1cbcc5e82cfd46a97328\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a57979a2-c6de-4664-96c1-8840c8e85cb2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/udop\", \"title\": \"UDOP\", \"tokens\": 1659, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1510f92cd9bf9229593b36ce8d1475ffc02ad70560c6ad062623971670b79a7c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4213, \"end_char_idx\": 6445, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3f2e41d6-9cc9-480a-ac9c-5e750cd18cc9", "doc_id": "3f2e41d6-9cc9-480a-ac9c-5e750cd18cc9", "ref_doc_id": "3f2e41d6-9cc9-480a-ac9c-5e750cd18cc9", "chroma:document": "Use of segment position embeddings typically results in better performance.\n- At inference time, it's recommended to use the `generate` method to autoregressively generate text given a document image.\n- The model has been pre-trained on both self-supervised and supervised objectives. One can use the various task prefixes (prompts) used during pre-training to test out the out-of-the-box capabilities. For instance, the model can be prompted with \"Question answering. What is the date?\", as \"Question answering.\" is the task prefix used during pre-training for DocVQA. Refer to the [paper](https://arxiv.org/abs/2212.02623) (table 1) for all task prefixes.\n- One can also fine-tune [`UdopEncoderModel`], which is the encoder-only part of UDOP, which can be seen as a LayoutLMv3-like Transformer encoder. For discriminative tasks, one can just add a linear classifier on top of it and fine-tune it on a labeled dataset.\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/microsoft/UDOP).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with UDOP. If\nyou're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll\nreview it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n- Demo notebooks regarding UDOP can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/UDOP) that show how\nto fine-tune UDOP on a custom dataset as well as inference. \ud83c\udf0e\n- [Document question answering task guide](../tasks/document_question_answering)\n\n## UdopConfig\n\n[[autodoc]] UdopConfig\n\n## UdopTokenizer\n\n[[autodoc]] UdopTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## UdopTokenizerFast\n\n[[autodoc]] UdopTokenizerFast\n\n## UdopProcessor\n\n[[autodoc]] UdopProcessor\n    - __call__\n\n## UdopModel\n\n[[autodoc]] UdopModel\n    - forward\n\n## UdopForConditionalGeneration\n\n[[autodoc]] UdopForConditionalGeneration\n    - forward\n\n## UdopEncoderModel\n\n[[autodoc]] UdopEncoderModel\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: ddbf05ff-cd23-4724-a962-272678959166
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llama3", "title": "Llama3", "tokens": 1415, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ddbf05ff-cd23-4724-a962-272678959166\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama3\", \"title\": \"Llama3\", \"tokens\": 1415, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"76cb02bd-1c97-4f83-96f9-dad056d035bb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama3\", \"title\": \"Llama3\", \"tokens\": 1415, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2c36a307f6db561d9ce6fffdacc8d81f809d3a1714d09c6e3cac163db98dc1b1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3a1343c8-3498-4a58-b450-434bab5eec88\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2ac8c6fd5b8eba52c1f6759982d23fc8f52cdb7ad9c736d321f30c0f55a4fc6b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3199, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "76cb02bd-1c97-4f83-96f9-dad056d035bb", "doc_id": "76cb02bd-1c97-4f83-96f9-dad056d035bb", "ref_doc_id": "76cb02bd-1c97-4f83-96f9-dad056d035bb", "chroma:document": "# Llama3\n\n```py3\nimport transformers\nimport torch\n\nmodel_id = \"meta-llama/Meta-Llama-3-8B\"\n\npipeline = transformers.pipeline(\"text-generation\", model=model_id, model_kwargs={\"torch_dtype\": torch.bfloat16}, device_map=\"auto\")\npipeline(\"Hey how are you doing today?\")\n```\n\n## Overview\n\nThe Llama3 model was proposed in [Introducing Meta Llama 3: The most capable openly available LLM to date](https://ai.meta.com/blog/meta-llama-3/) by the meta AI team.\n\nThe abstract from the blogpost is the following:\n\n*Today, we\u2019re excited to share the first two models of the next generation of Llama, Meta Llama 3, available for broad use. This release features pretrained and instruction-fine-tuned language models with 8B and 70B parameters that can support a broad range of use cases. This next generation of Llama demonstrates state-of-the-art performance on a wide range of industry benchmarks and offers new capabilities, including improved reasoning. We believe these are the best open source models of their class, period. In support of our longstanding open approach, we\u2019re putting Llama 3 in the hands of the community. We want to kickstart the next wave of innovation in AI across the stack\u2014from applications to developer tools to evals to inference optimizations and more. We can\u2019t wait to see what you build and look forward to your feedback.*\n\nCheckout all Llama3 model checkpoints [here](https://huggingface.co/models?search=llama3).\nThe original code of the authors can be found [here](https://github.com/meta-llama/llama3).\n\n## Usage tips\n\n<Tip warning={true}>\n\nThe `Llama3` models were trained using `bfloat16`, but the original inference uses `float16`. The checkpoints uploaded on the Hub use `torch_dtype = 'float16'`, which will be\nused by the `AutoModel` API to cast the checkpoints from `torch.float32` to `torch.float16`. \n\nThe `dtype` of the online weights is mostly irrelevant unless you are using `torch_dtype=\"auto\"` when initializing a model using `model = AutoModelForCausalLM.from_pretrained(\"path\", torch_dtype = \"auto\")`. The reason is that the model will first be downloaded ( using the `dtype` of the checkpoints online), then it will be casted to the default `dtype` of `torch` (becomes `torch.float32`), and finally, if there is a `torch_dtype` provided in the config, it will be used. \n\nTraining the model in `float16` is not recommended and is known to produce `nan`; as such, the model should be trained in `bfloat16`.\n\n</Tip>\n\nTips:\n\n- Weights for the Llama3 models can be obtained by filling out [this form](https://ai.meta.com/resources/models-and-libraries/llama-downloads/)\n- The architecture is exactly the same as Llama2.\n- The tokenizer is a BPE model based on [tiktoken](https://github.com/openai/tiktoken) (vs the one based on sentencepiece implementation for Llama2). The main difference that it ignores BPE merge rules when an input token is part of the vocab. This means that if no merge exist to produce `\"hugging\"`, instead of having the smallest units, like `[\"hug\",\"ging\"] form 2 tokens, if `\"hugging\"` is part of the vocab, it will be automatically returned as a token.\n- The original model uses `pad_id = -1` which means that there is no padding token."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 3a1343c8-3498-4a58-b450-434bab5eec88
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llama3", "title": "Llama3", "tokens": 1415, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3a1343c8-3498-4a58-b450-434bab5eec88\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama3\", \"title\": \"Llama3\", \"tokens\": 1415, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"76cb02bd-1c97-4f83-96f9-dad056d035bb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama3\", \"title\": \"Llama3\", \"tokens\": 1415, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2c36a307f6db561d9ce6fffdacc8d81f809d3a1714d09c6e3cac163db98dc1b1\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ddbf05ff-cd23-4724-a962-272678959166\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama3\", \"title\": \"Llama3\", \"tokens\": 1415, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3b88e1c5eb23cedba9faacf13dfedd943ebfd4341a8fdb1f44d4c6ef94cd21ea\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9e8db2c5-51d3-462e-b289-d1eb77f2149b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"78edebbbb2fda98dbdf33156ba4d8c3f88527a5ce2c2425bb0d0ca556e264b38\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1660, \"end_char_idx\": 4885, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "76cb02bd-1c97-4f83-96f9-dad056d035bb", "doc_id": "76cb02bd-1c97-4f83-96f9-dad056d035bb", "ref_doc_id": "76cb02bd-1c97-4f83-96f9-dad056d035bb", "chroma:document": "The checkpoints uploaded on the Hub use `torch_dtype = 'float16'`, which will be\nused by the `AutoModel` API to cast the checkpoints from `torch.float32` to `torch.float16`. \n\nThe `dtype` of the online weights is mostly irrelevant unless you are using `torch_dtype=\"auto\"` when initializing a model using `model = AutoModelForCausalLM.from_pretrained(\"path\", torch_dtype = \"auto\")`. The reason is that the model will first be downloaded ( using the `dtype` of the checkpoints online), then it will be casted to the default `dtype` of `torch` (becomes `torch.float32`), and finally, if there is a `torch_dtype` provided in the config, it will be used. \n\nTraining the model in `float16` is not recommended and is known to produce `nan`; as such, the model should be trained in `bfloat16`.\n\n</Tip>\n\nTips:\n\n- Weights for the Llama3 models can be obtained by filling out [this form](https://ai.meta.com/resources/models-and-libraries/llama-downloads/)\n- The architecture is exactly the same as Llama2.\n- The tokenizer is a BPE model based on [tiktoken](https://github.com/openai/tiktoken) (vs the one based on sentencepiece implementation for Llama2). The main difference that it ignores BPE merge rules when an input token is part of the vocab. This means that if no merge exist to produce `\"hugging\"`, instead of having the smallest units, like `[\"hug\",\"ging\"] form 2 tokens, if `\"hugging\"` is part of the vocab, it will be automatically returned as a token.\n- The original model uses `pad_id = -1` which means that there is no padding token. We can't have the same logic, make sure to add a padding token using `tokenizer.add_special_tokens({\"pad_token\":\"<pad>\"})` and resize the token embedding accordingly. You should also set the `model.config.pad_token_id`. The `embed_tokens` layer of the model is initialized with `self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.config.padding_idx)`, which makes sure that encoding the padding token will output zeros, so passing it when initializing is recommended.\n- The original checkpoint can be converted using the [conversion script](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/convert_llama_weights_to_hf.py). The script can be called with the following (example) command:\n\n```bash\npython src/transformers/models/llama/convert_llama_weights_to_hf.py \\\n    --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path --llama_version 3\n```\n\n- After conversion, the model and tokenizer can be loaded via:\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\"/output/path\")\nmodel = AutoModelForCausalLM.from_pretrained(\"/output/path\")\n```\n\nNote that executing the script requires enough CPU RAM to host the whole model in float16 precision (even if the biggest versions\ncome in several checkpoints they each contain a part of each weight of the model, so we need to load them all in RAM). For the 75B model, it's thus 145GB of RAM needed.\n\n- When using Flash Attention 2 via `attn_implementation=\"flash_attention_2\"`, don't pass `torch_dtype` to the `from_pretrained` class method and use Automatic Mixed-Precision training."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 9e8db2c5-51d3-462e-b289-d1eb77f2149b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llama3", "title": "Llama3", "tokens": 1415, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9e8db2c5-51d3-462e-b289-d1eb77f2149b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama3\", \"title\": \"Llama3\", \"tokens\": 1415, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"76cb02bd-1c97-4f83-96f9-dad056d035bb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama3\", \"title\": \"Llama3\", \"tokens\": 1415, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2c36a307f6db561d9ce6fffdacc8d81f809d3a1714d09c6e3cac163db98dc1b1\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3a1343c8-3498-4a58-b450-434bab5eec88\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama3\", \"title\": \"Llama3\", \"tokens\": 1415, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1c108ce2ac8c1d8ea4b09d44b53e83c361280d03bd4d77e043d54b6eeba13e0c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3200, \"end_char_idx\": 5285, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "76cb02bd-1c97-4f83-96f9-dad056d035bb", "doc_id": "76cb02bd-1c97-4f83-96f9-dad056d035bb", "ref_doc_id": "76cb02bd-1c97-4f83-96f9-dad056d035bb", "chroma:document": "We can't have the same logic, make sure to add a padding token using `tokenizer.add_special_tokens({\"pad_token\":\"<pad>\"})` and resize the token embedding accordingly. You should also set the `model.config.pad_token_id`. The `embed_tokens` layer of the model is initialized with `self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.config.padding_idx)`, which makes sure that encoding the padding token will output zeros, so passing it when initializing is recommended.\n- The original checkpoint can be converted using the [conversion script](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/convert_llama_weights_to_hf.py). The script can be called with the following (example) command:\n\n```bash\npython src/transformers/models/llama/convert_llama_weights_to_hf.py \\\n    --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path --llama_version 3\n```\n\n- After conversion, the model and tokenizer can be loaded via:\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\"/output/path\")\nmodel = AutoModelForCausalLM.from_pretrained(\"/output/path\")\n```\n\nNote that executing the script requires enough CPU RAM to host the whole model in float16 precision (even if the biggest versions\ncome in several checkpoints they each contain a part of each weight of the model, so we need to load them all in RAM). For the 75B model, it's thus 145GB of RAM needed.\n\n- When using Flash Attention 2 via `attn_implementation=\"flash_attention_2\"`, don't pass `torch_dtype` to the `from_pretrained` class method and use Automatic Mixed-Precision training. When using `Trainer`, it is simply specifying either `fp16` or `bf16` to `True`. Otherwise, make sure you are using `torch.autocast`. This is required because the Flash Attention only support `fp16` and `bf16` data type.\n\n## Resources\nA ton of cool resources are already available on the documentation page of [Llama2](./llama2), inviting contributors to add new resources curated for Llama3 here! \ud83e\udd17"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 4c686975-4492-4ee3-b144-1cdfe0610f97
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bert-japanese", "title": "BertJapanese", "tokens": 670, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4c686975-4492-4ee3-b144-1cdfe0610f97\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert-japanese\", \"title\": \"BertJapanese\", \"tokens\": 670, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2145c4c0-5654-414f-b157-f260a7a3bbc7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bert-japanese\", \"title\": \"BertJapanese\", \"tokens\": 670, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"34c3999e9470768dfc788f23fc66e1952bb83129a71c4160ac3ea9618b882398\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1862, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2145c4c0-5654-414f-b157-f260a7a3bbc7", "doc_id": "2145c4c0-5654-414f-b157-f260a7a3bbc7", "ref_doc_id": "2145c4c0-5654-414f-b157-f260a7a3bbc7", "chroma:document": "# BertJapanese\n\n## Overview\n\nThe BERT models trained on Japanese text.\n\nThere are models with two different tokenization methods:\n\n- Tokenize with MeCab and WordPiece. This requires some extra dependencies, [fugashi](https://github.com/polm/fugashi) which is a wrapper around [MeCab](https://taku910.github.io/mecab/).\n- Tokenize into characters.\n\nTo use *MecabTokenizer*, you should `pip install transformers[\"ja\"]` (or `pip install -e .[\"ja\"]` if you install\nfrom source) to install dependencies.\n\nSee [details on cl-tohoku repository](https://github.com/cl-tohoku/bert-japanese).\n\nExample of using a model with MeCab and WordPiece tokenization:\n\n```python\n>>> import torch\n>>> from transformers import AutoModel, AutoTokenizer\n\n>>> bertjapanese = AutoModel.from_pretrained(\"cl-tohoku/bert-base-japanese\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"cl-tohoku/bert-base-japanese\")\n\n>>> ## Input Japanese Text\n>>> line = \"\u543e\u8f29\u306f\u732b\u3067\u3042\u308b\u3002\"\n\n>>> inputs = tokenizer(line, return_tensors=\"pt\")\n\n>>> print(tokenizer.decode(inputs[\"input_ids\"][0]))\n[CLS] \u543e\u8f29 \u306f \u732b \u3067 \u3042\u308b \u3002 [SEP]\n\n>>> outputs = bertjapanese(**inputs)\n```\n\nExample of using a model with Character tokenization:\n\n```python\n>>> bertjapanese = AutoModel.from_pretrained(\"cl-tohoku/bert-base-japanese-char\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"cl-tohoku/bert-base-japanese-char\")\n\n>>> ## Input Japanese Text\n>>> line = \"\u543e\u8f29\u306f\u732b\u3067\u3042\u308b\u3002\"\n\n>>> inputs = tokenizer(line, return_tensors=\"pt\")\n\n>>> print(tokenizer.decode(inputs[\"input_ids\"][0]))\n[CLS] \u543e \u8f29 \u306f \u732b \u3067 \u3042 \u308b \u3002 [SEP]\n\n>>> outputs = bertjapanese(**inputs)\n```\n\nThis model was contributed by [cl-tohoku](https://huggingface.co/cl-tohoku).\n\n<Tip> \n\nThis implementation is the same as BERT, except for tokenization method. Refer to [BERT documentation](bert) for \nAPI reference information.  \n\n</Tip>\n\n\n## BertJapaneseTokenizer\n\n[[autodoc]] BertJapaneseTokenizer"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 00f8e287-d423-46af-9c47-e4fdb5282169
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/align", "title": "ALIGN", "tokens": 1281, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"00f8e287-d423-46af-9c47-e4fdb5282169\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/align\", \"title\": \"ALIGN\", \"tokens\": 1281, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5ce19799-4b8f-432f-b958-a0abb4024ad8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/align\", \"title\": \"ALIGN\", \"tokens\": 1281, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d73cb1df62e0391c744e0543d87805f667e500553fb06d91e9058ad45e06ff84\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f6cc2ab2-0db4-455c-9658-38485eebe691\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"23039f70f72a47ad99b8b4fc0457338739dc4ea1a0e4b3abdad3e265f4a7e302\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3296, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5ce19799-4b8f-432f-b958-a0abb4024ad8", "doc_id": "5ce19799-4b8f-432f-b958-a0abb4024ad8", "ref_doc_id": "5ce19799-4b8f-432f-b958-a0abb4024ad8", "chroma:document": "# ALIGN\n\n## Overview\n\nThe ALIGN model was proposed in [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://arxiv.org/abs/2102.05918) by Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yunhsuan Sung, Zhen Li, Tom Duerig. ALIGN is a multi-modal vision and language model. It can be used for image-text similarity and for zero-shot image classification. ALIGN features a dual-encoder architecture with [EfficientNet](efficientnet) as its vision encoder and [BERT](bert) as its text encoder, and learns to align visual and text representations with contrastive learning. Unlike previous work, ALIGN leverages a massive noisy dataset and shows that the scale of the corpus can be used to achieve SOTA representations with a simple recipe.\n\nThe abstract from the paper is the following:\n\n*Pre-trained representations are becoming crucial for many NLP and perception tasks. While representation learning in NLP has transitioned to training on raw text without human annotations, visual and vision-language representations still rely heavily on curated training datasets that are expensive or require expert knowledge. For vision applications, representations are mostly learned using datasets with explicit class labels such as ImageNet or OpenImages. For vision-language, popular datasets like Conceptual Captions, MSCOCO, or CLIP all involve a non-trivial data collection (and cleaning) process. This costly curation process limits the size of datasets and hence hinders the scaling of trained models. In this paper, we leverage a noisy dataset of over one billion image alt-text pairs, obtained without expensive filtering or post-processing steps in the Conceptual Captions dataset. A simple dual-encoder architecture learns to align visual and language representations of the image and text pairs using a contrastive loss. We show that the scale of our corpus can make up for its noise and leads to state-of-the-art representations even with such a simple learning scheme. Our visual representation achieves strong performance when transferred to classification tasks such as ImageNet and VTAB. The aligned visual and language representations enables zero-shot image classification and also set new state-of-the-art results on Flickr30K and MSCOCO image-text retrieval benchmarks, even when compared with more sophisticated cross-attention models. The representations also enable cross-modality search with complex text and text + image queries.*\n\nThis model was contributed by [Alara Dirik](https://huggingface.co/adirik).\nThe original code is not released, this implementation is based on the Kakao Brain implementation based on the original paper.\n\n## Usage example\n\nALIGN uses EfficientNet to get visual features and BERT to get the text features. Both the text and visual features are then projected to a latent space with identical dimension. The dot product between the projected image and text features is then used as a similarity score.\n\n[`AlignProcessor`] wraps [`EfficientNetImageProcessor`] and [`BertTokenizer`] into a single instance to both encode the text and preprocess the images. The following example shows how to get the image-text similarity scores using [`AlignProcessor`] and [`AlignModel`]."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f6cc2ab2-0db4-455c-9658-38485eebe691
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/align", "title": "ALIGN", "tokens": 1281, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f6cc2ab2-0db4-455c-9658-38485eebe691\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/align\", \"title\": \"ALIGN\", \"tokens\": 1281, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5ce19799-4b8f-432f-b958-a0abb4024ad8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/align\", \"title\": \"ALIGN\", \"tokens\": 1281, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d73cb1df62e0391c744e0543d87805f667e500553fb06d91e9058ad45e06ff84\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"00f8e287-d423-46af-9c47-e4fdb5282169\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/align\", \"title\": \"ALIGN\", \"tokens\": 1281, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a205e192e8a59273b8ea4173c59d2df0834e8678fb79e598cd41f4d61b36f69c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"af7d6393-174f-4714-8342-05cc26251761\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"971c9c808d8a25ede08518cb71ba0b6b5cc1519e88f514c92642ac688c0d1153\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1327, \"end_char_idx\": 4712, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5ce19799-4b8f-432f-b958-a0abb4024ad8", "doc_id": "5ce19799-4b8f-432f-b958-a0abb4024ad8", "ref_doc_id": "5ce19799-4b8f-432f-b958-a0abb4024ad8", "chroma:document": "For vision-language, popular datasets like Conceptual Captions, MSCOCO, or CLIP all involve a non-trivial data collection (and cleaning) process. This costly curation process limits the size of datasets and hence hinders the scaling of trained models. In this paper, we leverage a noisy dataset of over one billion image alt-text pairs, obtained without expensive filtering or post-processing steps in the Conceptual Captions dataset. A simple dual-encoder architecture learns to align visual and language representations of the image and text pairs using a contrastive loss. We show that the scale of our corpus can make up for its noise and leads to state-of-the-art representations even with such a simple learning scheme. Our visual representation achieves strong performance when transferred to classification tasks such as ImageNet and VTAB. The aligned visual and language representations enables zero-shot image classification and also set new state-of-the-art results on Flickr30K and MSCOCO image-text retrieval benchmarks, even when compared with more sophisticated cross-attention models. The representations also enable cross-modality search with complex text and text + image queries.*\n\nThis model was contributed by [Alara Dirik](https://huggingface.co/adirik).\nThe original code is not released, this implementation is based on the Kakao Brain implementation based on the original paper.\n\n## Usage example\n\nALIGN uses EfficientNet to get visual features and BERT to get the text features. Both the text and visual features are then projected to a latent space with identical dimension. The dot product between the projected image and text features is then used as a similarity score.\n\n[`AlignProcessor`] wraps [`EfficientNetImageProcessor`] and [`BertTokenizer`] into a single instance to both encode the text and preprocess the images. The following example shows how to get the image-text similarity scores using [`AlignProcessor`] and [`AlignModel`].\n\n```python\nimport requests\nimport torch\nfrom PIL import Image\nfrom transformers import AlignProcessor, AlignModel\n\nprocessor = AlignProcessor.from_pretrained(\"kakaobrain/align-base\")\nmodel = AlignModel.from_pretrained(\"kakaobrain/align-base\")\n\nurl = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nimage = Image.open(requests.get(url, stream=True).raw)\ncandidate_labels = [\"an image of a cat\", \"an image of a dog\"]\n\ninputs = processor(text=candidate_labels, images=image, return_tensors=\"pt\")\n\nwith torch.no_grad():\n    outputs = model(**inputs)\n\n# this is the image-text similarity score\nlogits_per_image = outputs.logits_per_image\n\n# we can take the softmax to get the label probabilities\nprobs = logits_per_image.softmax(dim=1)\nprint(probs)\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with ALIGN.\n\n- A blog post on [ALIGN and the COYO-700M dataset](https://huggingface.co/blog/vit-align).\n- A zero-shot image classification [demo](https://huggingface.co/spaces/adirik/ALIGN-zero-shot-image-classification).\n- [Model card](https://huggingface.co/kakaobrain/align-base) of `kakaobrain/align-base` model.\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it. The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: af7d6393-174f-4714-8342-05cc26251761
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/align", "title": "ALIGN", "tokens": 1281, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"af7d6393-174f-4714-8342-05cc26251761\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/align\", \"title\": \"ALIGN\", \"tokens\": 1281, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5ce19799-4b8f-432f-b958-a0abb4024ad8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/align\", \"title\": \"ALIGN\", \"tokens\": 1281, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d73cb1df62e0391c744e0543d87805f667e500553fb06d91e9058ad45e06ff84\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f6cc2ab2-0db4-455c-9658-38485eebe691\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/align\", \"title\": \"ALIGN\", \"tokens\": 1281, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5dfceeea1a7b227c57516cfa82fff2a442ebc632573017c21f44cd62af6024f3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3028, \"end_char_idx\": 5161, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5ce19799-4b8f-432f-b958-a0abb4024ad8", "doc_id": "5ce19799-4b8f-432f-b958-a0abb4024ad8", "ref_doc_id": "5ce19799-4b8f-432f-b958-a0abb4024ad8", "chroma:document": "[`AlignProcessor`] wraps [`EfficientNetImageProcessor`] and [`BertTokenizer`] into a single instance to both encode the text and preprocess the images. The following example shows how to get the image-text similarity scores using [`AlignProcessor`] and [`AlignModel`].\n\n```python\nimport requests\nimport torch\nfrom PIL import Image\nfrom transformers import AlignProcessor, AlignModel\n\nprocessor = AlignProcessor.from_pretrained(\"kakaobrain/align-base\")\nmodel = AlignModel.from_pretrained(\"kakaobrain/align-base\")\n\nurl = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nimage = Image.open(requests.get(url, stream=True).raw)\ncandidate_labels = [\"an image of a cat\", \"an image of a dog\"]\n\ninputs = processor(text=candidate_labels, images=image, return_tensors=\"pt\")\n\nwith torch.no_grad():\n    outputs = model(**inputs)\n\n# this is the image-text similarity score\nlogits_per_image = outputs.logits_per_image\n\n# we can take the softmax to get the label probabilities\nprobs = logits_per_image.softmax(dim=1)\nprint(probs)\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with ALIGN.\n\n- A blog post on [ALIGN and the COYO-700M dataset](https://huggingface.co/blog/vit-align).\n- A zero-shot image classification [demo](https://huggingface.co/spaces/adirik/ALIGN-zero-shot-image-classification).\n- [Model card](https://huggingface.co/kakaobrain/align-base) of `kakaobrain/align-base` model.\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it. The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## AlignConfig\n\n[[autodoc]] AlignConfig\n    - from_text_vision_configs\n\n## AlignTextConfig\n\n[[autodoc]] AlignTextConfig\n\n## AlignVisionConfig\n\n[[autodoc]] AlignVisionConfig\n\n## AlignProcessor\n\n[[autodoc]] AlignProcessor\n\n## AlignModel\n\n[[autodoc]] AlignModel\n    - forward\n    - get_text_features\n    - get_image_features\n\n## AlignTextModel\n\n[[autodoc]] AlignTextModel\n    - forward\n\n## AlignVisionModel\n\n[[autodoc]] AlignVisionModel\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: cad0eef8-76e8-434c-9233-c0254deeec1c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/dialogpt", "title": "DialoGPT", "tokens": 729, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"cad0eef8-76e8-434c-9233-c0254deeec1c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dialogpt\", \"title\": \"DialoGPT\", \"tokens\": 729, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a8e81dbf-0ad8-4c2c-ac77-3cfbcbe9cdcd\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dialogpt\", \"title\": \"DialoGPT\", \"tokens\": 729, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"acd00c53e1f6f865d113365069be34dd15e5153ec00030034f686aaade1a8b58\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2398, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a8e81dbf-0ad8-4c2c-ac77-3cfbcbe9cdcd", "doc_id": "a8e81dbf-0ad8-4c2c-ac77-3cfbcbe9cdcd", "ref_doc_id": "a8e81dbf-0ad8-4c2c-ac77-3cfbcbe9cdcd", "chroma:document": "# DialoGPT\n\n## Overview\n\nDialoGPT was proposed in [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://arxiv.org/abs/1911.00536) by Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao,\nJianfeng Gao, Jingjing Liu, Bill Dolan. It's a GPT2 Model trained on 147M conversation-like exchanges extracted from\nReddit.\n\nThe abstract from the paper is the following:\n\n*We present a large, tunable neural conversational response generation model, DialoGPT (dialogue generative pre-trained\ntransformer). Trained on 147M conversation-like exchanges extracted from Reddit comment chains over a period spanning\nfrom 2005 through 2017, DialoGPT extends the Hugging Face PyTorch transformer to attain a performance close to human\nboth in terms of automatic and human evaluation in single-turn dialogue settings. We show that conversational systems\nthat leverage DialoGPT generate more relevant, contentful and context-consistent responses than strong baseline\nsystems. The pre-trained model and training pipeline are publicly released to facilitate research into neural response\ngeneration and the development of more intelligent open-domain dialogue systems.*\n\nThe original code can be found [here](https://github.com/microsoft/DialoGPT).\n\n## Usage tips\n\n- DialoGPT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather\n  than the left.\n- DialoGPT was trained with a causal language modeling (CLM) objective on conversational data and is therefore powerful\n  at response generation in open-domain dialogue systems.\n- DialoGPT enables the user to create a chat bot in just 10 lines of code as shown on [DialoGPT's model card](https://huggingface.co/microsoft/DialoGPT-medium).\n\nTraining:\n\nIn order to train or fine-tune DialoGPT, one can use causal language modeling training. To cite the official paper: *We\nfollow the OpenAI GPT-2 to model a multiturn dialogue session as a long text and frame the generation task as language\nmodeling. We first concatenate all dialog turns within a dialogue session into a long text x_1,..., x_N (N is the\nsequence length), ended by the end-of-text token.* For more information please confer to the original paper.\n\n<Tip>\n\nDialoGPT's architecture is based on the GPT2 model, refer to [GPT2's documentation page](gpt2) for API reference and examples.\n\n</Tip>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 3d15d9c7-7139-4251-bea7-a661d086a7d8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/regnet", "title": "RegNet", "tokens": 981, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3d15d9c7-7139-4251-bea7-a661d086a7d8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/regnet\", \"title\": \"RegNet\", \"tokens\": 981, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c3f58060-abe7-45bf-ab39-81bcb6d2a585\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/regnet\", \"title\": \"RegNet\", \"tokens\": 981, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"25666284c633cc766faa610f9b3e9860b36485007df2712da6302236fa8fa7cc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"232ea0b6-d9a1-43d6-acac-6ef07c2292dc\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4f0ee605765d5c3213adb84b51d56f0286fbd26485f906658e943014a859d03d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3010, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c3f58060-abe7-45bf-ab39-81bcb6d2a585", "doc_id": "c3f58060-abe7-45bf-ab39-81bcb6d2a585", "ref_doc_id": "c3f58060-abe7-45bf-ab39-81bcb6d2a585", "chroma:document": "# RegNet\n\n## Overview\n\nThe RegNet model was proposed in [Designing Network Design Spaces](https://arxiv.org/abs/2003.13678) by Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, Piotr Doll\u00e1r.\n\nThe authors design search spaces to perform Neural Architecture Search (NAS). They first start from a high dimensional search space and iteratively reduce the search space by empirically applying constraints based on the best-performing models sampled by the current search space.\n\nThe abstract from the paper is the following:\n\n*In this work, we present a new network design paradigm. Our goal is to help advance the understanding of network design and discover design principles that generalize across settings. Instead of focusing on designing individual network instances, we design network design spaces that parametrize populations of networks. The overall process is analogous to classic manual design of networks, but elevated to the design space level. Using our methodology we explore the structure aspect of network design and arrive at a low-dimensional design space consisting of simple, regular networks that we call RegNet. The core insight of the RegNet parametrization is surprisingly simple: widths and depths of good networks can be explained by a quantized linear function. We analyze the RegNet design space and arrive at interesting findings that do not match the current practice of network design. The RegNet design space provides simple and fast networks that work well across a wide range of flop regimes. Under comparable training settings and flops, the RegNet models outperform the popular EfficientNet models while being up to 5x faster on GPUs.*\n\nThis model was contributed by [Francesco](https://huggingface.co/Francesco). The TensorFlow version of the model\nwas contributed by [sayakpaul](https://huggingface.co/sayakpaul) and [ariG23498](https://huggingface.co/ariG23498).\nThe original code can be found [here](https://github.com/facebookresearch/pycls).\n\nThe huge 10B model from [Self-supervised Pretraining of Visual Features in the Wild](https://arxiv.org/abs/2103.01988), \ntrained on  one billion Instagram images, is available on the [hub](https://huggingface.co/facebook/regnet-y-10b-seer)\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with RegNet.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`RegNetForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 232ea0b6-d9a1-43d6-acac-6ef07c2292dc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/regnet", "title": "RegNet", "tokens": 981, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"232ea0b6-d9a1-43d6-acac-6ef07c2292dc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/regnet\", \"title\": \"RegNet\", \"tokens\": 981, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c3f58060-abe7-45bf-ab39-81bcb6d2a585\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/regnet\", \"title\": \"RegNet\", \"tokens\": 981, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"25666284c633cc766faa610f9b3e9860b36485007df2712da6302236fa8fa7cc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3d15d9c7-7139-4251-bea7-a661d086a7d8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/regnet\", \"title\": \"RegNet\", \"tokens\": 981, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"61ea6b66784f3831368d140e72601bfa449404b6893b1ab83254ac47e767eac5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1304, \"end_char_idx\": 3582, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c3f58060-abe7-45bf-ab39-81bcb6d2a585", "doc_id": "c3f58060-abe7-45bf-ab39-81bcb6d2a585", "ref_doc_id": "c3f58060-abe7-45bf-ab39-81bcb6d2a585", "chroma:document": "We analyze the RegNet design space and arrive at interesting findings that do not match the current practice of network design. The RegNet design space provides simple and fast networks that work well across a wide range of flop regimes. Under comparable training settings and flops, the RegNet models outperform the popular EfficientNet models while being up to 5x faster on GPUs.*\n\nThis model was contributed by [Francesco](https://huggingface.co/Francesco). The TensorFlow version of the model\nwas contributed by [sayakpaul](https://huggingface.co/sayakpaul) and [ariG23498](https://huggingface.co/ariG23498).\nThe original code can be found [here](https://github.com/facebookresearch/pycls).\n\nThe huge 10B model from [Self-supervised Pretraining of Visual Features in the Wild](https://arxiv.org/abs/2103.01988), \ntrained on  one billion Instagram images, is available on the [hub](https://huggingface.co/facebook/regnet-y-10b-seer)\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with RegNet.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`RegNetForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## RegNetConfig\n\n[[autodoc]] RegNetConfig\n\n<frameworkcontent>\n<pt>\n\n## RegNetModel\n\n[[autodoc]] RegNetModel\n    - forward\n\n## RegNetForImageClassification\n\n[[autodoc]] RegNetForImageClassification\n    - forward\n\n</pt>\n<tf>\n\n## TFRegNetModel\n\n[[autodoc]] TFRegNetModel\n    - call\n\n## TFRegNetForImageClassification\n\n[[autodoc]] TFRegNetForImageClassification\n    - call\n\n</tf>\n<jax>\n\n## FlaxRegNetModel\n\n[[autodoc]] FlaxRegNetModel\n    - __call__\n\n## FlaxRegNetForImageClassification\n\n[[autodoc]] FlaxRegNetForImageClassification\n    - __call__\n</jax>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 67faa3b2-3d8c-4d08-b6b2-237fcb2f15fb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/depth_anything", "title": "Depth Anything", "tokens": 1332, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"67faa3b2-3d8c-4d08-b6b2-237fcb2f15fb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/depth_anything\", \"title\": \"Depth Anything\", \"tokens\": 1332, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ebe3e8a1-0d9b-4c11-8050-43b9567d9e21\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/depth_anything\", \"title\": \"Depth Anything\", \"tokens\": 1332, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"eb6c1de1530b1ad4af5fcd404d4b89d0e7407f1db35360580ccaafdb82361b8b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0d844a22-b3d0-495a-a7d2-425ed269e61c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fbd9d993ddbaa1750747c29fc7e5f658fcd5d5f385e50b6d419115539345120b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2703, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ebe3e8a1-0d9b-4c11-8050-43b9567d9e21", "doc_id": "ebe3e8a1-0d9b-4c11-8050-43b9567d9e21", "ref_doc_id": "ebe3e8a1-0d9b-4c11-8050-43b9567d9e21", "chroma:document": "# Depth Anything\n\n## Overview\n\nThe Depth Anything model was proposed in [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://arxiv.org/abs/2401.10891) by Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, Hengshuang Zhao. Depth Anything is based on the [DPT](dpt) architecture, trained on ~62 million images, obtaining state-of-the-art results for both relative and absolute depth estimation.\n\n<Tip>\n\n[Depth Anything V2](depth_anything_v2) was released in June 2024. It uses the same architecture as Depth Anything and therefore it is compatible with all code examples and existing workflows. However, it leverages synthetic data and a larger capacity teacher model to achieve much finer and robust depth predictions.\n\n</Tip>\n\nThe abstract from the paper is the following:\n\n*This work presents Depth Anything, a highly practical solution for robust monocular depth estimation. Without pursuing novel technical modules, we aim to build a simple yet powerful foundation model dealing with any images under any circumstances. To this end, we scale up the dataset by designing a data engine to collect and automatically annotate large-scale unlabeled data (~62M), which significantly enlarges the data coverage and thus is able to reduce the generalization error. We investigate two simple yet effective strategies that make data scaling-up promising. First, a more challenging optimization target is created by leveraging data augmentation tools. It compels the model to actively seek extra visual knowledge and acquire robust representations. Second, an auxiliary supervision is developed to enforce the model to inherit rich semantic priors from pre-trained encoders. We evaluate its zero-shot capabilities extensively, including six public datasets and randomly captured photos. It demonstrates impressive generalization ability. Further, through fine-tuning it with metric depth information from NYUv2 and KITTI, new SOTAs are set. Our better depth model also results in a better depth-conditioned ControlNet.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/depth_anything_overview.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Depth Anything overview. Taken from the <a href=\"https://arxiv.org/abs/2401.10891\">original paper</a>.</small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/LiheYoung/Depth-Anything).\n\n## Usage example\n\nThere are 2 main ways to use Depth Anything: either using the pipeline API, which abstracts away all the complexity for you, or by using the `DepthAnythingForDepthEstimation` class yourself."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0d844a22-b3d0-495a-a7d2-425ed269e61c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/depth_anything", "title": "Depth Anything", "tokens": 1332, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0d844a22-b3d0-495a-a7d2-425ed269e61c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/depth_anything\", \"title\": \"Depth Anything\", \"tokens\": 1332, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ebe3e8a1-0d9b-4c11-8050-43b9567d9e21\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/depth_anything\", \"title\": \"Depth Anything\", \"tokens\": 1332, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"eb6c1de1530b1ad4af5fcd404d4b89d0e7407f1db35360580ccaafdb82361b8b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"67faa3b2-3d8c-4d08-b6b2-237fcb2f15fb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/depth_anything\", \"title\": \"Depth Anything\", \"tokens\": 1332, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bb6bc6cbf27979716ade83b81609cb279f51a782357c8786c0fc8d54d664fa7c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f3221ad7-c1f2-4cf1-b97b-c14e8aad107d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d3ee77e0b0fead26cdefe4e9d335f68f62d6c752467ac8ae293802127f3af7c2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 914, \"end_char_idx\": 4524, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ebe3e8a1-0d9b-4c11-8050-43b9567d9e21", "doc_id": "ebe3e8a1-0d9b-4c11-8050-43b9567d9e21", "ref_doc_id": "ebe3e8a1-0d9b-4c11-8050-43b9567d9e21", "chroma:document": "Without pursuing novel technical modules, we aim to build a simple yet powerful foundation model dealing with any images under any circumstances. To this end, we scale up the dataset by designing a data engine to collect and automatically annotate large-scale unlabeled data (~62M), which significantly enlarges the data coverage and thus is able to reduce the generalization error. We investigate two simple yet effective strategies that make data scaling-up promising. First, a more challenging optimization target is created by leveraging data augmentation tools. It compels the model to actively seek extra visual knowledge and acquire robust representations. Second, an auxiliary supervision is developed to enforce the model to inherit rich semantic priors from pre-trained encoders. We evaluate its zero-shot capabilities extensively, including six public datasets and randomly captured photos. It demonstrates impressive generalization ability. Further, through fine-tuning it with metric depth information from NYUv2 and KITTI, new SOTAs are set. Our better depth model also results in a better depth-conditioned ControlNet.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/depth_anything_overview.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Depth Anything overview. Taken from the <a href=\"https://arxiv.org/abs/2401.10891\">original paper</a>.</small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/LiheYoung/Depth-Anything).\n\n## Usage example\n\nThere are 2 main ways to use Depth Anything: either using the pipeline API, which abstracts away all the complexity for you, or by using the `DepthAnythingForDepthEstimation` class yourself.\n\n### Pipeline API\n\nThe pipeline allows to use the model in a few lines of code:\n\n```python\n>>> from transformers import pipeline\n>>> from PIL import Image\n>>> import requests\n\n>>> # load pipe\n>>> pipe = pipeline(task=\"depth-estimation\", model=\"LiheYoung/depth-anything-small-hf\")\n\n>>> # load image\n>>> url = 'http://images.cocodataset.org/val2017/000000039769.jpg'\n>>> image = Image.open(requests.get(url, stream=True).raw)\n\n>>> # inference\n>>> depth = pipe(image)[\"depth\"]\n```\n\n### Using the model yourself\n\nIf you want to do the pre- and postprocessing yourself, here's how to do that:\n\n```python\n>>> from transformers import AutoImageProcessor, AutoModelForDepthEstimation\n>>> import torch\n>>> import numpy as np\n>>> from PIL import Image\n>>> import requests\n\n>>> url = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n\n>>> image_processor = AutoImageProcessor.from_pretrained(\"LiheYoung/depth-anything-small-hf\")\n>>> model = AutoModelForDepthEstimation.from_pretrained(\"LiheYoung/depth-anything-small-hf\")\n\n>>> # prepare image for the model\n>>> inputs = image_processor(images=image, return_tensors=\"pt\")\n\n>>> with torch.no_grad():\n...     outputs = model(**inputs)\n...     predicted_depth = outputs.predicted_depth\n\n>>> # interpolate to original size\n>>> prediction = torch.nn.functional.interpolate(\n...     predicted_depth.unsqueeze(1),\n...     size=image.size[::-1],\n...     mode=\"bicubic\",\n...     align_corners=False,\n... )\n\n>>> # visualize the prediction\n>>> output = prediction.squeeze().cpu().numpy()\n>>> formatted = (output * 255 / np.max(output)).astype(\"uint8\")\n>>> depth = Image.fromarray(formatted)\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Depth Anything."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f3221ad7-c1f2-4cf1-b97b-c14e8aad107d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/depth_anything", "title": "Depth Anything", "tokens": 1332, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f3221ad7-c1f2-4cf1-b97b-c14e8aad107d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/depth_anything\", \"title\": \"Depth Anything\", \"tokens\": 1332, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ebe3e8a1-0d9b-4c11-8050-43b9567d9e21\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/depth_anything\", \"title\": \"Depth Anything\", \"tokens\": 1332, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"eb6c1de1530b1ad4af5fcd404d4b89d0e7407f1db35360580ccaafdb82361b8b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0d844a22-b3d0-495a-a7d2-425ed269e61c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/depth_anything\", \"title\": \"Depth Anything\", \"tokens\": 1332, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ea96821cc2d41b2a57ac137d3326e9b3fbdb9df477f5d737bb9ea437f1f5050d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4526, \"end_char_idx\": 5210, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ebe3e8a1-0d9b-4c11-8050-43b9567d9e21", "doc_id": "ebe3e8a1-0d9b-4c11-8050-43b9567d9e21", "ref_doc_id": "ebe3e8a1-0d9b-4c11-8050-43b9567d9e21", "chroma:document": "- [Monocular depth estimation task guide](../tasks/depth_estimation)\n- A notebook showcasing inference with [`DepthAnythingForDepthEstimation`] can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/Depth%20Anything/Predicting_depth_in_an_image_with_Depth_Anything.ipynb). \ud83c\udf0e\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## DepthAnythingConfig\n\n[[autodoc]] DepthAnythingConfig\n\n## DepthAnythingForDepthEstimation\n\n[[autodoc]] DepthAnythingForDepthEstimation\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 88ec6d83-42d8-45f6-8578-8f6945916858
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/yolos", "title": "YOLOS", "tokens": 1419, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"88ec6d83-42d8-45f6-8578-8f6945916858\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/yolos\", \"title\": \"YOLOS\", \"tokens\": 1419, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c31ce59b-d53f-45c4-9a66-a81a2d10817f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/yolos\", \"title\": \"YOLOS\", \"tokens\": 1419, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"15e603461d5b610e0da53196578808c3a31bc7cd2f3d46b2d21862f97944ca20\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8f58e40e-3725-47e5-b0b9-ab2e85c984df\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c39c72ec2ac702891465730cf21f151721bb963b1b983f7f97b36876279dcd5b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3068, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c31ce59b-d53f-45c4-9a66-a81a2d10817f", "doc_id": "c31ce59b-d53f-45c4-9a66-a81a2d10817f", "ref_doc_id": "c31ce59b-d53f-45c4-9a66-a81a2d10817f", "chroma:document": "# YOLOS\n\n## Overview\n\nThe YOLOS model was proposed in [You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detection](https://arxiv.org/abs/2106.00666) by Yuxin Fang, Bencheng Liao, Xinggang Wang, Jiemin Fang, Jiyang Qi, Rui Wu, Jianwei Niu, Wenyu Liu.\nYOLOS proposes to just leverage the plain [Vision Transformer (ViT)](vit) for object detection, inspired by DETR. It turns out that a base-sized encoder-only Transformer can also achieve 42 AP on COCO, similar to DETR and much more complex frameworks such as Faster R-CNN.\n\nThe abstract from the paper is the following:\n\n*Can Transformer perform 2D object- and region-level recognition from a pure sequence-to-sequence perspective with minimal knowledge about the 2D spatial structure? To answer this question, we present You Only Look at One Sequence (YOLOS), a series of object detection models based on the vanilla Vision Transformer with the fewest possible modifications, region priors, as well as inductive biases of the target task. We find that YOLOS pre-trained on the mid-sized ImageNet-1k dataset only can already achieve quite competitive performance on the challenging COCO object detection benchmark, e.g., YOLOS-Base directly adopted from BERT-Base architecture can obtain 42.0 box AP on COCO val. We also discuss the impacts as well as limitations of current pre-train schemes and model scaling strategies for Transformer in vision through YOLOS.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/yolos_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> YOLOS architecture. Taken from the <a href=\"https://arxiv.org/abs/2106.00666\">original paper</a>.</small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/hustvl/YOLOS).\n\n## Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```\nfrom transformers import AutoModelForObjectDetection\nmodel = AutoModelForObjectDetection.from_pretrained(\"hustvl/yolos-base\", attn_implementation=\"sdpa\", torch_dtype=torch.float16)\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `hustvl/yolos-base` model, we saw the following speedups during inference."}


