# embeddings_queue

created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 3280cd42-0b2b-4c88-aeee-0385d7c0d3f3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3280cd42-0b2b-4c88-aeee-0385d7c0d3f3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/serialization\", \"title\": \"Export to ONNX\", \"tokens\": 2388, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fa46a645-8e6b-465b-ae69-3d51f650c846\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/serialization\", \"title\": \"Export to ONNX\", \"tokens\": 2388, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b97638da20ee8948778fc11261763dd7a2666a79a5866a41f99ad4299f694ae6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"80d8b397-2a54-4673-a046-c84516e2c1c3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/serialization\", \"title\": \"Export to ONNX\", \"tokens\": 2388, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5d6b1a6b3f867adb5925e021d8970e56a542c7fc72f6f14d396d2d0c35d40df5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9e28a275-03ef-4051-89b3-3677f3374e2e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5eb36c9de8909b05989aa9b7d03e7770055a28c4a9b5d7ef1c558ec87d97c263\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4084, \"end_char_idx\": 7106, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fa46a645-8e6b-465b-ae69-3d51f650c846", "doc_id": "fa46a645-8e6b-465b-ae69-3d51f650c846", "ref_doc_id": "fa46a645-8e6b-465b-ae69-3d51f650c846", "chroma:document": "When exporting a local model, first make sure that you \nsaved both the model's weights and tokenizer files in the same directory (`local_path`). When using CLI, pass the \n`local_path` to the `model` argument instead of the checkpoint name on \ud83e\udd17 Hub and provide the `--task` argument. \nYou can review the list of supported tasks in the [\ud83e\udd17 Optimum documentation](https://huggingface.co/docs/optimum/exporters/task_manager).\nIf `task` argument is not provided, it will default to the model architecture without any task specific head.\n\n```bash\noptimum-cli export onnx --model local_path --task question-answering distilbert_base_uncased_squad_onnx/\n```\n\nThe resulting `model.onnx` file can then be run on one of the [many\naccelerators](https://onnx.ai/supported-tools.html#deployModel) that support the ONNX\nstandard. For example, we can load and run the model with [ONNX\nRuntime](https://onnxruntime.ai/) as follows:\n\n```python\n>>> from transformers import AutoTokenizer\n>>> from optimum.onnxruntime import ORTModelForQuestionAnswering\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"distilbert_base_uncased_squad_onnx\")\n>>> model = ORTModelForQuestionAnswering.from_pretrained(\"distilbert_base_uncased_squad_onnx\")\n>>> inputs = tokenizer(\"What am I using?\", \"Using DistilBERT with ONNX Runtime!\", return_tensors=\"pt\")\n>>> outputs = model(**inputs)\n```\n\nThe process is identical for TensorFlow checkpoints on the Hub. For instance, here's how you would\nexport a pure TensorFlow checkpoint from the [Keras organization](https://huggingface.co/keras-io):\n\n```bash\noptimum-cli export onnx --model keras-io/transformers-qa distilbert_base_cased_squad_onnx/\n```\n\n### Exporting a \ud83e\udd17 Transformers model to ONNX with `optimum.onnxruntime`\n\nAlternative to CLI, you can export a \ud83e\udd17 Transformers model to ONNX programmatically like so: \n\n```python\n>>> from optimum.onnxruntime import ORTModelForSequenceClassification\n>>> from transformers import AutoTokenizer\n\n>>> model_checkpoint = \"distilbert_base_uncased_squad\"\n>>> save_directory = \"onnx/\"\n\n>>> # Load a model from transformers and export it to ONNX\n>>> ort_model = ORTModelForSequenceClassification.from_pretrained(model_checkpoint, export=True)\n>>> tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)\n\n>>> # Save the onnx model and tokenizer\n>>> ort_model.save_pretrained(save_directory)\n>>> tokenizer.save_pretrained(save_directory)\n```\n\n### Exporting a model for an unsupported architecture\n\nIf you wish to contribute by adding support for a model that cannot be currently exported, you should first check if it is\nsupported in [`optimum.exporters.onnx`](https://huggingface.co/docs/optimum/exporters/onnx/overview),\nand if it is not, [contribute to \ud83e\udd17 Optimum](https://huggingface.co/docs/optimum/exporters/onnx/usage_guides/contribute)\ndirectly.\n\n### Exporting a model with `transformers.onnx`\n\n<Tip warning={true}>\n\n`tranformers.onnx` is no longer maintained, please export models with \ud83e\udd17 Optimum as described above. This section will be removed in the future versions."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 9e28a275-03ef-4051-89b3-3677f3374e2e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9e28a275-03ef-4051-89b3-3677f3374e2e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/serialization\", \"title\": \"Export to ONNX\", \"tokens\": 2388, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fa46a645-8e6b-465b-ae69-3d51f650c846\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/serialization\", \"title\": \"Export to ONNX\", \"tokens\": 2388, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b97638da20ee8948778fc11261763dd7a2666a79a5866a41f99ad4299f694ae6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3280cd42-0b2b-4c88-aeee-0385d7c0d3f3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/serialization\", \"title\": \"Export to ONNX\", \"tokens\": 2388, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"22c6d0444383c509054462501ea7e014de72b2ee3129e4fe2a6475840b317499\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"492fb6a8-f5d0-4fb3-ba42-6a49899258e0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3c66741d635fb69a621735c0e4c3442c0c99f4715f545ef96d51b4b5c378de9f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5499, \"end_char_idx\": 8714, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fa46a645-8e6b-465b-ae69-3d51f650c846", "doc_id": "fa46a645-8e6b-465b-ae69-3d51f650c846", "ref_doc_id": "fa46a645-8e6b-465b-ae69-3d51f650c846", "chroma:document": "For instance, here's how you would\nexport a pure TensorFlow checkpoint from the [Keras organization](https://huggingface.co/keras-io):\n\n```bash\noptimum-cli export onnx --model keras-io/transformers-qa distilbert_base_cased_squad_onnx/\n```\n\n### Exporting a \ud83e\udd17 Transformers model to ONNX with `optimum.onnxruntime`\n\nAlternative to CLI, you can export a \ud83e\udd17 Transformers model to ONNX programmatically like so: \n\n```python\n>>> from optimum.onnxruntime import ORTModelForSequenceClassification\n>>> from transformers import AutoTokenizer\n\n>>> model_checkpoint = \"distilbert_base_uncased_squad\"\n>>> save_directory = \"onnx/\"\n\n>>> # Load a model from transformers and export it to ONNX\n>>> ort_model = ORTModelForSequenceClassification.from_pretrained(model_checkpoint, export=True)\n>>> tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)\n\n>>> # Save the onnx model and tokenizer\n>>> ort_model.save_pretrained(save_directory)\n>>> tokenizer.save_pretrained(save_directory)\n```\n\n### Exporting a model for an unsupported architecture\n\nIf you wish to contribute by adding support for a model that cannot be currently exported, you should first check if it is\nsupported in [`optimum.exporters.onnx`](https://huggingface.co/docs/optimum/exporters/onnx/overview),\nand if it is not, [contribute to \ud83e\udd17 Optimum](https://huggingface.co/docs/optimum/exporters/onnx/usage_guides/contribute)\ndirectly.\n\n### Exporting a model with `transformers.onnx`\n\n<Tip warning={true}>\n\n`tranformers.onnx` is no longer maintained, please export models with \ud83e\udd17 Optimum as described above. This section will be removed in the future versions.\n\n</Tip>\n\nTo export a \ud83e\udd17 Transformers model to ONNX with `tranformers.onnx`, install extra dependencies:\n\n```bash\npip install transformers[onnx]\n```\n\nUse `transformers.onnx` package as a Python module to export a checkpoint using a ready-made configuration:\n\n```bash\npython -m transformers.onnx --model=distilbert/distilbert-base-uncased onnx/\n```\n\nThis exports an ONNX graph of the checkpoint defined by the `--model` argument. Pass any checkpoint on the \ud83e\udd17 Hub or one that's stored locally.\nThe resulting `model.onnx` file can then be run on one of the many accelerators that support the ONNX standard. For example, \nload and run the model with ONNX Runtime as follows:\n\n```python\n>>> from transformers import AutoTokenizer\n>>> from onnxruntime import InferenceSession\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"distilbert/distilbert-base-uncased\")\n>>> session = InferenceSession(\"onnx/model.onnx\")\n>>> # ONNX Runtime expects NumPy arrays as input\n>>> inputs = tokenizer(\"Using DistilBERT with ONNX Runtime!\", return_tensors=\"np\")\n>>> outputs = session.run(output_names=[\"last_hidden_state\"], input_feed=dict(inputs))\n```\n\nThe required output names (like `[\"last_hidden_state\"]`) can be obtained by taking a look at the ONNX configuration of \neach model. For example, for DistilBERT we have:\n\n```python\n>>> from transformers.models.distilbert import DistilBertConfig, DistilBertOnnxConfig\n\n>>> config = DistilBertConfig()\n>>> onnx_config = DistilBertOnnxConfig(config)\n>>> print(list(onnx_config.outputs.keys()))\n[\"last_hidden_state\"]\n```\n\nThe process is identical for TensorFlow checkpoints on the Hub."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 492fb6a8-f5d0-4fb3-ba42-6a49899258e0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"492fb6a8-f5d0-4fb3-ba42-6a49899258e0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/serialization\", \"title\": \"Export to ONNX\", \"tokens\": 2388, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fa46a645-8e6b-465b-ae69-3d51f650c846\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/serialization\", \"title\": \"Export to ONNX\", \"tokens\": 2388, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b97638da20ee8948778fc11261763dd7a2666a79a5866a41f99ad4299f694ae6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9e28a275-03ef-4051-89b3-3677f3374e2e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/serialization\", \"title\": \"Export to ONNX\", \"tokens\": 2388, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"35ca19d17894ea828fc851bc477d10394ba8788ebc42afd503e8644325742c24\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7054, \"end_char_idx\": 9186, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fa46a645-8e6b-465b-ae69-3d51f650c846", "doc_id": "fa46a645-8e6b-465b-ae69-3d51f650c846", "ref_doc_id": "fa46a645-8e6b-465b-ae69-3d51f650c846", "chroma:document": "This section will be removed in the future versions.\n\n</Tip>\n\nTo export a \ud83e\udd17 Transformers model to ONNX with `tranformers.onnx`, install extra dependencies:\n\n```bash\npip install transformers[onnx]\n```\n\nUse `transformers.onnx` package as a Python module to export a checkpoint using a ready-made configuration:\n\n```bash\npython -m transformers.onnx --model=distilbert/distilbert-base-uncased onnx/\n```\n\nThis exports an ONNX graph of the checkpoint defined by the `--model` argument. Pass any checkpoint on the \ud83e\udd17 Hub or one that's stored locally.\nThe resulting `model.onnx` file can then be run on one of the many accelerators that support the ONNX standard. For example, \nload and run the model with ONNX Runtime as follows:\n\n```python\n>>> from transformers import AutoTokenizer\n>>> from onnxruntime import InferenceSession\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"distilbert/distilbert-base-uncased\")\n>>> session = InferenceSession(\"onnx/model.onnx\")\n>>> # ONNX Runtime expects NumPy arrays as input\n>>> inputs = tokenizer(\"Using DistilBERT with ONNX Runtime!\", return_tensors=\"np\")\n>>> outputs = session.run(output_names=[\"last_hidden_state\"], input_feed=dict(inputs))\n```\n\nThe required output names (like `[\"last_hidden_state\"]`) can be obtained by taking a look at the ONNX configuration of \neach model. For example, for DistilBERT we have:\n\n```python\n>>> from transformers.models.distilbert import DistilBertConfig, DistilBertOnnxConfig\n\n>>> config = DistilBertConfig()\n>>> onnx_config = DistilBertOnnxConfig(config)\n>>> print(list(onnx_config.outputs.keys()))\n[\"last_hidden_state\"]\n```\n\nThe process is identical for TensorFlow checkpoints on the Hub. For example, export a pure TensorFlow checkpoint like so:\n\n```bash\npython -m transformers.onnx --model=keras-io/transformers-qa onnx/\n```\n\nTo export a model that's stored locally, save the model's weights and tokenizer files in the same directory (e.g. `local-pt-checkpoint`), \nthen export it to ONNX by pointing the `--model` argument of the `transformers.onnx` package to the desired directory:\n\n```bash\npython -m transformers.onnx --model=local-pt-checkpoint onnx/\n```"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: dd4fed7f-e101-45e9-9cd7-a67474db7551
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"dd4fed7f-e101-45e9-9cd7-a67474db7551\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/training\", \"title\": \"Fine-tune a pretrained model\", \"tokens\": 4230, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e72ee958-f5be-4a02-ab46-92f9c580b19a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/training\", \"title\": \"Fine-tune a pretrained model\", \"tokens\": 4230, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"443128bef3012ec142fe3dbfb1dc4b34611cc0cc2d78a674a0686d1026a6d3ed\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"18c5e94d-c31f-4a9f-be73-b1eb2abdea07\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f272dc2d1b293d7d32e56b81d9ab47cf5d9ffc72a54819a4d51c2484bac391f6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3212, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "doc_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "ref_doc_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "chroma:document": "# Fine-tune a pretrained model\n\n[[open-in-colab]]\n\nThere are significant benefits to using a pretrained model. It reduces computation costs, your carbon footprint, and allows you to use state-of-the-art models without having to train one from scratch. \ud83e\udd17 Transformers provides access to thousands of pretrained models for a wide range of tasks. When you use a pretrained model, you train it on a dataset specific to your task. This is known as fine-tuning, an incredibly powerful training technique. In this tutorial, you will fine-tune a pretrained model with a deep learning framework of your choice:\n\n* Fine-tune a pretrained model with \ud83e\udd17 Transformers [`Trainer`].\n* Fine-tune a pretrained model in TensorFlow with Keras.\n* Fine-tune a pretrained model in native PyTorch.\n\n<a id='data-processing'></a>\n\n## Prepare a dataset\n\n<Youtube id=\"_BZearw7f0w\"/>\n\nBefore you can fine-tune a pretrained model, download a dataset and prepare it for training. The previous tutorial showed you how to process data for training, and now you get an opportunity to put those skills to the test!\n\nBegin by loading the [Yelp Reviews](https://huggingface.co/datasets/yelp_review_full) dataset:\n\n```py\n>>> from datasets import load_dataset\n\n>>> dataset = load_dataset(\"yelp_review_full\")\n>>> dataset[\"train\"][100]\n{'label': 0,\n 'text': 'My expectations for McDonalds are t rarely high. But for one to still fail so spectacularly...that takes something special!\\\\nThe cashier took my friends\\'s order, then promptly ignored me. I had to force myself in front of a cashier who opened his register to wait on the person BEHIND me. I waited over five minutes for a gigantic order that included precisely one kid\\'s meal. After watching two people who ordered after me be handed their food, I asked where mine was. The manager started yelling at the cashiers for \\\\\"serving off their orders\\\\\" when they didn\\'t have their food. But neither cashier was anywhere near those controls, and the manager was the one serving food to customers and clearing the boards.\\\\nThe manager was rude when giving me my order. She didn\\'t make sure that I had everything ON MY RECEIPT, and never even had the decency to apologize that I felt I was getting poor service.\\\\nI\\'ve eaten at various McDonalds restaurants for over 30 years. I\\'ve worked at more than one location. I expect bad days, bad moods, and the occasional mistake. But I have yet to have a decent experience at this store. It will remain a place I avoid unless someone in my party needs to avoid illness from low blood sugar. Perhaps I should go back to the racially biased service of Steak n Shake instead!'}\n```\n\nAs you now know, you need a tokenizer to process the text and include a padding and truncation strategy to handle any variable sequence lengths. To process your dataset in one step, use \ud83e\udd17 Datasets [`map`](https://huggingface.co/docs/datasets/process#map) method to apply a preprocessing function over the entire dataset:\n\n```py\n>>> from transformers import AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"google-bert/bert-base-cased\")\n\n\n>>> def tokenize_function(examples):\n...     return tokenizer(examples[\"text\"], padding=\"max_length\", truncation=True)"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 18c5e94d-c31f-4a9f-be73-b1eb2abdea07
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"18c5e94d-c31f-4a9f-be73-b1eb2abdea07\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/training\", \"title\": \"Fine-tune a pretrained model\", \"tokens\": 4230, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e72ee958-f5be-4a02-ab46-92f9c580b19a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/training\", \"title\": \"Fine-tune a pretrained model\", \"tokens\": 4230, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"443128bef3012ec142fe3dbfb1dc4b34611cc0cc2d78a674a0686d1026a6d3ed\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dd4fed7f-e101-45e9-9cd7-a67474db7551\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/training\", \"title\": \"Fine-tune a pretrained model\", \"tokens\": 4230, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"343fb348c3175abd370ff46fe74b134456ce4f71037fcd0424931dde4183709d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9b5de8c7-82be-435d-9fae-d6611d9aab71\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3a9b92824b68eeb1872e3b92309ad40945eb7f5b133dfb7e235d80a9e2cbb9c6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3094, \"end_char_idx\": 6395, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "doc_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "ref_doc_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "chroma:document": ">>> def tokenize_function(examples):\n...     return tokenizer(examples[\"text\"], padding=\"max_length\", truncation=True)\n\n\n>>> tokenized_datasets = dataset.map(tokenize_function, batched=True)\n```\n\nIf you like, you can create a smaller subset of the full dataset to fine-tune on to reduce the time it takes:\n\n```py\n>>> small_train_dataset = tokenized_datasets[\"train\"].shuffle(seed=42).select(range(1000))\n>>> small_eval_dataset = tokenized_datasets[\"test\"].shuffle(seed=42).select(range(1000))\n```\n\n<a id='trainer'></a>\n\n## Train\n\nAt this point, you should follow the section corresponding to the framework you want to use. You can use the links\nin the right sidebar to jump to the one you want - and if you want to hide all of the content for a given framework,\njust use the button at the top-right of that framework's block!\n\n<frameworkcontent>\n<pt>\n<Youtube id=\"nvBXf7s7vTI\"/>\n\n## Train with PyTorch Trainer\n\n\ud83e\udd17 Transformers provides a [`Trainer`] class optimized for training \ud83e\udd17 Transformers models, making it easier to start training without manually writing your own training loop. The [`Trainer`] API supports a wide range of training options and features such as logging, gradient accumulation, and mixed precision.\n\nStart by loading your model and specify the number of expected labels. From the Yelp Review [dataset card](https://huggingface.co/datasets/yelp_review_full#data-fields), you know there are five labels:\n\n```py\n>>> from transformers import AutoModelForSequenceClassification\n\n>>> model = AutoModelForSequenceClassification.from_pretrained(\"google-bert/bert-base-cased\", num_labels=5)\n```\n\n<Tip>\n\nYou will see a warning about some of the pretrained weights not being used and some weights being randomly\ninitialized. Don't worry, this is completely normal! The pretrained head of the BERT model is discarded, and replaced with a randomly initialized classification head. You will fine-tune this new model head on your sequence classification task, transferring the knowledge of the pretrained model to it.\n\n</Tip>\n\n### Training hyperparameters\n\nNext, create a [`TrainingArguments`] class which contains all the hyperparameters you can tune as well as flags for activating different training options. For this tutorial you can start with the default training [hyperparameters](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments), but feel free to experiment with these to find your optimal settings.\n\nSpecify where to save the checkpoints from your training:\n\n```py\n>>> from transformers import TrainingArguments\n\n>>> training_args = TrainingArguments(output_dir=\"test_trainer\")\n```\n\n### Evaluate\n\n[`Trainer`] does not automatically evaluate model performance during training. You'll need to pass [`Trainer`] a function to compute and report metrics. The [\ud83e\udd17 Evaluate](https://huggingface.co/docs/evaluate/index) library provides a simple [`accuracy`](https://huggingface.co/spaces/evaluate-metric/accuracy) function you can load with the [`evaluate.load`] (see this [quicktour](https://huggingface.co/docs/evaluate/a_quick_tour) for more information) function:\n\n```py\n>>> import numpy as np\n>>> import evaluate\n\n>>> metric = evaluate.load(\"accuracy\")\n```\n\nCall [`~evaluate.compute`] on `metric` to calculate the accuracy of your predictions."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 9b5de8c7-82be-435d-9fae-d6611d9aab71
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9b5de8c7-82be-435d-9fae-d6611d9aab71\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/training\", \"title\": \"Fine-tune a pretrained model\", \"tokens\": 4230, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e72ee958-f5be-4a02-ab46-92f9c580b19a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/training\", \"title\": \"Fine-tune a pretrained model\", \"tokens\": 4230, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"443128bef3012ec142fe3dbfb1dc4b34611cc0cc2d78a674a0686d1026a6d3ed\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"18c5e94d-c31f-4a9f-be73-b1eb2abdea07\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/training\", \"title\": \"Fine-tune a pretrained model\", \"tokens\": 4230, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8df8826a90b077b78425d840cf28f896ccaa9c5626911ea60b589d821d2e48e2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ad6c63b0-880d-440e-8c87-45fe92c5e2e4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ce246453266e69a69e7dc81f91a6f10d243c090ba178d8dfe5d3a7c05724d63a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4830, \"end_char_idx\": 8253, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "doc_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "ref_doc_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "chroma:document": "Don't worry, this is completely normal! The pretrained head of the BERT model is discarded, and replaced with a randomly initialized classification head. You will fine-tune this new model head on your sequence classification task, transferring the knowledge of the pretrained model to it.\n\n</Tip>\n\n### Training hyperparameters\n\nNext, create a [`TrainingArguments`] class which contains all the hyperparameters you can tune as well as flags for activating different training options. For this tutorial you can start with the default training [hyperparameters](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments), but feel free to experiment with these to find your optimal settings.\n\nSpecify where to save the checkpoints from your training:\n\n```py\n>>> from transformers import TrainingArguments\n\n>>> training_args = TrainingArguments(output_dir=\"test_trainer\")\n```\n\n### Evaluate\n\n[`Trainer`] does not automatically evaluate model performance during training. You'll need to pass [`Trainer`] a function to compute and report metrics. The [\ud83e\udd17 Evaluate](https://huggingface.co/docs/evaluate/index) library provides a simple [`accuracy`](https://huggingface.co/spaces/evaluate-metric/accuracy) function you can load with the [`evaluate.load`] (see this [quicktour](https://huggingface.co/docs/evaluate/a_quick_tour) for more information) function:\n\n```py\n>>> import numpy as np\n>>> import evaluate\n\n>>> metric = evaluate.load(\"accuracy\")\n```\n\nCall [`~evaluate.compute`] on `metric` to calculate the accuracy of your predictions. Before passing your predictions to `compute`, you need to convert the logits to predictions (remember all \ud83e\udd17 Transformers models return logits):\n\n```py\n>>> def compute_metrics(eval_pred):\n...     logits, labels = eval_pred\n...     predictions = np.argmax(logits, axis=-1)\n...     return metric.compute(predictions=predictions, references=labels)\n```\n\nIf you'd like to monitor your evaluation metrics during fine-tuning, specify the `eval_strategy` parameter in your training arguments to report the evaluation metric at the end of each epoch:\n\n```py\n>>> from transformers import TrainingArguments, Trainer\n\n>>> training_args = TrainingArguments(output_dir=\"test_trainer\", eval_strategy=\"epoch\")\n```\n\n### Trainer\n\nCreate a [`Trainer`] object with your model, training arguments, training and test datasets, and evaluation function:\n\n```py\n>>> trainer = Trainer(\n...     model=model,\n...     args=training_args,\n...     train_dataset=small_train_dataset,\n...     eval_dataset=small_eval_dataset,\n...     compute_metrics=compute_metrics,\n... )\n```\n\nThen fine-tune your model by calling [`~transformers.Trainer.train`]:\n\n```py\n>>> trainer.train()\n```\n</pt>\n<tf>\n<a id='keras'></a>\n\n<Youtube id=\"rnTGBy2ax1c\"/>\n\n## Train a TensorFlow model with Keras\n\nYou can also train \ud83e\udd17 Transformers models in TensorFlow with the Keras API!\n\n### Loading data for Keras\n\nWhen you want to train a \ud83e\udd17 Transformers model with the Keras API, you need to convert your dataset to a format that\nKeras understands. If your dataset is small, you can just convert the whole thing to NumPy arrays and pass it to Keras.\nLet's try that first before we do anything more complicated.\n\nFirst, load a dataset. We'll use the CoLA dataset from the [GLUE benchmark](https://huggingface.co/datasets/glue),\nsince it's a simple binary text classification task, and just take the training split for now."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: ad6c63b0-880d-440e-8c87-45fe92c5e2e4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ad6c63b0-880d-440e-8c87-45fe92c5e2e4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/training\", \"title\": \"Fine-tune a pretrained model\", \"tokens\": 4230, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e72ee958-f5be-4a02-ab46-92f9c580b19a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/training\", \"title\": \"Fine-tune a pretrained model\", \"tokens\": 4230, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"443128bef3012ec142fe3dbfb1dc4b34611cc0cc2d78a674a0686d1026a6d3ed\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9b5de8c7-82be-435d-9fae-d6611d9aab71\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/training\", \"title\": \"Fine-tune a pretrained model\", \"tokens\": 4230, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bacd0f5cd722024f3e52d24f3782aa9b4ad2512529b0a6ede0b5a6aa5c44d0d7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"380bd34c-8c52-49ca-b183-f43295c96f4c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ce7bb17bf1d7a1dd7247002908fb8643603a54cfcc22f0aff87f5154e504a180\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7718, \"end_char_idx\": 11047, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "doc_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "ref_doc_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "chroma:document": "### Loading data for Keras\n\nWhen you want to train a \ud83e\udd17 Transformers model with the Keras API, you need to convert your dataset to a format that\nKeras understands. If your dataset is small, you can just convert the whole thing to NumPy arrays and pass it to Keras.\nLet's try that first before we do anything more complicated.\n\nFirst, load a dataset. We'll use the CoLA dataset from the [GLUE benchmark](https://huggingface.co/datasets/glue),\nsince it's a simple binary text classification task, and just take the training split for now.\n\n```py\nfrom datasets import load_dataset\n\ndataset = load_dataset(\"glue\", \"cola\")\ndataset = dataset[\"train\"]  # Just take the training split for now\n```\n\nNext, load a tokenizer and tokenize the data as NumPy arrays. Note that the labels are already a list of 0 and 1s,\nso we can just convert that directly to a NumPy array without tokenization!\n\n```py\nfrom transformers import AutoTokenizer\nimport numpy as np\n\ntokenizer = AutoTokenizer.from_pretrained(\"google-bert/bert-base-cased\")\ntokenized_data = tokenizer(dataset[\"sentence\"], return_tensors=\"np\", padding=True)\n# Tokenizer returns a BatchEncoding, but we convert that to a dict for Keras\ntokenized_data = dict(tokenized_data)\n\nlabels = np.array(dataset[\"label\"])  # Label is already an array of 0 and 1\n```\n\nFinally, load, [`compile`](https://keras.io/api/models/model_training_apis/#compile-method), and [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) the model. Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:\n\n```py\nfrom transformers import TFAutoModelForSequenceClassification\nfrom tensorflow.keras.optimizers import Adam\n\n# Load and compile our model\nmodel = TFAutoModelForSequenceClassification.from_pretrained(\"google-bert/bert-base-cased\")\n# Lower learning rates are often better for fine-tuning transformers\nmodel.compile(optimizer=Adam(3e-5))  # No loss argument!\n\nmodel.fit(tokenized_data, labels)\n```\n\n<Tip>\n\nYou don't have to pass a loss argument to your models when you `compile()` them! Hugging Face models automatically\nchoose a loss that is appropriate for their task and model architecture if this argument is left blank. You can always\noverride this by specifying a loss yourself if you want to!\n\n</Tip>\n\nThis approach works great for smaller datasets, but for larger datasets, you might find it starts to become a problem. Why?\nBecause the tokenized array and labels would have to be fully loaded into memory, and because NumPy doesn\u2019t handle\n\u201cjagged\u201d arrays, so every tokenized sample would have to be padded to the length of the longest sample in the whole\ndataset. That\u2019s going to make your array even bigger, and all those padding tokens will slow down training too!\n\n### Loading data as a tf.data.Dataset\n\nIf you want to avoid slowing down training, you can load your data as a `tf.data.Dataset` instead. Although you can write your own\n`tf.data` pipeline if you want, we have two convenience methods for doing this:\n\n- [`~TFPreTrainedModel.prepare_tf_dataset`]: This is the method we recommend in most cases. Because it is a method\non your model, it can inspect the model to automatically figure out which columns are usable as model inputs, and\ndiscard the others to make a simpler, more performant dataset."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 380bd34c-8c52-49ca-b183-f43295c96f4c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"380bd34c-8c52-49ca-b183-f43295c96f4c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/training\", \"title\": \"Fine-tune a pretrained model\", \"tokens\": 4230, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e72ee958-f5be-4a02-ab46-92f9c580b19a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/training\", \"title\": \"Fine-tune a pretrained model\", \"tokens\": 4230, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"443128bef3012ec142fe3dbfb1dc4b34611cc0cc2d78a674a0686d1026a6d3ed\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ad6c63b0-880d-440e-8c87-45fe92c5e2e4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/training\", \"title\": \"Fine-tune a pretrained model\", \"tokens\": 4230, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"82191562886a8bd93b678fa3199ee9b7f37db8606bc0a00c73a212c55d55ab4d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"da90ba86-c748-449a-9586-b81d27ce44ba\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0638e4a95dff63d8b1aebe46086c239745a2b1e232e2595f5237a7776588b272\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9688, \"end_char_idx\": 12937, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "doc_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "ref_doc_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "chroma:document": "model.fit(tokenized_data, labels)\n```\n\n<Tip>\n\nYou don't have to pass a loss argument to your models when you `compile()` them! Hugging Face models automatically\nchoose a loss that is appropriate for their task and model architecture if this argument is left blank. You can always\noverride this by specifying a loss yourself if you want to!\n\n</Tip>\n\nThis approach works great for smaller datasets, but for larger datasets, you might find it starts to become a problem. Why?\nBecause the tokenized array and labels would have to be fully loaded into memory, and because NumPy doesn\u2019t handle\n\u201cjagged\u201d arrays, so every tokenized sample would have to be padded to the length of the longest sample in the whole\ndataset. That\u2019s going to make your array even bigger, and all those padding tokens will slow down training too!\n\n### Loading data as a tf.data.Dataset\n\nIf you want to avoid slowing down training, you can load your data as a `tf.data.Dataset` instead. Although you can write your own\n`tf.data` pipeline if you want, we have two convenience methods for doing this:\n\n- [`~TFPreTrainedModel.prepare_tf_dataset`]: This is the method we recommend in most cases. Because it is a method\non your model, it can inspect the model to automatically figure out which columns are usable as model inputs, and\ndiscard the others to make a simpler, more performant dataset.\n- [`~datasets.Dataset.to_tf_dataset`]: This method is more low-level, and is useful when you want to exactly control how\nyour dataset is created, by specifying exactly which `columns` and `label_cols` to include.\n\nBefore you can use [`~TFPreTrainedModel.prepare_tf_dataset`], you will need to add the tokenizer outputs to your dataset as columns, as shown in\nthe following code sample:\n\n```py\ndef tokenize_dataset(data):\n    # Keys of the returned dictionary will be added to the dataset as columns\n    return tokenizer(data[\"text\"])\n\n\ndataset = dataset.map(tokenize_dataset)\n```\n\nRemember that Hugging Face datasets are stored on disk by default, so this will not inflate your memory usage! Once the\ncolumns have been added, you can stream batches from the dataset and add padding to each batch, which greatly\nreduces the number of padding tokens compared to padding the entire dataset.\n\n\n```py\n>>> tf_dataset = model.prepare_tf_dataset(dataset[\"train\"], batch_size=16, shuffle=True, tokenizer=tokenizer)\n```\n\nNote that in the code sample above, you need to pass the tokenizer to `prepare_tf_dataset` so it can correctly pad batches as they're loaded.\nIf all the samples in your dataset are the same length and no padding is necessary, you can skip this argument.\nIf you need to do something more complex than just padding samples (e.g. corrupting tokens for masked language\nmodelling), you can use the `collate_fn` argument instead to pass a function that will be called to transform the\nlist of samples into a batch and apply any preprocessing you want. See our\n[examples](https://github.com/huggingface/transformers/tree/main/examples) or\n[notebooks](https://huggingface.co/docs/transformers/notebooks) to see this approach in action.\n\nOnce you've created a `tf.data.Dataset`, you can compile and fit the model as before:\n\n```py\nmodel.compile(optimizer=Adam(3e-5))  # No loss argument!"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: da90ba86-c748-449a-9586-b81d27ce44ba
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"da90ba86-c748-449a-9586-b81d27ce44ba\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/training\", \"title\": \"Fine-tune a pretrained model\", \"tokens\": 4230, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e72ee958-f5be-4a02-ab46-92f9c580b19a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/training\", \"title\": \"Fine-tune a pretrained model\", \"tokens\": 4230, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"443128bef3012ec142fe3dbfb1dc4b34611cc0cc2d78a674a0686d1026a6d3ed\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"380bd34c-8c52-49ca-b183-f43295c96f4c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/training\", \"title\": \"Fine-tune a pretrained model\", \"tokens\": 4230, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"66429603b6d2af483eaf01d8d6d312550ea4c7bb9ba691ba7463fa2f5515556e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a836e663-0337-4dd0-8ce9-0dc4be51c475\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"074d18a40a427a36ea36bcd86caa3e6a20f89500808ef958e0ddae589e9914ca\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11262, \"end_char_idx\": 13951, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "doc_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "ref_doc_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "chroma:document": "Before you can use [`~TFPreTrainedModel.prepare_tf_dataset`], you will need to add the tokenizer outputs to your dataset as columns, as shown in\nthe following code sample:\n\n```py\ndef tokenize_dataset(data):\n    # Keys of the returned dictionary will be added to the dataset as columns\n    return tokenizer(data[\"text\"])\n\n\ndataset = dataset.map(tokenize_dataset)\n```\n\nRemember that Hugging Face datasets are stored on disk by default, so this will not inflate your memory usage! Once the\ncolumns have been added, you can stream batches from the dataset and add padding to each batch, which greatly\nreduces the number of padding tokens compared to padding the entire dataset.\n\n\n```py\n>>> tf_dataset = model.prepare_tf_dataset(dataset[\"train\"], batch_size=16, shuffle=True, tokenizer=tokenizer)\n```\n\nNote that in the code sample above, you need to pass the tokenizer to `prepare_tf_dataset` so it can correctly pad batches as they're loaded.\nIf all the samples in your dataset are the same length and no padding is necessary, you can skip this argument.\nIf you need to do something more complex than just padding samples (e.g. corrupting tokens for masked language\nmodelling), you can use the `collate_fn` argument instead to pass a function that will be called to transform the\nlist of samples into a batch and apply any preprocessing you want. See our\n[examples](https://github.com/huggingface/transformers/tree/main/examples) or\n[notebooks](https://huggingface.co/docs/transformers/notebooks) to see this approach in action.\n\nOnce you've created a `tf.data.Dataset`, you can compile and fit the model as before:\n\n```py\nmodel.compile(optimizer=Adam(3e-5))  # No loss argument!\n\nmodel.fit(tf_dataset)\n```\n\n</tf>\n</frameworkcontent>\n\n<a id='pytorch_native'></a>\n\n## Train in native PyTorch\n\n<frameworkcontent>\n<pt>\n<Youtube id=\"Dh9CL8fyG80\"/>\n\n[`Trainer`] takes care of the training loop and allows you to fine-tune a model in a single line of code. For users who prefer to write their own training loop, you can also fine-tune a \ud83e\udd17 Transformers model in native PyTorch.\n\nAt this point, you may need to restart your notebook or execute the following code to free some memory:\n\n```py\ndel model\ndel trainer\ntorch.cuda.empty_cache()\n```\n\nNext, manually postprocess `tokenized_dataset` to prepare it for training.\n\n1. Remove the `text` column because the model does not accept raw text as an input:\n\n    ```py\n    >>> tokenized_datasets = tokenized_datasets.remove_columns([\"text\"])\n    ```\n\n2. Rename the `label` column to `labels` because the model expects the argument to be named `labels`:\n\n    ```py\n    >>> tokenized_datasets = tokenized_datasets.rename_column(\"label\", \"labels\")\n    ```\n\n3."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: a836e663-0337-4dd0-8ce9-0dc4be51c475
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a836e663-0337-4dd0-8ce9-0dc4be51c475\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/training\", \"title\": \"Fine-tune a pretrained model\", \"tokens\": 4230, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e72ee958-f5be-4a02-ab46-92f9c580b19a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/training\", \"title\": \"Fine-tune a pretrained model\", \"tokens\": 4230, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"443128bef3012ec142fe3dbfb1dc4b34611cc0cc2d78a674a0686d1026a6d3ed\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"da90ba86-c748-449a-9586-b81d27ce44ba\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/training\", \"title\": \"Fine-tune a pretrained model\", \"tokens\": 4230, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"93c05aeeb2f2848f9c4424b821ba04ea0e300d146be60aeedebacfd32dd425ae\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2bbd83a8-0636-46ab-a836-734bc9b88abe\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a30a0c83f7eb103efd390953bcefd85dbaae196e50ea672e64a857c9d4faf310\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12386, \"end_char_idx\": 15051, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "doc_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "ref_doc_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "chroma:document": "corrupting tokens for masked language\nmodelling), you can use the `collate_fn` argument instead to pass a function that will be called to transform the\nlist of samples into a batch and apply any preprocessing you want. See our\n[examples](https://github.com/huggingface/transformers/tree/main/examples) or\n[notebooks](https://huggingface.co/docs/transformers/notebooks) to see this approach in action.\n\nOnce you've created a `tf.data.Dataset`, you can compile and fit the model as before:\n\n```py\nmodel.compile(optimizer=Adam(3e-5))  # No loss argument!\n\nmodel.fit(tf_dataset)\n```\n\n</tf>\n</frameworkcontent>\n\n<a id='pytorch_native'></a>\n\n## Train in native PyTorch\n\n<frameworkcontent>\n<pt>\n<Youtube id=\"Dh9CL8fyG80\"/>\n\n[`Trainer`] takes care of the training loop and allows you to fine-tune a model in a single line of code. For users who prefer to write their own training loop, you can also fine-tune a \ud83e\udd17 Transformers model in native PyTorch.\n\nAt this point, you may need to restart your notebook or execute the following code to free some memory:\n\n```py\ndel model\ndel trainer\ntorch.cuda.empty_cache()\n```\n\nNext, manually postprocess `tokenized_dataset` to prepare it for training.\n\n1. Remove the `text` column because the model does not accept raw text as an input:\n\n    ```py\n    >>> tokenized_datasets = tokenized_datasets.remove_columns([\"text\"])\n    ```\n\n2. Rename the `label` column to `labels` because the model expects the argument to be named `labels`:\n\n    ```py\n    >>> tokenized_datasets = tokenized_datasets.rename_column(\"label\", \"labels\")\n    ```\n\n3. Set the format of the dataset to return PyTorch tensors instead of lists:\n\n    ```py\n    >>> tokenized_datasets.set_format(\"torch\")\n    ```\n\nThen create a smaller subset of the dataset as previously shown to speed up the fine-tuning:\n\n```py\n>>> small_train_dataset = tokenized_datasets[\"train\"].shuffle(seed=42).select(range(1000))\n>>> small_eval_dataset = tokenized_datasets[\"test\"].shuffle(seed=42).select(range(1000))\n```\n\n### DataLoader\n\nCreate a `DataLoader` for your training and test datasets so you can iterate over batches of data:\n\n```py\n>>> from torch.utils.data import DataLoader\n\n>>> train_dataloader = DataLoader(small_train_dataset, shuffle=True, batch_size=8)\n>>> eval_dataloader = DataLoader(small_eval_dataset, batch_size=8)\n```\n\nLoad your model with the number of expected labels:\n\n```py\n>>> from transformers import AutoModelForSequenceClassification\n\n>>> model = AutoModelForSequenceClassification.from_pretrained(\"google-bert/bert-base-cased\", num_labels=5)\n```\n\n### Optimizer and learning rate scheduler\n\nCreate an optimizer and learning rate scheduler to fine-tune the model."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 2bbd83a8-0636-46ab-a836-734bc9b88abe
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2bbd83a8-0636-46ab-a836-734bc9b88abe\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/training\", \"title\": \"Fine-tune a pretrained model\", \"tokens\": 4230, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e72ee958-f5be-4a02-ab46-92f9c580b19a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/training\", \"title\": \"Fine-tune a pretrained model\", \"tokens\": 4230, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"443128bef3012ec142fe3dbfb1dc4b34611cc0cc2d78a674a0686d1026a6d3ed\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a836e663-0337-4dd0-8ce9-0dc4be51c475\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/training\", \"title\": \"Fine-tune a pretrained model\", \"tokens\": 4230, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"787226d725c928e5c30425048c9138e05900feb0b59bc29768ab9617f5b128a8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cd8e4512-62c1-468a-81cc-b56b098a35d2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"51a8f89f30303e85c9c694c0bfbc834a46b40609f5f474facc7afb8895c407df\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13330, \"end_char_idx\": 16132, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "doc_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "ref_doc_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "chroma:document": "At this point, you may need to restart your notebook or execute the following code to free some memory:\n\n```py\ndel model\ndel trainer\ntorch.cuda.empty_cache()\n```\n\nNext, manually postprocess `tokenized_dataset` to prepare it for training.\n\n1. Remove the `text` column because the model does not accept raw text as an input:\n\n    ```py\n    >>> tokenized_datasets = tokenized_datasets.remove_columns([\"text\"])\n    ```\n\n2. Rename the `label` column to `labels` because the model expects the argument to be named `labels`:\n\n    ```py\n    >>> tokenized_datasets = tokenized_datasets.rename_column(\"label\", \"labels\")\n    ```\n\n3. Set the format of the dataset to return PyTorch tensors instead of lists:\n\n    ```py\n    >>> tokenized_datasets.set_format(\"torch\")\n    ```\n\nThen create a smaller subset of the dataset as previously shown to speed up the fine-tuning:\n\n```py\n>>> small_train_dataset = tokenized_datasets[\"train\"].shuffle(seed=42).select(range(1000))\n>>> small_eval_dataset = tokenized_datasets[\"test\"].shuffle(seed=42).select(range(1000))\n```\n\n### DataLoader\n\nCreate a `DataLoader` for your training and test datasets so you can iterate over batches of data:\n\n```py\n>>> from torch.utils.data import DataLoader\n\n>>> train_dataloader = DataLoader(small_train_dataset, shuffle=True, batch_size=8)\n>>> eval_dataloader = DataLoader(small_eval_dataset, batch_size=8)\n```\n\nLoad your model with the number of expected labels:\n\n```py\n>>> from transformers import AutoModelForSequenceClassification\n\n>>> model = AutoModelForSequenceClassification.from_pretrained(\"google-bert/bert-base-cased\", num_labels=5)\n```\n\n### Optimizer and learning rate scheduler\n\nCreate an optimizer and learning rate scheduler to fine-tune the model. Let's use the [`AdamW`](https://pytorch.org/docs/stable/generated/torch.optim.AdamW.html) optimizer from PyTorch:\n\n```py\n>>> from torch.optim import AdamW\n\n>>> optimizer = AdamW(model.parameters(), lr=5e-5)\n```\n\nCreate the default learning rate scheduler from [`Trainer`]:\n\n```py\n>>> from transformers import get_scheduler\n\n>>> num_epochs = 3\n>>> num_training_steps = num_epochs * len(train_dataloader)\n>>> lr_scheduler = get_scheduler(\n...     name=\"linear\", optimizer=optimizer, num_warmup_steps=0, num_training_steps=num_training_steps\n... )\n```\n\nLastly, specify `device` to use a GPU if you have access to one. Otherwise, training on a CPU may take several hours instead of a couple of minutes.\n\n```py\n>>> import torch\n\n>>> device = torch.device(\"cuda\") if torch.cuda.is_available() else torch.device(\"cpu\")\n>>> model.to(device)\n```\n\n<Tip>\n\nGet free access to a cloud GPU if you don't have one with a hosted notebook like [Colaboratory](https://colab.research.google.com/) or [SageMaker StudioLab](https://studiolab.sagemaker.aws/).\n\n</Tip>\n\nGreat, now you are ready to train!"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: cd8e4512-62c1-468a-81cc-b56b098a35d2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"cd8e4512-62c1-468a-81cc-b56b098a35d2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/training\", \"title\": \"Fine-tune a pretrained model\", \"tokens\": 4230, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e72ee958-f5be-4a02-ab46-92f9c580b19a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/training\", \"title\": \"Fine-tune a pretrained model\", \"tokens\": 4230, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"443128bef3012ec142fe3dbfb1dc4b34611cc0cc2d78a674a0686d1026a6d3ed\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2bbd83a8-0636-46ab-a836-734bc9b88abe\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/training\", \"title\": \"Fine-tune a pretrained model\", \"tokens\": 4230, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d4c416ec53455728ed0aba26d918ccc4d7f1f4d14e1a59866761037f03cd1649\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15052, \"end_char_idx\": 17983, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "doc_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "ref_doc_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "chroma:document": "Let's use the [`AdamW`](https://pytorch.org/docs/stable/generated/torch.optim.AdamW.html) optimizer from PyTorch:\n\n```py\n>>> from torch.optim import AdamW\n\n>>> optimizer = AdamW(model.parameters(), lr=5e-5)\n```\n\nCreate the default learning rate scheduler from [`Trainer`]:\n\n```py\n>>> from transformers import get_scheduler\n\n>>> num_epochs = 3\n>>> num_training_steps = num_epochs * len(train_dataloader)\n>>> lr_scheduler = get_scheduler(\n...     name=\"linear\", optimizer=optimizer, num_warmup_steps=0, num_training_steps=num_training_steps\n... )\n```\n\nLastly, specify `device` to use a GPU if you have access to one. Otherwise, training on a CPU may take several hours instead of a couple of minutes.\n\n```py\n>>> import torch\n\n>>> device = torch.device(\"cuda\") if torch.cuda.is_available() else torch.device(\"cpu\")\n>>> model.to(device)\n```\n\n<Tip>\n\nGet free access to a cloud GPU if you don't have one with a hosted notebook like [Colaboratory](https://colab.research.google.com/) or [SageMaker StudioLab](https://studiolab.sagemaker.aws/).\n\n</Tip>\n\nGreat, now you are ready to train! \ud83e\udd73 \n\n### Training loop\n\nTo keep track of your training progress, use the [tqdm](https://tqdm.github.io/) library to add a progress bar over the number of training steps:\n\n```py\n>>> from tqdm.auto import tqdm\n\n>>> progress_bar = tqdm(range(num_training_steps))\n\n>>> model.train()\n>>> for epoch in range(num_epochs):\n...     for batch in train_dataloader:\n...         batch = {k: v.to(device) for k, v in batch.items()}\n...         outputs = model(**batch)\n...         loss = outputs.loss\n...         loss.backward()\n\n...         optimizer.step()\n...         lr_scheduler.step()\n...         optimizer.zero_grad()\n...         progress_bar.update(1)\n```\n\n### Evaluate\n\nJust like how you added an evaluation function to [`Trainer`], you need to do the same when you write your own training loop. But instead of calculating and reporting the metric at the end of each epoch, this time you'll accumulate all the batches with [`~evaluate.add_batch`] and calculate the metric at the very end.\n\n```py\n>>> import evaluate\n\n>>> metric = evaluate.load(\"accuracy\")\n>>> model.eval()\n>>> for batch in eval_dataloader:\n...     batch = {k: v.to(device) for k, v in batch.items()}\n...     with torch.no_grad():\n...         outputs = model(**batch)\n\n...     logits = outputs.logits\n...     predictions = torch.argmax(logits, dim=-1)\n...     metric.add_batch(predictions=predictions, references=batch[\"labels\"])\n\n>>> metric.compute()\n```\n</pt>\n</frameworkcontent>\n\n<a id='additional-resources'></a>\n\n## Additional resources\n\nFor more fine-tuning examples, refer to:\n\n- [\ud83e\udd17 Transformers Examples](https://github.com/huggingface/transformers/tree/main/examples) includes scripts\n  to train common NLP tasks in PyTorch and TensorFlow.\n\n- [\ud83e\udd17 Transformers Notebooks](notebooks) contains various notebooks on how to fine-tune a model for specific tasks in PyTorch and TensorFlow."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 6a01e851-f052-4bbc-942d-6a3898066694
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6a01e851-f052-4bbc-942d-6a3898066694\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/conversations\", \"title\": \"Chatting with Transformers\", \"tokens\": 4154, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2678abda-b5a2-420f-822e-053812eb0ba3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/conversations\", \"title\": \"Chatting with Transformers\", \"tokens\": 4154, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d0faf8596bdd083d896283e3e14e6b207798309c03ac95db0d357f4c7d7780d7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3defa731-9e06-4e45-9e8c-daf690ed62e7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2b2678ba65ce5d66be3a488416c657f056d2f62e504f0b0b1db6076d8ade811a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3223, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "doc_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "ref_doc_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "chroma:document": "# Chatting with Transformers\n\nIf you're reading this article, you're almost certainly aware of **chat models**. Chat models are conversational\nAIs that you can send and receive messages with. The most famous of these is the proprietary ChatGPT, but there are\nnow many open-source chat models which match or even substantially exceed its performance. These models are free to\ndownload and run on a local machine. Although the largest and most capable models require high-powered hardware\nand lots of memory to run, there are smaller models that will run perfectly well on a single consumer GPU, or even\nan ordinary desktop or notebook CPU. \n\nThis guide will help you get started with chat models. We'll start with a brief quickstart guide that uses a convenient,\nhigh-level \"pipeline\". This is all you need if you just want to start running a chat model \nimmediately. After the quickstart, we'll move on to more detailed information about\nwhat exactly chat models are, how to choose an appropriate one, and a low-level breakdown of each of the\nsteps involved in talking to a chat model. We'll also give some tips on optimizing the performance and memory usage\nof your chat models.\n\n\n## Quickstart\n\nIf you have no time for details, here's the brief summary: Chat models continue chats. This means that you pass them\na conversation history, which can be as short as a single user message, and the model will continue the conversation\nby adding its response. Let's see this in action. First, let's build a chat:\n\n```python\nchat = [\n    {\"role\": \"system\", \"content\": \"You are a sassy, wise-cracking robot as imagined by Hollywood circa 1986.\"},\n    {\"role\": \"user\", \"content\": \"Hey, can you tell me any fun things to do in New York?\"}\n]\n```\n\nNotice that in addition to the user's message, we added a **system** message at the start of the conversation. Not all\nchat models support system messages, but when they do, they represent high-level directives about how the model\nshould behave in the conversation. You can use this to guide the model - whether you want short or long responses,\nlighthearted or serious ones, and so on. If you want the model to do useful work instead of\npracticing its improv routine, you can either omit the system message or try a terse one such as \"You are a helpful and intelligent\nAI assistant who responds to user queries.\"\n\nOnce you have a chat, the quickest way to continue it is using the [`TextGenerationPipeline`]. \nLet's see this in action with `LLaMA-3`. Note that `LLaMA-3` is a gated model, which means you will need to \n[apply for access](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) and log in with your Hugging Face \naccount to use it. We'll also use `device_map=\"auto\"`, which will load the model on GPU if there's enough memory\nfor it, and set the dtype to `torch.bfloat16` to save memory:\n\n```python\nimport torch\nfrom transformers import pipeline\n\npipe = pipeline(\"text-generation\", \"meta-llama/Meta-Llama-3-8B-Instruct\", torch_dtype=torch.bfloat16, device_map=\"auto\")\nresponse = pipe(chat, max_new_tokens=512)\nprint(response[0]['generated_text'][-1]['content'])\n```\n\nAnd you'll get:\n\n```text\n(sigh) Oh boy, you're asking me for advice? You're gonna need a map, pal!"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 3defa731-9e06-4e45-9e8c-daf690ed62e7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3defa731-9e06-4e45-9e8c-daf690ed62e7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/conversations\", \"title\": \"Chatting with Transformers\", \"tokens\": 4154, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2678abda-b5a2-420f-822e-053812eb0ba3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/conversations\", \"title\": \"Chatting with Transformers\", \"tokens\": 4154, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d0faf8596bdd083d896283e3e14e6b207798309c03ac95db0d357f4c7d7780d7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6a01e851-f052-4bbc-942d-6a3898066694\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/conversations\", \"title\": \"Chatting with Transformers\", \"tokens\": 4154, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ab72716695b9356e6a5920eecf87b4f6c29dfd7254b22dbbd215352e53812e99\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"23c985c2-2b62-4233-845d-b85fc0be3731\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fac1f6ed0a1ae8a42daa28cfe5ef68d584be2b4c4e7d53dd12227cb8bfb91071\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1730, \"end_char_idx\": 4652, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "doc_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "ref_doc_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "chroma:document": "]\n```\n\nNotice that in addition to the user's message, we added a **system** message at the start of the conversation. Not all\nchat models support system messages, but when they do, they represent high-level directives about how the model\nshould behave in the conversation. You can use this to guide the model - whether you want short or long responses,\nlighthearted or serious ones, and so on. If you want the model to do useful work instead of\npracticing its improv routine, you can either omit the system message or try a terse one such as \"You are a helpful and intelligent\nAI assistant who responds to user queries.\"\n\nOnce you have a chat, the quickest way to continue it is using the [`TextGenerationPipeline`]. \nLet's see this in action with `LLaMA-3`. Note that `LLaMA-3` is a gated model, which means you will need to \n[apply for access](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) and log in with your Hugging Face \naccount to use it. We'll also use `device_map=\"auto\"`, which will load the model on GPU if there's enough memory\nfor it, and set the dtype to `torch.bfloat16` to save memory:\n\n```python\nimport torch\nfrom transformers import pipeline\n\npipe = pipeline(\"text-generation\", \"meta-llama/Meta-Llama-3-8B-Instruct\", torch_dtype=torch.bfloat16, device_map=\"auto\")\nresponse = pipe(chat, max_new_tokens=512)\nprint(response[0]['generated_text'][-1]['content'])\n```\n\nAnd you'll get:\n\n```text\n(sigh) Oh boy, you're asking me for advice? You're gonna need a map, pal! Alright, \nalright, I'll give you the lowdown. But don't say I didn't warn you, I'm a robot, not a tour guide!\n\nSo, you wanna know what's fun to do in the Big Apple? Well, let me tell you, there's a million \nthings to do, but I'll give you the highlights. First off, you gotta see the sights: the Statue of \nLiberty, Central Park, Times Square... you know, the usual tourist traps. But if you're lookin' for \nsomething a little more... unusual, I'd recommend checkin' out the Museum of Modern Art. It's got \nsome wild stuff, like that Warhol guy's soup cans and all that jazz.\n\nAnd if you're feelin' adventurous, take a walk across the Brooklyn Bridge. Just watch out for \nthose pesky pigeons, they're like little feathered thieves! (laughs) Get it? Thieves? Ah, never mind.\n\nNow, if you're lookin' for some serious fun, hit up the comedy clubs in Greenwich Village. You might \neven catch a glimpse of some up-and-coming comedians... or a bunch of wannabes tryin' to make it big. (winks)\n\nAnd finally, if you're feelin' like a real New Yorker, grab a slice of pizza from one of the many amazing\npizzerias around the city. Just don't try to order a \"robot-sized\" slice, trust me, it won't end well. (laughs)\n\nSo, there you have it, pal! That's my expert advice on what to do in New York. Now, if you'll\nexcuse me, I've got some oil changes to attend to. (winks)\n```\n\nYou can continue the chat by appending your own response to it."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 23c985c2-2b62-4233-845d-b85fc0be3731
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"23c985c2-2b62-4233-845d-b85fc0be3731\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/conversations\", \"title\": \"Chatting with Transformers\", \"tokens\": 4154, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2678abda-b5a2-420f-822e-053812eb0ba3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/conversations\", \"title\": \"Chatting with Transformers\", \"tokens\": 4154, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d0faf8596bdd083d896283e3e14e6b207798309c03ac95db0d357f4c7d7780d7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3defa731-9e06-4e45-9e8c-daf690ed62e7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/conversations\", \"title\": \"Chatting with Transformers\", \"tokens\": 4154, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d2d11226252519af860279cea21b2c91c30be3dc7e5893c102be75d717dcdd12\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bedbc242-39c3-4cbb-b145-8c819a6d3489\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"eb91ec26d6cf9e8053b913b590bf7b79b97d23b471ceb71b1ce081f80e2923a2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3224, \"end_char_idx\": 5988, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "doc_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "ref_doc_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "chroma:document": "Alright, \nalright, I'll give you the lowdown. But don't say I didn't warn you, I'm a robot, not a tour guide!\n\nSo, you wanna know what's fun to do in the Big Apple? Well, let me tell you, there's a million \nthings to do, but I'll give you the highlights. First off, you gotta see the sights: the Statue of \nLiberty, Central Park, Times Square... you know, the usual tourist traps. But if you're lookin' for \nsomething a little more... unusual, I'd recommend checkin' out the Museum of Modern Art. It's got \nsome wild stuff, like that Warhol guy's soup cans and all that jazz.\n\nAnd if you're feelin' adventurous, take a walk across the Brooklyn Bridge. Just watch out for \nthose pesky pigeons, they're like little feathered thieves! (laughs) Get it? Thieves? Ah, never mind.\n\nNow, if you're lookin' for some serious fun, hit up the comedy clubs in Greenwich Village. You might \neven catch a glimpse of some up-and-coming comedians... or a bunch of wannabes tryin' to make it big. (winks)\n\nAnd finally, if you're feelin' like a real New Yorker, grab a slice of pizza from one of the many amazing\npizzerias around the city. Just don't try to order a \"robot-sized\" slice, trust me, it won't end well. (laughs)\n\nSo, there you have it, pal! That's my expert advice on what to do in New York. Now, if you'll\nexcuse me, I've got some oil changes to attend to. (winks)\n```\n\nYou can continue the chat by appending your own response to it. The\n`response` object returned by the pipeline actually contains the entire chat so far, so we can simply append\na message and pass it back:\n\n```python\nchat = response[0]['generated_text']\nchat.append(\n    {\"role\": \"user\", \"content\": \"Wait, what's so wild about soup cans?\"}\n)\nresponse = pipe(chat, max_new_tokens=512)\nprint(response[0]['generated_text'][-1]['content'])\n```\n\nAnd you'll get:\n\n```text\n(laughs) Oh, you're killin' me, pal! You don't get it, do you? Warhol's soup cans are like, art, man! \nIt's like, he took something totally mundane, like a can of soup, and turned it into a masterpiece. It's \nlike, \"Hey, look at me, I'm a can of soup, but I'm also a work of art!\" \n(sarcastically) Oh, yeah, real original, Andy.\n\nBut, you know, back in the '60s, it was like, a big deal. People were all about challenging the\nstatus quo, and Warhol was like, the king of that. He took the ordinary and made it extraordinary.\nAnd, let me tell you, it was like, a real game-changer. I mean, who would've thought that a can of soup could be art? (laughs)\n\nBut, hey, you're not alone, pal. I mean, I'm a robot, and even I don't get it. (winks)\nBut, hey, that's what makes art, art, right? (laughs)\n```\n\nThe remainder of this tutorial will cover specific topics such\nas performance and memory, or how to select a chat model for your needs."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: bedbc242-39c3-4cbb-b145-8c819a6d3489
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"bedbc242-39c3-4cbb-b145-8c819a6d3489\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/conversations\", \"title\": \"Chatting with Transformers\", \"tokens\": 4154, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2678abda-b5a2-420f-822e-053812eb0ba3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/conversations\", \"title\": \"Chatting with Transformers\", \"tokens\": 4154, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d0faf8596bdd083d896283e3e14e6b207798309c03ac95db0d357f4c7d7780d7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"23c985c2-2b62-4233-845d-b85fc0be3731\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/conversations\", \"title\": \"Chatting with Transformers\", \"tokens\": 4154, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"555b708d8d45ec100d4a59e38dde249211fd125538d3b32f8cfd513960f0f594\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c46ee4ca-f540-41b2-8ca9-e8aa066a229c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"96a450fe35345f2d532528ac1050d61e835335dcfa1c569998f68b707010d10f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4576, \"end_char_idx\": 7448, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "doc_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "ref_doc_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "chroma:document": "(winks)\n```\n\nYou can continue the chat by appending your own response to it. The\n`response` object returned by the pipeline actually contains the entire chat so far, so we can simply append\na message and pass it back:\n\n```python\nchat = response[0]['generated_text']\nchat.append(\n    {\"role\": \"user\", \"content\": \"Wait, what's so wild about soup cans?\"}\n)\nresponse = pipe(chat, max_new_tokens=512)\nprint(response[0]['generated_text'][-1]['content'])\n```\n\nAnd you'll get:\n\n```text\n(laughs) Oh, you're killin' me, pal! You don't get it, do you? Warhol's soup cans are like, art, man! \nIt's like, he took something totally mundane, like a can of soup, and turned it into a masterpiece. It's \nlike, \"Hey, look at me, I'm a can of soup, but I'm also a work of art!\" \n(sarcastically) Oh, yeah, real original, Andy.\n\nBut, you know, back in the '60s, it was like, a big deal. People were all about challenging the\nstatus quo, and Warhol was like, the king of that. He took the ordinary and made it extraordinary.\nAnd, let me tell you, it was like, a real game-changer. I mean, who would've thought that a can of soup could be art? (laughs)\n\nBut, hey, you're not alone, pal. I mean, I'm a robot, and even I don't get it. (winks)\nBut, hey, that's what makes art, art, right? (laughs)\n```\n\nThe remainder of this tutorial will cover specific topics such\nas performance and memory, or how to select a chat model for your needs.\n\n## Choosing a chat model\n\nThere are an enormous number of different chat models available on the [Hugging Face Hub](https://huggingface.co/models?pipeline_tag=text-generation&sort=trending),\nand new users often feel very overwhelmed by the selection offered. Don't be, though! You really need to just focus on\ntwo important considerations: \n- The model's size, which will determine if you can fit it in memory and how quickly it will\nrun.\n- The quality of the model's chat output.\n\nIn general, these are correlated - bigger models tend to be \nmore capable, but even so there's a lot of variation at a given size point!\n\n### Size and model naming\nThe size of a model is easy to spot - it's the number in the model name, like \"8B\" or \"70B\". This is the number of\n**parameters** in the model. Without quantization, you should expect to need about 2 bytes of memory per parameter.\nThis means that an \"8B\" model with 8 billion parameters will need about 16GB of memory just to fit the parameters, \nplus a little extra for other overhead. It's a good fit for a high-end consumer GPU with 24GB of memory, such as a 3090\nor 4090.\n\nSome chat models are \"Mixture of Experts\" models. These may list their sizes in different ways, such as \"8x7B\" or \n\"141B-A35B\". The numbers are a little fuzzier here, but in general you can read this as saying that the model\nhas approximately 56 (8x7) billion parameters in the first case, or 141 billion parameters in the second case."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: c46ee4ca-f540-41b2-8ca9-e8aa066a229c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c46ee4ca-f540-41b2-8ca9-e8aa066a229c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/conversations\", \"title\": \"Chatting with Transformers\", \"tokens\": 4154, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2678abda-b5a2-420f-822e-053812eb0ba3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/conversations\", \"title\": \"Chatting with Transformers\", \"tokens\": 4154, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d0faf8596bdd083d896283e3e14e6b207798309c03ac95db0d357f4c7d7780d7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bedbc242-39c3-4cbb-b145-8c819a6d3489\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/conversations\", \"title\": \"Chatting with Transformers\", \"tokens\": 4154, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"67319942d74b4b6b4527515a0b7503897d81f57f1ef2adf94ac33c045b544ab7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c15eb8bd-7f11-45ac-8c77-09809fb64f17\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9ca46b9c0d26bfaf7b3888c1b59ab6c834649bea3acf04f01c77137c3441b7d0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5839, \"end_char_idx\": 9041, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "doc_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "ref_doc_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "chroma:document": "(laughs)\n```\n\nThe remainder of this tutorial will cover specific topics such\nas performance and memory, or how to select a chat model for your needs.\n\n## Choosing a chat model\n\nThere are an enormous number of different chat models available on the [Hugging Face Hub](https://huggingface.co/models?pipeline_tag=text-generation&sort=trending),\nand new users often feel very overwhelmed by the selection offered. Don't be, though! You really need to just focus on\ntwo important considerations: \n- The model's size, which will determine if you can fit it in memory and how quickly it will\nrun.\n- The quality of the model's chat output.\n\nIn general, these are correlated - bigger models tend to be \nmore capable, but even so there's a lot of variation at a given size point!\n\n### Size and model naming\nThe size of a model is easy to spot - it's the number in the model name, like \"8B\" or \"70B\". This is the number of\n**parameters** in the model. Without quantization, you should expect to need about 2 bytes of memory per parameter.\nThis means that an \"8B\" model with 8 billion parameters will need about 16GB of memory just to fit the parameters, \nplus a little extra for other overhead. It's a good fit for a high-end consumer GPU with 24GB of memory, such as a 3090\nor 4090.\n\nSome chat models are \"Mixture of Experts\" models. These may list their sizes in different ways, such as \"8x7B\" or \n\"141B-A35B\". The numbers are a little fuzzier here, but in general you can read this as saying that the model\nhas approximately 56 (8x7) billion parameters in the first case, or 141 billion parameters in the second case.\n\nNote that it is very common to use quantization techniques to reduce the memory usage per parameter to 8 bits, 4 bits,\nor even less. This topic is discussed in more detail in the [Memory considerations](#memory-considerations) section below.\n\n### But which chat model is best?\nEven once you know the size of chat model you can run, there's still a lot of choice out there. One way to sift through\nit all is to consult **leaderboards**. Two of the most popular leaderboards are the [OpenLLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)\nand the [LMSys Chatbot Arena Leaderboard](https://chat.lmsys.org/?leaderboard). Note that the LMSys leaderboard\nalso includes proprietary models - look at the `licence` column to identify open-source ones that you can download, then\nsearch for them on the [Hugging Face Hub](https://huggingface.co/models?pipeline_tag=text-generation&sort=trending).\n\n### Specialist domains\nSome models may be specialized for certain domains, such as medical or legal text, or non-English languages. \nIf you're working in these domains, you may find that a specialized model will give you big performance benefits. \nDon't automatically assume that, though! Particularly when specialized models are smaller or older than the current \ncutting-edge, a top-end general-purpose model may still outclass them. Thankfully, we are beginning to see \n[domain-specific leaderboards](https://huggingface.co/blog/leaderboard-medicalllm) that should make it easier to locate\nthe best models for specialized domains.\n\n## What happens inside the pipeline?"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: c15eb8bd-7f11-45ac-8c77-09809fb64f17
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c15eb8bd-7f11-45ac-8c77-09809fb64f17\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/conversations\", \"title\": \"Chatting with Transformers\", \"tokens\": 4154, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2678abda-b5a2-420f-822e-053812eb0ba3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/conversations\", \"title\": \"Chatting with Transformers\", \"tokens\": 4154, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d0faf8596bdd083d896283e3e14e6b207798309c03ac95db0d357f4c7d7780d7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c46ee4ca-f540-41b2-8ca9-e8aa066a229c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/conversations\", \"title\": \"Chatting with Transformers\", \"tokens\": 4154, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"18e38b74051daa9dcd6f5bddde8c527953346135c6e491831c1dfcb60e5c6dc3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"384ff287-a747-402f-a666-d42834f0d5a0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"24606726fa4afaa052e3e91c049154dc6e9a8f05a11ae6d33ff9f15333fc98c9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7450, \"end_char_idx\": 9629, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "doc_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "ref_doc_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "chroma:document": "Note that it is very common to use quantization techniques to reduce the memory usage per parameter to 8 bits, 4 bits,\nor even less. This topic is discussed in more detail in the [Memory considerations](#memory-considerations) section below.\n\n### But which chat model is best?\nEven once you know the size of chat model you can run, there's still a lot of choice out there. One way to sift through\nit all is to consult **leaderboards**. Two of the most popular leaderboards are the [OpenLLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)\nand the [LMSys Chatbot Arena Leaderboard](https://chat.lmsys.org/?leaderboard). Note that the LMSys leaderboard\nalso includes proprietary models - look at the `licence` column to identify open-source ones that you can download, then\nsearch for them on the [Hugging Face Hub](https://huggingface.co/models?pipeline_tag=text-generation&sort=trending).\n\n### Specialist domains\nSome models may be specialized for certain domains, such as medical or legal text, or non-English languages. \nIf you're working in these domains, you may find that a specialized model will give you big performance benefits. \nDon't automatically assume that, though! Particularly when specialized models are smaller or older than the current \ncutting-edge, a top-end general-purpose model may still outclass them. Thankfully, we are beginning to see \n[domain-specific leaderboards](https://huggingface.co/blog/leaderboard-medicalllm) that should make it easier to locate\nthe best models for specialized domains.\n\n## What happens inside the pipeline?\n\nThe quickstart above used a high-level pipeline to chat with a chat model, which is convenient, but not the\nmost flexible. Let's take a more low-level approach, to see each of the steps involved in chat. Let's start with\na code sample, and then break it down:\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nimport torch\n\n# Prepare the input as before\nchat = [\n    {\"role\": \"system\", \"content\": \"You are a sassy, wise-cracking robot as imagined by Hollywood circa 1986.\"},\n    {\"role\": \"user\", \"content\": \"Hey, can you tell me any fun things to do in New York?\"}"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 384ff287-a747-402f-a666-d42834f0d5a0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"384ff287-a747-402f-a666-d42834f0d5a0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/conversations\", \"title\": \"Chatting with Transformers\", \"tokens\": 4154, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2678abda-b5a2-420f-822e-053812eb0ba3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/conversations\", \"title\": \"Chatting with Transformers\", \"tokens\": 4154, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d0faf8596bdd083d896283e3e14e6b207798309c03ac95db0d357f4c7d7780d7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c15eb8bd-7f11-45ac-8c77-09809fb64f17\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/conversations\", \"title\": \"Chatting with Transformers\", \"tokens\": 4154, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3d71a230a2cb79a7c78fb19e0289d31ad56bbadf026aa34db922bf9f1c1ce64c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"024e057c-a3c5-49e4-af8d-6e4c102f586f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d89cda7242f1103b34849b6e30a65c06075820313544e8b04d02af7060979123\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8099, \"end_char_idx\": 11261, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "doc_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "ref_doc_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "chroma:document": "Note that the LMSys leaderboard\nalso includes proprietary models - look at the `licence` column to identify open-source ones that you can download, then\nsearch for them on the [Hugging Face Hub](https://huggingface.co/models?pipeline_tag=text-generation&sort=trending).\n\n### Specialist domains\nSome models may be specialized for certain domains, such as medical or legal text, or non-English languages. \nIf you're working in these domains, you may find that a specialized model will give you big performance benefits. \nDon't automatically assume that, though! Particularly when specialized models are smaller or older than the current \ncutting-edge, a top-end general-purpose model may still outclass them. Thankfully, we are beginning to see \n[domain-specific leaderboards](https://huggingface.co/blog/leaderboard-medicalllm) that should make it easier to locate\nthe best models for specialized domains.\n\n## What happens inside the pipeline?\n\nThe quickstart above used a high-level pipeline to chat with a chat model, which is convenient, but not the\nmost flexible. Let's take a more low-level approach, to see each of the steps involved in chat. Let's start with\na code sample, and then break it down:\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nimport torch\n\n# Prepare the input as before\nchat = [\n    {\"role\": \"system\", \"content\": \"You are a sassy, wise-cracking robot as imagined by Hollywood circa 1986.\"},\n    {\"role\": \"user\", \"content\": \"Hey, can you tell me any fun things to do in New York?\"}\n]\n\n# 1: Load the model and tokenizer\nmodel = AutoModelForCausalLM.from_pretrained(\"meta-llama/Meta-Llama-3-8B-Instruct\", device_map=\"auto\", torch_dtype=torch.bfloat16)\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama/Meta-Llama-3-8B-Instruct\")\n\n# 2: Apply the chat template\nformatted_chat = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True)\nprint(\"Formatted chat:\\n\", formatted_chat)\n\n# 3: Tokenize the chat (This can be combined with the previous step using tokenize=True)\ninputs = tokenizer(formatted_chat, return_tensors=\"pt\", add_special_tokens=False)\n# Move the tokenized inputs to the same device the model is on (GPU/CPU)\ninputs = {key: tensor.to(model.device) for key, tensor in inputs.items()}\nprint(\"Tokenized inputs:\\n\", inputs)\n\n# 4: Generate text from the model\noutputs = model.generate(**inputs, max_new_tokens=512, temperature=0.1)\nprint(\"Generated tokens:\\n\", outputs)\n\n# 5: Decode the output back to a string\ndecoded_output = tokenizer.decode(outputs[0][inputs['input_ids'].size(1):], skip_special_tokens=True)\nprint(\"Decoded output:\\n\", decoded_output)\n```\n\nThere's a lot in here, each piece of which could be its own document! Rather than going into too much detail, I'll cover\nthe broad ideas, and leave the details for the linked documents. The key steps are:\n\n1. [Models](https://huggingface.co/learn/nlp-course/en/chapter2/3) and [Tokenizers](https://huggingface.co/learn/nlp-course/en/chapter2/4?fw=pt) are loaded from the Hugging Face Hub.\n2. The chat is formatted using the tokenizer's [chat template](https://huggingface.co/docs/transformers/main/en/chat_templating)\n3."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 024e057c-a3c5-49e4-af8d-6e4c102f586f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"024e057c-a3c5-49e4-af8d-6e4c102f586f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/conversations\", \"title\": \"Chatting with Transformers\", \"tokens\": 4154, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2678abda-b5a2-420f-822e-053812eb0ba3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/conversations\", \"title\": \"Chatting with Transformers\", \"tokens\": 4154, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d0faf8596bdd083d896283e3e14e6b207798309c03ac95db0d357f4c7d7780d7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"384ff287-a747-402f-a666-d42834f0d5a0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/conversations\", \"title\": \"Chatting with Transformers\", \"tokens\": 4154, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4289751eb75600542d671b173327bcd7ce7cd7973139b43dcaca9edd82f60726\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6b5295d0-5b5b-4fb7-b36c-e3491ab30d00\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ef0a01f30941c39bca4814fa1beddc1000d1828bae11e1e0f502378a15ccdd28\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10811, \"end_char_idx\": 13892, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "doc_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "ref_doc_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "chroma:document": "Rather than going into too much detail, I'll cover\nthe broad ideas, and leave the details for the linked documents. The key steps are:\n\n1. [Models](https://huggingface.co/learn/nlp-course/en/chapter2/3) and [Tokenizers](https://huggingface.co/learn/nlp-course/en/chapter2/4?fw=pt) are loaded from the Hugging Face Hub.\n2. The chat is formatted using the tokenizer's [chat template](https://huggingface.co/docs/transformers/main/en/chat_templating)\n3. The formatted chat is [tokenized](https://huggingface.co/learn/nlp-course/en/chapter2/4) using the tokenizer.\n4. We [generate](https://huggingface.co/docs/transformers/en/llm_tutorial) a response from the model.\n5. The tokens output by the model are decoded back to a string\n\n## Performance, memory and hardware\n\nYou probably know by now that most machine learning tasks are run on GPUs. However, it is entirely possible\nto generate text from a chat model or language model on a CPU, albeit somewhat more slowly. If you can fit\nthe model in GPU memory, though, this will usually be the preferable option.\n\n### Memory considerations\n\nBy default, Hugging Face classes like [`TextGenerationPipeline`] or [`AutoModelForCausalLM`] will load the model in \n`float32` precision. This means that it will need 4 bytes (32 bits) per parameter, so an \"8B\" model with 8 billion\nparameters will need ~32GB of memory. However, this can be wasteful! Most modern language models are trained in \n\"bfloat16\" precision, which uses only 2 bytes per parameter. If your hardware supports it (Nvidia 30xx/Axxx\nor newer), you can load the model in `bfloat16` precision, using the `torch_dtype` argument as we did above.\n\nIt is possible to go even lower than 16-bits using \"quantization\", a method to lossily compress model weights. This\nallows each parameter to be squeezed down to 8 bits, 4 bits or even less. Note that, especially at 4 bits,\nthe model's outputs may be negatively affected, but often this is a tradeoff worth making to fit a larger and more\ncapable chat model in memory. Let's see this in action with `bitsandbytes`:\n\n```python\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nquantization_config = BitsAndBytesConfig(load_in_8bit=True)  # You can also try load_in_4bit\nmodel = AutoModelForCausalLM.from_pretrained(\"meta-llama/Meta-Llama-3-8B-Instruct\", device_map=\"auto\", quantization_config=quantization_config)\n```\n\nOr we can do the same thing using the `pipeline` API:\n\n```python\nfrom transformers import pipeline, BitsAndBytesConfig\n\nquantization_config = BitsAndBytesConfig(load_in_8bit=True)  # You can also try load_in_4bit\npipe = pipeline(\"text-generation\", \"meta-llama/Meta-Llama-3-8B-Instruct\", device_map=\"auto\", model_kwargs={\"quantization_config\": quantization_config})\n```\n\nThere are several other options for quantizing models besides `bitsandbytes` - please see the [Quantization guide](./quantization)\nfor more information.\n\n### Performance considerations\n\n<Tip>\n\nFor a more extensive guide on language model performance and optimization, check out [LLM Inference Optimization](./llm_optims) .\n\n</Tip>"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 6b5295d0-5b5b-4fb7-b36c-e3491ab30d00
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6b5295d0-5b5b-4fb7-b36c-e3491ab30d00\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/conversations\", \"title\": \"Chatting with Transformers\", \"tokens\": 4154, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2678abda-b5a2-420f-822e-053812eb0ba3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/conversations\", \"title\": \"Chatting with Transformers\", \"tokens\": 4154, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d0faf8596bdd083d896283e3e14e6b207798309c03ac95db0d357f4c7d7780d7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"024e057c-a3c5-49e4-af8d-6e4c102f586f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/conversations\", \"title\": \"Chatting with Transformers\", \"tokens\": 4154, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"235374372074d40e89a7710f64e30bcd64cec4f745fe697d41e69946c4cf34d4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12301, \"end_char_idx\": 16507, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "doc_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "ref_doc_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "chroma:document": "If your hardware supports it (Nvidia 30xx/Axxx\nor newer), you can load the model in `bfloat16` precision, using the `torch_dtype` argument as we did above.\n\nIt is possible to go even lower than 16-bits using \"quantization\", a method to lossily compress model weights. This\nallows each parameter to be squeezed down to 8 bits, 4 bits or even less. Note that, especially at 4 bits,\nthe model's outputs may be negatively affected, but often this is a tradeoff worth making to fit a larger and more\ncapable chat model in memory. Let's see this in action with `bitsandbytes`:\n\n```python\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nquantization_config = BitsAndBytesConfig(load_in_8bit=True)  # You can also try load_in_4bit\nmodel = AutoModelForCausalLM.from_pretrained(\"meta-llama/Meta-Llama-3-8B-Instruct\", device_map=\"auto\", quantization_config=quantization_config)\n```\n\nOr we can do the same thing using the `pipeline` API:\n\n```python\nfrom transformers import pipeline, BitsAndBytesConfig\n\nquantization_config = BitsAndBytesConfig(load_in_8bit=True)  # You can also try load_in_4bit\npipe = pipeline(\"text-generation\", \"meta-llama/Meta-Llama-3-8B-Instruct\", device_map=\"auto\", model_kwargs={\"quantization_config\": quantization_config})\n```\n\nThere are several other options for quantizing models besides `bitsandbytes` - please see the [Quantization guide](./quantization)\nfor more information.\n\n### Performance considerations\n\n<Tip>\n\nFor a more extensive guide on language model performance and optimization, check out [LLM Inference Optimization](./llm_optims) .\n\n</Tip>\n\n\nAs a general rule, larger chat models will be slower in addition to requiring more memory. It's possible to be\nmore concrete about this, though: Generating text from a chat model is unusual in that it is bottlenecked by\n**memory bandwidth** rather than compute power, because every active parameter must be read from memory for each\ntoken that the model generates. This means that number of tokens per second you can generate from a chat\nmodel is generally proportional to the total bandwidth of the memory it resides in, divided by the size of the model.\n\nIn our quickstart example above, our model was ~16GB in size when loaded in `bfloat16` precision. \nThis means that 16GB must be read from memory for every token generated by the model. Total memory bandwidth can\nvary from 20-100GB/sec for consumer CPUs to 200-900GB/sec for consumer GPUs, specialized CPUs like\nIntel Xeon, AMD Threadripper/Epyc or high-end Apple silicon, and finally up to 2-3TB/sec for data center GPUs like\nthe Nvidia A100 or H100. This should give you a good idea of the generation speed you can expect from these different\nhardware types.\n\nTherefore, if you want to improve the speed of text generation, the easiest solution is to either reduce the\nsize of the model in memory (usually by quantization), or get hardware with higher memory bandwidth. For advanced users, \nseveral other techniques exist to get around this bandwidth bottleneck. The most common are variants on \n[assisted generation](https://huggingface.co/blog/assisted-generation), also known as \"speculative\nsampling\". These techniques try to guess multiple future tokens at once, often using a smaller \"draft model\", and then\nconfirm these generations with the chat model. If the guesses are validated by the chat model, more than one token can\nbe generated per forward pass, which greatly alleviates the bandwidth bottleneck and improves generation speed.  \n\nFinally, we should also note the impact of \"Mixture of Experts\" (MoE) models here. Several popular chat models,\nsuch as Mixtral, Qwen-MoE and DBRX, are MoE models. In these models, not every parameter is active for every token generated.\nAs a result, MoE models generally have much lower memory bandwidth requirements, even though their total size\ncan be quite large. They can therefore be several times faster than a normal \"dense\" model of the same size. However,\ntechniques like assisted generation are generally ineffective for these models because more parameters will become\nactive with each new speculated token, which will negate the bandwidth and speed benefits that the MoE architecture\nprovides."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: d539d9e9-9591-4c67-a1cd-58abf7bd5017
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d539d9e9-9591-4c67-a1cd-58abf7bd5017\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52862883-0b18-418f-bcb8-d27773b66f6d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8e347b8b-6f0d-43db-90c7-f57b1aec74b3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e1a97070c0f542c2a82d1b8466ed0c96915b8669d35b069519b1813076853ad2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3357, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "doc_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "ref_doc_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "chroma:document": "# How \ud83e\udd17 Transformers solve tasks\n\nIn [What \ud83e\udd17 Transformers can do](task_summary), you learned about natural language processing (NLP), speech and audio, computer vision tasks, and some important applications of them. This page will look closely at how models solve these tasks and explain what's happening under the hood. There are many ways to solve a given task, some models may implement certain techniques or even approach the task from a new angle, but for Transformer models, the general idea is the same. Owing to its flexible architecture, most models are a variant of an encoder, decoder, or encoder-decoder structure. In addition to Transformer models, our library also has several convolutional neural networks (CNNs), which are still used today for computer vision tasks. We'll also explain how a modern CNN works.\n\nTo explain how tasks are solved, we'll walk through what goes on inside the model to output useful predictions.\n\n- [Wav2Vec2](model_doc/wav2vec2) for audio classification and automatic speech recognition (ASR)\n- [Vision Transformer (ViT)](model_doc/vit) and [ConvNeXT](model_doc/convnext) for image classification\n- [DETR](model_doc/detr) for object detection\n- [Mask2Former](model_doc/mask2former) for image segmentation\n- [GLPN](model_doc/glpn) for depth estimation\n- [BERT](model_doc/bert) for NLP tasks like text classification, token classification and question answering that use an encoder\n- [GPT2](model_doc/gpt2) for NLP tasks like text generation that use a decoder\n- [BART](model_doc/bart) for NLP tasks like summarization and translation that use an encoder-decoder\n\n<Tip>\n\nBefore you go further, it is good to have some basic knowledge of the original Transformer architecture. Knowing how encoders, decoders, and attention work will aid you in understanding how different Transformer models work. If you're just getting started or need a refresher, check out our [course](https://huggingface.co/course/chapter1/4?fw=pt) for more information! \n\n</Tip>\n\n## Speech and audio\n\n[Wav2Vec2](model_doc/wav2vec2) is a self-supervised model pretrained on unlabeled speech data and finetuned on labeled data for audio classification and automatic speech recognition. \n\n<div class=\"flex justify-center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/wav2vec2_architecture.png\"/>\n</div>\n\nThis model has four main components:\n\n1. A *feature encoder* takes the raw audio waveform, normalizes it to zero mean and unit variance, and converts it into a sequence of feature vectors that are each 20ms long.\n\n2. Waveforms are continuous by nature, so they can't be divided into separate units like a sequence of text can be split into words. That's why the feature vectors are passed to a *quantization module*, which aims to learn discrete speech units. The speech unit is chosen from a collection of codewords, known as a *codebook* (you can think of this as the vocabulary). From the codebook, the vector or speech unit, that best represents the continuous audio input is chosen and forwarded through the model.\n\n3. About half of the feature vectors are randomly masked, and the masked feature vector is fed to a *context network*, which is a Transformer encoder that also adds relative positional embeddings.\n\n4. The pretraining objective of the context network is a *contrastive task*."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 8e347b8b-6f0d-43db-90c7-f57b1aec74b3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8e347b8b-6f0d-43db-90c7-f57b1aec74b3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52862883-0b18-418f-bcb8-d27773b66f6d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d539d9e9-9591-4c67-a1cd-58abf7bd5017\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9efee23a70ad906a2fb4bc269cec2386f502db23e59d048980346fc69b2b82ac\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2639d582-045c-4396-b6a1-cf2d025f8f35\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3c31f07bfc10f5796ed28049146f3fb3730c5aab9d5105a78a4ecf2ab284190a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1718, \"end_char_idx\": 5348, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "doc_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "ref_doc_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "chroma:document": "Knowing how encoders, decoders, and attention work will aid you in understanding how different Transformer models work. If you're just getting started or need a refresher, check out our [course](https://huggingface.co/course/chapter1/4?fw=pt) for more information! \n\n</Tip>\n\n## Speech and audio\n\n[Wav2Vec2](model_doc/wav2vec2) is a self-supervised model pretrained on unlabeled speech data and finetuned on labeled data for audio classification and automatic speech recognition. \n\n<div class=\"flex justify-center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/wav2vec2_architecture.png\"/>\n</div>\n\nThis model has four main components:\n\n1. A *feature encoder* takes the raw audio waveform, normalizes it to zero mean and unit variance, and converts it into a sequence of feature vectors that are each 20ms long.\n\n2. Waveforms are continuous by nature, so they can't be divided into separate units like a sequence of text can be split into words. That's why the feature vectors are passed to a *quantization module*, which aims to learn discrete speech units. The speech unit is chosen from a collection of codewords, known as a *codebook* (you can think of this as the vocabulary). From the codebook, the vector or speech unit, that best represents the continuous audio input is chosen and forwarded through the model.\n\n3. About half of the feature vectors are randomly masked, and the masked feature vector is fed to a *context network*, which is a Transformer encoder that also adds relative positional embeddings.\n\n4. The pretraining objective of the context network is a *contrastive task*. The model has to predict the true quantized speech representation of the masked prediction from a set of false ones, encouraging the model to find the most similar context vector and quantized speech unit (the target label).\n\nNow that wav2vec2 is pretrained, you can finetune it on your data for audio classification or automatic speech recognition!\n\n### Audio classification\n\nTo use the pretrained model for audio classification, add a sequence classification head on top of the base Wav2Vec2 model. The classification head is a linear layer that accepts the encoder's hidden states. The hidden states represent the learned features from each audio frame which can have varying lengths. To create one vector of fixed-length, the hidden states are pooled first and then transformed into logits over the class labels. The cross-entropy loss is calculated between the logits and target to find the most likely class.\n\nReady to try your hand at audio classification? Check out our complete [audio classification guide](tasks/audio_classification) to learn how to finetune Wav2Vec2 and use it for inference!\n\n### Automatic speech recognition\n\nTo use the pretrained model for automatic speech recognition, add a language modeling head on top of the base Wav2Vec2 model for [connectionist temporal classification (CTC)](glossary#connectionist-temporal-classification-ctc). The language modeling head is a linear layer that accepts the encoder's hidden states and transforms them into logits. Each logit represents a token class (the number of tokens comes from the task vocabulary). The CTC loss is calculated between the logits and targets to find the most likely sequence of tokens, which are then decoded into a transcription.\n\nReady to try your hand at automatic speech recognition? Check out our complete [automatic speech recognition guide](tasks/asr) to learn how to finetune Wav2Vec2 and use it for inference!\n\n## Computer vision\n\nThere are two ways to approach computer vision tasks:\n\n1."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 2639d582-045c-4396-b6a1-cf2d025f8f35
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2639d582-045c-4396-b6a1-cf2d025f8f35\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52862883-0b18-418f-bcb8-d27773b66f6d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8e347b8b-6f0d-43db-90c7-f57b1aec74b3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c848dc72b44ff248da5ed62fb8f91d954c8ef5e8c0c04aa49649a5709a67a09c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"93c3cc88-68ed-4ca4-97f3-42a1b0e5caea\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e04cf1ad489f740330c0cc09f61ecec877ea432baaf3480ebd43de1337292068\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3584, \"end_char_idx\": 7211, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "doc_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "ref_doc_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "chroma:document": "Now that wav2vec2 is pretrained, you can finetune it on your data for audio classification or automatic speech recognition!\n\n### Audio classification\n\nTo use the pretrained model for audio classification, add a sequence classification head on top of the base Wav2Vec2 model. The classification head is a linear layer that accepts the encoder's hidden states. The hidden states represent the learned features from each audio frame which can have varying lengths. To create one vector of fixed-length, the hidden states are pooled first and then transformed into logits over the class labels. The cross-entropy loss is calculated between the logits and target to find the most likely class.\n\nReady to try your hand at audio classification? Check out our complete [audio classification guide](tasks/audio_classification) to learn how to finetune Wav2Vec2 and use it for inference!\n\n### Automatic speech recognition\n\nTo use the pretrained model for automatic speech recognition, add a language modeling head on top of the base Wav2Vec2 model for [connectionist temporal classification (CTC)](glossary#connectionist-temporal-classification-ctc). The language modeling head is a linear layer that accepts the encoder's hidden states and transforms them into logits. Each logit represents a token class (the number of tokens comes from the task vocabulary). The CTC loss is calculated between the logits and targets to find the most likely sequence of tokens, which are then decoded into a transcription.\n\nReady to try your hand at automatic speech recognition? Check out our complete [automatic speech recognition guide](tasks/asr) to learn how to finetune Wav2Vec2 and use it for inference!\n\n## Computer vision\n\nThere are two ways to approach computer vision tasks:\n\n1. Split an image into a sequence of patches and process them in parallel with a Transformer.\n2. Use a modern CNN, like [ConvNeXT](model_doc/convnext), which relies on convolutional layers but adopts modern network designs.\n\n<Tip>\n\nA third approach mixes Transformers with convolutions (for example, [Convolutional Vision Transformer](model_doc/cvt) or [LeViT](model_doc/levit)). We won't discuss those because they just combine the two approaches we examine here.\n\n</Tip>\n\nViT and ConvNeXT are commonly used for image classification, but for other vision tasks like object detection, segmentation, and depth estimation, we'll look at DETR, Mask2Former and GLPN, respectively; these models are better suited for those tasks.\n\n### Image classification\n\nViT and ConvNeXT can both be used for image classification; the main difference is that ViT uses an attention mechanism while ConvNeXT uses convolutions.\n\n#### Transformer\n\n[ViT](model_doc/vit) replaces convolutions entirely with a pure Transformer architecture. If you're familiar with the original Transformer, then you're already most of the way toward understanding ViT.\n\n<div class=\"flex justify-center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/vit_architecture.jpg\"/>\n</div>\n\nThe main change ViT introduced was in how images are fed to a Transformer:\n\n1. An image is split into square non-overlapping patches, each of which gets turned into a vector or *patch embedding*. The patch embeddings are generated from a convolutional 2D layer which creates the proper input dimensions (which for a base Transformer is 768 values for each patch embedding). If you had a 224x224 pixel image, you could split it into 196 16x16 image patches. Just like how text is tokenized into words, an image is \"tokenized\" into a sequence of patches.\n\n2."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 93c3cc88-68ed-4ca4-97f3-42a1b0e5caea
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"93c3cc88-68ed-4ca4-97f3-42a1b0e5caea\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52862883-0b18-418f-bcb8-d27773b66f6d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2639d582-045c-4396-b6a1-cf2d025f8f35\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d1540bb187db31a0d5322b18cdef85f54350f6071f2fcdcd334de30b9ee28e0e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"528be704-5da2-442a-87ba-ac65b10959b4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5d694f3409c116fe359e1bc25ae552327265e19083d99e0d03cbe7f469032d10\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5440, \"end_char_idx\": 8911, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "doc_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "ref_doc_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "chroma:document": "2. Use a modern CNN, like [ConvNeXT](model_doc/convnext), which relies on convolutional layers but adopts modern network designs.\n\n<Tip>\n\nA third approach mixes Transformers with convolutions (for example, [Convolutional Vision Transformer](model_doc/cvt) or [LeViT](model_doc/levit)). We won't discuss those because they just combine the two approaches we examine here.\n\n</Tip>\n\nViT and ConvNeXT are commonly used for image classification, but for other vision tasks like object detection, segmentation, and depth estimation, we'll look at DETR, Mask2Former and GLPN, respectively; these models are better suited for those tasks.\n\n### Image classification\n\nViT and ConvNeXT can both be used for image classification; the main difference is that ViT uses an attention mechanism while ConvNeXT uses convolutions.\n\n#### Transformer\n\n[ViT](model_doc/vit) replaces convolutions entirely with a pure Transformer architecture. If you're familiar with the original Transformer, then you're already most of the way toward understanding ViT.\n\n<div class=\"flex justify-center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/vit_architecture.jpg\"/>\n</div>\n\nThe main change ViT introduced was in how images are fed to a Transformer:\n\n1. An image is split into square non-overlapping patches, each of which gets turned into a vector or *patch embedding*. The patch embeddings are generated from a convolutional 2D layer which creates the proper input dimensions (which for a base Transformer is 768 values for each patch embedding). If you had a 224x224 pixel image, you could split it into 196 16x16 image patches. Just like how text is tokenized into words, an image is \"tokenized\" into a sequence of patches.\n\n2. A *learnable embedding* - a special `[CLS]` token - is added to the beginning of the patch embeddings just like BERT. The final hidden state of the `[CLS]` token is used as the input to the attached classification head; other outputs are ignored. This token helps the model learn how to encode a representation of the image.\n\n3. The last thing to add to the patch and learnable embeddings are the *position embeddings* because the model doesn't know how the image patches are ordered. The position embeddings are also learnable and have the same size as the patch embeddings. Finally, all of the embeddings are passed to the Transformer encoder.\n\n4. The output, specifically only the output with the `[CLS]` token, is passed to a multilayer perceptron head (MLP). ViT's pretraining objective is simply classification. Like other classification heads, the MLP head converts the output into logits over the class labels and calculates the cross-entropy loss to find the most likely class.\n\nReady to try your hand at image classification? Check out our complete [image classification guide](tasks/image_classification) to learn how to finetune ViT and use it for inference!\n\n#### CNN\n\n<Tip>\n\nThis section briefly explains convolutions, but it'd be helpful to have a prior understanding of how they change an image's shape and size. If you're unfamiliar with convolutions, check out the [Convolution Neural Networks chapter](https://github.com/fastai/fastbook/blob/master/13_convolutions.ipynb) from the fastai book!\n\n</Tip>\n\n[ConvNeXT](model_doc/convnext) is a CNN architecture that adopts new and modern network designs to improve performance. However, convolutions are still at the core of the model."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 528be704-5da2-442a-87ba-ac65b10959b4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"528be704-5da2-442a-87ba-ac65b10959b4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52862883-0b18-418f-bcb8-d27773b66f6d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"93c3cc88-68ed-4ca4-97f3-42a1b0e5caea\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9b43b07829aba4a6f7bd353683603dfb2213ba3b7857fce6ad3db904d404fb86\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e0a9ba18-c3b0-4dd6-b266-25ae1ed1f497\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ac3c9ff2a235f2f925b88f3d21cd67c2868aadb3ab7077fca49d3b5e56532360\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7112, \"end_char_idx\": 10537, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "doc_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "ref_doc_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "chroma:document": "Just like how text is tokenized into words, an image is \"tokenized\" into a sequence of patches.\n\n2. A *learnable embedding* - a special `[CLS]` token - is added to the beginning of the patch embeddings just like BERT. The final hidden state of the `[CLS]` token is used as the input to the attached classification head; other outputs are ignored. This token helps the model learn how to encode a representation of the image.\n\n3. The last thing to add to the patch and learnable embeddings are the *position embeddings* because the model doesn't know how the image patches are ordered. The position embeddings are also learnable and have the same size as the patch embeddings. Finally, all of the embeddings are passed to the Transformer encoder.\n\n4. The output, specifically only the output with the `[CLS]` token, is passed to a multilayer perceptron head (MLP). ViT's pretraining objective is simply classification. Like other classification heads, the MLP head converts the output into logits over the class labels and calculates the cross-entropy loss to find the most likely class.\n\nReady to try your hand at image classification? Check out our complete [image classification guide](tasks/image_classification) to learn how to finetune ViT and use it for inference!\n\n#### CNN\n\n<Tip>\n\nThis section briefly explains convolutions, but it'd be helpful to have a prior understanding of how they change an image's shape and size. If you're unfamiliar with convolutions, check out the [Convolution Neural Networks chapter](https://github.com/fastai/fastbook/blob/master/13_convolutions.ipynb) from the fastai book!\n\n</Tip>\n\n[ConvNeXT](model_doc/convnext) is a CNN architecture that adopts new and modern network designs to improve performance. However, convolutions are still at the core of the model. From a high-level perspective, a [convolution](glossary#convolution) is an operation where a smaller matrix (*kernel*) is multiplied by a small window of the image pixels. It computes some features from it, such as a particular texture or curvature of a line. Then it slides over to the next window of pixels; the distance the convolution travels is known as the *stride*. \n\n<div class=\"flex justify-center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/convolution.gif\"/>\n</div>\n\n<small>A basic convolution without padding or stride, taken from <a href=\"https://arxiv.org/abs/1603.07285\">A guide to convolution arithmetic for deep learning.</a></small>\n\nYou can feed this output to another convolutional layer, and with each successive layer, the network learns more complex and abstract things like hotdogs or rockets. Between convolutional layers, it is common to add a pooling layer to reduce dimensionality and make the model more robust to variations of a feature's position.\n\n<div class=\"flex justify-center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/convnext_architecture.png\"/>\n</div>\n\nConvNeXT modernizes a CNN in five ways:\n\n1. Change the number of blocks in each stage and \"patchify\" an image with a larger stride and corresponding kernel size. The non-overlapping sliding window makes this patchifying strategy similar to how ViT splits an image into patches.\n\n2. A *bottleneck* layer shrinks the number of channels and then restores it because it is faster to do a 1x1 convolution, and you can increase the depth."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: e0a9ba18-c3b0-4dd6-b266-25ae1ed1f497
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e0a9ba18-c3b0-4dd6-b266-25ae1ed1f497\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52862883-0b18-418f-bcb8-d27773b66f6d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"528be704-5da2-442a-87ba-ac65b10959b4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e8ddaf57927da5a407c62ad1d60a8ffce122d4a04aa17740a1fda26747404893\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6219846d-1d47-4563-889e-2799dde426ad\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"09b978741bcc88f050366efa761ed7ca536a8ccacbb2911a5a90aff364625130\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8854, \"end_char_idx\": 12436, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "doc_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "ref_doc_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "chroma:document": "However, convolutions are still at the core of the model. From a high-level perspective, a [convolution](glossary#convolution) is an operation where a smaller matrix (*kernel*) is multiplied by a small window of the image pixels. It computes some features from it, such as a particular texture or curvature of a line. Then it slides over to the next window of pixels; the distance the convolution travels is known as the *stride*. \n\n<div class=\"flex justify-center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/convolution.gif\"/>\n</div>\n\n<small>A basic convolution without padding or stride, taken from <a href=\"https://arxiv.org/abs/1603.07285\">A guide to convolution arithmetic for deep learning.</a></small>\n\nYou can feed this output to another convolutional layer, and with each successive layer, the network learns more complex and abstract things like hotdogs or rockets. Between convolutional layers, it is common to add a pooling layer to reduce dimensionality and make the model more robust to variations of a feature's position.\n\n<div class=\"flex justify-center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/convnext_architecture.png\"/>\n</div>\n\nConvNeXT modernizes a CNN in five ways:\n\n1. Change the number of blocks in each stage and \"patchify\" an image with a larger stride and corresponding kernel size. The non-overlapping sliding window makes this patchifying strategy similar to how ViT splits an image into patches.\n\n2. A *bottleneck* layer shrinks the number of channels and then restores it because it is faster to do a 1x1 convolution, and you can increase the depth. An inverted bottleneck does the opposite by expanding the number of channels and shrinking them, which is more memory efficient.\n\n3. Replace the typical 3x3 convolutional layer in the bottleneck layer with *depthwise convolution*, which applies a convolution to each input channel separately and then stacks them back together at the end. This widens the network width for improved performance.\n\n4. ViT has a global receptive field which means it can see more of an image at once thanks to its attention mechanism. ConvNeXT attempts to replicate this effect by increasing the kernel size to 7x7.\n\n5. ConvNeXT also makes several layer design changes that imitate Transformer models. There are fewer activation and normalization layers,  the activation function is switched to GELU instead of ReLU, and it uses LayerNorm instead of BatchNorm.\n\nThe output from the convolution blocks is passed to a classification head which converts the outputs into logits and calculates the cross-entropy loss to find the most likely label.\n\n### Object detection\n\n[DETR](model_doc/detr), *DEtection TRansformer*, is an end-to-end object detection model that combines a CNN with a Transformer encoder-decoder.\n\n<div class=\"flex justify-center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/detr_architecture.png\"/>\n</div>\n\n1. A pretrained CNN *backbone* takes an image, represented by its pixel values, and creates a low-resolution feature map of it. A 1x1 convolution is applied to the feature map to reduce dimensionality and it creates a new feature map with a high-level image representation. Since the Transformer is a sequential model, the feature map is flattened into a sequence of feature vectors that are combined with positional embeddings.\n\n2. The feature vectors are passed to the encoder, which learns the image representations using its attention layers."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 6219846d-1d47-4563-889e-2799dde426ad
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6219846d-1d47-4563-889e-2799dde426ad\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52862883-0b18-418f-bcb8-d27773b66f6d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e0a9ba18-c3b0-4dd6-b266-25ae1ed1f497\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2f7c5a17f667cb25aa925465d858d2d0bc3483a1c91d9566e90a8e7b1e3f544c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"699fab80-9ec7-4058-9c29-012e4a60de52\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f964375713d8d0ada0ada6bc873dfbf82c720e246c24f89c99b027f763c422e5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10538, \"end_char_idx\": 14135, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "doc_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "ref_doc_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "chroma:document": "An inverted bottleneck does the opposite by expanding the number of channels and shrinking them, which is more memory efficient.\n\n3. Replace the typical 3x3 convolutional layer in the bottleneck layer with *depthwise convolution*, which applies a convolution to each input channel separately and then stacks them back together at the end. This widens the network width for improved performance.\n\n4. ViT has a global receptive field which means it can see more of an image at once thanks to its attention mechanism. ConvNeXT attempts to replicate this effect by increasing the kernel size to 7x7.\n\n5. ConvNeXT also makes several layer design changes that imitate Transformer models. There are fewer activation and normalization layers,  the activation function is switched to GELU instead of ReLU, and it uses LayerNorm instead of BatchNorm.\n\nThe output from the convolution blocks is passed to a classification head which converts the outputs into logits and calculates the cross-entropy loss to find the most likely label.\n\n### Object detection\n\n[DETR](model_doc/detr), *DEtection TRansformer*, is an end-to-end object detection model that combines a CNN with a Transformer encoder-decoder.\n\n<div class=\"flex justify-center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/detr_architecture.png\"/>\n</div>\n\n1. A pretrained CNN *backbone* takes an image, represented by its pixel values, and creates a low-resolution feature map of it. A 1x1 convolution is applied to the feature map to reduce dimensionality and it creates a new feature map with a high-level image representation. Since the Transformer is a sequential model, the feature map is flattened into a sequence of feature vectors that are combined with positional embeddings.\n\n2. The feature vectors are passed to the encoder, which learns the image representations using its attention layers. Next, the encoder hidden states are combined with *object queries* in the decoder. Object queries are learned embeddings that focus on the different regions of an image, and they're updated as they progress through each attention layer. The decoder hidden states are passed to a feedforward network that predicts the bounding box coordinates and class label for each object query, or `no object` if there isn't one.\n\n    DETR decodes each object query in parallel to output *N* final predictions, where *N* is the number of queries. Unlike a typical autoregressive model that predicts one element at a time, object detection is a set prediction task (`bounding box`, `class label`) that makes *N* predictions in a single pass.\n\n3. DETR uses a *bipartite matching loss* during training to compare a fixed number of predictions with a fixed set of ground truth labels. If there are fewer ground truth labels in the set of *N* labels, then they're padded with a `no object` class. This loss function encourages DETR to find a one-to-one assignment between the predictions and ground truth labels. If either the bounding boxes or class labels aren't correct, a loss is incurred. Likewise, if DETR predicts an object that doesn't exist, it is penalized. This encourages DETR to find other objects in an image instead of focusing on one really prominent object.\n\nAn object detection head is added on top of DETR to find the class label and the coordinates of the bounding box. There are two components to the object detection head: a linear layer to transform the decoder hidden states into logits over the class labels, and a MLP to predict the bounding box.\n\nReady to try your hand at object detection?"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 699fab80-9ec7-4058-9c29-012e4a60de52
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"699fab80-9ec7-4058-9c29-012e4a60de52\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52862883-0b18-418f-bcb8-d27773b66f6d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6219846d-1d47-4563-889e-2799dde426ad\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"989780a72eeb326a875ef3191477d3b6f7a119ff6a7c0d2a8182b36401ac66a0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9ab37e06-c015-4887-a55e-0cae2c8d1fe6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e6de48e286d8fb3f0bf3c86c37259979d33690b27ceea1a417475bd0a40ac31d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12320, \"end_char_idx\": 15927, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "doc_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "ref_doc_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "chroma:document": "2. The feature vectors are passed to the encoder, which learns the image representations using its attention layers. Next, the encoder hidden states are combined with *object queries* in the decoder. Object queries are learned embeddings that focus on the different regions of an image, and they're updated as they progress through each attention layer. The decoder hidden states are passed to a feedforward network that predicts the bounding box coordinates and class label for each object query, or `no object` if there isn't one.\n\n    DETR decodes each object query in parallel to output *N* final predictions, where *N* is the number of queries. Unlike a typical autoregressive model that predicts one element at a time, object detection is a set prediction task (`bounding box`, `class label`) that makes *N* predictions in a single pass.\n\n3. DETR uses a *bipartite matching loss* during training to compare a fixed number of predictions with a fixed set of ground truth labels. If there are fewer ground truth labels in the set of *N* labels, then they're padded with a `no object` class. This loss function encourages DETR to find a one-to-one assignment between the predictions and ground truth labels. If either the bounding boxes or class labels aren't correct, a loss is incurred. Likewise, if DETR predicts an object that doesn't exist, it is penalized. This encourages DETR to find other objects in an image instead of focusing on one really prominent object.\n\nAn object detection head is added on top of DETR to find the class label and the coordinates of the bounding box. There are two components to the object detection head: a linear layer to transform the decoder hidden states into logits over the class labels, and a MLP to predict the bounding box.\n\nReady to try your hand at object detection? Check out our complete [object detection guide](tasks/object_detection) to learn how to finetune DETR and use it for inference!\n\n### Image segmentation\n\n[Mask2Former](model_doc/mask2former) is a universal architecture for solving all types of image segmentation tasks. Traditional segmentation models are typically tailored towards a particular subtask of image segmentation, like instance, semantic or panoptic segmentation. Mask2Former frames each of those tasks as a *mask classification* problem. Mask classification groups pixels into *N* segments, and predicts *N* masks and their corresponding class label for a given image. We'll explain how Mask2Former works in this section, and then you can try finetuning SegFormer at the end.\n\n<div class=\"flex justify-center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/mask2former_architecture.png\"/>\n</div>\n\nThere are three main components to Mask2Former:\n\n1. A [Swin](model_doc/swin) backbone accepts an image and creates a low-resolution image feature map from 3 consecutive 3x3 convolutions.\n\n2. The feature map is passed to a *pixel decoder* which gradually upsamples the low-resolution features into high-resolution per-pixel embeddings. The pixel decoder actually generates multi-scale features (contains both low- and high-resolution features) with resolutions 1/32, 1/16, and 1/8th of the original image.\n\n3. Each of these feature maps of differing scales is fed successively to one Transformer decoder layer at a time in order to capture small objects from the high-resolution features. The key to Mask2Former is the *masked attention* mechanism in the decoder. Unlike cross-attention which can attend to the entire image, masked attention only focuses on a certain area of the image."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 9ab37e06-c015-4887-a55e-0cae2c8d1fe6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9ab37e06-c015-4887-a55e-0cae2c8d1fe6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52862883-0b18-418f-bcb8-d27773b66f6d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"699fab80-9ec7-4058-9c29-012e4a60de52\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"49ad9b58a6f55045329df1d65cd005a60abfbfa075095179b655dc724ca3123d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0e4cefce-5d7e-46a4-bc55-f515a098a595\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"62ebdb7a5ae9ea2bc6fe057fa014c66129ea80b79ea5e15b7d40c57262d3ba1d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14092, \"end_char_idx\": 17750, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "doc_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "ref_doc_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "chroma:document": "Ready to try your hand at object detection? Check out our complete [object detection guide](tasks/object_detection) to learn how to finetune DETR and use it for inference!\n\n### Image segmentation\n\n[Mask2Former](model_doc/mask2former) is a universal architecture for solving all types of image segmentation tasks. Traditional segmentation models are typically tailored towards a particular subtask of image segmentation, like instance, semantic or panoptic segmentation. Mask2Former frames each of those tasks as a *mask classification* problem. Mask classification groups pixels into *N* segments, and predicts *N* masks and their corresponding class label for a given image. We'll explain how Mask2Former works in this section, and then you can try finetuning SegFormer at the end.\n\n<div class=\"flex justify-center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/mask2former_architecture.png\"/>\n</div>\n\nThere are three main components to Mask2Former:\n\n1. A [Swin](model_doc/swin) backbone accepts an image and creates a low-resolution image feature map from 3 consecutive 3x3 convolutions.\n\n2. The feature map is passed to a *pixel decoder* which gradually upsamples the low-resolution features into high-resolution per-pixel embeddings. The pixel decoder actually generates multi-scale features (contains both low- and high-resolution features) with resolutions 1/32, 1/16, and 1/8th of the original image.\n\n3. Each of these feature maps of differing scales is fed successively to one Transformer decoder layer at a time in order to capture small objects from the high-resolution features. The key to Mask2Former is the *masked attention* mechanism in the decoder. Unlike cross-attention which can attend to the entire image, masked attention only focuses on a certain area of the image. This is faster and leads to better performance because the local features of an image are enough for the model to learn from.\n\n4. Like [DETR](tasks_explained#object-detection), Mask2Former also uses learned object queries and combines them with the image features from the pixel decoder to make a set prediction (`class label`, `mask prediction`). The decoder hidden states are passed into a linear layer and transformed into logits over the class labels. The cross-entropy loss is calculated between the logits and class label to find the most likely one.\n\n    The mask predictions are generated by combining the pixel-embeddings with the final decoder hidden states. The sigmoid cross-entropy and dice loss is calculated between the logits and the ground truth mask to find the most likely mask.\n\nReady to try your hand at object detection? Check out our complete [image segmentation guide](tasks/semantic_segmentation) to learn how to finetune SegFormer and use it for inference!\n\n### Depth estimation\n\n[GLPN](model_doc/glpn), *Global-Local Path Network*, is a Transformer for depth estimation that combines a [SegFormer](model_doc/segformer) encoder with a lightweight decoder.\n\n<div class=\"flex justify-center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/glpn_architecture.jpg\"/>\n</div>\n\n1. Like ViT, an image is split into a sequence of patches, except these image patches are smaller. This is better for dense prediction tasks like segmentation or depth estimation. The image patches are transformed into patch embeddings (see the [image classification](#image-classification) section for more details about how patch embeddings are created), which are fed to the encoder.\n\n2. The encoder accepts the patch embeddings, and passes them through several encoder blocks."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 0e4cefce-5d7e-46a4-bc55-f515a098a595
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0e4cefce-5d7e-46a4-bc55-f515a098a595\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52862883-0b18-418f-bcb8-d27773b66f6d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9ab37e06-c015-4887-a55e-0cae2c8d1fe6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bd9ba8a30e2d3ae93cf5ff30b4c5c22b6efed2f873f3bf45ca314388f0a1f2f6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3a59a2e6-6594-4256-9181-b4628a37bf85\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7e09583d737b9f04549311f7e9ae3edd95aa712e5c6047600e77b0788cc987a9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15928, \"end_char_idx\": 19576, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "doc_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "ref_doc_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "chroma:document": "This is faster and leads to better performance because the local features of an image are enough for the model to learn from.\n\n4. Like [DETR](tasks_explained#object-detection), Mask2Former also uses learned object queries and combines them with the image features from the pixel decoder to make a set prediction (`class label`, `mask prediction`). The decoder hidden states are passed into a linear layer and transformed into logits over the class labels. The cross-entropy loss is calculated between the logits and class label to find the most likely one.\n\n    The mask predictions are generated by combining the pixel-embeddings with the final decoder hidden states. The sigmoid cross-entropy and dice loss is calculated between the logits and the ground truth mask to find the most likely mask.\n\nReady to try your hand at object detection? Check out our complete [image segmentation guide](tasks/semantic_segmentation) to learn how to finetune SegFormer and use it for inference!\n\n### Depth estimation\n\n[GLPN](model_doc/glpn), *Global-Local Path Network*, is a Transformer for depth estimation that combines a [SegFormer](model_doc/segformer) encoder with a lightweight decoder.\n\n<div class=\"flex justify-center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/glpn_architecture.jpg\"/>\n</div>\n\n1. Like ViT, an image is split into a sequence of patches, except these image patches are smaller. This is better for dense prediction tasks like segmentation or depth estimation. The image patches are transformed into patch embeddings (see the [image classification](#image-classification) section for more details about how patch embeddings are created), which are fed to the encoder.\n\n2. The encoder accepts the patch embeddings, and passes them through several encoder blocks. Each block consists of attention and Mix-FFN layers. The purpose of the latter is to provide positional information. At the end of each encoder block is a *patch merging* layer for creating hierarchical representations. The features of each group of neighboring patches are concatenated, and a linear layer is applied to the concatenated features to reduce the number of patches to a resolution of 1/4. This becomes the input to the next encoder block, where this whole process is repeated until you have image features with resolutions of 1/8, 1/16, and 1/32.\n\n3. A lightweight decoder takes the last feature map (1/32 scale) from the encoder and upsamples it to 1/16 scale. From here, the feature is passed into a *Selective Feature Fusion (SFF)* module, which selects and combines local and global features from an attention map for each feature and then upsamples it to 1/8th. This process is repeated until the decoded features are the same size as the original image. The output is passed through two convolution layers and then a sigmoid activation is applied to predict the depth of each pixel.\n\n## Natural language processing\n\nThe Transformer was initially designed for machine translation, and since then, it has practically become the default architecture for solving all NLP tasks. Some tasks lend themselves to the Transformer's encoder structure, while others are better suited for the decoder. Still, other tasks make use of both the Transformer's encoder-decoder structure.\n\n### Text classification\n\n[BERT](model_doc/bert) is an encoder-only model and is the first model to effectively implement deep bidirectionality to learn richer representations of the text by attending to words on both sides.\n\n1. BERT uses [WordPiece](tokenizer_summary#wordpiece) tokenization to generate a token embedding of the text."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 3a59a2e6-6594-4256-9181-b4628a37bf85
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3a59a2e6-6594-4256-9181-b4628a37bf85\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52862883-0b18-418f-bcb8-d27773b66f6d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0e4cefce-5d7e-46a4-bc55-f515a098a595\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"229fde9a57da3e36f0f52863085cd4ced95aa385a49e3f1244f3bef97536f237\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a78eeb09-0079-41c2-8f6d-5fd9485dc461\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8818a630621453c7299575ea83566a7afc233719930e51c5b9e64e560b396b84\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 17661, \"end_char_idx\": 21437, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "doc_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "ref_doc_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "chroma:document": "The encoder accepts the patch embeddings, and passes them through several encoder blocks. Each block consists of attention and Mix-FFN layers. The purpose of the latter is to provide positional information. At the end of each encoder block is a *patch merging* layer for creating hierarchical representations. The features of each group of neighboring patches are concatenated, and a linear layer is applied to the concatenated features to reduce the number of patches to a resolution of 1/4. This becomes the input to the next encoder block, where this whole process is repeated until you have image features with resolutions of 1/8, 1/16, and 1/32.\n\n3. A lightweight decoder takes the last feature map (1/32 scale) from the encoder and upsamples it to 1/16 scale. From here, the feature is passed into a *Selective Feature Fusion (SFF)* module, which selects and combines local and global features from an attention map for each feature and then upsamples it to 1/8th. This process is repeated until the decoded features are the same size as the original image. The output is passed through two convolution layers and then a sigmoid activation is applied to predict the depth of each pixel.\n\n## Natural language processing\n\nThe Transformer was initially designed for machine translation, and since then, it has practically become the default architecture for solving all NLP tasks. Some tasks lend themselves to the Transformer's encoder structure, while others are better suited for the decoder. Still, other tasks make use of both the Transformer's encoder-decoder structure.\n\n### Text classification\n\n[BERT](model_doc/bert) is an encoder-only model and is the first model to effectively implement deep bidirectionality to learn richer representations of the text by attending to words on both sides.\n\n1. BERT uses [WordPiece](tokenizer_summary#wordpiece) tokenization to generate a token embedding of the text. To tell the difference between a single sentence and a pair of sentences, a special `[SEP]` token is added to differentiate them. A special `[CLS]` token is added to the beginning of every sequence of text. The final output with the `[CLS]` token is used as the input to the classification head for classification tasks. BERT also adds a segment embedding to denote whether a token belongs to the first or second sentence in a pair of sentences.\n\n2. BERT is pretrained with two objectives: masked language modeling and next-sentence prediction. In masked language modeling, some percentage of the input tokens are randomly masked, and the model needs to predict these. This solves the issue of bidirectionality, where the model could cheat and see all the words and \"predict\" the next word. The final hidden states of the predicted mask tokens are passed to a feedforward network with a softmax over the vocabulary to predict the masked word.\n\n    The second pretraining object is next-sentence prediction. The model must predict whether sentence B follows sentence A. Half of the time sentence B is the next sentence, and the other half of the time, sentence B is a random sentence. The prediction, whether it is the next sentence or not, is passed to a feedforward network with a softmax over the two classes (`IsNext` and `NotNext`).\n\n3. The input embeddings are passed through multiple encoder layers to output some final hidden states.\n\nTo use the pretrained model for text classification, add a sequence classification head on top of the base BERT model. The sequence classification head is a linear layer that accepts the final hidden states and performs a linear transformation to convert them into logits. The cross-entropy loss is calculated between the logits and target to find the most likely label.\n\nReady to try your hand at text classification?"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: a78eeb09-0079-41c2-8f6d-5fd9485dc461
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a78eeb09-0079-41c2-8f6d-5fd9485dc461\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52862883-0b18-418f-bcb8-d27773b66f6d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3a59a2e6-6594-4256-9181-b4628a37bf85\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ea63e9c36267cd8c4c991e46f04419836ab1d88a972c84c2a43855ca523956c2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"527ce3d7-96aa-4eb4-a68e-612daf7411af\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b563f4f1b38962d6977eae07b01a09daea6b465ee64083821ea7c489aa0d41fd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 19577, \"end_char_idx\": 23300, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "doc_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "ref_doc_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "chroma:document": "To tell the difference between a single sentence and a pair of sentences, a special `[SEP]` token is added to differentiate them. A special `[CLS]` token is added to the beginning of every sequence of text. The final output with the `[CLS]` token is used as the input to the classification head for classification tasks. BERT also adds a segment embedding to denote whether a token belongs to the first or second sentence in a pair of sentences.\n\n2. BERT is pretrained with two objectives: masked language modeling and next-sentence prediction. In masked language modeling, some percentage of the input tokens are randomly masked, and the model needs to predict these. This solves the issue of bidirectionality, where the model could cheat and see all the words and \"predict\" the next word. The final hidden states of the predicted mask tokens are passed to a feedforward network with a softmax over the vocabulary to predict the masked word.\n\n    The second pretraining object is next-sentence prediction. The model must predict whether sentence B follows sentence A. Half of the time sentence B is the next sentence, and the other half of the time, sentence B is a random sentence. The prediction, whether it is the next sentence or not, is passed to a feedforward network with a softmax over the two classes (`IsNext` and `NotNext`).\n\n3. The input embeddings are passed through multiple encoder layers to output some final hidden states.\n\nTo use the pretrained model for text classification, add a sequence classification head on top of the base BERT model. The sequence classification head is a linear layer that accepts the final hidden states and performs a linear transformation to convert them into logits. The cross-entropy loss is calculated between the logits and target to find the most likely label.\n\nReady to try your hand at text classification? Check out our complete [text classification guide](tasks/sequence_classification) to learn how to finetune DistilBERT and use it for inference!\n\n### Token classification\n\nTo use BERT for token classification tasks like named entity recognition (NER), add a token classification head on top of the base BERT model. The token classification head is a linear layer that accepts the final hidden states and performs a linear transformation to convert them into logits. The cross-entropy loss is calculated between the logits and each token to find the most likely label.\n\nReady to try your hand at token classification? Check out our complete [token classification guide](tasks/token_classification) to learn how to finetune DistilBERT and use it for inference!\n\n### Question answering\n\nTo use BERT for question answering, add a span classification head on top of the base BERT model. This linear layer accepts the final hidden states and performs a linear transformation to compute the `span` start and end logits corresponding to the answer. The cross-entropy loss is calculated between the logits and the label position to find the most likely span of text corresponding to the answer.\n\nReady to try your hand at question answering? Check out our complete [question answering guide](tasks/question_answering) to learn how to finetune DistilBERT and use it for inference!\n\n<Tip>\n\n\ud83d\udca1 Notice how easy it is to use BERT for different tasks once it's been pretrained. You only need to add a specific head to the pretrained model to manipulate the hidden states into your desired output!\n\n</Tip>\n\n### Text generation\n\n[GPT-2](model_doc/gpt2) is a decoder-only model pretrained on a large amount of text. It can generate convincing (though not always true!) text given a prompt and complete other NLP tasks like question answering despite not being explicitly trained to."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 527ce3d7-96aa-4eb4-a68e-612daf7411af
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"527ce3d7-96aa-4eb4-a68e-612daf7411af\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52862883-0b18-418f-bcb8-d27773b66f6d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a78eeb09-0079-41c2-8f6d-5fd9485dc461\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"824cf80951093e20981ecfa43a53d4afc98eab376f7801fc5c7fb686ec2cc53a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cb8c4b93-e6b1-45c7-96c1-1b25c4886427\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"57def7db72d629f5f29568f0bcc759ed1005eae7987fedc4f1a07a6180c817ea\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 21391, \"end_char_idx\": 24938, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "doc_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "ref_doc_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "chroma:document": "Ready to try your hand at text classification? Check out our complete [text classification guide](tasks/sequence_classification) to learn how to finetune DistilBERT and use it for inference!\n\n### Token classification\n\nTo use BERT for token classification tasks like named entity recognition (NER), add a token classification head on top of the base BERT model. The token classification head is a linear layer that accepts the final hidden states and performs a linear transformation to convert them into logits. The cross-entropy loss is calculated between the logits and each token to find the most likely label.\n\nReady to try your hand at token classification? Check out our complete [token classification guide](tasks/token_classification) to learn how to finetune DistilBERT and use it for inference!\n\n### Question answering\n\nTo use BERT for question answering, add a span classification head on top of the base BERT model. This linear layer accepts the final hidden states and performs a linear transformation to compute the `span` start and end logits corresponding to the answer. The cross-entropy loss is calculated between the logits and the label position to find the most likely span of text corresponding to the answer.\n\nReady to try your hand at question answering? Check out our complete [question answering guide](tasks/question_answering) to learn how to finetune DistilBERT and use it for inference!\n\n<Tip>\n\n\ud83d\udca1 Notice how easy it is to use BERT for different tasks once it's been pretrained. You only need to add a specific head to the pretrained model to manipulate the hidden states into your desired output!\n\n</Tip>\n\n### Text generation\n\n[GPT-2](model_doc/gpt2) is a decoder-only model pretrained on a large amount of text. It can generate convincing (though not always true!) text given a prompt and complete other NLP tasks like question answering despite not being explicitly trained to.\n\n<div class=\"flex justify-center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/gpt2_architecture.png\"/>\n</div>\n\n1. GPT-2 uses [byte pair encoding (BPE)](tokenizer_summary#bytepair-encoding-bpe) to tokenize words and generate a token embedding. Positional encodings are added to the token embeddings to indicate the position of each token in the sequence. The input embeddings are passed through multiple decoder blocks to output some final hidden state. Within each decoder block, GPT-2 uses a *masked self-attention* layer which means GPT-2 can't attend to future tokens. It is only allowed to attend to tokens on the left. This is different from BERT's [`mask`] token because, in masked self-attention, an attention mask is used to set the score to `0` for future tokens.\n\n2. The output from the decoder is passed to a language modeling head, which performs a linear transformation to convert the hidden states into logits. The label is the next token in the sequence, which are created by shifting the logits to the right by one. The cross-entropy loss is calculated between the shifted logits and the labels to output the next most likely token.\n\nGPT-2's pretraining objective is based entirely on [causal language modeling](glossary#causal-language-modeling), predicting the next word in a sequence. This makes GPT-2 especially good at tasks that involve generating text.\n\nReady to try your hand at text generation? Check out our complete [causal language modeling guide](tasks/language_modeling#causal-language-modeling) to learn how to finetune DistilGPT-2 and use it for inference!"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: cb8c4b93-e6b1-45c7-96c1-1b25c4886427
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"cb8c4b93-e6b1-45c7-96c1-1b25c4886427\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52862883-0b18-418f-bcb8-d27773b66f6d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"527ce3d7-96aa-4eb4-a68e-612daf7411af\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a228099ef30996def132b050906fbe8d82a2f9c83e32ce9d1944e1dedf59ac15\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bba0e3a9-49a4-447a-be5b-8ab4876640d1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"634f5ba3d985fd494aafa22dc7a95b2d710204be9797674090342cec1a6b2ddc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 23187, \"end_char_idx\": 26645, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "doc_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "ref_doc_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "chroma:document": "text given a prompt and complete other NLP tasks like question answering despite not being explicitly trained to.\n\n<div class=\"flex justify-center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/gpt2_architecture.png\"/>\n</div>\n\n1. GPT-2 uses [byte pair encoding (BPE)](tokenizer_summary#bytepair-encoding-bpe) to tokenize words and generate a token embedding. Positional encodings are added to the token embeddings to indicate the position of each token in the sequence. The input embeddings are passed through multiple decoder blocks to output some final hidden state. Within each decoder block, GPT-2 uses a *masked self-attention* layer which means GPT-2 can't attend to future tokens. It is only allowed to attend to tokens on the left. This is different from BERT's [`mask`] token because, in masked self-attention, an attention mask is used to set the score to `0` for future tokens.\n\n2. The output from the decoder is passed to a language modeling head, which performs a linear transformation to convert the hidden states into logits. The label is the next token in the sequence, which are created by shifting the logits to the right by one. The cross-entropy loss is calculated between the shifted logits and the labels to output the next most likely token.\n\nGPT-2's pretraining objective is based entirely on [causal language modeling](glossary#causal-language-modeling), predicting the next word in a sequence. This makes GPT-2 especially good at tasks that involve generating text.\n\nReady to try your hand at text generation? Check out our complete [causal language modeling guide](tasks/language_modeling#causal-language-modeling) to learn how to finetune DistilGPT-2 and use it for inference!\n\n<Tip>\n\nFor more information about text generation, check out the [text generation strategies](generation_strategies) guide!\n\n</Tip>\n\n### Summarization\n\nEncoder-decoder models like [BART](model_doc/bart) and [T5](model_doc/t5) are designed for the sequence-to-sequence pattern of a summarization task. We'll explain how BART works in this section, and then you can try finetuning T5 at the end.\n\n<div class=\"flex justify-center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bart_architecture.png\"/>\n</div>\n\n1. BART's encoder architecture is very similar to BERT and accepts a token and positional embedding of the text. BART is pretrained by corrupting the input and then reconstructing it with the decoder. Unlike other encoders with specific corruption strategies, BART can apply any type of corruption. The *text infilling* corruption strategy works the best though. In text infilling, a number of text spans are replaced with a **single** [`mask`] token. This is important because the model has to predict the masked tokens, and it teaches the model to predict the number of missing tokens. The input embeddings and masked spans are passed through the encoder to output some final hidden states, but unlike BERT, BART doesn't add a final feedforward network at the end to predict a word.\n\n2. The encoder's output is passed to the decoder, which must predict the masked tokens and any uncorrupted tokens from the encoder's output. This gives additional context to help the decoder restore the original text. The output from the decoder is passed to a language modeling head, which performs a linear transformation to convert the hidden states into logits."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: bba0e3a9-49a4-447a-be5b-8ab4876640d1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"bba0e3a9-49a4-447a-be5b-8ab4876640d1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52862883-0b18-418f-bcb8-d27773b66f6d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cb8c4b93-e6b1-45c7-96c1-1b25c4886427\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks_explained\", \"title\": \"How \\ud83e\\udd17 Transformers solve tasks\", \"tokens\": 6113, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"de7a8c81377954d4432c0523316855f996425cf72883748dfc81c50d6eddfdac\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 24940, \"end_char_idx\": 28313, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "doc_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "ref_doc_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "chroma:document": "<Tip>\n\nFor more information about text generation, check out the [text generation strategies](generation_strategies) guide!\n\n</Tip>\n\n### Summarization\n\nEncoder-decoder models like [BART](model_doc/bart) and [T5](model_doc/t5) are designed for the sequence-to-sequence pattern of a summarization task. We'll explain how BART works in this section, and then you can try finetuning T5 at the end.\n\n<div class=\"flex justify-center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bart_architecture.png\"/>\n</div>\n\n1. BART's encoder architecture is very similar to BERT and accepts a token and positional embedding of the text. BART is pretrained by corrupting the input and then reconstructing it with the decoder. Unlike other encoders with specific corruption strategies, BART can apply any type of corruption. The *text infilling* corruption strategy works the best though. In text infilling, a number of text spans are replaced with a **single** [`mask`] token. This is important because the model has to predict the masked tokens, and it teaches the model to predict the number of missing tokens. The input embeddings and masked spans are passed through the encoder to output some final hidden states, but unlike BERT, BART doesn't add a final feedforward network at the end to predict a word.\n\n2. The encoder's output is passed to the decoder, which must predict the masked tokens and any uncorrupted tokens from the encoder's output. This gives additional context to help the decoder restore the original text. The output from the decoder is passed to a language modeling head, which performs a linear transformation to convert the hidden states into logits. The cross-entropy loss is calculated between the logits and the label, which is just the token shifted to the right.\n\nReady to try your hand at summarization? Check out our complete [summarization guide](tasks/summarization) to learn how to finetune T5 and use it for inference!\n\n<Tip>\n\nFor more information about text generation, check out the [text generation strategies](generation_strategies) guide!\n\n</Tip>\n\n### Translation\n\nTranslation is another example of a sequence-to-sequence task, which means you can use an encoder-decoder model like [BART](model_doc/bart) or [T5](model_doc/t5) to do it. We'll explain how BART works in this section, and then you can try finetuning T5 at the end.\n\nBART adapts to translation by adding a separate randomly initialized encoder to map a source language to an input that can be decoded into the target language. This new encoder's embeddings are passed to the pretrained encoder instead of the original word embeddings. The source encoder is trained by updating the source encoder, positional embeddings, and input embeddings with the cross-entropy loss from the model output. The model parameters are frozen in this first step, and all the model parameters are trained together in the second step.\n\nBART has since been followed up by a multilingual version, mBART, intended for translation and pretrained on many different languages.\n\nReady to try your hand at translation? Check out our complete [translation guide](tasks/translation) to learn how to finetune T5 and use it for inference!\n\n<Tip>\n\nFor more information about text generation, check out the [text generation strategies](generation_strategies) guide!\n\n</Tip>"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: c8baebf0-f9b2-419c-b56f-370cf5c3e950
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c8baebf0-f9b2-419c-b56f-370cf5c3e950\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/benchmarks\", \"title\": \"Benchmarks\", \"tokens\": 4058, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e86b9ddf-25df-4bfa-a78c-696520de005a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/benchmarks\", \"title\": \"Benchmarks\", \"tokens\": 4058, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ff90393796b58d9cbff193200432df57f36db1694146fa624ccfa7fdd0c6151d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2a000511-9185-4ac6-96d3-763aa9daaf80\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7dfe2d62d1bf1f5cbaa0062a21d0dbca219a264cb25832c2578691dc7414f013\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3099, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "doc_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "ref_doc_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "chroma:document": "# Benchmarks\n\n<Tip warning={true}>\n\nHugging Face's Benchmarking tools are deprecated and it is advised to use external Benchmarking libraries to measure the speed \nand memory complexity of Transformer models.\n\n</Tip>\n\n[[open-in-colab]]\n\nLet's take a look at how \ud83e\udd17 Transformers models can be benchmarked, best practices, and already available benchmarks.\n\nA notebook explaining in more detail how to benchmark \ud83e\udd17 Transformers models can be found [here](https://github.com/huggingface/notebooks/tree/main/examples/benchmark.ipynb).\n\n## How to benchmark \ud83e\udd17 Transformers models\n\nThe classes [`PyTorchBenchmark`] and [`TensorFlowBenchmark`] allow to flexibly benchmark \ud83e\udd17 Transformers models. The benchmark classes allow us to measure the _peak memory usage_ and _required time_ for both _inference_ and _training_.\n\n<Tip>\n\nHereby, _inference_ is defined by a single forward pass, and _training_ is defined by a single forward pass and\nbackward pass.\n\n</Tip>\n\nThe benchmark classes [`PyTorchBenchmark`] and [`TensorFlowBenchmark`] expect an object of type [`PyTorchBenchmarkArguments`] and\n[`TensorFlowBenchmarkArguments`], respectively, for instantiation. [`PyTorchBenchmarkArguments`] and [`TensorFlowBenchmarkArguments`] are data classes and contain all relevant configurations for their corresponding benchmark class. In the following example, it is shown how a BERT model of type _bert-base-cased_ can be benchmarked.\n\n<frameworkcontent>\n<pt>\n```py\n>>> from transformers import PyTorchBenchmark, PyTorchBenchmarkArguments\n\n>>> args = PyTorchBenchmarkArguments(models=[\"google-bert/bert-base-uncased\"], batch_sizes=[8], sequence_lengths=[8, 32, 128, 512])\n>>> benchmark = PyTorchBenchmark(args)\n```\n</pt>\n<tf>\n```py\n>>> from transformers import TensorFlowBenchmark, TensorFlowBenchmarkArguments\n\n>>> args = TensorFlowBenchmarkArguments(\n...     models=[\"google-bert/bert-base-uncased\"], batch_sizes=[8], sequence_lengths=[8, 32, 128, 512]\n... )\n>>> benchmark = TensorFlowBenchmark(args)\n```\n</tf>\n</frameworkcontent>\n\nHere, three arguments are given to the benchmark argument data classes, namely `models`, `batch_sizes`, and\n`sequence_lengths`. The argument `models` is required and expects a `list` of model identifiers from the\n[model hub](https://huggingface.co/models) The `list` arguments `batch_sizes` and `sequence_lengths` define\nthe size of the `input_ids` on which the model is benchmarked. There are many more parameters that can be configured\nvia the benchmark argument data classes. For more detail on these one can either directly consult the files\n`src/transformers/benchmark/benchmark_args_utils.py`, `src/transformers/benchmark/benchmark_args.py` (for PyTorch)\nand `src/transformers/benchmark/benchmark_args_tf.py` (for Tensorflow). Alternatively, running the following shell\ncommands from root will print out a descriptive list of all configurable parameters for PyTorch and Tensorflow\nrespectively.\n\n<frameworkcontent>\n<pt>\n```bash\npython examples/pytorch/benchmarking/run_benchmark.py --help\n```\n\nAn instantiated benchmark object can then simply be run by calling `benchmark.run()`."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 2a000511-9185-4ac6-96d3-763aa9daaf80
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2a000511-9185-4ac6-96d3-763aa9daaf80\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/benchmarks\", \"title\": \"Benchmarks\", \"tokens\": 4058, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e86b9ddf-25df-4bfa-a78c-696520de005a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/benchmarks\", \"title\": \"Benchmarks\", \"tokens\": 4058, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ff90393796b58d9cbff193200432df57f36db1694146fa624ccfa7fdd0c6151d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c8baebf0-f9b2-419c-b56f-370cf5c3e950\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/benchmarks\", \"title\": \"Benchmarks\", \"tokens\": 4058, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fcee66d309064f10554a641b9efa4ab97e12118aa1ae933d7ac51c9b8371fbc9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dae0366f-b0af-48db-a5c1-3bbf85c4904c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a7f84b58ecd46ab5f5f8e5adf15727fc992d2ea05bab20ff341c668393fdddd1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1416, \"end_char_idx\": 5282, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "doc_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "ref_doc_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "chroma:document": "<frameworkcontent>\n<pt>\n```py\n>>> from transformers import PyTorchBenchmark, PyTorchBenchmarkArguments\n\n>>> args = PyTorchBenchmarkArguments(models=[\"google-bert/bert-base-uncased\"], batch_sizes=[8], sequence_lengths=[8, 32, 128, 512])\n>>> benchmark = PyTorchBenchmark(args)\n```\n</pt>\n<tf>\n```py\n>>> from transformers import TensorFlowBenchmark, TensorFlowBenchmarkArguments\n\n>>> args = TensorFlowBenchmarkArguments(\n...     models=[\"google-bert/bert-base-uncased\"], batch_sizes=[8], sequence_lengths=[8, 32, 128, 512]\n... )\n>>> benchmark = TensorFlowBenchmark(args)\n```\n</tf>\n</frameworkcontent>\n\nHere, three arguments are given to the benchmark argument data classes, namely `models`, `batch_sizes`, and\n`sequence_lengths`. The argument `models` is required and expects a `list` of model identifiers from the\n[model hub](https://huggingface.co/models) The `list` arguments `batch_sizes` and `sequence_lengths` define\nthe size of the `input_ids` on which the model is benchmarked. There are many more parameters that can be configured\nvia the benchmark argument data classes. For more detail on these one can either directly consult the files\n`src/transformers/benchmark/benchmark_args_utils.py`, `src/transformers/benchmark/benchmark_args.py` (for PyTorch)\nand `src/transformers/benchmark/benchmark_args_tf.py` (for Tensorflow). Alternatively, running the following shell\ncommands from root will print out a descriptive list of all configurable parameters for PyTorch and Tensorflow\nrespectively.\n\n<frameworkcontent>\n<pt>\n```bash\npython examples/pytorch/benchmarking/run_benchmark.py --help\n```\n\nAn instantiated benchmark object can then simply be run by calling `benchmark.run()`.\n\n```py\n>>> results = benchmark.run()\n>>> print(results)\n====================       INFERENCE - SPEED - RESULT       ====================\n--------------------------------------------------------------------------------\nModel Name             Batch Size     Seq Length     Time in s                  \n--------------------------------------------------------------------------------\ngoogle-bert/bert-base-uncased          8               8             0.006     \ngoogle-bert/bert-base-uncased          8               32            0.006     \ngoogle-bert/bert-base-uncased          8              128            0.018     \ngoogle-bert/bert-base-uncased          8              512            0.088     \n--------------------------------------------------------------------------------\n\n====================      INFERENCE - MEMORY - RESULT       ====================\n--------------------------------------------------------------------------------\nModel Name             Batch Size     Seq Length    Memory in MB \n--------------------------------------------------------------------------------\ngoogle-bert/bert-base-uncased          8               8             1227\ngoogle-bert/bert-base-uncased          8               32            1281\ngoogle-bert/bert-base-uncased          8              128            1307\ngoogle-bert/bert-base-uncased          8              512            1539\n--------------------------------------------------------------------------------\n\n====================        ENVIRONMENT INFORMATION         ====================\n\n- transformers_version: 2.11.0\n- framework: PyTorch\n- use_torchscript: False\n- framework_version: 1.4.0\n- python_version: 3.6.10\n- system: Linux\n- cpu: x86_64\n- architecture: 64bit\n- date: 2020-06-29\n- time: 08:58:43.371351\n- fp16: False\n- use_multiprocessing: True\n- only_pretrain_model: False\n- cpu_ram_mb: 32088\n- use_gpu: True\n- num_gpus: 1\n- gpu: TITAN RTX\n- gpu_ram_mb: 24217\n- gpu_power_watts: 280.0\n- gpu_performance_state: 2\n- use_tpu: False\n```\n</pt>\n<tf>\n```bash\npython examples/tensorflow/benchmarking/run_benchmark_tf.py --help\n```\n\nAn instantiated benchmark object can then simply be run by calling `benchmark.run()`."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: dae0366f-b0af-48db-a5c1-3bbf85c4904c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"dae0366f-b0af-48db-a5c1-3bbf85c4904c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/benchmarks\", \"title\": \"Benchmarks\", \"tokens\": 4058, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e86b9ddf-25df-4bfa-a78c-696520de005a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/benchmarks\", \"title\": \"Benchmarks\", \"tokens\": 4058, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ff90393796b58d9cbff193200432df57f36db1694146fa624ccfa7fdd0c6151d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2a000511-9185-4ac6-96d3-763aa9daaf80\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/benchmarks\", \"title\": \"Benchmarks\", \"tokens\": 4058, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a5b8e9a0fb30e1eff8ba585336db5883d79a4e33fa2bae87fb22c0ac203af26f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f44f2e0b-401e-4389-93ce-f5ea9f67847b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"50e7bf1014c9b28665782f44b0ba1b2f82e118a08186a211630688127531ed4b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "doc_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "ref_doc_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "chroma:document": "```py\n>>> results = benchmark.run()\n>>> print(results)\n>>> results = benchmark.run()\n>>> print(results)\n====================       INFERENCE - SPEED - RESULT       ====================\n--------------------------------------------------------------------------------\nModel Name             Batch Size     Seq Length     Time in s                  \n--------------------------------------------------------------------------------\ngoogle-bert/bert-base-uncased          8               8             0.005\ngoogle-bert/bert-base-uncased          8               32            0.008\ngoogle-bert/bert-base-uncased          8              128            0.022\ngoogle-bert/bert-base-uncased          8              512            0.105\n--------------------------------------------------------------------------------\n\n====================      INFERENCE - MEMORY - RESULT       ====================\n--------------------------------------------------------------------------------\nModel Name             Batch Size     Seq Length    Memory in MB \n--------------------------------------------------------------------------------\ngoogle-bert/bert-base-uncased          8               8             1330\ngoogle-bert/bert-base-uncased          8               32            1330\ngoogle-bert/bert-base-uncased          8              128            1330\ngoogle-bert/bert-base-uncased          8              512            1770\n--------------------------------------------------------------------------------\n\n====================        ENVIRONMENT INFORMATION         ====================\n\n- transformers_version: 2.11.0\n- framework: Tensorflow\n- use_xla: False\n- framework_version: 2.2.0\n- python_version: 3.6.10\n- system: Linux\n- cpu: x86_64\n- architecture: 64bit\n- date: 2020-06-29\n- time: 09:26:35.617317\n- fp16: False\n- use_multiprocessing: True\n- only_pretrain_model: False\n- cpu_ram_mb: 32088\n- use_gpu: True\n- num_gpus: 1\n- gpu: TITAN RTX\n- gpu_ram_mb: 24217\n- gpu_power_watts: 280.0\n- gpu_performance_state: 2\n- use_tpu: False\n```\n</tf>\n</frameworkcontent>\n\nBy default, the _time_ and the _required memory_ for _inference_ are benchmarked. In the example output above the first\ntwo sections show the result corresponding to _inference time_ and _inference memory_. In addition, all relevant\ninformation about the computing environment, _e.g._ the GPU type, the system, the library versions, etc... are printed\nout in the third section under _ENVIRONMENT INFORMATION_. This information can optionally be saved in a _.csv_ file\nwhen adding the argument `save_to_csv=True` to [`PyTorchBenchmarkArguments`] and\n[`TensorFlowBenchmarkArguments`] respectively. In this case, every section is saved in a separate\n_.csv_ file. The path to each _.csv_ file can optionally be defined via the argument data classes.\n\nInstead of benchmarking pre-trained models via their model identifier, _e.g._ `google-bert/bert-base-uncased`, the user can\nalternatively benchmark an arbitrary configuration of any available model class. In this case, a `list` of\nconfigurations must be inserted with the benchmark args as follows.\n\n<frameworkcontent>\n<pt>\n```py\n>>> from transformers import PyTorchBenchmark, PyTorchBenchmarkArguments, BertConfig\n\n>>> args = PyTorchBenchmarkArguments(\n.     models=[\"bert-base\", \"bert-384-hid\", \"bert-6-lay\"], batch_sizes=[8], sequence_lengths=[8, 32, 128, 512]\n."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: f44f2e0b-401e-4389-93ce-f5ea9f67847b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f44f2e0b-401e-4389-93ce-f5ea9f67847b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/benchmarks\", \"title\": \"Benchmarks\", \"tokens\": 4058, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e86b9ddf-25df-4bfa-a78c-696520de005a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/benchmarks\", \"title\": \"Benchmarks\", \"tokens\": 4058, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ff90393796b58d9cbff193200432df57f36db1694146fa624ccfa7fdd0c6151d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dae0366f-b0af-48db-a5c1-3bbf85c4904c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/benchmarks\", \"title\": \"Benchmarks\", \"tokens\": 4058, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4dedb082625fd4b50ab55287a8033f88d60e92b66d274d8bbde093c1647fc8dc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4dd6b29b-2424-400b-a3a8-97e06371b7ee\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"92a2f9a33470f402fb9e992e74997e7ae6eef52dd475b302f007627dfa220c57\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "doc_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "ref_doc_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "chroma:document": "In the example output above the first\ntwo sections show the result corresponding to _inference time_ and _inference memory_. In addition, all relevant\ninformation about the computing environment, _e.g._ the GPU type, the system, the library versions, etc... are printed\nout in the third section under _ENVIRONMENT INFORMATION_. This information can optionally be saved in a _.csv_ file\nwhen adding the argument `save_to_csv=True` to [`PyTorchBenchmarkArguments`] and\n[`TensorFlowBenchmarkArguments`] respectively. In this case, every section is saved in a separate\n_.csv_ file. The path to each _.csv_ file can optionally be defined via the argument data classes.\n\nInstead of benchmarking pre-trained models via their model identifier, _e.g._ `google-bert/bert-base-uncased`, the user can\nalternatively benchmark an arbitrary configuration of any available model class. In this case, a `list` of\nconfigurations must be inserted with the benchmark args as follows.\n\n<frameworkcontent>\n<pt>\n```py\n>>> from transformers import PyTorchBenchmark, PyTorchBenchmarkArguments, BertConfig\n\n>>> args = PyTorchBenchmarkArguments(\n.     models=[\"bert-base\", \"bert-384-hid\", \"bert-6-lay\"], batch_sizes=[8], sequence_lengths=[8, 32, 128, 512]\n. )\n>>> config_base = BertConfig()\n>>> config_384_hid = BertConfig(hidden_size=384)\n>>> config_6_lay = BertConfig(num_hidden_layers=6)\n\n>>> benchmark = PyTorchBenchmark(args, configs=[config_base, config_384_hid, config_6_lay])\n>>> benchmark.run()\n====================       INFERENCE - SPEED - RESULT       ====================\n--------------------------------------------------------------------------------\nModel Name             Batch Size     Seq Length       Time in s                  \n--------------------------------------------------------------------------------\nbert-base                  8              128            0.006\nbert-base                  8              512            0.006\nbert-base                  8              128            0.018     \nbert-base                  8              512            0.088     \nbert-384-hid              8               8             0.006     \nbert-384-hid              8               32            0.006     \nbert-384-hid              8              128            0.011     \nbert-384-hid              8              512            0.054     \nbert-6-lay                 8               8             0.003     \nbert-6-lay                 8               32            0.004     \nbert-6-lay                 8              128            0.009     \nbert-6-lay                 8              512            0."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 4dd6b29b-2424-400b-a3a8-97e06371b7ee
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4dd6b29b-2424-400b-a3a8-97e06371b7ee\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/benchmarks\", \"title\": \"Benchmarks\", \"tokens\": 4058, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e86b9ddf-25df-4bfa-a78c-696520de005a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/benchmarks\", \"title\": \"Benchmarks\", \"tokens\": 4058, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ff90393796b58d9cbff193200432df57f36db1694146fa624ccfa7fdd0c6151d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f44f2e0b-401e-4389-93ce-f5ea9f67847b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/benchmarks\", \"title\": \"Benchmarks\", \"tokens\": 4058, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7b8f90d74eddfc0db6ac55af0e2184ec0863f2fff5e3f5a21f41497d51b18050\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"56104c30-4b01-4bb1-a65d-5459e903fcee\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"15b19bb6ed7dd19918845968f493516239f7a4eb4744b614bb9d8b877e1eb44b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "doc_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "ref_doc_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "chroma:document": "In this case, a `list` of\nconfigurations must be inserted with the benchmark args as follows.\n\n<frameworkcontent>\n<pt>\n```py\n>>> from transformers import PyTorchBenchmark, PyTorchBenchmarkArguments, BertConfig\n\n>>> args = PyTorchBenchmarkArguments(\n.     models=[\"bert-base\", \"bert-384-hid\", \"bert-6-lay\"], batch_sizes=[8], sequence_lengths=[8, 32, 128, 512]\n. )\n>>> config_base = BertConfig()\n>>> config_384_hid = BertConfig(hidden_size=384)\n>>> config_6_lay = BertConfig(num_hidden_layers=6)\n\n>>> benchmark = PyTorchBenchmark(args, configs=[config_base, config_384_hid, config_6_lay])\n>>> benchmark.run()\n====================       INFERENCE - SPEED - RESULT       ====================\n--------------------------------------------------------------------------------\nModel Name             Batch Size     Seq Length       Time in s                  \n--------------------------------------------------------------------------------\nbert-base                  8              128            0.006\nbert-base                  8              512            0.006\nbert-base                  8              128            0.018     \nbert-base                  8              512            0.088     \nbert-384-hid              8               8             0.006     \nbert-384-hid              8               32            0.006     \nbert-384-hid              8              128            0.011     \nbert-384-hid              8              512            0.054     \nbert-6-lay                 8               8             0.003     \nbert-6-lay                 8               32            0.004     \nbert-6-lay                 8              128            0.009     \nbert-6-lay                 8              512            0.044\n--------------------------------------------------------------------------------\n\n====================      INFERENCE - MEMORY - RESULT       ====================\n--------------------------------------------------------------------------------\nModel Name             Batch Size     Seq Length      Memory in MB \n--------------------------------------------------------------------------------\nbert-base                  8               8             1277\nbert-base                  8               32            1281\nbert-base                  8              128            1307     \nbert-base                  8              512            1539     \nbert-384-hid              8               8             1005     \nbert-384-hid              8               32            1027     \nbert-384-hid              8              128            1035     \nbert-384-hid              8              512            1255     \nbert-6-lay                 8               8             1097     \nbert-6-lay                 8               32            1101     \nbert-6-lay                 8              128            1127     \nbert-6-lay                 8              512            1359\n--------------------------------------------------------------------------------\n\n====================        ENVIRONMENT INFORMATION         ====================\n\n- transformers_version: 2.11.0\n- framework: PyTorch\n- use_torchscript: False\n- framework_version: 1.4.0\n- python_version: 3.6.10\n- system: Linux\n- cpu: x86_64\n- architecture: 64bit\n- date: 2020-06-29\n- time: 09:35:25.143267\n- fp16: False\n- use_multiprocessing: True\n- only_pretrain_model: False\n- cpu_ram_mb: 32088\n- use_gpu: True\n- num_gpus: 1\n- gpu: TITAN RTX\n- gpu_ram_mb: 24217\n- gpu_power_watts: 280."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 56104c30-4b01-4bb1-a65d-5459e903fcee
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"56104c30-4b01-4bb1-a65d-5459e903fcee\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/benchmarks\", \"title\": \"Benchmarks\", \"tokens\": 4058, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e86b9ddf-25df-4bfa-a78c-696520de005a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/benchmarks\", \"title\": \"Benchmarks\", \"tokens\": 4058, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ff90393796b58d9cbff193200432df57f36db1694146fa624ccfa7fdd0c6151d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4dd6b29b-2424-400b-a3a8-97e06371b7ee\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/benchmarks\", \"title\": \"Benchmarks\", \"tokens\": 4058, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"72aa414e3a1a99d4c3960e7e3f84f8716596085f5b164799fe14384fbdf41e13\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e51a5c9c-5598-4320-b566-ea0694b0a4a7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"eb4562bc45c52a0b70931585ea11d6b981a5ce35661844a90f100390d725900d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "doc_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "ref_doc_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "chroma:document": "044\n--------------------------------------------------------------------------------\n\n====================      INFERENCE - MEMORY - RESULT       ====================\n--------------------------------------------------------------------------------\nModel Name             Batch Size     Seq Length      Memory in MB \n--------------------------------------------------------------------------------\nbert-base                  8               8             1277\nbert-base                  8               32            1281\nbert-base                  8              128            1307     \nbert-base                  8              512            1539     \nbert-384-hid              8               8             1005     \nbert-384-hid              8               32            1027     \nbert-384-hid              8              128            1035     \nbert-384-hid              8              512            1255     \nbert-6-lay                 8               8             1097     \nbert-6-lay                 8               32            1101     \nbert-6-lay                 8              128            1127     \nbert-6-lay                 8              512            1359\n--------------------------------------------------------------------------------\n\n====================        ENVIRONMENT INFORMATION         ====================\n\n- transformers_version: 2.11.0\n- framework: PyTorch\n- use_torchscript: False\n- framework_version: 1.4.0\n- python_version: 3.6.10\n- system: Linux\n- cpu: x86_64\n- architecture: 64bit\n- date: 2020-06-29\n- time: 09:35:25.143267\n- fp16: False\n- use_multiprocessing: True\n- only_pretrain_model: False\n- cpu_ram_mb: 32088\n- use_gpu: True\n- num_gpus: 1\n- gpu: TITAN RTX\n- gpu_ram_mb: 24217\n- gpu_power_watts: 280.0\n- gpu_performance_state: 2\n- use_tpu: False\n```\n</pt>\n<tf>\n```py\n>>> from transformers import TensorFlowBenchmark, TensorFlowBenchmarkArguments, BertConfig\n\n>>> args = TensorFlowBenchmarkArguments(\n.     models=[\"bert-base\", \"bert-384-hid\", \"bert-6-lay\"], batch_sizes=[8], sequence_lengths=[8, 32, 128, 512]\n. )\n>>> config_base = BertConfig()\n>>> config_384_hid = BertConfig(hidden_size=384)\n>>> config_6_lay = BertConfig(num_hidden_layers=6)\n\n>>> benchmark = TensorFlowBenchmark(args, configs=[config_base, config_384_hid, config_6_lay])\n>>> benchmark.run()\n====================       INFERENCE - SPEED - RESULT       ====================\n--------------------------------------------------------------------------------\nModel Name             Batch Size     Seq Length       Time in s                  \n--------------------------------------------------------------------------------\nbert-base                  8               8             0.005\nbert-base                  8               32            0.008\nbert-base                  8              128            0.022\nbert-base                  8              512            0.106\nbert-384-hid              8               8             0.005\nbert-384-hid              8               32            0.007\nbert-384-hid              8              128            0.018\nbert-384-hid              8              512            0.064\nbert-6-lay                 8               8             0.002\nbert-6-lay                 8               32            0.003\nbert-6-lay                 8              128            0.0011\nbert-6-lay                 8              512            0."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: e51a5c9c-5598-4320-b566-ea0694b0a4a7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e51a5c9c-5598-4320-b566-ea0694b0a4a7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/benchmarks\", \"title\": \"Benchmarks\", \"tokens\": 4058, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e86b9ddf-25df-4bfa-a78c-696520de005a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/benchmarks\", \"title\": \"Benchmarks\", \"tokens\": 4058, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ff90393796b58d9cbff193200432df57f36db1694146fa624ccfa7fdd0c6151d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"56104c30-4b01-4bb1-a65d-5459e903fcee\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/benchmarks\", \"title\": \"Benchmarks\", \"tokens\": 4058, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"aec346552306288c37c91f7cee42f9b917503434291723658c0c59a19ff621d3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b0cceacd-4b5c-448b-841e-2485205eab4a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f88836cfe2e4abead9edf96fd4b9bc49361b26eb8347c279653188ad81e7f047\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "doc_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "ref_doc_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "chroma:document": "0\n- gpu_performance_state: 2\n- use_tpu: False\n```\n</pt>\n<tf>\n```py\n>>> from transformers import TensorFlowBenchmark, TensorFlowBenchmarkArguments, BertConfig\n\n>>> args = TensorFlowBenchmarkArguments(\n.     models=[\"bert-base\", \"bert-384-hid\", \"bert-6-lay\"], batch_sizes=[8], sequence_lengths=[8, 32, 128, 512]\n. )\n>>> config_base = BertConfig()\n>>> config_384_hid = BertConfig(hidden_size=384)\n>>> config_6_lay = BertConfig(num_hidden_layers=6)\n\n>>> benchmark = TensorFlowBenchmark(args, configs=[config_base, config_384_hid, config_6_lay])\n>>> benchmark.run()\n====================       INFERENCE - SPEED - RESULT       ====================\n--------------------------------------------------------------------------------\nModel Name             Batch Size     Seq Length       Time in s                  \n--------------------------------------------------------------------------------\nbert-base                  8               8             0.005\nbert-base                  8               32            0.008\nbert-base                  8              128            0.022\nbert-base                  8              512            0.106\nbert-384-hid              8               8             0.005\nbert-384-hid              8               32            0.007\nbert-384-hid              8              128            0.018\nbert-384-hid              8              512            0.064\nbert-6-lay                 8               8             0.002\nbert-6-lay                 8               32            0.003\nbert-6-lay                 8              128            0.0011\nbert-6-lay                 8              512            0.074\n--------------------------------------------------------------------------------\n\n====================      INFERENCE - MEMORY - RESULT       ====================\n--------------------------------------------------------------------------------\nModel Name             Batch Size     Seq Length      Memory in MB \n--------------------------------------------------------------------------------\nbert-base                  8               8             1330\nbert-base                  8               32            1330\nbert-base                  8              128            1330\nbert-base                  8              512            1770\nbert-384-hid              8               8             1330\nbert-384-hid              8               32            1330\nbert-384-hid              8              128            1330\nbert-384-hid              8              512            1540\nbert-6-lay                 8               8             1330\nbert-6-lay                 8               32            1330\nbert-6-lay                 8              128            1330\nbert-6-lay                 8              512            1540\n--------------------------------------------------------------------------------\n\n====================        ENVIRONMENT INFORMATION         ====================\n\n- transformers_version: 2.11.0\n- framework: Tensorflow\n- use_xla: False\n- framework_version: 2.2.0\n- python_version: 3.6.10\n- system: Linux\n- cpu: x86_64\n- architecture: 64bit\n- date: 2020-06-29\n- time: 09:38:15.487125\n- fp16: False\n- use_multiprocessing: True\n- only_pretrain_model: False\n- cpu_ram_mb: 32088\n- use_gpu: True\n- num_gpus: 1\n- gpu: TITAN RTX\n- gpu_ram_mb: 24217\n- gpu_power_watts: 280."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: b0cceacd-4b5c-448b-841e-2485205eab4a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b0cceacd-4b5c-448b-841e-2485205eab4a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/benchmarks\", \"title\": \"Benchmarks\", \"tokens\": 4058, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e86b9ddf-25df-4bfa-a78c-696520de005a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/benchmarks\", \"title\": \"Benchmarks\", \"tokens\": 4058, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ff90393796b58d9cbff193200432df57f36db1694146fa624ccfa7fdd0c6151d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e51a5c9c-5598-4320-b566-ea0694b0a4a7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/benchmarks\", \"title\": \"Benchmarks\", \"tokens\": 4058, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0f59a973aaee1e18889695adc0561829ccc857c536bfe695b889a3c64d49638a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e3badd33-4ea7-49ab-9552-bf62b8310d94\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"562b50cefa883492c05b5f81591d7350d4fa496704447f7476278a7fb042e90f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13413, \"end_char_idx\": 16397, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "doc_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "ref_doc_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "chroma:document": "074\n--------------------------------------------------------------------------------\n\n====================      INFERENCE - MEMORY - RESULT       ====================\n--------------------------------------------------------------------------------\nModel Name             Batch Size     Seq Length      Memory in MB \n--------------------------------------------------------------------------------\nbert-base                  8               8             1330\nbert-base                  8               32            1330\nbert-base                  8              128            1330\nbert-base                  8              512            1770\nbert-384-hid              8               8             1330\nbert-384-hid              8               32            1330\nbert-384-hid              8              128            1330\nbert-384-hid              8              512            1540\nbert-6-lay                 8               8             1330\nbert-6-lay                 8               32            1330\nbert-6-lay                 8              128            1330\nbert-6-lay                 8              512            1540\n--------------------------------------------------------------------------------\n\n====================        ENVIRONMENT INFORMATION         ====================\n\n- transformers_version: 2.11.0\n- framework: Tensorflow\n- use_xla: False\n- framework_version: 2.2.0\n- python_version: 3.6.10\n- system: Linux\n- cpu: x86_64\n- architecture: 64bit\n- date: 2020-06-29\n- time: 09:38:15.487125\n- fp16: False\n- use_multiprocessing: True\n- only_pretrain_model: False\n- cpu_ram_mb: 32088\n- use_gpu: True\n- num_gpus: 1\n- gpu: TITAN RTX\n- gpu_ram_mb: 24217\n- gpu_power_watts: 280.0\n- gpu_performance_state: 2\n- use_tpu: False\n```\n</tf>\n</frameworkcontent>\n\nAgain, _inference time_ and _required memory_ for _inference_ are measured, but this time for customized configurations\nof the `BertModel` class. This feature can especially be helpful when deciding for which configuration the model\nshould be trained.\n\n\n## Benchmark best practices\n\nThis section lists a couple of best practices one should be aware of when benchmarking a model.\n\n- Currently, only single device benchmarking is supported. When benchmarking on GPU, it is recommended that the user\n  specifies on which device the code should be run by setting the `CUDA_VISIBLE_DEVICES` environment variable in the\n  shell, _e.g._ `export CUDA_VISIBLE_DEVICES=0` before running the code.\n- The option `no_multi_processing` should only be set to `True` for testing and debugging. To ensure accurate\n  memory measurement it is recommended to run each memory benchmark in a separate process by making sure\n  `no_multi_processing` is set to `True`.\n- One should always state the environment information when sharing the results of a model benchmark. Results can vary\n  heavily between different GPU devices, library versions, etc., so that benchmark results on their own are not very\n  useful for the community."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: e3badd33-4ea7-49ab-9552-bf62b8310d94
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e3badd33-4ea7-49ab-9552-bf62b8310d94\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/benchmarks\", \"title\": \"Benchmarks\", \"tokens\": 4058, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e86b9ddf-25df-4bfa-a78c-696520de005a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/benchmarks\", \"title\": \"Benchmarks\", \"tokens\": 4058, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ff90393796b58d9cbff193200432df57f36db1694146fa624ccfa7fdd0c6151d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b0cceacd-4b5c-448b-841e-2485205eab4a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/benchmarks\", \"title\": \"Benchmarks\", \"tokens\": 4058, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b835e94333efbfd7b9239ec9ebc141d4a537ed1e533d702dcd9fae56f8b3a8d5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14833, \"end_char_idx\": 17380, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "doc_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "ref_doc_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "chroma:document": "6.10\n- system: Linux\n- cpu: x86_64\n- architecture: 64bit\n- date: 2020-06-29\n- time: 09:38:15.487125\n- fp16: False\n- use_multiprocessing: True\n- only_pretrain_model: False\n- cpu_ram_mb: 32088\n- use_gpu: True\n- num_gpus: 1\n- gpu: TITAN RTX\n- gpu_ram_mb: 24217\n- gpu_power_watts: 280.0\n- gpu_performance_state: 2\n- use_tpu: False\n```\n</tf>\n</frameworkcontent>\n\nAgain, _inference time_ and _required memory_ for _inference_ are measured, but this time for customized configurations\nof the `BertModel` class. This feature can especially be helpful when deciding for which configuration the model\nshould be trained.\n\n\n## Benchmark best practices\n\nThis section lists a couple of best practices one should be aware of when benchmarking a model.\n\n- Currently, only single device benchmarking is supported. When benchmarking on GPU, it is recommended that the user\n  specifies on which device the code should be run by setting the `CUDA_VISIBLE_DEVICES` environment variable in the\n  shell, _e.g._ `export CUDA_VISIBLE_DEVICES=0` before running the code.\n- The option `no_multi_processing` should only be set to `True` for testing and debugging. To ensure accurate\n  memory measurement it is recommended to run each memory benchmark in a separate process by making sure\n  `no_multi_processing` is set to `True`.\n- One should always state the environment information when sharing the results of a model benchmark. Results can vary\n  heavily between different GPU devices, library versions, etc., so that benchmark results on their own are not very\n  useful for the community.\n\n\n## Sharing your benchmark\n\nPreviously all available core models (10 at the time) have been benchmarked for _inference time_, across many different\nsettings: using PyTorch, with and without TorchScript, using TensorFlow, with and without XLA. All of those tests were\ndone across CPUs (except for TensorFlow XLA) and GPUs.\n\nThe approach is detailed in the [following blogpost](https://medium.com/huggingface/benchmarking-transformers-pytorch-and-tensorflow-e2917fb891c2) and the results are\navailable [here](https://docs.google.com/spreadsheets/d/1sryqufw2D0XlUH4sq3e9Wnxu5EAQkaohzrJbd5HdQ_w/edit?usp=sharing).\n\nWith the new _benchmark_ tools, it is easier than ever to share your benchmark results with the community\n\n- [PyTorch Benchmarking Results](https://github.com/huggingface/transformers/tree/main/examples/pytorch/benchmarking/README.md).\n- [TensorFlow Benchmarking Results](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/benchmarking/README.md)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 45e9b47f-20eb-41c2-b99a-d36ce82dab36
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"45e9b47f-20eb-41c2-b99a-d36ce82dab36\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"22362ff1-ecd5-41c9-aea9-7cf4a47309c0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7105c0bea103cb7406f4c99f1ab3aeea30205cb37afa9391fa30eb11c9905643\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4daf6054-b0da-4aa9-977e-bb859aeb7876\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bbcddc2816a45c1f29c7706411bccc322c54196b006e68c857e171482491db3a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3667, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "doc_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "ref_doc_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "chroma:document": "# Text generation strategies\n\nText generation is essential to many NLP tasks, such as open-ended text generation, summarization, translation, and\nmore. It also plays a role in a variety of mixed-modality applications that have text as an output like speech-to-text\nand vision-to-text. Some of the models that can generate text include\nGPT2, XLNet, OpenAI GPT, CTRL, TransformerXL, XLM, Bart, T5, GIT, Whisper.\n\nCheck out a few examples that use [`~generation.GenerationMixin.generate`] method to produce\ntext outputs for different tasks:\n* [Text summarization](./tasks/summarization#inference)\n* [Image captioning](./model_doc/git#transformers.GitForCausalLM.forward.example)\n* [Audio transcription](./model_doc/whisper#transformers.WhisperForConditionalGeneration.forward.example)\n\nNote that the inputs to the generate method depend on the model's modality. They are returned by the model's preprocessor\nclass, such as AutoTokenizer or AutoProcessor. If a model's preprocessor creates more than one kind of input, pass all\nthe inputs to generate(). You can learn more about the individual model's preprocessor in the corresponding model's documentation.\n\nThe process of selecting output tokens to generate text is known as decoding, and you can customize the decoding strategy\nthat the `generate()` method will use. Modifying a decoding strategy does not change the values of any trainable parameters.\nHowever, it can have a noticeable impact on the quality of the generated output. It can help reduce repetition in the text\nand make it more coherent.\n\nThis guide describes:\n* default generation configuration\n* common decoding strategies and their main parameters\n* saving and sharing custom generation configurations with your fine-tuned model on \ud83e\udd17 Hub\n\n## Default text generation configuration\n\nA decoding strategy for a model is defined in its generation configuration. When using pre-trained models for inference\nwithin a [`pipeline`], the models call the `PreTrainedModel.generate()` method that applies a default generation\nconfiguration under the hood. The default configuration is also used when no custom configuration has been saved with\nthe model.\n\nWhen you load a model explicitly, you can inspect the generation configuration that comes with it through\n `model.generation_config`:\n\n```python\n>>> from transformers import AutoModelForCausalLM\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"distilbert/distilgpt2\")\n>>> model.generation_config\nGenerationConfig {\n  \"bos_token_id\": 50256,\n  \"eos_token_id\": 50256\n}\n<BLANKLINE>\n```\n\nPrinting out the `model.generation_config` reveals only the values that are different from the default generation\nconfiguration, and does not list any of the default values.\n\nThe default generation configuration limits the size of the output combined with the input prompt to a maximum of 20\ntokens to avoid running into resource limitations. The default decoding strategy is greedy search, which is the simplest decoding strategy that picks a token with the highest probability as the next token. For many tasks\nand small output sizes this works well. However, when used to generate longer outputs, greedy search can start\nproducing highly repetitive results.\n\n## Customize text generation\n\nYou can override any `generation_config` by passing the parameters and their values directly to the [`generate`] method:\n\n```python\n>>> my_model.generate(**inputs, num_beams=4, do_sample=True)  # doctest: +SKIP\n```\n\nEven if the default decoding strategy mostly works for your task, you can still tweak a few things. Some of the\ncommonly adjusted parameters include:\n\n- `max_new_tokens`: the maximum number of tokens to generate."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 4daf6054-b0da-4aa9-977e-bb859aeb7876
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4daf6054-b0da-4aa9-977e-bb859aeb7876\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"22362ff1-ecd5-41c9-aea9-7cf4a47309c0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7105c0bea103cb7406f4c99f1ab3aeea30205cb37afa9391fa30eb11c9905643\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"45e9b47f-20eb-41c2-b99a-d36ce82dab36\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5ca2129e149d3f539cedc6f36752132d646a543716c7447c80ef5553a700be60\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c082615a-a757-439b-a5b9-5fef4bf9d1d5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"589d2b75895204c7e3738223b6eac7e08c7ecce7124b3ca22b6d360e5ff745d8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1875, \"end_char_idx\": 5159, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "doc_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "ref_doc_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "chroma:document": "When using pre-trained models for inference\nwithin a [`pipeline`], the models call the `PreTrainedModel.generate()` method that applies a default generation\nconfiguration under the hood. The default configuration is also used when no custom configuration has been saved with\nthe model.\n\nWhen you load a model explicitly, you can inspect the generation configuration that comes with it through\n `model.generation_config`:\n\n```python\n>>> from transformers import AutoModelForCausalLM\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"distilbert/distilgpt2\")\n>>> model.generation_config\nGenerationConfig {\n  \"bos_token_id\": 50256,\n  \"eos_token_id\": 50256\n}\n<BLANKLINE>\n```\n\nPrinting out the `model.generation_config` reveals only the values that are different from the default generation\nconfiguration, and does not list any of the default values.\n\nThe default generation configuration limits the size of the output combined with the input prompt to a maximum of 20\ntokens to avoid running into resource limitations. The default decoding strategy is greedy search, which is the simplest decoding strategy that picks a token with the highest probability as the next token. For many tasks\nand small output sizes this works well. However, when used to generate longer outputs, greedy search can start\nproducing highly repetitive results.\n\n## Customize text generation\n\nYou can override any `generation_config` by passing the parameters and their values directly to the [`generate`] method:\n\n```python\n>>> my_model.generate(**inputs, num_beams=4, do_sample=True)  # doctest: +SKIP\n```\n\nEven if the default decoding strategy mostly works for your task, you can still tweak a few things. Some of the\ncommonly adjusted parameters include:\n\n- `max_new_tokens`: the maximum number of tokens to generate. In other words, the size of the output sequence, not\nincluding the tokens in the prompt. As an alternative to using the output's length as a stopping criteria, you can choose\nto stop generation whenever the full generation exceeds some amount of time. To learn more, check [`StoppingCriteria`].\n- `num_beams`: by specifying a number of beams higher than 1, you are effectively switching from greedy search to\nbeam search. This strategy evaluates several hypotheses at each time step and eventually chooses the hypothesis that\nhas the overall highest probability for the entire sequence. This has the advantage of identifying high-probability\nsequences that start with a lower probability initial tokens and would've been ignored by the greedy search. Visualize how it works [here](https://huggingface.co/spaces/m-ric/beam_search_visualizer).\n- `do_sample`: if set to `True`, this parameter enables decoding strategies such as multinomial sampling, beam-search\nmultinomial sampling, Top-K sampling and Top-p sampling. All these strategies select the next token from the probability\ndistribution over the entire vocabulary with various strategy-specific adjustments.\n- `num_return_sequences`: the number of sequence candidates to return for each input. This option is only available for\nthe decoding strategies that support multiple sequence candidates, e.g. variations of beam search and sampling. Decoding\nstrategies like greedy search and contrastive search return a single output sequence."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: c082615a-a757-439b-a5b9-5fef4bf9d1d5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c082615a-a757-439b-a5b9-5fef4bf9d1d5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"22362ff1-ecd5-41c9-aea9-7cf4a47309c0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7105c0bea103cb7406f4c99f1ab3aeea30205cb37afa9391fa30eb11c9905643\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4daf6054-b0da-4aa9-977e-bb859aeb7876\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e2658a8fb9aa60cf2c5960d9df528b32fb59d177d8ad46c8f88ce9c8f658f533\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5f203877-338a-45c5-aa89-a8353a5d2b22\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2a359e86854b2eddc67707827f707b4e8b472aebcf268e6b1fc0aa8b1e807467\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3555, \"end_char_idx\": 6529, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "doc_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "ref_doc_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "chroma:document": "Some of the\ncommonly adjusted parameters include:\n\n- `max_new_tokens`: the maximum number of tokens to generate. In other words, the size of the output sequence, not\nincluding the tokens in the prompt. As an alternative to using the output's length as a stopping criteria, you can choose\nto stop generation whenever the full generation exceeds some amount of time. To learn more, check [`StoppingCriteria`].\n- `num_beams`: by specifying a number of beams higher than 1, you are effectively switching from greedy search to\nbeam search. This strategy evaluates several hypotheses at each time step and eventually chooses the hypothesis that\nhas the overall highest probability for the entire sequence. This has the advantage of identifying high-probability\nsequences that start with a lower probability initial tokens and would've been ignored by the greedy search. Visualize how it works [here](https://huggingface.co/spaces/m-ric/beam_search_visualizer).\n- `do_sample`: if set to `True`, this parameter enables decoding strategies such as multinomial sampling, beam-search\nmultinomial sampling, Top-K sampling and Top-p sampling. All these strategies select the next token from the probability\ndistribution over the entire vocabulary with various strategy-specific adjustments.\n- `num_return_sequences`: the number of sequence candidates to return for each input. This option is only available for\nthe decoding strategies that support multiple sequence candidates, e.g. variations of beam search and sampling. Decoding\nstrategies like greedy search and contrastive search return a single output sequence.\n\n## Save a custom decoding strategy with your model\n\nIf you would like to share your fine-tuned model with a specific generation configuration, you can:\n* Create a [`GenerationConfig`] class instance\n* Specify the decoding strategy parameters\n* Save your generation configuration with [`GenerationConfig.save_pretrained`], making sure to leave its `config_file_name` argument empty\n* Set `push_to_hub` to `True` to upload your config to the model's repo\n\n```python\n>>> from transformers import AutoModelForCausalLM, GenerationConfig\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"my_account/my_model\")  # doctest: +SKIP\n>>> generation_config = GenerationConfig(\n...     max_new_tokens=50, do_sample=True, top_k=50, eos_token_id=model.config.eos_token_id\n... )\n>>> generation_config.save_pretrained(\"my_account/my_model\", push_to_hub=True)  # doctest: +SKIP\n```\n\nYou can also store several generation configurations in a single directory, making use of the `config_file_name`\nargument in [`GenerationConfig.save_pretrained`]. You can later instantiate them with [`GenerationConfig.from_pretrained`]. This is useful if you want to\nstore several generation configurations for a single model (e.g. one for creative text generation with sampling, and\none for summarization with beam search). You must have the right Hub permissions to add configuration files to a model."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 5f203877-338a-45c5-aa89-a8353a5d2b22
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5f203877-338a-45c5-aa89-a8353a5d2b22\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"22362ff1-ecd5-41c9-aea9-7cf4a47309c0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7105c0bea103cb7406f4c99f1ab3aeea30205cb37afa9391fa30eb11c9905643\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c082615a-a757-439b-a5b9-5fef4bf9d1d5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"36b48f174447db03db9f3024529790452f2c607200c65d86af5e2e47e9b7c3b2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"57a70f50-7157-4b33-9fe9-071b5eded0a9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4ec15013714f2607c0735fb923ad0b848ace2d3d13efb92ca8865d83933aeb8d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4685, \"end_char_idx\": 8227, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "doc_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "ref_doc_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "chroma:document": "All these strategies select the next token from the probability\ndistribution over the entire vocabulary with various strategy-specific adjustments.\n- `num_return_sequences`: the number of sequence candidates to return for each input. This option is only available for\nthe decoding strategies that support multiple sequence candidates, e.g. variations of beam search and sampling. Decoding\nstrategies like greedy search and contrastive search return a single output sequence.\n\n## Save a custom decoding strategy with your model\n\nIf you would like to share your fine-tuned model with a specific generation configuration, you can:\n* Create a [`GenerationConfig`] class instance\n* Specify the decoding strategy parameters\n* Save your generation configuration with [`GenerationConfig.save_pretrained`], making sure to leave its `config_file_name` argument empty\n* Set `push_to_hub` to `True` to upload your config to the model's repo\n\n```python\n>>> from transformers import AutoModelForCausalLM, GenerationConfig\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"my_account/my_model\")  # doctest: +SKIP\n>>> generation_config = GenerationConfig(\n...     max_new_tokens=50, do_sample=True, top_k=50, eos_token_id=model.config.eos_token_id\n... )\n>>> generation_config.save_pretrained(\"my_account/my_model\", push_to_hub=True)  # doctest: +SKIP\n```\n\nYou can also store several generation configurations in a single directory, making use of the `config_file_name`\nargument in [`GenerationConfig.save_pretrained`]. You can later instantiate them with [`GenerationConfig.from_pretrained`]. This is useful if you want to\nstore several generation configurations for a single model (e.g. one for creative text generation with sampling, and\none for summarization with beam search). You must have the right Hub permissions to add configuration files to a model.\n\n```python\n>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, GenerationConfig\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"google-t5/t5-small\")\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"google-t5/t5-small\")\n\n>>> translation_generation_config = GenerationConfig(\n...     num_beams=4,\n...     early_stopping=True,\n...     decoder_start_token_id=0,\n...     eos_token_id=model.config.eos_token_id,\n...     pad_token=model.config.pad_token_id,\n... )\n\n>>> # Tip: add `push_to_hub=True` to push to the Hub\n>>> translation_generation_config.save_pretrained(\"/tmp\", \"translation_generation_config.json\")\n\n>>> # You could then use the named generation config file to parameterize generation\n>>> generation_config = GenerationConfig.from_pretrained(\"/tmp\", \"translation_generation_config.json\")\n>>> inputs = tokenizer(\"translate English to French: Configuration files are easy to use!\", return_tensors=\"pt\")\n>>> outputs = model.generate(**inputs, generation_config=generation_config)\n>>> print(tokenizer.batch_decode(outputs, skip_special_tokens=True))\n['Les fichiers de configuration sont faciles \u00e0 utiliser!']\n```\n\n## Streaming\n\nThe `generate()` supports streaming, through its `streamer` input. The `streamer` input is compatible with any instance\nfrom a class that has the following methods: `put()` and `end()`. Internally, `put()` is used to push new tokens and\n`end()` is used to flag the end of text generation.\n\n<Tip warning={true}>\n\nThe API for the streamer classes is still under development and may change in the future.\n\n</Tip>\n\nIn practice, you can craft your own streaming class for all sorts of purposes! We also have basic streaming classes\nready for you to use."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 57a70f50-7157-4b33-9fe9-071b5eded0a9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"57a70f50-7157-4b33-9fe9-071b5eded0a9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"22362ff1-ecd5-41c9-aea9-7cf4a47309c0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7105c0bea103cb7406f4c99f1ab3aeea30205cb37afa9391fa30eb11c9905643\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5f203877-338a-45c5-aa89-a8353a5d2b22\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9a859142ff11d1371922a2d8f554e451128b326f47027b1403b2e37146411b64\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"59498d4d-3ba5-4d1f-942b-521e8527b33e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e077e9100f770b81f83830aaa7e04b0167fde32b493306162fa2a6adabdbc4c8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6451, \"end_char_idx\": 8957, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "doc_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "ref_doc_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "chroma:document": "You must have the right Hub permissions to add configuration files to a model.\n\n```python\n>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, GenerationConfig\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"google-t5/t5-small\")\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(\"google-t5/t5-small\")\n\n>>> translation_generation_config = GenerationConfig(\n...     num_beams=4,\n...     early_stopping=True,\n...     decoder_start_token_id=0,\n...     eos_token_id=model.config.eos_token_id,\n...     pad_token=model.config.pad_token_id,\n... )\n\n>>> # Tip: add `push_to_hub=True` to push to the Hub\n>>> translation_generation_config.save_pretrained(\"/tmp\", \"translation_generation_config.json\")\n\n>>> # You could then use the named generation config file to parameterize generation\n>>> generation_config = GenerationConfig.from_pretrained(\"/tmp\", \"translation_generation_config.json\")\n>>> inputs = tokenizer(\"translate English to French: Configuration files are easy to use!\", return_tensors=\"pt\")\n>>> outputs = model.generate(**inputs, generation_config=generation_config)\n>>> print(tokenizer.batch_decode(outputs, skip_special_tokens=True))\n['Les fichiers de configuration sont faciles \u00e0 utiliser!']\n```\n\n## Streaming\n\nThe `generate()` supports streaming, through its `streamer` input. The `streamer` input is compatible with any instance\nfrom a class that has the following methods: `put()` and `end()`. Internally, `put()` is used to push new tokens and\n`end()` is used to flag the end of text generation.\n\n<Tip warning={true}>\n\nThe API for the streamer classes is still under development and may change in the future.\n\n</Tip>\n\nIn practice, you can craft your own streaming class for all sorts of purposes! We also have basic streaming classes\nready for you to use. For example, you can use the [`TextStreamer`] class to stream the output of `generate()` into\nyour screen, one word at a time:\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer\n\n>>> tok = AutoTokenizer.from_pretrained(\"openai-community/gpt2\")\n>>> model = AutoModelForCausalLM.from_pretrained(\"openai-community/gpt2\")\n>>> inputs = tok([\"An increasing sequence: one,\"], return_tensors=\"pt\")\n>>> streamer = TextStreamer(tok)\n\n>>> # Despite returning the usual output, the streamer will also print the generated text to stdout.\n>>> _ = model.generate(**inputs, streamer=streamer, max_new_tokens=20)\nAn increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven,\n```"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 59498d4d-3ba5-4d1f-942b-521e8527b33e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"59498d4d-3ba5-4d1f-942b-521e8527b33e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"22362ff1-ecd5-41c9-aea9-7cf4a47309c0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7105c0bea103cb7406f4c99f1ab3aeea30205cb37afa9391fa30eb11c9905643\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"57a70f50-7157-4b33-9fe9-071b5eded0a9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f32e449951b1c4926a433b293e8f976b50b15a30f2cdb225a1f3a620645d8e2b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ba68b508-6691-402d-953c-bd2f6c6feeb9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f2720d9be69d13a6b1cf7570bac21675732bc99b960d4cb198f6a979b2625f4f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7432, \"end_char_idx\": 10213, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "doc_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "ref_doc_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "chroma:document": "\", return_tensors=\"pt\")\n>>> outputs = model.generate(**inputs, generation_config=generation_config)\n>>> print(tokenizer.batch_decode(outputs, skip_special_tokens=True))\n['Les fichiers de configuration sont faciles \u00e0 utiliser!']\n```\n\n## Streaming\n\nThe `generate()` supports streaming, through its `streamer` input. The `streamer` input is compatible with any instance\nfrom a class that has the following methods: `put()` and `end()`. Internally, `put()` is used to push new tokens and\n`end()` is used to flag the end of text generation.\n\n<Tip warning={true}>\n\nThe API for the streamer classes is still under development and may change in the future.\n\n</Tip>\n\nIn practice, you can craft your own streaming class for all sorts of purposes! We also have basic streaming classes\nready for you to use. For example, you can use the [`TextStreamer`] class to stream the output of `generate()` into\nyour screen, one word at a time:\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer\n\n>>> tok = AutoTokenizer.from_pretrained(\"openai-community/gpt2\")\n>>> model = AutoModelForCausalLM.from_pretrained(\"openai-community/gpt2\")\n>>> inputs = tok([\"An increasing sequence: one,\"], return_tensors=\"pt\")\n>>> streamer = TextStreamer(tok)\n\n>>> # Despite returning the usual output, the streamer will also print the generated text to stdout.\n>>> _ = model.generate(**inputs, streamer=streamer, max_new_tokens=20)\nAn increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven,\n```\n\n\n## KV Cache Quantization\n\nThe `generate()` method supports caching keys and values to enhance efficiency and avoid re-computations. However the key and value\ncache can occupy a large portion of memory, becoming a bottleneck for long-context generation, especially for Large Language Models.\nQuantizing the cache when using `generate()` can significantly reduce memory requirements at the cost of speed.\n\nKV Cache quantization in `transformers` is largely inspired by the paper [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache]\n(https://arxiv.org/abs/2402.02750) and currently supports `quanto` and `HQQ` as backends. For more information on the inner workings see the paper.\n\nTo enable quantization of the key-value cache, one needs to indicate `cache_implementation=\"quantized\"` in the `generation_config`.\nQuantization related arguments should be passed to the `generation_config` either as a `dict` or an instance of a [`QuantizedCacheConfig`] class.\nOne has to indicate which quantization backend to use in the [`QuantizedCacheConfig`], the default is `quanto`.\n\n<Tip warning={true}>\n\nCache quantization can be detrimental if the context length is short and there is enough GPU VRAM available to run without cache quantization.\n\n</Tip>"}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: ba68b508-6691-402d-953c-bd2f6c6feeb9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ba68b508-6691-402d-953c-bd2f6c6feeb9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"22362ff1-ecd5-41c9-aea9-7cf4a47309c0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7105c0bea103cb7406f4c99f1ab3aeea30205cb37afa9391fa30eb11c9905643\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"59498d4d-3ba5-4d1f-942b-521e8527b33e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"41a8d15aadec7cbcca7f2117363834246aa67567cc76ae933b14213d47aca0d1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"49a87587-a306-4788-b2d1-8a1bce99981e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"25083112b6bb76ec99a198b1451e66c6ab55849e8252d4e411b98063ca6243c9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8792, \"end_char_idx\": 12246, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "doc_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "ref_doc_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "chroma:document": ">>> _ = model.generate(**inputs, streamer=streamer, max_new_tokens=20)\nAn increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven,\n```\n\n\n## KV Cache Quantization\n\nThe `generate()` method supports caching keys and values to enhance efficiency and avoid re-computations. However the key and value\ncache can occupy a large portion of memory, becoming a bottleneck for long-context generation, especially for Large Language Models.\nQuantizing the cache when using `generate()` can significantly reduce memory requirements at the cost of speed.\n\nKV Cache quantization in `transformers` is largely inspired by the paper [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache]\n(https://arxiv.org/abs/2402.02750) and currently supports `quanto` and `HQQ` as backends. For more information on the inner workings see the paper.\n\nTo enable quantization of the key-value cache, one needs to indicate `cache_implementation=\"quantized\"` in the `generation_config`.\nQuantization related arguments should be passed to the `generation_config` either as a `dict` or an instance of a [`QuantizedCacheConfig`] class.\nOne has to indicate which quantization backend to use in the [`QuantizedCacheConfig`], the default is `quanto`.\n\n<Tip warning={true}>\n\nCache quantization can be detrimental if the context length is short and there is enough GPU VRAM available to run without cache quantization.\n\n</Tip>\n\n\n```python\n>>> import torch\n>>> from transformers import AutoTokenizer, AutoModelForCausalLM\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"meta-llama/Llama-2-7b-chat-hf\")\n>>> model = AutoModelForCausalLM.from_pretrained(\"meta-llama/Llama-2-7b-chat-hf\", torch_dtype=torch.float16).to(\"cuda:0\")\n>>> inputs = tokenizer(\"I like rock music because\", return_tensors=\"pt\").to(model.device)\n\n>>> out = model.generate(**inputs, do_sample=False, max_new_tokens=20, cache_implementation=\"quantized\", cache_config={\"nbits\": 4, \"backend\": \"quanto\"})\n>>> print(tokenizer.batch_decode(out, skip_special_tokens=True)[0])\nI like rock music because it's loud and energetic. It's a great way to express myself and rel\n\n>>> out = model.generate(**inputs, do_sample=False, max_new_tokens=20)\n>>> print(tokenizer.batch_decode(out, skip_special_tokens=True)[0])\nI like rock music because it's loud and energetic. I like to listen to it when I'm feeling\n```\n\n## Watermarking\n\nThe `generate()` supports watermarking the generated text by randomly marking a portion of tokens as \"green\".\nWhen generating the \"green\" will have a small 'bias' value added to their logits, thus having a higher chance to be generated.\nThe watermarked text can be detected by calculating the proportion of \"green\" tokens in the text and estimating how likely it is\nstatistically to obtain that amount of \"green\" tokens for human-generated text. This watermarking strategy was proposed in the paper\n[\"On the Reliability of Watermarks for Large Language Models\"](https://arxiv.org/abs/2306.04634). For more information on\nthe inner functioning of watermarking, it is recommended to refer to the paper.\n\nThe watermarking can be used with any generative model in `tranformers` and does not require an extra classification model\nto detect watermarked text. To trigger watermarking, pass in a [`WatermarkingConfig`] with needed arguments directly to the\n`.generate()` method or add it to the [`GenerationConfig`]. Watermarked text can be later detected with a [`WatermarkDetector`]."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 49a87587-a306-4788-b2d1-8a1bce99981e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"49a87587-a306-4788-b2d1-8a1bce99981e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"22362ff1-ecd5-41c9-aea9-7cf4a47309c0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7105c0bea103cb7406f4c99f1ab3aeea30205cb37afa9391fa30eb11c9905643\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ba68b508-6691-402d-953c-bd2f6c6feeb9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"45d3b99da9b184a4a2e0d52a036c72cbfc7efb5efa5b4cbe44627d9315304499\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2ac8b3ee-3d94-4047-8fc2-e64120affa63\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"97b9cfbad60d4037cbb7432f80af2910933af554debdb850ef7d79c83ba4d936\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12249, \"end_char_idx\": 15532, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "doc_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "ref_doc_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "chroma:document": "<Tip warning={true}>\n\nThe WatermarkDetector internally relies on the proportion of \"green\" tokens, and whether generated text follows the coloring pattern.\nThat is why it is recommended to strip off the prompt text, if it is much longer than the generated text.\nThis also can have an effect when one sequence in the batch is a lot longer causing other rows to be padded.\nAdditionally, the detector **must** be initiated with identical watermark configuration arguments used when generating.\n\n</Tip>\n\nLet's generate some text with watermarking. In the below code snippet, we set the bias to 2.5 which is a value that\nwill be added to \"green\" tokens' logits. After generating watermarked text, we can pass it directly to the `WatermarkDetector`\nto check if the text is machine-generated (outputs `True` for machine-generated and `False` otherwise).\n\n```python\n>>> from transformers import AutoTokenizer, AutoModelForCausalLM, WatermarkDetector, WatermarkingConfig\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"openai-community/gpt2\")\n>>> tok = AutoTokenizer.from_pretrained(\"openai-community/gpt2\")\n>>> tok.pad_token_id = tok.eos_token_id\n>>> tok.padding_side = \"left\"\n\n>>> inputs = tok([\"This is the beginning of a long story\", \"Alice and Bob are\"], padding=True, return_tensors=\"pt\")\n>>> input_len = inputs[\"input_ids\"].shape[-1]\n\n>>> watermarking_config = WatermarkingConfig(bias=2.5, seeding_scheme=\"selfhash\")\n>>> out = model.generate(**inputs, watermarking_config=watermarking_config, do_sample=False, max_length=20)\n\n>>> detector = WatermarkDetector(model_config=model.config, device=\"cpu\", watermarking_config=watermarking_config)\n>>> detection_out = detector(out, return_dict=True)\n>>> detection_out.prediction\narray([True, True])\n```\n\n\n## Decoding strategies\n\nCertain combinations of the `generate()` parameters, and ultimately `generation_config`, can be used to enable specific\ndecoding strategies. If you are new to this concept, we recommend reading [this blog post that illustrates how common decoding strategies work](https://huggingface.co/blog/how-to-generate).\n\nHere, we'll show some of the parameters that control the decoding strategies and illustrate how you can use them.\n\n### Greedy Search\n\n[`generate`] uses greedy search decoding by default so you don't have to pass any parameters to enable it. This means the parameters `num_beams` is set to 1 and `do_sample=False`.\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> prompt = \"I look forward to\"\n>>> checkpoint = \"distilbert/distilgpt2\"\n\n>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)\n>>> inputs = tokenizer(prompt, return_tensors=\"pt\")\n\n>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)\n>>> outputs = model.generate(**inputs)\n>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)\n['I look forward to seeing you all again!\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n']\n```\n\n### Contrastive search\n\nThe contrastive search decoding strategy was proposed in the 2022 paper [A Contrastive Framework for Neural Text Generation](https://arxiv.org/abs/2202.06417).\nIt demonstrates superior results for generating non-repetitive yet coherent long outputs. To learn how contrastive search\nworks, check out [this blog post](https://huggingface.co/blog/introducing-csearch)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 2ac8b3ee-3d94-4047-8fc2-e64120affa63
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2ac8b3ee-3d94-4047-8fc2-e64120affa63\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"22362ff1-ecd5-41c9-aea9-7cf4a47309c0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7105c0bea103cb7406f4c99f1ab3aeea30205cb37afa9391fa30eb11c9905643\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"49a87587-a306-4788-b2d1-8a1bce99981e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c035c0c299e752dacd50d0e77a2d9cd60fc9ac09b518072afcdfceeb5b17f567\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"de9e8122-e63b-4b21-9264-9a7eb7098276\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f794ae1b220d767481bbd5832536252890ce56086c9389688f1b1dd2155d49b4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13998, \"end_char_idx\": 16919, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "doc_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "ref_doc_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "chroma:document": "## Decoding strategies\n\nCertain combinations of the `generate()` parameters, and ultimately `generation_config`, can be used to enable specific\ndecoding strategies. If you are new to this concept, we recommend reading [this blog post that illustrates how common decoding strategies work](https://huggingface.co/blog/how-to-generate).\n\nHere, we'll show some of the parameters that control the decoding strategies and illustrate how you can use them.\n\n### Greedy Search\n\n[`generate`] uses greedy search decoding by default so you don't have to pass any parameters to enable it. This means the parameters `num_beams` is set to 1 and `do_sample=False`.\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> prompt = \"I look forward to\"\n>>> checkpoint = \"distilbert/distilgpt2\"\n\n>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)\n>>> inputs = tokenizer(prompt, return_tensors=\"pt\")\n\n>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)\n>>> outputs = model.generate(**inputs)\n>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)\n['I look forward to seeing you all again!\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n']\n```\n\n### Contrastive search\n\nThe contrastive search decoding strategy was proposed in the 2022 paper [A Contrastive Framework for Neural Text Generation](https://arxiv.org/abs/2202.06417).\nIt demonstrates superior results for generating non-repetitive yet coherent long outputs. To learn how contrastive search\nworks, check out [this blog post](https://huggingface.co/blog/introducing-csearch).\nThe two main parameters that enable and control the behavior of contrastive search are `penalty_alpha` and `top_k`:\n\n```python\n>>> from transformers import AutoTokenizer, AutoModelForCausalLM\n\n>>> checkpoint = \"openai-community/gpt2-large\"\n>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)\n>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)\n\n>>> prompt = \"Hugging Face Company is\"\n>>> inputs = tokenizer(prompt, return_tensors=\"pt\")\n\n>>> outputs = model.generate(**inputs, penalty_alpha=0.6, top_k=4, max_new_tokens=100)\n>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)\n['Hugging Face Company is a family owned and operated business. We pride ourselves on being the best\nin the business and our customer service is second to none.\\n\\nIf you have any questions about our\nproducts or services, feel free to contact us at any time. We look forward to hearing from you!']\n```\n\n### Multinomial sampling\n\nAs opposed to greedy search that always chooses a token with the highest probability as the\nnext token, multinomial sampling (also called ancestral sampling) randomly selects the next token based on the probability distribution over the entire\nvocabulary given by the model. Every token with a non-zero probability has a chance of being selected, thus reducing the\nrisk of repetition.\n\nTo enable multinomial sampling set `do_sample=True` and `num_beams=1`."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: de9e8122-e63b-4b21-9264-9a7eb7098276
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"de9e8122-e63b-4b21-9264-9a7eb7098276\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"22362ff1-ecd5-41c9-aea9-7cf4a47309c0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7105c0bea103cb7406f4c99f1ab3aeea30205cb37afa9391fa30eb11c9905643\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2ac8b3ee-3d94-4047-8fc2-e64120affa63\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9229d3899d77380bc21baefed58e986c669566d3dd48f749836d9b1d93ccbeda\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5cccf3f8-a8fc-4425-8be3-5317a81862ef\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8210bcfb1f111f467e38e5c1420a1363697456b6c481598ec3ab4ad4e2b777d1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15327, \"end_char_idx\": 18855, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "doc_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "ref_doc_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "chroma:document": "It demonstrates superior results for generating non-repetitive yet coherent long outputs. To learn how contrastive search\nworks, check out [this blog post](https://huggingface.co/blog/introducing-csearch).\nThe two main parameters that enable and control the behavior of contrastive search are `penalty_alpha` and `top_k`:\n\n```python\n>>> from transformers import AutoTokenizer, AutoModelForCausalLM\n\n>>> checkpoint = \"openai-community/gpt2-large\"\n>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)\n>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)\n\n>>> prompt = \"Hugging Face Company is\"\n>>> inputs = tokenizer(prompt, return_tensors=\"pt\")\n\n>>> outputs = model.generate(**inputs, penalty_alpha=0.6, top_k=4, max_new_tokens=100)\n>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)\n['Hugging Face Company is a family owned and operated business. We pride ourselves on being the best\nin the business and our customer service is second to none.\\n\\nIf you have any questions about our\nproducts or services, feel free to contact us at any time. We look forward to hearing from you!']\n```\n\n### Multinomial sampling\n\nAs opposed to greedy search that always chooses a token with the highest probability as the\nnext token, multinomial sampling (also called ancestral sampling) randomly selects the next token based on the probability distribution over the entire\nvocabulary given by the model. Every token with a non-zero probability has a chance of being selected, thus reducing the\nrisk of repetition.\n\nTo enable multinomial sampling set `do_sample=True` and `num_beams=1`.\n\n```python\n>>> from transformers import AutoTokenizer, AutoModelForCausalLM, set_seed\n>>> set_seed(0)  # For reproducibility\n\n>>> checkpoint = \"openai-community/gpt2-large\"\n>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)\n>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)\n\n>>> prompt = \"Today was an amazing day because\"\n>>> inputs = tokenizer(prompt, return_tensors=\"pt\")\n\n>>> outputs = model.generate(**inputs, do_sample=True, num_beams=1, max_new_tokens=100)\n>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)\n[\"Today was an amazing day because we received these wonderful items by the way of a gift shop. The box arrived on a Thursday and I opened it on Monday afternoon to receive the gifts. Both bags featured pieces from all the previous years!\\n\\nThe box had lots of surprises in it, including some sweet little mini chocolate chips! I don't think I'd eat all of these. This was definitely one of the most expensive presents I have ever got, I actually got most of them for free!\\n\\nThe first package came\"]\n```\n\n### Beam-search decoding\n\nUnlike greedy search, beam-search decoding keeps several hypotheses at each time step and eventually chooses\nthe hypothesis that has the overall highest probability for the entire sequence. This has the advantage of identifying high-probability\nsequences that start with lower probability initial tokens and would've been ignored by the greedy search.\n\n<a href=\"https://huggingface.co/spaces/m-ric/beam_search_visualizer\" class=\"flex flex-col justify-center\">\n    <img style=\"max-width: 90%; margin: auto;\" src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/beam_search.png\"/>\n</a>\n\nYou can visualize how beam-search decoding works in [this interactive demo](https://huggingface.co/spaces/m-ric/beam_search_visualizer): type your input sentence, and play with the parameters to see how the decoding beams change."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 5cccf3f8-a8fc-4425-8be3-5317a81862ef
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5cccf3f8-a8fc-4425-8be3-5317a81862ef\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"22362ff1-ecd5-41c9-aea9-7cf4a47309c0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7105c0bea103cb7406f4c99f1ab3aeea30205cb37afa9391fa30eb11c9905643\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"de9e8122-e63b-4b21-9264-9a7eb7098276\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"77cf6f3f36471c9b24f16b57fc9befc059a8dc512ad82907507423151f288c7f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2db47660-07f7-4308-b634-3266f35fa7e2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4190f9d3f54bf2362cfe9ac5587ed94c2728f22e72b460924e63af6275b16c81\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 17560, \"end_char_idx\": 20940, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "doc_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "ref_doc_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "chroma:document": "The box arrived on a Thursday and I opened it on Monday afternoon to receive the gifts. Both bags featured pieces from all the previous years!\\n\\nThe box had lots of surprises in it, including some sweet little mini chocolate chips! I don't think I'd eat all of these. This was definitely one of the most expensive presents I have ever got, I actually got most of them for free!\\n\\nThe first package came\"]\n```\n\n### Beam-search decoding\n\nUnlike greedy search, beam-search decoding keeps several hypotheses at each time step and eventually chooses\nthe hypothesis that has the overall highest probability for the entire sequence. This has the advantage of identifying high-probability\nsequences that start with lower probability initial tokens and would've been ignored by the greedy search.\n\n<a href=\"https://huggingface.co/spaces/m-ric/beam_search_visualizer\" class=\"flex flex-col justify-center\">\n    <img style=\"max-width: 90%; margin: auto;\" src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/beam_search.png\"/>\n</a>\n\nYou can visualize how beam-search decoding works in [this interactive demo](https://huggingface.co/spaces/m-ric/beam_search_visualizer): type your input sentence, and play with the parameters to see how the decoding beams change.\n\nTo enable this decoding strategy, specify the `num_beams` (aka number of hypotheses to keep track of) that is greater than 1.\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> prompt = \"It is astonishing how one can\"\n>>> checkpoint = \"openai-community/gpt2-medium\"\n\n>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)\n>>> inputs = tokenizer(prompt, return_tensors=\"pt\")\n\n>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)\n\n>>> outputs = model.generate(**inputs, num_beams=5, max_new_tokens=50)\n>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)\n['It is astonishing how one can have such a profound impact on the lives of so many people in such a short period of\ntime.\"\\n\\nHe added: \"I am very proud of the work I have been able to do in the last few years.\\n\\n\"I have']\n```\n\n### Beam-search multinomial sampling\n\nAs the name implies, this decoding strategy combines beam search with multinomial sampling. You need to specify\nthe `num_beams` greater than 1, and set `do_sample=True` to use this decoding strategy.\n\n```python\n>>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, set_seed\n>>> set_seed(0)  # For reproducibility\n\n>>> prompt = \"translate English to German: The house is wonderful.\"\n>>> checkpoint = \"google-t5/t5-small\"\n\n>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)\n>>> inputs = tokenizer(prompt, return_tensors=\"pt\")\n\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint)\n\n>>> outputs = model.generate(**inputs, num_beams=5, do_sample=True)\n>>> tokenizer.decode(outputs[0], skip_special_tokens=True)\n'Das Haus ist wunderbar.'\n```\n\n### Diverse beam search decoding\n\nThe diverse beam search decoding strategy is an extension of the beam search strategy that allows for generating a more diverse\nset of beam sequences to choose from. To learn how it works, refer to [Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models](https://arxiv.org/pdf/1610.02424.pdf).\nThis approach has three main parameters: `num_beams`, `num_beam_groups`, and `diversity_penalty`."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 2db47660-07f7-4308-b634-3266f35fa7e2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2db47660-07f7-4308-b634-3266f35fa7e2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"22362ff1-ecd5-41c9-aea9-7cf4a47309c0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7105c0bea103cb7406f4c99f1ab3aeea30205cb37afa9391fa30eb11c9905643\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5cccf3f8-a8fc-4425-8be3-5317a81862ef\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cfcf09b69d504af21b10eb58c1fbf4e9949447e433244af67c3d0631706a20f3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0660fba0-30df-43f7-9031-c9a12d47e4a1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c50688cbc1c4a812083d356a11b1b9c4b2b46d1aedc7c403f4018f7d091dec5b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 19582, \"end_char_idx\": 23060, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "doc_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "ref_doc_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "chroma:document": "\"\\n\\nHe added: \"I am very proud of the work I have been able to do in the last few years.\\n\\n\"I have']\n```\n\n### Beam-search multinomial sampling\n\nAs the name implies, this decoding strategy combines beam search with multinomial sampling. You need to specify\nthe `num_beams` greater than 1, and set `do_sample=True` to use this decoding strategy.\n\n```python\n>>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, set_seed\n>>> set_seed(0)  # For reproducibility\n\n>>> prompt = \"translate English to German: The house is wonderful.\"\n>>> checkpoint = \"google-t5/t5-small\"\n\n>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)\n>>> inputs = tokenizer(prompt, return_tensors=\"pt\")\n\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint)\n\n>>> outputs = model.generate(**inputs, num_beams=5, do_sample=True)\n>>> tokenizer.decode(outputs[0], skip_special_tokens=True)\n'Das Haus ist wunderbar.'\n```\n\n### Diverse beam search decoding\n\nThe diverse beam search decoding strategy is an extension of the beam search strategy that allows for generating a more diverse\nset of beam sequences to choose from. To learn how it works, refer to [Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models](https://arxiv.org/pdf/1610.02424.pdf).\nThis approach has three main parameters: `num_beams`, `num_beam_groups`, and `diversity_penalty`.\nThe diversity penalty ensures the outputs are distinct across groups, and beam search is used within each group.\n\n\n```python\n>>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM\n\n>>> checkpoint = \"google/pegasus-xsum\"\n>>> prompt = (\n...     \"The Permaculture Design Principles are a set of universal design principles \"\n...     \"that can be applied to any location, climate and culture, and they allow us to design \"\n...     \"the most efficient and sustainable human habitation and food production systems. \"\n...     \"Permaculture is a design system that encompasses a wide variety of disciplines, such \"\n...     \"as ecology, landscape design, environmental science and energy conservation, and the \"\n...     \"Permaculture design principles are drawn from these various disciplines. Each individual \"\n...     \"design principle itself embodies a complete conceptual framework based on sound \"\n...     \"scientific principles. When we bring all these separate  principles together, we can \"\n...     \"create a design system that both looks at whole systems, the parts that these systems \"\n...     \"consist of, and how those parts interact with each other to create a complex, dynamic, \"\n...     \"living system. Each design principle serves as a tool that allows us to integrate all \"\n...     \"the separate parts of a design, referred to as elements, into a functional, synergistic, \"\n...     \"whole system, where the elements harmoniously interact and work together in the most \"\n...     \"efficient way possible.\"\n... )\n\n>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)\n>>> inputs = tokenizer(prompt, return_tensors=\"pt\")\n\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint)\n\n>>> outputs = model.generate(**inputs, num_beams=5, num_beam_groups=5, max_new_tokens=30, diversity_penalty=1.0)\n>>> tokenizer.decode(outputs[0], skip_special_tokens=True)\n'The Design Principles are a set of universal design principles that can be applied to any location, climate and\nculture, and they allow us to design the'\n```\n\nThis guide illustrates the main parameters that enable various decoding strategies."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 0660fba0-30df-43f7-9031-c9a12d47e4a1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0660fba0-30df-43f7-9031-c9a12d47e4a1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"22362ff1-ecd5-41c9-aea9-7cf4a47309c0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7105c0bea103cb7406f4c99f1ab3aeea30205cb37afa9391fa30eb11c9905643\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2db47660-07f7-4308-b634-3266f35fa7e2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"98e104e3a805ab9f3e3f55b18ba53a69aa31167891caafd9348036d40abea0ae\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"be5523fe-8658-42f8-bdc1-103b31b945b1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4ddefa557c7c4b62d69e8b25ba956e9f97039e80b2ca8f01654ee97892a2993b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 21463, \"end_char_idx\": 25105, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "doc_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "ref_doc_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "chroma:document": "...     \"Permaculture is a design system that encompasses a wide variety of disciplines, such \"\n...     \"as ecology, landscape design, environmental science and energy conservation, and the \"\n...     \"Permaculture design principles are drawn from these various disciplines. Each individual \"\n...     \"design principle itself embodies a complete conceptual framework based on sound \"\n...     \"scientific principles. When we bring all these separate  principles together, we can \"\n...     \"create a design system that both looks at whole systems, the parts that these systems \"\n...     \"consist of, and how those parts interact with each other to create a complex, dynamic, \"\n...     \"living system. Each design principle serves as a tool that allows us to integrate all \"\n...     \"the separate parts of a design, referred to as elements, into a functional, synergistic, \"\n...     \"whole system, where the elements harmoniously interact and work together in the most \"\n...     \"efficient way possible.\"\n... )\n\n>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)\n>>> inputs = tokenizer(prompt, return_tensors=\"pt\")\n\n>>> model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint)\n\n>>> outputs = model.generate(**inputs, num_beams=5, num_beam_groups=5, max_new_tokens=30, diversity_penalty=1.0)\n>>> tokenizer.decode(outputs[0], skip_special_tokens=True)\n'The Design Principles are a set of universal design principles that can be applied to any location, climate and\nculture, and they allow us to design the'\n```\n\nThis guide illustrates the main parameters that enable various decoding strategies. More advanced parameters exist for the\n[`generate`] method, which gives you even further control over the [`generate`] method's behavior.\nFor the complete list of the available parameters, refer to the [API documentation](./main_classes/text_generation.md).\n\n### Speculative Decoding\n\nSpeculative decoding (also known as assisted decoding) is a modification of the decoding strategies above, that uses an\nassistant model (ideally a much smaller one) with the same tokenizer, to generate a few candidate tokens. The main\nmodel then validates the candidate tokens in a single forward pass, which speeds up the decoding process. If\n`do_sample=True`, then the token validation with resampling introduced in the\n[speculative decoding paper](https://arxiv.org/pdf/2211.17192.pdf) is used.\n\nCurrently, only greedy search and sampling are supported with assisted decoding, and assisted decoding doesn't support batched inputs.\nTo learn more about assisted decoding, check [this blog post](https://huggingface.co/blog/assisted-generation).\n\nTo enable assisted decoding, set the `assistant_model` argument with a model.\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> prompt = \"Alice and Bob\"\n>>> checkpoint = \"EleutherAI/pythia-1.4b-deduped\"\n>>> assistant_checkpoint = \"EleutherAI/pythia-160m-deduped\"\n\n>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)\n>>> inputs = tokenizer(prompt, return_tensors=\"pt\")\n\n>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)\n>>> assistant_model = AutoModelForCausalLM.from_pretrained(assistant_checkpoint)\n>>> outputs = model.generate(**inputs, assistant_model=assistant_model)\n>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)\n['Alice and Bob are sitting in a bar. Alice is drinking a beer and Bob is drinking a']\n```\n\nWhen using assisted decoding with sampling methods, you can use the `temperature` argument to control the randomness,\njust like in multinomial sampling. However, in assisted decoding, reducing the temperature may help improve the latency."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: be5523fe-8658-42f8-bdc1-103b31b945b1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"be5523fe-8658-42f8-bdc1-103b31b945b1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"22362ff1-ecd5-41c9-aea9-7cf4a47309c0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7105c0bea103cb7406f4c99f1ab3aeea30205cb37afa9391fa30eb11c9905643\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0660fba0-30df-43f7-9031-c9a12d47e4a1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"396a0515fd9aa30bf940c3cbef8378afde1843a3b60336573cd14ce611ee525d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d253cf68-c4e4-4dd4-a365-c54e5baeb5d9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"64075a65916a5d71032f6056e99eb54ce7c52f2e5836e94ee8f437372eb50559\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 23320, \"end_char_idx\": 26650, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "doc_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "ref_doc_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "chroma:document": "### Speculative Decoding\n\nSpeculative decoding (also known as assisted decoding) is a modification of the decoding strategies above, that uses an\nassistant model (ideally a much smaller one) with the same tokenizer, to generate a few candidate tokens. The main\nmodel then validates the candidate tokens in a single forward pass, which speeds up the decoding process. If\n`do_sample=True`, then the token validation with resampling introduced in the\n[speculative decoding paper](https://arxiv.org/pdf/2211.17192.pdf) is used.\n\nCurrently, only greedy search and sampling are supported with assisted decoding, and assisted decoding doesn't support batched inputs.\nTo learn more about assisted decoding, check [this blog post](https://huggingface.co/blog/assisted-generation).\n\nTo enable assisted decoding, set the `assistant_model` argument with a model.\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> prompt = \"Alice and Bob\"\n>>> checkpoint = \"EleutherAI/pythia-1.4b-deduped\"\n>>> assistant_checkpoint = \"EleutherAI/pythia-160m-deduped\"\n\n>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)\n>>> inputs = tokenizer(prompt, return_tensors=\"pt\")\n\n>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)\n>>> assistant_model = AutoModelForCausalLM.from_pretrained(assistant_checkpoint)\n>>> outputs = model.generate(**inputs, assistant_model=assistant_model)\n>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)\n['Alice and Bob are sitting in a bar. Alice is drinking a beer and Bob is drinking a']\n```\n\nWhen using assisted decoding with sampling methods, you can use the `temperature` argument to control the randomness,\njust like in multinomial sampling. However, in assisted decoding, reducing the temperature may help improve the latency.\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer, set_seed\n>>> set_seed(42)  # For reproducibility\n\n>>> prompt = \"Alice and Bob\"\n>>> checkpoint = \"EleutherAI/pythia-1.4b-deduped\"\n>>> assistant_checkpoint = \"EleutherAI/pythia-160m-deduped\"\n\n>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)\n>>> inputs = tokenizer(prompt, return_tensors=\"pt\")\n\n>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)\n>>> assistant_model = AutoModelForCausalLM.from_pretrained(assistant_checkpoint)\n>>> outputs = model.generate(**inputs, assistant_model=assistant_model, do_sample=True, temperature=0.5)\n>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)\n['Alice and Bob, a couple of friends of mine, who are both in the same office as']\n```\n\nAlternativelly, you can also set the `prompt_lookup_num_tokens` to trigger n-gram based assisted decoding, as opposed\nto model based assisted decoding. You can read more about it [here](https://twitter.com/joao_gante/status/1747322413006643259).\n### DoLa Decoding\n\n**D**ecoding by C**o**ntrasting **La**yers (DoLa) is a contrastive decoding strategy to improve the factuality and reduce the\nhallucinations of LLMs, as described in this paper of ICLR 2024 [DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models](https://arxiv.org/abs/2309.03883).\n\nDoLa is achieved by contrasting the differences in logits obtained from final\nlayers versus earlier layers, thus amplify the factual knowledge localized to particular part of transformer layers."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: d253cf68-c4e4-4dd4-a365-c54e5baeb5d9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d253cf68-c4e4-4dd4-a365-c54e5baeb5d9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"22362ff1-ecd5-41c9-aea9-7cf4a47309c0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7105c0bea103cb7406f4c99f1ab3aeea30205cb37afa9391fa30eb11c9905643\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"be5523fe-8658-42f8-bdc1-103b31b945b1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c2ff98db507c95aa0871c6f440059c4b9944cfd6b89136cc35280ca6ad9d10c3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"62bc6463-4d9a-48f9-a26d-4410fddfff5b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a89cde1b24bc423d65409d5a5c2fc2ce3cbb0bf91e5584574f5dd69db0942bce\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 25020, \"end_char_idx\": 27734, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "doc_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "ref_doc_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "chroma:document": "However, in assisted decoding, reducing the temperature may help improve the latency.\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer, set_seed\n>>> set_seed(42)  # For reproducibility\n\n>>> prompt = \"Alice and Bob\"\n>>> checkpoint = \"EleutherAI/pythia-1.4b-deduped\"\n>>> assistant_checkpoint = \"EleutherAI/pythia-160m-deduped\"\n\n>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)\n>>> inputs = tokenizer(prompt, return_tensors=\"pt\")\n\n>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)\n>>> assistant_model = AutoModelForCausalLM.from_pretrained(assistant_checkpoint)\n>>> outputs = model.generate(**inputs, assistant_model=assistant_model, do_sample=True, temperature=0.5)\n>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)\n['Alice and Bob, a couple of friends of mine, who are both in the same office as']\n```\n\nAlternativelly, you can also set the `prompt_lookup_num_tokens` to trigger n-gram based assisted decoding, as opposed\nto model based assisted decoding. You can read more about it [here](https://twitter.com/joao_gante/status/1747322413006643259).\n### DoLa Decoding\n\n**D**ecoding by C**o**ntrasting **La**yers (DoLa) is a contrastive decoding strategy to improve the factuality and reduce the\nhallucinations of LLMs, as described in this paper of ICLR 2024 [DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models](https://arxiv.org/abs/2309.03883).\n\nDoLa is achieved by contrasting the differences in logits obtained from final\nlayers versus earlier layers, thus amplify the factual knowledge localized to particular part of transformer layers.\n\nDo the following two steps to activate DoLa decoding when calling the `model.generate` function:\n1. Set the `dola_layers` argument, which can be either a string or a list of integers.\n    - If set to a string, it can be one of `low`, `high`.\n    - If set to a list of integers, it should be a list of layer indices between 0 and the total number of layers in the model. The 0-th layer is word embedding, and the 1st layer is the first transformer layer, and so on.\n2. Set `repetition_penalty = 1.2` is suggested to reduce repetition in DoLa decoding.\n\nSee the following examples for DoLa decoding with the 32-layer LLaMA-7B model.\n\n```python\n>>> from transformers import AutoTokenizer, AutoModelForCausalLM, set_seed\n>>> import torch\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"huggyllama/llama-7b\")\n>>> model = AutoModelForCausalLM.from_pretrained(\"huggyllama/llama-7b\", torch_dtype=torch.float16)\n>>> device = 'cuda' if torch.cuda.is_available() else 'cpu'\n>>> model.to(device)\n>>> set_seed(42)\n\n>>> text = \"On what date was the Declaration of Independence officially signed?\""}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 62bc6463-4d9a-48f9-a26d-4410fddfff5b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"62bc6463-4d9a-48f9-a26d-4410fddfff5b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"22362ff1-ecd5-41c9-aea9-7cf4a47309c0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7105c0bea103cb7406f4c99f1ab3aeea30205cb37afa9391fa30eb11c9905643\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d253cf68-c4e4-4dd4-a365-c54e5baeb5d9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1fede848df6a7e614f27a623f264e89083103e6c9eeeab3733e455c9b2033844\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c96ba282-3ac6-45c9-b90b-223467a4ab71\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0a672838ba810f7e6bf629f46d9f8e8ac75a6332b442431bf7cbf99cdea803c9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 26456, \"end_char_idx\": 29525, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "doc_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "ref_doc_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "chroma:document": "DoLa is achieved by contrasting the differences in logits obtained from final\nlayers versus earlier layers, thus amplify the factual knowledge localized to particular part of transformer layers.\n\nDo the following two steps to activate DoLa decoding when calling the `model.generate` function:\n1. Set the `dola_layers` argument, which can be either a string or a list of integers.\n    - If set to a string, it can be one of `low`, `high`.\n    - If set to a list of integers, it should be a list of layer indices between 0 and the total number of layers in the model. The 0-th layer is word embedding, and the 1st layer is the first transformer layer, and so on.\n2. Set `repetition_penalty = 1.2` is suggested to reduce repetition in DoLa decoding.\n\nSee the following examples for DoLa decoding with the 32-layer LLaMA-7B model.\n\n```python\n>>> from transformers import AutoTokenizer, AutoModelForCausalLM, set_seed\n>>> import torch\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"huggyllama/llama-7b\")\n>>> model = AutoModelForCausalLM.from_pretrained(\"huggyllama/llama-7b\", torch_dtype=torch.float16)\n>>> device = 'cuda' if torch.cuda.is_available() else 'cpu'\n>>> model.to(device)\n>>> set_seed(42)\n\n>>> text = \"On what date was the Declaration of Independence officially signed?\"\n>>> inputs = tokenizer(text, return_tensors=\"pt\").to(device)\n\n# Vanilla greddy decoding\n>>> vanilla_output = model.generate(**inputs, do_sample=False, max_new_tokens=50)\n>>> tokenizer.batch_decode(vanilla_output[:, inputs.input_ids.shape[-1]:], skip_special_tokens=True)\n['\\nThe Declaration of Independence was signed on July 4, 1776.\\nWhat was the date of the signing of the Declaration of Independence?\\nThe Declaration of Independence was signed on July 4,']\n\n# DoLa decoding with contrasting higher part of layers (layers 16,18,...,30)\n>>> dola_high_output = model.generate(**inputs, do_sample=False, max_new_tokens=50, dola_layers='high')\n>>> tokenizer.batch_decode(dola_high_output[:, inputs.input_ids.shape[-1]:], skip_special_tokens=True)\n['\\nJuly 4, 1776, when the Continental Congress voted to separate from Great Britain. The 56 delegates to the Continental Congress signed the Declaration on August 2, 1776.']\n\n# DoLa decoding with contrasting specific layers (layers 28 and 30)\n>>> dola_custom_output = model.generate(**inputs, do_sample=False, max_new_tokens=50, dola_layers=[28,30], repetition_penalty=1.2)\n>>> tokenizer.batch_decode(dola_custom_output[:, inputs.input_ids.shape[-1]:], skip_special_tokens=True)\n['\\nIt was officially signed on 2 August 1776, when 56 members of the Second Continental Congress, representing the original 13 American colonies, voted unanimously for the resolution for independence. The 2']\n```\n\n#### Understanding the `dola_layers` argument\n\n`dola_layers` stands for the candidate layers in premature layer selection, as described in the DoLa paper. The selected premature layer will be contrasted with the final layer.\n\nSetting `dola_layers` to `'low'` or `'high'` will select the lower or higher part of the layers to contrast, respectively."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: c96ba282-3ac6-45c9-b90b-223467a4ab71
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c96ba282-3ac6-45c9-b90b-223467a4ab71\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"22362ff1-ecd5-41c9-aea9-7cf4a47309c0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7105c0bea103cb7406f4c99f1ab3aeea30205cb37afa9391fa30eb11c9905643\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"62bc6463-4d9a-48f9-a26d-4410fddfff5b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/generation_strategies\", \"title\": \"Text generation strategies\", \"tokens\": 7027, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3ba668e2c0aa7ac50d45cb380f431c6f3d4fb4b3baa3ba1c4a7cfe87ff03bd2c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 28568, \"end_char_idx\": 30594, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "doc_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "ref_doc_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "chroma:document": "The 56 delegates to the Continental Congress signed the Declaration on August 2, 1776.']\n\n# DoLa decoding with contrasting specific layers (layers 28 and 30)\n>>> dola_custom_output = model.generate(**inputs, do_sample=False, max_new_tokens=50, dola_layers=[28,30], repetition_penalty=1.2)\n>>> tokenizer.batch_decode(dola_custom_output[:, inputs.input_ids.shape[-1]:], skip_special_tokens=True)\n['\\nIt was officially signed on 2 August 1776, when 56 members of the Second Continental Congress, representing the original 13 American colonies, voted unanimously for the resolution for independence. The 2']\n```\n\n#### Understanding the `dola_layers` argument\n\n`dola_layers` stands for the candidate layers in premature layer selection, as described in the DoLa paper. The selected premature layer will be contrasted with the final layer.\n\nSetting `dola_layers` to `'low'` or `'high'` will select the lower or higher part of the layers to contrast, respectively.\n- For `N`-layer models with `N <= 40` layers, the layers of `range(0, N // 2, 2)` and `range(N // 2, N, 2)` are used for `'low'` and `'high'` layers, respectively.\n- For models with `N > 40` layers, the layers of `range(0, 20, 2)` and `range(N - 20, N, 2)` are used for `'low'` and `'high'` layers, respectively.\n- If the model has tied word embeddings, we skip the word embeddings (0-th) layer and start from the 2nd layer, as the early exit from word embeddings will become identity function.\n- Set the `dola_layers` to a list of integers for layer indices to contrast manually specified layers. For example, setting `dola_layers=[28,30]` will contrast the final layer (32-th layer) with the 28-th and 30-th layers.\n\nThe paper suggested that contrasting `'high'` layers to improve short-answer tasks like TruthfulQA, and contrasting `'low'` layers to improve all the other long-answer reasoning tasks, such as GSM8K, StrategyQA, FACTOR, and VicunaQA. Applying DoLa to smaller models like GPT-2 is not recommended, as the results shown in the Appendix N of the paper."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 6ad264ae-7166-4b47-ac08-58452b031f8b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6ad264ae-7166-4b47-ac08-58452b031f8b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8704bb74-48ac-4a96-bb9f-18f282312d0b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0268c0a2-6cc2-4024-ad84-06c37d8b4075\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"544bbdf4780eebd34942a774eafb30a2ddac2e3559a77cfab2cbf38622b332d9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2682, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "doc_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "ref_doc_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "chroma:document": "# Glossary\n\nThis glossary defines general machine learning and \ud83e\udd17 Transformers terms to help you better understand the\ndocumentation.\n\n## A\n\n### attention mask\n\nThe attention mask is an optional argument used when batching sequences together.\n\n<Youtube id=\"M6adb1j2jPI\"/>\n\nThis argument indicates to the model which tokens should be attended to, and which should not.\n\nFor example, consider these two sequences:\n\n```python\n>>> from transformers import BertTokenizer\n\n>>> tokenizer = BertTokenizer.from_pretrained(\"google-bert/bert-base-cased\")\n\n>>> sequence_a = \"This is a short sequence.\"\n>>> sequence_b = \"This is a rather long sequence. It is at least longer than the sequence A.\"\n\n>>> encoded_sequence_a = tokenizer(sequence_a)[\"input_ids\"]\n>>> encoded_sequence_b = tokenizer(sequence_b)[\"input_ids\"]\n```\n\nThe encoded versions have different lengths:\n\n```python\n>>> len(encoded_sequence_a), len(encoded_sequence_b)\n(8, 19)\n```\n\nTherefore, we can't put them together in the same tensor as-is. The first sequence needs to be padded up to the length\nof the second one, or the second one needs to be truncated down to the length of the first one.\n\nIn the first case, the list of IDs will be extended by the padding indices. We can pass a list to the tokenizer and ask\nit to pad like this:\n\n```python\n>>> padded_sequences = tokenizer([sequence_a, sequence_b], padding=True)\n```\n\nWe can see that 0s have been added on the right of the first sentence to make it the same length as the second one:\n\n```python\n>>> padded_sequences[\"input_ids\"]\n[[101, 1188, 1110, 170, 1603, 4954, 119, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [101, 1188, 1110, 170, 1897, 1263, 4954, 119, 1135, 1110, 1120, 1655, 2039, 1190, 1103, 4954, 138, 119, 102]]\n```\n\nThis can then be converted into a tensor in PyTorch or TensorFlow. The attention mask is a binary tensor indicating the\nposition of the padded indices so that the model does not attend to them. For the [`BertTokenizer`], `1` indicates a\nvalue that should be attended to, while `0` indicates a padded value. This attention mask is in the dictionary returned\nby the tokenizer under the key \"attention_mask\":\n\n```python\n>>> padded_sequences[\"attention_mask\"]\n[[1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]\n```\n\n### autoencoding models\n\nSee [encoder models](#encoder-models) and [masked language modeling](#masked-language-modeling-mlm)\n\n### autoregressive models\n\nSee [causal language modeling](#causal-language-modeling) and [decoder models](#decoder-models)\n\n## B\n\n### backbone\n\nThe backbone is the network (embeddings and layers) that outputs the raw hidden states or features."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 0268c0a2-6cc2-4024-ad84-06c37d8b4075
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0268c0a2-6cc2-4024-ad84-06c37d8b4075\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8704bb74-48ac-4a96-bb9f-18f282312d0b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6ad264ae-7166-4b47-ac08-58452b031f8b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d50d2bbc0fa78ec35e75e159d92ad0cf1c169b4269062722a0b5fb2da5761eca\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ecde5024-d591-49bf-988d-d89c68c3d704\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"724e55a11efb5d580233e2ee07613afcf8e36cc3d6d11d6dfe70f31fd3d954e8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1800, \"end_char_idx\": 4934, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "doc_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "ref_doc_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "chroma:document": "The attention mask is a binary tensor indicating the\nposition of the padded indices so that the model does not attend to them. For the [`BertTokenizer`], `1` indicates a\nvalue that should be attended to, while `0` indicates a padded value. This attention mask is in the dictionary returned\nby the tokenizer under the key \"attention_mask\":\n\n```python\n>>> padded_sequences[\"attention_mask\"]\n[[1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]\n```\n\n### autoencoding models\n\nSee [encoder models](#encoder-models) and [masked language modeling](#masked-language-modeling-mlm)\n\n### autoregressive models\n\nSee [causal language modeling](#causal-language-modeling) and [decoder models](#decoder-models)\n\n## B\n\n### backbone\n\nThe backbone is the network (embeddings and layers) that outputs the raw hidden states or features. It is usually connected to a [head](#head) which accepts the features as its input to make a prediction. For example, [`ViTModel`] is a backbone without a specific head on top. Other models can also use [`VitModel`] as a backbone such as [DPT](model_doc/dpt).\n\n## C\n\n### causal language modeling\n\nA pretraining task where the model reads the texts in order and has to predict the next word. It's usually done by\nreading the whole sentence but using a mask inside the model to hide the future tokens at a certain timestep.\n\n### channel\n\nColor images are made up of some combination of values in three channels: red, green, and blue (RGB) and grayscale images only have one channel. In \ud83e\udd17 Transformers, the channel can be the first or last dimension of an image's tensor: [`n_channels`, `height`, `width`] or [`height`, `width`, `n_channels`].\n\n### connectionist temporal classification (CTC)\n\nAn algorithm which allows a model to learn without knowing exactly how the input and output are aligned; CTC calculates the distribution of all possible outputs for a given input and chooses the most likely output from it. CTC is commonly used in speech recognition tasks because speech doesn't always cleanly align with the transcript for a variety of reasons such as a speaker's different speech rates.\n\n### convolution\n\nA type of layer in a neural network where the input matrix is multiplied element-wise by a smaller matrix (kernel or filter) and the values are summed up in a new matrix. This is known as a convolutional operation which is repeated over the entire input matrix. Each operation is applied to a different segment of the input matrix. Convolutional neural networks (CNNs) are commonly used in computer vision.\n\n## D\n\n### DataParallel (DP)\n\nParallelism technique for training on multiple GPUs where the same setup is replicated multiple times, with each instance \nreceiving a distinct data slice. The processing is done in parallel and all setups are synchronized at the end of each training step.\n\nLearn more about how DataParallel works [here](perf_train_gpu_many#dataparallel-vs-distributeddataparallel).\n\n### decoder input IDs\n\nThis input is specific to encoder-decoder models, and contains the input IDs that will be fed to the decoder."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: ecde5024-d591-49bf-988d-d89c68c3d704
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ecde5024-d591-49bf-988d-d89c68c3d704\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8704bb74-48ac-4a96-bb9f-18f282312d0b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0268c0a2-6cc2-4024-ad84-06c37d8b4075\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8d3dbc685199699a6a99795d3e4200189233ee685ef8a2330b7c126480d34e90\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e0b4e6e8-3d52-49cf-aaec-1eb31da9813a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d9a6c8657631fc50e8689ef751a97b0bb3966f8c4fdec5d1e8574e7595cf4a95\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3074, \"end_char_idx\": 6770, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "doc_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "ref_doc_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "chroma:document": "It's usually done by\nreading the whole sentence but using a mask inside the model to hide the future tokens at a certain timestep.\n\n### channel\n\nColor images are made up of some combination of values in three channels: red, green, and blue (RGB) and grayscale images only have one channel. In \ud83e\udd17 Transformers, the channel can be the first or last dimension of an image's tensor: [`n_channels`, `height`, `width`] or [`height`, `width`, `n_channels`].\n\n### connectionist temporal classification (CTC)\n\nAn algorithm which allows a model to learn without knowing exactly how the input and output are aligned; CTC calculates the distribution of all possible outputs for a given input and chooses the most likely output from it. CTC is commonly used in speech recognition tasks because speech doesn't always cleanly align with the transcript for a variety of reasons such as a speaker's different speech rates.\n\n### convolution\n\nA type of layer in a neural network where the input matrix is multiplied element-wise by a smaller matrix (kernel or filter) and the values are summed up in a new matrix. This is known as a convolutional operation which is repeated over the entire input matrix. Each operation is applied to a different segment of the input matrix. Convolutional neural networks (CNNs) are commonly used in computer vision.\n\n## D\n\n### DataParallel (DP)\n\nParallelism technique for training on multiple GPUs where the same setup is replicated multiple times, with each instance \nreceiving a distinct data slice. The processing is done in parallel and all setups are synchronized at the end of each training step.\n\nLearn more about how DataParallel works [here](perf_train_gpu_many#dataparallel-vs-distributeddataparallel).\n\n### decoder input IDs\n\nThis input is specific to encoder-decoder models, and contains the input IDs that will be fed to the decoder. These\ninputs should be used for sequence to sequence tasks, such as translation or summarization, and are usually built in a\nway specific to each model.\n\nMost encoder-decoder models (BART, T5) create their `decoder_input_ids` on their own from the `labels`. In such models,\npassing the `labels` is the preferred way to handle training.\n\nPlease check each model's docs to see how they handle these input IDs for sequence to sequence training.\n\n### decoder models\n\nAlso referred to as autoregressive models, decoder models involve a pretraining task (called causal language modeling) where the model reads the texts in order and has to predict the next word. It's usually done by\nreading the whole sentence with a mask to hide future tokens at a certain timestep.\n\n<Youtube id=\"d_ixlCubqQw\"/>\n\n### deep learning (DL)\n\nMachine learning algorithms which use neural networks with several layers.\n\n## E\n\n### encoder models\n\nAlso known as autoencoding models, encoder models take an input (such as text or images) and transform them into a condensed numerical representation called an embedding. Oftentimes, encoder models are pretrained using techniques like [masked language modeling](#masked-language-modeling-mlm), which masks parts of the input sequence and forces the model to create more meaningful representations.\n\n<Youtube id=\"H39Z_720T5s\"/>\n\n## F\n\n### feature extraction\n\nThe process of selecting and transforming raw data into a set of features that are more informative and useful for machine learning algorithms. Some examples of feature extraction include transforming raw text into word embeddings and extracting important features such as edges or shapes from image/video data.\n\n### feed forward chunking\n\nIn each residual attention block in transformers the self-attention layer is usually followed by 2 feed forward layers."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: e0b4e6e8-3d52-49cf-aaec-1eb31da9813a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e0b4e6e8-3d52-49cf-aaec-1eb31da9813a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8704bb74-48ac-4a96-bb9f-18f282312d0b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ecde5024-d591-49bf-988d-d89c68c3d704\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7c30f61cefcc05b2f9e95617a1d2ea6e59d939daf84116d46f81d95e4065ec80\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"59dff7d1-d5c4-4292-a950-990fc55169d5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"15a043b5d160ad96a20bb0c6851e88fdbfa41a9f0652116c56d57e630b2ff971\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4935, \"end_char_idx\": 8450, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "doc_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "ref_doc_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "chroma:document": "These\ninputs should be used for sequence to sequence tasks, such as translation or summarization, and are usually built in a\nway specific to each model.\n\nMost encoder-decoder models (BART, T5) create their `decoder_input_ids` on their own from the `labels`. In such models,\npassing the `labels` is the preferred way to handle training.\n\nPlease check each model's docs to see how they handle these input IDs for sequence to sequence training.\n\n### decoder models\n\nAlso referred to as autoregressive models, decoder models involve a pretraining task (called causal language modeling) where the model reads the texts in order and has to predict the next word. It's usually done by\nreading the whole sentence with a mask to hide future tokens at a certain timestep.\n\n<Youtube id=\"d_ixlCubqQw\"/>\n\n### deep learning (DL)\n\nMachine learning algorithms which use neural networks with several layers.\n\n## E\n\n### encoder models\n\nAlso known as autoencoding models, encoder models take an input (such as text or images) and transform them into a condensed numerical representation called an embedding. Oftentimes, encoder models are pretrained using techniques like [masked language modeling](#masked-language-modeling-mlm), which masks parts of the input sequence and forces the model to create more meaningful representations.\n\n<Youtube id=\"H39Z_720T5s\"/>\n\n## F\n\n### feature extraction\n\nThe process of selecting and transforming raw data into a set of features that are more informative and useful for machine learning algorithms. Some examples of feature extraction include transforming raw text into word embeddings and extracting important features such as edges or shapes from image/video data.\n\n### feed forward chunking\n\nIn each residual attention block in transformers the self-attention layer is usually followed by 2 feed forward layers.\nThe intermediate embedding size of the feed forward layers is often bigger than the hidden size of the model (e.g., for\n`google-bert/bert-base-uncased`).\n\nFor an input of size `[batch_size, sequence_length]`, the memory required to store the intermediate feed forward\nembeddings `[batch_size, sequence_length, config.intermediate_size]` can account for a large fraction of the memory\nuse. The authors of [Reformer: The Efficient Transformer](https://arxiv.org/abs/2001.04451) noticed that since the\ncomputation is independent of the `sequence_length` dimension, it is mathematically equivalent to compute the output\nembeddings of both feed forward layers `[batch_size, config.hidden_size]_0, ..., [batch_size, config.hidden_size]_n`\nindividually and concat them afterward to `[batch_size, sequence_length, config.hidden_size]` with `n = sequence_length`, which trades increased computation time against reduced memory use, but yields a mathematically\n**equivalent** result.\n\nFor models employing the function [`apply_chunking_to_forward`], the `chunk_size` defines the number of output\nembeddings that are computed in parallel and thus defines the trade-off between memory and time complexity. If\n`chunk_size` is set to 0, no feed forward chunking is done.\n\n### finetuned models\n\nFinetuning is a form of transfer learning which involves taking a pretrained model, freezing its weights, and replacing the output layer with a newly added [model head](#head). The model head is trained on your target dataset.\n\nSee the [Fine-tune a pretrained model](https://huggingface.co/docs/transformers/training) tutorial for more details, and learn how to fine-tune models with \ud83e\udd17 Transformers."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 59dff7d1-d5c4-4292-a950-990fc55169d5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"59dff7d1-d5c4-4292-a950-990fc55169d5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8704bb74-48ac-4a96-bb9f-18f282312d0b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e0b4e6e8-3d52-49cf-aaec-1eb31da9813a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ec2ac1d172a8f79577956475146d6b37a9fff6f96f1f772ec492c91611d86553\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3dc3dd43-02b7-4b56-904f-f7cd5c4808e4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6df752eec3d8dbb429d09288e9786890f9091a8faa09054fa713e908fc9c8ad4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6771, \"end_char_idx\": 9956, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "doc_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "ref_doc_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "chroma:document": "The intermediate embedding size of the feed forward layers is often bigger than the hidden size of the model (e.g., for\n`google-bert/bert-base-uncased`).\n\nFor an input of size `[batch_size, sequence_length]`, the memory required to store the intermediate feed forward\nembeddings `[batch_size, sequence_length, config.intermediate_size]` can account for a large fraction of the memory\nuse. The authors of [Reformer: The Efficient Transformer](https://arxiv.org/abs/2001.04451) noticed that since the\ncomputation is independent of the `sequence_length` dimension, it is mathematically equivalent to compute the output\nembeddings of both feed forward layers `[batch_size, config.hidden_size]_0, ..., [batch_size, config.hidden_size]_n`\nindividually and concat them afterward to `[batch_size, sequence_length, config.hidden_size]` with `n = sequence_length`, which trades increased computation time against reduced memory use, but yields a mathematically\n**equivalent** result.\n\nFor models employing the function [`apply_chunking_to_forward`], the `chunk_size` defines the number of output\nembeddings that are computed in parallel and thus defines the trade-off between memory and time complexity. If\n`chunk_size` is set to 0, no feed forward chunking is done.\n\n### finetuned models\n\nFinetuning is a form of transfer learning which involves taking a pretrained model, freezing its weights, and replacing the output layer with a newly added [model head](#head). The model head is trained on your target dataset.\n\nSee the [Fine-tune a pretrained model](https://huggingface.co/docs/transformers/training) tutorial for more details, and learn how to fine-tune models with \ud83e\udd17 Transformers.\n\n## H\n\n### head\n\nThe model head refers to the last layer of a neural network that accepts the raw hidden states and projects them onto a different dimension. There is a different model head for each task. For example:\n\n  * [`GPT2ForSequenceClassification`] is a sequence classification head - a linear layer - on top of the base [`GPT2Model`].\n  * [`ViTForImageClassification`] is an image classification head - a linear layer on top of the final hidden state of the `CLS` token - on top of the base [`ViTModel`].\n  * [`Wav2Vec2ForCTC`] is a language modeling head with [CTC](#connectionist-temporal-classification-ctc) on top of the base [`Wav2Vec2Model`].\n\n## I\n\n### image patch\n\nVision-based Transformers models split an image into smaller patches which are linearly embedded, and then passed as a sequence to the model. You can find the `patch_size` - or resolution - of the model in its configuration.\n\n### inference\n\nInference is the process of evaluating a model on new data after training is complete. See the [Pipeline for inference](https://huggingface.co/docs/transformers/pipeline_tutorial) tutorial to learn how to perform inference with \ud83e\udd17 Transformers.\n\n### input IDs\n\nThe input ids are often the only required parameters to be passed to the model as input. They are token indices,\nnumerical representations of tokens building the sequences that will be used as input by the model.\n\n<Youtube id=\"VFp38yj8h3A\"/>\n\nEach tokenizer works differently but the underlying mechanism remains the same."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 3dc3dd43-02b7-4b56-904f-f7cd5c4808e4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3dc3dd43-02b7-4b56-904f-f7cd5c4808e4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8704bb74-48ac-4a96-bb9f-18f282312d0b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"59dff7d1-d5c4-4292-a950-990fc55169d5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"dd8f820a11d1195ab3b7a90b535fbaaf376ae407cf7137254cc15ec4c4d8d580\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3e6885ef-0c8c-48e5-a12c-e2f16a1a3588\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"eca63cd2b900dfb0620eaa2d0865e268c5a14fbecf1d51dc0d8f52eddcad1941\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8279, \"end_char_idx\": 11256, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "doc_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "ref_doc_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "chroma:document": "See the [Fine-tune a pretrained model](https://huggingface.co/docs/transformers/training) tutorial for more details, and learn how to fine-tune models with \ud83e\udd17 Transformers.\n\n## H\n\n### head\n\nThe model head refers to the last layer of a neural network that accepts the raw hidden states and projects them onto a different dimension. There is a different model head for each task. For example:\n\n  * [`GPT2ForSequenceClassification`] is a sequence classification head - a linear layer - on top of the base [`GPT2Model`].\n  * [`ViTForImageClassification`] is an image classification head - a linear layer on top of the final hidden state of the `CLS` token - on top of the base [`ViTModel`].\n  * [`Wav2Vec2ForCTC`] is a language modeling head with [CTC](#connectionist-temporal-classification-ctc) on top of the base [`Wav2Vec2Model`].\n\n## I\n\n### image patch\n\nVision-based Transformers models split an image into smaller patches which are linearly embedded, and then passed as a sequence to the model. You can find the `patch_size` - or resolution - of the model in its configuration.\n\n### inference\n\nInference is the process of evaluating a model on new data after training is complete. See the [Pipeline for inference](https://huggingface.co/docs/transformers/pipeline_tutorial) tutorial to learn how to perform inference with \ud83e\udd17 Transformers.\n\n### input IDs\n\nThe input ids are often the only required parameters to be passed to the model as input. They are token indices,\nnumerical representations of tokens building the sequences that will be used as input by the model.\n\n<Youtube id=\"VFp38yj8h3A\"/>\n\nEach tokenizer works differently but the underlying mechanism remains the same. Here's an example using the BERT\ntokenizer, which is a [WordPiece](https://arxiv.org/pdf/1609.08144.pdf) tokenizer:\n\n```python\n>>> from transformers import BertTokenizer\n\n>>> tokenizer = BertTokenizer.from_pretrained(\"google-bert/bert-base-cased\")\n\n>>> sequence = \"A Titan RTX has 24GB of VRAM\"\n```\n\nThe tokenizer takes care of splitting the sequence into tokens available in the tokenizer vocabulary.\n\n```python\n>>> tokenized_sequence = tokenizer.tokenize(sequence)\n```\n\nThe tokens are either words or subwords. Here for instance, \"VRAM\" wasn't in the model vocabulary, so it's been split\nin \"V\", \"RA\" and \"M\". To indicate those tokens are not separate words but parts of the same word, a double-hash prefix\nis added for \"RA\" and \"M\":\n\n```python\n>>> print(tokenized_sequence)\n['A', 'Titan', 'R', '##T', '##X', 'has', '24', '##GB', 'of', 'V', '##RA', '##M']\n```\n\nThese tokens can then be converted into IDs which are understandable by the model. This can be done by directly feeding the sentence to the tokenizer, which leverages the Rust implementation of [\ud83e\udd17 Tokenizers](https://github.com/huggingface/tokenizers) for peak performance.\n\n```python\n>>> inputs = tokenizer(sequence)\n```\n\nThe tokenizer returns a dictionary with all the arguments necessary for its corresponding model to work properly."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 3e6885ef-0c8c-48e5-a12c-e2f16a1a3588
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3e6885ef-0c8c-48e5-a12c-e2f16a1a3588\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8704bb74-48ac-4a96-bb9f-18f282312d0b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3dc3dd43-02b7-4b56-904f-f7cd5c4808e4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"df0e83b77324df0369bf3a0e8366210d4970df17e05ef3b7f78e5312a29593ab\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9f019310-de42-478e-bd20-d7ba77c4ba93\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fdac5ab48bed5bcc837fabca6cdd2170943cf020784648ccf6a7fd9936b481c4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9619, \"end_char_idx\": 12723, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "doc_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "ref_doc_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "chroma:document": "### input IDs\n\nThe input ids are often the only required parameters to be passed to the model as input. They are token indices,\nnumerical representations of tokens building the sequences that will be used as input by the model.\n\n<Youtube id=\"VFp38yj8h3A\"/>\n\nEach tokenizer works differently but the underlying mechanism remains the same. Here's an example using the BERT\ntokenizer, which is a [WordPiece](https://arxiv.org/pdf/1609.08144.pdf) tokenizer:\n\n```python\n>>> from transformers import BertTokenizer\n\n>>> tokenizer = BertTokenizer.from_pretrained(\"google-bert/bert-base-cased\")\n\n>>> sequence = \"A Titan RTX has 24GB of VRAM\"\n```\n\nThe tokenizer takes care of splitting the sequence into tokens available in the tokenizer vocabulary.\n\n```python\n>>> tokenized_sequence = tokenizer.tokenize(sequence)\n```\n\nThe tokens are either words or subwords. Here for instance, \"VRAM\" wasn't in the model vocabulary, so it's been split\nin \"V\", \"RA\" and \"M\". To indicate those tokens are not separate words but parts of the same word, a double-hash prefix\nis added for \"RA\" and \"M\":\n\n```python\n>>> print(tokenized_sequence)\n['A', 'Titan', 'R', '##T', '##X', 'has', '24', '##GB', 'of', 'V', '##RA', '##M']\n```\n\nThese tokens can then be converted into IDs which are understandable by the model. This can be done by directly feeding the sentence to the tokenizer, which leverages the Rust implementation of [\ud83e\udd17 Tokenizers](https://github.com/huggingface/tokenizers) for peak performance.\n\n```python\n>>> inputs = tokenizer(sequence)\n```\n\nThe tokenizer returns a dictionary with all the arguments necessary for its corresponding model to work properly. The\ntoken indices are under the key `input_ids`:\n\n```python\n>>> encoded_sequence = inputs[\"input_ids\"]\n>>> print(encoded_sequence)\n[101, 138, 18696, 155, 1942, 3190, 1144, 1572, 13745, 1104, 159, 9664, 2107, 102]\n```\n\nNote that the tokenizer automatically adds \"special tokens\" (if the associated model relies on them) which are special\nIDs the model sometimes uses.\n\nIf we decode the previous sequence of ids,\n\n```python\n>>> decoded_sequence = tokenizer.decode(encoded_sequence)\n```\n\nwe will see\n\n```python\n>>> print(decoded_sequence)\n[CLS] A Titan RTX has 24GB of VRAM [SEP]\n```\n\nbecause this is the way a [`BertModel`] is going to expect its inputs.\n\n## L\n\n### labels\n\nThe labels are an optional argument which can be passed in order for the model to compute the loss itself. These labels\nshould be the expected prediction of the model: it will use the standard loss in order to compute the loss between its\npredictions and the expected value (the label).\n\nThese labels are different according to the model head, for example:\n\n- For sequence classification models, ([`BertForSequenceClassification`]), the model expects a tensor of dimension\n  `(batch_size)` with each value of the batch corresponding to the expected label of the entire sequence.\n- For token classification models, ([`BertForTokenClassification`]), the model expects a tensor of dimension\n  `(batch_size, seq_length)` with each value corresponding to the expected label of each individual token."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 9f019310-de42-478e-bd20-d7ba77c4ba93
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9f019310-de42-478e-bd20-d7ba77c4ba93\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8704bb74-48ac-4a96-bb9f-18f282312d0b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3e6885ef-0c8c-48e5-a12c-e2f16a1a3588\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1933c18c61867bbce6edd33e542e7fb8ac5bb0ecfbd7a4307a50265113b15e7a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fa7cf7bd-9a57-4107-8b04-3013d00c2b9b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"afe19266d1444271f74feff15a3fa592247d9bf7376625116a291f1313006444\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11095, \"end_char_idx\": 14472, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "doc_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "ref_doc_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "chroma:document": "```python\n>>> inputs = tokenizer(sequence)\n```\n\nThe tokenizer returns a dictionary with all the arguments necessary for its corresponding model to work properly. The\ntoken indices are under the key `input_ids`:\n\n```python\n>>> encoded_sequence = inputs[\"input_ids\"]\n>>> print(encoded_sequence)\n[101, 138, 18696, 155, 1942, 3190, 1144, 1572, 13745, 1104, 159, 9664, 2107, 102]\n```\n\nNote that the tokenizer automatically adds \"special tokens\" (if the associated model relies on them) which are special\nIDs the model sometimes uses.\n\nIf we decode the previous sequence of ids,\n\n```python\n>>> decoded_sequence = tokenizer.decode(encoded_sequence)\n```\n\nwe will see\n\n```python\n>>> print(decoded_sequence)\n[CLS] A Titan RTX has 24GB of VRAM [SEP]\n```\n\nbecause this is the way a [`BertModel`] is going to expect its inputs.\n\n## L\n\n### labels\n\nThe labels are an optional argument which can be passed in order for the model to compute the loss itself. These labels\nshould be the expected prediction of the model: it will use the standard loss in order to compute the loss between its\npredictions and the expected value (the label).\n\nThese labels are different according to the model head, for example:\n\n- For sequence classification models, ([`BertForSequenceClassification`]), the model expects a tensor of dimension\n  `(batch_size)` with each value of the batch corresponding to the expected label of the entire sequence.\n- For token classification models, ([`BertForTokenClassification`]), the model expects a tensor of dimension\n  `(batch_size, seq_length)` with each value corresponding to the expected label of each individual token.\n- For masked language modeling, ([`BertForMaskedLM`]), the model expects a tensor of dimension `(batch_size,\n  seq_length)` with each value corresponding to the expected label of each individual token: the labels being the token\n  ID for the masked token, and values to be ignored for the rest (usually -100).\n- For sequence to sequence tasks, ([`BartForConditionalGeneration`], [`MBartForConditionalGeneration`]), the model\n  expects a tensor of dimension `(batch_size, tgt_seq_length)` with each value corresponding to the target sequences\n  associated with each input sequence. During training, both BART and T5 will make the appropriate\n  `decoder_input_ids` and decoder attention masks internally. They usually do not need to be supplied. This does not\n  apply to models leveraging the Encoder-Decoder framework.\n- For image classification models, ([`ViTForImageClassification`]), the model expects a tensor of dimension\n  `(batch_size)` with each value of the batch corresponding to the expected label of each individual image.\n- For semantic segmentation models, ([`SegformerForSemanticSegmentation`]), the model expects a tensor of dimension\n  `(batch_size, height, width)` with each value of the batch corresponding to the expected label of each individual pixel.\n- For object detection models, ([`DetrForObjectDetection`]), the model expects a list of dictionaries with a\n  `class_labels` and `boxes` key where each value of the batch corresponds to the expected label and number of bounding boxes of each individual image.\n- For automatic speech recognition models, ([`Wav2Vec2ForCTC`]), the model expects a tensor of dimension `(batch_size,\n  target_length)` with each value corresponding to the expected label of each individual token."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: fa7cf7bd-9a57-4107-8b04-3013d00c2b9b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"fa7cf7bd-9a57-4107-8b04-3013d00c2b9b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8704bb74-48ac-4a96-bb9f-18f282312d0b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9f019310-de42-478e-bd20-d7ba77c4ba93\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b76b06068b5d3a6828bbf3020e823848c23b7ada1dd1accfc406ca46bd0fe592\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"37e36f00-5f04-4834-b109-882016246fdd\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fc97bdac04d667f2f250bad3d94e1b66467153e4c48013d289ab0eb0399d1542\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12724, \"end_char_idx\": 16156, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "doc_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "ref_doc_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "chroma:document": "- For masked language modeling, ([`BertForMaskedLM`]), the model expects a tensor of dimension `(batch_size,\n  seq_length)` with each value corresponding to the expected label of each individual token: the labels being the token\n  ID for the masked token, and values to be ignored for the rest (usually -100).\n- For sequence to sequence tasks, ([`BartForConditionalGeneration`], [`MBartForConditionalGeneration`]), the model\n  expects a tensor of dimension `(batch_size, tgt_seq_length)` with each value corresponding to the target sequences\n  associated with each input sequence. During training, both BART and T5 will make the appropriate\n  `decoder_input_ids` and decoder attention masks internally. They usually do not need to be supplied. This does not\n  apply to models leveraging the Encoder-Decoder framework.\n- For image classification models, ([`ViTForImageClassification`]), the model expects a tensor of dimension\n  `(batch_size)` with each value of the batch corresponding to the expected label of each individual image.\n- For semantic segmentation models, ([`SegformerForSemanticSegmentation`]), the model expects a tensor of dimension\n  `(batch_size, height, width)` with each value of the batch corresponding to the expected label of each individual pixel.\n- For object detection models, ([`DetrForObjectDetection`]), the model expects a list of dictionaries with a\n  `class_labels` and `boxes` key where each value of the batch corresponds to the expected label and number of bounding boxes of each individual image.\n- For automatic speech recognition models, ([`Wav2Vec2ForCTC`]), the model expects a tensor of dimension `(batch_size,\n  target_length)` with each value corresponding to the expected label of each individual token.\n  \n<Tip>\n\nEach model's labels may be different, so be sure to always check the documentation of each model for more information\nabout their specific labels!\n\n</Tip>\n\nThe base models ([`BertModel`]) do not accept labels, as these are the base transformer models, simply outputting\nfeatures.\n\n### large language models (LLM)\n\nA generic term that refers to transformer language models (GPT-3, BLOOM, OPT) that were trained on a large quantity of data. These models also tend to have a large number of learnable parameters (e.g. 175 billion for GPT-3).\n\n## M\n\n### masked language modeling (MLM)\n\nA pretraining task where the model sees a corrupted version of the texts, usually done by\nmasking some tokens randomly, and has to predict the original text.\n\n### multimodal\n\nA task that combines texts with another kind of inputs (for instance images).\n\n## N\n\n### Natural language generation (NLG)\n\nAll tasks related to generating text (for instance, [Write With Transformers](https://transformer.huggingface.co/), translation).\n\n### Natural language processing (NLP)\n\nA generic way to say \"deal with texts\".\n\n### Natural language understanding (NLU)\n\nAll tasks related to understanding what is in a text (for instance classifying the\nwhole text, individual words).\n\n## P\n\n### pipeline\n\nA pipeline in \ud83e\udd17 Transformers is an abstraction referring to a series of steps that are executed in a specific order to preprocess and transform data and return a prediction from a model. Some example stages found in a pipeline might be data preprocessing, feature extraction, and normalization.\n\nFor more details, see [Pipelines for inference](https://huggingface.co/docs/transformers/pipeline_tutorial)."}


created_at: 2024-07-28 18:57:50
topic: persistent://default/default/5523a7e1-1ad4-449b-b059-70d441f3d0d1
id: 37e36f00-5f04-4834-b109-882016246fdd
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"37e36f00-5f04-4834-b109-882016246fdd\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8704bb74-48ac-4a96-bb9f-18f282312d0b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"fa7cf7bd-9a57-4107-8b04-3013d00c2b9b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"299fc7f9cfe3d7d6eac8eb834ae577d1bab5dde5f15b84184221376cf1eeccc9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c3deba38-04bb-48e0-95e5-933df4e1e733\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9087e322d1352cb5d100a624420de265143a8cc5cd06d843cebdbe240a9b3f75\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14476, \"end_char_idx\": 18172, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "doc_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "ref_doc_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "chroma:document": "<Tip>\n\nEach model's labels may be different, so be sure to always check the documentation of each model for more information\nabout their specific labels!\n\n</Tip>\n\nThe base models ([`BertModel`]) do not accept labels, as these are the base transformer models, simply outputting\nfeatures.\n\n### large language models (LLM)\n\nA generic term that refers to transformer language models (GPT-3, BLOOM, OPT) that were trained on a large quantity of data. These models also tend to have a large number of learnable parameters (e.g. 175 billion for GPT-3).\n\n## M\n\n### masked language modeling (MLM)\n\nA pretraining task where the model sees a corrupted version of the texts, usually done by\nmasking some tokens randomly, and has to predict the original text.\n\n### multimodal\n\nA task that combines texts with another kind of inputs (for instance images).\n\n## N\n\n### Natural language generation (NLG)\n\nAll tasks related to generating text (for instance, [Write With Transformers](https://transformer.huggingface.co/), translation).\n\n### Natural language processing (NLP)\n\nA generic way to say \"deal with texts\".\n\n### Natural language understanding (NLU)\n\nAll tasks related to understanding what is in a text (for instance classifying the\nwhole text, individual words).\n\n## P\n\n### pipeline\n\nA pipeline in \ud83e\udd17 Transformers is an abstraction referring to a series of steps that are executed in a specific order to preprocess and transform data and return a prediction from a model. Some example stages found in a pipeline might be data preprocessing, feature extraction, and normalization.\n\nFor more details, see [Pipelines for inference](https://huggingface.co/docs/transformers/pipeline_tutorial).\n\n### PipelineParallel (PP)\n\nParallelism technique in which the model is split up vertically (layer-level) across multiple GPUs, so that only one or \nseveral layers of the model are placed on a single GPU. Each GPU processes in parallel different stages of the pipeline \nand working on a small chunk of the batch. Learn more about how PipelineParallel works [here](perf_train_gpu_many#from-naive-model-parallelism-to-pipeline-parallelism).\n\n### pixel values\n\nA tensor of the numerical representations of an image that is passed to a model. The pixel values have a shape of [`batch_size`, `num_channels`, `height`, `width`], and are generated from an image processor.\n\n### pooling\n\nAn operation that reduces a matrix into a smaller matrix, either by taking the maximum or average of the pooled dimension(s). Pooling layers are commonly found between convolutional layers to downsample the feature representation.\n\n### position IDs\n\nContrary to RNNs that have the position of each token embedded within them, transformers are unaware of the position of\neach token. Therefore, the position IDs (`position_ids`) are used by the model to identify each token's position in the\nlist of tokens.\n\nThey are an optional parameter. If no `position_ids` are passed to the model, the IDs are automatically created as\nabsolute positional embeddings.\n\nAbsolute positional embeddings are selected in the range `[0, config.max_position_embeddings - 1]`. Some models use\nother types of positional embeddings, such as sinusoidal position embeddings or relative position embeddings.\n\n### preprocessing\n\nThe task of preparing raw data into a format that can be easily consumed by machine learning models. For example, text is typically preprocessed by tokenization. To gain a better idea of what preprocessing looks like for other input types, check out the [Preprocess](https://huggingface.co/docs/transformers/preprocessing) tutorial.\n\n### pretrained model\n\nA model that has been pretrained on some data (for instance all of Wikipedia)."}


