# embeddings_queue

created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: bf7edf5a-5e48-4599-b06b-8d9f1e876136
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/glossary/", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"bf7edf5a-5e48-4599-b06b-8d9f1e876136\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary/\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d163ba88-0765-467b-b349-0214ba8f97c2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary/\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"c251a0702ad97d16ec8329c65cebe21f3fb572cd0bc6998a445f16e90d04cb51\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"92a1f2e8-3686-4706-b8d8-7b201255ffc5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary/\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"1399cad008ccf34db659454ffd79a9fc03cdefc10d30854c06f0160e13ca3b2f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"843d5eb4-6900-44cf-8b34-7f300b54bb41\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c3a7a4e15a3260f545c0a7a29a610fb2c510e14501dff3997f8930c8ac5ddd62\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 16362, \"end_char_idx\": 19966, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d163ba88-0765-467b-b349-0214ba8f97c2", "doc_id": "d163ba88-0765-467b-b349-0214ba8f97c2", "ref_doc_id": "d163ba88-0765-467b-b349-0214ba8f97c2", "chroma:document": "Each GPU processes in parallel different stages of the pipeline \nand working on a small chunk of the batch. Learn more about how PipelineParallel works [here](perf_train_gpu_many#from-naive-model-parallelism-to-pipeline-parallelism).\n\n### pixel values\n\nA tensor of the numerical representations of an image that is passed to a model. The pixel values have a shape of [`batch_size`, `num_channels`, `height`, `width`], and are generated from an image processor.\n\n### pooling\n\nAn operation that reduces a matrix into a smaller matrix, either by taking the maximum or average of the pooled dimension(s). Pooling layers are commonly found between convolutional layers to downsample the feature representation.\n\n### position IDs\n\nContrary to RNNs that have the position of each token embedded within them, transformers are unaware of the position of\neach token. Therefore, the position IDs (`position_ids`) are used by the model to identify each token's position in the\nlist of tokens.\n\nThey are an optional parameter. If no `position_ids` are passed to the model, the IDs are automatically created as\nabsolute positional embeddings.\n\nAbsolute positional embeddings are selected in the range `[0, config.max_position_embeddings - 1]`. Some models use\nother types of positional embeddings, such as sinusoidal position embeddings or relative position embeddings.\n\n### preprocessing\n\nThe task of preparing raw data into a format that can be easily consumed by machine learning models. For example, text is typically preprocessed by tokenization. To gain a better idea of what preprocessing looks like for other input types, check out the [Preprocess](https://huggingface.co/docs/transformers/preprocessing) tutorial.\n\n### pretrained model\n\nA model that has been pretrained on some data (for instance all of Wikipedia). Pretraining methods involve a\nself-supervised objective, which can be reading the text and trying to predict the next word (see [causal language\nmodeling](#causal-language-modeling)) or masking some words and trying to predict them (see [masked language\nmodeling](#masked-language-modeling-mlm)). \n\nSpeech and vision models have their own pretraining objectives. For example, Wav2Vec2 is a speech model pretrained on a contrastive task which requires the model to identify the \"true\" speech representation from a set of \"false\" speech representations. On the other hand, BEiT is a vision model pretrained on a masked image modeling task which masks some of the image patches and requires the model to predict the masked patches (similar to the masked language modeling objective).\n\n## R\n\n### recurrent neural network (RNN)\n\nA type of model that uses a loop over a layer to process texts.\n\n### representation learning\n\nA subfield of machine learning which focuses on learning meaningful representations of raw data. Some examples of representation learning techniques include word embeddings, autoencoders, and Generative Adversarial Networks (GANs).\n\n## S\n\n### sampling rate\n\nA measurement in hertz of the number of samples (the audio signal) taken per second. The sampling rate is a result of discretizing a continuous signal such as speech.\n\n### self-attention\n\nEach element of the input finds out which other elements of the input they should attend to.\n\n### self-supervised learning \n\nA category of machine learning techniques in which a model creates its own learning objective from unlabeled data. It differs from [unsupervised learning](#unsupervised-learning) and [supervised learning](#supervised-learning) in that the learning process is supervised, but not explicitly from the user."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 843d5eb4-6900-44cf-8b34-7f300b54bb41
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/glossary/", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"843d5eb4-6900-44cf-8b34-7f300b54bb41\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary/\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d163ba88-0765-467b-b349-0214ba8f97c2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary/\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"c251a0702ad97d16ec8329c65cebe21f3fb572cd0bc6998a445f16e90d04cb51\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bf7edf5a-5e48-4599-b06b-8d9f1e876136\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary/\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"150c7b19516bef7a0913d284aea3291f29f0944d169b47332e4b671aa357f6ae\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0fd7e278-5142-47c5-b0c1-ec110b5b2871\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"84cf497223feeb5b11ef3bc6a703d790e376496aed1cab97791ff4b9be76a373\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 18072, \"end_char_idx\": 21661, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d163ba88-0765-467b-b349-0214ba8f97c2", "doc_id": "d163ba88-0765-467b-b349-0214ba8f97c2", "ref_doc_id": "d163ba88-0765-467b-b349-0214ba8f97c2", "chroma:document": "### pretrained model\n\nA model that has been pretrained on some data (for instance all of Wikipedia). Pretraining methods involve a\nself-supervised objective, which can be reading the text and trying to predict the next word (see [causal language\nmodeling](#causal-language-modeling)) or masking some words and trying to predict them (see [masked language\nmodeling](#masked-language-modeling-mlm)). \n\nSpeech and vision models have their own pretraining objectives. For example, Wav2Vec2 is a speech model pretrained on a contrastive task which requires the model to identify the \"true\" speech representation from a set of \"false\" speech representations. On the other hand, BEiT is a vision model pretrained on a masked image modeling task which masks some of the image patches and requires the model to predict the masked patches (similar to the masked language modeling objective).\n\n## R\n\n### recurrent neural network (RNN)\n\nA type of model that uses a loop over a layer to process texts.\n\n### representation learning\n\nA subfield of machine learning which focuses on learning meaningful representations of raw data. Some examples of representation learning techniques include word embeddings, autoencoders, and Generative Adversarial Networks (GANs).\n\n## S\n\n### sampling rate\n\nA measurement in hertz of the number of samples (the audio signal) taken per second. The sampling rate is a result of discretizing a continuous signal such as speech.\n\n### self-attention\n\nEach element of the input finds out which other elements of the input they should attend to.\n\n### self-supervised learning \n\nA category of machine learning techniques in which a model creates its own learning objective from unlabeled data. It differs from [unsupervised learning](#unsupervised-learning) and [supervised learning](#supervised-learning) in that the learning process is supervised, but not explicitly from the user. \n\nOne example of self-supervised learning is [masked language modeling](#masked-language-modeling-mlm), where a model is passed sentences with a proportion of its tokens removed and learns to predict the missing tokens.\n\n### semi-supervised learning\n\nA broad category of machine learning training techniques that leverages a small amount of labeled data with a larger quantity of unlabeled data to improve the accuracy of a model, unlike [supervised learning](#supervised-learning) and [unsupervised learning](#unsupervised-learning).\n\nAn example of a semi-supervised learning approach is \"self-training\", in which a model is trained on labeled data, and then used to make predictions on the unlabeled data. The portion of the unlabeled data that the model predicts with the most confidence gets added to the labeled dataset and used to retrain the model.\n\n### sequence-to-sequence (seq2seq)\n\nModels that generate a new sequence from an input, like translation models, or summarization models (such as\n[Bart](model_doc/bart) or [T5](model_doc/t5)).\n\n### Sharded DDP\n\nAnother name for the foundational [ZeRO](#zero-redundancy-optimizer-zero) concept as used by various other implementations of ZeRO.\n\n### stride\n\nIn [convolution](#convolution) or [pooling](#pooling), the stride refers to the distance the kernel is moved over a matrix. A stride of 1 means the kernel is moved one pixel over at a time, and a stride of 2 means the kernel is moved two pixels over at a time.\n\n### supervised learning\n\nA form of model training that directly uses labeled data to correct and instruct model performance. Data is fed into the model being trained, and its predictions are compared to the known labels."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 0fd7e278-5142-47c5-b0c1-ec110b5b2871
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/glossary/", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"0fd7e278-5142-47c5-b0c1-ec110b5b2871\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary/\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d163ba88-0765-467b-b349-0214ba8f97c2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary/\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"c251a0702ad97d16ec8329c65cebe21f3fb572cd0bc6998a445f16e90d04cb51\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"843d5eb4-6900-44cf-8b34-7f300b54bb41\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary/\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"7780203afc84966769520978d3ffb7426e58085defdf9b697f66142e67c24fc8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8adc34ac-e818-44bc-8299-be53b54ba51f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8bb2e16160716b62be4ddb3f60bd6c01d50330634fa4e43a5753e4b5497d2c3e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 19969, \"end_char_idx\": 23395, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d163ba88-0765-467b-b349-0214ba8f97c2", "doc_id": "d163ba88-0765-467b-b349-0214ba8f97c2", "ref_doc_id": "d163ba88-0765-467b-b349-0214ba8f97c2", "chroma:document": "One example of self-supervised learning is [masked language modeling](#masked-language-modeling-mlm), where a model is passed sentences with a proportion of its tokens removed and learns to predict the missing tokens.\n\n### semi-supervised learning\n\nA broad category of machine learning training techniques that leverages a small amount of labeled data with a larger quantity of unlabeled data to improve the accuracy of a model, unlike [supervised learning](#supervised-learning) and [unsupervised learning](#unsupervised-learning).\n\nAn example of a semi-supervised learning approach is \"self-training\", in which a model is trained on labeled data, and then used to make predictions on the unlabeled data. The portion of the unlabeled data that the model predicts with the most confidence gets added to the labeled dataset and used to retrain the model.\n\n### sequence-to-sequence (seq2seq)\n\nModels that generate a new sequence from an input, like translation models, or summarization models (such as\n[Bart](model_doc/bart) or [T5](model_doc/t5)).\n\n### Sharded DDP\n\nAnother name for the foundational [ZeRO](#zero-redundancy-optimizer-zero) concept as used by various other implementations of ZeRO.\n\n### stride\n\nIn [convolution](#convolution) or [pooling](#pooling), the stride refers to the distance the kernel is moved over a matrix. A stride of 1 means the kernel is moved one pixel over at a time, and a stride of 2 means the kernel is moved two pixels over at a time.\n\n### supervised learning\n\nA form of model training that directly uses labeled data to correct and instruct model performance. Data is fed into the model being trained, and its predictions are compared to the known labels. The model updates its weights based on how incorrect its predictions were, and the process is repeated to optimize model performance.\n\n## T\n\n### Tensor Parallelism (TP)\n\nParallelism technique for training on multiple GPUs in which each tensor is split up into multiple chunks, so instead of \nhaving the whole tensor reside on a single GPU, each shard of the tensor resides on its designated GPU. Shards gets \nprocessed separately and in parallel on different GPUs and the results are synced at the end of the processing step. \nThis is what is sometimes called horizontal parallelism, as the splitting happens on horizontal level.\nLearn more about Tensor Parallelism [here](perf_train_gpu_many#tensor-parallelism).\n\n### token\n\nA part of a sentence, usually a word, but can also be a subword (non-common words are often split in subwords) or a\npunctuation symbol.\n\n### token Type IDs\n\nSome models' purpose is to do classification on pairs of sentences or question answering.\n\n<Youtube id=\"0u3ioSwev3s\"/>\n\nThese require two different sequences to be joined in a single \"input_ids\" entry, which usually is performed with the\nhelp of special tokens, such as the classifier (`[CLS]`) and separator (`[SEP]`) tokens. For example, the BERT model\nbuilds its two sequence input as such:\n\n```python\n>>> # [CLS] SEQUENCE_A [SEP] SEQUENCE_B [SEP]\n```\n\nWe can use our tokenizer to automatically generate such a sentence by passing the two sequences to `tokenizer` as two\narguments (and not a list, like before) like this:\n\n```python\n>>> from transformers import BertTokenizer\n\n>>> tokenizer = BertTokenizer.from_pretrained(\"google-bert/bert-base-cased\")\n>>> sequence_a = \"HuggingFace is based in NYC\"\n>>> sequence_b = \"Where is HuggingFace based?\""}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 8adc34ac-e818-44bc-8299-be53b54ba51f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/glossary/", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"8adc34ac-e818-44bc-8299-be53b54ba51f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary/\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d163ba88-0765-467b-b349-0214ba8f97c2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary/\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"c251a0702ad97d16ec8329c65cebe21f3fb572cd0bc6998a445f16e90d04cb51\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0fd7e278-5142-47c5-b0c1-ec110b5b2871\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary/\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5263754338195b3f19748331ed6b69dd6ab5410d73f90933219b3f0f666af692\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a633150c-62e0-44d7-b5e1-e1bb8a38e420\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fde4c4570d7f72e7333237d6ca27730b8b3d4e54b6069a8bd84acfac5f481a8f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 21662, \"end_char_idx\": 25025, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d163ba88-0765-467b-b349-0214ba8f97c2", "doc_id": "d163ba88-0765-467b-b349-0214ba8f97c2", "ref_doc_id": "d163ba88-0765-467b-b349-0214ba8f97c2", "chroma:document": "The model updates its weights based on how incorrect its predictions were, and the process is repeated to optimize model performance.\n\n## T\n\n### Tensor Parallelism (TP)\n\nParallelism technique for training on multiple GPUs in which each tensor is split up into multiple chunks, so instead of \nhaving the whole tensor reside on a single GPU, each shard of the tensor resides on its designated GPU. Shards gets \nprocessed separately and in parallel on different GPUs and the results are synced at the end of the processing step. \nThis is what is sometimes called horizontal parallelism, as the splitting happens on horizontal level.\nLearn more about Tensor Parallelism [here](perf_train_gpu_many#tensor-parallelism).\n\n### token\n\nA part of a sentence, usually a word, but can also be a subword (non-common words are often split in subwords) or a\npunctuation symbol.\n\n### token Type IDs\n\nSome models' purpose is to do classification on pairs of sentences or question answering.\n\n<Youtube id=\"0u3ioSwev3s\"/>\n\nThese require two different sequences to be joined in a single \"input_ids\" entry, which usually is performed with the\nhelp of special tokens, such as the classifier (`[CLS]`) and separator (`[SEP]`) tokens. For example, the BERT model\nbuilds its two sequence input as such:\n\n```python\n>>> # [CLS] SEQUENCE_A [SEP] SEQUENCE_B [SEP]\n```\n\nWe can use our tokenizer to automatically generate such a sentence by passing the two sequences to `tokenizer` as two\narguments (and not a list, like before) like this:\n\n```python\n>>> from transformers import BertTokenizer\n\n>>> tokenizer = BertTokenizer.from_pretrained(\"google-bert/bert-base-cased\")\n>>> sequence_a = \"HuggingFace is based in NYC\"\n>>> sequence_b = \"Where is HuggingFace based?\"\n\n>>> encoded_dict = tokenizer(sequence_a, sequence_b)\n>>> decoded = tokenizer.decode(encoded_dict[\"input_ids\"])\n```\n\nwhich will return:\n\n```python\n>>> print(decoded)\n[CLS] HuggingFace is based in NYC [SEP] Where is HuggingFace based? [SEP]\n```\n\nThis is enough for some models to understand where one sequence ends and where another begins. However, other models,\nsuch as BERT, also deploy token type IDs (also called segment IDs). They are represented as a binary mask identifying\nthe two types of sequence in the model.\n\nThe tokenizer returns this mask as the \"token_type_ids\" entry:\n\n```python\n>>> encoded_dict[\"token_type_ids\"]\n[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n```\n\nThe first sequence, the \"context\" used for the question, has all its tokens represented by a `0`, whereas the second\nsequence, corresponding to the \"question\", has all its tokens represented by a `1`.\n\nSome models, like [`XLNetModel`] use an additional token represented by a `2`.\n\n### transfer learning\n\nA technique that involves taking a pretrained model and adapting it to a dataset specific to your task. Instead of training a model from scratch, you can leverage knowledge obtained from an existing model as a starting point. This speeds up the learning process and reduces the amount of training data needed.\n\n### transformer\n\nSelf-attention based deep learning model architecture.\n\n## U\n\n### unsupervised learning\n\nA form of model training in which data provided to the model is not labeled. Unsupervised learning techniques leverage statistical information of the data distribution to find patterns useful for the task at hand."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: a633150c-62e0-44d7-b5e1-e1bb8a38e420
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/glossary/", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"a633150c-62e0-44d7-b5e1-e1bb8a38e420\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary/\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d163ba88-0765-467b-b349-0214ba8f97c2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary/\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"c251a0702ad97d16ec8329c65cebe21f3fb572cd0bc6998a445f16e90d04cb51\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8adc34ac-e818-44bc-8299-be53b54ba51f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/glossary/\", \"title\": \"Glossary\", \"tokens\": 5921, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"245ea591010479a26381395afd1065bed0c9c35bd9b714023bbbb1b76462eec3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 23397, \"end_char_idx\": 25502, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d163ba88-0765-467b-b349-0214ba8f97c2", "doc_id": "d163ba88-0765-467b-b349-0214ba8f97c2", "ref_doc_id": "d163ba88-0765-467b-b349-0214ba8f97c2", "chroma:document": ">>> encoded_dict = tokenizer(sequence_a, sequence_b)\n>>> decoded = tokenizer.decode(encoded_dict[\"input_ids\"])\n```\n\nwhich will return:\n\n```python\n>>> print(decoded)\n[CLS] HuggingFace is based in NYC [SEP] Where is HuggingFace based? [SEP]\n```\n\nThis is enough for some models to understand where one sequence ends and where another begins. However, other models,\nsuch as BERT, also deploy token type IDs (also called segment IDs). They are represented as a binary mask identifying\nthe two types of sequence in the model.\n\nThe tokenizer returns this mask as the \"token_type_ids\" entry:\n\n```python\n>>> encoded_dict[\"token_type_ids\"]\n[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n```\n\nThe first sequence, the \"context\" used for the question, has all its tokens represented by a `0`, whereas the second\nsequence, corresponding to the \"question\", has all its tokens represented by a `1`.\n\nSome models, like [`XLNetModel`] use an additional token represented by a `2`.\n\n### transfer learning\n\nA technique that involves taking a pretrained model and adapting it to a dataset specific to your task. Instead of training a model from scratch, you can leverage knowledge obtained from an existing model as a starting point. This speeds up the learning process and reduces the amount of training data needed.\n\n### transformer\n\nSelf-attention based deep learning model architecture.\n\n## U\n\n### unsupervised learning\n\nA form of model training in which data provided to the model is not labeled. Unsupervised learning techniques leverage statistical information of the data distribution to find patterns useful for the task at hand.\n\n## Z\n\n### Zero Redundancy Optimizer (ZeRO)\n\nParallelism technique which performs sharding of the tensors somewhat similar to [TensorParallel](#tensor-parallelism-tp), \nexcept the whole tensor gets reconstructed in time for a forward or backward computation, therefore the model doesn't need \nto be modified. This method also supports various offloading techniques to compensate for limited GPU memory. \nLearn more about ZeRO [here](perf_train_gpu_many#zero-data-parallelism)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 14890659-7424-40ef-a9a8-6056181a881b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tf_xla/", "title": "XLA Integration for TensorFlow Models", "tokens": 2215, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"14890659-7424-40ef-a9a8-6056181a881b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tf_xla/\", \"title\": \"XLA Integration for TensorFlow Models\", \"tokens\": 2215, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7e99c004-d533-455d-afcc-f51420fecb6d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tf_xla/\", \"title\": \"XLA Integration for TensorFlow Models\", \"tokens\": 2215, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"c296e6e1692cf8b10862e4415724abc48c7753fd2e153b313d877b94c6089e73\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c086fa41-ac90-4079-93df-d3d6a4df7437\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"cb1acb123801198053de1788131088b2385d44d17b82393016640fdedd49f18c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3326, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7e99c004-d533-455d-afcc-f51420fecb6d", "doc_id": "7e99c004-d533-455d-afcc-f51420fecb6d", "ref_doc_id": "7e99c004-d533-455d-afcc-f51420fecb6d", "chroma:document": "# XLA Integration for TensorFlow Models\n\n[[open-in-colab]]\n\nAccelerated Linear Algebra, dubbed XLA, is a compiler for accelerating the runtime of TensorFlow Models. From the [official documentation](https://www.tensorflow.org/xla):\n\nXLA (Accelerated Linear Algebra) is a domain-specific compiler for linear algebra that can accelerate TensorFlow models with potentially no source code changes.\n\nUsing XLA in TensorFlow is simple \u2013 it comes packaged inside the `tensorflow` library, and it can be triggered with the `jit_compile` argument in any graph-creating function such as [`tf.function`](https://www.tensorflow.org/guide/intro_to_graphs). When using Keras methods like `fit()` and `predict()`, you can enable XLA simply by passing the `jit_compile` argument to `model.compile()`. However, XLA is not limited to these methods - it can also be used to accelerate any arbitrary `tf.function`.\n\nSeveral TensorFlow methods in \ud83e\udd17 Transformers have been rewritten to be XLA-compatible, including text generation for models such as [GPT2](https://huggingface.co/docs/transformers/model_doc/gpt2), [T5](https://huggingface.co/docs/transformers/model_doc/t5) and [OPT](https://huggingface.co/docs/transformers/model_doc/opt), as well as speech processing for models such as [Whisper](https://huggingface.co/docs/transformers/model_doc/whisper).\n\nWhile the exact amount of speed-up is very much model-dependent, for TensorFlow text generation models inside \ud83e\udd17 Transformers, we noticed a speed-up of ~100x. This document will explain how you can use XLA for these models to get the maximum amount of performance. We\u2019ll also provide links to additional resources if you\u2019re interested to learn more about the benchmarks and our design philosophy behind the XLA integration.\n\n## Running TF functions with XLA\n\nLet us consider the following model in TensorFlow:\n\n```py\nimport tensorflow as tf\n\nmodel = tf.keras.Sequential(\n    [tf.keras.layers.Dense(10, input_shape=(10,), activation=\"relu\"), tf.keras.layers.Dense(5, activation=\"softmax\")]\n)\n```\n\nThe above model accepts inputs having a dimension of `(10, )`. We can use the model for running a forward pass like so:\n\n```py\n# Generate random inputs for the model.\nbatch_size = 16\ninput_vector_dim = 10\nrandom_inputs = tf.random.normal((batch_size, input_vector_dim))\n\n# Run a forward pass.\n_ = model(random_inputs)\n```\n\nIn order to run the forward pass with an XLA-compiled function, we\u2019d need to do:\n\n```py\nxla_fn = tf.function(model, jit_compile=True)\n_ = xla_fn(random_inputs)\n```\n\nThe default `call()` function of the `model` is used for compiling the XLA graph. But if there\u2019s any other model function you want to compile into XLA that\u2019s also possible with:\n\n```py\nmy_xla_fn = tf.function(model.my_xla_fn, jit_compile=True)\n```\n\n## Running a TF text generation model with XLA from \ud83e\udd17 Transformers\n\nTo enable XLA-accelerated generation within \ud83e\udd17 Transformers, you need to have a recent version of `transformers` installed. You can install it by running:\n\n```bash\npip install transformers --upgrade\n```\n\nAnd then you can run the following code:\n\n```py\nimport tensorflow as tf\nfrom transformers import AutoTokenizer, TFAutoModelForCausalLM\n\n# Will error if the minimal version of Transformers is not installed.\nfrom transformers.utils import check_min_version\n\ncheck_min_version(\"4.21.0\")"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: c086fa41-ac90-4079-93df-d3d6a4df7437
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tf_xla/", "title": "XLA Integration for TensorFlow Models", "tokens": 2215, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"c086fa41-ac90-4079-93df-d3d6a4df7437\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tf_xla/\", \"title\": \"XLA Integration for TensorFlow Models\", \"tokens\": 2215, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7e99c004-d533-455d-afcc-f51420fecb6d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tf_xla/\", \"title\": \"XLA Integration for TensorFlow Models\", \"tokens\": 2215, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"c296e6e1692cf8b10862e4415724abc48c7753fd2e153b313d877b94c6089e73\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"14890659-7424-40ef-a9a8-6056181a881b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tf_xla/\", \"title\": \"XLA Integration for TensorFlow Models\", \"tokens\": 2215, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"bb15f032a80bc7f40af3893e8ba6d6e07cd7aaafc9eff7e36cafdac6a2785f57\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3fbbc840-435c-469d-a8be-f63e1125b3da\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"cfbf756928d6bb56c92e13fa29acca586645e08f5d7727ab763d5f3ae43eef82\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3329, \"end_char_idx\": 6144, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7e99c004-d533-455d-afcc-f51420fecb6d", "doc_id": "7e99c004-d533-455d-afcc-f51420fecb6d", "ref_doc_id": "7e99c004-d533-455d-afcc-f51420fecb6d", "chroma:document": "tokenizer = AutoTokenizer.from_pretrained(\"openai-community/gpt2\", padding_side=\"left\", pad_token=\"</s>\")\nmodel = TFAutoModelForCausalLM.from_pretrained(\"openai-community/gpt2\")\ninput_string = [\"TensorFlow is\"]\n\n# One line to create an XLA generation function\nxla_generate = tf.function(model.generate, jit_compile=True)\n\ntokenized_input = tokenizer(input_string, return_tensors=\"tf\")\ngenerated_tokens = xla_generate(**tokenized_input, num_beams=2)\n\ndecoded_text = tokenizer.decode(generated_tokens[0], skip_special_tokens=True)\nprint(f\"Generated -- {decoded_text}\")\n# Generated -- TensorFlow is an open-source, open-source, distributed-source application # framework for the\n```\n\nAs you can notice, enabling XLA on `generate()` is just a single line of code. The rest of the code remains unchanged. However, there are a couple of gotchas in the above code snippet that are specific to XLA. You need to be aware of those to realize the speed-ups that XLA can bring in. We discuss these in the following section. \n\n## Gotchas to be aware of\n\nWhen you are executing an XLA-enabled function (like `xla_generate()` above) for the first time, it will internally try to infer the computation graph, which is time-consuming.  This process is known as [\u201ctracing\u201d](https://www.tensorflow.org/guide/intro_to_graphs#when_is_a_function_tracing). \n\nYou might notice that the generation time is not fast. Successive calls of `xla_generate()` (or any other XLA-enabled function) won\u2019t have to infer the computation graph, given the inputs to the function follow the same shape with which the computation graph was initially built. While this is not a problem for modalities with fixed input shapes (e.g., images), you must pay attention if you are working with variable input shape modalities (e.g., text).\n\nTo ensure `xla_generate()` always operates with the same input shapes, you can specify the `padding` arguments when calling the tokenizer. \n\n```py\nimport tensorflow as tf\nfrom transformers import AutoTokenizer, TFAutoModelForCausalLM\n\ntokenizer = AutoTokenizer.from_pretrained(\"openai-community/gpt2\", padding_side=\"left\", pad_token=\"</s>\")\nmodel = TFAutoModelForCausalLM.from_pretrained(\"openai-community/gpt2\")\ninput_string = [\"TensorFlow is\"]\n\nxla_generate = tf.function(model.generate, jit_compile=True)\n\n# Here, we call the tokenizer with padding options.\ntokenized_input = tokenizer(input_string, pad_to_multiple_of=8, padding=True, return_tensors=\"tf\")\n\ngenerated_tokens = xla_generate(**tokenized_input, num_beams=2)\ndecoded_text = tokenizer.decode(generated_tokens[0], skip_special_tokens=True)\nprint(f\"Generated -- {decoded_text}\")\n```\n\nThis way, you can ensure that the inputs to `xla_generate()` will always receive inputs with the shape it was traced with and thus leading to speed-ups in the generation time."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 3fbbc840-435c-469d-a8be-f63e1125b3da
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tf_xla/", "title": "XLA Integration for TensorFlow Models", "tokens": 2215, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"3fbbc840-435c-469d-a8be-f63e1125b3da\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tf_xla/\", \"title\": \"XLA Integration for TensorFlow Models\", \"tokens\": 2215, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7e99c004-d533-455d-afcc-f51420fecb6d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tf_xla/\", \"title\": \"XLA Integration for TensorFlow Models\", \"tokens\": 2215, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"c296e6e1692cf8b10862e4415724abc48c7753fd2e153b313d877b94c6089e73\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c086fa41-ac90-4079-93df-d3d6a4df7437\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tf_xla/\", \"title\": \"XLA Integration for TensorFlow Models\", \"tokens\": 2215, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"e1ee66f74b5a8e7996f3a2e3bc4f0146e41bb581830eb3d16f823e3fc73bef59\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f42bc9bb-2031-4ff7-9aed-72c920aa2c14\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"cc48ceb3ce183e859398a95b15f5e5efb054174a2d033fd3fb6da2477579bb88\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4548, \"end_char_idx\": 7631, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7e99c004-d533-455d-afcc-f51420fecb6d", "doc_id": "7e99c004-d533-455d-afcc-f51420fecb6d", "ref_doc_id": "7e99c004-d533-455d-afcc-f51420fecb6d", "chroma:document": "This process is known as [\u201ctracing\u201d](https://www.tensorflow.org/guide/intro_to_graphs#when_is_a_function_tracing). \n\nYou might notice that the generation time is not fast. Successive calls of `xla_generate()` (or any other XLA-enabled function) won\u2019t have to infer the computation graph, given the inputs to the function follow the same shape with which the computation graph was initially built. While this is not a problem for modalities with fixed input shapes (e.g., images), you must pay attention if you are working with variable input shape modalities (e.g., text).\n\nTo ensure `xla_generate()` always operates with the same input shapes, you can specify the `padding` arguments when calling the tokenizer. \n\n```py\nimport tensorflow as tf\nfrom transformers import AutoTokenizer, TFAutoModelForCausalLM\n\ntokenizer = AutoTokenizer.from_pretrained(\"openai-community/gpt2\", padding_side=\"left\", pad_token=\"</s>\")\nmodel = TFAutoModelForCausalLM.from_pretrained(\"openai-community/gpt2\")\ninput_string = [\"TensorFlow is\"]\n\nxla_generate = tf.function(model.generate, jit_compile=True)\n\n# Here, we call the tokenizer with padding options.\ntokenized_input = tokenizer(input_string, pad_to_multiple_of=8, padding=True, return_tensors=\"tf\")\n\ngenerated_tokens = xla_generate(**tokenized_input, num_beams=2)\ndecoded_text = tokenizer.decode(generated_tokens[0], skip_special_tokens=True)\nprint(f\"Generated -- {decoded_text}\")\n```\n\nThis way, you can ensure that the inputs to `xla_generate()` will always receive inputs with the shape it was traced with and thus leading to speed-ups in the generation time. You can verify this with the code below:\n\n```py\nimport time\nimport tensorflow as tf\nfrom transformers import AutoTokenizer, TFAutoModelForCausalLM\n\ntokenizer = AutoTokenizer.from_pretrained(\"openai-community/gpt2\", padding_side=\"left\", pad_token=\"</s>\")\nmodel = TFAutoModelForCausalLM.from_pretrained(\"openai-community/gpt2\")\n\nxla_generate = tf.function(model.generate, jit_compile=True)\n\nfor input_string in [\"TensorFlow is\", \"TensorFlow is a\", \"TFLite is a\"]:\n    tokenized_input = tokenizer(input_string, pad_to_multiple_of=8, padding=True, return_tensors=\"tf\")\n    start = time.time_ns()\n    generated_tokens = xla_generate(**tokenized_input, num_beams=2)\n    end = time.time_ns()\n    print(f\"Execution time -- {(end - start) / 1e6:.1f} ms\\n\")\n```\n\nOn a Tesla T4 GPU, you can expect the outputs like so:\n\n```bash\nExecution time -- 30819.6 ms\n\nExecution time -- 79.0 ms\n\nExecution time -- 78.9 ms\n```\nThe first call to `xla_generate()` is time-consuming because of tracing, but the successive calls are orders of magnitude faster. Keep in mind that any change in the generation options at any point with trigger re-tracing and thus leading to slow-downs in the generation time. \n\nWe didn\u2019t cover all the text generation options \ud83e\udd17 Transformers provides in this document. We encourage you to read the documentation for advanced use cases.\n\n## Additional Resources\n\nHere, we leave you with some additional resources if you want to delve deeper into XLA in \ud83e\udd17 Transformers and in general."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: f42bc9bb-2031-4ff7-9aed-72c920aa2c14
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tf_xla/", "title": "XLA Integration for TensorFlow Models", "tokens": 2215, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"f42bc9bb-2031-4ff7-9aed-72c920aa2c14\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tf_xla/\", \"title\": \"XLA Integration for TensorFlow Models\", \"tokens\": 2215, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7e99c004-d533-455d-afcc-f51420fecb6d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tf_xla/\", \"title\": \"XLA Integration for TensorFlow Models\", \"tokens\": 2215, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"c296e6e1692cf8b10862e4415724abc48c7753fd2e153b313d877b94c6089e73\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3fbbc840-435c-469d-a8be-f63e1125b3da\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tf_xla/\", \"title\": \"XLA Integration for TensorFlow Models\", \"tokens\": 2215, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b10ed11c475459d418a329d3756ad7f09ed9f4f232e8c359695b9a7db175d1bc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6145, \"end_char_idx\": 8819, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7e99c004-d533-455d-afcc-f51420fecb6d", "doc_id": "7e99c004-d533-455d-afcc-f51420fecb6d", "ref_doc_id": "7e99c004-d533-455d-afcc-f51420fecb6d", "chroma:document": "You can verify this with the code below:\n\n```py\nimport time\nimport tensorflow as tf\nfrom transformers import AutoTokenizer, TFAutoModelForCausalLM\n\ntokenizer = AutoTokenizer.from_pretrained(\"openai-community/gpt2\", padding_side=\"left\", pad_token=\"</s>\")\nmodel = TFAutoModelForCausalLM.from_pretrained(\"openai-community/gpt2\")\n\nxla_generate = tf.function(model.generate, jit_compile=True)\n\nfor input_string in [\"TensorFlow is\", \"TensorFlow is a\", \"TFLite is a\"]:\n    tokenized_input = tokenizer(input_string, pad_to_multiple_of=8, padding=True, return_tensors=\"tf\")\n    start = time.time_ns()\n    generated_tokens = xla_generate(**tokenized_input, num_beams=2)\n    end = time.time_ns()\n    print(f\"Execution time -- {(end - start) / 1e6:.1f} ms\\n\")\n```\n\nOn a Tesla T4 GPU, you can expect the outputs like so:\n\n```bash\nExecution time -- 30819.6 ms\n\nExecution time -- 79.0 ms\n\nExecution time -- 78.9 ms\n```\nThe first call to `xla_generate()` is time-consuming because of tracing, but the successive calls are orders of magnitude faster. Keep in mind that any change in the generation options at any point with trigger re-tracing and thus leading to slow-downs in the generation time. \n\nWe didn\u2019t cover all the text generation options \ud83e\udd17 Transformers provides in this document. We encourage you to read the documentation for advanced use cases.\n\n## Additional Resources\n\nHere, we leave you with some additional resources if you want to delve deeper into XLA in \ud83e\udd17 Transformers and in general. \n \n* [This Colab Notebook](https://colab.research.google.com/github/huggingface/blog/blob/main/notebooks/91_tf_xla_generate.ipynb) provides an interactive demonstration if you want to fiddle with the XLA-compatible encoder-decoder (like [T5](https://huggingface.co/docs/transformers/model_doc/t5)) and decoder-only (like [GPT2](https://huggingface.co/docs/transformers/model_doc/gpt2)) text generation models. \n* [This blog post](https://huggingface.co/blog/tf-xla-generate) provides an overview of the comparison benchmarks for XLA-compatible models along with a friendly introduction to XLA in TensorFlow. \n* [This blog post](https://blog.tensorflow.org/2022/11/how-hugging-face-improved-text-generation-performance-with-xla.html) discusses our design philosophy behind adding XLA support to the TensorFlow models in \ud83e\udd17 Transformers. \n* Recommended posts for learning more about XLA and TensorFlow graphs in general:\n    * [XLA: Optimizing Compiler for Machine Learning](https://www.tensorflow.org/xla)\n    * [Introduction to graphs and tf.function](https://www.tensorflow.org/guide/intro_to_graphs)\n    * [Better performance with tf.function](https://www.tensorflow.org/guide/function)"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 78cc7d0e-02df-4973-9909-5adf51da48b2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/llm_optims/", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"78cc7d0e-02df-4973-9909-5adf51da48b2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_optims/\", \"title\": \"LLM inference optimization\", \"tokens\": 4298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2b48628a-751d-4860-850f-fe9c7ff43443\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_optims/\", \"title\": \"LLM inference optimization\", \"tokens\": 4298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5e989f3e75139db0c03329e4a2eeabb974a2ca02fb5f3ff905ef61b01914e6af\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"245768bf-9753-4037-b296-e42f2a59a7f6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1420a39d4a15d3485de9266e9b857d1860c108999adaa6fc8a910a87a417a93e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3304, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2b48628a-751d-4860-850f-fe9c7ff43443", "doc_id": "2b48628a-751d-4860-850f-fe9c7ff43443", "ref_doc_id": "2b48628a-751d-4860-850f-fe9c7ff43443", "chroma:document": "# LLM inference optimization\n\nLarge language models (LLMs) have pushed text generation applications, such as chat and code completion models, to the next level by producing text that displays a high level of understanding and fluency. But what makes LLMs so powerful - namely their size - also presents challenges for inference.\n\nBasic inference is slow because LLMs have to be called repeatedly to generate the next token. The input sequence increases as generation progresses, which takes longer and longer for the LLM to process. LLMs also have billions of parameters, making it a challenge to store and handle all those weights in memory.\n\nThis guide will show you how to use the optimization techniques available in Transformers to accelerate LLM inference.\n\n> [!TIP]\n> Hugging Face also provides [Text Generation Inference (TGI)](https://hf.co/docs/text-generation-inference), a library dedicated to deploying and serving highly optimized LLMs for inference. It includes more optimization features not included in Transformers, such as continuous batching for increasing throughput and tensor parallelism for multi-GPU inference.\n\n## Static kv-cache and torch.compile\n\nDuring decoding, a LLM computes the key-value (kv) values for each input token and since it is autoregressive, it computes the same kv values each time because the generated output becomes part of the input now. This is not very efficient because you're recomputing the same kv values each time.\n\nTo optimize this, you can use a kv-cache to store the past keys and values instead of recomputing them each time. However, since the kv-cache grows with each generation step and is dynamic, it prevents you from taking advantage of [torch.compile](./perf_torch_compile), a powerful optimization tool that fuses PyTorch code into fast and optimized kernels.\n\nThe *static kv-cache* solves this issue by pre-allocating the kv-cache size to a maximum value which allows you to combine it with torch.compile for up to a 4x speed up.\n\n> [!WARNING]\n> Currently, only [Llama](./model_doc/llama2) and a few other models support static kv-cache and torch.compile. Check [this issue](https://github.com/huggingface/transformers/issues/28981) for a live model compatibility list.\n\nFor this example, let's load the [Gemma](https://hf.co/google/gemma-2b) model.\n\n```py\nfrom transformers import AutoTokenizer, AutoModelForCausalLM\n\ntokenizer = AutoTokenizer.from_pretrained(\"google/gemma-2b\")\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"google/gemma-2b\", device_map=\"auto\"\n)\n```\n\nThere are two ways you can configure the model to use a static kv-cache. For a 7B model on an A100, both methods get a 4x speed up in the forward pass. Your speed up may vary depending on the model size (larger models have a smaller speed up) and hardware. If you're using the [`~GenerationMixin.generate`] method, the speed up is ~3x. The forward pass (which still gets 4x speed up) is only a part of the whole [`~GenerationMixin.generate`] code.\n\n<hfoptions id=\"static-kv\">\n<hfoption id=\"generation_config\">\n\nAccess the model's `generation_config` attribute and set the `cache_implementation` to \"static\".\n\n```py\nmodel.generation_config.cache_implementation = \"static\"\n```\n\nCall torch.compile on the model to compile the forward pass with the static kv-cache."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 245768bf-9753-4037-b296-e42f2a59a7f6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/llm_optims/", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"245768bf-9753-4037-b296-e42f2a59a7f6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_optims/\", \"title\": \"LLM inference optimization\", \"tokens\": 4298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2b48628a-751d-4860-850f-fe9c7ff43443\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_optims/\", \"title\": \"LLM inference optimization\", \"tokens\": 4298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5e989f3e75139db0c03329e4a2eeabb974a2ca02fb5f3ff905ef61b01914e6af\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"78cc7d0e-02df-4973-9909-5adf51da48b2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_optims/\", \"title\": \"LLM inference optimization\", \"tokens\": 4298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"20af77c272d80ba9d97c7ee66c29dc4a27762060b370e1224505d1a6468d5fa2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f306694b-f453-4db0-8105-0f83afa655f1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d51094c491733fca5eadfe6128eb6dbf3b58524838592ddeb04714cbebd1a8be\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1829, \"end_char_idx\": 4791, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2b48628a-751d-4860-850f-fe9c7ff43443", "doc_id": "2b48628a-751d-4860-850f-fe9c7ff43443", "ref_doc_id": "2b48628a-751d-4860-850f-fe9c7ff43443", "chroma:document": "The *static kv-cache* solves this issue by pre-allocating the kv-cache size to a maximum value which allows you to combine it with torch.compile for up to a 4x speed up.\n\n> [!WARNING]\n> Currently, only [Llama](./model_doc/llama2) and a few other models support static kv-cache and torch.compile. Check [this issue](https://github.com/huggingface/transformers/issues/28981) for a live model compatibility list.\n\nFor this example, let's load the [Gemma](https://hf.co/google/gemma-2b) model.\n\n```py\nfrom transformers import AutoTokenizer, AutoModelForCausalLM\n\ntokenizer = AutoTokenizer.from_pretrained(\"google/gemma-2b\")\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"google/gemma-2b\", device_map=\"auto\"\n)\n```\n\nThere are two ways you can configure the model to use a static kv-cache. For a 7B model on an A100, both methods get a 4x speed up in the forward pass. Your speed up may vary depending on the model size (larger models have a smaller speed up) and hardware. If you're using the [`~GenerationMixin.generate`] method, the speed up is ~3x. The forward pass (which still gets 4x speed up) is only a part of the whole [`~GenerationMixin.generate`] code.\n\n<hfoptions id=\"static-kv\">\n<hfoption id=\"generation_config\">\n\nAccess the model's `generation_config` attribute and set the `cache_implementation` to \"static\".\n\n```py\nmodel.generation_config.cache_implementation = \"static\"\n```\n\nCall torch.compile on the model to compile the forward pass with the static kv-cache.\n\n```py\ncompiled_model = torch.compile(model, mode=\"reduce-overhead\", fullgraph=True)\ninput_text = \"The theory of special relativity states \"\ninput_ids = tokenizer(input_text, return_tensors=\"pt\").to(\"cuda\")\n\noutputs = compiled_model.generate(**input_ids)\ntokenizer.batch_decode(outputs, skip_special_tokens=True)\n['The theory of special relativity states 1. The speed of light is constant in all inertial reference']\n```\n\nUnder the hood, `generate` will attempt to reuse the same cache object, removing the need for re-compilation at each call. However, if the batch size or the maximum output length increase between calls, the cache will have to be reinitialized, triggering a new compilation.\n\n</hfoption>\n<hfoption id=\"Static Cache\">\n\nA [`StaticCache`] object can be passed to the model's forward pass under the `past_key_values` argument, enabling the use of this object as a static kv-cache. Using this strategy, you can write your own function to decode the next token given the current token and position and cache position of previously generated tokens. You can also pass the [`StaticCache`] object to [`~GenerationMixin.generate`] and use it across calls, like you would do with a dynamic cache.\n\n```py\nfrom transformers import LlamaTokenizer, LlamaForCausalLM, StaticCache, logging\nfrom transformers.testing_utils import CaptureLogger\nimport torch\n\nprompts = [\n    \"Simply put, the theory of relativity states that \",\n    \"My favorite all time favorite condiment is ketchup."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: f306694b-f453-4db0-8105-0f83afa655f1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/llm_optims/", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"f306694b-f453-4db0-8105-0f83afa655f1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_optims/\", \"title\": \"LLM inference optimization\", \"tokens\": 4298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2b48628a-751d-4860-850f-fe9c7ff43443\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_optims/\", \"title\": \"LLM inference optimization\", \"tokens\": 4298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5e989f3e75139db0c03329e4a2eeabb974a2ca02fb5f3ff905ef61b01914e6af\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"245768bf-9753-4037-b296-e42f2a59a7f6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_optims/\", \"title\": \"LLM inference optimization\", \"tokens\": 4298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b24ce7356346bfd1a1006fcc4e39288c05f8ddcd0cfba8bf2970b56fd66428d9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b82e9071-5b82-4ed6-bbfe-665231fff577\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"89fb2f3b85dac2e1ac66f99674d0ab181e6c1f15e4304f06566121bd9aa608c9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2992, \"end_char_idx\": 6184, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2b48628a-751d-4860-850f-fe9c7ff43443", "doc_id": "2b48628a-751d-4860-850f-fe9c7ff43443", "ref_doc_id": "2b48628a-751d-4860-850f-fe9c7ff43443", "chroma:document": "<hfoptions id=\"static-kv\">\n<hfoption id=\"generation_config\">\n\nAccess the model's `generation_config` attribute and set the `cache_implementation` to \"static\".\n\n```py\nmodel.generation_config.cache_implementation = \"static\"\n```\n\nCall torch.compile on the model to compile the forward pass with the static kv-cache.\n\n```py\ncompiled_model = torch.compile(model, mode=\"reduce-overhead\", fullgraph=True)\ninput_text = \"The theory of special relativity states \"\ninput_ids = tokenizer(input_text, return_tensors=\"pt\").to(\"cuda\")\n\noutputs = compiled_model.generate(**input_ids)\ntokenizer.batch_decode(outputs, skip_special_tokens=True)\n['The theory of special relativity states 1. The speed of light is constant in all inertial reference']\n```\n\nUnder the hood, `generate` will attempt to reuse the same cache object, removing the need for re-compilation at each call. However, if the batch size or the maximum output length increase between calls, the cache will have to be reinitialized, triggering a new compilation.\n\n</hfoption>\n<hfoption id=\"Static Cache\">\n\nA [`StaticCache`] object can be passed to the model's forward pass under the `past_key_values` argument, enabling the use of this object as a static kv-cache. Using this strategy, you can write your own function to decode the next token given the current token and position and cache position of previously generated tokens. You can also pass the [`StaticCache`] object to [`~GenerationMixin.generate`] and use it across calls, like you would do with a dynamic cache.\n\n```py\nfrom transformers import LlamaTokenizer, LlamaForCausalLM, StaticCache, logging\nfrom transformers.testing_utils import CaptureLogger\nimport torch\n\nprompts = [\n    \"Simply put, the theory of relativity states that \",\n    \"My favorite all time favorite condiment is ketchup.\",\n]\n\nNUM_TOKENS_TO_GENERATE = 40\ntorch_device = \"cuda\"\n\ntokenizer = LlamaTokenizer.from_pretrained(\"meta-llama/Llama-2-7b-hf\", pad_token=\"</s>\", padding_side=\"right\")\nmodel = LlamaForCausalLM.from_pretrained(\"meta-llama/Llama-2-7b-hf\", device_map=\"sequential\")\ninputs = tokenizer(prompts, return_tensors=\"pt\", padding=True).to(model.device)\n\ndef decode_one_tokens(model, cur_token, input_pos, cache_position, past_key_values):\n    logits = model(\n        cur_token,\n        position_ids=input_pos,\n        cache_position=cache_position,\n        past_key_values=past_key_values,\n        return_dict=False,\n        use_cache=True\n    )[0]\n    new_token = torch.argmax(logits[:, -1], dim=-1)[:, None]\n    return new_token\n```\n\nThere are a few important things you must do to enable static kv-cache and torch.compile with the `StaticCache` method:\n\n1. Initialize the [`StaticCache`] instance before using the model for inference. There you can configure parameters like the maximum batch size and sequence length.\n\n2. Call torch.compile on the model to compile the forward pass with the static kv-cache.\n\n3. Set `enable_math=True` in the [torch.backends.cuda.sdp_kernel](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html) context manager to enable the native PyTorch C++ implementation of scaled dot product attention to speed up inference even more."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: b82e9071-5b82-4ed6-bbfe-665231fff577
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/llm_optims/", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"b82e9071-5b82-4ed6-bbfe-665231fff577\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_optims/\", \"title\": \"LLM inference optimization\", \"tokens\": 4298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2b48628a-751d-4860-850f-fe9c7ff43443\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_optims/\", \"title\": \"LLM inference optimization\", \"tokens\": 4298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5e989f3e75139db0c03329e4a2eeabb974a2ca02fb5f3ff905ef61b01914e6af\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f306694b-f453-4db0-8105-0f83afa655f1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_optims/\", \"title\": \"LLM inference optimization\", \"tokens\": 4298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"abb78410b5018ae706a95e459d533aec734dc5f07242ad6ad438f54085d0258f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"eabff45d-70ce-4774-9d1d-08ae75120b0f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f9879ced09f4fccaa6e91d962e1d87cc1529c06568ec9f30c71cfdf26df20a26\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4513, \"end_char_idx\": 7825, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2b48628a-751d-4860-850f-fe9c7ff43443", "doc_id": "2b48628a-751d-4860-850f-fe9c7ff43443", "ref_doc_id": "2b48628a-751d-4860-850f-fe9c7ff43443", "chroma:document": "```py\nfrom transformers import LlamaTokenizer, LlamaForCausalLM, StaticCache, logging\nfrom transformers.testing_utils import CaptureLogger\nimport torch\n\nprompts = [\n    \"Simply put, the theory of relativity states that \",\n    \"My favorite all time favorite condiment is ketchup.\",\n]\n\nNUM_TOKENS_TO_GENERATE = 40\ntorch_device = \"cuda\"\n\ntokenizer = LlamaTokenizer.from_pretrained(\"meta-llama/Llama-2-7b-hf\", pad_token=\"</s>\", padding_side=\"right\")\nmodel = LlamaForCausalLM.from_pretrained(\"meta-llama/Llama-2-7b-hf\", device_map=\"sequential\")\ninputs = tokenizer(prompts, return_tensors=\"pt\", padding=True).to(model.device)\n\ndef decode_one_tokens(model, cur_token, input_pos, cache_position, past_key_values):\n    logits = model(\n        cur_token,\n        position_ids=input_pos,\n        cache_position=cache_position,\n        past_key_values=past_key_values,\n        return_dict=False,\n        use_cache=True\n    )[0]\n    new_token = torch.argmax(logits[:, -1], dim=-1)[:, None]\n    return new_token\n```\n\nThere are a few important things you must do to enable static kv-cache and torch.compile with the `StaticCache` method:\n\n1. Initialize the [`StaticCache`] instance before using the model for inference. There you can configure parameters like the maximum batch size and sequence length.\n\n2. Call torch.compile on the model to compile the forward pass with the static kv-cache.\n\n3. Set `enable_math=True` in the [torch.backends.cuda.sdp_kernel](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html) context manager to enable the native PyTorch C++ implementation of scaled dot product attention to speed up inference even more.\n\n```py\nbatch_size, seq_length = inputs[\"input_ids\"].shape\nwith torch.no_grad():\n    past_key_values = StaticCache(\n        config=model.config, max_batch_size=2, max_cache_len=4096, device=torch_device, dtype=model.dtype\n    )\n    cache_position = torch.arange(seq_length, device=torch_device)\n    generated_ids = torch.zeros(\n        batch_size, seq_length + NUM_TOKENS_TO_GENERATE + 1, dtype=torch.int, device=torch_device\n    )\n    generated_ids[:, cache_position] = inputs[\"input_ids\"].to(torch_device).to(torch.int)\n\n    logits = model(\n        **inputs, cache_position=cache_position, past_key_values=past_key_values,return_dict=False, use_cache=True\n    )[0]\n    next_token = torch.argmax(logits[:, -1], dim=-1)[:, None]\n    generated_ids[:, seq_length] = next_token[:, 0]\n\n    decode_one_tokens = torch.compile(decode_one_tokens, mode=\"reduce-overhead\", fullgraph=True)\n    cache_position = torch.tensor([seq_length + 1], device=torch_device)\n    for _ in range(1, NUM_TOKENS_TO_GENERATE):\n        with torch.backends.cuda.sdp_kernel(enable_flash=False, enable_mem_efficient=False, enable_math=True):\n            next_token = decode_one_tokens(model, next_token.clone(), None, cache_position, past_key_values)\n            generated_ids[:, cache_position] = next_token.int()\n        cache_position += 1\n\ntext = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)\ntext\n['Simply put, the theory of relativity states that 1) the speed of light is constant, 2) the speed of light is the same for all observers, and 3) the laws of physics are the same for all observers.',\n 'My favorite all time favorite condiment is ketchup."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: eabff45d-70ce-4774-9d1d-08ae75120b0f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/llm_optims/", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"eabff45d-70ce-4774-9d1d-08ae75120b0f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_optims/\", \"title\": \"LLM inference optimization\", \"tokens\": 4298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2b48628a-751d-4860-850f-fe9c7ff43443\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_optims/\", \"title\": \"LLM inference optimization\", \"tokens\": 4298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5e989f3e75139db0c03329e4a2eeabb974a2ca02fb5f3ff905ef61b01914e6af\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b82e9071-5b82-4ed6-bbfe-665231fff577\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_optims/\", \"title\": \"LLM inference optimization\", \"tokens\": 4298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8bfd9373e09433dab36d0c1fde265d34b70d00e1e701edf140c9806257953480\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"417c2679-5905-4abf-9a27-54345ce94b1f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d3fb57e868efc03fbb147f29fb32d46007fe9391013b8174827453d6e9056f16\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6186, \"end_char_idx\": 9507, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2b48628a-751d-4860-850f-fe9c7ff43443", "doc_id": "2b48628a-751d-4860-850f-fe9c7ff43443", "ref_doc_id": "2b48628a-751d-4860-850f-fe9c7ff43443", "chroma:document": "```py\nbatch_size, seq_length = inputs[\"input_ids\"].shape\nwith torch.no_grad():\n    past_key_values = StaticCache(\n        config=model.config, max_batch_size=2, max_cache_len=4096, device=torch_device, dtype=model.dtype\n    )\n    cache_position = torch.arange(seq_length, device=torch_device)\n    generated_ids = torch.zeros(\n        batch_size, seq_length + NUM_TOKENS_TO_GENERATE + 1, dtype=torch.int, device=torch_device\n    )\n    generated_ids[:, cache_position] = inputs[\"input_ids\"].to(torch_device).to(torch.int)\n\n    logits = model(\n        **inputs, cache_position=cache_position, past_key_values=past_key_values,return_dict=False, use_cache=True\n    )[0]\n    next_token = torch.argmax(logits[:, -1], dim=-1)[:, None]\n    generated_ids[:, seq_length] = next_token[:, 0]\n\n    decode_one_tokens = torch.compile(decode_one_tokens, mode=\"reduce-overhead\", fullgraph=True)\n    cache_position = torch.tensor([seq_length + 1], device=torch_device)\n    for _ in range(1, NUM_TOKENS_TO_GENERATE):\n        with torch.backends.cuda.sdp_kernel(enable_flash=False, enable_mem_efficient=False, enable_math=True):\n            next_token = decode_one_tokens(model, next_token.clone(), None, cache_position, past_key_values)\n            generated_ids[:, cache_position] = next_token.int()\n        cache_position += 1\n\ntext = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)\ntext\n['Simply put, the theory of relativity states that 1) the speed of light is constant, 2) the speed of light is the same for all observers, and 3) the laws of physics are the same for all observers.',\n 'My favorite all time favorite condiment is ketchup. I love it on everything. I love it on my eggs, my fries, my chicken, my burgers, my hot dogs, my sandwiches, my salads, my p']\n```\n\n> [!TIP]\n> If you want to reuse the [`StaticCache`] object on a new prompt, be sure to reset its contents with the `.reset()` method\n\n</hfoption>\n</hfoptions>\n\n## Speculative decoding\n\n> [!TIP]\n> For a more in-depth explanation, take a look at the [Assisted Generation: a new direction toward low-latency text generation](https://hf.co/blog/assisted-generation) blog post!\n\nAnother issue with autoregression is that for each input token you need to load the model weights each time during the forward pass. This is slow and cumbersome for LLMs which have billions of parameters. Speculative decoding alleviates this slowdown by using a second smaller and faster assistant model to generate candidate tokens that are verified by the larger LLM in a single forward pass. If the verified tokens are correct, the LLM essentially gets them for \"free\" without having to generate them itself. There is no degradation in accuracy because the verification forward pass ensures the same outputs are generated as if the LLM had generated them on its own.\n\nTo get the largest speed up, the assistant model should be a lot smaller than the LLM so that it can generate tokens quickly. The assistant and LLM model must also share the same tokenizer to avoid re-encoding and decoding tokens.\n\n> [!WARNING]\n> Speculative decoding is only supported for the greedy search and sampling decoding strategies, and it also doesn't support batched inputs.\n\nEnable speculative decoding by loading an assistant model and passing it to the [`~GenerationMixin.generate`] method."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 417c2679-5905-4abf-9a27-54345ce94b1f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/llm_optims/", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"417c2679-5905-4abf-9a27-54345ce94b1f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_optims/\", \"title\": \"LLM inference optimization\", \"tokens\": 4298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2b48628a-751d-4860-850f-fe9c7ff43443\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_optims/\", \"title\": \"LLM inference optimization\", \"tokens\": 4298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5e989f3e75139db0c03329e4a2eeabb974a2ca02fb5f3ff905ef61b01914e6af\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"eabff45d-70ce-4774-9d1d-08ae75120b0f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_optims/\", \"title\": \"LLM inference optimization\", \"tokens\": 4298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"ef0e47f33a56d137bedbb62f7c23d85901b1a85e671997a8f07017811cbe2bec\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e21adef2-ecf6-4e28-b551-45e15fdc1be8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"76f67fe6a8a479472994d93f390addc7e3fb3746a26ee2951655e09a39eab1a2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7769, \"end_char_idx\": 10446, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2b48628a-751d-4860-850f-fe9c7ff43443", "doc_id": "2b48628a-751d-4860-850f-fe9c7ff43443", "ref_doc_id": "2b48628a-751d-4860-850f-fe9c7ff43443", "chroma:document": "',\n 'My favorite all time favorite condiment is ketchup. I love it on everything. I love it on my eggs, my fries, my chicken, my burgers, my hot dogs, my sandwiches, my salads, my p']\n```\n\n> [!TIP]\n> If you want to reuse the [`StaticCache`] object on a new prompt, be sure to reset its contents with the `.reset()` method\n\n</hfoption>\n</hfoptions>\n\n## Speculative decoding\n\n> [!TIP]\n> For a more in-depth explanation, take a look at the [Assisted Generation: a new direction toward low-latency text generation](https://hf.co/blog/assisted-generation) blog post!\n\nAnother issue with autoregression is that for each input token you need to load the model weights each time during the forward pass. This is slow and cumbersome for LLMs which have billions of parameters. Speculative decoding alleviates this slowdown by using a second smaller and faster assistant model to generate candidate tokens that are verified by the larger LLM in a single forward pass. If the verified tokens are correct, the LLM essentially gets them for \"free\" without having to generate them itself. There is no degradation in accuracy because the verification forward pass ensures the same outputs are generated as if the LLM had generated them on its own.\n\nTo get the largest speed up, the assistant model should be a lot smaller than the LLM so that it can generate tokens quickly. The assistant and LLM model must also share the same tokenizer to avoid re-encoding and decoding tokens.\n\n> [!WARNING]\n> Speculative decoding is only supported for the greedy search and sampling decoding strategies, and it also doesn't support batched inputs.\n\nEnable speculative decoding by loading an assistant model and passing it to the [`~GenerationMixin.generate`] method.\n\n<hfoptions id=\"spec-decoding\">\n<hfoption id=\"greedy search\">\n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nimport torch\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n\ntokenizer = AutoTokenizer.from_pretrained(\"facebook/opt-1.3b\")\ninputs = tokenizer(\"Einstein's theory of relativity states\", return_tensors=\"pt\").to(device)\n\nmodel = AutoModelForCausalLM.from_pretrained(\"facebook/opt-1.3b\").to(device)\nassistant_model = AutoModelForCausalLM.from_pretrained(\"facebook/opt-125m\").to(device)\noutputs = model.generate(**inputs, assistant_model=assistant_model)\ntokenizer.batch_decode(outputs, skip_special_tokens=True)\n[\"Einstein's theory of relativity states that the speed of light is constant.    \"]\n```\n\n</hfoption>\n<hfoption id=\"sampling\">\n\nFor speculative sampling decoding, add the `do_sample` and `temperature` parameters to the [`~GenerationMixin.generate`] method in addition to the assistant model."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: e21adef2-ecf6-4e28-b551-45e15fdc1be8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/llm_optims/", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"e21adef2-ecf6-4e28-b551-45e15fdc1be8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_optims/\", \"title\": \"LLM inference optimization\", \"tokens\": 4298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2b48628a-751d-4860-850f-fe9c7ff43443\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_optims/\", \"title\": \"LLM inference optimization\", \"tokens\": 4298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5e989f3e75139db0c03329e4a2eeabb974a2ca02fb5f3ff905ef61b01914e6af\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"417c2679-5905-4abf-9a27-54345ce94b1f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_optims/\", \"title\": \"LLM inference optimization\", \"tokens\": 4298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"800587032c69b2df8b3cbef659c2cbd91362ef1bd635aba8859386d329ccd1ca\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2c6016d8-a68a-4904-a292-e218e459d993\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0248ab152570f78fe982fae2b54ae145f941996d0e9198c5bc89fb8b8a4f81e0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8727, \"end_char_idx\": 11780, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2b48628a-751d-4860-850f-fe9c7ff43443", "doc_id": "2b48628a-751d-4860-850f-fe9c7ff43443", "ref_doc_id": "2b48628a-751d-4860-850f-fe9c7ff43443", "chroma:document": "If the verified tokens are correct, the LLM essentially gets them for \"free\" without having to generate them itself. There is no degradation in accuracy because the verification forward pass ensures the same outputs are generated as if the LLM had generated them on its own.\n\nTo get the largest speed up, the assistant model should be a lot smaller than the LLM so that it can generate tokens quickly. The assistant and LLM model must also share the same tokenizer to avoid re-encoding and decoding tokens.\n\n> [!WARNING]\n> Speculative decoding is only supported for the greedy search and sampling decoding strategies, and it also doesn't support batched inputs.\n\nEnable speculative decoding by loading an assistant model and passing it to the [`~GenerationMixin.generate`] method.\n\n<hfoptions id=\"spec-decoding\">\n<hfoption id=\"greedy search\">\n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nimport torch\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n\ntokenizer = AutoTokenizer.from_pretrained(\"facebook/opt-1.3b\")\ninputs = tokenizer(\"Einstein's theory of relativity states\", return_tensors=\"pt\").to(device)\n\nmodel = AutoModelForCausalLM.from_pretrained(\"facebook/opt-1.3b\").to(device)\nassistant_model = AutoModelForCausalLM.from_pretrained(\"facebook/opt-125m\").to(device)\noutputs = model.generate(**inputs, assistant_model=assistant_model)\ntokenizer.batch_decode(outputs, skip_special_tokens=True)\n[\"Einstein's theory of relativity states that the speed of light is constant.    \"]\n```\n\n</hfoption>\n<hfoption id=\"sampling\">\n\nFor speculative sampling decoding, add the `do_sample` and `temperature` parameters to the [`~GenerationMixin.generate`] method in addition to the assistant model.\n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nimport torch\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n\ntokenizer = AutoTokenizer.from_pretrained(\"facebook/opt-1.3b\")\ninputs = tokenizer(\"Einstein's theory of relativity states\", return_tensors=\"pt\").to(device)\n\nmodel = AutoModelForCausalLM.from_pretrained(\"facebook/opt-1.3b\").to(device)\nassistant_model = AutoModelForCausalLM.from_pretrained(\"facebook/opt-125m\").to(device)\noutputs = model.generate(**inputs, assistant_model=assistant_model, do_sample=True, temperature=0.7)\nprint(tokenizer.batch_decode(outputs, skip_special_tokens=True))\n[\"Einstein's theory of relativity states that motion in the universe is not a straight line.\\n\"]\n```\n\n</hfoption>\n</hfoptions>\n\n### Prompt lookup decoding\n\nPrompt lookup decoding is a variant of speculative decoding that is also compatible with greedy search and sampling. Prompt lookup works especially well for input-grounded tasks - such as summarization - where there is often overlapping words between the prompt and output. These overlapping n-grams are used as the LLM candidate tokens.\n\nTo enable prompt lookup decoding, specify the number of tokens that should be overlapping in the `prompt_lookup_num_tokens` parameter. Then you can pass this parameter to the [`~GenerationMixin.generate`] method."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 2c6016d8-a68a-4904-a292-e218e459d993
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/llm_optims/", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"2c6016d8-a68a-4904-a292-e218e459d993\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_optims/\", \"title\": \"LLM inference optimization\", \"tokens\": 4298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2b48628a-751d-4860-850f-fe9c7ff43443\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_optims/\", \"title\": \"LLM inference optimization\", \"tokens\": 4298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5e989f3e75139db0c03329e4a2eeabb974a2ca02fb5f3ff905ef61b01914e6af\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e21adef2-ecf6-4e28-b551-45e15fdc1be8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_optims/\", \"title\": \"LLM inference optimization\", \"tokens\": 4298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b048561d8ed3641021aa6e604ecf6c297c7456d731ea69800287fcb161ded509\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b5674e1a-0ca5-4ac3-8550-4bad7d83bcda\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7fc372a42e14d6ff22e90542b5e54b512405505d3bf281d4b48e93fda64a512a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10240, \"end_char_idx\": 13616, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2b48628a-751d-4860-850f-fe9c7ff43443", "doc_id": "2b48628a-751d-4860-850f-fe9c7ff43443", "ref_doc_id": "2b48628a-751d-4860-850f-fe9c7ff43443", "chroma:document": "```\n\n</hfoption>\n<hfoption id=\"sampling\">\n\nFor speculative sampling decoding, add the `do_sample` and `temperature` parameters to the [`~GenerationMixin.generate`] method in addition to the assistant model.\n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nimport torch\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n\ntokenizer = AutoTokenizer.from_pretrained(\"facebook/opt-1.3b\")\ninputs = tokenizer(\"Einstein's theory of relativity states\", return_tensors=\"pt\").to(device)\n\nmodel = AutoModelForCausalLM.from_pretrained(\"facebook/opt-1.3b\").to(device)\nassistant_model = AutoModelForCausalLM.from_pretrained(\"facebook/opt-125m\").to(device)\noutputs = model.generate(**inputs, assistant_model=assistant_model, do_sample=True, temperature=0.7)\nprint(tokenizer.batch_decode(outputs, skip_special_tokens=True))\n[\"Einstein's theory of relativity states that motion in the universe is not a straight line.\\n\"]\n```\n\n</hfoption>\n</hfoptions>\n\n### Prompt lookup decoding\n\nPrompt lookup decoding is a variant of speculative decoding that is also compatible with greedy search and sampling. Prompt lookup works especially well for input-grounded tasks - such as summarization - where there is often overlapping words between the prompt and output. These overlapping n-grams are used as the LLM candidate tokens.\n\nTo enable prompt lookup decoding, specify the number of tokens that should be overlapping in the `prompt_lookup_num_tokens` parameter. Then you can pass this parameter to the [`~GenerationMixin.generate`] method.\n\n<hfoptions id=\"pld\">\n<hfoption id=\"greedy decoding\">\n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nimport torch\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n\ntokenizer = AutoTokenizer.from_pretrained(\"facebook/opt-1.3b\")\ninputs = tokenizer(\"The second law of thermodynamics states\", return_tensors=\"pt\").to(device)\n\nmodel = AutoModelForCausalLM.from_pretrained(\"facebook/opt-1.3b\").to(device)\nassistant_model = AutoModelForCausalLM.from_pretrained(\"facebook/opt-125m\").to(device)\noutputs = model.generate(**inputs, prompt_lookup_num_tokens=3)\nprint(tokenizer.batch_decode(outputs, skip_special_tokens=True))\n['The second law of thermodynamics states that entropy increases with temperature.      ']\n```\n\n</hfoption>\n<hfoption id=\"sampling\">\n\nFor prompt lookup decoding with sampling, add the `do_sample` and `temperature` parameters to the [`~GenerationMixin.generate`] method.\n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nimport torch\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n\ntokenizer = AutoTokenizer.from_pretrained(\"facebook/opt-1.3b\")\ninputs = tokenizer(\"The second law of thermodynamics states\", return_tensors=\"pt\").to(device)\n\nmodel = AutoModelForCausalLM.from_pretrained(\"facebook/opt-1.3b\").to(device)\noutputs = model.generate(**inputs, prompt_lookup_num_tokens=3, do_sample=True, temperature=0.7)\nprint(tokenizer.batch_decode(outputs, skip_special_tokens=True))\n[\"The second law of thermodynamics states that energy cannot be created nor destroyed. It's not a\"]\n```\n\n</hfoption>\n</hfoptions>\n\n## Attention optimizations\n\nA known issue with transformer models is that the self-attention mechanism grows quadratically in compute and memory with the number of input tokens. This limitation is only magnified in LLMs which handles much longer sequences."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: b5674e1a-0ca5-4ac3-8550-4bad7d83bcda
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/llm_optims/", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"b5674e1a-0ca5-4ac3-8550-4bad7d83bcda\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_optims/\", \"title\": \"LLM inference optimization\", \"tokens\": 4298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2b48628a-751d-4860-850f-fe9c7ff43443\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_optims/\", \"title\": \"LLM inference optimization\", \"tokens\": 4298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5e989f3e75139db0c03329e4a2eeabb974a2ca02fb5f3ff905ef61b01914e6af\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2c6016d8-a68a-4904-a292-e218e459d993\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_optims/\", \"title\": \"LLM inference optimization\", \"tokens\": 4298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a0bfe8947d253f8ae8781fcb2e8337511c1aa07d0e22beb6f8eb3484e3dd2748\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d96c3da6-4e5b-4056-8ed3-4652ce37daf5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9df4b0d3fcec539fd596e2ae2b6eef0c03184a5e677d462d574d2206890bf9bb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12515, \"end_char_idx\": 15865, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2b48628a-751d-4860-850f-fe9c7ff43443", "doc_id": "2b48628a-751d-4860-850f-fe9c7ff43443", "ref_doc_id": "2b48628a-751d-4860-850f-fe9c7ff43443", "chroma:document": "```\n\n</hfoption>\n<hfoption id=\"sampling\">\n\nFor prompt lookup decoding with sampling, add the `do_sample` and `temperature` parameters to the [`~GenerationMixin.generate`] method.\n\n```py\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nimport torch\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n\ntokenizer = AutoTokenizer.from_pretrained(\"facebook/opt-1.3b\")\ninputs = tokenizer(\"The second law of thermodynamics states\", return_tensors=\"pt\").to(device)\n\nmodel = AutoModelForCausalLM.from_pretrained(\"facebook/opt-1.3b\").to(device)\noutputs = model.generate(**inputs, prompt_lookup_num_tokens=3, do_sample=True, temperature=0.7)\nprint(tokenizer.batch_decode(outputs, skip_special_tokens=True))\n[\"The second law of thermodynamics states that energy cannot be created nor destroyed. It's not a\"]\n```\n\n</hfoption>\n</hfoptions>\n\n## Attention optimizations\n\nA known issue with transformer models is that the self-attention mechanism grows quadratically in compute and memory with the number of input tokens. This limitation is only magnified in LLMs which handles much longer sequences. To address this, try FlashAttention2 or PyTorch's scaled dot product attention (SDPA), which are more memory efficient attention implementations and can accelerate inference.\n\n### FlashAttention-2\n\nFlashAttention and [FlashAttention-2](./perf_infer_gpu_one#flashattention-2) break up the attention computation into smaller chunks and reduces the number of intermediate read/write operations to GPU memory to speed up inference. FlashAttention-2 improves on the original FlashAttention algorithm by also parallelizing over sequence length dimension and better partitioning work on the hardware to reduce synchronization and communication overhead.\n\nTo use FlashAttention-2, set `attn_implementation=\"flash_attention_2\"` in the [`~PreTrainedModel.from_pretrained`] method.\n\n```py\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nquant_config = BitsAndBytesConfig(load_in_8bit=True)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"google/gemma-2b\",\n    quantization_config=quant_config,\n    torch_dtype=torch.bfloat16,\n    attn_implementation=\"flash_attention_2\",\n)\n```\n\n### PyTorch scaled dot product attention\n\nScaled dot product attention (SDPA) is automatically enabled in PyTorch 2.0 and it supports FlashAttention, xFormers, and PyTorch's C++ implementation. SDPA chooses the most performant attention algorithm if you're using a CUDA backend. For other backends, SDPA defaults to the PyTorch C++ implementation.\n\n> [!TIP]\n> SDPA supports FlashAttention-2 as long as you have the latest PyTorch version installed.\n\nUse the [torch.backends.cuda.sdp_kernel](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html) context manager to explicitly enable or disable any of the three attention algorithms. For example, set `enable_flash=True` to enable FlashAttention.\n\n```py\nimport torch\nfrom transformers import AutoModelForCausalLM\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"google/gemma-2b\",\n    torch_dtype=torch.bfloat16,\n)\n\nwith torch.backends.cuda.sdp_kernel(enable_flash=True, enable_math=False, enable_mem_efficient=False):\n    outputs = model.generate(**inputs)\n```\n\n## Quantization\n\nQuantization reduces the size of the LLM weights by storing them in a lower precision."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: d96c3da6-4e5b-4056-8ed3-4652ce37daf5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/llm_optims/", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"d96c3da6-4e5b-4056-8ed3-4652ce37daf5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_optims/\", \"title\": \"LLM inference optimization\", \"tokens\": 4298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2b48628a-751d-4860-850f-fe9c7ff43443\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_optims/\", \"title\": \"LLM inference optimization\", \"tokens\": 4298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5e989f3e75139db0c03329e4a2eeabb974a2ca02fb5f3ff905ef61b01914e6af\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b5674e1a-0ca5-4ac3-8550-4bad7d83bcda\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_optims/\", \"title\": \"LLM inference optimization\", \"tokens\": 4298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"4f05189f9e23a4cabf3fbaef72c8f90833056a72c445c76d2e1e2fec8f01b2c0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d46120ab-883f-44d0-a07c-052bbdfcbf58\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5e16265a671d5180fb197170a1fdd86732e0b971c90b35594dc39477ad678e0d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14265, \"end_char_idx\": 17243, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2b48628a-751d-4860-850f-fe9c7ff43443", "doc_id": "2b48628a-751d-4860-850f-fe9c7ff43443", "ref_doc_id": "2b48628a-751d-4860-850f-fe9c7ff43443", "chroma:document": "To use FlashAttention-2, set `attn_implementation=\"flash_attention_2\"` in the [`~PreTrainedModel.from_pretrained`] method.\n\n```py\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nquant_config = BitsAndBytesConfig(load_in_8bit=True)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"google/gemma-2b\",\n    quantization_config=quant_config,\n    torch_dtype=torch.bfloat16,\n    attn_implementation=\"flash_attention_2\",\n)\n```\n\n### PyTorch scaled dot product attention\n\nScaled dot product attention (SDPA) is automatically enabled in PyTorch 2.0 and it supports FlashAttention, xFormers, and PyTorch's C++ implementation. SDPA chooses the most performant attention algorithm if you're using a CUDA backend. For other backends, SDPA defaults to the PyTorch C++ implementation.\n\n> [!TIP]\n> SDPA supports FlashAttention-2 as long as you have the latest PyTorch version installed.\n\nUse the [torch.backends.cuda.sdp_kernel](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html) context manager to explicitly enable or disable any of the three attention algorithms. For example, set `enable_flash=True` to enable FlashAttention.\n\n```py\nimport torch\nfrom transformers import AutoModelForCausalLM\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"google/gemma-2b\",\n    torch_dtype=torch.bfloat16,\n)\n\nwith torch.backends.cuda.sdp_kernel(enable_flash=True, enable_math=False, enable_mem_efficient=False):\n    outputs = model.generate(**inputs)\n```\n\n## Quantization\n\nQuantization reduces the size of the LLM weights by storing them in a lower precision. This translates to lower memory usage and makes loading LLMs for inference more accessible if you're constrained by your GPUs memory. If you aren't limited by your GPU, you don't necessarily need to quantize your model because it can incur a small latency cost (except for AWQ and fused AWQ modules) due to the extra step required to quantize and dequantize the weights.\n\n> [!TIP]\n> There are many quantization libraries (see the [Quantization](./quantization) guide for more details) available, such as Quanto, AQLM, AWQ, and AutoGPTQ. Feel free to try them out and see which one works best for your use case. We also recommend reading the [Overview of natively supported quantization schemes in \ud83e\udd17 Transformers](https://hf.co/blog/overview-quantization-transformers) blog post which compares AutoGPTQ and bitsandbytes.\n\nUse the Model Memory Calculator below to estimate and compare how much memory is required to load a model. For example, try estimating how much memory it costs to load [Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1).\n\n<iframe\n\tsrc=\"https://hf-accelerate-model-memory-usage.hf.space\"\n\tframeborder=\"0\"\n\twidth=\"850\"\n\theight=\"450\"\n></iframe>\n\nTo load Mistral-7B-v0.1 in half-precision, set the `torch_dtype` parameter in the [`~transformers.AutoModelForCausalLM.from_pretrained`] method to `torch.bfloat16`. This requires 13.74GB of memory."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: d46120ab-883f-44d0-a07c-052bbdfcbf58
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/llm_optims/", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"d46120ab-883f-44d0-a07c-052bbdfcbf58\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_optims/\", \"title\": \"LLM inference optimization\", \"tokens\": 4298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2b48628a-751d-4860-850f-fe9c7ff43443\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_optims/\", \"title\": \"LLM inference optimization\", \"tokens\": 4298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5e989f3e75139db0c03329e4a2eeabb974a2ca02fb5f3ff905ef61b01914e6af\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d96c3da6-4e5b-4056-8ed3-4652ce37daf5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_optims/\", \"title\": \"LLM inference optimization\", \"tokens\": 4298, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b51f44ace530d1f7c2e93fc2c0dbd1089ba732f0bd87ce2e18e6134377db8d9c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15866, \"end_char_idx\": 17988, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2b48628a-751d-4860-850f-fe9c7ff43443", "doc_id": "2b48628a-751d-4860-850f-fe9c7ff43443", "ref_doc_id": "2b48628a-751d-4860-850f-fe9c7ff43443", "chroma:document": "This translates to lower memory usage and makes loading LLMs for inference more accessible if you're constrained by your GPUs memory. If you aren't limited by your GPU, you don't necessarily need to quantize your model because it can incur a small latency cost (except for AWQ and fused AWQ modules) due to the extra step required to quantize and dequantize the weights.\n\n> [!TIP]\n> There are many quantization libraries (see the [Quantization](./quantization) guide for more details) available, such as Quanto, AQLM, AWQ, and AutoGPTQ. Feel free to try them out and see which one works best for your use case. We also recommend reading the [Overview of natively supported quantization schemes in \ud83e\udd17 Transformers](https://hf.co/blog/overview-quantization-transformers) blog post which compares AutoGPTQ and bitsandbytes.\n\nUse the Model Memory Calculator below to estimate and compare how much memory is required to load a model. For example, try estimating how much memory it costs to load [Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1).\n\n<iframe\n\tsrc=\"https://hf-accelerate-model-memory-usage.hf.space\"\n\tframeborder=\"0\"\n\twidth=\"850\"\n\theight=\"450\"\n></iframe>\n\nTo load Mistral-7B-v0.1 in half-precision, set the `torch_dtype` parameter in the [`~transformers.AutoModelForCausalLM.from_pretrained`] method to `torch.bfloat16`. This requires 13.74GB of memory.\n\n```py\nfrom transformers import AutoTokenizer, AutoModelForCausalLM\nimport torch\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"mistralai/Mistral-7B-v0.1\", torch_dtype=torch.bfloat16, device_map=\"auto\",\n)\n```\n\nTo load a quantized model (8-bit or 4-bit) for inference, try [bitsandbytes](https://hf.co/docs/bitsandbytes) and set the `load_in_4bit` or `load_in_8bit` parameters to `True`. Loading the model in 8-bits only requires 6.87 GB of memory.\n\n```py\nfrom transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig\nimport torch\n\nquant_config = BitsAndBytesConfig(load_in_8bit=True)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"mistralai/Mistral-7B-v0.1\", quantization_config=quant_config, device_map=\"auto\"\n)\n```"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 6ab33cb5-c864-4af3-af1d-7db202327271
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/llm_tutorial/", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"6ab33cb5-c864-4af3-af1d-7db202327271\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_tutorial/\", \"title\": \"Generation with LLMs\", \"tokens\": 3594, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3ecee411-8e31-4efa-af29-533880c0a63f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_tutorial/\", \"title\": \"Generation with LLMs\", \"tokens\": 3594, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"87f417fea89e56403f8d3d126dd0191fa18fec192e999880f186a8528147e92c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ba956f73-06bb-4190-9a10-90edfc9cb584\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8ade258cbf1c0901b8893588a0fa93d16296686d236c09f3f7400d435c838a07\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3615, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3ecee411-8e31-4efa-af29-533880c0a63f", "doc_id": "3ecee411-8e31-4efa-af29-533880c0a63f", "ref_doc_id": "3ecee411-8e31-4efa-af29-533880c0a63f", "chroma:document": "# Generation with LLMs\n\n[[open-in-colab]]\n\nLLMs, or Large Language Models, are the key component behind text generation. In a nutshell, they consist of large pretrained transformer models trained to predict the next word (or, more precisely, token) given some input text. Since they predict one token at a time, you need to do something more elaborate to generate new sentences other than just calling the model -- you need to do autoregressive generation.\n\nAutoregressive generation is the inference-time procedure of iteratively calling a model with its own generated outputs, given a few initial inputs. In \ud83e\udd17 Transformers, this is handled by the [`~generation.GenerationMixin.generate`] method, which is available to all models with generative capabilities.\n\nThis tutorial will show you how to:\n\n* Generate text with an LLM\n* Avoid common pitfalls\n* Next steps to help you get the most out of your LLM\n\nBefore you begin, make sure you have all the necessary libraries installed:\n\n```bash\npip install transformers bitsandbytes>=0.39.0 -q\n```\n\n\n## Generate text\n\nA language model trained for [causal language modeling](tasks/language_modeling) takes a sequence of text tokens as input and returns the probability distribution for the next token.\n\n<!-- [GIF 1 -- FWD PASS] -->\n<figure class=\"image table text-center m-0 w-full\">\n    <video\n        style=\"max-width: 90%; margin: auto;\"\n        autoplay loop muted playsinline\n        src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/assisted-generation/gif_1_1080p.mov\"\n    ></video>\n    <figcaption>\"Forward pass of an LLM\"</figcaption>\n</figure>\n\nA critical aspect of autoregressive generation with LLMs is how to select the next token from this probability distribution. Anything goes in this step as long as you end up with a token for the next iteration. This means it can be as simple as selecting the most likely token from the probability distribution or as complex as applying a dozen transformations before sampling from the resulting distribution.\n\n<!-- [GIF 2 -- TEXT GENERATION] -->\n<figure class=\"image table text-center m-0 w-full\">\n    <video\n        style=\"max-width: 90%; margin: auto;\"\n        autoplay loop muted playsinline\n        src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/assisted-generation/gif_2_1080p.mov\"\n    ></video>\n    <figcaption>\"Autoregressive generation iteratively selects the next token from a probability distribution to generate text\"</figcaption>\n</figure>\n\nThe process depicted above is repeated iteratively until some stopping condition is reached. Ideally, the stopping condition is dictated by the model, which should learn when to output an end-of-sequence (`EOS`) token. If this is not the case, generation stops when some predefined maximum length is reached.\n\nProperly setting up the token selection step and the stopping condition is essential to make your model behave as you'd expect on your task. That is why we have a [`~generation.GenerationConfig`] file associated with each model, which contains a good default generative parameterization and is loaded alongside your model.\n\nLet's talk code!\n\n<Tip>\n\nIf you're interested in basic LLM usage, our high-level [`Pipeline`](pipeline_tutorial) interface is a great starting point. However, LLMs often require advanced features like quantization and fine control of the token selection step, which is best done through [`~generation.GenerationMixin.generate`]. Autoregressive generation with LLMs is also resource-intensive and should be executed on a GPU for adequate throughput."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: ba956f73-06bb-4190-9a10-90edfc9cb584
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/llm_tutorial/", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"ba956f73-06bb-4190-9a10-90edfc9cb584\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_tutorial/\", \"title\": \"Generation with LLMs\", \"tokens\": 3594, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3ecee411-8e31-4efa-af29-533880c0a63f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_tutorial/\", \"title\": \"Generation with LLMs\", \"tokens\": 3594, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"87f417fea89e56403f8d3d126dd0191fa18fec192e999880f186a8528147e92c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6ab33cb5-c864-4af3-af1d-7db202327271\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_tutorial/\", \"title\": \"Generation with LLMs\", \"tokens\": 3594, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"45ef0345b8abeb1aa929b7edd17978e8407991dc47bbf25e7d7b084eeff0b9e4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3fcce074-2442-4e56-85ad-0d6cb4410a50\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ed92f45d2f6f0334ecc0a16537d491eb128a24b5c930acab9e2960e54f0f23e1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1763, \"end_char_idx\": 5251, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3ecee411-8e31-4efa-af29-533880c0a63f", "doc_id": "3ecee411-8e31-4efa-af29-533880c0a63f", "ref_doc_id": "3ecee411-8e31-4efa-af29-533880c0a63f", "chroma:document": "Anything goes in this step as long as you end up with a token for the next iteration. This means it can be as simple as selecting the most likely token from the probability distribution or as complex as applying a dozen transformations before sampling from the resulting distribution.\n\n<!-- [GIF 2 -- TEXT GENERATION] -->\n<figure class=\"image table text-center m-0 w-full\">\n    <video\n        style=\"max-width: 90%; margin: auto;\"\n        autoplay loop muted playsinline\n        src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/assisted-generation/gif_2_1080p.mov\"\n    ></video>\n    <figcaption>\"Autoregressive generation iteratively selects the next token from a probability distribution to generate text\"</figcaption>\n</figure>\n\nThe process depicted above is repeated iteratively until some stopping condition is reached. Ideally, the stopping condition is dictated by the model, which should learn when to output an end-of-sequence (`EOS`) token. If this is not the case, generation stops when some predefined maximum length is reached.\n\nProperly setting up the token selection step and the stopping condition is essential to make your model behave as you'd expect on your task. That is why we have a [`~generation.GenerationConfig`] file associated with each model, which contains a good default generative parameterization and is loaded alongside your model.\n\nLet's talk code!\n\n<Tip>\n\nIf you're interested in basic LLM usage, our high-level [`Pipeline`](pipeline_tutorial) interface is a great starting point. However, LLMs often require advanced features like quantization and fine control of the token selection step, which is best done through [`~generation.GenerationMixin.generate`]. Autoregressive generation with LLMs is also resource-intensive and should be executed on a GPU for adequate throughput.\n\n</Tip>\n\nFirst, you need to load the model.\n\n```py\n>>> from transformers import AutoModelForCausalLM\n\n>>> model = AutoModelForCausalLM.from_pretrained(\n...     \"mistralai/Mistral-7B-v0.1\", device_map=\"auto\", load_in_4bit=True\n... )\n```\n\nYou'll notice two flags in the `from_pretrained` call:\n\n - `device_map` ensures the model is moved to your GPU(s)\n - `load_in_4bit` applies [4-bit dynamic quantization](main_classes/quantization) to massively reduce the resource requirements\n\nThere are other ways to initialize a model, but this is a good baseline to begin with an LLM.\n\nNext, you need to preprocess your text input with a [tokenizer](tokenizer_summary).\n\n```py\n>>> from transformers import AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"mistralai/Mistral-7B-v0.1\", padding_side=\"left\")\n>>> model_inputs = tokenizer([\"A list of colors: red, blue\"], return_tensors=\"pt\").to(\"cuda\")\n```\n\nThe `model_inputs` variable holds the tokenized text input, as well as the attention mask. While [`~generation.GenerationMixin.generate`] does its best effort to infer the attention mask when it is not passed, we recommend passing it whenever possible for optimal results.\n\nAfter tokenizing the inputs, you can call the [`~generation.GenerationMixin.generate`] method to returns the generated tokens. The generated tokens then should be converted to text before printing.\n\n```py\n>>> generated_ids = model.generate(**model_inputs)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]\n'A list of colors: red, blue, green, yellow, orange, purple, pink,'\n```\n\nFinally, you don't need to do it one sequence at a time!"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 3fcce074-2442-4e56-85ad-0d6cb4410a50
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/llm_tutorial/", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"3fcce074-2442-4e56-85ad-0d6cb4410a50\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_tutorial/\", \"title\": \"Generation with LLMs\", \"tokens\": 3594, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3ecee411-8e31-4efa-af29-533880c0a63f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_tutorial/\", \"title\": \"Generation with LLMs\", \"tokens\": 3594, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"87f417fea89e56403f8d3d126dd0191fa18fec192e999880f186a8528147e92c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ba956f73-06bb-4190-9a10-90edfc9cb584\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_tutorial/\", \"title\": \"Generation with LLMs\", \"tokens\": 3594, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f982e6777473571e9c778879b55756892197dd5b728c30a279fd608a4c6d7849\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4ca38496-e838-4feb-b28c-14907d3df4f8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a11e967a82ca12cf79fcdc2c1c41f470651fe99f7ac274a0492d888249a6c76a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3617, \"end_char_idx\": 6010, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3ecee411-8e31-4efa-af29-533880c0a63f", "doc_id": "3ecee411-8e31-4efa-af29-533880c0a63f", "ref_doc_id": "3ecee411-8e31-4efa-af29-533880c0a63f", "chroma:document": "</Tip>\n\nFirst, you need to load the model.\n\n```py\n>>> from transformers import AutoModelForCausalLM\n\n>>> model = AutoModelForCausalLM.from_pretrained(\n...     \"mistralai/Mistral-7B-v0.1\", device_map=\"auto\", load_in_4bit=True\n... )\n```\n\nYou'll notice two flags in the `from_pretrained` call:\n\n - `device_map` ensures the model is moved to your GPU(s)\n - `load_in_4bit` applies [4-bit dynamic quantization](main_classes/quantization) to massively reduce the resource requirements\n\nThere are other ways to initialize a model, but this is a good baseline to begin with an LLM.\n\nNext, you need to preprocess your text input with a [tokenizer](tokenizer_summary).\n\n```py\n>>> from transformers import AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"mistralai/Mistral-7B-v0.1\", padding_side=\"left\")\n>>> model_inputs = tokenizer([\"A list of colors: red, blue\"], return_tensors=\"pt\").to(\"cuda\")\n```\n\nThe `model_inputs` variable holds the tokenized text input, as well as the attention mask. While [`~generation.GenerationMixin.generate`] does its best effort to infer the attention mask when it is not passed, we recommend passing it whenever possible for optimal results.\n\nAfter tokenizing the inputs, you can call the [`~generation.GenerationMixin.generate`] method to returns the generated tokens. The generated tokens then should be converted to text before printing.\n\n```py\n>>> generated_ids = model.generate(**model_inputs)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]\n'A list of colors: red, blue, green, yellow, orange, purple, pink,'\n```\n\nFinally, you don't need to do it one sequence at a time! You can batch your inputs, which will greatly improve the throughput at a small latency and memory cost. All you need to do is to make sure you pad your inputs properly (more on that below).\n\n```py\n>>> tokenizer.pad_token = tokenizer.eos_token  # Most LLMs don't have a pad token by default\n>>> model_inputs = tokenizer(\n...     [\"A list of colors: red, blue\", \"Portugal is\"], return_tensors=\"pt\", padding=True\n... ).to(\"cuda\")\n>>> generated_ids = model.generate(**model_inputs)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)\n['A list of colors: red, blue, green, yellow, orange, purple, pink,',\n'Portugal is a country in southwestern Europe, on the Iber']\n```\n\nAnd that's it! In a few lines of code, you can harness the power of an LLM."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 4ca38496-e838-4feb-b28c-14907d3df4f8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/llm_tutorial/", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"4ca38496-e838-4feb-b28c-14907d3df4f8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_tutorial/\", \"title\": \"Generation with LLMs\", \"tokens\": 3594, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3ecee411-8e31-4efa-af29-533880c0a63f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_tutorial/\", \"title\": \"Generation with LLMs\", \"tokens\": 3594, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"87f417fea89e56403f8d3d126dd0191fa18fec192e999880f186a8528147e92c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3fcce074-2442-4e56-85ad-0d6cb4410a50\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_tutorial/\", \"title\": \"Generation with LLMs\", \"tokens\": 3594, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"d00139dfca77d1ca658c02540f4efac5361b6d4ede89a571b7adf96ca9407831\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4dc989a4-6a0c-4365-9654-5612eb0302da\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"53d392ad922752b7299d8be8c25a38dd47f1c6dd0db6ee72ace1baf2fa6b2d34\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4612, \"end_char_idx\": 7132, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3ecee411-8e31-4efa-af29-533880c0a63f", "doc_id": "3ecee411-8e31-4efa-af29-533880c0a63f", "ref_doc_id": "3ecee411-8e31-4efa-af29-533880c0a63f", "chroma:document": "While [`~generation.GenerationMixin.generate`] does its best effort to infer the attention mask when it is not passed, we recommend passing it whenever possible for optimal results.\n\nAfter tokenizing the inputs, you can call the [`~generation.GenerationMixin.generate`] method to returns the generated tokens. The generated tokens then should be converted to text before printing.\n\n```py\n>>> generated_ids = model.generate(**model_inputs)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]\n'A list of colors: red, blue, green, yellow, orange, purple, pink,'\n```\n\nFinally, you don't need to do it one sequence at a time! You can batch your inputs, which will greatly improve the throughput at a small latency and memory cost. All you need to do is to make sure you pad your inputs properly (more on that below).\n\n```py\n>>> tokenizer.pad_token = tokenizer.eos_token  # Most LLMs don't have a pad token by default\n>>> model_inputs = tokenizer(\n...     [\"A list of colors: red, blue\", \"Portugal is\"], return_tensors=\"pt\", padding=True\n... ).to(\"cuda\")\n>>> generated_ids = model.generate(**model_inputs)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)\n['A list of colors: red, blue, green, yellow, orange, purple, pink,',\n'Portugal is a country in southwestern Europe, on the Iber']\n```\n\nAnd that's it! In a few lines of code, you can harness the power of an LLM.\n\n\n## Common pitfalls\n\nThere are many [generation strategies](generation_strategies), and sometimes the default values may not be appropriate for your use case. If your outputs aren't aligned with what you're expecting, we've created a list of the most common pitfalls and how to avoid them.\n\n```py\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"mistralai/Mistral-7B-v0.1\")\n>>> tokenizer.pad_token = tokenizer.eos_token  # Most LLMs don't have a pad token by default\n>>> model = AutoModelForCausalLM.from_pretrained(\n...     \"mistralai/Mistral-7B-v0.1\", device_map=\"auto\", load_in_4bit=True\n... )\n```\n\n### Generated output is too short/long\n\nIf not specified in the [`~generation.GenerationConfig`] file, `generate` returns up to 20 tokens by default. We highly recommend manually setting `max_new_tokens` in your `generate` call to control the maximum number of new tokens it can return. Keep in mind LLMs (more precisely, [decoder-only models](https://huggingface.co/learn/nlp-course/chapter1/6?fw=pt)) also return the input prompt as part of the output."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 4dc989a4-6a0c-4365-9654-5612eb0302da
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/llm_tutorial/", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"4dc989a4-6a0c-4365-9654-5612eb0302da\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_tutorial/\", \"title\": \"Generation with LLMs\", \"tokens\": 3594, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3ecee411-8e31-4efa-af29-533880c0a63f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_tutorial/\", \"title\": \"Generation with LLMs\", \"tokens\": 3594, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"87f417fea89e56403f8d3d126dd0191fa18fec192e999880f186a8528147e92c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4ca38496-e838-4feb-b28c-14907d3df4f8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_tutorial/\", \"title\": \"Generation with LLMs\", \"tokens\": 3594, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"18f0b82e66875d5e524655b77b286eac476d91467d2aa3fd3a273c8b8dad30dd\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"becdd942-28e6-45cf-85e0-84caf4a7d226\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"113cdfb0e5a37e902b267751b2595782b1841f7982bab0bb5c4da5af328728cf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5950, \"end_char_idx\": 9027, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3ecee411-8e31-4efa-af29-533880c0a63f", "doc_id": "3ecee411-8e31-4efa-af29-533880c0a63f", "ref_doc_id": "3ecee411-8e31-4efa-af29-533880c0a63f", "chroma:document": "In a few lines of code, you can harness the power of an LLM.\n\n\n## Common pitfalls\n\nThere are many [generation strategies](generation_strategies), and sometimes the default values may not be appropriate for your use case. If your outputs aren't aligned with what you're expecting, we've created a list of the most common pitfalls and how to avoid them.\n\n```py\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"mistralai/Mistral-7B-v0.1\")\n>>> tokenizer.pad_token = tokenizer.eos_token  # Most LLMs don't have a pad token by default\n>>> model = AutoModelForCausalLM.from_pretrained(\n...     \"mistralai/Mistral-7B-v0.1\", device_map=\"auto\", load_in_4bit=True\n... )\n```\n\n### Generated output is too short/long\n\nIf not specified in the [`~generation.GenerationConfig`] file, `generate` returns up to 20 tokens by default. We highly recommend manually setting `max_new_tokens` in your `generate` call to control the maximum number of new tokens it can return. Keep in mind LLMs (more precisely, [decoder-only models](https://huggingface.co/learn/nlp-course/chapter1/6?fw=pt)) also return the input prompt as part of the output.\n\n\n```py\n>>> model_inputs = tokenizer([\"A sequence of numbers: 1, 2\"], return_tensors=\"pt\").to(\"cuda\")\n\n>>> # By default, the output will contain up to 20 tokens\n>>> generated_ids = model.generate(**model_inputs)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]\n'A sequence of numbers: 1, 2, 3, 4, 5'\n\n>>> # Setting `max_new_tokens` allows you to control the maximum length\n>>> generated_ids = model.generate(**model_inputs, max_new_tokens=50)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]\n'A sequence of numbers: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16,'\n```\n\n### Incorrect generation mode\n\nBy default, and unless specified in the [`~generation.GenerationConfig`] file, `generate` selects the most likely token at each iteration (greedy decoding). Depending on your task, this may be undesirable; creative tasks like chatbots or writing an essay benefit from sampling. On the other hand, input-grounded tasks like audio transcription or translation benefit from greedy decoding. Enable sampling with `do_sample=True`, and you can learn more about this topic in this [blog post](https://huggingface.co/blog/how-to-generate).\n\n```py\n>>> # Set seed or reproducibility -- you don't need this unless you want full reproducibility\n>>> from transformers import set_seed\n>>> set_seed(42)\n\n>>> model_inputs = tokenizer([\"I am a cat.\"], return_tensors=\"pt\").to(\"cuda\")\n\n>>> # LLM + greedy decoding = repetitive, boring output\n>>> generated_ids = model.generate(**model_inputs)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]\n'I am a cat. I am a cat. I am a cat. I am a cat'\n\n>>> # With sampling, the output becomes more creative!\n>>> generated_ids = model.generate(**model_inputs, do_sample=True)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]\n'I am a cat.  Specifically, I am an indoor-only cat."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: becdd942-28e6-45cf-85e0-84caf4a7d226
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/llm_tutorial/", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"becdd942-28e6-45cf-85e0-84caf4a7d226\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_tutorial/\", \"title\": \"Generation with LLMs\", \"tokens\": 3594, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3ecee411-8e31-4efa-af29-533880c0a63f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_tutorial/\", \"title\": \"Generation with LLMs\", \"tokens\": 3594, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"87f417fea89e56403f8d3d126dd0191fa18fec192e999880f186a8528147e92c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4dc989a4-6a0c-4365-9654-5612eb0302da\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_tutorial/\", \"title\": \"Generation with LLMs\", \"tokens\": 3594, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"82fec938d93e159032195b730593f8173f2a6ec5cbffcabb38446be519a053d3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ebbaf416-74c3-45f8-ac85-d93b4eb36c12\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c4ae41c5777a36ba310ec9edb457541ab67e3b5379e0b262c94d74b7cd631834\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7942, \"end_char_idx\": 10796, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3ecee411-8e31-4efa-af29-533880c0a63f", "doc_id": "3ecee411-8e31-4efa-af29-533880c0a63f", "ref_doc_id": "3ecee411-8e31-4efa-af29-533880c0a63f", "chroma:document": "Depending on your task, this may be undesirable; creative tasks like chatbots or writing an essay benefit from sampling. On the other hand, input-grounded tasks like audio transcription or translation benefit from greedy decoding. Enable sampling with `do_sample=True`, and you can learn more about this topic in this [blog post](https://huggingface.co/blog/how-to-generate).\n\n```py\n>>> # Set seed or reproducibility -- you don't need this unless you want full reproducibility\n>>> from transformers import set_seed\n>>> set_seed(42)\n\n>>> model_inputs = tokenizer([\"I am a cat.\"], return_tensors=\"pt\").to(\"cuda\")\n\n>>> # LLM + greedy decoding = repetitive, boring output\n>>> generated_ids = model.generate(**model_inputs)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]\n'I am a cat. I am a cat. I am a cat. I am a cat'\n\n>>> # With sampling, the output becomes more creative!\n>>> generated_ids = model.generate(**model_inputs, do_sample=True)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]\n'I am a cat.  Specifically, I am an indoor-only cat.  I'\n```\n\n### Wrong padding side\n\nLLMs are [decoder-only](https://huggingface.co/learn/nlp-course/chapter1/6?fw=pt) architectures, meaning they continue to iterate on your input prompt. If your inputs do not have the same length, they need to be padded. Since LLMs are not trained to continue from pad tokens, your input needs to be left-padded. Make sure you also don't forget to pass the attention mask to generate!\n\n```py\n>>> # The tokenizer initialized above has right-padding active by default: the 1st sequence,\n>>> # which is shorter, has padding on the right side. Generation fails to capture the logic.\n>>> model_inputs = tokenizer(\n...     [\"1, 2, 3\", \"A, B, C, D, E\"], padding=True, return_tensors=\"pt\"\n... ).to(\"cuda\")\n>>> generated_ids = model.generate(**model_inputs)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]\n'1, 2, 33333333333'\n\n>>> # With left-padding, it works as expected!\n>>> tokenizer = AutoTokenizer.from_pretrained(\"mistralai/Mistral-7B-v0.1\", padding_side=\"left\")\n>>> tokenizer.pad_token = tokenizer.eos_token  # Most LLMs don't have a pad token by default\n>>> model_inputs = tokenizer(\n...     [\"1, 2, 3\", \"A, B, C, D, E\"], padding=True, return_tensors=\"pt\"\n... ).to(\"cuda\")\n>>> generated_ids = model.generate(**model_inputs)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]\n'1, 2, 3, 4, 5, 6,'\n```\n\n### Wrong prompt\n\nSome models and tasks expect a certain input prompt format to work properly. When this format is not applied, you will get a silent performance degradation: the model kinda works, but not as well as if you were following the expected prompt. More information about prompting, including which models and tasks need to be careful, is available in this [guide](tasks/prompting)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: ebbaf416-74c3-45f8-ac85-d93b4eb36c12
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/llm_tutorial/", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"ebbaf416-74c3-45f8-ac85-d93b4eb36c12\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_tutorial/\", \"title\": \"Generation with LLMs\", \"tokens\": 3594, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3ecee411-8e31-4efa-af29-533880c0a63f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_tutorial/\", \"title\": \"Generation with LLMs\", \"tokens\": 3594, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"87f417fea89e56403f8d3d126dd0191fa18fec192e999880f186a8528147e92c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"becdd942-28e6-45cf-85e0-84caf4a7d226\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_tutorial/\", \"title\": \"Generation with LLMs\", \"tokens\": 3594, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"662e52ce184091124a19c57bcd9737da77da0602ce0491405c5f2054026fb796\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a2782a97-7778-4d3b-90b5-30d3f7c8cffc\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a89520078fde28d8d36aadfd5688f9bed45706a5660444e89f25497e5b21f980\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9281, \"end_char_idx\": 12048, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3ecee411-8e31-4efa-af29-533880c0a63f", "doc_id": "3ecee411-8e31-4efa-af29-533880c0a63f", "ref_doc_id": "3ecee411-8e31-4efa-af29-533880c0a63f", "chroma:document": "Since LLMs are not trained to continue from pad tokens, your input needs to be left-padded. Make sure you also don't forget to pass the attention mask to generate!\n\n```py\n>>> # The tokenizer initialized above has right-padding active by default: the 1st sequence,\n>>> # which is shorter, has padding on the right side. Generation fails to capture the logic.\n>>> model_inputs = tokenizer(\n...     [\"1, 2, 3\", \"A, B, C, D, E\"], padding=True, return_tensors=\"pt\"\n... ).to(\"cuda\")\n>>> generated_ids = model.generate(**model_inputs)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]\n'1, 2, 33333333333'\n\n>>> # With left-padding, it works as expected!\n>>> tokenizer = AutoTokenizer.from_pretrained(\"mistralai/Mistral-7B-v0.1\", padding_side=\"left\")\n>>> tokenizer.pad_token = tokenizer.eos_token  # Most LLMs don't have a pad token by default\n>>> model_inputs = tokenizer(\n...     [\"1, 2, 3\", \"A, B, C, D, E\"], padding=True, return_tensors=\"pt\"\n... ).to(\"cuda\")\n>>> generated_ids = model.generate(**model_inputs)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]\n'1, 2, 3, 4, 5, 6,'\n```\n\n### Wrong prompt\n\nSome models and tasks expect a certain input prompt format to work properly. When this format is not applied, you will get a silent performance degradation: the model kinda works, but not as well as if you were following the expected prompt. More information about prompting, including which models and tasks need to be careful, is available in this [guide](tasks/prompting). Let's see an example with a chat LLM, which makes use of [chat templating](chat_templating):\n\n```python\n>>> tokenizer = AutoTokenizer.from_pretrained(\"HuggingFaceH4/zephyr-7b-alpha\")\n>>> model = AutoModelForCausalLM.from_pretrained(\n...     \"HuggingFaceH4/zephyr-7b-alpha\", device_map=\"auto\", load_in_4bit=True\n... )\n>>> set_seed(0)\n>>> prompt = \"\"\"How many helicopters can a human eat in one sitting? Reply as a thug.\"\"\"\n>>> model_inputs = tokenizer([prompt], return_tensors=\"pt\").to(\"cuda\")\n>>> input_length = model_inputs.input_ids.shape[1]\n>>> generated_ids = model.generate(**model_inputs, max_new_tokens=20)\n>>> print(tokenizer.batch_decode(generated_ids[:, input_length:], skip_special_tokens=True)[0])\n\"I'm not a thug, but i can tell you that a human cannot eat\"\n>>> # Oh no, it did not follow our instruction to reply as a thug! Let's see what happens when we write\n>>> # a better prompt and use the right template for this model (through `tokenizer.apply_chat_template`)\n\n>>> set_seed(0)\n>>> messages = [\n...     {\n...         \"role\": \"system\",\n...         \"content\": \"You are a friendly chatbot who always responds in the style of a thug\",\n...     },\n...     {\"role\": \"user\", \"content\": \"How many helicopters can a human eat in one sitting?"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: a2782a97-7778-4d3b-90b5-30d3f7c8cffc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/llm_tutorial/", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"a2782a97-7778-4d3b-90b5-30d3f7c8cffc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_tutorial/\", \"title\": \"Generation with LLMs\", \"tokens\": 3594, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3ecee411-8e31-4efa-af29-533880c0a63f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_tutorial/\", \"title\": \"Generation with LLMs\", \"tokens\": 3594, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"87f417fea89e56403f8d3d126dd0191fa18fec192e999880f186a8528147e92c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ebbaf416-74c3-45f8-ac85-d93b4eb36c12\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_tutorial/\", \"title\": \"Generation with LLMs\", \"tokens\": 3594, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"4ae5fc98018d7205edf19ce69f3b6c5ce4382a2b1f1b98e159549412b5cbd064\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8832e1e1-0c42-4871-9216-90d0b19f0b5b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"10a427314a0a5650b9c27bd9a76b199f2b2314d69d4b7604ecbef6aab780a3d5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10498, \"end_char_idx\": 13715, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3ecee411-8e31-4efa-af29-533880c0a63f", "doc_id": "3ecee411-8e31-4efa-af29-533880c0a63f", "ref_doc_id": "3ecee411-8e31-4efa-af29-533880c0a63f", "chroma:document": "When this format is not applied, you will get a silent performance degradation: the model kinda works, but not as well as if you were following the expected prompt. More information about prompting, including which models and tasks need to be careful, is available in this [guide](tasks/prompting). Let's see an example with a chat LLM, which makes use of [chat templating](chat_templating):\n\n```python\n>>> tokenizer = AutoTokenizer.from_pretrained(\"HuggingFaceH4/zephyr-7b-alpha\")\n>>> model = AutoModelForCausalLM.from_pretrained(\n...     \"HuggingFaceH4/zephyr-7b-alpha\", device_map=\"auto\", load_in_4bit=True\n... )\n>>> set_seed(0)\n>>> prompt = \"\"\"How many helicopters can a human eat in one sitting? Reply as a thug.\"\"\"\n>>> model_inputs = tokenizer([prompt], return_tensors=\"pt\").to(\"cuda\")\n>>> input_length = model_inputs.input_ids.shape[1]\n>>> generated_ids = model.generate(**model_inputs, max_new_tokens=20)\n>>> print(tokenizer.batch_decode(generated_ids[:, input_length:], skip_special_tokens=True)[0])\n\"I'm not a thug, but i can tell you that a human cannot eat\"\n>>> # Oh no, it did not follow our instruction to reply as a thug! Let's see what happens when we write\n>>> # a better prompt and use the right template for this model (through `tokenizer.apply_chat_template`)\n\n>>> set_seed(0)\n>>> messages = [\n...     {\n...         \"role\": \"system\",\n...         \"content\": \"You are a friendly chatbot who always responds in the style of a thug\",\n...     },\n...     {\"role\": \"user\", \"content\": \"How many helicopters can a human eat in one sitting?\"},\n... ]\n>>> model_inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors=\"pt\").to(\"cuda\")\n>>> input_length = model_inputs.shape[1]\n>>> generated_ids = model.generate(model_inputs, do_sample=True, max_new_tokens=20)\n>>> print(tokenizer.batch_decode(generated_ids[:, input_length:], skip_special_tokens=True)[0])\n'None, you thug. How bout you try to focus on more useful questions?'\n>>> # As we can see, it followed a proper thug style \ud83d\ude0e\n```\n\n## Further resources\n\nWhile the autoregressive generation process is relatively straightforward, making the most out of your LLM can be a challenging endeavor because there are many moving parts. For your next steps to help you dive deeper into LLM usage and understanding:\n\n### Advanced generate usage\n\n1. Guide on how to [control different generation methods](generation_strategies), how to set up the generation configuration file, and how to stream the output;\n2. [Accelerating text generation](llm_optims);\n3. [Prompt templates for chat LLMs](chat_templating);\n4. [Prompt design guide](tasks/prompting);\n5. API reference on [`~generation.GenerationConfig`], [`~generation.GenerationMixin.generate`], and [generate-related classes](internal/generation_utils). Most of the classes, including the logits processors, have usage examples!\n\n### LLM leaderboards\n\n1. [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard), which focuses on the quality of the open-source models;\n2. [Open LLM-Perf Leaderboard](https://huggingface.co/spaces/optimum/llm-perf-leaderboard), which focuses on LLM throughput.\n\n### Latency, throughput and memory utilization\n\n1."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 8832e1e1-0c42-4871-9216-90d0b19f0b5b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/llm_tutorial/", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"8832e1e1-0c42-4871-9216-90d0b19f0b5b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_tutorial/\", \"title\": \"Generation with LLMs\", \"tokens\": 3594, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3ecee411-8e31-4efa-af29-533880c0a63f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_tutorial/\", \"title\": \"Generation with LLMs\", \"tokens\": 3594, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"87f417fea89e56403f8d3d126dd0191fa18fec192e999880f186a8528147e92c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a2782a97-7778-4d3b-90b5-30d3f7c8cffc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/llm_tutorial/\", \"title\": \"Generation with LLMs\", \"tokens\": 3594, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"be787d5e68952bc6194edc5656ae85b58daac23b2a32f91a230b48d24046b353\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12048, \"end_char_idx\": 14486, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3ecee411-8e31-4efa-af29-533880c0a63f", "doc_id": "3ecee411-8e31-4efa-af29-533880c0a63f", "ref_doc_id": "3ecee411-8e31-4efa-af29-533880c0a63f", "chroma:document": "\"},\n... ]\n>>> model_inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors=\"pt\").to(\"cuda\")\n>>> input_length = model_inputs.shape[1]\n>>> generated_ids = model.generate(model_inputs, do_sample=True, max_new_tokens=20)\n>>> print(tokenizer.batch_decode(generated_ids[:, input_length:], skip_special_tokens=True)[0])\n'None, you thug. How bout you try to focus on more useful questions?'\n>>> # As we can see, it followed a proper thug style \ud83d\ude0e\n```\n\n## Further resources\n\nWhile the autoregressive generation process is relatively straightforward, making the most out of your LLM can be a challenging endeavor because there are many moving parts. For your next steps to help you dive deeper into LLM usage and understanding:\n\n### Advanced generate usage\n\n1. Guide on how to [control different generation methods](generation_strategies), how to set up the generation configuration file, and how to stream the output;\n2. [Accelerating text generation](llm_optims);\n3. [Prompt templates for chat LLMs](chat_templating);\n4. [Prompt design guide](tasks/prompting);\n5. API reference on [`~generation.GenerationConfig`], [`~generation.GenerationMixin.generate`], and [generate-related classes](internal/generation_utils). Most of the classes, including the logits processors, have usage examples!\n\n### LLM leaderboards\n\n1. [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard), which focuses on the quality of the open-source models;\n2. [Open LLM-Perf Leaderboard](https://huggingface.co/spaces/optimum/llm-perf-leaderboard), which focuses on LLM throughput.\n\n### Latency, throughput and memory utilization\n\n1. Guide on how to [optimize LLMs for speed and memory](llm_tutorial_optimization);\n2. Guide on [quantization](main_classes/quantization) such as bitsandbytes and autogptq, which shows you how to drastically reduce your memory requirements.\n\n### Related libraries\n\n1. [`optimum`](https://github.com/huggingface/optimum), an extension of \ud83e\udd17 Transformers that optimizes for specific hardware devices.\n2. [`outlines`](https://github.com/outlines-dev/outlines), a library where you can constrain text generation (e.g. to generate JSON files);\n3. [`text-generation-inference`](https://github.com/huggingface/text-generation-inference), a production-ready server for LLMs;\n4. [`text-generation-webui`](https://github.com/oobabooga/text-generation-webui), a UI for text generation;"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 4027a52e-5e56-4697-8b8e-b6540805578d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/torchscript/", "title": "Export to TorchScript", "tokens": 2219, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"4027a52e-5e56-4697-8b8e-b6540805578d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/torchscript/\", \"title\": \"Export to TorchScript\", \"tokens\": 2219, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5488e8e7-df1b-469e-9d4c-cf3b39ae31af\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/torchscript/\", \"title\": \"Export to TorchScript\", \"tokens\": 2219, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"dbe7f2f375c2112c34c1b7ce0a45743df1c5bc33d4786c28f846ede83925cbcc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6a9542aa-ec25-4bbf-98b3-562e087e1007\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"52e5ba33aed87ce49846f926e41404dc4fc7d133f0b6f5e5ad56f603cf8d21ef\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3273, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5488e8e7-df1b-469e-9d4c-cf3b39ae31af", "doc_id": "5488e8e7-df1b-469e-9d4c-cf3b39ae31af", "ref_doc_id": "5488e8e7-df1b-469e-9d4c-cf3b39ae31af", "chroma:document": "# Export to TorchScript\n\n<Tip>\n\nThis is the very beginning of our experiments with TorchScript and we are still\nexploring its capabilities with variable-input-size models. It is a focus of interest to\nus and we will deepen our analysis in upcoming releases, with more code examples, a more\nflexible implementation, and benchmarks comparing Python-based codes with compiled\nTorchScript.\n\n</Tip>\n\nAccording to the [TorchScript documentation](https://pytorch.org/docs/stable/jit.html):\n\n> TorchScript is a way to create serializable and optimizable models from PyTorch code.\n\nThere are two PyTorch modules, [JIT and\nTRACE](https://pytorch.org/docs/stable/jit.html), that allow developers to export their\nmodels to be reused in other programs like efficiency-oriented C++ programs.\n\nWe provide an interface that allows you to export \ud83e\udd17 Transformers models to TorchScript\nso they can be reused in a different environment than PyTorch-based Python programs.\nHere, we explain how to export and use our models using TorchScript.\n\nExporting a model requires two things:\n\n- model instantiation with the `torchscript` flag\n- a forward pass with dummy inputs\n\nThese necessities imply several things developers should be careful about as detailed\nbelow.\n\n## TorchScript flag and tied weights\n\nThe `torchscript` flag is necessary because most of the \ud83e\udd17 Transformers language models\nhave tied weights between their `Embedding` layer and their `Decoding` layer.\nTorchScript does not allow you to export models that have tied weights, so it is\nnecessary to untie and clone the weights beforehand.\n\nModels instantiated with the `torchscript` flag have their `Embedding` layer and\n`Decoding` layer separated, which means that they should not be trained down the line.\nTraining would desynchronize the two layers, leading to unexpected results.\n\nThis is not the case for models that do not have a language model head, as those do not\nhave tied weights. These models can be safely exported without the `torchscript` flag.\n\n## Dummy inputs and standard lengths\n\nThe dummy inputs are used for a models forward pass. While the inputs' values are\npropagated through the layers, PyTorch keeps track of the different operations executed\non each tensor. These recorded operations are then used to create the *trace* of the\nmodel.\n\nThe trace is created relative to the inputs' dimensions. It is therefore constrained by\nthe dimensions of the dummy input, and will not work for any other sequence length or\nbatch size. When trying with a different size, the following error is raised:\n\n```\n`The expanded size of the tensor (3) must match the existing size (7) at non-singleton dimension 2`\n```\n\nWe recommended you trace the model with a dummy input size at least as large as the\nlargest input that will be fed to the model during inference. Padding can help fill the\nmissing values. However, since the model is traced with a larger input size, the\ndimensions of the matrix will also be large, resulting in more calculations.\n\nBe careful of the total number of operations done on each input and follow the\nperformance closely when exporting varying sequence-length models.\n\n## Using TorchScript in Python\n\nThis section demonstrates how to save and load models as well as how to use the trace\nfor inference."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 6a9542aa-ec25-4bbf-98b3-562e087e1007
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/torchscript/", "title": "Export to TorchScript", "tokens": 2219, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"6a9542aa-ec25-4bbf-98b3-562e087e1007\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/torchscript/\", \"title\": \"Export to TorchScript\", \"tokens\": 2219, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5488e8e7-df1b-469e-9d4c-cf3b39ae31af\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/torchscript/\", \"title\": \"Export to TorchScript\", \"tokens\": 2219, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"dbe7f2f375c2112c34c1b7ce0a45743df1c5bc33d4786c28f846ede83925cbcc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4027a52e-5e56-4697-8b8e-b6540805578d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/torchscript/\", \"title\": \"Export to TorchScript\", \"tokens\": 2219, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"79e20a55b702951e1fded9c7c149e4ce8070feea5919efbad7655a5b970a99c2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bcb660d9-fb70-4fd7-8d51-47199ef7fbd1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3e01464f563ce2f71b32676684c7482e3face88131c2997f8c07b500a5c2834d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1444, \"end_char_idx\": 4263, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5488e8e7-df1b-469e-9d4c-cf3b39ae31af", "doc_id": "5488e8e7-df1b-469e-9d4c-cf3b39ae31af", "ref_doc_id": "5488e8e7-df1b-469e-9d4c-cf3b39ae31af", "chroma:document": "TorchScript does not allow you to export models that have tied weights, so it is\nnecessary to untie and clone the weights beforehand.\n\nModels instantiated with the `torchscript` flag have their `Embedding` layer and\n`Decoding` layer separated, which means that they should not be trained down the line.\nTraining would desynchronize the two layers, leading to unexpected results.\n\nThis is not the case for models that do not have a language model head, as those do not\nhave tied weights. These models can be safely exported without the `torchscript` flag.\n\n## Dummy inputs and standard lengths\n\nThe dummy inputs are used for a models forward pass. While the inputs' values are\npropagated through the layers, PyTorch keeps track of the different operations executed\non each tensor. These recorded operations are then used to create the *trace* of the\nmodel.\n\nThe trace is created relative to the inputs' dimensions. It is therefore constrained by\nthe dimensions of the dummy input, and will not work for any other sequence length or\nbatch size. When trying with a different size, the following error is raised:\n\n```\n`The expanded size of the tensor (3) must match the existing size (7) at non-singleton dimension 2`\n```\n\nWe recommended you trace the model with a dummy input size at least as large as the\nlargest input that will be fed to the model during inference. Padding can help fill the\nmissing values. However, since the model is traced with a larger input size, the\ndimensions of the matrix will also be large, resulting in more calculations.\n\nBe careful of the total number of operations done on each input and follow the\nperformance closely when exporting varying sequence-length models.\n\n## Using TorchScript in Python\n\nThis section demonstrates how to save and load models as well as how to use the trace\nfor inference.\n\n### Saving a model\n\nTo export a `BertModel` with TorchScript, instantiate `BertModel` from the `BertConfig`\nclass and then save it to disk under the filename `traced_bert.pt`:\n\n```python\nfrom transformers import BertModel, BertTokenizer, BertConfig\nimport torch\n\nenc = BertTokenizer.from_pretrained(\"google-bert/bert-base-uncased\")\n\n# Tokenizing input text\ntext = \"[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]\"\ntokenized_text = enc.tokenize(text)\n\n# Masking one of the input tokens\nmasked_index = 8\ntokenized_text[masked_index] = \"[MASK]\"\nindexed_tokens = enc.convert_tokens_to_ids(tokenized_text)\nsegments_ids = [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1]\n\n# Creating a dummy input\ntokens_tensor = torch.tensor([indexed_tokens])\nsegments_tensors = torch.tensor([segments_ids])\ndummy_input = [tokens_tensor, segments_tensors]\n\n# Initializing the model with the torchscript flag\n# Flag set to True even though it is not necessary as this model does not have an LM Head."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: bcb660d9-fb70-4fd7-8d51-47199ef7fbd1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/torchscript/", "title": "Export to TorchScript", "tokens": 2219, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"bcb660d9-fb70-4fd7-8d51-47199ef7fbd1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/torchscript/\", \"title\": \"Export to TorchScript\", \"tokens\": 2219, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5488e8e7-df1b-469e-9d4c-cf3b39ae31af\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/torchscript/\", \"title\": \"Export to TorchScript\", \"tokens\": 2219, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"dbe7f2f375c2112c34c1b7ce0a45743df1c5bc33d4786c28f846ede83925cbcc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6a9542aa-ec25-4bbf-98b3-562e087e1007\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/torchscript/\", \"title\": \"Export to TorchScript\", \"tokens\": 2219, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"86eefb45f86937df42d55a68673b2c2051eae8379077366f56bd3a186475e8af\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c9dc1fdb-7b99-45d9-a7b6-2be16e38e778\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"903ed2d58ac048bf9b1a1582a8992dc03fa1b128394875b48eb5f15d182779d9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2809, \"end_char_idx\": 5978, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5488e8e7-df1b-469e-9d4c-cf3b39ae31af", "doc_id": "5488e8e7-df1b-469e-9d4c-cf3b39ae31af", "ref_doc_id": "5488e8e7-df1b-469e-9d4c-cf3b39ae31af", "chroma:document": "Padding can help fill the\nmissing values. However, since the model is traced with a larger input size, the\ndimensions of the matrix will also be large, resulting in more calculations.\n\nBe careful of the total number of operations done on each input and follow the\nperformance closely when exporting varying sequence-length models.\n\n## Using TorchScript in Python\n\nThis section demonstrates how to save and load models as well as how to use the trace\nfor inference.\n\n### Saving a model\n\nTo export a `BertModel` with TorchScript, instantiate `BertModel` from the `BertConfig`\nclass and then save it to disk under the filename `traced_bert.pt`:\n\n```python\nfrom transformers import BertModel, BertTokenizer, BertConfig\nimport torch\n\nenc = BertTokenizer.from_pretrained(\"google-bert/bert-base-uncased\")\n\n# Tokenizing input text\ntext = \"[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]\"\ntokenized_text = enc.tokenize(text)\n\n# Masking one of the input tokens\nmasked_index = 8\ntokenized_text[masked_index] = \"[MASK]\"\nindexed_tokens = enc.convert_tokens_to_ids(tokenized_text)\nsegments_ids = [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1]\n\n# Creating a dummy input\ntokens_tensor = torch.tensor([indexed_tokens])\nsegments_tensors = torch.tensor([segments_ids])\ndummy_input = [tokens_tensor, segments_tensors]\n\n# Initializing the model with the torchscript flag\n# Flag set to True even though it is not necessary as this model does not have an LM Head.\nconfig = BertConfig(\n    vocab_size_or_config_json_file=32000,\n    hidden_size=768,\n    num_hidden_layers=12,\n    num_attention_heads=12,\n    intermediate_size=3072,\n    torchscript=True,\n)\n\n# Instantiating the model\nmodel = BertModel(config)\n\n# The model needs to be in evaluation mode\nmodel.eval()\n\n# If you are instantiating the model with *from_pretrained* you can also easily set the TorchScript flag\nmodel = BertModel.from_pretrained(\"google-bert/bert-base-uncased\", torchscript=True)\n\n# Creating the trace\ntraced_model = torch.jit.trace(model, [tokens_tensor, segments_tensors])\ntorch.jit.save(traced_model, \"traced_bert.pt\")\n```\n\n### Loading a model\n\nNow you can load the previously saved `BertModel`, `traced_bert.pt`, from disk and use\nit on the previously initialised `dummy_input`:\n\n```python\nloaded_model = torch.jit.load(\"traced_bert.pt\")\nloaded_model.eval()\n\nall_encoder_layers, pooled_output = loaded_model(*dummy_input)\n```\n\n### Using a traced model for inference\n\nUse the traced model for inference by using its `__call__` dunder method:\n\n```python\ntraced_model(tokens_tensor, segments_tensors)\n```\n\n## Deploy Hugging Face TorchScript models to AWS with the Neuron SDK\n\nAWS introduced the [Amazon EC2 Inf1](https://aws.amazon.com/ec2/instance-types/inf1/)\ninstance family for low cost, high performance machine learning inference in the cloud.\nThe Inf1 instances are powered by the AWS Inferentia chip, a custom-built hardware\naccelerator, specializing in deep learning inferencing workloads. [AWS\nNeuron](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/#) is the SDK for\nInferentia that supports tracing and optimizing transformers models for deployment on\nInf1. The Neuron SDK provides:"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: c9dc1fdb-7b99-45d9-a7b6-2be16e38e778
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/torchscript/", "title": "Export to TorchScript", "tokens": 2219, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"c9dc1fdb-7b99-45d9-a7b6-2be16e38e778\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/torchscript/\", \"title\": \"Export to TorchScript\", \"tokens\": 2219, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5488e8e7-df1b-469e-9d4c-cf3b39ae31af\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/torchscript/\", \"title\": \"Export to TorchScript\", \"tokens\": 2219, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"dbe7f2f375c2112c34c1b7ce0a45743df1c5bc33d4786c28f846ede83925cbcc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bcb660d9-fb70-4fd7-8d51-47199ef7fbd1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/torchscript/\", \"title\": \"Export to TorchScript\", \"tokens\": 2219, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8a3fc84f4d5a505e5a7d36e017b485488b19b4500b2f3299ac38151774351aee\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5626, \"end_char_idx\": 8863, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5488e8e7-df1b-469e-9d4c-cf3b39ae31af", "doc_id": "5488e8e7-df1b-469e-9d4c-cf3b39ae31af", "ref_doc_id": "5488e8e7-df1b-469e-9d4c-cf3b39ae31af", "chroma:document": "The Inf1 instances are powered by the AWS Inferentia chip, a custom-built hardware\naccelerator, specializing in deep learning inferencing workloads. [AWS\nNeuron](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/#) is the SDK for\nInferentia that supports tracing and optimizing transformers models for deployment on\nInf1. The Neuron SDK provides:\n\n\n1. Easy-to-use API with one line of code change to trace and optimize a TorchScript\n   model for inference in the cloud.\n2. Out of the box performance optimizations for [improved\n   cost-performance](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-guide/benchmark/>).\n3. Support for Hugging Face transformers models built with either\n   [PyTorch](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/src/examples/pytorch/bert_tutorial/tutorial_pretrained_bert.html)\n   or\n   [TensorFlow](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/src/examples/tensorflow/huggingface_bert/huggingface_bert.html).\n\n### Implications\n\nTransformers models based on the [BERT (Bidirectional Encoder Representations from\nTransformers)](https://huggingface.co/docs/transformers/main/model_doc/bert)\narchitecture, or its variants such as\n[distilBERT](https://huggingface.co/docs/transformers/main/model_doc/distilbert) and\n[roBERTa](https://huggingface.co/docs/transformers/main/model_doc/roberta) run best on\nInf1 for non-generative tasks such as extractive question answering, sequence\nclassification, and token classification. However, text generation tasks can still be\nadapted to run on Inf1 according to this [AWS Neuron MarianMT\ntutorial](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/src/examples/pytorch/transformers-marianmt.html).\nMore information about models that can be converted out of the box on Inferentia can be\nfound in the [Model Architecture\nFit](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-guide/models/models-inferentia.html#models-inferentia)\nsection of the Neuron documentation.\n\n### Dependencies\n\nUsing AWS Neuron to convert models requires a [Neuron SDK\nenvironment](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-guide/neuron-frameworks/pytorch-neuron/index.html#installation-guide)\nwhich comes preconfigured on [AWS Deep Learning\nAMI](https://docs.aws.amazon.com/dlami/latest/devguide/tutorial-inferentia-launching.html).\n\n### Converting a model for AWS Neuron\n\nConvert a model for AWS NEURON using the same code from [Using TorchScript in\nPython](torchscript#using-torchscript-in-python) to trace a `BertModel`. Import the\n`torch.neuron` framework extension to access the components of the Neuron SDK through a\nPython API:\n\n```python\nfrom transformers import BertModel, BertTokenizer, BertConfig\nimport torch\nimport torch.neuron\n```\n\nYou only need to modify the following line:\n\n```diff\n- torch.jit.trace(model, [tokens_tensor, segments_tensors])\n+ torch.neuron.trace(model, [token_tensor, segments_tensors])\n```\n\nThis enables the Neuron SDK to trace the model and optimize it for Inf1 instances.\n\nTo learn more about AWS Neuron SDK features, tools, example tutorials and latest\nupdates, please see the [AWS NeuronSDK\ndocumentation](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/index.html)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 2a137cf0-b5ba-4c86-a95b-56942746cfb7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf/", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"2a137cf0-b5ba-4c86-a95b-56942746cfb7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_tpu_tf/\", \"title\": \"Training on TPU with TensorFlow\", \"tokens\": 3205, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ea1b958a-2bac-447b-9416-fa8b4a2ff755\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_tpu_tf/\", \"title\": \"Training on TPU with TensorFlow\", \"tokens\": 3205, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b74dfc96b97a641db4d95da420a47e4c3448d0eec08ce939ced40c51264461d5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fde06c30-8b87-4a2e-a6d6-d1dbc82752c2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d2d5e0eddc9d1b1deab50d5955d2a86fa61a989c75273fe3b9038b327fa732ab\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3323, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ea1b958a-2bac-447b-9416-fa8b4a2ff755", "doc_id": "ea1b958a-2bac-447b-9416-fa8b4a2ff755", "ref_doc_id": "ea1b958a-2bac-447b-9416-fa8b4a2ff755", "chroma:document": "# Training on TPU with TensorFlow\n\n<Tip>\n\nIf you don't need long explanations and just want TPU code samples to get started with, check out [our TPU example notebook!](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/tpu_training-tf.ipynb)\n\n</Tip>\n\n### What is a TPU?\n\nA TPU is a **Tensor Processing Unit.** They are hardware designed by Google, which are used to greatly speed up the tensor computations within neural networks, much like GPUs. They can be used for both network training and inference. They are generally accessed through Google\u2019s cloud services, but small TPUs can also be accessed directly for free through Google Colab and Kaggle Kernels.\n\nBecause [all TensorFlow models in \ud83e\udd17 Transformers are Keras models](https://huggingface.co/blog/tensorflow-philosophy), most of the methods in this document are generally applicable to TPU training for any Keras model! However, there are a few points that are specific to the HuggingFace ecosystem (hug-o-system?) of Transformers and Datasets, and we\u2019ll make sure to flag them up when we get to them.\n\n### What kinds of TPU are available?\n\nNew users are often very confused by the range of TPUs, and the different ways to access them. The first key distinction to understand is the difference between **TPU Nodes** and **TPU VMs.**\n\nWhen you use a **TPU Node**, you are effectively indirectly accessing a remote TPU. You will need a separate VM, which will initialize your network and data pipeline and then forward them to the remote node. When you use a TPU on Google Colab, you are accessing it in the **TPU Node** style.\n\nUsing TPU Nodes can have some quite unexpected behaviour for people who aren\u2019t used to them! In particular, because the TPU is located on a physically different system to the machine you\u2019re running your Python code on, your data cannot be local to your machine - any data pipeline that loads from your machine\u2019s internal storage will totally fail! Instead, data must be stored in Google Cloud Storage where your data pipeline can still access it, even when the pipeline is running on the remote TPU node.\n\n<Tip>\n\nIf you can fit all your data in memory as `np.ndarray` or `tf.Tensor`, then you can `fit()` on that data even when using Colab or a TPU Node, without needing to upload it to Google Cloud Storage.\n\n</Tip>\n\n<Tip>\n\n**\ud83e\udd17Specific Hugging Face Tip\ud83e\udd17:** The methods `Dataset.to_tf_dataset()` and its higher-level wrapper `model.prepare_tf_dataset()` , which you will see throughout our TF code examples, will both fail on a TPU Node. The reason for this is that even though they create a `tf.data.Dataset` it is not a \u201cpure\u201d `tf.data` pipeline and uses `tf.numpy_function` or `Dataset.from_generator()` to stream data from the underlying HuggingFace `Dataset`. This HuggingFace `Dataset` is backed by data that is on a local disc and which the remote TPU Node will not be able to read.\n\n</Tip>\n\nThe second way to access a TPU is via a **TPU VM.** When using a TPU VM, you connect directly to the machine that the TPU is attached to, much like training on a GPU VM. TPU VMs are generally easier to work with, particularly when it comes to your data pipeline. All of the above warnings do not apply to TPU VMs!\n\nThis is an opinionated document, so here\u2019s our opinion: **Avoid using TPU Node if possible."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: fde06c30-8b87-4a2e-a6d6-d1dbc82752c2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf/", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"fde06c30-8b87-4a2e-a6d6-d1dbc82752c2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_tpu_tf/\", \"title\": \"Training on TPU with TensorFlow\", \"tokens\": 3205, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ea1b958a-2bac-447b-9416-fa8b4a2ff755\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_tpu_tf/\", \"title\": \"Training on TPU with TensorFlow\", \"tokens\": 3205, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b74dfc96b97a641db4d95da420a47e4c3448d0eec08ce939ced40c51264461d5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2a137cf0-b5ba-4c86-a95b-56942746cfb7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_tpu_tf/\", \"title\": \"Training on TPU with TensorFlow\", \"tokens\": 3205, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"4ca8acbf58680a9bfe46c25f65c3f91a76f25b2f43e27df163841119b4fad291\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"82599d5b-cfac-4a5e-ba7d-f16ddbe41f70\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e50e8902689a2231c6ea9c642a024754c791fa7623a555fa69bef0f87a71637b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1711, \"end_char_idx\": 4800, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ea1b958a-2bac-447b-9416-fa8b4a2ff755", "doc_id": "ea1b958a-2bac-447b-9416-fa8b4a2ff755", "ref_doc_id": "ea1b958a-2bac-447b-9416-fa8b4a2ff755", "chroma:document": "In particular, because the TPU is located on a physically different system to the machine you\u2019re running your Python code on, your data cannot be local to your machine - any data pipeline that loads from your machine\u2019s internal storage will totally fail! Instead, data must be stored in Google Cloud Storage where your data pipeline can still access it, even when the pipeline is running on the remote TPU node.\n\n<Tip>\n\nIf you can fit all your data in memory as `np.ndarray` or `tf.Tensor`, then you can `fit()` on that data even when using Colab or a TPU Node, without needing to upload it to Google Cloud Storage.\n\n</Tip>\n\n<Tip>\n\n**\ud83e\udd17Specific Hugging Face Tip\ud83e\udd17:** The methods `Dataset.to_tf_dataset()` and its higher-level wrapper `model.prepare_tf_dataset()` , which you will see throughout our TF code examples, will both fail on a TPU Node. The reason for this is that even though they create a `tf.data.Dataset` it is not a \u201cpure\u201d `tf.data` pipeline and uses `tf.numpy_function` or `Dataset.from_generator()` to stream data from the underlying HuggingFace `Dataset`. This HuggingFace `Dataset` is backed by data that is on a local disc and which the remote TPU Node will not be able to read.\n\n</Tip>\n\nThe second way to access a TPU is via a **TPU VM.** When using a TPU VM, you connect directly to the machine that the TPU is attached to, much like training on a GPU VM. TPU VMs are generally easier to work with, particularly when it comes to your data pipeline. All of the above warnings do not apply to TPU VMs!\n\nThis is an opinionated document, so here\u2019s our opinion: **Avoid using TPU Node if possible.** It is more confusing and more difficult to debug than TPU VMs. It is also likely to be unsupported in future - Google\u2019s latest TPU, TPUv4, can only be accessed as a TPU VM, which suggests that TPU Nodes are increasingly going to become a \u201clegacy\u201d access method. However, we understand that the only free TPU access is on Colab and Kaggle Kernels, which uses TPU Node - so we\u2019ll try to explain how to handle it if you have to! Check the [TPU example notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/tpu_training-tf.ipynb) for code samples that explain this in more detail.\n\n### What sizes of TPU are available?\n\nA single TPU (a v2-8/v3-8/v4-8) runs 8 replicas. TPUs exist in **pods** that can run hundreds or thousands of replicas simultaneously. When you use more than a single TPU but less than a whole pod (for example, a v3-32), your TPU fleet is referred to as a **pod slice.**\n\nWhen you access a free TPU via Colab, you generally get a single v2-8 TPU.\n\n### I keep hearing about this XLA thing. What\u2019s XLA, and how does it relate to TPUs?\n\nXLA is an optimizing compiler, used by both TensorFlow and JAX. In JAX it is the only compiler, whereas in TensorFlow it is optional (but mandatory on TPU!). The easiest way to enable it when training a Keras model is to pass the argument `jit_compile=True` to `model.compile()`. If you don\u2019t get any errors and performance is good, that\u2019s a great sign that you\u2019re ready to move to TPU!"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 82599d5b-cfac-4a5e-ba7d-f16ddbe41f70
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf/", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"82599d5b-cfac-4a5e-ba7d-f16ddbe41f70\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_tpu_tf/\", \"title\": \"Training on TPU with TensorFlow\", \"tokens\": 3205, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ea1b958a-2bac-447b-9416-fa8b4a2ff755\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_tpu_tf/\", \"title\": \"Training on TPU with TensorFlow\", \"tokens\": 3205, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b74dfc96b97a641db4d95da420a47e4c3448d0eec08ce939ced40c51264461d5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"fde06c30-8b87-4a2e-a6d6-d1dbc82752c2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_tpu_tf/\", \"title\": \"Training on TPU with TensorFlow\", \"tokens\": 3205, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"4911e6af3202b9165402b18b93c7998940e16e19c7a8ba47a11810b595b920df\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"48ae939f-8e40-4d00-a06e-b65a877efae5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e0c77fa278d5c05fb22c5b5b9024535e7dbd512a5a7d187adf1c1ee610501ecc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3323, \"end_char_idx\": 6338, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ea1b958a-2bac-447b-9416-fa8b4a2ff755", "doc_id": "ea1b958a-2bac-447b-9416-fa8b4a2ff755", "ref_doc_id": "ea1b958a-2bac-447b-9416-fa8b4a2ff755", "chroma:document": "** It is more confusing and more difficult to debug than TPU VMs. It is also likely to be unsupported in future - Google\u2019s latest TPU, TPUv4, can only be accessed as a TPU VM, which suggests that TPU Nodes are increasingly going to become a \u201clegacy\u201d access method. However, we understand that the only free TPU access is on Colab and Kaggle Kernels, which uses TPU Node - so we\u2019ll try to explain how to handle it if you have to! Check the [TPU example notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/tpu_training-tf.ipynb) for code samples that explain this in more detail.\n\n### What sizes of TPU are available?\n\nA single TPU (a v2-8/v3-8/v4-8) runs 8 replicas. TPUs exist in **pods** that can run hundreds or thousands of replicas simultaneously. When you use more than a single TPU but less than a whole pod (for example, a v3-32), your TPU fleet is referred to as a **pod slice.**\n\nWhen you access a free TPU via Colab, you generally get a single v2-8 TPU.\n\n### I keep hearing about this XLA thing. What\u2019s XLA, and how does it relate to TPUs?\n\nXLA is an optimizing compiler, used by both TensorFlow and JAX. In JAX it is the only compiler, whereas in TensorFlow it is optional (but mandatory on TPU!). The easiest way to enable it when training a Keras model is to pass the argument `jit_compile=True` to `model.compile()`. If you don\u2019t get any errors and performance is good, that\u2019s a great sign that you\u2019re ready to move to TPU!\n\nDebugging on TPU is generally a bit harder than on CPU/GPU, so we recommend getting your code running on CPU/GPU with XLA first before trying it on TPU. You don\u2019t have to train for long, of course - just for a few steps to make sure that your model and data pipeline are working like you expect them to.\n\n<Tip>\n\nXLA compiled code is usually faster - so even if you\u2019re not planning to run on TPU, adding `jit_compile=True` can improve your performance. Be sure to note the caveats below about XLA compatibility, though!\n\n</Tip>\n\n<Tip warning={true}>\n\n**Tip born of painful experience:** Although using `jit_compile=True` is a good way to get a speed boost and test if your CPU/GPU code is XLA-compatible, it can actually cause a lot of problems if you leave it in when actually training on TPU. XLA compilation will happen implicitly on TPU, so remember to remove that line before actually running your code on a TPU!\n\n</Tip>\n\n### How do I make my model XLA compatible?\n\nIn many cases, your code is probably XLA-compatible already! However, there are a few things that work in normal TensorFlow that don\u2019t work in XLA. We\u2019ve distilled them into three core rules below:\n\n<Tip>\n\n**\ud83e\udd17Specific HuggingFace Tip\ud83e\udd17:** We\u2019ve put a lot of effort into rewriting our TensorFlow models and loss functions to be XLA-compatible. Our models and loss functions generally obey rule #1 and #2 by default, so you can skip over them if you\u2019re using `transformers` models. Don\u2019t forget about these rules when writing your own models and loss functions, though!"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 48ae939f-8e40-4d00-a06e-b65a877efae5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf/", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"48ae939f-8e40-4d00-a06e-b65a877efae5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_tpu_tf/\", \"title\": \"Training on TPU with TensorFlow\", \"tokens\": 3205, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ea1b958a-2bac-447b-9416-fa8b4a2ff755\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_tpu_tf/\", \"title\": \"Training on TPU with TensorFlow\", \"tokens\": 3205, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b74dfc96b97a641db4d95da420a47e4c3448d0eec08ce939ced40c51264461d5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"82599d5b-cfac-4a5e-ba7d-f16ddbe41f70\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_tpu_tf/\", \"title\": \"Training on TPU with TensorFlow\", \"tokens\": 3205, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"af83313da1d1862991138655f1d564470c24f58310f4916315502c054c01c675\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9ea3724e-8404-4cd5-bc0e-604ebbb53fa6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8b013ddea85fd54f6b6dfa37639c70ee7a4ae87425db59ff8eb0620cd7d8fc65\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4694, \"end_char_idx\": 7840, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ea1b958a-2bac-447b-9416-fa8b4a2ff755", "doc_id": "ea1b958a-2bac-447b-9416-fa8b4a2ff755", "ref_doc_id": "ea1b958a-2bac-447b-9416-fa8b4a2ff755", "chroma:document": "If you don\u2019t get any errors and performance is good, that\u2019s a great sign that you\u2019re ready to move to TPU!\n\nDebugging on TPU is generally a bit harder than on CPU/GPU, so we recommend getting your code running on CPU/GPU with XLA first before trying it on TPU. You don\u2019t have to train for long, of course - just for a few steps to make sure that your model and data pipeline are working like you expect them to.\n\n<Tip>\n\nXLA compiled code is usually faster - so even if you\u2019re not planning to run on TPU, adding `jit_compile=True` can improve your performance. Be sure to note the caveats below about XLA compatibility, though!\n\n</Tip>\n\n<Tip warning={true}>\n\n**Tip born of painful experience:** Although using `jit_compile=True` is a good way to get a speed boost and test if your CPU/GPU code is XLA-compatible, it can actually cause a lot of problems if you leave it in when actually training on TPU. XLA compilation will happen implicitly on TPU, so remember to remove that line before actually running your code on a TPU!\n\n</Tip>\n\n### How do I make my model XLA compatible?\n\nIn many cases, your code is probably XLA-compatible already! However, there are a few things that work in normal TensorFlow that don\u2019t work in XLA. We\u2019ve distilled them into three core rules below:\n\n<Tip>\n\n**\ud83e\udd17Specific HuggingFace Tip\ud83e\udd17:** We\u2019ve put a lot of effort into rewriting our TensorFlow models and loss functions to be XLA-compatible. Our models and loss functions generally obey rule #1 and #2 by default, so you can skip over them if you\u2019re using `transformers` models. Don\u2019t forget about these rules when writing your own models and loss functions, though!\n\n</Tip>\n\n#### XLA Rule #1: Your code cannot have \u201cdata-dependent conditionals\u201d\n\nWhat that means is that any `if` statement cannot depend on values inside a `tf.Tensor`. For example, this code block cannot be compiled with XLA!\n\n```python\nif tf.reduce_sum(tensor) > 10:\n    tensor = tensor / 2.0\n```\n\nThis might seem very restrictive at first, but most neural net code doesn\u2019t need to do this. You can often get around this restriction by using `tf.cond` (see the documentation [here](https://www.tensorflow.org/api_docs/python/tf/cond)) or by removing the conditional and finding a clever math trick with indicator variables instead, like so:\n\n```python\nsum_over_10 = tf.cast(tf.reduce_sum(tensor) > 10, tf.float32)\ntensor = tensor / (1.0 + sum_over_10)\n```\n\nThis code has exactly the same effect as the code above, but by avoiding a conditional, we ensure it will compile with XLA without problems!\n\n#### XLA Rule #2: Your code cannot have \u201cdata-dependent shapes\u201d\n\nWhat this means is that the shape of all of the `tf.Tensor` objects in your code cannot depend on their values. For example, the function `tf.unique` cannot be compiled with XLA, because it returns a `tensor` containing one instance of each unique value in the input. The shape of this output will obviously be different depending on how repetitive the input `Tensor` was, and so XLA refuses to handle it!\n\nIn general, most neural network code obeys rule #2 by default. However, there are a few common cases where it becomes a problem."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 9ea3724e-8404-4cd5-bc0e-604ebbb53fa6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf/", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"9ea3724e-8404-4cd5-bc0e-604ebbb53fa6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_tpu_tf/\", \"title\": \"Training on TPU with TensorFlow\", \"tokens\": 3205, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ea1b958a-2bac-447b-9416-fa8b4a2ff755\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_tpu_tf/\", \"title\": \"Training on TPU with TensorFlow\", \"tokens\": 3205, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b74dfc96b97a641db4d95da420a47e4c3448d0eec08ce939ced40c51264461d5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"48ae939f-8e40-4d00-a06e-b65a877efae5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_tpu_tf/\", \"title\": \"Training on TPU with TensorFlow\", \"tokens\": 3205, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b56620b114a803aa406473db411fceb4e291558ba6615c550ae4ddeb5aec197f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"38f6cb4d-f4d6-42b2-97d1-67e4ab57c7a5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"58230d26aa0812f7a2d4e8e8ad5753b88a7bca534ec119532e0f8f34be89ea9e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6251, \"end_char_idx\": 9602, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ea1b958a-2bac-447b-9416-fa8b4a2ff755", "doc_id": "ea1b958a-2bac-447b-9416-fa8b4a2ff755", "ref_doc_id": "ea1b958a-2bac-447b-9416-fa8b4a2ff755", "chroma:document": "Don\u2019t forget about these rules when writing your own models and loss functions, though!\n\n</Tip>\n\n#### XLA Rule #1: Your code cannot have \u201cdata-dependent conditionals\u201d\n\nWhat that means is that any `if` statement cannot depend on values inside a `tf.Tensor`. For example, this code block cannot be compiled with XLA!\n\n```python\nif tf.reduce_sum(tensor) > 10:\n    tensor = tensor / 2.0\n```\n\nThis might seem very restrictive at first, but most neural net code doesn\u2019t need to do this. You can often get around this restriction by using `tf.cond` (see the documentation [here](https://www.tensorflow.org/api_docs/python/tf/cond)) or by removing the conditional and finding a clever math trick with indicator variables instead, like so:\n\n```python\nsum_over_10 = tf.cast(tf.reduce_sum(tensor) > 10, tf.float32)\ntensor = tensor / (1.0 + sum_over_10)\n```\n\nThis code has exactly the same effect as the code above, but by avoiding a conditional, we ensure it will compile with XLA without problems!\n\n#### XLA Rule #2: Your code cannot have \u201cdata-dependent shapes\u201d\n\nWhat this means is that the shape of all of the `tf.Tensor` objects in your code cannot depend on their values. For example, the function `tf.unique` cannot be compiled with XLA, because it returns a `tensor` containing one instance of each unique value in the input. The shape of this output will obviously be different depending on how repetitive the input `Tensor` was, and so XLA refuses to handle it!\n\nIn general, most neural network code obeys rule #2 by default. However, there are a few common cases where it becomes a problem. One very common one is when you use **label masking**, setting your labels to a negative value to indicate that those positions should be ignored when computing the loss. If you look at NumPy or PyTorch loss functions that support label masking, you will often see code like this that uses [boolean indexing](https://numpy.org/doc/stable/user/basics.indexing.html#boolean-array-indexing):\n\n```python\nlabel_mask = labels >= 0\nmasked_outputs = outputs[label_mask]\nmasked_labels = labels[label_mask]\nloss = compute_loss(masked_outputs, masked_labels)\nmean_loss = torch.mean(loss)\n```\n\nThis code is totally fine in NumPy or PyTorch, but it breaks in XLA! Why? Because the shape of `masked_outputs` and `masked_labels` depends on how many positions are masked - that makes it a **data-dependent shape.** However, just like for rule #1, we can often rewrite this code to yield exactly the same output without any data-dependent shapes.\n\n```python\nlabel_mask = tf.cast(labels >= 0, tf.float32)\nloss = compute_loss(outputs, labels)\nloss = loss * label_mask  # Set negative label positions to 0\nmean_loss = tf.reduce_sum(loss) / tf.reduce_sum(label_mask)\n```\n\nHere, we avoid data-dependent shapes by computing the loss for every position, but zeroing out the masked positions in both the numerator and denominator when we calculate the mean, which yields exactly the same result as the first block while maintaining XLA compatibility. Note that we use the same trick as in rule #1 - converting a `tf.bool` to `tf.float32` and using it as an indicator variable. This is a really useful trick, so remember it if you need to convert your own code to XLA!\n\n#### XLA Rule #3: XLA will need to recompile your model for every different input shape it sees\n\nThis is the big one."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 38f6cb4d-f4d6-42b2-97d1-67e4ab57c7a5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf/", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"38f6cb4d-f4d6-42b2-97d1-67e4ab57c7a5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_tpu_tf/\", \"title\": \"Training on TPU with TensorFlow\", \"tokens\": 3205, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ea1b958a-2bac-447b-9416-fa8b4a2ff755\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_tpu_tf/\", \"title\": \"Training on TPU with TensorFlow\", \"tokens\": 3205, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b74dfc96b97a641db4d95da420a47e4c3448d0eec08ce939ced40c51264461d5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9ea3724e-8404-4cd5-bc0e-604ebbb53fa6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_tpu_tf/\", \"title\": \"Training on TPU with TensorFlow\", \"tokens\": 3205, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"90df39d07433e274e49a5599e0b2023fca069a714b22960626a3368835a3bb39\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ed9cb157-3176-47f0-98af-115e7235791c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bceda79f31cca7fc833bfbee6c3cd792c41f7c236242780e746b2fac36212354\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8012, \"end_char_idx\": 11285, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ea1b958a-2bac-447b-9416-fa8b4a2ff755", "doc_id": "ea1b958a-2bac-447b-9416-fa8b4a2ff755", "ref_doc_id": "ea1b958a-2bac-447b-9416-fa8b4a2ff755", "chroma:document": "If you look at NumPy or PyTorch loss functions that support label masking, you will often see code like this that uses [boolean indexing](https://numpy.org/doc/stable/user/basics.indexing.html#boolean-array-indexing):\n\n```python\nlabel_mask = labels >= 0\nmasked_outputs = outputs[label_mask]\nmasked_labels = labels[label_mask]\nloss = compute_loss(masked_outputs, masked_labels)\nmean_loss = torch.mean(loss)\n```\n\nThis code is totally fine in NumPy or PyTorch, but it breaks in XLA! Why? Because the shape of `masked_outputs` and `masked_labels` depends on how many positions are masked - that makes it a **data-dependent shape.** However, just like for rule #1, we can often rewrite this code to yield exactly the same output without any data-dependent shapes.\n\n```python\nlabel_mask = tf.cast(labels >= 0, tf.float32)\nloss = compute_loss(outputs, labels)\nloss = loss * label_mask  # Set negative label positions to 0\nmean_loss = tf.reduce_sum(loss) / tf.reduce_sum(label_mask)\n```\n\nHere, we avoid data-dependent shapes by computing the loss for every position, but zeroing out the masked positions in both the numerator and denominator when we calculate the mean, which yields exactly the same result as the first block while maintaining XLA compatibility. Note that we use the same trick as in rule #1 - converting a `tf.bool` to `tf.float32` and using it as an indicator variable. This is a really useful trick, so remember it if you need to convert your own code to XLA!\n\n#### XLA Rule #3: XLA will need to recompile your model for every different input shape it sees\n\nThis is the big one. What this means is that if your input shapes are very variable, XLA will have to recompile your model over and over, which will create huge performance problems. This commonly arises in NLP models, where input texts have variable lengths after tokenization. In other modalities, static shapes are more common and this rule is much less of a problem.\n\nHow can you get around rule #3? The key is **padding** - if you pad all your inputs to the same length, and then use an `attention_mask`, you can get the same results as you\u2019d get from variable shapes, but without any XLA issues. However, excessive padding can cause severe slowdown too - if you pad all your samples to the maximum length in the whole dataset, you might end up with batches consisting endless padding tokens, which will waste a lot of compute and memory!\n\nThere isn\u2019t a perfect solution to this problem. However, you can try some tricks. One very useful trick is to **pad batches of samples up to a multiple of a number like 32 or 64 tokens.** This often only increases the number of tokens by a small amount, but it hugely reduces the number of unique input shapes, because every input shape now has to be a multiple of 32 or 64. Fewer unique input shapes means fewer XLA compilations!\n\n<Tip>\n\n**\ud83e\udd17Specific HuggingFace Tip\ud83e\udd17:** Our tokenizers and data collators have methods that can help you here. You can use `padding=\"max_length\"` or `padding=\"longest\"` when calling tokenizers to get them to output padded data. Our tokenizers and data collators also have a `pad_to_multiple_of` argument that you can use to reduce the number of unique input shapes you see!\n\n</Tip>\n\n### How do I actually train my model on TPU?"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: ed9cb157-3176-47f0-98af-115e7235791c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf/", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"ed9cb157-3176-47f0-98af-115e7235791c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_tpu_tf/\", \"title\": \"Training on TPU with TensorFlow\", \"tokens\": 3205, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ea1b958a-2bac-447b-9416-fa8b4a2ff755\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_tpu_tf/\", \"title\": \"Training on TPU with TensorFlow\", \"tokens\": 3205, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b74dfc96b97a641db4d95da420a47e4c3448d0eec08ce939ced40c51264461d5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"38f6cb4d-f4d6-42b2-97d1-67e4ab57c7a5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_tpu_tf/\", \"title\": \"Training on TPU with TensorFlow\", \"tokens\": 3205, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"cea4f346c438ead48ef6b705ce5de2de3fed5877667ed2b32b01af457765c2b5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b6179de9-4d54-4970-973c-405a6ea83f05\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f8eba4a6ba0f7f20c573ecc5e00d4e1fc61d30b1e627ac6a81006893a89e2db5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9603, \"end_char_idx\": 12855, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ea1b958a-2bac-447b-9416-fa8b4a2ff755", "doc_id": "ea1b958a-2bac-447b-9416-fa8b4a2ff755", "ref_doc_id": "ea1b958a-2bac-447b-9416-fa8b4a2ff755", "chroma:document": "What this means is that if your input shapes are very variable, XLA will have to recompile your model over and over, which will create huge performance problems. This commonly arises in NLP models, where input texts have variable lengths after tokenization. In other modalities, static shapes are more common and this rule is much less of a problem.\n\nHow can you get around rule #3? The key is **padding** - if you pad all your inputs to the same length, and then use an `attention_mask`, you can get the same results as you\u2019d get from variable shapes, but without any XLA issues. However, excessive padding can cause severe slowdown too - if you pad all your samples to the maximum length in the whole dataset, you might end up with batches consisting endless padding tokens, which will waste a lot of compute and memory!\n\nThere isn\u2019t a perfect solution to this problem. However, you can try some tricks. One very useful trick is to **pad batches of samples up to a multiple of a number like 32 or 64 tokens.** This often only increases the number of tokens by a small amount, but it hugely reduces the number of unique input shapes, because every input shape now has to be a multiple of 32 or 64. Fewer unique input shapes means fewer XLA compilations!\n\n<Tip>\n\n**\ud83e\udd17Specific HuggingFace Tip\ud83e\udd17:** Our tokenizers and data collators have methods that can help you here. You can use `padding=\"max_length\"` or `padding=\"longest\"` when calling tokenizers to get them to output padded data. Our tokenizers and data collators also have a `pad_to_multiple_of` argument that you can use to reduce the number of unique input shapes you see!\n\n</Tip>\n\n### How do I actually train my model on TPU?\n\nOnce your training is XLA-compatible and (if you\u2019re using TPU Node / Colab) your dataset has been prepared appropriately, running on TPU is surprisingly easy! All you really need to change in your code is to add a few lines to initialize your TPU, and to ensure that your model and dataset are created inside a `TPUStrategy` scope. Take a look at [our TPU example notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/tpu_training-tf.ipynb) to see this in action!\n\n### Summary\n\nThere was a lot in here, so let\u2019s summarize with a quick checklist you can follow when you want to get your model ready for TPU training:\n\n- Make sure your code follows the three rules of XLA\n- Compile your model with `jit_compile=True` on CPU/GPU and confirm that you can train it with XLA\n- Either load your dataset into memory or use a TPU-compatible dataset loading approach (see [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/tpu_training-tf.ipynb))\n- Migrate your code either to Colab (with accelerator set to \u201cTPU\u201d) or a TPU VM on Google Cloud\n- Add TPU initializer code (see [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/tpu_training-tf.ipynb))\n- Create your `TPUStrategy` and make sure dataset loading and model creation are inside the `strategy.scope()` (see [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/tpu_training-tf.ipynb))\n- Don\u2019t forget to take `jit_compile=True` out again when you move to TPU!"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: b6179de9-4d54-4970-973c-405a6ea83f05
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf/", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"b6179de9-4d54-4970-973c-405a6ea83f05\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_tpu_tf/\", \"title\": \"Training on TPU with TensorFlow\", \"tokens\": 3205, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ea1b958a-2bac-447b-9416-fa8b4a2ff755\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_tpu_tf/\", \"title\": \"Training on TPU with TensorFlow\", \"tokens\": 3205, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b74dfc96b97a641db4d95da420a47e4c3448d0eec08ce939ced40c51264461d5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ed9cb157-3176-47f0-98af-115e7235791c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/perf_train_tpu_tf/\", \"title\": \"Training on TPU with TensorFlow\", \"tokens\": 3205, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"25cd77b7f8ff545b14802fa6f7ef702821338744c78a817cd2794eceaeef6473\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11287, \"end_char_idx\": 12897, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ea1b958a-2bac-447b-9416-fa8b4a2ff755", "doc_id": "ea1b958a-2bac-447b-9416-fa8b4a2ff755", "ref_doc_id": "ea1b958a-2bac-447b-9416-fa8b4a2ff755", "chroma:document": "Once your training is XLA-compatible and (if you\u2019re using TPU Node / Colab) your dataset has been prepared appropriately, running on TPU is surprisingly easy! All you really need to change in your code is to add a few lines to initialize your TPU, and to ensure that your model and dataset are created inside a `TPUStrategy` scope. Take a look at [our TPU example notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/tpu_training-tf.ipynb) to see this in action!\n\n### Summary\n\nThere was a lot in here, so let\u2019s summarize with a quick checklist you can follow when you want to get your model ready for TPU training:\n\n- Make sure your code follows the three rules of XLA\n- Compile your model with `jit_compile=True` on CPU/GPU and confirm that you can train it with XLA\n- Either load your dataset into memory or use a TPU-compatible dataset loading approach (see [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/tpu_training-tf.ipynb))\n- Migrate your code either to Colab (with accelerator set to \u201cTPU\u201d) or a TPU VM on Google Cloud\n- Add TPU initializer code (see [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/tpu_training-tf.ipynb))\n- Create your `TPUStrategy` and make sure dataset loading and model creation are inside the `strategy.scope()` (see [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/tpu_training-tf.ipynb))\n- Don\u2019t forget to take `jit_compile=True` out again when you move to TPU!\n- \ud83d\ude4f\ud83d\ude4f\ud83d\ude4f\ud83e\udd7a\ud83e\udd7a\ud83e\udd7a\n- Call model.fit()\n- You did it!"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: c0a798ad-d97d-4727-8bee-64cf906488f5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"c0a798ad-d97d-4727-8bee-64cf906488f5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"69f0817e-23c0-4bca-846c-5ace9895af0a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"3df8fa8f9e9d30020137186799a1a245301f837f2e21ea3015ffe1543676fe0f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"19cacafc-b25c-4d5b-9268-b920de77d67d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2f4bcf3a09487322d635c84899dffdb6348017b7807ed1898cc5f0d8429de56f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 4461, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "doc_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "ref_doc_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "chroma:document": "# Quick tour\n\n[[open-in-colab]]\n\nGet up and running with \ud83e\udd17 Transformers! Whether you're a developer or an everyday user, this quick tour will help you get started and show you how to use the [`pipeline`] for inference, load a pretrained model and preprocessor with an [AutoClass](./model_doc/auto), and quickly train a model with PyTorch or TensorFlow. If you're a beginner, we recommend checking out our tutorials or [course](https://huggingface.co/course/chapter1/1) next for more in-depth explanations of the concepts introduced here.\n\nBefore you begin, make sure you have all the necessary libraries installed:\n\n```bash\n!pip install transformers datasets evaluate accelerate\n```\n\nYou'll also need to install your preferred machine learning framework:\n\n<frameworkcontent>\n<pt>\n\n```bash\npip install torch\n```\n</pt>\n<tf>\n\n```bash\npip install tensorflow\n```\n</tf>\n</frameworkcontent>\n\n## Pipeline\n\n<Youtube id=\"tiZFewofSLM\"/>\n\nThe [`pipeline`] is the easiest and fastest way to use a pretrained model for inference. You can use the [`pipeline`] out-of-the-box for many tasks across different modalities, some of which are shown in the table below:\n\n<Tip>\n\nFor a complete list of available tasks, check out the [pipeline API reference](./main_classes/pipelines).\n\n</Tip>\n\n| **Task**                     | **Description**                                                                                              | **Modality**    | **Pipeline identifier**                       |\n|------------------------------|--------------------------------------------------------------------------------------------------------------|-----------------|-----------------------------------------------|\n| Text classification          | assign a label to a given sequence of text                                                                   | NLP             | pipeline(task=\u201csentiment-analysis\u201d)           |\n| Text generation              | generate text given a prompt                                                                                 | NLP             | pipeline(task=\u201ctext-generation\u201d)              |\n| Summarization                | generate a summary of a sequence of text or document                                                         | NLP             | pipeline(task=\u201csummarization\u201d)                |\n| Image classification         | assign a label to an image                                                                                   | Computer vision | pipeline(task=\u201cimage-classification\u201d)         |\n| Image segmentation           | assign a label to each individual pixel of an image (supports semantic, panoptic, and instance segmentation) | Computer vision | pipeline(task=\u201cimage-segmentation\u201d)           |\n| Object detection             | predict the bounding boxes and classes of objects in an image                                                | Computer vision | pipeline(task=\u201cobject-detection\u201d)             |\n| Audio classification         | assign a label to some audio data                                                                            | Audio           | pipeline(task=\u201caudio-classification\u201d)         |\n| Automatic speech recognition | transcribe speech into text                                                                                  | Audio           | pipeline(task=\u201cautomatic-speech-recognition\u201d) |\n| Visual question answering    | answer a question about the image, given an image and a question                                             | Multimodal      | pipeline(task=\u201cvqa\u201d)                          |\n| Document question answering  | answer a question about the document, given a document and a question                                        | Multimodal      | pipeline(task=\"document-question-answering\")  |\n| Image captioning             | generate a caption for a given image                                                                         | Multimodal      | pipeline(task=\"image-to-text\")                |\n\nStart by creating an instance of [`pipeline`] and specifying a task you want to use it for. In this guide, you'll use the [`pipeline`] for sentiment analysis as an example:\n\n```py\n>>> from transformers import pipeline\n\n>>> classifier = pipeline(\"sentiment-analysis\")\n```\n\nThe [`pipeline`] downloads and caches a default [pretrained model](https://huggingface.co/distilbert/distilbert-base-uncased-finetuned-sst-2-english) and tokenizer for sentiment analysis."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 19cacafc-b25c-4d5b-9268-b920de77d67d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"19cacafc-b25c-4d5b-9268-b920de77d67d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"69f0817e-23c0-4bca-846c-5ace9895af0a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"3df8fa8f9e9d30020137186799a1a245301f837f2e21ea3015ffe1543676fe0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c0a798ad-d97d-4727-8bee-64cf906488f5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"cd8be0b66c53cc8930b55d924306482f9592a86448d971cde3a7f3e9151a3cbb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a3ba50bf-f05f-4156-964a-8e133d5af31a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"32067a2c374b2adeb814b4ffbebeffbf7c02b7ab29a8600dafdd1ceee5d13045\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4094, \"end_char_idx\": 6199, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "doc_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "ref_doc_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "chroma:document": "In this guide, you'll use the [`pipeline`] for sentiment analysis as an example:\n\n```py\n>>> from transformers import pipeline\n\n>>> classifier = pipeline(\"sentiment-analysis\")\n```\n\nThe [`pipeline`] downloads and caches a default [pretrained model](https://huggingface.co/distilbert/distilbert-base-uncased-finetuned-sst-2-english) and tokenizer for sentiment analysis. Now you can use the `classifier` on your target text:\n\n```py\n>>> classifier(\"We are very happy to show you the \ud83e\udd17 Transformers library.\")\n[{'label': 'POSITIVE', 'score': 0.9998}]\n```\n\nIf you have more than one input, pass your inputs as a list to the [`pipeline`] to return a list of dictionaries:\n\n```py\n>>> results = classifier([\"We are very happy to show you the \ud83e\udd17 Transformers library.\", \"We hope you don't hate it.\"])\n>>> for result in results:\n...     print(f\"label: {result['label']}, with score: {round(result['score'], 4)}\")\nlabel: POSITIVE, with score: 0.9998\nlabel: NEGATIVE, with score: 0.5309\n```\n\nThe [`pipeline`] can also iterate over an entire dataset for any task you like. For this example, let's choose automatic speech recognition as our task:\n\n```py\n>>> import torch\n>>> from transformers import pipeline\n\n>>> speech_recognizer = pipeline(\"automatic-speech-recognition\", model=\"facebook/wav2vec2-base-960h\")\n```\n\nLoad an audio dataset (see the \ud83e\udd17 Datasets [Quick Start](https://huggingface.co/docs/datasets/quickstart#audio) for more details) you'd like to iterate over. For example, load the [MInDS-14](https://huggingface.co/datasets/PolyAI/minds14) dataset:\n\n```py\n>>> from datasets import load_dataset, Audio\n\n>>> dataset = load_dataset(\"PolyAI/minds14\", name=\"en-US\", split=\"train\")  # doctest: +IGNORE_RESULT\n```\n\nYou need to make sure the sampling rate of the dataset matches the sampling \nrate [`facebook/wav2vec2-base-960h`](https://huggingface.co/facebook/wav2vec2-base-960h) was trained on:\n\n```py\n>>> dataset = dataset.cast_column(\"audio\", Audio(sampling_rate=speech_recognizer.feature_extractor.sampling_rate))\n```\n\nThe audio files are automatically loaded and resampled when calling the `\"audio\"` column."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: a3ba50bf-f05f-4156-964a-8e133d5af31a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"a3ba50bf-f05f-4156-964a-8e133d5af31a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"69f0817e-23c0-4bca-846c-5ace9895af0a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"3df8fa8f9e9d30020137186799a1a245301f837f2e21ea3015ffe1543676fe0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"19cacafc-b25c-4d5b-9268-b920de77d67d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"bd177c37a7dd2f1057a114729cda3c76012e491bbc46ceb6053ed0e4bc4248bc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"25ba36cf-99b8-41c1-9ef5-348a4fe9cffd\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"edf7908e6722d5c497727af26f915dbb5a8f1176cf8766e11f4db6119f4a7f50\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4850, \"end_char_idx\": 7514, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "doc_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "ref_doc_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "chroma:document": "\", \"We hope you don't hate it.\"])\n>>> for result in results:\n...     print(f\"label: {result['label']}, with score: {round(result['score'], 4)}\")\nlabel: POSITIVE, with score: 0.9998\nlabel: NEGATIVE, with score: 0.5309\n```\n\nThe [`pipeline`] can also iterate over an entire dataset for any task you like. For this example, let's choose automatic speech recognition as our task:\n\n```py\n>>> import torch\n>>> from transformers import pipeline\n\n>>> speech_recognizer = pipeline(\"automatic-speech-recognition\", model=\"facebook/wav2vec2-base-960h\")\n```\n\nLoad an audio dataset (see the \ud83e\udd17 Datasets [Quick Start](https://huggingface.co/docs/datasets/quickstart#audio) for more details) you'd like to iterate over. For example, load the [MInDS-14](https://huggingface.co/datasets/PolyAI/minds14) dataset:\n\n```py\n>>> from datasets import load_dataset, Audio\n\n>>> dataset = load_dataset(\"PolyAI/minds14\", name=\"en-US\", split=\"train\")  # doctest: +IGNORE_RESULT\n```\n\nYou need to make sure the sampling rate of the dataset matches the sampling \nrate [`facebook/wav2vec2-base-960h`](https://huggingface.co/facebook/wav2vec2-base-960h) was trained on:\n\n```py\n>>> dataset = dataset.cast_column(\"audio\", Audio(sampling_rate=speech_recognizer.feature_extractor.sampling_rate))\n```\n\nThe audio files are automatically loaded and resampled when calling the `\"audio\"` column.\nExtract the raw waveform arrays from the first 4 samples and pass it as a list to the pipeline:\n\n```py\n>>> result = speech_recognizer(dataset[:4][\"audio\"])\n>>> print([d[\"text\"] for d in result])\n['I WOULD LIKE TO SET UP A JOINT ACCOUNT WITH MY PARTNER HOW DO I PROCEED WITH DOING THAT', \"FONDERING HOW I'D SET UP A JOIN TO HELL T WITH MY WIFE AND WHERE THE AP MIGHT BE\", \"I I'D LIKE TOY SET UP A JOINT ACCOUNT WITH MY PARTNER I'M NOT SEEING THE OPTION TO DO IT ON THE APSO I CALLED IN TO GET SOME HELP CAN I JUST DO IT OVER THE PHONE WITH YOU AND GIVE YOU THE INFORMATION OR SHOULD I DO IT IN THE AP AN I'M MISSING SOMETHING UQUETTE HAD PREFERRED TO JUST DO IT OVER THE PHONE OF POSSIBLE THINGS\", 'HOW DO I FURN A JOINA COUT']\n```\n\nFor larger datasets where the inputs are big (like in speech or vision), you'll want to pass a generator instead of a list to load all the inputs in memory. Take a look at the [pipeline API reference](./main_classes/pipelines) for more information.\n\n### Use another model and tokenizer in the pipeline\n\nThe [`pipeline`] can accommodate any model from the [Hub](https://huggingface.co/models), making it easy to adapt the [`pipeline`] for other use-cases. For example, if you'd like a model capable of handling French text, use the tags on the Hub to filter for an appropriate model."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 25ba36cf-99b8-41c1-9ef5-348a4fe9cffd
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"25ba36cf-99b8-41c1-9ef5-348a4fe9cffd\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"69f0817e-23c0-4bca-846c-5ace9895af0a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"3df8fa8f9e9d30020137186799a1a245301f837f2e21ea3015ffe1543676fe0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a3ba50bf-f05f-4156-964a-8e133d5af31a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"7e645fd1a6c69de3e1ca9e1169b918abb9ca8a6411728282534304b12d9cd207\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3c3de771-63e3-40a0-ab42-d4cfc3ad25de\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6d2f53bb2cf2f564603ea8e51ba990c158529239531bd87c96707ac5af69e07c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6200, \"end_char_idx\": 9335, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "doc_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "ref_doc_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "chroma:document": "Extract the raw waveform arrays from the first 4 samples and pass it as a list to the pipeline:\n\n```py\n>>> result = speech_recognizer(dataset[:4][\"audio\"])\n>>> print([d[\"text\"] for d in result])\n['I WOULD LIKE TO SET UP A JOINT ACCOUNT WITH MY PARTNER HOW DO I PROCEED WITH DOING THAT', \"FONDERING HOW I'D SET UP A JOIN TO HELL T WITH MY WIFE AND WHERE THE AP MIGHT BE\", \"I I'D LIKE TOY SET UP A JOINT ACCOUNT WITH MY PARTNER I'M NOT SEEING THE OPTION TO DO IT ON THE APSO I CALLED IN TO GET SOME HELP CAN I JUST DO IT OVER THE PHONE WITH YOU AND GIVE YOU THE INFORMATION OR SHOULD I DO IT IN THE AP AN I'M MISSING SOMETHING UQUETTE HAD PREFERRED TO JUST DO IT OVER THE PHONE OF POSSIBLE THINGS\", 'HOW DO I FURN A JOINA COUT']\n```\n\nFor larger datasets where the inputs are big (like in speech or vision), you'll want to pass a generator instead of a list to load all the inputs in memory. Take a look at the [pipeline API reference](./main_classes/pipelines) for more information.\n\n### Use another model and tokenizer in the pipeline\n\nThe [`pipeline`] can accommodate any model from the [Hub](https://huggingface.co/models), making it easy to adapt the [`pipeline`] for other use-cases. For example, if you'd like a model capable of handling French text, use the tags on the Hub to filter for an appropriate model. The top filtered result returns a multilingual [BERT model](https://huggingface.co/nlptown/bert-base-multilingual-uncased-sentiment) finetuned for sentiment analysis you can use for French text:\n\n```py\n>>> model_name = \"nlptown/bert-base-multilingual-uncased-sentiment\"\n```\n\n<frameworkcontent>\n<pt>\nUse [`AutoModelForSequenceClassification`] and [`AutoTokenizer`] to load the pretrained model and it's associated tokenizer (more on an `AutoClass` in the next section):\n\n```py\n>>> from transformers import AutoTokenizer, AutoModelForSequenceClassification\n\n>>> model = AutoModelForSequenceClassification.from_pretrained(model_name)\n>>> tokenizer = AutoTokenizer.from_pretrained(model_name)\n```\n</pt>\n<tf>\nUse [`TFAutoModelForSequenceClassification`] and [`AutoTokenizer`] to load the pretrained model and it's associated tokenizer (more on an `TFAutoClass` in the next section):\n\n```py\n>>> from transformers import AutoTokenizer, TFAutoModelForSequenceClassification\n\n>>> model = TFAutoModelForSequenceClassification.from_pretrained(model_name)\n>>> tokenizer = AutoTokenizer.from_pretrained(model_name)\n```\n</tf>\n</frameworkcontent>\n\nSpecify the model and tokenizer in the [`pipeline`], and now you can apply the `classifier` on French text:\n\n```py\n>>> classifier = pipeline(\"sentiment-analysis\", model=model, tokenizer=tokenizer)\n>>> classifier(\"Nous sommes tr\u00e8s heureux de vous pr\u00e9senter la biblioth\u00e8que \ud83e\udd17 Transformers.\")\n[{'label': '5 stars', 'score': 0.7273}]\n```\n\nIf you can't find a model for your use-case, you'll need to finetune a pretrained model on your data. Take a look at our [finetuning tutorial](./training) to learn how. Finally, after you've finetuned your pretrained model, please consider [sharing](./model_sharing) the model with the community on the Hub to democratize machine learning for everyone!"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 3c3de771-63e3-40a0-ab42-d4cfc3ad25de
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"3c3de771-63e3-40a0-ab42-d4cfc3ad25de\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"69f0817e-23c0-4bca-846c-5ace9895af0a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"3df8fa8f9e9d30020137186799a1a245301f837f2e21ea3015ffe1543676fe0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"25ba36cf-99b8-41c1-9ef5-348a4fe9cffd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"336b20c14803a0331e5df2436f4a51455cda01f666fa1b4a6fb0081508af5b10\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"03fe8ad3-05f4-4206-a2c0-8f526e808a74\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"72d817d147d8d680bdfe0ddc949cc311d091d9696f1860abdac2c976400ec12a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8938, \"end_char_idx\": 11779, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "doc_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "ref_doc_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "chroma:document": "[{'label': '5 stars', 'score': 0.7273}]\n```\n\nIf you can't find a model for your use-case, you'll need to finetune a pretrained model on your data. Take a look at our [finetuning tutorial](./training) to learn how. Finally, after you've finetuned your pretrained model, please consider [sharing](./model_sharing) the model with the community on the Hub to democratize machine learning for everyone! \ud83e\udd17\n\n## AutoClass\n\n<Youtube id=\"AhChOFRegn4\"/>\n\nUnder the hood, the [`AutoModelForSequenceClassification`] and [`AutoTokenizer`] classes work together to power the [`pipeline`] you used above. An [AutoClass](./model_doc/auto) is a shortcut that automatically retrieves the architecture of a pretrained model from its name or path. You only need to select the appropriate `AutoClass` for your task and it's associated preprocessing class. \n\nLet's return to the example from the previous section and see how you can use the `AutoClass` to replicate the results of the [`pipeline`].\n\n### AutoTokenizer\n\nA tokenizer is responsible for preprocessing text into an array of numbers as inputs to a model. There are multiple rules that govern the tokenization process, including how to split a word and at what level words should be split (learn more about tokenization in the [tokenizer summary](./tokenizer_summary)). The most important thing to remember is you need to instantiate a tokenizer with the same model name to ensure you're using the same tokenization rules a model was pretrained with.\n\nLoad a tokenizer with [`AutoTokenizer`]:\n\n```py\n>>> from transformers import AutoTokenizer\n\n>>> model_name = \"nlptown/bert-base-multilingual-uncased-sentiment\"\n>>> tokenizer = AutoTokenizer.from_pretrained(model_name)\n```\n\nPass your text to the tokenizer:\n\n```py\n>>> encoding = tokenizer(\"We are very happy to show you the \ud83e\udd17 Transformers library.\")\n>>> print(encoding)\n{'input_ids': [101, 11312, 10320, 12495, 19308, 10114, 11391, 10855, 10103, 100, 58263, 13299, 119, 102],\n 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}\n```\n\nThe tokenizer returns a dictionary containing:\n\n* [input_ids](./glossary#input-ids): numerical representations of your tokens.\n* [attention_mask](./glossary#attention-mask): indicates which tokens should be attended to.\n\nA tokenizer can also accept a list of inputs, and pad and truncate the text to return a batch with uniform length:\n\n<frameworkcontent>\n<pt>\n\n```py\n>>> pt_batch = tokenizer(\n...     [\"We are very happy to show you the \ud83e\udd17 Transformers library.\", \"We hope you don't hate it.\"],\n...     padding=True,\n...     truncation=True,\n...     max_length=512,\n...     return_tensors=\"pt\",\n... )\n```\n</pt>\n<tf>\n\n```py\n>>> tf_batch = tokenizer(\n...     [\"We are very happy to show you the \ud83e\udd17 Transformers library.\", \"We hope you don't hate it."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 03fe8ad3-05f4-4206-a2c0-8f526e808a74
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"03fe8ad3-05f4-4206-a2c0-8f526e808a74\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"69f0817e-23c0-4bca-846c-5ace9895af0a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"3df8fa8f9e9d30020137186799a1a245301f837f2e21ea3015ffe1543676fe0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3c3de771-63e3-40a0-ab42-d4cfc3ad25de\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"e9b0118e2e0adcb3df1d073ab704429b508c0b9e274ee9bbde921612ac3818bb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2f0cd4e4-c3bc-465c-86c7-f6c2b6263ada\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4bda205eb754f0a8adfb712a013aba8e32a115c1437d23bfd32022cf56b9aed6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10776, \"end_char_idx\": 13097, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "doc_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "ref_doc_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "chroma:document": ">>> print(encoding)\n{'input_ids': [101, 11312, 10320, 12495, 19308, 10114, 11391, 10855, 10103, 100, 58263, 13299, 119, 102],\n 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}\n```\n\nThe tokenizer returns a dictionary containing:\n\n* [input_ids](./glossary#input-ids): numerical representations of your tokens.\n* [attention_mask](./glossary#attention-mask): indicates which tokens should be attended to.\n\nA tokenizer can also accept a list of inputs, and pad and truncate the text to return a batch with uniform length:\n\n<frameworkcontent>\n<pt>\n\n```py\n>>> pt_batch = tokenizer(\n...     [\"We are very happy to show you the \ud83e\udd17 Transformers library.\", \"We hope you don't hate it.\"],\n...     padding=True,\n...     truncation=True,\n...     max_length=512,\n...     return_tensors=\"pt\",\n... )\n```\n</pt>\n<tf>\n\n```py\n>>> tf_batch = tokenizer(\n...     [\"We are very happy to show you the \ud83e\udd17 Transformers library.\", \"We hope you don't hate it.\"],\n...     padding=True,\n...     truncation=True,\n...     max_length=512,\n...     return_tensors=\"tf\",\n... )\n```\n</tf>\n</frameworkcontent>\n\n<Tip>\n\nCheck out the [preprocess](./preprocessing) tutorial for more details about tokenization, and how to use an [`AutoImageProcessor`], [`AutoFeatureExtractor`] and [`AutoProcessor`] to preprocess image, audio, and multimodal inputs.\n\n</Tip>\n\n### AutoModel\n\n<frameworkcontent>\n<pt>\n\ud83e\udd17 Transformers provides a simple and unified way to load pretrained instances. This means you can load an [`AutoModel`] like you would load an [`AutoTokenizer`]. The only difference is selecting the correct [`AutoModel`] for the task. For text (or sequence) classification, you should load [`AutoModelForSequenceClassification`]:\n\n```py\n>>> from transformers import AutoModelForSequenceClassification\n\n>>> model_name = \"nlptown/bert-base-multilingual-uncased-sentiment\"\n>>> pt_model = AutoModelForSequenceClassification.from_pretrained(model_name)\n```\n\n<Tip>\n\nSee the [task summary](./task_summary) for tasks supported by an [`AutoModel`] class.\n\n</Tip>\n\nNow pass your preprocessed batch of inputs directly to the model. You just have to unpack the dictionary by adding `**`:\n\n```py\n>>> pt_outputs = pt_model(**pt_batch)\n```\n\nThe model outputs the final activations in the `logits` attribute."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 2f0cd4e4-c3bc-465c-86c7-f6c2b6263ada
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"2f0cd4e4-c3bc-465c-86c7-f6c2b6263ada\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"69f0817e-23c0-4bca-846c-5ace9895af0a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"3df8fa8f9e9d30020137186799a1a245301f837f2e21ea3015ffe1543676fe0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"03fe8ad3-05f4-4206-a2c0-8f526e808a74\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"3513a1ce50218dd02356ea83a236c6b0fb65a1230620db3237c1c66f73d64184\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"22e1d760-de46-41cb-8059-7aea43ba9829\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fd600be74cf70414fd62fd10e349dd281905975cbed45424c415d31b56104e78\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11494, \"end_char_idx\": 14322, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "doc_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "ref_doc_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "chroma:document": "\", \"We hope you don't hate it.\"],\n...     padding=True,\n...     truncation=True,\n...     max_length=512,\n...     return_tensors=\"pt\",\n... )\n```\n</pt>\n<tf>\n\n```py\n>>> tf_batch = tokenizer(\n...     [\"We are very happy to show you the \ud83e\udd17 Transformers library.\", \"We hope you don't hate it.\"],\n...     padding=True,\n...     truncation=True,\n...     max_length=512,\n...     return_tensors=\"tf\",\n... )\n```\n</tf>\n</frameworkcontent>\n\n<Tip>\n\nCheck out the [preprocess](./preprocessing) tutorial for more details about tokenization, and how to use an [`AutoImageProcessor`], [`AutoFeatureExtractor`] and [`AutoProcessor`] to preprocess image, audio, and multimodal inputs.\n\n</Tip>\n\n### AutoModel\n\n<frameworkcontent>\n<pt>\n\ud83e\udd17 Transformers provides a simple and unified way to load pretrained instances. This means you can load an [`AutoModel`] like you would load an [`AutoTokenizer`]. The only difference is selecting the correct [`AutoModel`] for the task. For text (or sequence) classification, you should load [`AutoModelForSequenceClassification`]:\n\n```py\n>>> from transformers import AutoModelForSequenceClassification\n\n>>> model_name = \"nlptown/bert-base-multilingual-uncased-sentiment\"\n>>> pt_model = AutoModelForSequenceClassification.from_pretrained(model_name)\n```\n\n<Tip>\n\nSee the [task summary](./task_summary) for tasks supported by an [`AutoModel`] class.\n\n</Tip>\n\nNow pass your preprocessed batch of inputs directly to the model. You just have to unpack the dictionary by adding `**`:\n\n```py\n>>> pt_outputs = pt_model(**pt_batch)\n```\n\nThe model outputs the final activations in the `logits` attribute. Apply the softmax function to the `logits` to retrieve the probabilities:\n\n```py\n>>> from torch import nn\n\n>>> pt_predictions = nn.functional.softmax(pt_outputs.logits, dim=-1)\n>>> print(pt_predictions)\ntensor([[0.0021, 0.0018, 0.0115, 0.2121, 0.7725],\n        [0.2084, 0.1826, 0.1969, 0.1755, 0.2365]], grad_fn=<SoftmaxBackward0>)\n```\n</pt>\n<tf>\n\ud83e\udd17 Transformers provides a simple and unified way to load pretrained instances. This means you can load an [`TFAutoModel`] like you would load an [`AutoTokenizer`]. The only difference is selecting the correct [`TFAutoModel`] for the task. For text (or sequence) classification, you should load [`TFAutoModelForSequenceClassification`]:\n\n```py\n>>> from transformers import TFAutoModelForSequenceClassification\n\n>>> model_name = \"nlptown/bert-base-multilingual-uncased-sentiment\"\n>>> tf_model = TFAutoModelForSequenceClassification.from_pretrained(model_name)\n```\n\n<Tip>\n\nSee the [task summary](./task_summary) for tasks supported by an [`AutoModel`] class.\n\n</Tip>\n\nNow pass your preprocessed batch of inputs directly to the model. You can pass the tensors as-is:\n\n```py\n>>> tf_outputs = tf_model(tf_batch)\n```\n\nThe model outputs the final activations in the `logits` attribute."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 22e1d760-de46-41cb-8059-7aea43ba9829
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"22e1d760-de46-41cb-8059-7aea43ba9829\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"69f0817e-23c0-4bca-846c-5ace9895af0a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"3df8fa8f9e9d30020137186799a1a245301f837f2e21ea3015ffe1543676fe0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2f0cd4e4-c3bc-465c-86c7-f6c2b6263ada\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a91be36d14f078ccb78b9495b2704a80321535abc03244a5f2dc4db906a19a16\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"45d23c0d-c060-4633-b0d4-9b63837d6165\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"994a8fa4f2c43b35b4b170b5cfd214a12b0b64a18abe435becdaea554f2c7cc5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12852, \"end_char_idx\": 15017, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "doc_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "ref_doc_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "chroma:document": "</Tip>\n\nNow pass your preprocessed batch of inputs directly to the model. You just have to unpack the dictionary by adding `**`:\n\n```py\n>>> pt_outputs = pt_model(**pt_batch)\n```\n\nThe model outputs the final activations in the `logits` attribute. Apply the softmax function to the `logits` to retrieve the probabilities:\n\n```py\n>>> from torch import nn\n\n>>> pt_predictions = nn.functional.softmax(pt_outputs.logits, dim=-1)\n>>> print(pt_predictions)\ntensor([[0.0021, 0.0018, 0.0115, 0.2121, 0.7725],\n        [0.2084, 0.1826, 0.1969, 0.1755, 0.2365]], grad_fn=<SoftmaxBackward0>)\n```\n</pt>\n<tf>\n\ud83e\udd17 Transformers provides a simple and unified way to load pretrained instances. This means you can load an [`TFAutoModel`] like you would load an [`AutoTokenizer`]. The only difference is selecting the correct [`TFAutoModel`] for the task. For text (or sequence) classification, you should load [`TFAutoModelForSequenceClassification`]:\n\n```py\n>>> from transformers import TFAutoModelForSequenceClassification\n\n>>> model_name = \"nlptown/bert-base-multilingual-uncased-sentiment\"\n>>> tf_model = TFAutoModelForSequenceClassification.from_pretrained(model_name)\n```\n\n<Tip>\n\nSee the [task summary](./task_summary) for tasks supported by an [`AutoModel`] class.\n\n</Tip>\n\nNow pass your preprocessed batch of inputs directly to the model. You can pass the tensors as-is:\n\n```py\n>>> tf_outputs = tf_model(tf_batch)\n```\n\nThe model outputs the final activations in the `logits` attribute. Apply the softmax function to the `logits` to retrieve the probabilities:\n\n```py\n>>> import tensorflow as tf\n\n>>> tf_predictions = tf.nn.softmax(tf_outputs.logits, axis=-1)\n>>> tf_predictions  # doctest: +IGNORE_RESULT\n```\n</tf>\n</frameworkcontent>\n\n<Tip>\n\nAll \ud83e\udd17 Transformers models (PyTorch or TensorFlow) output the tensors *before* the final activation\nfunction (like softmax) because the final activation function is often fused with the loss. Model outputs are special dataclasses so their attributes are autocompleted in an IDE. The model outputs behave like a tuple or a dictionary (you can index with an integer, a slice or a string) in which case, attributes that are None are ignored."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 45d23c0d-c060-4633-b0d4-9b63837d6165
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"45d23c0d-c060-4633-b0d4-9b63837d6165\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"69f0817e-23c0-4bca-846c-5ace9895af0a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"3df8fa8f9e9d30020137186799a1a245301f837f2e21ea3015ffe1543676fe0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"22e1d760-de46-41cb-8059-7aea43ba9829\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"739c0888f3584dd8e5d8cc3fb3152ee96945dd133b2fa895647bfcfb51e84ab7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c09b860d-4b95-4f92-a8a9-b6a49c0d97f9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2baf2fec40273bdf1b3868a72f659f29affd0f09d5161aaa179d466f2c429ccb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13524, \"end_char_idx\": 16219, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "doc_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "ref_doc_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "chroma:document": "This means you can load an [`TFAutoModel`] like you would load an [`AutoTokenizer`]. The only difference is selecting the correct [`TFAutoModel`] for the task. For text (or sequence) classification, you should load [`TFAutoModelForSequenceClassification`]:\n\n```py\n>>> from transformers import TFAutoModelForSequenceClassification\n\n>>> model_name = \"nlptown/bert-base-multilingual-uncased-sentiment\"\n>>> tf_model = TFAutoModelForSequenceClassification.from_pretrained(model_name)\n```\n\n<Tip>\n\nSee the [task summary](./task_summary) for tasks supported by an [`AutoModel`] class.\n\n</Tip>\n\nNow pass your preprocessed batch of inputs directly to the model. You can pass the tensors as-is:\n\n```py\n>>> tf_outputs = tf_model(tf_batch)\n```\n\nThe model outputs the final activations in the `logits` attribute. Apply the softmax function to the `logits` to retrieve the probabilities:\n\n```py\n>>> import tensorflow as tf\n\n>>> tf_predictions = tf.nn.softmax(tf_outputs.logits, axis=-1)\n>>> tf_predictions  # doctest: +IGNORE_RESULT\n```\n</tf>\n</frameworkcontent>\n\n<Tip>\n\nAll \ud83e\udd17 Transformers models (PyTorch or TensorFlow) output the tensors *before* the final activation\nfunction (like softmax) because the final activation function is often fused with the loss. Model outputs are special dataclasses so their attributes are autocompleted in an IDE. The model outputs behave like a tuple or a dictionary (you can index with an integer, a slice or a string) in which case, attributes that are None are ignored.\n\n</Tip>\n\n### Save a model\n\n<frameworkcontent>\n<pt>\nOnce your model is fine-tuned, you can save it with its tokenizer using [`PreTrainedModel.save_pretrained`]:\n\n```py\n>>> pt_save_directory = \"./pt_save_pretrained\"\n>>> tokenizer.save_pretrained(pt_save_directory)  # doctest: +IGNORE_RESULT\n>>> pt_model.save_pretrained(pt_save_directory)\n```\n\nWhen you are ready to use the model again, reload it with [`PreTrainedModel.from_pretrained`]:\n\n```py\n>>> pt_model = AutoModelForSequenceClassification.from_pretrained(\"./pt_save_pretrained\")\n```\n</pt>\n<tf>\nOnce your model is fine-tuned, you can save it with its tokenizer using [`TFPreTrainedModel.save_pretrained`]:\n\n```py\n>>> tf_save_directory = \"./tf_save_pretrained\"\n>>> tokenizer.save_pretrained(tf_save_directory)  # doctest: +IGNORE_RESULT\n>>> tf_model.save_pretrained(tf_save_directory)\n```\n\nWhen you are ready to use the model again, reload it with [`TFPreTrainedModel.from_pretrained`]:\n\n```py\n>>> tf_model = TFAutoModelForSequenceClassification.from_pretrained(\"./tf_save_pretrained\")\n```\n</tf>\n</frameworkcontent>\n\nOne particularly cool \ud83e\udd17 Transformers feature is the ability to save a model and reload it as either a PyTorch or TensorFlow model."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: c09b860d-4b95-4f92-a8a9-b6a49c0d97f9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"c09b860d-4b95-4f92-a8a9-b6a49c0d97f9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"69f0817e-23c0-4bca-846c-5ace9895af0a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"3df8fa8f9e9d30020137186799a1a245301f837f2e21ea3015ffe1543676fe0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"45d23c0d-c060-4633-b0d4-9b63837d6165\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5c982ad8551c417b6b4cf331d16215cbc608cbb0faace1de4e5950fe8ae9fe68\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"18b1306c-48a5-4818-81d0-d66f73942fa4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b428c2405a58a5cbe5d38fd98ca9ac2d25e96810c349ffee301f0b22621ed4c4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14771, \"end_char_idx\": 18154, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "doc_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "ref_doc_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "chroma:document": "Model outputs are special dataclasses so their attributes are autocompleted in an IDE. The model outputs behave like a tuple or a dictionary (you can index with an integer, a slice or a string) in which case, attributes that are None are ignored.\n\n</Tip>\n\n### Save a model\n\n<frameworkcontent>\n<pt>\nOnce your model is fine-tuned, you can save it with its tokenizer using [`PreTrainedModel.save_pretrained`]:\n\n```py\n>>> pt_save_directory = \"./pt_save_pretrained\"\n>>> tokenizer.save_pretrained(pt_save_directory)  # doctest: +IGNORE_RESULT\n>>> pt_model.save_pretrained(pt_save_directory)\n```\n\nWhen you are ready to use the model again, reload it with [`PreTrainedModel.from_pretrained`]:\n\n```py\n>>> pt_model = AutoModelForSequenceClassification.from_pretrained(\"./pt_save_pretrained\")\n```\n</pt>\n<tf>\nOnce your model is fine-tuned, you can save it with its tokenizer using [`TFPreTrainedModel.save_pretrained`]:\n\n```py\n>>> tf_save_directory = \"./tf_save_pretrained\"\n>>> tokenizer.save_pretrained(tf_save_directory)  # doctest: +IGNORE_RESULT\n>>> tf_model.save_pretrained(tf_save_directory)\n```\n\nWhen you are ready to use the model again, reload it with [`TFPreTrainedModel.from_pretrained`]:\n\n```py\n>>> tf_model = TFAutoModelForSequenceClassification.from_pretrained(\"./tf_save_pretrained\")\n```\n</tf>\n</frameworkcontent>\n\nOne particularly cool \ud83e\udd17 Transformers feature is the ability to save a model and reload it as either a PyTorch or TensorFlow model. The `from_pt` or `from_tf` parameter can convert the model from one framework to the other:\n\n<frameworkcontent>\n<pt>\n\n```py\n>>> from transformers import AutoModel\n\n>>> tokenizer = AutoTokenizer.from_pretrained(tf_save_directory)\n>>> pt_model = AutoModelForSequenceClassification.from_pretrained(tf_save_directory, from_tf=True)\n```\n</pt>\n<tf>\n\n```py\n>>> from transformers import TFAutoModel\n\n>>> tokenizer = AutoTokenizer.from_pretrained(pt_save_directory)\n>>> tf_model = TFAutoModelForSequenceClassification.from_pretrained(pt_save_directory, from_pt=True)\n```\n</tf>\n</frameworkcontent>\n\n## Custom model builds\n\nYou can modify the model's configuration class to change how a model is built. The configuration specifies a model's attributes, such as the number of hidden layers or attention heads. You start from scratch when you initialize a model from a custom configuration class. The model attributes are randomly initialized, and you'll need to train the model before you can use it to get meaningful results.\n\nStart by importing [`AutoConfig`], and then load the pretrained model you want to modify. Within [`AutoConfig.from_pretrained`], you can specify the attribute you want to change, such as the number of attention heads:\n\n```py\n>>> from transformers import AutoConfig\n\n>>> my_config = AutoConfig.from_pretrained(\"distilbert/distilbert-base-uncased\", n_heads=12)\n```\n\n<frameworkcontent>\n<pt>\nCreate a model from your custom configuration with [`AutoModel.from_config`]:\n\n```py\n>>> from transformers import AutoModel\n\n>>> my_model = AutoModel.from_config(my_config)\n```\n</pt>\n<tf>\nCreate a model from your custom configuration with [`TFAutoModel.from_config`]:\n\n```py\n>>> from transformers import TFAutoModel\n\n>>> my_model = TFAutoModel.from_config(my_config)\n```\n</tf>\n</frameworkcontent>\n\nTake a look at the [Create a custom architecture](./create_a_model) guide for more information about building custom configurations."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 18b1306c-48a5-4818-81d0-d66f73942fa4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"18b1306c-48a5-4818-81d0-d66f73942fa4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"69f0817e-23c0-4bca-846c-5ace9895af0a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"3df8fa8f9e9d30020137186799a1a245301f837f2e21ea3015ffe1543676fe0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c09b860d-4b95-4f92-a8a9-b6a49c0d97f9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"d5aa84a94af2ee233166e26e748d94ca547122d7f56b02b638dbd1607e38a086\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d82282e1-6b50-4a00-b469-f031e04d7f50\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"75e61de65b9b3126c2f3b0f785aeae98034f23cbc1c8198e6faa886ec8b6c9a3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 16912, \"end_char_idx\": 20239, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "doc_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "ref_doc_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "chroma:document": "The configuration specifies a model's attributes, such as the number of hidden layers or attention heads. You start from scratch when you initialize a model from a custom configuration class. The model attributes are randomly initialized, and you'll need to train the model before you can use it to get meaningful results.\n\nStart by importing [`AutoConfig`], and then load the pretrained model you want to modify. Within [`AutoConfig.from_pretrained`], you can specify the attribute you want to change, such as the number of attention heads:\n\n```py\n>>> from transformers import AutoConfig\n\n>>> my_config = AutoConfig.from_pretrained(\"distilbert/distilbert-base-uncased\", n_heads=12)\n```\n\n<frameworkcontent>\n<pt>\nCreate a model from your custom configuration with [`AutoModel.from_config`]:\n\n```py\n>>> from transformers import AutoModel\n\n>>> my_model = AutoModel.from_config(my_config)\n```\n</pt>\n<tf>\nCreate a model from your custom configuration with [`TFAutoModel.from_config`]:\n\n```py\n>>> from transformers import TFAutoModel\n\n>>> my_model = TFAutoModel.from_config(my_config)\n```\n</tf>\n</frameworkcontent>\n\nTake a look at the [Create a custom architecture](./create_a_model) guide for more information about building custom configurations.\n\n## Trainer - a PyTorch optimized training loop\n\nAll models are a standard [`torch.nn.Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) so you can use them in any typical training loop. While you can write your own training loop, \ud83e\udd17 Transformers provides a [`Trainer`] class for PyTorch, which contains the basic training loop and adds additional functionality for features like distributed training, mixed precision, and more.\n\nDepending on your task, you'll typically pass the following parameters to [`Trainer`]:\n\n1. You'll start with a [`PreTrainedModel`] or a [`torch.nn.Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module):\n\n   ```py\n   >>> from transformers import AutoModelForSequenceClassification\n\n   >>> model = AutoModelForSequenceClassification.from_pretrained(\"distilbert/distilbert-base-uncased\")\n   ```\n\n2. [`TrainingArguments`] contains the model hyperparameters you can change like learning rate, batch size, and the number of epochs to train for. The default values are used if you don't specify any training arguments:\n\n   ```py\n   >>> from transformers import TrainingArguments\n\n   >>> training_args = TrainingArguments(\n   ...     output_dir=\"path/to/save/folder/\",\n   ...     learning_rate=2e-5,\n   ...     per_device_train_batch_size=8,\n   ...     per_device_eval_batch_size=8,\n   ...     num_train_epochs=2,\n   ... )\n   ```\n\n3. Load a preprocessing class like a tokenizer, image processor, feature extractor, or processor:\n\n   ```py\n   >>> from transformers import AutoTokenizer\n\n   >>> tokenizer = AutoTokenizer.from_pretrained(\"distilbert/distilbert-base-uncased\")\n   ```\n\n4. Load a dataset:\n\n   ```py\n   >>> from datasets import load_dataset\n\n   >>> dataset = load_dataset(\"rotten_tomatoes\")  # doctest: +IGNORE_RESULT\n   ```\n\n5. Create a function to tokenize the dataset:\n\n   ```py\n   >>> def tokenize_dataset(dataset):\n   ...     return tokenizer(dataset[\"text\"])\n   ```\n\n   Then apply it over the entire dataset with [`~datasets.Dataset.map`]:\n\n   ```py\n   >>> dataset = dataset.map(tokenize_dataset, batched=True)\n   ```\n\n6."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: d82282e1-6b50-4a00-b469-f031e04d7f50
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"d82282e1-6b50-4a00-b469-f031e04d7f50\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"69f0817e-23c0-4bca-846c-5ace9895af0a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"3df8fa8f9e9d30020137186799a1a245301f837f2e21ea3015ffe1543676fe0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"18b1306c-48a5-4818-81d0-d66f73942fa4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"c2e137ce0247c07837d893b321b7fc9bbde1aafe5ad0487a8c046d35ef16af0f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fbd00022-f900-4a68-a1fa-1b216d7f4f21\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"61a8a9c1a5e20ea7b470219acc1c9008b652fbc04245b056eb7b91ddb9f590d6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 18599, \"end_char_idx\": 21737, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "doc_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "ref_doc_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "chroma:document": "Depending on your task, you'll typically pass the following parameters to [`Trainer`]:\n\n1. You'll start with a [`PreTrainedModel`] or a [`torch.nn.Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module):\n\n   ```py\n   >>> from transformers import AutoModelForSequenceClassification\n\n   >>> model = AutoModelForSequenceClassification.from_pretrained(\"distilbert/distilbert-base-uncased\")\n   ```\n\n2. [`TrainingArguments`] contains the model hyperparameters you can change like learning rate, batch size, and the number of epochs to train for. The default values are used if you don't specify any training arguments:\n\n   ```py\n   >>> from transformers import TrainingArguments\n\n   >>> training_args = TrainingArguments(\n   ...     output_dir=\"path/to/save/folder/\",\n   ...     learning_rate=2e-5,\n   ...     per_device_train_batch_size=8,\n   ...     per_device_eval_batch_size=8,\n   ...     num_train_epochs=2,\n   ... )\n   ```\n\n3. Load a preprocessing class like a tokenizer, image processor, feature extractor, or processor:\n\n   ```py\n   >>> from transformers import AutoTokenizer\n\n   >>> tokenizer = AutoTokenizer.from_pretrained(\"distilbert/distilbert-base-uncased\")\n   ```\n\n4. Load a dataset:\n\n   ```py\n   >>> from datasets import load_dataset\n\n   >>> dataset = load_dataset(\"rotten_tomatoes\")  # doctest: +IGNORE_RESULT\n   ```\n\n5. Create a function to tokenize the dataset:\n\n   ```py\n   >>> def tokenize_dataset(dataset):\n   ...     return tokenizer(dataset[\"text\"])\n   ```\n\n   Then apply it over the entire dataset with [`~datasets.Dataset.map`]:\n\n   ```py\n   >>> dataset = dataset.map(tokenize_dataset, batched=True)\n   ```\n\n6. A [`DataCollatorWithPadding`] to create a batch of examples from your dataset:\n\n   ```py\n   >>> from transformers import DataCollatorWithPadding\n\n   >>> data_collator = DataCollatorWithPadding(tokenizer=tokenizer)\n   ```\n\nNow gather all these classes in [`Trainer`]:\n\n```py\n>>> from transformers import Trainer\n\n>>> trainer = Trainer(\n...     model=model,\n...     args=training_args,\n...     train_dataset=dataset[\"train\"],\n...     eval_dataset=dataset[\"test\"],\n...     tokenizer=tokenizer,\n...     data_collator=data_collator,\n... )  # doctest: +SKIP\n```\n\nWhen you're ready, call [`~Trainer.train`] to start training:\n\n```py\n>>> trainer.train()  # doctest: +SKIP\n```\n\n<Tip>\n\nFor tasks - like translation or summarization - that use a sequence-to-sequence model, use the [`Seq2SeqTrainer`] and [`Seq2SeqTrainingArguments`] classes instead.\n\n</Tip>\n\nYou can customize the training loop behavior by subclassing the methods inside [`Trainer`]. This allows you to customize features such as the loss function, optimizer, and scheduler. Take a look at the [`Trainer`] reference for which methods can be subclassed. \n\nThe other way to customize the training loop is by using [Callbacks](./main_classes/callback). You can use callbacks to integrate with other libraries and inspect the training loop to report on progress or stop the training early. Callbacks do not modify anything in the training loop itself. To customize something like the loss function, you need to subclass the [`Trainer`] instead."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: fbd00022-f900-4a68-a1fa-1b216d7f4f21
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"fbd00022-f900-4a68-a1fa-1b216d7f4f21\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"69f0817e-23c0-4bca-846c-5ace9895af0a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"3df8fa8f9e9d30020137186799a1a245301f837f2e21ea3015ffe1543676fe0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d82282e1-6b50-4a00-b469-f031e04d7f50\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0ea1240657f41850f3ec6410b2930ccf17b54802db76c11b0c45aa349718857e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7b69c8a8-950b-425f-84d9-5cfe9f09c543\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e44743883676472df595ad995b2d82e1f791f6645c551eaf22ae8065266fee36\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 20240, \"end_char_idx\": 23501, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "doc_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "ref_doc_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "chroma:document": "A [`DataCollatorWithPadding`] to create a batch of examples from your dataset:\n\n   ```py\n   >>> from transformers import DataCollatorWithPadding\n\n   >>> data_collator = DataCollatorWithPadding(tokenizer=tokenizer)\n   ```\n\nNow gather all these classes in [`Trainer`]:\n\n```py\n>>> from transformers import Trainer\n\n>>> trainer = Trainer(\n...     model=model,\n...     args=training_args,\n...     train_dataset=dataset[\"train\"],\n...     eval_dataset=dataset[\"test\"],\n...     tokenizer=tokenizer,\n...     data_collator=data_collator,\n... )  # doctest: +SKIP\n```\n\nWhen you're ready, call [`~Trainer.train`] to start training:\n\n```py\n>>> trainer.train()  # doctest: +SKIP\n```\n\n<Tip>\n\nFor tasks - like translation or summarization - that use a sequence-to-sequence model, use the [`Seq2SeqTrainer`] and [`Seq2SeqTrainingArguments`] classes instead.\n\n</Tip>\n\nYou can customize the training loop behavior by subclassing the methods inside [`Trainer`]. This allows you to customize features such as the loss function, optimizer, and scheduler. Take a look at the [`Trainer`] reference for which methods can be subclassed. \n\nThe other way to customize the training loop is by using [Callbacks](./main_classes/callback). You can use callbacks to integrate with other libraries and inspect the training loop to report on progress or stop the training early. Callbacks do not modify anything in the training loop itself. To customize something like the loss function, you need to subclass the [`Trainer`] instead.\n\n## Train with TensorFlow\n\nAll models are a standard [`tf.keras.Model`](https://www.tensorflow.org/api_docs/python/tf/keras/Model) so they can be trained in TensorFlow with the [Keras](https://keras.io/) API. \ud83e\udd17 Transformers provides the [`~TFPreTrainedModel.prepare_tf_dataset`] method to easily load your dataset as a `tf.data.Dataset` so you can start training right away with Keras' [`compile`](https://keras.io/api/models/model_training_apis/#compile-method) and [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) methods.\n\n1. You'll start with a [`TFPreTrainedModel`] or a [`tf.keras.Model`](https://www.tensorflow.org/api_docs/python/tf/keras/Model):\n\n   ```py\n   >>> from transformers import TFAutoModelForSequenceClassification\n\n   >>> model = TFAutoModelForSequenceClassification.from_pretrained(\"distilbert/distilbert-base-uncased\")\n   ```\n\n2. Load a preprocessing class like a tokenizer, image processor, feature extractor, or processor:\n\n   ```py\n   >>> from transformers import AutoTokenizer\n\n   >>> tokenizer = AutoTokenizer.from_pretrained(\"distilbert/distilbert-base-uncased\")\n   ```\n\n3. Create a function to tokenize the dataset:\n\n   ```py\n   >>> def tokenize_dataset(dataset):\n   ...     return tokenizer(dataset[\"text\"])  # doctest: +SKIP\n   ```\n\n4. Apply the tokenizer over the entire dataset with [`~datasets.Dataset.map`] and then pass the dataset and tokenizer to [`~TFPreTrainedModel.prepare_tf_dataset`]. You can also change the batch size and shuffle the dataset here if you'd like:\n\n   ```py\n   >>> dataset = dataset.map(tokenize_dataset)  # doctest: +SKIP\n   >>> tf_dataset = model.prepare_tf_dataset(\n   ...     dataset[\"train\"], batch_size=16, shuffle=True, tokenizer=tokenizer\n   ... )  # doctest: +SKIP\n   ```\n\n5."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 7b69c8a8-950b-425f-84d9-5cfe9f09c543
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/quicktour/", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"7b69c8a8-950b-425f-84d9-5cfe9f09c543\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"69f0817e-23c0-4bca-846c-5ace9895af0a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"3df8fa8f9e9d30020137186799a1a245301f837f2e21ea3015ffe1543676fe0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"fbd00022-f900-4a68-a1fa-1b216d7f4f21\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/quicktour/\", \"title\": \"Quick tour\", \"tokens\": 5735, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"d0c389fceb632b1a1ac647f273d3735da2868714bf90898663f4be89e8ebea03\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 21947, \"end_char_idx\": 24255, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "doc_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "ref_doc_id": "69f0817e-23c0-4bca-846c-5ace9895af0a", "chroma:document": "\ud83e\udd17 Transformers provides the [`~TFPreTrainedModel.prepare_tf_dataset`] method to easily load your dataset as a `tf.data.Dataset` so you can start training right away with Keras' [`compile`](https://keras.io/api/models/model_training_apis/#compile-method) and [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) methods.\n\n1. You'll start with a [`TFPreTrainedModel`] or a [`tf.keras.Model`](https://www.tensorflow.org/api_docs/python/tf/keras/Model):\n\n   ```py\n   >>> from transformers import TFAutoModelForSequenceClassification\n\n   >>> model = TFAutoModelForSequenceClassification.from_pretrained(\"distilbert/distilbert-base-uncased\")\n   ```\n\n2. Load a preprocessing class like a tokenizer, image processor, feature extractor, or processor:\n\n   ```py\n   >>> from transformers import AutoTokenizer\n\n   >>> tokenizer = AutoTokenizer.from_pretrained(\"distilbert/distilbert-base-uncased\")\n   ```\n\n3. Create a function to tokenize the dataset:\n\n   ```py\n   >>> def tokenize_dataset(dataset):\n   ...     return tokenizer(dataset[\"text\"])  # doctest: +SKIP\n   ```\n\n4. Apply the tokenizer over the entire dataset with [`~datasets.Dataset.map`] and then pass the dataset and tokenizer to [`~TFPreTrainedModel.prepare_tf_dataset`]. You can also change the batch size and shuffle the dataset here if you'd like:\n\n   ```py\n   >>> dataset = dataset.map(tokenize_dataset)  # doctest: +SKIP\n   >>> tf_dataset = model.prepare_tf_dataset(\n   ...     dataset[\"train\"], batch_size=16, shuffle=True, tokenizer=tokenizer\n   ... )  # doctest: +SKIP\n   ```\n\n5. When you're ready, you can call `compile` and `fit` to start training. Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:\n\n   ```py\n   >>> from tensorflow.keras.optimizers import Adam\n\n   >>> model.compile(optimizer='adam')  # No loss argument!\n   >>> model.fit(tf_dataset)  # doctest: +SKIP\n   ```\n\n## What's next?\n\nNow that you've completed the \ud83e\udd17 Transformers quick tour, check out our guides and learn how to do more specific things like writing a custom model, fine-tuning a model for a task, and how to train a model with a script. If you're interested in learning more about \ud83e\udd17 Transformers core concepts, grab a cup of coffee and take a look at our Conceptual Guides!"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: a27b5ed6-5fad-4e93-83d4-d8b119ec3685
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"a27b5ed6-5fad-4e93-83d4-d8b119ec3685\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f70c357f-3346-456a-b50a-8bba28db2c88\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8e5e9e146ea1d1d455499f2fdf931f83db97537ae996f5b608186a3fa2a50bfa\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a4950503-fbd5-4367-a378-93fd58e427f8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c73989040327e25a5f79f2fa66909924a0b5781cbb3e570b886f45e06b1107f1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3469, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "doc_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "ref_doc_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "chroma:document": "# Trainer\n\nThe [`Trainer`] is a complete training and evaluation loop for PyTorch models implemented in the Transformers library. You only need to pass it the necessary pieces for training (model, tokenizer, dataset, evaluation function, training hyperparameters, etc.), and the [`Trainer`] class takes care of the rest. This makes it easier to start training faster without manually writing your own training loop. But at the same time, [`Trainer`] is very customizable and offers a ton of training options so you can tailor it to your exact training needs.\n\n<Tip>\n\nIn addition to the [`Trainer`] class, Transformers also provides a [`Seq2SeqTrainer`] class for sequence-to-sequence tasks like translation or summarization. There is also the [`~trl.SFTTrainer`] class from the [TRL](https://hf.co/docs/trl) library which wraps the [`Trainer`] class and is optimized for training language models like Llama-2 and Mistral with autoregressive techniques. [`~trl.SFTTrainer`] also supports features like sequence packing, LoRA, quantization, and DeepSpeed for efficiently scaling to any model size.\n\n<br>\n\nFeel free to check out the [API reference](./main_classes/trainer) for these other [`Trainer`]-type classes to learn more about when to use which one. In general, [`Trainer`] is the most versatile option and is appropriate for a broad spectrum of tasks. [`Seq2SeqTrainer`] is designed for sequence-to-sequence tasks and [`~trl.SFTTrainer`] is designed for training language models.\n\n</Tip>\n\nBefore you start, make sure [Accelerate](https://hf.co/docs/accelerate) - a library for enabling and running PyTorch training across distributed environments - is installed.\n\n```bash\npip install accelerate\n\n# upgrade\npip install accelerate --upgrade\n```\n\nThis guide provides an overview of the [`Trainer`] class.\n\n## Basic usage\n\n[`Trainer`] includes all the code you'll find in a basic training loop:\n\n1. perform a training step to calculate the loss\n2. calculate the gradients with the [`~accelerate.Accelerator.backward`] method\n3. update the weights based on the gradients\n4. repeat this process until you've reached a predetermined number of epochs\n\nThe [`Trainer`] class abstracts all of this code away so you don't have to worry about manually writing a training loop every time or if you're just getting started with PyTorch and training. You only need to provide the essential components required for training, such as a model and a dataset, and the [`Trainer`] class handles everything else.\n\nIf you want to specify any training options or hyperparameters, you can find them in the [`TrainingArguments`] class. For example, let's define where to save the model in `output_dir` and push the model to the Hub after training with `push_to_hub=True`.\n\n```py\nfrom transformers import TrainingArguments\n\ntraining_args = TrainingArguments(\n    output_dir=\"your-model\",\n    learning_rate=2e-5,\n    per_device_train_batch_size=16,\n    per_device_eval_batch_size=16,\n    num_train_epochs=2,\n    weight_decay=0.01,\n    eval_strategy=\"epoch\",\n    save_strategy=\"epoch\",\n    load_best_model_at_end=True,\n    push_to_hub=True,\n)\n```\n\nPass `training_args` to the [`Trainer`] along with a model, dataset, something to preprocess the dataset with (depending on your data type it could be a tokenizer, feature extractor or image processor), a data collator, and a function to compute the metrics you want to track during training.\n\nFinally, call [`~Trainer.train`] to start training!"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: a4950503-fbd5-4367-a378-93fd58e427f8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"a4950503-fbd5-4367-a378-93fd58e427f8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f70c357f-3346-456a-b50a-8bba28db2c88\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8e5e9e146ea1d1d455499f2fdf931f83db97537ae996f5b608186a3fa2a50bfa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a27b5ed6-5fad-4e93-83d4-d8b119ec3685\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"2ae0b3f3203e112949fbcdf79c05cd37b7c1ba7fc9a8c5d73cae7c1ca9a3f0ef\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"50f181fd-0f29-4492-8b32-8f67ad50ab42\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b83cef4bfd4a7f7a31fdd599e3d7927cf609105e8ab369dbf3436366f21f9a0a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1808, \"end_char_idx\": 5274, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "doc_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "ref_doc_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "chroma:document": "## Basic usage\n\n[`Trainer`] includes all the code you'll find in a basic training loop:\n\n1. perform a training step to calculate the loss\n2. calculate the gradients with the [`~accelerate.Accelerator.backward`] method\n3. update the weights based on the gradients\n4. repeat this process until you've reached a predetermined number of epochs\n\nThe [`Trainer`] class abstracts all of this code away so you don't have to worry about manually writing a training loop every time or if you're just getting started with PyTorch and training. You only need to provide the essential components required for training, such as a model and a dataset, and the [`Trainer`] class handles everything else.\n\nIf you want to specify any training options or hyperparameters, you can find them in the [`TrainingArguments`] class. For example, let's define where to save the model in `output_dir` and push the model to the Hub after training with `push_to_hub=True`.\n\n```py\nfrom transformers import TrainingArguments\n\ntraining_args = TrainingArguments(\n    output_dir=\"your-model\",\n    learning_rate=2e-5,\n    per_device_train_batch_size=16,\n    per_device_eval_batch_size=16,\n    num_train_epochs=2,\n    weight_decay=0.01,\n    eval_strategy=\"epoch\",\n    save_strategy=\"epoch\",\n    load_best_model_at_end=True,\n    push_to_hub=True,\n)\n```\n\nPass `training_args` to the [`Trainer`] along with a model, dataset, something to preprocess the dataset with (depending on your data type it could be a tokenizer, feature extractor or image processor), a data collator, and a function to compute the metrics you want to track during training.\n\nFinally, call [`~Trainer.train`] to start training!\n\n```py\nfrom transformers import Trainer\n\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=dataset[\"train\"],\n    eval_dataset=dataset[\"test\"],\n    tokenizer=tokenizer,\n    data_collator=data_collator,\n    compute_metrics=compute_metrics,\n)\n\ntrainer.train()\n```\n\n### Checkpoints\n\nThe [`Trainer`] class saves your model checkpoints to the directory specified in the `output_dir` parameter of [`TrainingArguments`]. You'll find the checkpoints saved in a `checkpoint-000` subfolder where the numbers at the end correspond to the training step. Saving checkpoints are useful for resuming training later.\n\n```py\n# resume from latest checkpoint\ntrainer.train(resume_from_checkpoint=True)\n\n# resume from specific checkpoint saved in output directory\ntrainer.train(resume_from_checkpoint=\"your-model/checkpoint-1000\")\n```\n\nYou can save your checkpoints (the optimizer state is not saved by default) to the Hub by setting `push_to_hub=True` in [`TrainingArguments`] to commit and push them. Other options for deciding how your checkpoints are saved are set up in the [`hub_strategy`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.hub_strategy) parameter:\n\n* `hub_strategy=\"checkpoint\"` pushes the latest checkpoint to a subfolder named \"last-checkpoint\" from which you can resume training\n* `hub_strategy=\"all_checkpoints\"` pushes all checkpoints to the directory defined in `output_dir` (you'll see one checkpoint per folder in your model repository)\n\nWhen you resume training from a checkpoint, the [`Trainer`] tries to keep the Python, NumPy, and PyTorch RNG states the same as they were when the checkpoint was saved. But because PyTorch has various non-deterministic default settings, the RNG states aren't guaranteed to be the same."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 50f181fd-0f29-4492-8b32-8f67ad50ab42
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"50f181fd-0f29-4492-8b32-8f67ad50ab42\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f70c357f-3346-456a-b50a-8bba28db2c88\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8e5e9e146ea1d1d455499f2fdf931f83db97537ae996f5b608186a3fa2a50bfa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a4950503-fbd5-4367-a378-93fd58e427f8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0e8ee71a435ce365b890aeeff7d763c574b18669bd1b124328866db733a977ca\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b3e035da-4276-4281-b0e2-1c59f06bfe34\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3ac69d12e3599402cd8789085c6f2e23b4d38232fc5618ec5dd5fbd556e8bc2b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3471, \"end_char_idx\": 7020, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "doc_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "ref_doc_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "chroma:document": "```py\nfrom transformers import Trainer\n\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=dataset[\"train\"],\n    eval_dataset=dataset[\"test\"],\n    tokenizer=tokenizer,\n    data_collator=data_collator,\n    compute_metrics=compute_metrics,\n)\n\ntrainer.train()\n```\n\n### Checkpoints\n\nThe [`Trainer`] class saves your model checkpoints to the directory specified in the `output_dir` parameter of [`TrainingArguments`]. You'll find the checkpoints saved in a `checkpoint-000` subfolder where the numbers at the end correspond to the training step. Saving checkpoints are useful for resuming training later.\n\n```py\n# resume from latest checkpoint\ntrainer.train(resume_from_checkpoint=True)\n\n# resume from specific checkpoint saved in output directory\ntrainer.train(resume_from_checkpoint=\"your-model/checkpoint-1000\")\n```\n\nYou can save your checkpoints (the optimizer state is not saved by default) to the Hub by setting `push_to_hub=True` in [`TrainingArguments`] to commit and push them. Other options for deciding how your checkpoints are saved are set up in the [`hub_strategy`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.hub_strategy) parameter:\n\n* `hub_strategy=\"checkpoint\"` pushes the latest checkpoint to a subfolder named \"last-checkpoint\" from which you can resume training\n* `hub_strategy=\"all_checkpoints\"` pushes all checkpoints to the directory defined in `output_dir` (you'll see one checkpoint per folder in your model repository)\n\nWhen you resume training from a checkpoint, the [`Trainer`] tries to keep the Python, NumPy, and PyTorch RNG states the same as they were when the checkpoint was saved. But because PyTorch has various non-deterministic default settings, the RNG states aren't guaranteed to be the same. If you want to enable full determinism, take a look at the [Controlling sources of randomness](https://pytorch.org/docs/stable/notes/randomness#controlling-sources-of-randomness) guide to learn what you can enable to make your training fully deterministic. Keep in mind though that by making certain settings deterministic, training may be slower.\n\n## Customize the Trainer\n\nWhile the [`Trainer`] class is designed to be accessible and easy-to-use, it also offers a lot of customizability for more adventurous users. Many of the [`Trainer`]'s method can be subclassed and overridden to support the functionality you want, without having to rewrite the entire training loop from scratch to accommodate it. These methods include:\n\n* [`~Trainer.get_train_dataloader`] creates a training DataLoader\n* [`~Trainer.get_eval_dataloader`] creates an evaluation DataLoader\n* [`~Trainer.get_test_dataloader`] creates a test DataLoader\n* [`~Trainer.log`] logs information on the various objects that watch training\n* [`~Trainer.create_optimizer_and_scheduler`] creates an optimizer and learning rate scheduler if they weren't passed in the `__init__`; these can also be separately customized with [`~Trainer.create_optimizer`] and [`~Trainer.create_scheduler`] respectively\n* [`~Trainer.compute_loss`] computes the loss on a batch of training inputs\n* [`~Trainer.training_step`] performs the training step\n* [`~Trainer.prediction_step`] performs the prediction and test step\n* [`~Trainer.evaluate`] evaluates the model and returns the evaluation metrics\n* [`~Trainer.predict`] makes predictions (with metrics if labels are available) on the test set\n\nFor example, if you want to customize the [`~Trainer.compute_loss`] method to use a weighted loss instead."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: b3e035da-4276-4281-b0e2-1c59f06bfe34
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"b3e035da-4276-4281-b0e2-1c59f06bfe34\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f70c357f-3346-456a-b50a-8bba28db2c88\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8e5e9e146ea1d1d455499f2fdf931f83db97537ae996f5b608186a3fa2a50bfa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"50f181fd-0f29-4492-8b32-8f67ad50ab42\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"56a462f16131784ef5b237cef28d3ff0814b24f1860014e2421574792c7e4014\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5e196db1-2152-437a-aaf3-21a45fb4c2f5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8096f1b996cc2c648faada935544920a581958d5fb48441d5ee0bb077dd6baaa\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5275, \"end_char_idx\": 8613, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "doc_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "ref_doc_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "chroma:document": "If you want to enable full determinism, take a look at the [Controlling sources of randomness](https://pytorch.org/docs/stable/notes/randomness#controlling-sources-of-randomness) guide to learn what you can enable to make your training fully deterministic. Keep in mind though that by making certain settings deterministic, training may be slower.\n\n## Customize the Trainer\n\nWhile the [`Trainer`] class is designed to be accessible and easy-to-use, it also offers a lot of customizability for more adventurous users. Many of the [`Trainer`]'s method can be subclassed and overridden to support the functionality you want, without having to rewrite the entire training loop from scratch to accommodate it. These methods include:\n\n* [`~Trainer.get_train_dataloader`] creates a training DataLoader\n* [`~Trainer.get_eval_dataloader`] creates an evaluation DataLoader\n* [`~Trainer.get_test_dataloader`] creates a test DataLoader\n* [`~Trainer.log`] logs information on the various objects that watch training\n* [`~Trainer.create_optimizer_and_scheduler`] creates an optimizer and learning rate scheduler if they weren't passed in the `__init__`; these can also be separately customized with [`~Trainer.create_optimizer`] and [`~Trainer.create_scheduler`] respectively\n* [`~Trainer.compute_loss`] computes the loss on a batch of training inputs\n* [`~Trainer.training_step`] performs the training step\n* [`~Trainer.prediction_step`] performs the prediction and test step\n* [`~Trainer.evaluate`] evaluates the model and returns the evaluation metrics\n* [`~Trainer.predict`] makes predictions (with metrics if labels are available) on the test set\n\nFor example, if you want to customize the [`~Trainer.compute_loss`] method to use a weighted loss instead.\n\n```py\nfrom torch import nn\nfrom transformers import Trainer\n\nclass CustomTrainer(Trainer):\n    def compute_loss(self, model, inputs, return_outputs=False):\n        labels = inputs.pop(\"labels\")\n        # forward pass\n        outputs = model(**inputs)\n        logits = outputs.get(\"logits\")\n        # compute custom loss for 3 labels with different weights\n        loss_fct = nn.CrossEntropyLoss(weight=torch.tensor([1.0, 2.0, 3.0], device=model.device))\n        loss = loss_fct(logits.view(-1, self.model.config.num_labels), labels.view(-1))\n        return (loss, outputs) if return_outputs else loss\n```\n\n### Callbacks\n\nAnother option for customizing the [`Trainer`] is to use [callbacks](callbacks). Callbacks *don't change* anything in the training loop. They inspect the training loop state and then execute some action (early stopping, logging results, etc.) depending on the state. In other words, a callback can't be used to implement something like a custom loss function and you'll need to subclass and override the [`~Trainer.compute_loss`] method for that.\n\nFor example, if you want to add an early stopping callback to the training loop after 10 steps.\n\n```py\nfrom transformers import TrainerCallback\n\nclass EarlyStoppingCallback(TrainerCallback):\n    def __init__(self, num_steps=10):\n        self.num_steps = num_steps\n    \n    def on_step_end(self, args, state, control, **kwargs):\n        if state.global_step >= self.num_steps:\n            return {\"should_training_stop\": True}\n        else:\n            return {}\n```\n\nThen pass it to the [`Trainer`]'s `callback` parameter."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 5e196db1-2152-437a-aaf3-21a45fb4c2f5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"5e196db1-2152-437a-aaf3-21a45fb4c2f5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f70c357f-3346-456a-b50a-8bba28db2c88\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8e5e9e146ea1d1d455499f2fdf931f83db97537ae996f5b608186a3fa2a50bfa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b3e035da-4276-4281-b0e2-1c59f06bfe34\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"93f42bbec48de48d071215fe083f31fdeae8dc237455dfed1e7bbf2435bc8571\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"18c566d8-5550-4947-acca-0c5d1b431eb4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e4a692305e1d9e04ce8d5de4ed4a2d52c947f5bb02a47025960726f306177531\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7022, \"end_char_idx\": 10172, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "doc_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "ref_doc_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "chroma:document": "```py\nfrom torch import nn\nfrom transformers import Trainer\n\nclass CustomTrainer(Trainer):\n    def compute_loss(self, model, inputs, return_outputs=False):\n        labels = inputs.pop(\"labels\")\n        # forward pass\n        outputs = model(**inputs)\n        logits = outputs.get(\"logits\")\n        # compute custom loss for 3 labels with different weights\n        loss_fct = nn.CrossEntropyLoss(weight=torch.tensor([1.0, 2.0, 3.0], device=model.device))\n        loss = loss_fct(logits.view(-1, self.model.config.num_labels), labels.view(-1))\n        return (loss, outputs) if return_outputs else loss\n```\n\n### Callbacks\n\nAnother option for customizing the [`Trainer`] is to use [callbacks](callbacks). Callbacks *don't change* anything in the training loop. They inspect the training loop state and then execute some action (early stopping, logging results, etc.) depending on the state. In other words, a callback can't be used to implement something like a custom loss function and you'll need to subclass and override the [`~Trainer.compute_loss`] method for that.\n\nFor example, if you want to add an early stopping callback to the training loop after 10 steps.\n\n```py\nfrom transformers import TrainerCallback\n\nclass EarlyStoppingCallback(TrainerCallback):\n    def __init__(self, num_steps=10):\n        self.num_steps = num_steps\n    \n    def on_step_end(self, args, state, control, **kwargs):\n        if state.global_step >= self.num_steps:\n            return {\"should_training_stop\": True}\n        else:\n            return {}\n```\n\nThen pass it to the [`Trainer`]'s `callback` parameter.\n\n```py\nfrom transformers import Trainer\n\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=dataset[\"train\"],\n    eval_dataset=dataset[\"test\"],\n    tokenizer=tokenizer,\n    data_collator=data_collator,\n    compute_metrics=compute_metrics,\n    callback=[EarlyStoppingCallback()],\n)\n```\n\n## Logging\n\n<Tip>\n\nCheck out the [logging](./main_classes/logging) API reference for more information about the different logging levels.\n\n</Tip>\n\nThe [`Trainer`] is set to `logging.INFO` by default which reports errors, warnings, and other basic information. A [`Trainer`] replica - in distributed environments - is set to `logging.WARNING` which only reports errors and warnings. You can change the logging level with the [`log_level`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.log_level) and [`log_level_replica`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.log_level_replica) parameters in [`TrainingArguments`].\n\nTo configure the log level setting for each node, use the [`log_on_each_node`](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.TrainingArguments.log_on_each_node) parameter to determine whether to use the log level on each node or only on the main node.\n\n<Tip>\n\n[`Trainer`] sets the log level separately for each node in the [`Trainer.__init__`] method, so you may want to consider setting this sooner if you're using other Transformers functionalities before creating the [`Trainer`] object."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 18c566d8-5550-4947-acca-0c5d1b431eb4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"18c566d8-5550-4947-acca-0c5d1b431eb4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f70c357f-3346-456a-b50a-8bba28db2c88\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8e5e9e146ea1d1d455499f2fdf931f83db97537ae996f5b608186a3fa2a50bfa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5e196db1-2152-437a-aaf3-21a45fb4c2f5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"fa968655c4a983a8d781d6dd507829bf6b8fc6bd95144eb63367ea01de8a11e6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e75846be-12af-4f6f-ae78-1833c5a0375b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"400af10df9f3ce12b740746bb79063b84391fdd89592d53c2e52ec082b39823b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8615, \"end_char_idx\": 11854, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "doc_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "ref_doc_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "chroma:document": "```py\nfrom transformers import Trainer\n\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=dataset[\"train\"],\n    eval_dataset=dataset[\"test\"],\n    tokenizer=tokenizer,\n    data_collator=data_collator,\n    compute_metrics=compute_metrics,\n    callback=[EarlyStoppingCallback()],\n)\n```\n\n## Logging\n\n<Tip>\n\nCheck out the [logging](./main_classes/logging) API reference for more information about the different logging levels.\n\n</Tip>\n\nThe [`Trainer`] is set to `logging.INFO` by default which reports errors, warnings, and other basic information. A [`Trainer`] replica - in distributed environments - is set to `logging.WARNING` which only reports errors and warnings. You can change the logging level with the [`log_level`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.log_level) and [`log_level_replica`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.log_level_replica) parameters in [`TrainingArguments`].\n\nTo configure the log level setting for each node, use the [`log_on_each_node`](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.TrainingArguments.log_on_each_node) parameter to determine whether to use the log level on each node or only on the main node.\n\n<Tip>\n\n[`Trainer`] sets the log level separately for each node in the [`Trainer.__init__`] method, so you may want to consider setting this sooner if you're using other Transformers functionalities before creating the [`Trainer`] object.\n\n</Tip>\n\nFor example, to set your main code and modules to use the same log level according to each node:\n\n```py\nlogger = logging.getLogger(__name__)\n\nlogging.basicConfig(\n    format=\"%(asctime)s - %(levelname)s - %(name)s - %(message)s\",\n    datefmt=\"%m/%d/%Y %H:%M:%S\",\n    handlers=[logging.StreamHandler(sys.stdout)],\n)\n\nlog_level = training_args.get_process_log_level()\nlogger.setLevel(log_level)\ndatasets.utils.logging.set_verbosity(log_level)\ntransformers.utils.logging.set_verbosity(log_level)\n\ntrainer = Trainer(...)\n```\n\nUse different combinations of `log_level` and `log_level_replica` to configure what gets logged on each of the nodes.\n\n<hfoptions id=\"logging\">\n<hfoption id=\"single node\">\n\n```bash\nmy_app.py ... --log_level warning --log_level_replica error\n```\n\n</hfoption>\n<hfoption id=\"multi-node\">\n\nAdd the `log_on_each_node 0` parameter for multi-node environments.\n\n```bash\nmy_app.py ... --log_level warning --log_level_replica error --log_on_each_node 0\n\n# set to only report errors\nmy_app.py ... --log_level error --log_level_replica error --log_on_each_node 0\n```\n\n</hfoption>\n</hfoptions>\n\n## NEFTune\n\n[NEFTune](https://hf.co/papers/2310.05914) is a technique that can improve performance by adding noise to the embedding vectors during training. To enable it in [`Trainer`], set the `neftune_noise_alpha` parameter in [`TrainingArguments`] to control how much noise is added.\n\n```py\nfrom transformers import TrainingArguments, Trainer\n\ntraining_args = TrainingArguments(..., neftune_noise_alpha=0.1)\ntrainer = Trainer(..., args=training_args)\n```\n\nNEFTune is disabled after training to restore the original embedding layer to avoid any unexpected behavior."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: e75846be-12af-4f6f-ae78-1833c5a0375b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"e75846be-12af-4f6f-ae78-1833c5a0375b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f70c357f-3346-456a-b50a-8bba28db2c88\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8e5e9e146ea1d1d455499f2fdf931f83db97537ae996f5b608186a3fa2a50bfa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"18c566d8-5550-4947-acca-0c5d1b431eb4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f11fd9635fede0965b253ab910bd1f9e1388073c0a9799d33198a27454f7da5d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"efbbcd84-a4c4-4b3c-9be5-d1b95110a5d0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"69d73554b5a1af28f4384900b686b50d8950010d1b591e3f4024b2fabeb756de\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10823, \"end_char_idx\": 12411, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "doc_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "ref_doc_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "chroma:document": "<hfoptions id=\"logging\">\n<hfoption id=\"single node\">\n\n```bash\nmy_app.py ... --log_level warning --log_level_replica error\n```\n\n</hfoption>\n<hfoption id=\"multi-node\">\n\nAdd the `log_on_each_node 0` parameter for multi-node environments.\n\n```bash\nmy_app.py ... --log_level warning --log_level_replica error --log_on_each_node 0\n\n# set to only report errors\nmy_app.py ... --log_level error --log_level_replica error --log_on_each_node 0\n```\n\n</hfoption>\n</hfoptions>\n\n## NEFTune\n\n[NEFTune](https://hf.co/papers/2310.05914) is a technique that can improve performance by adding noise to the embedding vectors during training. To enable it in [`Trainer`], set the `neftune_noise_alpha` parameter in [`TrainingArguments`] to control how much noise is added.\n\n```py\nfrom transformers import TrainingArguments, Trainer\n\ntraining_args = TrainingArguments(..., neftune_noise_alpha=0.1)\ntrainer = Trainer(..., args=training_args)\n```\n\nNEFTune is disabled after training to restore the original embedding layer to avoid any unexpected behavior.\n\n## GaLore\n\nGradient Low-Rank Projection (GaLore) is a memory-efficient low-rank training strategy that allows full-parameter learning but is more memory-efficient than common low-rank adaptation methods, such as LoRA.\n\nFirst make sure to install GaLore official repository:\n\n```bash\npip install galore-torch\n```\n\nThen simply add one of `[\"galore_adamw\", \"galore_adafactor\", \"galore_adamw_8bit\"]` in `optim` together with `optim_target_modules`, which can be a list of strings, regex or full path corresponding to the target module names you want to adapt."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: efbbcd84-a4c4-4b3c-9be5-d1b95110a5d0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"efbbcd84-a4c4-4b3c-9be5-d1b95110a5d0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f70c357f-3346-456a-b50a-8bba28db2c88\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8e5e9e146ea1d1d455499f2fdf931f83db97537ae996f5b608186a3fa2a50bfa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e75846be-12af-4f6f-ae78-1833c5a0375b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"ecf092d95b5960db7b27332137afd2dd9a208a0d763c24de528df0c47b45cb06\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"56813570-13c5-4d91-83f3-1ec1376a062e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"18d01c6f0a32ae4a4e8cacd52dd342424179017e1952a967df19dc70f5ccb613\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10823, \"end_char_idx\": 14292, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "doc_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "ref_doc_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "chroma:document": "<hfoptions id=\"logging\">\n<hfoption id=\"single node\">\n\n```bash\nmy_app.py ... --log_level warning --log_level_replica error\n```\n\n</hfoption>\n<hfoption id=\"multi-node\">\n\nAdd the `log_on_each_node 0` parameter for multi-node environments.\n\n```bash\nmy_app.py ... --log_level warning --log_level_replica error --log_on_each_node 0\n\n# set to only report errors\nmy_app.py ... --log_level error --log_level_replica error --log_on_each_node 0\n```\n\n</hfoption>\n</hfoptions>\n\n## NEFTune\n\n[NEFTune](https://hf.co/papers/2310.05914) is a technique that can improve performance by adding noise to the embedding vectors during training. To enable it in [`Trainer`], set the `neftune_noise_alpha` parameter in [`TrainingArguments`] to control how much noise is added.\n\n```py\nfrom transformers import TrainingArguments, Trainer\n\ntraining_args = TrainingArguments(..., neftune_noise_alpha=0.1)\ntrainer = Trainer(..., args=training_args)\n```\n\nNEFTune is disabled after training to restore the original embedding layer to avoid any unexpected behavior.\n\n## GaLore\n\nGradient Low-Rank Projection (GaLore) is a memory-efficient low-rank training strategy that allows full-parameter learning but is more memory-efficient than common low-rank adaptation methods, such as LoRA.\n\nFirst make sure to install GaLore official repository:\n\n```bash\npip install galore-torch\n```\n\nThen simply add one of `[\"galore_adamw\", \"galore_adafactor\", \"galore_adamw_8bit\"]` in `optim` together with `optim_target_modules`, which can be a list of strings, regex or full path corresponding to the target module names you want to adapt. Below is an end-to-end example script (make sure to `pip install trl datasets`):\n\n```python\nimport torch\nimport datasets\nimport trl\n\nfrom transformers import TrainingArguments, AutoConfig, AutoTokenizer, AutoModelForCausalLM\n\ntrain_dataset = datasets.load_dataset('imdb', split='train')\n\nargs = TrainingArguments(\n    output_dir=\"./test-galore\",\n    max_steps=100,\n    per_device_train_batch_size=2,\n    optim=\"galore_adamw\",\n    optim_target_modules=[\"attn\", \"mlp\"]\n)\n\nmodel_id = \"google/gemma-2b\"\n\nconfig = AutoConfig.from_pretrained(model_id)\n\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nmodel = AutoModelForCausalLM.from_config(config).to(0)\n\ntrainer = trl.SFTTrainer(\n    model=model, \n    args=args,\n    train_dataset=train_dataset,\n    dataset_text_field='text',\n    max_seq_length=512,\n)\n\ntrainer.train()\n```\n\nTo pass extra arguments supports by GaLore, you should pass correctly `optim_args`, for example:\n\n```python\nimport torch\nimport datasets\nimport trl\n\nfrom transformers import TrainingArguments, AutoConfig, AutoTokenizer, AutoModelForCausalLM\n\ntrain_dataset = datasets.load_dataset('imdb', split='train')\n\nargs = TrainingArguments(\n    output_dir=\"./test-galore\",\n    max_steps=100,\n    per_device_train_batch_size=2,\n    optim=\"galore_adamw\",\n    optim_target_modules=[\"attn\", \"mlp\"],\n    optim_args=\"rank=64, update_proj_gap=100, scale=0.10\",\n)\n\nmodel_id = \"google/gemma-2b\"\n\nconfig = AutoConfig.from_pretrained(model_id)\n\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nmodel = AutoModelForCausalLM.from_config(config).to(0)\n\ntrainer = trl.SFTTrainer(\n    model=model, \n    args=args,\n    train_dataset=train_dataset,\n    dataset_text_field='text',\n    max_seq_length=512,\n)\n\ntrainer.train()\n```\n\nYou can read more about the method in the [original repository](https://github.com/jiaweizzhao/GaLore) or the [paper](https://arxiv.org/abs/2403.03507)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 56813570-13c5-4d91-83f3-1ec1376a062e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"56813570-13c5-4d91-83f3-1ec1376a062e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f70c357f-3346-456a-b50a-8bba28db2c88\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8e5e9e146ea1d1d455499f2fdf931f83db97537ae996f5b608186a3fa2a50bfa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"efbbcd84-a4c4-4b3c-9be5-d1b95110a5d0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"13375c2d5758885b34051a3feb435f05d5ac8408b583b1146e0878b6f364d5d8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"af49548b-3b9a-47b4-99aa-2fa0f52189cc\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0e1106b978fc14ae02ea553da24672cc236801000700933f3ba3b4912abeb356\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14294, \"end_char_idx\": 16849, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "doc_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "ref_doc_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "chroma:document": "Currently you can only train Linear layers that are considered as GaLore layers and will use low-rank decomposition to be trained while remaining layers will be optimized in the conventional manner.\n\nNote it will take a bit of time before starting the training (~3 minutes for a 2B model on a NVIDIA A100), but training should go smoothly afterwards.\n\nYou can also perform layer-wise optimization by post-pending the optimizer name with `layerwise` like below:\n\n```python\nimport torch\nimport datasets\nimport trl\n\nfrom transformers import TrainingArguments, AutoConfig, AutoTokenizer, AutoModelForCausalLM\n\ntrain_dataset = datasets.load_dataset('imdb', split='train')\n\nargs = TrainingArguments(\n    output_dir=\"./test-galore\",\n    max_steps=100,\n    per_device_train_batch_size=2,\n    optim=\"galore_adamw_layerwise\",\n    optim_target_modules=[\"attn\", \"mlp\"]\n)\n\nmodel_id = \"google/gemma-2b\"\n\nconfig = AutoConfig.from_pretrained(model_id)\n\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nmodel = AutoModelForCausalLM.from_config(config).to(0)\n\ntrainer = trl.SFTTrainer(\n    model=model, \n    args=args,\n    train_dataset=train_dataset,\n    dataset_text_field='text',\n    max_seq_length=512,\n)\n\ntrainer.train()\n```\n\nNote layerwise optimization is a bit experimental and does not support DDP (Distributed Data Parallel), thus you can run the training script only on a single GPU. Please see [this appropriate section](https://github.com/jiaweizzhao/GaLore?tab=readme-ov-file#train-7b-model-with-a-single-gpu-with-24gb-memory) for more details. Other features such as gradient clipping, DeepSpeed, etc might not be supported out of the box. Please [raise an issue on GitHub](https://github.com/huggingface/transformers/issues) if you encounter such issue.\n\n## LOMO optimizer\n\nThe LOMO optimizers have been introduced in [Full Parameter Fine-Tuning for Large Language Models with Limited Resources](https://hf.co/papers/2306.09782) and [AdaLomo: Low-memory Optimization with Adaptive Learning Rate](https://hf.co/papers/2310.10195). \nThey both consist of an efficient full-parameter fine-tuning method. These optimizers fuse the gradient computation and the parameter update in one step to reduce memory usage. Supported optimizers for LOMO are `\"lomo\"` and `\"adalomo\"`. First either install LOMO from pypi `pip install lomo-optim` or install it from source with `pip install git+https://github.com/OpenLMLab/LOMO.git`. \n\n<Tip>\n\nAccording to the authors, it is recommended to use `AdaLomo` without `grad_norm` to get better performance and higher throughput."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: af49548b-3b9a-47b4-99aa-2fa0f52189cc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"af49548b-3b9a-47b4-99aa-2fa0f52189cc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f70c357f-3346-456a-b50a-8bba28db2c88\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8e5e9e146ea1d1d455499f2fdf931f83db97537ae996f5b608186a3fa2a50bfa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"56813570-13c5-4d91-83f3-1ec1376a062e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5c3b7d436a06b7e256d324c1b547dbd12e359e715a1af59003f31c9ea93e288b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b964ab35-75c8-452a-8124-c4e3a4e2b96b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4d26d021b0bb1ef0193d0213ecd443fdfa444ffc43406d77aa57c29dad3060cd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15673, \"end_char_idx\": 18667, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "doc_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "ref_doc_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "chroma:document": "Please see [this appropriate section](https://github.com/jiaweizzhao/GaLore?tab=readme-ov-file#train-7b-model-with-a-single-gpu-with-24gb-memory) for more details. Other features such as gradient clipping, DeepSpeed, etc might not be supported out of the box. Please [raise an issue on GitHub](https://github.com/huggingface/transformers/issues) if you encounter such issue.\n\n## LOMO optimizer\n\nThe LOMO optimizers have been introduced in [Full Parameter Fine-Tuning for Large Language Models with Limited Resources](https://hf.co/papers/2306.09782) and [AdaLomo: Low-memory Optimization with Adaptive Learning Rate](https://hf.co/papers/2310.10195). \nThey both consist of an efficient full-parameter fine-tuning method. These optimizers fuse the gradient computation and the parameter update in one step to reduce memory usage. Supported optimizers for LOMO are `\"lomo\"` and `\"adalomo\"`. First either install LOMO from pypi `pip install lomo-optim` or install it from source with `pip install git+https://github.com/OpenLMLab/LOMO.git`. \n\n<Tip>\n\nAccording to the authors, it is recommended to use `AdaLomo` without `grad_norm` to get better performance and higher throughput.\n\n</Tip>\n\nBelow is a simple script to demonstrate how to fine-tune [google/gemma-2b](https://huggingface.co/google/gemma-2b) on IMDB dataset in full precision:\n\n```python\nimport torch\nimport datasets\nfrom transformers import TrainingArguments, AutoTokenizer, AutoModelForCausalLM\nimport trl\n\ntrain_dataset = datasets.load_dataset('imdb', split='train')\n\nargs = TrainingArguments(\n    output_dir=\"./test-lomo\",\n    max_steps=1000,\n    per_device_train_batch_size=4,\n    optim=\"adalomo\",\n    gradient_checkpointing=True,\n    logging_strategy=\"steps\",\n    logging_steps=1,\n    learning_rate=2e-6,\n    save_strategy=\"no\",\n    run_name=\"lomo-imdb\",\n)\n\nmodel_id = \"google/gemma-2b\"\n\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nmodel = AutoModelForCausalLM.from_pretrained(model_id, low_cpu_mem_usage=True).to(0)\n\ntrainer = trl.SFTTrainer(\n    model=model, \n    args=args,\n    train_dataset=train_dataset,\n    dataset_text_field='text',\n    max_seq_length=1024,\n)\n\ntrainer.train()\n```\n\n## Accelerate and Trainer\n\nThe [`Trainer`] class is powered by [Accelerate](https://hf.co/docs/accelerate), a library for easily training PyTorch models in distributed environments with support for integrations such as [FullyShardedDataParallel (FSDP)](https://pytorch.org/blog/introducing-pytorch-fully-sharded-data-parallel-api/) and [DeepSpeed](https://www.deepspeed.ai/).\n\n<Tip>\n\nLearn more about FSDP sharding strategies, CPU offloading, and more with the [`Trainer`] in the [Fully Sharded Data Parallel](fsdp) guide.\n\n</Tip>\n\nTo use Accelerate with [`Trainer`], run the [`accelerate.config`](https://huggingface.co/docs/accelerate/package_reference/cli#accelerate-config) command to set up training for your training environment. This command creates a `config_file.yaml` that'll be used when you launch your training script."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: b964ab35-75c8-452a-8124-c4e3a4e2b96b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"b964ab35-75c8-452a-8124-c4e3a4e2b96b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f70c357f-3346-456a-b50a-8bba28db2c88\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8e5e9e146ea1d1d455499f2fdf931f83db97537ae996f5b608186a3fa2a50bfa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"af49548b-3b9a-47b4-99aa-2fa0f52189cc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"ba946ff824f0fdf5ede7d02e83a44c5e04113f480a259be1512365ddca367609\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e94485e2-d25e-454d-ac88-5129c0359c45\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b86580875dbfe86cc0625fe7769988277b2a76725355a704803eb4d60bf484cc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 18213, \"end_char_idx\": 21542, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "doc_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "ref_doc_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "chroma:document": "<Tip>\n\nLearn more about FSDP sharding strategies, CPU offloading, and more with the [`Trainer`] in the [Fully Sharded Data Parallel](fsdp) guide.\n\n</Tip>\n\nTo use Accelerate with [`Trainer`], run the [`accelerate.config`](https://huggingface.co/docs/accelerate/package_reference/cli#accelerate-config) command to set up training for your training environment. This command creates a `config_file.yaml` that'll be used when you launch your training script. For example, some example configurations you can setup are:\n\n<hfoptions id=\"config\">\n<hfoption id=\"DistributedDataParallel\">\n\n```yml\ncompute_environment: LOCAL_MACHINE                                                                                             \ndistributed_type: MULTI_GPU                                                                                                    \ndowncast_bf16: 'no'\ngpu_ids: all\nmachine_rank: 0 #change rank as per the node\nmain_process_ip: 192.168.20.1\nmain_process_port: 9898\nmain_training_function: main\nmixed_precision: fp16\nnum_machines: 2\nnum_processes: 8\nrdzv_backend: static\nsame_network: true\ntpu_env: []\ntpu_use_cluster: false\ntpu_use_sudo: false\nuse_cpu: false\n```\n\n</hfoption>\n<hfoption id=\"FSDP\">\n\n```yml\ncompute_environment: LOCAL_MACHINE\ndistributed_type: FSDP\ndowncast_bf16: 'no'\nfsdp_config:\n  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP\n  fsdp_backward_prefetch_policy: BACKWARD_PRE\n  fsdp_forward_prefetch: true\n  fsdp_offload_params: false\n  fsdp_sharding_strategy: 1\n  fsdp_state_dict_type: FULL_STATE_DICT\n  fsdp_sync_module_states: true\n  fsdp_transformer_layer_cls_to_wrap: BertLayer\n  fsdp_use_orig_params: true\nmachine_rank: 0\nmain_training_function: main\nmixed_precision: bf16\nnum_machines: 1\nnum_processes: 2\nrdzv_backend: static\nsame_network: true\ntpu_env: []\ntpu_use_cluster: false\ntpu_use_sudo: false\nuse_cpu: false\n```\n\n</hfoption>\n<hfoption id=\"DeepSpeed\">\n\n```yml\ncompute_environment: LOCAL_MACHINE\ndeepspeed_config:\n  deepspeed_config_file: /home/user/configs/ds_zero3_config.json\n  zero3_init_flag: true\ndistributed_type: DEEPSPEED\ndowncast_bf16: 'no'\nmachine_rank: 0\nmain_training_function: main\nnum_machines: 1\nnum_processes: 4\nrdzv_backend: static\nsame_network: true\ntpu_env: []\ntpu_use_cluster: false\ntpu_use_sudo: false\nuse_cpu: false\n```\n\n</hfoption>\n<hfoption id=\"DeepSpeed with Accelerate plugin\">\n\n```yml\ncompute_environment: LOCAL_MACHINE                                                                                             \ndeepspeed_config:                                                                                                              \n  gradient_accumulation_steps: 1\n  gradient_clipping: 0.7\n  offload_optimizer_device: cpu\n  offload_param_device: cpu\n  zero3_init_flag: true\n  zero_stage: 2\ndistributed_type: DEEPSPEED\ndowncast_bf16: 'no'\nmachine_rank: 0\nmain_training_function: main\nmixed_precision: bf16\nnum_machines: 1\nnum_processes: 4\nrdzv_backend: static\nsame_network: true\ntpu_env: []\ntpu_use_cluster: false\ntpu_use_sudo: false\nuse_cpu: false\n```\n\n</hfoption>\n</hfoptions>\n\nThe [`accelerate_launch`](https://huggingface.co/docs/accelerate/package_reference/cli#accelerate-launch) command is the recommended way to launch your training script on a distributed system with Accelerate and [`Trainer`] with the parameters specified in `config_file.yaml`."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: e94485e2-d25e-454d-ac88-5129c0359c45
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/trainer/", "title": "Trainer", "tokens": 5652, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"e94485e2-d25e-454d-ac88-5129c0359c45\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f70c357f-3346-456a-b50a-8bba28db2c88\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8e5e9e146ea1d1d455499f2fdf931f83db97537ae996f5b608186a3fa2a50bfa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b964ab35-75c8-452a-8124-c4e3a4e2b96b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/trainer/\", \"title\": \"Trainer\", \"tokens\": 5652, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"486b2d28e76b6c93b8283ea374a5d5ffb5baacc6be23a6f4588883357b58247b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 21543, \"end_char_idx\": 23199, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "doc_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "ref_doc_id": "f70c357f-3346-456a-b50a-8bba28db2c88", "chroma:document": "This file is saved to the Accelerate cache folder and automatically loaded when you run `accelerate_launch`.\n\nFor example, to run the [run_glue.py](https://github.com/huggingface/transformers/blob/f4db565b695582891e43a5e042e5d318e28f20b8/examples/pytorch/text-classification/run_glue.py#L4) training script with the FSDP configuration:\n\n```bash\naccelerate launch \\\n    ./examples/pytorch/text-classification/run_glue.py \\\n    --model_name_or_path google-bert/bert-base-cased \\\n    --task_name $TASK_NAME \\\n    --do_train \\\n    --do_eval \\\n    --max_seq_length 128 \\\n    --per_device_train_batch_size 16 \\\n    --learning_rate 5e-5 \\\n    --num_train_epochs 3 \\\n    --output_dir /tmp/$TASK_NAME/ \\\n    --overwrite_output_dir\n```\n\nYou could also specify the parameters from the `config_file.yaml` file directly in the command line:\n\n```bash\naccelerate launch --num_processes=2 \\\n    --use_fsdp \\\n    --mixed_precision=bf16 \\\n    --fsdp_auto_wrap_policy=TRANSFORMER_BASED_WRAP  \\\n    --fsdp_transformer_layer_cls_to_wrap=\"BertLayer\" \\\n    --fsdp_sharding_strategy=1 \\\n    --fsdp_state_dict_type=FULL_STATE_DICT \\\n    ./examples/pytorch/text-classification/run_glue.py\n    --model_name_or_path google-bert/bert-base-cased \\\n    --task_name $TASK_NAME \\\n    --do_train \\\n    --do_eval \\\n    --max_seq_length 128 \\\n    --per_device_train_batch_size 16 \\\n    --learning_rate 5e-5 \\\n    --num_train_epochs 3 \\\n    --output_dir /tmp/$TASK_NAME/ \\\n    --overwrite_output_dir\n```\n\nCheck out the [Launching your Accelerate scripts](https://huggingface.co/docs/accelerate/basic_tutorials/launch) tutorial to learn more about `accelerate_launch` and custom configurations."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 1b16d9bc-c728-4491-9319-20c3321b7b01
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"1b16d9bc-c728-4491-9319-20c3321b7b01\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/run_scripts/\", \"title\": \"Train with a script\", \"tokens\": 4440, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c3a407ba-8fce-4d64-8e90-12d39d21c156\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/run_scripts/\", \"title\": \"Train with a script\", \"tokens\": 4440, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b25a057f637ab1096d5ffb47c8fa86ab7cb723e23de95e8dded0e0e3cac2489f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4be9ba97-0e37-4ebe-8a16-09837e23a1a4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"69d81e0588f6a06e36c9b5d8e4f960e87c81dbbb4bee32daa3509f7272d6abc0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3004, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "doc_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "ref_doc_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "chroma:document": "# Train with a script\n\nAlong with the \ud83e\udd17 Transformers [notebooks](./notebooks), there are also example scripts demonstrating how to train a model for a task with [PyTorch](https://github.com/huggingface/transformers/tree/main/examples/pytorch), [TensorFlow](https://github.com/huggingface/transformers/tree/main/examples/tensorflow), or [JAX/Flax](https://github.com/huggingface/transformers/tree/main/examples/flax).\n\nYou will also find scripts we've used in our [research projects](https://github.com/huggingface/transformers/tree/main/examples/research_projects) and [legacy examples](https://github.com/huggingface/transformers/tree/main/examples/legacy) which are mostly community contributed. These scripts are not actively maintained and require a specific version of \ud83e\udd17 Transformers that will most likely be incompatible with the latest version of the library.\n\nThe example scripts are not expected to work out-of-the-box on every problem, and you may need to adapt the script to the problem you're trying to solve. To help you with this, most of the scripts fully expose how data is preprocessed, allowing you to edit it as necessary for your use case.\n\nFor any feature you'd like to implement in an example script, please discuss it on the [forum](https://discuss.huggingface.co/) or in an [issue](https://github.com/huggingface/transformers/issues) before submitting a Pull Request. While we welcome bug fixes, it is unlikely we will merge a Pull Request that adds more functionality at the cost of readability.\n\nThis guide will show you how to run an example summarization training script in [PyTorch](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization) and [TensorFlow](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/summarization). All examples are expected to work with both frameworks unless otherwise specified.\n\n## Setup\n\nTo successfully run the latest version of the example scripts, you have to **install \ud83e\udd17 Transformers from source** in a new virtual environment:\n\n```bash\ngit clone https://github.com/huggingface/transformers\ncd transformers\npip install .\n```\n\nFor older versions of the example scripts, click on the toggle below:\n\n<details>\n  <summary>Examples for older versions of \ud83e\udd17 Transformers</summary>\n\t<ul>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v4.5.1/examples\">v4.5.1</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v4.4.2/examples\">v4.4.2</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v4.3.3/examples\">v4.3.3</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v4.2.2/examples\">v4.2.2</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v4.1.1/examples\">v4.1.1</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v4.0.1/examples\">v4.0.1</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v3.5.1/examples\">v3.5.1</a></li>\n\t\t<li><a href=\"https://github."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 4be9ba97-0e37-4ebe-8a16-09837e23a1a4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"4be9ba97-0e37-4ebe-8a16-09837e23a1a4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/run_scripts/\", \"title\": \"Train with a script\", \"tokens\": 4440, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c3a407ba-8fce-4d64-8e90-12d39d21c156\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/run_scripts/\", \"title\": \"Train with a script\", \"tokens\": 4440, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b25a057f637ab1096d5ffb47c8fa86ab7cb723e23de95e8dded0e0e3cac2489f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1b16d9bc-c728-4491-9319-20c3321b7b01\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/run_scripts/\", \"title\": \"Train with a script\", \"tokens\": 4440, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"81194525bbd140e949ff731ea8465f3acb9b9e6645196a34325ce3463207f5a4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"691b5b28-0e79-490f-a5b2-c0c0e42edab4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"458b5668fdbef749238e11a3f8337f43be0500b308455f510069d1633a212001\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1805, \"end_char_idx\": 4034, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "doc_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "ref_doc_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "chroma:document": "All examples are expected to work with both frameworks unless otherwise specified.\n\n## Setup\n\nTo successfully run the latest version of the example scripts, you have to **install \ud83e\udd17 Transformers from source** in a new virtual environment:\n\n```bash\ngit clone https://github.com/huggingface/transformers\ncd transformers\npip install .\n```\n\nFor older versions of the example scripts, click on the toggle below:\n\n<details>\n  <summary>Examples for older versions of \ud83e\udd17 Transformers</summary>\n\t<ul>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v4.5.1/examples\">v4.5.1</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v4.4.2/examples\">v4.4.2</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v4.3.3/examples\">v4.3.3</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v4.2.2/examples\">v4.2.2</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v4.1.1/examples\">v4.1.1</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v4.0.1/examples\">v4.0.1</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v3.5.1/examples\">v3.5.1</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v3.4.0/examples\">v3.4.0</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v3.3.1/examples\">v3.3.1</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v3.2.0/examples\">v3.2.0</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v3.1.0/examples\">v3.1.0</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v3.0.2/examples\">v3.0.2</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v2.11.0/examples\">v2.11.0</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v2.10.0/examples\">v2.10.0</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v2.9.1/examples\">v2.9.1</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v2.8.0/examples\">v2.8.0</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v2.7.0/examples\">v2.7.0</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v2.6.0/examples\">v2.6."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 691b5b28-0e79-490f-a5b2-c0c0e42edab4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"691b5b28-0e79-490f-a5b2-c0c0e42edab4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/run_scripts/\", \"title\": \"Train with a script\", \"tokens\": 4440, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c3a407ba-8fce-4d64-8e90-12d39d21c156\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/run_scripts/\", \"title\": \"Train with a script\", \"tokens\": 4440, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b25a057f637ab1096d5ffb47c8fa86ab7cb723e23de95e8dded0e0e3cac2489f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4be9ba97-0e37-4ebe-8a16-09837e23a1a4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/run_scripts/\", \"title\": \"Train with a script\", \"tokens\": 4440, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"fe46d89f61a70d145390246ad7ff0342190635d32555166271726a8a4dfd72ec\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"967628fb-ac4c-4955-b87b-7c01bebffacf\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a675129f05031861d2a77a33bee7e5eb4300fa8c381eeee7ca4c27744c7713ab\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3041, \"end_char_idx\": 5066, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "doc_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "ref_doc_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "chroma:document": "4.0/examples\">v3.4.0</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v3.3.1/examples\">v3.3.1</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v3.2.0/examples\">v3.2.0</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v3.1.0/examples\">v3.1.0</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v3.0.2/examples\">v3.0.2</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v2.11.0/examples\">v2.11.0</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v2.10.0/examples\">v2.10.0</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v2.9.1/examples\">v2.9.1</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v2.8.0/examples\">v2.8.0</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v2.7.0/examples\">v2.7.0</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v2.6.0/examples\">v2.6.0</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v2.5.1/examples\">v2.5.1</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v2.4.0/examples\">v2.4.0</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v2.3.0/examples\">v2.3.0</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v2.2.0/examples\">v2.2.0</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v2.1.0/examples\">v2.1.1</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v2.0.0/examples\">v2.0.0</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v1.2.0/examples\">v1.2.0</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v1.1.0/examples\">v1.1.0</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v1.0.0/examples\">v1.0.0</a></li>\n\t</ul>\n</details>\n\nThen switch your current clone of \ud83e\udd17 Transformers to a specific version, like v3.5.1 for example:\n\n```bash\ngit checkout tags/v3.5."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 967628fb-ac4c-4955-b87b-7c01bebffacf
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"967628fb-ac4c-4955-b87b-7c01bebffacf\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/run_scripts/\", \"title\": \"Train with a script\", \"tokens\": 4440, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c3a407ba-8fce-4d64-8e90-12d39d21c156\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/run_scripts/\", \"title\": \"Train with a script\", \"tokens\": 4440, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b25a057f637ab1096d5ffb47c8fa86ab7cb723e23de95e8dded0e0e3cac2489f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"691b5b28-0e79-490f-a5b2-c0c0e42edab4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/run_scripts/\", \"title\": \"Train with a script\", \"tokens\": 4440, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"676fa360e1a3caac595995afe7fb2546f91deb376e2fe03e02271b2d6366a1eb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"25f64b7e-6d6a-455c-8a99-8a2c6229fd02\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b4e9aefd194a9462fde45f20ed8dcc2f3017d3c3280fea6c45b2194f1bd8fa91\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4017, \"end_char_idx\": 6577, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "doc_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "ref_doc_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "chroma:document": "0/examples\">v2.6.0</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v2.5.1/examples\">v2.5.1</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v2.4.0/examples\">v2.4.0</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v2.3.0/examples\">v2.3.0</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v2.2.0/examples\">v2.2.0</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v2.1.0/examples\">v2.1.1</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v2.0.0/examples\">v2.0.0</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v1.2.0/examples\">v1.2.0</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v1.1.0/examples\">v1.1.0</a></li>\n\t\t<li><a href=\"https://github.com/huggingface/transformers/tree/v1.0.0/examples\">v1.0.0</a></li>\n\t</ul>\n</details>\n\nThen switch your current clone of \ud83e\udd17 Transformers to a specific version, like v3.5.1 for example:\n\n```bash\ngit checkout tags/v3.5.1\n```\n\nAfter you've setup the correct library version, navigate to the example folder of your choice and install the example specific requirements:\n\n```bash\npip install -r requirements.txt\n```\n\n## Run a script\n\n<frameworkcontent>\n<pt>\nThe example script downloads and preprocesses a dataset from the \ud83e\udd17 [Datasets](https://huggingface.co/docs/datasets/) library. Then the script fine-tunes a dataset with the [Trainer](https://huggingface.co/docs/transformers/main_classes/trainer) on an architecture that supports summarization. The following example shows how to fine-tune [T5-small](https://huggingface.co/google-t5/t5-small) on the [CNN/DailyMail](https://huggingface.co/datasets/cnn_dailymail) dataset. The T5 model requires an additional `source_prefix` argument due to how it was trained. This prompt lets T5 know this is a summarization task.\n\n```bash\npython examples/pytorch/summarization/run_summarization.py \\\n    --model_name_or_path google-t5/t5-small \\\n    --do_train \\\n    --do_eval \\\n    --dataset_name cnn_dailymail \\\n    --dataset_config \"3.0.0\" \\\n    --source_prefix \"summarize: \" \\\n    --output_dir /tmp/tst-summarization \\\n    --per_device_train_batch_size=4 \\\n    --per_device_eval_batch_size=4 \\\n    --overwrite_output_dir \\\n    --predict_with_generate\n```\n</pt>\n<tf>\nThe example script downloads and preprocesses a dataset from the \ud83e\udd17 [Datasets](https://huggingface.co/docs/datasets/) library. Then the script fine-tunes a dataset using Keras on an architecture that supports summarization."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 25f64b7e-6d6a-455c-8a99-8a2c6229fd02
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"25f64b7e-6d6a-455c-8a99-8a2c6229fd02\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/run_scripts/\", \"title\": \"Train with a script\", \"tokens\": 4440, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c3a407ba-8fce-4d64-8e90-12d39d21c156\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/run_scripts/\", \"title\": \"Train with a script\", \"tokens\": 4440, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b25a057f637ab1096d5ffb47c8fa86ab7cb723e23de95e8dded0e0e3cac2489f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"967628fb-ac4c-4955-b87b-7c01bebffacf\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/run_scripts/\", \"title\": \"Train with a script\", \"tokens\": 4440, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"1834ed163e104c00e2d17d2797e0a5d91efe9315a30d534aad111edd35d6180d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7b306508-aa63-45ac-9be6-1ae3bc33cb09\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"73402c8f930f349a1f30656c4a78f8fda594359a4fbe07a47e5cc1e6a12b55eb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5019, \"end_char_idx\": 7666, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "doc_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "ref_doc_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "chroma:document": "1 for example:\n\n```bash\ngit checkout tags/v3.5.1\n```\n\nAfter you've setup the correct library version, navigate to the example folder of your choice and install the example specific requirements:\n\n```bash\npip install -r requirements.txt\n```\n\n## Run a script\n\n<frameworkcontent>\n<pt>\nThe example script downloads and preprocesses a dataset from the \ud83e\udd17 [Datasets](https://huggingface.co/docs/datasets/) library. Then the script fine-tunes a dataset with the [Trainer](https://huggingface.co/docs/transformers/main_classes/trainer) on an architecture that supports summarization. The following example shows how to fine-tune [T5-small](https://huggingface.co/google-t5/t5-small) on the [CNN/DailyMail](https://huggingface.co/datasets/cnn_dailymail) dataset. The T5 model requires an additional `source_prefix` argument due to how it was trained. This prompt lets T5 know this is a summarization task.\n\n```bash\npython examples/pytorch/summarization/run_summarization.py \\\n    --model_name_or_path google-t5/t5-small \\\n    --do_train \\\n    --do_eval \\\n    --dataset_name cnn_dailymail \\\n    --dataset_config \"3.0.0\" \\\n    --source_prefix \"summarize: \" \\\n    --output_dir /tmp/tst-summarization \\\n    --per_device_train_batch_size=4 \\\n    --per_device_eval_batch_size=4 \\\n    --overwrite_output_dir \\\n    --predict_with_generate\n```\n</pt>\n<tf>\nThe example script downloads and preprocesses a dataset from the \ud83e\udd17 [Datasets](https://huggingface.co/docs/datasets/) library. Then the script fine-tunes a dataset using Keras on an architecture that supports summarization. The following example shows how to fine-tune [T5-small](https://huggingface.co/google-t5/t5-small) on the [CNN/DailyMail](https://huggingface.co/datasets/cnn_dailymail) dataset. The T5 model requires an additional `source_prefix` argument due to how it was trained. This prompt lets T5 know this is a summarization task.\n\n```bash\npython examples/tensorflow/summarization/run_summarization.py  \\\n    --model_name_or_path google-t5/t5-small \\\n    --dataset_name cnn_dailymail \\\n    --dataset_config \"3.0.0\" \\\n    --output_dir /tmp/tst-summarization  \\\n    --per_device_train_batch_size 8 \\\n    --per_device_eval_batch_size 16 \\\n    --num_train_epochs 3 \\\n    --do_train \\\n    --do_eval\n```\n</tf>\n</frameworkcontent>\n\n## Distributed training and mixed precision\n\nThe [Trainer](https://huggingface.co/docs/transformers/main_classes/trainer) supports distributed training and mixed precision, which means you can also use it in a script. To enable both of these features:\n\n- Add the `fp16` argument to enable mixed precision.\n- Set the number of GPUs to use with the `nproc_per_node` argument."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 7b306508-aa63-45ac-9be6-1ae3bc33cb09
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"7b306508-aa63-45ac-9be6-1ae3bc33cb09\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/run_scripts/\", \"title\": \"Train with a script\", \"tokens\": 4440, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c3a407ba-8fce-4d64-8e90-12d39d21c156\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/run_scripts/\", \"title\": \"Train with a script\", \"tokens\": 4440, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b25a057f637ab1096d5ffb47c8fa86ab7cb723e23de95e8dded0e0e3cac2489f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"25f64b7e-6d6a-455c-8a99-8a2c6229fd02\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/run_scripts/\", \"title\": \"Train with a script\", \"tokens\": 4440, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"3c3539dbb75a87e07eda29df9b1f1db917885def146d6bb3e4ee2e69efa24038\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f04a6330-8086-44d2-aeca-515ed13ec2db\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"db016caf71e3c5138db8a105c02f04fe9b299a401e03dfc07de1d80fb4bb9f10\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6481, \"end_char_idx\": 9409, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "doc_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "ref_doc_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "chroma:document": "Then the script fine-tunes a dataset using Keras on an architecture that supports summarization. The following example shows how to fine-tune [T5-small](https://huggingface.co/google-t5/t5-small) on the [CNN/DailyMail](https://huggingface.co/datasets/cnn_dailymail) dataset. The T5 model requires an additional `source_prefix` argument due to how it was trained. This prompt lets T5 know this is a summarization task.\n\n```bash\npython examples/tensorflow/summarization/run_summarization.py  \\\n    --model_name_or_path google-t5/t5-small \\\n    --dataset_name cnn_dailymail \\\n    --dataset_config \"3.0.0\" \\\n    --output_dir /tmp/tst-summarization  \\\n    --per_device_train_batch_size 8 \\\n    --per_device_eval_batch_size 16 \\\n    --num_train_epochs 3 \\\n    --do_train \\\n    --do_eval\n```\n</tf>\n</frameworkcontent>\n\n## Distributed training and mixed precision\n\nThe [Trainer](https://huggingface.co/docs/transformers/main_classes/trainer) supports distributed training and mixed precision, which means you can also use it in a script. To enable both of these features:\n\n- Add the `fp16` argument to enable mixed precision.\n- Set the number of GPUs to use with the `nproc_per_node` argument.\n\n```bash\ntorchrun \\\n    --nproc_per_node 8 pytorch/summarization/run_summarization.py \\\n    --fp16 \\\n    --model_name_or_path google-t5/t5-small \\\n    --do_train \\\n    --do_eval \\\n    --dataset_name cnn_dailymail \\\n    --dataset_config \"3.0.0\" \\\n    --source_prefix \"summarize: \" \\\n    --output_dir /tmp/tst-summarization \\\n    --per_device_train_batch_size=4 \\\n    --per_device_eval_batch_size=4 \\\n    --overwrite_output_dir \\\n    --predict_with_generate\n```\n\nTensorFlow scripts utilize a [`MirroredStrategy`](https://www.tensorflow.org/guide/distributed_training#mirroredstrategy) for distributed training, and you don't need to add any additional arguments to the training script. The TensorFlow script will use multiple GPUs by default if they are available.\n\n## Run a script on a TPU\n\n<frameworkcontent>\n<pt>\nTensor Processing Units (TPUs) are specifically designed to accelerate performance. PyTorch supports TPUs with the [XLA](https://www.tensorflow.org/xla) deep learning compiler (see [here](https://github.com/pytorch/xla/blob/master/README.md) for more details). To use a TPU, launch the `xla_spawn.py` script and use the `num_cores` argument to set the number of TPU cores you want to use.\n\n```bash\npython xla_spawn.py --num_cores 8 \\\n    summarization/run_summarization.py \\\n    --model_name_or_path google-t5/t5-small \\\n    --do_train \\\n    --do_eval \\\n    --dataset_name cnn_dailymail \\\n    --dataset_config \"3.0.0\" \\\n    --source_prefix \"summarize: \" \\\n    --output_dir /tmp/tst-summarization \\\n    --per_device_train_batch_size=4 \\\n    --per_device_eval_batch_size=4 \\\n    --overwrite_output_dir \\\n    --predict_with_generate\n```\n</pt>\n<tf>\nTensor Processing Units (TPUs) are specifically designed to accelerate performance."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: f04a6330-8086-44d2-aeca-515ed13ec2db
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"f04a6330-8086-44d2-aeca-515ed13ec2db\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/run_scripts/\", \"title\": \"Train with a script\", \"tokens\": 4440, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c3a407ba-8fce-4d64-8e90-12d39d21c156\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/run_scripts/\", \"title\": \"Train with a script\", \"tokens\": 4440, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b25a057f637ab1096d5ffb47c8fa86ab7cb723e23de95e8dded0e0e3cac2489f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7b306508-aa63-45ac-9be6-1ae3bc33cb09\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/run_scripts/\", \"title\": \"Train with a script\", \"tokens\": 4440, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"98cc8dcd27d455ec4c822dcc0f605c9c77add1984142c01c794a4c9669d5903e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f0358a28-7c7f-454d-9939-d5def0f9d6d6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e054c4c38fb87a613a063b5b3a907454323311f6c90d085eda22374dcc27a79c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8351, \"end_char_idx\": 10750, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "doc_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "ref_doc_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "chroma:document": "The TensorFlow script will use multiple GPUs by default if they are available.\n\n## Run a script on a TPU\n\n<frameworkcontent>\n<pt>\nTensor Processing Units (TPUs) are specifically designed to accelerate performance. PyTorch supports TPUs with the [XLA](https://www.tensorflow.org/xla) deep learning compiler (see [here](https://github.com/pytorch/xla/blob/master/README.md) for more details). To use a TPU, launch the `xla_spawn.py` script and use the `num_cores` argument to set the number of TPU cores you want to use.\n\n```bash\npython xla_spawn.py --num_cores 8 \\\n    summarization/run_summarization.py \\\n    --model_name_or_path google-t5/t5-small \\\n    --do_train \\\n    --do_eval \\\n    --dataset_name cnn_dailymail \\\n    --dataset_config \"3.0.0\" \\\n    --source_prefix \"summarize: \" \\\n    --output_dir /tmp/tst-summarization \\\n    --per_device_train_batch_size=4 \\\n    --per_device_eval_batch_size=4 \\\n    --overwrite_output_dir \\\n    --predict_with_generate\n```\n</pt>\n<tf>\nTensor Processing Units (TPUs) are specifically designed to accelerate performance. TensorFlow scripts utilize a [`TPUStrategy`](https://www.tensorflow.org/guide/distributed_training#tpustrategy) for training on TPUs. To use a TPU, pass the name of the TPU resource to the `tpu` argument.\n\n```bash\npython run_summarization.py  \\\n    --tpu name_of_tpu_resource \\\n    --model_name_or_path google-t5/t5-small \\\n    --dataset_name cnn_dailymail \\\n    --dataset_config \"3.0.0\" \\\n    --output_dir /tmp/tst-summarization  \\\n    --per_device_train_batch_size 8 \\\n    --per_device_eval_batch_size 16 \\\n    --num_train_epochs 3 \\\n    --do_train \\\n    --do_eval\n```\n</tf>\n</frameworkcontent>\n\n## Run a script with \ud83e\udd17 Accelerate\n\n\ud83e\udd17 [Accelerate](https://huggingface.co/docs/accelerate) is a PyTorch-only library that offers a unified method for training a model on several types of setups (CPU-only, multiple GPUs, TPUs) while maintaining complete visibility into the PyTorch training loop. Make sure you have \ud83e\udd17 Accelerate installed if you don't already have it:\n\n> Note: As Accelerate is rapidly developing, the git version of accelerate must be installed to run the scripts\n```bash\npip install git+https://github.com/huggingface/accelerate\n```\n\nInstead of the `run_summarization.py` script, you need to use the `run_summarization_no_trainer.py` script. \ud83e\udd17 Accelerate supported scripts will have a `task_no_trainer.py` file in the folder."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: f0358a28-7c7f-454d-9939-d5def0f9d6d6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"f0358a28-7c7f-454d-9939-d5def0f9d6d6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/run_scripts/\", \"title\": \"Train with a script\", \"tokens\": 4440, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c3a407ba-8fce-4d64-8e90-12d39d21c156\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/run_scripts/\", \"title\": \"Train with a script\", \"tokens\": 4440, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b25a057f637ab1096d5ffb47c8fa86ab7cb723e23de95e8dded0e0e3cac2489f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f04a6330-8086-44d2-aeca-515ed13ec2db\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/run_scripts/\", \"title\": \"Train with a script\", \"tokens\": 4440, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a8462377349e20b8ad7b0533073cffd737b7b3e0cb810ddbc83297ee6b481244\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c0b858fe-f154-45b3-8c88-f595ef471807\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0c2ce3951a5b8f78bb76ac0d2b436d4947b0aaf83957d6d38a05272713026635\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9410, \"end_char_idx\": 12467, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "doc_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "ref_doc_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "chroma:document": "TensorFlow scripts utilize a [`TPUStrategy`](https://www.tensorflow.org/guide/distributed_training#tpustrategy) for training on TPUs. To use a TPU, pass the name of the TPU resource to the `tpu` argument.\n\n```bash\npython run_summarization.py  \\\n    --tpu name_of_tpu_resource \\\n    --model_name_or_path google-t5/t5-small \\\n    --dataset_name cnn_dailymail \\\n    --dataset_config \"3.0.0\" \\\n    --output_dir /tmp/tst-summarization  \\\n    --per_device_train_batch_size 8 \\\n    --per_device_eval_batch_size 16 \\\n    --num_train_epochs 3 \\\n    --do_train \\\n    --do_eval\n```\n</tf>\n</frameworkcontent>\n\n## Run a script with \ud83e\udd17 Accelerate\n\n\ud83e\udd17 [Accelerate](https://huggingface.co/docs/accelerate) is a PyTorch-only library that offers a unified method for training a model on several types of setups (CPU-only, multiple GPUs, TPUs) while maintaining complete visibility into the PyTorch training loop. Make sure you have \ud83e\udd17 Accelerate installed if you don't already have it:\n\n> Note: As Accelerate is rapidly developing, the git version of accelerate must be installed to run the scripts\n```bash\npip install git+https://github.com/huggingface/accelerate\n```\n\nInstead of the `run_summarization.py` script, you need to use the `run_summarization_no_trainer.py` script. \ud83e\udd17 Accelerate supported scripts will have a `task_no_trainer.py` file in the folder. Begin by running the following command to create and save a configuration file:\n\n```bash\naccelerate config\n```\n\nTest your setup to make sure it is configured correctly:\n\n```bash\naccelerate test\n```\n\nNow you are ready to launch the training:\n\n```bash\naccelerate launch run_summarization_no_trainer.py \\\n    --model_name_or_path google-t5/t5-small \\\n    --dataset_name cnn_dailymail \\\n    --dataset_config \"3.0.0\" \\\n    --source_prefix \"summarize: \" \\\n    --output_dir ~/tmp/tst-summarization\n```\n\n## Use a custom dataset\n\nThe summarization script supports custom datasets as long as they are a CSV or JSON Line file. When you use your own dataset, you need to specify several additional arguments:\n\n- `train_file` and `validation_file` specify the path to your training and validation files.\n- `text_column` is the input text to summarize.\n- `summary_column` is the target text to output.\n\nA summarization script using a custom dataset would look like this:\n\n```bash\npython examples/pytorch/summarization/run_summarization.py \\\n    --model_name_or_path google-t5/t5-small \\\n    --do_train \\\n    --do_eval \\\n    --train_file path_to_csv_or_jsonlines_file \\\n    --validation_file path_to_csv_or_jsonlines_file \\\n    --text_column text_column_name \\\n    --summary_column summary_column_name \\\n    --source_prefix \"summarize: \" \\\n    --output_dir /tmp/tst-summarization \\\n    --overwrite_output_dir \\\n    --per_device_train_batch_size=4 \\\n    --per_device_eval_batch_size=4 \\\n    --predict_with_generate\n```\n\n## Test a script\n\nIt is often a good idea to run your script on a smaller number of dataset examples to ensure everything works as expected before committing to an entire dataset which may take hours to complete."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: c0b858fe-f154-45b3-8c88-f595ef471807
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/run_scripts/", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"c0b858fe-f154-45b3-8c88-f595ef471807\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/run_scripts/\", \"title\": \"Train with a script\", \"tokens\": 4440, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c3a407ba-8fce-4d64-8e90-12d39d21c156\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/run_scripts/\", \"title\": \"Train with a script\", \"tokens\": 4440, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b25a057f637ab1096d5ffb47c8fa86ab7cb723e23de95e8dded0e0e3cac2489f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f0358a28-7c7f-454d-9939-d5def0f9d6d6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/run_scripts/\", \"title\": \"Train with a script\", \"tokens\": 4440, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"9c495a79934e6b4cd7e742a8eae4233ab92e62c506a0be0db76e6da964146957\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"952cd0a2-b08f-448a-8cf2-fb5fca6455c1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"785d9ac72f06f758f47f399e8d1fd78d291a871f9cbff4d221e7a83bc34cae0b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11367, \"end_char_idx\": 14432, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "doc_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "ref_doc_id": "c3a407ba-8fce-4d64-8e90-12d39d21c156", "chroma:document": "When you use your own dataset, you need to specify several additional arguments:\n\n- `train_file` and `validation_file` specify the path to your training and validation files.\n- `text_column` is the input text to summarize.\n- `summary_column` is the target text to output.\n\nA summarization script using a custom dataset would look like this:\n\n```bash\npython examples/pytorch/summarization/run_summarization.py \\\n    --model_name_or_path google-t5/t5-small \\\n    --do_train \\\n    --do_eval \\\n    --train_file path_to_csv_or_jsonlines_file \\\n    --validation_file path_to_csv_or_jsonlines_file \\\n    --text_column text_column_name \\\n    --summary_column summary_column_name \\\n    --source_prefix \"summarize: \" \\\n    --output_dir /tmp/tst-summarization \\\n    --overwrite_output_dir \\\n    --per_device_train_batch_size=4 \\\n    --per_device_eval_batch_size=4 \\\n    --predict_with_generate\n```\n\n## Test a script\n\nIt is often a good idea to run your script on a smaller number of dataset examples to ensure everything works as expected before committing to an entire dataset which may take hours to complete. Use the following arguments to truncate the dataset to a maximum number of samples:\n\n- `max_train_samples`\n- `max_eval_samples`\n- `max_predict_samples`\n\n```bash\npython examples/pytorch/summarization/run_summarization.py \\\n    --model_name_or_path google-t5/t5-small \\\n    --max_train_samples 50 \\\n    --max_eval_samples 50 \\\n    --max_predict_samples 50 \\\n    --do_train \\\n    --do_eval \\\n    --dataset_name cnn_dailymail \\\n    --dataset_config \"3.0.0\" \\\n    --source_prefix \"summarize: \" \\\n    --output_dir /tmp/tst-summarization \\\n    --per_device_train_batch_size=4 \\\n    --per_device_eval_batch_size=4 \\\n    --overwrite_output_dir \\\n    --predict_with_generate\n```\n\nNot all example scripts support the `max_predict_samples` argument. If you aren't sure whether your script supports this argument, add the `-h` argument to check:\n\n```bash\nexamples/pytorch/summarization/run_summarization.py -h\n```\n\n## Resume training from checkpoint\n\nAnother helpful option to enable is resuming training from a previous checkpoint. This will ensure you can pick up where you left off without starting over if your training gets interrupted. There are two methods to resume training from a checkpoint.\n\nThe first method uses the `output_dir previous_output_dir` argument to resume training from the latest checkpoint stored in `output_dir`. In this case, you should remove `overwrite_output_dir`:\n\n```bash\npython examples/pytorch/summarization/run_summarization.py\n    --model_name_or_path google-t5/t5-small \\\n    --do_train \\\n    --do_eval \\\n    --dataset_name cnn_dailymail \\\n    --dataset_config \"3.0.0\" \\\n    --source_prefix \"summarize: \" \\\n    --output_dir /tmp/tst-summarization \\\n    --per_device_train_batch_size=4 \\\n    --per_device_eval_batch_size=4 \\\n    --output_dir previous_output_dir \\\n    --predict_with_generate\n```\n\nThe second method uses the `resume_from_checkpoint path_to_specific_checkpoint` argument to resume training from a specific checkpoint folder."}


