# embeddings_queue

created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a0dff318-8d41-4545-9a93-1843e0a6306c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/question_answering", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a0dff318-8d41-4545-9a93-1843e0a6306c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/question_answering\", \"title\": \"Question answering\", \"tokens\": 3690, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0644af4b-e7db-4d46-a9bc-a8d41dc816fb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/question_answering\", \"title\": \"Question answering\", \"tokens\": 3690, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d40ef902670412e44c6eab15348ba95f1cc6da7be2d725be4f054d04caa135cd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6d8decf9-34b1-4d00-88ad-7c2317023217\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/question_answering\", \"title\": \"Question answering\", \"tokens\": 3690, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d2f83049344b93da410f966930130c2e19572c8309ef8523d01cb2c686d47b71\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f917a26e-b4a5-4d5f-a448-647b0e76ac28\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3d5c739a1ce353b979ff2d97ff8deb958eb9426bc5c6320b404f8d709ae0369c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1509, \"end_char_idx\": 4025, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0644af4b-e7db-4d46-a9bc-a8d41dc816fb", "doc_id": "0644af4b-e7db-4d46-a9bc-a8d41dc816fb", "ref_doc_id": "0644af4b-e7db-4d46-a9bc-a8d41dc816fb", "chroma:document": "```py\n>>> from datasets import load_dataset\n\n>>> squad = load_dataset(\"squad\", split=\"train[:5000]\")\n```\n\nSplit the dataset's `train` split into a train and test set with the [`~datasets.Dataset.train_test_split`] method:\n\n```py\n>>> squad = squad.train_test_split(test_size=0.2)\n```\n\nThen take a look at an example:\n\n```py\n>>> squad[\"train\"][0]\n{'answers': {'answer_start': [515], 'text': ['Saint Bernadette Soubirous']},\n 'context': 'Architecturally, the school has a Catholic character. Atop the Main Building\\'s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend \"Venite Ad Me Omnes\". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.',\n 'id': '5733be284776f41900661182',\n 'question': 'To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?',\n 'title': 'University_of_Notre_Dame'\n}\n```\n\nThere are several important fields here:\n\n- `answers`: the starting location of the answer token and the answer text.\n- `context`: background information from which the model needs to extract the answer.\n- `question`: the question a model should answer.\n\n## Preprocess\n\n<Youtube id=\"qgaM0weJHpA\"/>\n\nThe next step is to load a DistilBERT tokenizer to process the `question` and `context` fields:\n\n```py\n>>> from transformers import AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"distilbert/distilbert-base-uncased\")\n```\n\nThere are a few preprocessing steps particular to question answering tasks you should be aware of:\n\n1. Some examples in a dataset may have a very long `context` that exceeds the maximum input length of the model. To deal with longer sequences, truncate only the `context` by setting `truncation=\"only_second\"`.\n2. Next, map the start and end positions of the answer to the original `context` by setting\n   `return_offset_mapping=True`.\n3. With the mapping in hand, now you can find the start and end tokens of the answer. Use the [`~tokenizers.Encoding.sequence_ids`] method to\n   find which part of the offset corresponds to the `question` and which corresponds to the `context`."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f917a26e-b4a5-4d5f-a448-647b0e76ac28
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/question_answering", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f917a26e-b4a5-4d5f-a448-647b0e76ac28\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/question_answering\", \"title\": \"Question answering\", \"tokens\": 3690, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0644af4b-e7db-4d46-a9bc-a8d41dc816fb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/question_answering\", \"title\": \"Question answering\", \"tokens\": 3690, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d40ef902670412e44c6eab15348ba95f1cc6da7be2d725be4f054d04caa135cd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a0dff318-8d41-4545-9a93-1843e0a6306c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/question_answering\", \"title\": \"Question answering\", \"tokens\": 3690, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"617dbeb042f879f87b56b73453b035c043d7a5ffacbfc85dc360b5fc582c63c9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c4d74967-5474-4687-91c1-94c9d455c1fd\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2f3b76818a8c4b90d8ef8b0e99271d348b4623d0126a1366d5c7318719e5f997\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2367, \"end_char_idx\": 6117, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0644af4b-e7db-4d46-a9bc-a8d41dc816fb", "doc_id": "0644af4b-e7db-4d46-a9bc-a8d41dc816fb", "ref_doc_id": "0644af4b-e7db-4d46-a9bc-a8d41dc816fb", "chroma:document": "It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.',\n 'id': '5733be284776f41900661182',\n 'question': 'To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?',\n 'title': 'University_of_Notre_Dame'\n}\n```\n\nThere are several important fields here:\n\n- `answers`: the starting location of the answer token and the answer text.\n- `context`: background information from which the model needs to extract the answer.\n- `question`: the question a model should answer.\n\n## Preprocess\n\n<Youtube id=\"qgaM0weJHpA\"/>\n\nThe next step is to load a DistilBERT tokenizer to process the `question` and `context` fields:\n\n```py\n>>> from transformers import AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"distilbert/distilbert-base-uncased\")\n```\n\nThere are a few preprocessing steps particular to question answering tasks you should be aware of:\n\n1. Some examples in a dataset may have a very long `context` that exceeds the maximum input length of the model. To deal with longer sequences, truncate only the `context` by setting `truncation=\"only_second\"`.\n2. Next, map the start and end positions of the answer to the original `context` by setting\n   `return_offset_mapping=True`.\n3. With the mapping in hand, now you can find the start and end tokens of the answer. Use the [`~tokenizers.Encoding.sequence_ids`] method to\n   find which part of the offset corresponds to the `question` and which corresponds to the `context`.\n\nHere is how you can create a function to truncate and map the start and end tokens of the `answer` to the `context`:\n\n```py\n>>> def preprocess_function(examples):\n...     questions = [q.strip() for q in examples[\"question\"]]\n...     inputs = tokenizer(\n...         questions,\n...         examples[\"context\"],\n...         max_length=384,\n...         truncation=\"only_second\",\n...         return_offsets_mapping=True,\n...         padding=\"max_length\",\n...     )\n\n...     offset_mapping = inputs.pop(\"offset_mapping\")\n...     answers = examples[\"answers\"]\n...     start_positions = []\n...     end_positions = []\n\n...     for i, offset in enumerate(offset_mapping):\n...         answer = answers[i]\n...         start_char = answer[\"answer_start\"][0]\n...         end_char = answer[\"answer_start\"][0] + len(answer[\"text\"][0])\n...         sequence_ids = inputs.sequence_ids(i)\n\n...         # Find the start and end of the context\n...         idx = 0\n...         while sequence_ids[idx] != 1:\n...             idx += 1\n...         context_start = idx\n...         while sequence_ids[idx] == 1:\n...             idx += 1\n...         context_end = idx - 1\n\n...         # If the answer is not fully inside the context, label it (0, 0)\n...         if offset[context_start][0] > end_char or offset[context_end][1] < start_char:\n...             start_positions.append(0)\n...             end_positions.append(0)\n...         else:\n...             # Otherwise it's the start and end token positions\n...             idx = context_start\n...             while idx <= context_end and offset[idx][0] <= start_char:\n...                 idx += 1\n...             start_positions.append(idx - 1)\n\n...             idx = context_end\n...             while idx >= context_start and offset[idx][1] >= end_char:\n...                 idx -= 1\n...             end_positions.append(idx + 1)\n\n...     inputs[\"start_positions\"] = start_positions\n...     inputs[\"end_positions\"] = end_positions\n...     return inputs\n```\n\nTo apply the preprocessing function over the entire dataset, use \ud83e\udd17 Datasets [`~datasets.Dataset.map`] function."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: c4d74967-5474-4687-91c1-94c9d455c1fd
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/question_answering", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c4d74967-5474-4687-91c1-94c9d455c1fd\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/question_answering\", \"title\": \"Question answering\", \"tokens\": 3690, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0644af4b-e7db-4d46-a9bc-a8d41dc816fb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/question_answering\", \"title\": \"Question answering\", \"tokens\": 3690, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d40ef902670412e44c6eab15348ba95f1cc6da7be2d725be4f054d04caa135cd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f917a26e-b4a5-4d5f-a448-647b0e76ac28\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/question_answering\", \"title\": \"Question answering\", \"tokens\": 3690, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1fca7f106b0140b26877eb60556b7466e3b53355742e8cb8393dc4841e1497f6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9d01c23e-5f3d-435d-a8a5-ad24de457f22\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f2632235133c1214f724dcec6853072d824dcae3146ced97f687cdc77f0981e9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6118, \"end_char_idx\": 8779, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0644af4b-e7db-4d46-a9bc-a8d41dc816fb", "doc_id": "0644af4b-e7db-4d46-a9bc-a8d41dc816fb", "ref_doc_id": "0644af4b-e7db-4d46-a9bc-a8d41dc816fb", "chroma:document": "You can speed up the `map` function by setting `batched=True` to process multiple elements of the dataset at once. Remove any columns you don't need:\n\n```py\n>>> tokenized_squad = squad.map(preprocess_function, batched=True, remove_columns=squad[\"train\"].column_names)\n```\n\nNow create a batch of examples using [`DefaultDataCollator`]. Unlike other data collators in \ud83e\udd17 Transformers, the [`DefaultDataCollator`] does not apply any additional preprocessing such as padding.\n\n<frameworkcontent>\n<pt>\n```py\n>>> from transformers import DefaultDataCollator\n\n>>> data_collator = DefaultDataCollator()\n```\n</pt>\n<tf>\n```py\n>>> from transformers import DefaultDataCollator\n\n>>> data_collator = DefaultDataCollator(return_tensors=\"tf\")\n```\n</tf>\n</frameworkcontent>\n\n## Train\n\n<frameworkcontent>\n<pt>\n<Tip>\n\nIf you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!\n\n</Tip>\n\nYou're ready to start training your model now! Load DistilBERT with [`AutoModelForQuestionAnswering`]:\n\n```py\n>>> from transformers import AutoModelForQuestionAnswering, TrainingArguments, Trainer\n\n>>> model = AutoModelForQuestionAnswering.from_pretrained(\"distilbert/distilbert-base-uncased\")\n```\n\nAt this point, only three steps remain:\n\n1. Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model).\n2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, and data collator.\n3. Call [`~Trainer.train`] to finetune your model.\n\n```py\n>>> training_args = TrainingArguments(\n...     output_dir=\"my_awesome_qa_model\",\n...     eval_strategy=\"epoch\",\n...     learning_rate=2e-5,\n...     per_device_train_batch_size=16,\n...     per_device_eval_batch_size=16,\n...     num_train_epochs=3,\n...     weight_decay=0.01,\n...     push_to_hub=True,\n... )\n\n>>> trainer = Trainer(\n...     model=model,\n...     args=training_args,\n...     train_dataset=tokenized_squad[\"train\"],\n...     eval_dataset=tokenized_squad[\"test\"],\n...     tokenizer=tokenizer,\n...     data_collator=data_collator,\n... )\n\n>>> trainer.train()\n```\n\nOnce training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:\n\n```py\n>>> trainer.push_to_hub()\n```\n</pt>\n<tf>\n<Tip>\n\nIf you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 9d01c23e-5f3d-435d-a8a5-ad24de457f22
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/question_answering", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9d01c23e-5f3d-435d-a8a5-ad24de457f22\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/question_answering\", \"title\": \"Question answering\", \"tokens\": 3690, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0644af4b-e7db-4d46-a9bc-a8d41dc816fb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/question_answering\", \"title\": \"Question answering\", \"tokens\": 3690, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d40ef902670412e44c6eab15348ba95f1cc6da7be2d725be4f054d04caa135cd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c4d74967-5474-4687-91c1-94c9d455c1fd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/question_answering\", \"title\": \"Question answering\", \"tokens\": 3690, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8ccdba0217de475f4b01d9a75e1e429a77ce71871088112fd42e654483736a30\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"85f38fe4-bfca-46b9-b31d-90b41973eebc\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a03f9849ba499f8975613becf42236d30b92d2585f0381d7cc7662419648d4ce\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7419, \"end_char_idx\": 10249, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0644af4b-e7db-4d46-a9bc-a8d41dc816fb", "doc_id": "0644af4b-e7db-4d46-a9bc-a8d41dc816fb", "ref_doc_id": "0644af4b-e7db-4d46-a9bc-a8d41dc816fb", "chroma:document": "Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model).\n2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, and data collator.\n3. Call [`~Trainer.train`] to finetune your model.\n\n```py\n>>> training_args = TrainingArguments(\n...     output_dir=\"my_awesome_qa_model\",\n...     eval_strategy=\"epoch\",\n...     learning_rate=2e-5,\n...     per_device_train_batch_size=16,\n...     per_device_eval_batch_size=16,\n...     num_train_epochs=3,\n...     weight_decay=0.01,\n...     push_to_hub=True,\n... )\n\n>>> trainer = Trainer(\n...     model=model,\n...     args=training_args,\n...     train_dataset=tokenized_squad[\"train\"],\n...     eval_dataset=tokenized_squad[\"test\"],\n...     tokenizer=tokenizer,\n...     data_collator=data_collator,\n... )\n\n>>> trainer.train()\n```\n\nOnce training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:\n\n```py\n>>> trainer.push_to_hub()\n```\n</pt>\n<tf>\n<Tip>\n\nIf you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!\n\n</Tip>\nTo finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:\n\n```py\n>>> from transformers import create_optimizer\n\n>>> batch_size = 16\n>>> num_epochs = 2\n>>> total_train_steps = (len(tokenized_squad[\"train\"]) // batch_size) * num_epochs\n>>> optimizer, schedule = create_optimizer(\n...     init_lr=2e-5,\n...     num_warmup_steps=0,\n...     num_train_steps=total_train_steps,\n... )\n```\n\nThen you can load DistilBERT with [`TFAutoModelForQuestionAnswering`]:\n\n```py\n>>> from transformers import TFAutoModelForQuestionAnswering\n\n>>> model = TFAutoModelForQuestionAnswering.from_pretrained(\"distilbert/distilbert-base-uncased\")\n```\n\nConvert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:\n\n```py\n>>> tf_train_set = model.prepare_tf_dataset(\n...     tokenized_squad[\"train\"],\n...     shuffle=True,\n...     batch_size=16,\n...     collate_fn=data_collator,\n... )\n\n>>> tf_validation_set = model.prepare_tf_dataset(\n...     tokenized_squad[\"test\"],\n...     shuffle=False,\n...     batch_size=16,\n...     collate_fn=data_collator,\n... )\n```\n\nConfigure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method):\n\n```py\n>>> import tensorflow as tf\n\n>>> model.compile(optimizer=optimizer)\n```\n\nThe last thing to setup before you start training is to provide a way to push your model to the Hub."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 85f38fe4-bfca-46b9-b31d-90b41973eebc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/question_answering", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"85f38fe4-bfca-46b9-b31d-90b41973eebc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/question_answering\", \"title\": \"Question answering\", \"tokens\": 3690, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0644af4b-e7db-4d46-a9bc-a8d41dc816fb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/question_answering\", \"title\": \"Question answering\", \"tokens\": 3690, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d40ef902670412e44c6eab15348ba95f1cc6da7be2d725be4f054d04caa135cd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9d01c23e-5f3d-435d-a8a5-ad24de457f22\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/question_answering\", \"title\": \"Question answering\", \"tokens\": 3690, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1fc4068755746ca17b32691c421cbeab8e89e7e5e5bd51a61b9d6077a1aab74e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"36e4976a-28e4-4014-9e34-046e8b66217d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5633c5115d0f00fe317080a6833cd9457e3c13cb02e8e80f4f639cb8966e7f99\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8781, \"end_char_idx\": 12050, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0644af4b-e7db-4d46-a9bc-a8d41dc816fb", "doc_id": "0644af4b-e7db-4d46-a9bc-a8d41dc816fb", "ref_doc_id": "0644af4b-e7db-4d46-a9bc-a8d41dc816fb", "chroma:document": "</Tip>\nTo finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:\n\n```py\n>>> from transformers import create_optimizer\n\n>>> batch_size = 16\n>>> num_epochs = 2\n>>> total_train_steps = (len(tokenized_squad[\"train\"]) // batch_size) * num_epochs\n>>> optimizer, schedule = create_optimizer(\n...     init_lr=2e-5,\n...     num_warmup_steps=0,\n...     num_train_steps=total_train_steps,\n... )\n```\n\nThen you can load DistilBERT with [`TFAutoModelForQuestionAnswering`]:\n\n```py\n>>> from transformers import TFAutoModelForQuestionAnswering\n\n>>> model = TFAutoModelForQuestionAnswering.from_pretrained(\"distilbert/distilbert-base-uncased\")\n```\n\nConvert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:\n\n```py\n>>> tf_train_set = model.prepare_tf_dataset(\n...     tokenized_squad[\"train\"],\n...     shuffle=True,\n...     batch_size=16,\n...     collate_fn=data_collator,\n... )\n\n>>> tf_validation_set = model.prepare_tf_dataset(\n...     tokenized_squad[\"test\"],\n...     shuffle=False,\n...     batch_size=16,\n...     collate_fn=data_collator,\n... )\n```\n\nConfigure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method):\n\n```py\n>>> import tensorflow as tf\n\n>>> model.compile(optimizer=optimizer)\n```\n\nThe last thing to setup before you start training is to provide a way to push your model to the Hub. This can be done by specifying where to push your model and tokenizer in the [`~transformers.PushToHubCallback`]:\n\n```py\n>>> from transformers.keras_callbacks import PushToHubCallback\n\n>>> callback = PushToHubCallback(\n...     output_dir=\"my_awesome_qa_model\",\n...     tokenizer=tokenizer,\n... )\n```\n\nFinally, you're ready to start training your model! Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callback to finetune the model:\n\n```py\n>>> model.fit(x=tf_train_set, validation_data=tf_validation_set, epochs=3, callbacks=[callback])\n```\nOnce training is completed, your model is automatically uploaded to the Hub so everyone can use it!\n</tf>\n</frameworkcontent>\n\n<Tip>\n\nFor a more in-depth example of how to finetune a model for question answering, take a look at the corresponding\n[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb)\nor [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).\n\n</Tip>\n\n## Evaluate\n\nEvaluation for question answering requires a significant amount of postprocessing. To avoid taking up too much of your time, this guide skips the evaluation step. The [`Trainer`] still calculates the evaluation loss during training so you're not completely in the dark about your model's performance.\n\nIf have more time and you're interested in how to evaluate your model for question answering, take a look at the [Question answering](https://huggingface.co/course/chapter7/7?fw=pt#post-processing) chapter from the \ud83e\udd17 Hugging Face Course!\n\n## Inference\n\nGreat, now that you've finetuned a model, you can use it for inference!"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 36e4976a-28e4-4014-9e34-046e8b66217d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/question_answering", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"36e4976a-28e4-4014-9e34-046e8b66217d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/question_answering\", \"title\": \"Question answering\", \"tokens\": 3690, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0644af4b-e7db-4d46-a9bc-a8d41dc816fb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/question_answering\", \"title\": \"Question answering\", \"tokens\": 3690, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d40ef902670412e44c6eab15348ba95f1cc6da7be2d725be4f054d04caa135cd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"85f38fe4-bfca-46b9-b31d-90b41973eebc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/question_answering\", \"title\": \"Question answering\", \"tokens\": 3690, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"11abd61fdc01ff27847243c95279f32c9fbc31357cf52d47a2f343543d6a3de2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f124629b-f5ea-4dcb-8e7e-ccc98c625370\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e746484cd609606399329f9f411df3abdbc236165789816fc3041c0b37e58bfe\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10603, \"end_char_idx\": 12435, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0644af4b-e7db-4d46-a9bc-a8d41dc816fb", "doc_id": "0644af4b-e7db-4d46-a9bc-a8d41dc816fb", "ref_doc_id": "0644af4b-e7db-4d46-a9bc-a8d41dc816fb", "chroma:document": "Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callback to finetune the model:\n\n```py\n>>> model.fit(x=tf_train_set, validation_data=tf_validation_set, epochs=3, callbacks=[callback])\n```\nOnce training is completed, your model is automatically uploaded to the Hub so everyone can use it!\n</tf>\n</frameworkcontent>\n\n<Tip>\n\nFor a more in-depth example of how to finetune a model for question answering, take a look at the corresponding\n[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb)\nor [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).\n\n</Tip>\n\n## Evaluate\n\nEvaluation for question answering requires a significant amount of postprocessing. To avoid taking up too much of your time, this guide skips the evaluation step. The [`Trainer`] still calculates the evaluation loss during training so you're not completely in the dark about your model's performance.\n\nIf have more time and you're interested in how to evaluate your model for question answering, take a look at the [Question answering](https://huggingface.co/course/chapter7/7?fw=pt#post-processing) chapter from the \ud83e\udd17 Hugging Face Course!\n\n## Inference\n\nGreat, now that you've finetuned a model, you can use it for inference!\n\nCome up with a question and some context you'd like the model to predict:\n\n```py\n>>> question = \"How many programming languages does BLOOM support?\"\n>>> context = \"BLOOM has 176 billion parameters and can generate text in 46 languages natural languages and 13 programming languages.\"\n```\n\nThe simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f124629b-f5ea-4dcb-8e7e-ccc98c625370
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/question_answering", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f124629b-f5ea-4dcb-8e7e-ccc98c625370\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/question_answering\", \"title\": \"Question answering\", \"tokens\": 3690, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0644af4b-e7db-4d46-a9bc-a8d41dc816fb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/question_answering\", \"title\": \"Question answering\", \"tokens\": 3690, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d40ef902670412e44c6eab15348ba95f1cc6da7be2d725be4f054d04caa135cd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"36e4976a-28e4-4014-9e34-046e8b66217d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/question_answering\", \"title\": \"Question answering\", \"tokens\": 3690, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"74132e8ba1519a1033c900f4a81915caeca6f5f82e4287f0fff9116c7b84ec6b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10992, \"end_char_idx\": 15021, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0644af4b-e7db-4d46-a9bc-a8d41dc816fb", "doc_id": "0644af4b-e7db-4d46-a9bc-a8d41dc816fb", "ref_doc_id": "0644af4b-e7db-4d46-a9bc-a8d41dc816fb", "chroma:document": "</tf>\n</frameworkcontent>\n\n<Tip>\n\nFor a more in-depth example of how to finetune a model for question answering, take a look at the corresponding\n[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb)\nor [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).\n\n</Tip>\n\n## Evaluate\n\nEvaluation for question answering requires a significant amount of postprocessing. To avoid taking up too much of your time, this guide skips the evaluation step. The [`Trainer`] still calculates the evaluation loss during training so you're not completely in the dark about your model's performance.\n\nIf have more time and you're interested in how to evaluate your model for question answering, take a look at the [Question answering](https://huggingface.co/course/chapter7/7?fw=pt#post-processing) chapter from the \ud83e\udd17 Hugging Face Course!\n\n## Inference\n\nGreat, now that you've finetuned a model, you can use it for inference!\n\nCome up with a question and some context you'd like the model to predict:\n\n```py\n>>> question = \"How many programming languages does BLOOM support?\"\n>>> context = \"BLOOM has 176 billion parameters and can generate text in 46 languages natural languages and 13 programming languages.\"\n```\n\nThe simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for question answering with your model, and pass your text to it:\n\n```py\n>>> from transformers import pipeline\n\n>>> question_answerer = pipeline(\"question-answering\", model=\"my_awesome_qa_model\")\n>>> question_answerer(question=question, context=context)\n{'score': 0.2058267742395401,\n 'start': 10,\n 'end': 95,\n 'answer': '176 billion parameters and can generate text in 46 languages natural languages and 13'}\n```\n\nYou can also manually replicate the results of the `pipeline` if you'd like:\n\n<frameworkcontent>\n<pt>\nTokenize the text and return PyTorch tensors:\n\n```py\n>>> from transformers import AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"my_awesome_qa_model\")\n>>> inputs = tokenizer(question, context, return_tensors=\"pt\")\n```\n\nPass your inputs to the model and return the `logits`:\n\n```py\n>>> import torch\n>>> from transformers import AutoModelForQuestionAnswering\n\n>>> model = AutoModelForQuestionAnswering.from_pretrained(\"my_awesome_qa_model\")\n>>> with torch.no_grad():\n...     outputs = model(**inputs)\n```\n\nGet the highest probability from the model output for the start and end positions:\n\n```py\n>>> answer_start_index = outputs.start_logits.argmax()\n>>> answer_end_index = outputs.end_logits.argmax()\n```\n\nDecode the predicted tokens to get the answer:\n\n```py\n>>> predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1]\n>>> tokenizer.decode(predict_answer_tokens)\n'176 billion parameters and can generate text in 46 languages natural languages and 13'\n```\n</pt>\n<tf>\nTokenize the text and return TensorFlow tensors:\n\n```py\n>>> from transformers import AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"my_awesome_qa_model\")\n>>> inputs = tokenizer(question, text, return_tensors=\"tf\")\n```\n\nPass your inputs to the model and return the `logits`:\n\n```py\n>>> from transformers import TFAutoModelForQuestionAnswering\n\n>>> model = TFAutoModelForQuestionAnswering.from_pretrained(\"my_awesome_qa_model\")\n>>> outputs = model(**inputs)\n```\n\nGet the highest probability from the model output for the start and end positions:\n\n```py\n>>> answer_start_index = int(tf.math.argmax(outputs.start_logits, axis=-1)[0])\n>>> answer_end_index = int(tf.math.argmax(outputs.end_logits, axis=-1)[0])\n```\n\nDecode the predicted tokens to get the answer:\n\n```py\n>>> predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1]\n>>> tokenizer.decode(predict_answer_tokens)\n'176 billion parameters and can generate text in 46 languages natural languages and 13'\n```\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 8910169d-f5d6-46fd-9564-3c17dcdb6468
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8910169d-f5d6-46fd-9564-3c17dcdb6468\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/masked_language_modeling\", \"title\": \"Masked language modeling\", \"tokens\": 4346, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"442ccb6b-a9d8-40fd-8399-d315c49ca8c2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/masked_language_modeling\", \"title\": \"Masked language modeling\", \"tokens\": 4346, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9b3e8180ef07283ca8562e6514bd7d0c93a63972b09cdd91afad2c57ce3af303\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"82d9ee8d-754f-43cf-9d73-73d647a29d85\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"786ad593ccb8b4a3c4f82237545ba3d694d5f0622a9cadbe988c5ea9379a0904\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3077, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "doc_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "ref_doc_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "chroma:document": "# Masked language modeling\n\n[[open-in-colab]]\n\n<Youtube id=\"mqElG5QJWUg\"/>\n\nMasked language modeling predicts a masked token in a sequence, and the model can attend to tokens bidirectionally. This\nmeans the model has full access to the tokens on the left and right. Masked language modeling is great for tasks that\nrequire a good contextual understanding of an entire sequence. BERT is an example of a masked language model.\n\nThis guide will show you how to:\n\n1. Finetune [DistilRoBERTa](https://huggingface.co/distilbert/distilroberta-base) on the [r/askscience](https://www.reddit.com/r/askscience/) subset of the [ELI5](https://huggingface.co/datasets/eli5) dataset.\n2. Use your finetuned model for inference.\n\n<Tip>\n\nTo see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/fill-mask)\n\n</Tip>\n\nBefore you begin, make sure you have all the necessary libraries installed:\n\n```bash\npip install transformers datasets evaluate\n```\n\nWe encourage you to log in to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to log in:\n\n```py\n>>> from huggingface_hub import notebook_login\n\n>>> notebook_login()\n```\n\n## Load ELI5 dataset\n\nStart by loading the first 5000 examples from the [ELI5-Category](https://huggingface.co/datasets/eli5_category) dataset with the \ud83e\udd17 Datasets library. This'll give you a chance to experiment and make sure everything works before spending more time training on the full dataset.\n\n```py\n>>> from datasets import load_dataset\n\n>>> eli5 = load_dataset(\"eli5_category\", split=\"train[:5000]\")\n```\n\nSplit the dataset's `train` split into a train and test set with the [`~datasets.Dataset.train_test_split`] method:\n\n```py\n>>> eli5 = eli5.train_test_split(test_size=0.2)\n```\n\nThen take a look at an example:\n\n```py\n>>> eli5[\"train\"][0]\n{'q_id': '7h191n',\n 'title': 'What does the tax bill that was passed today mean? How will it affect Americans in each tax bracket?',\n 'selftext': '',\n 'category': 'Economics',\n 'subreddit': 'explainlikeimfive',\n 'answers': {'a_id': ['dqnds8l', 'dqnd1jl', 'dqng3i1', 'dqnku5x'],\n  'text': [\"The tax bill is 500 pages long and there were a lot of changes still going on right to the end. It's not just an adjustment to the income tax brackets, it's a whole bunch of changes. As such there is no good answer to your question. The big take aways are: - Big reduction in corporate income tax rate will make large companies very happy. - Pass through rate change will make certain styles of business (law firms, hedge funds) extremely happy - Income tax changes are moderate, and are set to expire (though it's the kind of thing that might just always get re-applied without being made permanent) - People in high tax states (California, New York) lose out, and many of them will end up with their taxes raised.\",\n   'None yet. It has to be reconciled with a vastly different house bill and then passed again.',\n   'Also: does this apply to 2017 taxes? Or does it start with 2018 taxes?"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 82d9ee8d-754f-43cf-9d73-73d647a29d85
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"82d9ee8d-754f-43cf-9d73-73d647a29d85\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/masked_language_modeling\", \"title\": \"Masked language modeling\", \"tokens\": 4346, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"442ccb6b-a9d8-40fd-8399-d315c49ca8c2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/masked_language_modeling\", \"title\": \"Masked language modeling\", \"tokens\": 4346, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9b3e8180ef07283ca8562e6514bd7d0c93a63972b09cdd91afad2c57ce3af303\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8910169d-f5d6-46fd-9564-3c17dcdb6468\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/masked_language_modeling\", \"title\": \"Masked language modeling\", \"tokens\": 4346, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e1c45bb09fe4bbd82106fcf3f7f521d7146e05b09afb7d89b271ed15ee3f470d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"338043a9-6d87-4820-9e54-8e57088d6de9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fca688bf07822e2d57f1ac5e1f49a2ba723bb41da1cfb7d25f8a8a9b210a2db6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1978, \"end_char_idx\": 4884, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "doc_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "ref_doc_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "chroma:document": "How will it affect Americans in each tax bracket?',\n 'selftext': '',\n 'category': 'Economics',\n 'subreddit': 'explainlikeimfive',\n 'answers': {'a_id': ['dqnds8l', 'dqnd1jl', 'dqng3i1', 'dqnku5x'],\n  'text': [\"The tax bill is 500 pages long and there were a lot of changes still going on right to the end. It's not just an adjustment to the income tax brackets, it's a whole bunch of changes. As such there is no good answer to your question. The big take aways are: - Big reduction in corporate income tax rate will make large companies very happy. - Pass through rate change will make certain styles of business (law firms, hedge funds) extremely happy - Income tax changes are moderate, and are set to expire (though it's the kind of thing that might just always get re-applied without being made permanent) - People in high tax states (California, New York) lose out, and many of them will end up with their taxes raised.\",\n   'None yet. It has to be reconciled with a vastly different house bill and then passed again.',\n   'Also: does this apply to 2017 taxes? Or does it start with 2018 taxes?',\n   'This article explains both the House and senate bills, including the proposed changes to your income taxes based on your income level. URL_0'],\n  'score': [21, 19, 5, 3],\n  'text_urls': [[],\n   [],\n   [],\n   ['https://www.investopedia.com/news/trumps-tax-reform-what-can-be-done/']]},\n 'title_urls': ['url'],\n 'selftext_urls': ['url']}\n```\n\nWhile this may look like a lot, you're only really interested in the `text` field. What's cool about language modeling tasks is you don't need labels (also known as an unsupervised task) because the next word *is* the label.\n\n## Preprocess\n\n<Youtube id=\"8PmhEIXhBvI\"/>\n\nFor masked language modeling, the next step is to load a DistilRoBERTa tokenizer to process the `text` subfield:\n\n```py\n>>> from transformers import AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"distilbert/distilroberta-base\")\n```\n\nYou'll notice from the example above, the `text` field is actually nested inside `answers`. This means you'll need to extract the `text` subfield from its nested structure with the [`flatten`](https://huggingface.co/docs/datasets/process#flatten) method:\n\n```py\n>>> eli5 = eli5.flatten()\n>>> eli5[\"train\"][0]\n{'q_id': '7h191n',\n 'title': 'What does the tax bill that was passed today mean? How will it affect Americans in each tax bracket?',\n 'selftext': '',\n 'category': 'Economics',\n 'subreddit': 'explainlikeimfive',\n 'answers.a_id': ['dqnds8l', 'dqnd1jl', 'dqng3i1', 'dqnku5x'],\n 'answers.text': [\"The tax bill is 500 pages long and there were a lot of changes still going on right to the end. It's not just an adjustment to the income tax brackets, it's a whole bunch of changes. As such there is no good answer to your question. The big take aways are: - Big reduction in corporate income tax rate will make large companies very happy."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 338043a9-6d87-4820-9e54-8e57088d6de9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"338043a9-6d87-4820-9e54-8e57088d6de9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/masked_language_modeling\", \"title\": \"Masked language modeling\", \"tokens\": 4346, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"442ccb6b-a9d8-40fd-8399-d315c49ca8c2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/masked_language_modeling\", \"title\": \"Masked language modeling\", \"tokens\": 4346, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9b3e8180ef07283ca8562e6514bd7d0c93a63972b09cdd91afad2c57ce3af303\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"82d9ee8d-754f-43cf-9d73-73d647a29d85\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/masked_language_modeling\", \"title\": \"Masked language modeling\", \"tokens\": 4346, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9e93fc2b2bcdd25a760aa43c35151ad75215336f3d5ba4a40aa6d070cf76a691\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2040222c-6019-4906-a900-09b18bff1a30\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"89cb920f142f182cdbb73afb89187d9623f0dcaf139c3ecf7ea62308aad3444f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3508, \"end_char_idx\": 6534, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "doc_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "ref_doc_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "chroma:document": "What's cool about language modeling tasks is you don't need labels (also known as an unsupervised task) because the next word *is* the label.\n\n## Preprocess\n\n<Youtube id=\"8PmhEIXhBvI\"/>\n\nFor masked language modeling, the next step is to load a DistilRoBERTa tokenizer to process the `text` subfield:\n\n```py\n>>> from transformers import AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"distilbert/distilroberta-base\")\n```\n\nYou'll notice from the example above, the `text` field is actually nested inside `answers`. This means you'll need to extract the `text` subfield from its nested structure with the [`flatten`](https://huggingface.co/docs/datasets/process#flatten) method:\n\n```py\n>>> eli5 = eli5.flatten()\n>>> eli5[\"train\"][0]\n{'q_id': '7h191n',\n 'title': 'What does the tax bill that was passed today mean? How will it affect Americans in each tax bracket?',\n 'selftext': '',\n 'category': 'Economics',\n 'subreddit': 'explainlikeimfive',\n 'answers.a_id': ['dqnds8l', 'dqnd1jl', 'dqng3i1', 'dqnku5x'],\n 'answers.text': [\"The tax bill is 500 pages long and there were a lot of changes still going on right to the end. It's not just an adjustment to the income tax brackets, it's a whole bunch of changes. As such there is no good answer to your question. The big take aways are: - Big reduction in corporate income tax rate will make large companies very happy. - Pass through rate change will make certain styles of business (law firms, hedge funds) extremely happy - Income tax changes are moderate, and are set to expire (though it's the kind of thing that might just always get re-applied without being made permanent) - People in high tax states (California, New York) lose out, and many of them will end up with their taxes raised.\",\n  'None yet. It has to be reconciled with a vastly different house bill and then passed again.',\n  'Also: does this apply to 2017 taxes? Or does it start with 2018 taxes?',\n  'This article explains both the House and senate bills, including the proposed changes to your income taxes based on your income level. URL_0'],\n 'answers.score': [21, 19, 5, 3],\n 'answers.text_urls': [[],\n  [],\n  [],\n  ['https://www.investopedia.com/news/trumps-tax-reform-what-can-be-done/']],\n 'title_urls': ['url'],\n 'selftext_urls': ['url']}\n```\n\nEach subfield is now a separate column as indicated by the `answers` prefix, and the `text` field is a list now. Instead\nof tokenizing each sentence separately, convert the list to a string so you can jointly tokenize them.\n\nHere is a first preprocessing function to join the list of strings for each example and tokenize the result:\n\n```py\n>>> def preprocess_function(examples):\n...     return tokenizer([\" \".join(x) for x in examples[\"answers.text\"]])\n```\n\nTo apply this preprocessing function over the entire dataset, use the \ud83e\udd17 Datasets [`~datasets.Dataset.map`] method. You can speed up the `map` function by setting `batched=True` to process multiple elements of the dataset at once, and increasing the number of processes with `num_proc`."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 2040222c-6019-4906-a900-09b18bff1a30
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2040222c-6019-4906-a900-09b18bff1a30\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/masked_language_modeling\", \"title\": \"Masked language modeling\", \"tokens\": 4346, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"442ccb6b-a9d8-40fd-8399-d315c49ca8c2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/masked_language_modeling\", \"title\": \"Masked language modeling\", \"tokens\": 4346, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9b3e8180ef07283ca8562e6514bd7d0c93a63972b09cdd91afad2c57ce3af303\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"338043a9-6d87-4820-9e54-8e57088d6de9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/masked_language_modeling\", \"title\": \"Masked language modeling\", \"tokens\": 4346, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"520ef1063244a362f6e7ee0b8732aa6d4834e17a4239847a9e81e96404999c6b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7fe04d43-840d-4bb6-a2fe-9271da77a828\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ad41d22d1e30f42d94be711d57f6ce5cf1e72f8e7c9d2f9bc2e73c94392769c7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4885, \"end_char_idx\": 8091, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "doc_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "ref_doc_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "chroma:document": "- Pass through rate change will make certain styles of business (law firms, hedge funds) extremely happy - Income tax changes are moderate, and are set to expire (though it's the kind of thing that might just always get re-applied without being made permanent) - People in high tax states (California, New York) lose out, and many of them will end up with their taxes raised.\",\n  'None yet. It has to be reconciled with a vastly different house bill and then passed again.',\n  'Also: does this apply to 2017 taxes? Or does it start with 2018 taxes?',\n  'This article explains both the House and senate bills, including the proposed changes to your income taxes based on your income level. URL_0'],\n 'answers.score': [21, 19, 5, 3],\n 'answers.text_urls': [[],\n  [],\n  [],\n  ['https://www.investopedia.com/news/trumps-tax-reform-what-can-be-done/']],\n 'title_urls': ['url'],\n 'selftext_urls': ['url']}\n```\n\nEach subfield is now a separate column as indicated by the `answers` prefix, and the `text` field is a list now. Instead\nof tokenizing each sentence separately, convert the list to a string so you can jointly tokenize them.\n\nHere is a first preprocessing function to join the list of strings for each example and tokenize the result:\n\n```py\n>>> def preprocess_function(examples):\n...     return tokenizer([\" \".join(x) for x in examples[\"answers.text\"]])\n```\n\nTo apply this preprocessing function over the entire dataset, use the \ud83e\udd17 Datasets [`~datasets.Dataset.map`] method. You can speed up the `map` function by setting `batched=True` to process multiple elements of the dataset at once, and increasing the number of processes with `num_proc`. Remove any columns you don't need:\n\n```py\n>>> tokenized_eli5 = eli5.map(\n...     preprocess_function,\n...     batched=True,\n...     num_proc=4,\n...     remove_columns=eli5[\"train\"].column_names,\n... )\n```\n\nThis dataset contains the token sequences, but some of these are longer than the maximum input length for the model.\n\nYou can now use a second preprocessing function to\n- concatenate all the sequences\n- split the concatenated sequences into shorter chunks defined by `block_size`, which should be both shorter than the maximum input length and short enough for your GPU RAM. \n\n```py\n>>> block_size = 128\n\n\n>>> def group_texts(examples):\n...     # Concatenate all texts.\n...     concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}\n...     total_length = len(concatenated_examples[list(examples.keys())[0]])\n...     # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can\n...     # customize this part to your needs.\n...     if total_length >= block_size:\n...         total_length = (total_length // block_size) * block_size\n...     # Split by chunks of block_size.\n...     result = {\n...         k: [t[i : i + block_size] for i in range(0, total_length, block_size)]\n...         for k, t in concatenated_examples.items()\n...     }\n...     return result\n```\n\nApply the `group_texts` function over the entire dataset:\n\n```py\n>>> lm_dataset = tokenized_eli5.map(group_texts, batched=True, num_proc=4)\n```\n\nNow create a batch of examples using [`DataCollatorForLanguageModeling`]."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 7fe04d43-840d-4bb6-a2fe-9271da77a828
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7fe04d43-840d-4bb6-a2fe-9271da77a828\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/masked_language_modeling\", \"title\": \"Masked language modeling\", \"tokens\": 4346, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"442ccb6b-a9d8-40fd-8399-d315c49ca8c2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/masked_language_modeling\", \"title\": \"Masked language modeling\", \"tokens\": 4346, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9b3e8180ef07283ca8562e6514bd7d0c93a63972b09cdd91afad2c57ce3af303\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2040222c-6019-4906-a900-09b18bff1a30\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/masked_language_modeling\", \"title\": \"Masked language modeling\", \"tokens\": 4346, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"06ea08715d3c26618a18632a6ee32cd737b0233fac6912bb71b5cb6a0eb01fc6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"49cb58ba-702a-48d1-907a-5b7038b8b25b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"58f107a3b1885214cff3a2ab1f050d40d6500978da4f6cb166f8a6596466ec5f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6535, \"end_char_idx\": 9774, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "doc_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "ref_doc_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "chroma:document": "Remove any columns you don't need:\n\n```py\n>>> tokenized_eli5 = eli5.map(\n...     preprocess_function,\n...     batched=True,\n...     num_proc=4,\n...     remove_columns=eli5[\"train\"].column_names,\n... )\n```\n\nThis dataset contains the token sequences, but some of these are longer than the maximum input length for the model.\n\nYou can now use a second preprocessing function to\n- concatenate all the sequences\n- split the concatenated sequences into shorter chunks defined by `block_size`, which should be both shorter than the maximum input length and short enough for your GPU RAM. \n\n```py\n>>> block_size = 128\n\n\n>>> def group_texts(examples):\n...     # Concatenate all texts.\n...     concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}\n...     total_length = len(concatenated_examples[list(examples.keys())[0]])\n...     # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can\n...     # customize this part to your needs.\n...     if total_length >= block_size:\n...         total_length = (total_length // block_size) * block_size\n...     # Split by chunks of block_size.\n...     result = {\n...         k: [t[i : i + block_size] for i in range(0, total_length, block_size)]\n...         for k, t in concatenated_examples.items()\n...     }\n...     return result\n```\n\nApply the `group_texts` function over the entire dataset:\n\n```py\n>>> lm_dataset = tokenized_eli5.map(group_texts, batched=True, num_proc=4)\n```\n\nNow create a batch of examples using [`DataCollatorForLanguageModeling`]. It's more efficient to *dynamically pad* the sentences to the longest length in a batch during collation, instead of padding the whole dataset to the maximum length.\n\n<frameworkcontent>\n<pt>\n\nUse the end-of-sequence token as the padding token and specify `mlm_probability` to randomly mask tokens each time you iterate over the data:\n\n```py\n>>> from transformers import DataCollatorForLanguageModeling\n\n>>> tokenizer.pad_token = tokenizer.eos_token\n>>> data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm_probability=0.15)\n```\n</pt>\n<tf>\n\nUse the end-of-sequence token as the padding token and specify `mlm_probability` to randomly mask tokens each time you iterate over the data:\n\n```py\n>>> from transformers import DataCollatorForLanguageModeling\n\n>>> data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm_probability=0.15, return_tensors=\"tf\")\n```\n</tf>\n</frameworkcontent>\n\n## Train\n\n<frameworkcontent>\n<pt>\n<Tip>\n\nIf you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!\n\n</Tip>\n\nYou're ready to start training your model now! Load DistilRoBERTa with [`AutoModelForMaskedLM`]:\n\n```py\n>>> from transformers import AutoModelForMaskedLM\n\n>>> model = AutoModelForMaskedLM.from_pretrained(\"distilbert/distilroberta-base\")\n```\n\nAt this point, only three steps remain:\n\n1. Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 49cb58ba-702a-48d1-907a-5b7038b8b25b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"49cb58ba-702a-48d1-907a-5b7038b8b25b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/masked_language_modeling\", \"title\": \"Masked language modeling\", \"tokens\": 4346, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"442ccb6b-a9d8-40fd-8399-d315c49ca8c2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/masked_language_modeling\", \"title\": \"Masked language modeling\", \"tokens\": 4346, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9b3e8180ef07283ca8562e6514bd7d0c93a63972b09cdd91afad2c57ce3af303\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7fe04d43-840d-4bb6-a2fe-9271da77a828\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/masked_language_modeling\", \"title\": \"Masked language modeling\", \"tokens\": 4346, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"11f4d344258e22141fa192861d40d2a933e22915521886bc9dce98ff12bacd30\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c36fb013-51eb-4827-b65c-3c4571171dc3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e141bd8886cdbc40dfed409f0728ad146184a514e3a86ac4a89df08cc96f6911\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8259, \"end_char_idx\": 10989, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "doc_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "ref_doc_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "chroma:document": "<frameworkcontent>\n<pt>\n\nUse the end-of-sequence token as the padding token and specify `mlm_probability` to randomly mask tokens each time you iterate over the data:\n\n```py\n>>> from transformers import DataCollatorForLanguageModeling\n\n>>> tokenizer.pad_token = tokenizer.eos_token\n>>> data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm_probability=0.15)\n```\n</pt>\n<tf>\n\nUse the end-of-sequence token as the padding token and specify `mlm_probability` to randomly mask tokens each time you iterate over the data:\n\n```py\n>>> from transformers import DataCollatorForLanguageModeling\n\n>>> data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm_probability=0.15, return_tensors=\"tf\")\n```\n</tf>\n</frameworkcontent>\n\n## Train\n\n<frameworkcontent>\n<pt>\n<Tip>\n\nIf you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!\n\n</Tip>\n\nYou're ready to start training your model now! Load DistilRoBERTa with [`AutoModelForMaskedLM`]:\n\n```py\n>>> from transformers import AutoModelForMaskedLM\n\n>>> model = AutoModelForMaskedLM.from_pretrained(\"distilbert/distilroberta-base\")\n```\n\nAt this point, only three steps remain:\n\n1. Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model).\n2. Pass the training arguments to [`Trainer`] along with the model, datasets, and data collator.\n3. Call [`~Trainer.train`] to finetune your model.\n\n```py\n>>> training_args = TrainingArguments(\n...     output_dir=\"my_awesome_eli5_mlm_model\",\n...     eval_strategy=\"epoch\",\n...     learning_rate=2e-5,\n...     num_train_epochs=3,\n...     weight_decay=0.01,\n...     push_to_hub=True,\n... )\n\n>>> trainer = Trainer(\n...     model=model,\n...     args=training_args,\n...     train_dataset=lm_dataset[\"train\"],\n...     eval_dataset=lm_dataset[\"test\"],\n...     data_collator=data_collator,\n... )\n\n>>> trainer.train()\n```\n\nOnce training is completed, use the [`~transformers.Trainer.evaluate`] method to evaluate your model and get its perplexity:\n\n```py\n>>> import math\n\n>>> eval_results = trainer.evaluate()\n>>> print(f\"Perplexity: {math.exp(eval_results['eval_loss']):.2f}\")\nPerplexity: 8.76\n```\n\nThen share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:\n\n```py\n>>> trainer.push_to_hub()\n```\n</pt>\n<tf>\n<Tip>\n\nIf you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: c36fb013-51eb-4827-b65c-3c4571171dc3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c36fb013-51eb-4827-b65c-3c4571171dc3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/masked_language_modeling\", \"title\": \"Masked language modeling\", \"tokens\": 4346, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"442ccb6b-a9d8-40fd-8399-d315c49ca8c2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/masked_language_modeling\", \"title\": \"Masked language modeling\", \"tokens\": 4346, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9b3e8180ef07283ca8562e6514bd7d0c93a63972b09cdd91afad2c57ce3af303\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"49cb58ba-702a-48d1-907a-5b7038b8b25b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/masked_language_modeling\", \"title\": \"Masked language modeling\", \"tokens\": 4346, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4eb27ff2203f1946019b00a4d0fe987df7f164df6e4149002d14dd54feccc385\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2d1e09e1-dc06-4927-ba2e-835c53593956\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d9ad6c526f3cdbb490f3b38e09507f8418d81ae632ff0e2baf0144a461ca17c2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9497, \"end_char_idx\": 12288, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "doc_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "ref_doc_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "chroma:document": "Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model).\n2. Pass the training arguments to [`Trainer`] along with the model, datasets, and data collator.\n3. Call [`~Trainer.train`] to finetune your model.\n\n```py\n>>> training_args = TrainingArguments(\n...     output_dir=\"my_awesome_eli5_mlm_model\",\n...     eval_strategy=\"epoch\",\n...     learning_rate=2e-5,\n...     num_train_epochs=3,\n...     weight_decay=0.01,\n...     push_to_hub=True,\n... )\n\n>>> trainer = Trainer(\n...     model=model,\n...     args=training_args,\n...     train_dataset=lm_dataset[\"train\"],\n...     eval_dataset=lm_dataset[\"test\"],\n...     data_collator=data_collator,\n... )\n\n>>> trainer.train()\n```\n\nOnce training is completed, use the [`~transformers.Trainer.evaluate`] method to evaluate your model and get its perplexity:\n\n```py\n>>> import math\n\n>>> eval_results = trainer.evaluate()\n>>> print(f\"Perplexity: {math.exp(eval_results['eval_loss']):.2f}\")\nPerplexity: 8.76\n```\n\nThen share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:\n\n```py\n>>> trainer.push_to_hub()\n```\n</pt>\n<tf>\n<Tip>\n\nIf you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!\n\n</Tip>\nTo finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:\n\n```py\n>>> from transformers import create_optimizer, AdamWeightDecay\n\n>>> optimizer = AdamWeightDecay(learning_rate=2e-5, weight_decay_rate=0.01)\n```\n\nThen you can load DistilRoBERTa with [`TFAutoModelForMaskedLM`]:\n\n```py\n>>> from transformers import TFAutoModelForMaskedLM\n\n>>> model = TFAutoModelForMaskedLM.from_pretrained(\"distilbert/distilroberta-base\")\n```\n\nConvert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:\n\n```py\n>>> tf_train_set = model.prepare_tf_dataset(\n...     lm_dataset[\"train\"],\n...     shuffle=True,\n...     batch_size=16,\n...     collate_fn=data_collator,\n... )\n\n>>> tf_test_set = model.prepare_tf_dataset(\n...     lm_dataset[\"test\"],\n...     shuffle=False,\n...     batch_size=16,\n...     collate_fn=data_collator,\n... )\n```\n\nConfigure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method). Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:\n\n```py\n>>> import tensorflow as tf\n\n>>> model.compile(optimizer=optimizer)  # No loss argument!"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 2d1e09e1-dc06-4927-ba2e-835c53593956
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2d1e09e1-dc06-4927-ba2e-835c53593956\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/masked_language_modeling\", \"title\": \"Masked language modeling\", \"tokens\": 4346, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"442ccb6b-a9d8-40fd-8399-d315c49ca8c2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/masked_language_modeling\", \"title\": \"Masked language modeling\", \"tokens\": 4346, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9b3e8180ef07283ca8562e6514bd7d0c93a63972b09cdd91afad2c57ce3af303\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c36fb013-51eb-4827-b65c-3c4571171dc3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/masked_language_modeling\", \"title\": \"Masked language modeling\", \"tokens\": 4346, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"39ecce2cf6263810c28d222487d2df58024ccd1fd87aac3e6e3af5283258e53b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"be75df2c-ee7f-40cb-aa3a-0b3173a87033\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b693c2d971025a507c713a496013b00da3022f285966af65e4247c161de42532\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10991, \"end_char_idx\": 13911, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "doc_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "ref_doc_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "chroma:document": "</Tip>\nTo finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:\n\n```py\n>>> from transformers import create_optimizer, AdamWeightDecay\n\n>>> optimizer = AdamWeightDecay(learning_rate=2e-5, weight_decay_rate=0.01)\n```\n\nThen you can load DistilRoBERTa with [`TFAutoModelForMaskedLM`]:\n\n```py\n>>> from transformers import TFAutoModelForMaskedLM\n\n>>> model = TFAutoModelForMaskedLM.from_pretrained(\"distilbert/distilroberta-base\")\n```\n\nConvert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:\n\n```py\n>>> tf_train_set = model.prepare_tf_dataset(\n...     lm_dataset[\"train\"],\n...     shuffle=True,\n...     batch_size=16,\n...     collate_fn=data_collator,\n... )\n\n>>> tf_test_set = model.prepare_tf_dataset(\n...     lm_dataset[\"test\"],\n...     shuffle=False,\n...     batch_size=16,\n...     collate_fn=data_collator,\n... )\n```\n\nConfigure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method). Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:\n\n```py\n>>> import tensorflow as tf\n\n>>> model.compile(optimizer=optimizer)  # No loss argument!\n```\n\nThis can be done by specifying where to push your model and tokenizer in the [`~transformers.PushToHubCallback`]:\n\n```py\n>>> from transformers.keras_callbacks import PushToHubCallback\n\n>>> callback = PushToHubCallback(\n...     output_dir=\"my_awesome_eli5_mlm_model\",\n...     tokenizer=tokenizer,\n... )\n```\n\nFinally, you're ready to start training your model! Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callback to finetune the model:\n\n```py\n>>> model.fit(x=tf_train_set, validation_data=tf_test_set, epochs=3, callbacks=[callback])\n```\n\nOnce training is completed, your model is automatically uploaded to the Hub so everyone can use it!\n</tf>\n</frameworkcontent>\n\n<Tip>\n\nFor a more in-depth example of how to finetune a model for masked language modeling, take a look at the corresponding\n[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb)\nor [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n\n</Tip>\n\n## Inference\n\nGreat, now that you've finetuned a model, you can use it for inference!\n\nCome up with some text you'd like the model to fill in the blank with, and use the special `<mask>` token to indicate the blank:\n\n```py\n>>> text = \"The Milky Way is a <mask> galaxy.\"\n```\n\nThe simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for fill-mask with your model, and pass your text to it."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: be75df2c-ee7f-40cb-aa3a-0b3173a87033
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"be75df2c-ee7f-40cb-aa3a-0b3173a87033\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/masked_language_modeling\", \"title\": \"Masked language modeling\", \"tokens\": 4346, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"442ccb6b-a9d8-40fd-8399-d315c49ca8c2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/masked_language_modeling\", \"title\": \"Masked language modeling\", \"tokens\": 4346, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9b3e8180ef07283ca8562e6514bd7d0c93a63972b09cdd91afad2c57ce3af303\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2d1e09e1-dc06-4927-ba2e-835c53593956\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/masked_language_modeling\", \"title\": \"Masked language modeling\", \"tokens\": 4346, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0a34f9edb7eee16ee69a5ec1a0892d3d3a70de22ddc2dde1f343fa9af2eda5e4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"84408b48-cb58-41ad-9410-d9788ed7e543\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5b3617d5ea882657a699cfcbadd744546e96406d02d90e3b5947fb7a308b50c6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12289, \"end_char_idx\": 14630, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "doc_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "ref_doc_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "chroma:document": "```\n\nThis can be done by specifying where to push your model and tokenizer in the [`~transformers.PushToHubCallback`]:\n\n```py\n>>> from transformers.keras_callbacks import PushToHubCallback\n\n>>> callback = PushToHubCallback(\n...     output_dir=\"my_awesome_eli5_mlm_model\",\n...     tokenizer=tokenizer,\n... )\n```\n\nFinally, you're ready to start training your model! Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callback to finetune the model:\n\n```py\n>>> model.fit(x=tf_train_set, validation_data=tf_test_set, epochs=3, callbacks=[callback])\n```\n\nOnce training is completed, your model is automatically uploaded to the Hub so everyone can use it!\n</tf>\n</frameworkcontent>\n\n<Tip>\n\nFor a more in-depth example of how to finetune a model for masked language modeling, take a look at the corresponding\n[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb)\nor [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n\n</Tip>\n\n## Inference\n\nGreat, now that you've finetuned a model, you can use it for inference!\n\nCome up with some text you'd like the model to fill in the blank with, and use the special `<mask>` token to indicate the blank:\n\n```py\n>>> text = \"The Milky Way is a <mask> galaxy.\"\n```\n\nThe simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for fill-mask with your model, and pass your text to it. If you like, you can use the `top_k` parameter to specify how many predictions to return:\n\n```py\n>>> from transformers import pipeline\n\n>>> mask_filler = pipeline(\"fill-mask\", \"username/my_awesome_eli5_mlm_model\")\n>>> mask_filler(text, top_k=3)\n[{'score': 0.5150994658470154,\n  'token': 21300,\n  'token_str': ' spiral',\n  'sequence': 'The Milky Way is a spiral galaxy.'},\n {'score': 0.07087188959121704,\n  'token': 2232,\n  'token_str': ' massive',\n  'sequence': 'The Milky Way is a massive galaxy.'},\n {'score': 0.06434620916843414,\n  'token': 650,\n  'token_str': ' small',\n  'sequence': 'The Milky Way is a small galaxy.'}]\n```\n\n<frameworkcontent>\n<pt>\nTokenize the text and return the `input_ids` as PyTorch tensors."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 84408b48-cb58-41ad-9410-d9788ed7e543
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"84408b48-cb58-41ad-9410-d9788ed7e543\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/masked_language_modeling\", \"title\": \"Masked language modeling\", \"tokens\": 4346, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"442ccb6b-a9d8-40fd-8399-d315c49ca8c2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/masked_language_modeling\", \"title\": \"Masked language modeling\", \"tokens\": 4346, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9b3e8180ef07283ca8562e6514bd7d0c93a63972b09cdd91afad2c57ce3af303\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"be75df2c-ee7f-40cb-aa3a-0b3173a87033\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/masked_language_modeling\", \"title\": \"Masked language modeling\", \"tokens\": 4346, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"56dad74a8ba1b3ae1e4e515a550a50675e3174097a1fea659dfc85e8d671c0f3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"99c361fe-9524-4f73-86d8-7c42c8f17283\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"829a88a74a4b39ff287a2f9e7093138bfaf6a7ff78359e5bf2132e5653f0584d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13452, \"end_char_idx\": 15751, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "doc_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "ref_doc_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "chroma:document": "</Tip>\n\n## Inference\n\nGreat, now that you've finetuned a model, you can use it for inference!\n\nCome up with some text you'd like the model to fill in the blank with, and use the special `<mask>` token to indicate the blank:\n\n```py\n>>> text = \"The Milky Way is a <mask> galaxy.\"\n```\n\nThe simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for fill-mask with your model, and pass your text to it. If you like, you can use the `top_k` parameter to specify how many predictions to return:\n\n```py\n>>> from transformers import pipeline\n\n>>> mask_filler = pipeline(\"fill-mask\", \"username/my_awesome_eli5_mlm_model\")\n>>> mask_filler(text, top_k=3)\n[{'score': 0.5150994658470154,\n  'token': 21300,\n  'token_str': ' spiral',\n  'sequence': 'The Milky Way is a spiral galaxy.'},\n {'score': 0.07087188959121704,\n  'token': 2232,\n  'token_str': ' massive',\n  'sequence': 'The Milky Way is a massive galaxy.'},\n {'score': 0.06434620916843414,\n  'token': 650,\n  'token_str': ' small',\n  'sequence': 'The Milky Way is a small galaxy.'}]\n```\n\n<frameworkcontent>\n<pt>\nTokenize the text and return the `input_ids` as PyTorch tensors. You'll also need to specify the position of the `<mask>` token:\n\n```py\n>>> from transformers import AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"username/my_awesome_eli5_mlm_model\")\n>>> inputs = tokenizer(text, return_tensors=\"pt\")\n>>> mask_token_index = torch.where(inputs[\"input_ids\"] == tokenizer.mask_token_id)[1]\n```\n\nPass your inputs to the model and return the `logits` of the masked token:\n\n```py\n>>> from transformers import AutoModelForMaskedLM\n\n>>> model = AutoModelForMaskedLM.from_pretrained(\"username/my_awesome_eli5_mlm_model\")\n>>> logits = model(**inputs).logits\n>>> mask_token_logits = logits[0, mask_token_index, :]\n```\n\nThen return the three masked tokens with the highest probability and print them out:\n\n```py\n>>> top_3_tokens = torch.topk(mask_token_logits, 3, dim=1).indices[0].tolist()\n\n>>> for token in top_3_tokens:\n...     print(text.replace(tokenizer.mask_token, tokenizer.decode([token])))\nThe Milky Way is a spiral galaxy.\nThe Milky Way is a massive galaxy.\nThe Milky Way is a small galaxy.\n```\n</pt>\n<tf>\nTokenize the text and return the `input_ids` as TensorFlow tensors."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 99c361fe-9524-4f73-86d8-7c42c8f17283
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"99c361fe-9524-4f73-86d8-7c42c8f17283\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/masked_language_modeling\", \"title\": \"Masked language modeling\", \"tokens\": 4346, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"442ccb6b-a9d8-40fd-8399-d315c49ca8c2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/masked_language_modeling\", \"title\": \"Masked language modeling\", \"tokens\": 4346, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9b3e8180ef07283ca8562e6514bd7d0c93a63972b09cdd91afad2c57ce3af303\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"84408b48-cb58-41ad-9410-d9788ed7e543\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/masked_language_modeling\", \"title\": \"Masked language modeling\", \"tokens\": 4346, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a25674e94f355fe8a34891f6ced1ef90472df4741160a83d9b0e5c8358257243\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14280, \"end_char_idx\": 16815, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "doc_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "ref_doc_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "chroma:document": "'},\n {'score': 0.07087188959121704,\n  'token': 2232,\n  'token_str': ' massive',\n  'sequence': 'The Milky Way is a massive galaxy.'},\n {'score': 0.06434620916843414,\n  'token': 650,\n  'token_str': ' small',\n  'sequence': 'The Milky Way is a small galaxy.'}]\n```\n\n<frameworkcontent>\n<pt>\nTokenize the text and return the `input_ids` as PyTorch tensors. You'll also need to specify the position of the `<mask>` token:\n\n```py\n>>> from transformers import AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"username/my_awesome_eli5_mlm_model\")\n>>> inputs = tokenizer(text, return_tensors=\"pt\")\n>>> mask_token_index = torch.where(inputs[\"input_ids\"] == tokenizer.mask_token_id)[1]\n```\n\nPass your inputs to the model and return the `logits` of the masked token:\n\n```py\n>>> from transformers import AutoModelForMaskedLM\n\n>>> model = AutoModelForMaskedLM.from_pretrained(\"username/my_awesome_eli5_mlm_model\")\n>>> logits = model(**inputs).logits\n>>> mask_token_logits = logits[0, mask_token_index, :]\n```\n\nThen return the three masked tokens with the highest probability and print them out:\n\n```py\n>>> top_3_tokens = torch.topk(mask_token_logits, 3, dim=1).indices[0].tolist()\n\n>>> for token in top_3_tokens:\n...     print(text.replace(tokenizer.mask_token, tokenizer.decode([token])))\nThe Milky Way is a spiral galaxy.\nThe Milky Way is a massive galaxy.\nThe Milky Way is a small galaxy.\n```\n</pt>\n<tf>\nTokenize the text and return the `input_ids` as TensorFlow tensors. You'll also need to specify the position of the `<mask>` token:\n\n```py\n>>> from transformers import AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"username/my_awesome_eli5_mlm_model\")\n>>> inputs = tokenizer(text, return_tensors=\"tf\")\n>>> mask_token_index = tf.where(inputs[\"input_ids\"] == tokenizer.mask_token_id)[0, 1]\n```\n\nPass your inputs to the model and return the `logits` of the masked token:\n\n```py\n>>> from transformers import TFAutoModelForMaskedLM\n\n>>> model = TFAutoModelForMaskedLM.from_pretrained(\"username/my_awesome_eli5_mlm_model\")\n>>> logits = model(**inputs).logits\n>>> mask_token_logits = logits[0, mask_token_index, :]\n```\n\nThen return the three masked tokens with the highest probability and print them out:\n\n```py\n>>> top_3_tokens = tf.math.top_k(mask_token_logits, 3).indices.numpy()\n\n>>> for token in top_3_tokens:\n...     print(text.replace(tokenizer.mask_token, tokenizer.decode([token])))\nThe Milky Way is a spiral galaxy.\nThe Milky Way is a massive galaxy.\nThe Milky Way is a small galaxy.\n```\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 904ebbbe-9880-4148-97eb-ddd3aacfbbc8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/image_to_image", "title": "Image-to-Image Task Guide", "tokens": 1389, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"904ebbbe-9880-4148-97eb-ddd3aacfbbc8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_to_image\", \"title\": \"Image-to-Image Task Guide\", \"tokens\": 1389, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"05348b65-cb37-495e-bb92-f3c0159872a1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_to_image\", \"title\": \"Image-to-Image Task Guide\", \"tokens\": 1389, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c18d02b9c0db9f863d39da18348e899e9067197b11d79e1493fcf4c633a0694d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"eeebfafe-72c8-4121-9e06-69d22a776728\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1ff1cef968511d48382d6cbe7ee3c4b063282962845720e40a5569c0e026c6ed\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2778, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "05348b65-cb37-495e-bb92-f3c0159872a1", "doc_id": "05348b65-cb37-495e-bb92-f3c0159872a1", "ref_doc_id": "05348b65-cb37-495e-bb92-f3c0159872a1", "chroma:document": "# Image-to-Image Task Guide\n\n[[open-in-colab]]\n\nImage-to-Image task is the task where an application receives an image and outputs another image. This has various subtasks, including image enhancement (super resolution, low light enhancement, deraining and so on), image inpainting, and more. \n\nThis guide will show you how to:\n- Use an image-to-image pipeline for super resolution task,\n- Run image-to-image models for same task without a pipeline.\n\nNote that as of the time this guide is released, `image-to-image` pipeline only supports super resolution task.\n\nLet's begin by installing the necessary libraries.\n\n```bash\npip install transformers\n```\n\nWe can now initialize the pipeline with a [Swin2SR model](https://huggingface.co/caidas/swin2SR-lightweight-x2-64). We can then infer with the pipeline by calling it with an image. As of now, only [Swin2SR models](https://huggingface.co/models?sort=trending&search=swin2sr) are supported in this pipeline. \n\n```python\nfrom transformers import pipeline\n\ndevice = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\npipe = pipeline(task=\"image-to-image\", model=\"caidas/swin2SR-lightweight-x2-64\", device=device)\n```\n\nNow, let's load an image.\n\n```python\nfrom PIL import Image\nimport requests\n\nurl = \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/cat.jpg\"\nimage = Image.open(requests.get(url, stream=True).raw)\n\nprint(image.size)\n```\n```bash\n# (532, 432)\n```\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/cat.jpg\" alt=\"Photo of a cat\"/>\n</div>\n\nWe can now do inference with the pipeline. We will get an upscaled version of the cat image. \n\n```python\nupscaled = pipe(image)\nprint(upscaled.size)\n```\n```bash\n# (1072, 880)\n```\n\nIf you wish to do inference yourself with no pipeline, you can use the `Swin2SRForImageSuperResolution` and `Swin2SRImageProcessor` classes of transformers. We will use the same model checkpoint for this. Let's initialize the model and the processor.\n\n```python\nfrom transformers import Swin2SRForImageSuperResolution, Swin2SRImageProcessor \n\nmodel = Swin2SRForImageSuperResolution.from_pretrained(\"caidas/swin2SR-lightweight-x2-64\").to(device)\nprocessor = Swin2SRImageProcessor(\"caidas/swin2SR-lightweight-x2-64\")\n```\n\n`pipeline` abstracts away the preprocessing and postprocessing steps that we have to do ourselves, so let's preprocess the image. We will pass the image to the processor and then move the pixel values to GPU. \n\n```python\npixel_values = processor(image, return_tensors=\"pt\").pixel_values\nprint(pixel_values.shape)\n\npixel_values = pixel_values.to(device)\n```\n\nWe can now infer the image by passing pixel values to the model."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: eeebfafe-72c8-4121-9e06-69d22a776728
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/image_to_image", "title": "Image-to-Image Task Guide", "tokens": 1389, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"eeebfafe-72c8-4121-9e06-69d22a776728\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_to_image\", \"title\": \"Image-to-Image Task Guide\", \"tokens\": 1389, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"05348b65-cb37-495e-bb92-f3c0159872a1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_to_image\", \"title\": \"Image-to-Image Task Guide\", \"tokens\": 1389, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c18d02b9c0db9f863d39da18348e899e9067197b11d79e1493fcf4c633a0694d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"904ebbbe-9880-4148-97eb-ddd3aacfbbc8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_to_image\", \"title\": \"Image-to-Image Task Guide\", \"tokens\": 1389, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e84eba29c003a8326201ca7e50761ad69640de84033b84f9cebd7b5a060d3dec\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b34f1a36-0f13-4c26-9283-7ab7a25c3396\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c28a6859c39eb1a4971ec2bd86ffee8bb3aa9bd5d60ec072f553d0346e80214a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1205, \"end_char_idx\": 3746, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "05348b65-cb37-495e-bb92-f3c0159872a1", "doc_id": "05348b65-cb37-495e-bb92-f3c0159872a1", "ref_doc_id": "05348b65-cb37-495e-bb92-f3c0159872a1", "chroma:document": "```python\nfrom PIL import Image\nimport requests\n\nurl = \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/cat.jpg\"\nimage = Image.open(requests.get(url, stream=True).raw)\n\nprint(image.size)\n```\n```bash\n# (532, 432)\n```\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/cat.jpg\" alt=\"Photo of a cat\"/>\n</div>\n\nWe can now do inference with the pipeline. We will get an upscaled version of the cat image. \n\n```python\nupscaled = pipe(image)\nprint(upscaled.size)\n```\n```bash\n# (1072, 880)\n```\n\nIf you wish to do inference yourself with no pipeline, you can use the `Swin2SRForImageSuperResolution` and `Swin2SRImageProcessor` classes of transformers. We will use the same model checkpoint for this. Let's initialize the model and the processor.\n\n```python\nfrom transformers import Swin2SRForImageSuperResolution, Swin2SRImageProcessor \n\nmodel = Swin2SRForImageSuperResolution.from_pretrained(\"caidas/swin2SR-lightweight-x2-64\").to(device)\nprocessor = Swin2SRImageProcessor(\"caidas/swin2SR-lightweight-x2-64\")\n```\n\n`pipeline` abstracts away the preprocessing and postprocessing steps that we have to do ourselves, so let's preprocess the image. We will pass the image to the processor and then move the pixel values to GPU. \n\n```python\npixel_values = processor(image, return_tensors=\"pt\").pixel_values\nprint(pixel_values.shape)\n\npixel_values = pixel_values.to(device)\n```\n\nWe can now infer the image by passing pixel values to the model.\n\n```python\nimport torch\n\nwith torch.no_grad():\n  outputs = model(pixel_values)\n```\nOutput is an object of type `ImageSuperResolutionOutput` that looks like below \ud83d\udc47 \n\n```\n(loss=None, reconstruction=tensor([[[[0.8270, 0.8269, 0.8275,  ..., 0.7463, 0.7446, 0.7453],\n          [0.8287, 0.8278, 0.8283,  ..., 0.7451, 0.7448, 0.7457],\n          [0.8280, 0.8273, 0.8269,  ..., 0.7447, 0.7446, 0.7452],\n          ...,\n          [0.5923, 0.5933, 0.5924,  ..., 0.0697, 0.0695, 0.0706],\n          [0.5926, 0.5932, 0.5926,  ..., 0.0673, 0.0687, 0.0705],\n          [0.5927, 0.5914, 0.5922,  ..., 0.0664, 0.0694, 0.0718]]]],\n       device='cuda:0'), hidden_states=None, attentions=None)\n```\nWe need to get the `reconstruction` and post-process it for visualization. Let's see how it looks like.\n\n```python\noutputs.reconstruction.data.shape\n# torch.Size([1, 3, 880, 1072])\n```\n\nWe need to squeeze the output and get rid of axis 0, clip the values, then convert it to be numpy float."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: b34f1a36-0f13-4c26-9283-7ab7a25c3396
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/image_to_image", "title": "Image-to-Image Task Guide", "tokens": 1389, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b34f1a36-0f13-4c26-9283-7ab7a25c3396\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_to_image\", \"title\": \"Image-to-Image Task Guide\", \"tokens\": 1389, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"05348b65-cb37-495e-bb92-f3c0159872a1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_to_image\", \"title\": \"Image-to-Image Task Guide\", \"tokens\": 1389, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c18d02b9c0db9f863d39da18348e899e9067197b11d79e1493fcf4c633a0694d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"eeebfafe-72c8-4121-9e06-69d22a776728\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/image_to_image\", \"title\": \"Image-to-Image Task Guide\", \"tokens\": 1389, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3391ce1631deab628406cd98a52706fb957902e2f614dbaf06464bb4de071111\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2780, \"end_char_idx\": 4405, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "05348b65-cb37-495e-bb92-f3c0159872a1", "doc_id": "05348b65-cb37-495e-bb92-f3c0159872a1", "ref_doc_id": "05348b65-cb37-495e-bb92-f3c0159872a1", "chroma:document": "```python\nimport torch\n\nwith torch.no_grad():\n  outputs = model(pixel_values)\n```\nOutput is an object of type `ImageSuperResolutionOutput` that looks like below \ud83d\udc47 \n\n```\n(loss=None, reconstruction=tensor([[[[0.8270, 0.8269, 0.8275,  ..., 0.7463, 0.7446, 0.7453],\n          [0.8287, 0.8278, 0.8283,  ..., 0.7451, 0.7448, 0.7457],\n          [0.8280, 0.8273, 0.8269,  ..., 0.7447, 0.7446, 0.7452],\n          ...,\n          [0.5923, 0.5933, 0.5924,  ..., 0.0697, 0.0695, 0.0706],\n          [0.5926, 0.5932, 0.5926,  ..., 0.0673, 0.0687, 0.0705],\n          [0.5927, 0.5914, 0.5922,  ..., 0.0664, 0.0694, 0.0718]]]],\n       device='cuda:0'), hidden_states=None, attentions=None)\n```\nWe need to get the `reconstruction` and post-process it for visualization. Let's see how it looks like.\n\n```python\noutputs.reconstruction.data.shape\n# torch.Size([1, 3, 880, 1072])\n```\n\nWe need to squeeze the output and get rid of axis 0, clip the values, then convert it to be numpy float. Then we will arrange axes to have the shape [1072, 880], and finally, bring the output back to range [0, 255].\n\n```python\nimport numpy as np\n\n# squeeze, take to CPU and clip the values\noutput = outputs.reconstruction.data.squeeze().cpu().clamp_(0, 1).numpy()\n# rearrange the axes\noutput = np.moveaxis(output, source=0, destination=-1)\n# bring values back to pixel values range\noutput = (output * 255.0).round().astype(np.uint8)\nImage.fromarray(output)\n```\n<div class=\"flex justify-center\">\n     <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/cat_upscaled.png\" alt=\"Upscaled photo of a cat\"/>\n</div>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 40465b30-8822-4844-b953-69da0acd1464
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"40465b30-8822-4844-b953-69da0acd1464\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"161d7d29-43e1-4306-bfa1-36cd04a2474d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3ed0d14d3467b190aad4d7f8c3c47f3f942dcbd36a3398099319a304d5bc33f6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a22188ed-63d2-4995-88a9-b712e114fba8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"243206efbd7b313c37e7fbee6b3f6512683901eef66261c089b61a728730135d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2872, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "doc_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "ref_doc_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "chroma:document": "# Video classification\n\n[[open-in-colab]]\n\nVideo classification is the task of assigning a label or class to an entire video. Videos are expected to have only one class for each video. Video classification models take a video as input and return a prediction about which class the video belongs to. These models can be used to categorize what a video is all about. A real-world application of video classification is action / activity recognition, which is useful for fitness applications. It is also helpful for vision-impaired individuals, especially when they are commuting.\n\nThis guide will show you how to:\n\n1. Fine-tune [VideoMAE](https://huggingface.co/docs/transformers/main/en/model_doc/videomae) on a subset of the [UCF101](https://www.crcv.ucf.edu/data/UCF101.php) dataset.\n2. Use your fine-tuned model for inference.\n\n<Tip>\n\nTo see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/video-classification).\n\n</Tip>\n\nBefore you begin, make sure you have all the necessary libraries installed:\n\n```bash\npip install -q pytorchvideo transformers evaluate\n```\n\nYou will use [PyTorchVideo](https://pytorchvideo.org/) (dubbed `pytorchvideo`) to process and prepare the videos.\n\nWe encourage you to log in to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to log in:\n\n```py\n>>> from huggingface_hub import notebook_login\n\n>>> notebook_login()\n```\n\n## Load UCF101 dataset\n\nStart by loading a subset of the [UCF-101 dataset](https://www.crcv.ucf.edu/data/UCF101.php). This will give you a chance to experiment and make sure everything works before spending more time training on the full dataset.\n\n```py\n>>> from huggingface_hub import hf_hub_download\n\n>>> hf_dataset_identifier = \"sayakpaul/ucf101-subset\"\n>>> filename = \"UCF101_subset.tar.gz\"\n>>> file_path = hf_hub_download(repo_id=hf_dataset_identifier, filename=filename, repo_type=\"dataset\")\n```\n\nAfter the subset has been downloaded, you need to extract the compressed archive:\n\n```py \n>>> import tarfile\n\n>>> with tarfile.open(file_path) as t:\n...      t.extractall(\".\")\n```\n\nAt a high level, the dataset is organized like so:\n\n```bash\nUCF101_subset/\n    train/\n        BandMarching/\n            video_1.mp4\n            video_2.mp4\n            ...\n        Archery\n            video_1.mp4\n            video_2.mp4\n            ...\n        ...\n    val/\n        BandMarching/\n            video_1.mp4\n            video_2.mp4\n            ...\n        Archery\n            video_1.mp4\n            video_2.mp4\n            ...\n        ...\n    test/\n        BandMarching/\n            video_1.mp4\n            video_2.mp4\n            ...\n        Archery\n            video_1.mp4\n            video_2.mp4\n            ...\n        ...\n```\n\nYou can then count the number of total videos."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a22188ed-63d2-4995-88a9-b712e114fba8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a22188ed-63d2-4995-88a9-b712e114fba8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"161d7d29-43e1-4306-bfa1-36cd04a2474d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3ed0d14d3467b190aad4d7f8c3c47f3f942dcbd36a3398099319a304d5bc33f6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"40465b30-8822-4844-b953-69da0acd1464\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2b2f1221b39fa7f1a43310d7c4e67edf7d1c39dbedc5b18702b1e759a8514115\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e28a0280-33f2-4b85-af5b-a6798054b85a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"966439955c865a87bec8253a1b82354754a78595997bbe9126fd7f9877ef072a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1259, \"end_char_idx\": 4199, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "doc_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "ref_doc_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "chroma:document": "We encourage you to log in to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to log in:\n\n```py\n>>> from huggingface_hub import notebook_login\n\n>>> notebook_login()\n```\n\n## Load UCF101 dataset\n\nStart by loading a subset of the [UCF-101 dataset](https://www.crcv.ucf.edu/data/UCF101.php). This will give you a chance to experiment and make sure everything works before spending more time training on the full dataset.\n\n```py\n>>> from huggingface_hub import hf_hub_download\n\n>>> hf_dataset_identifier = \"sayakpaul/ucf101-subset\"\n>>> filename = \"UCF101_subset.tar.gz\"\n>>> file_path = hf_hub_download(repo_id=hf_dataset_identifier, filename=filename, repo_type=\"dataset\")\n```\n\nAfter the subset has been downloaded, you need to extract the compressed archive:\n\n```py \n>>> import tarfile\n\n>>> with tarfile.open(file_path) as t:\n...      t.extractall(\".\")\n```\n\nAt a high level, the dataset is organized like so:\n\n```bash\nUCF101_subset/\n    train/\n        BandMarching/\n            video_1.mp4\n            video_2.mp4\n            ...\n        Archery\n            video_1.mp4\n            video_2.mp4\n            ...\n        ...\n    val/\n        BandMarching/\n            video_1.mp4\n            video_2.mp4\n            ...\n        Archery\n            video_1.mp4\n            video_2.mp4\n            ...\n        ...\n    test/\n        BandMarching/\n            video_1.mp4\n            video_2.mp4\n            ...\n        Archery\n            video_1.mp4\n            video_2.mp4\n            ...\n        ...\n```\n\nYou can then count the number of total videos.\n\n```py \n>>> import pathlib\n>>> dataset_root_path = \"UCF101_subset\"\n>>> dataset_root_path = pathlib.Path(dataset_root_path)\n```\n\n```py \n>>> video_count_train = len(list(dataset_root_path.glob(\"train/*/*.avi\")))\n>>> video_count_val = len(list(dataset_root_path.glob(\"val/*/*.avi\")))\n>>> video_count_test = len(list(dataset_root_path.glob(\"test/*/*.avi\")))\n>>> video_total = video_count_train + video_count_val + video_count_test\n>>> print(f\"Total videos: {video_total}\")\n```\n\n```py \n>>> all_video_file_paths = (\n...     list(dataset_root_path.glob(\"train/*/*.avi\"))\n...     + list(dataset_root_path.glob(\"val/*/*.avi\"))\n...     + list(dataset_root_path.glob(\"test/*/*.avi\"))\n...  )\n>>> all_video_file_paths[:5]\n```\n\nThe (`sorted`) video paths appear like so:\n\n```bash\n...\n'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g07_c04.avi',\n'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g07_c06.avi',\n'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g08_c01.avi',\n'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g09_c02.avi',\n'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g09_c06.avi'\n...\n```\n\nYou will notice that there are video clips belonging to the same group / scene where group is denoted by `g` in the video file paths. `v_ApplyEyeMakeup_g07_c04.avi` and `v_ApplyEyeMakeup_g07_c06.avi`, for example."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e28a0280-33f2-4b85-af5b-a6798054b85a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e28a0280-33f2-4b85-af5b-a6798054b85a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"161d7d29-43e1-4306-bfa1-36cd04a2474d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3ed0d14d3467b190aad4d7f8c3c47f3f942dcbd36a3398099319a304d5bc33f6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a22188ed-63d2-4995-88a9-b712e114fba8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"604f440e9b9b86622a88fc08bc7abdb2279679621b1b3245e7dbe054d843489f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"184f590e-009e-4cdc-9a0c-d5c3bdfa4270\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"78251cfd5b51be30a3e650acfea274c93ff6b9873ed11e6aebdadac6a9126842\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2874, \"end_char_idx\": 5601, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "doc_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "ref_doc_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "chroma:document": "```py \n>>> import pathlib\n>>> dataset_root_path = \"UCF101_subset\"\n>>> dataset_root_path = pathlib.Path(dataset_root_path)\n```\n\n```py \n>>> video_count_train = len(list(dataset_root_path.glob(\"train/*/*.avi\")))\n>>> video_count_val = len(list(dataset_root_path.glob(\"val/*/*.avi\")))\n>>> video_count_test = len(list(dataset_root_path.glob(\"test/*/*.avi\")))\n>>> video_total = video_count_train + video_count_val + video_count_test\n>>> print(f\"Total videos: {video_total}\")\n```\n\n```py \n>>> all_video_file_paths = (\n...     list(dataset_root_path.glob(\"train/*/*.avi\"))\n...     + list(dataset_root_path.glob(\"val/*/*.avi\"))\n...     + list(dataset_root_path.glob(\"test/*/*.avi\"))\n...  )\n>>> all_video_file_paths[:5]\n```\n\nThe (`sorted`) video paths appear like so:\n\n```bash\n...\n'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g07_c04.avi',\n'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g07_c06.avi',\n'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g08_c01.avi',\n'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g09_c02.avi',\n'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g09_c06.avi'\n...\n```\n\nYou will notice that there are video clips belonging to the same group / scene where group is denoted by `g` in the video file paths. `v_ApplyEyeMakeup_g07_c04.avi` and `v_ApplyEyeMakeup_g07_c06.avi`, for example.\n\nFor the validation and evaluation splits, you wouldn't want to have video clips from the same group / scene to prevent [data leakage](https://www.kaggle.com/code/alexisbcook/data-leakage). The subset that you are using in this tutorial takes this information into account.\n\nNext up, you will derive the set of labels present in the dataset. Also, create two dictionaries that'll be helpful when initializing the model:\n\n* `label2id`: maps the class names to integers.\n* `id2label`: maps the integers to class names. \n\n```py \n>>> class_labels = sorted({str(path).split(\"/\")[2] for path in all_video_file_paths})\n>>> label2id = {label: i for i, label in enumerate(class_labels)}\n>>> id2label = {i: label for label, i in label2id.items()}\n\n>>> print(f\"Unique classes: {list(label2id.keys())}.\")\n\n# Unique classes: ['ApplyEyeMakeup', 'ApplyLipstick', 'Archery', 'BabyCrawling', 'BalanceBeam', 'BandMarching', 'BaseballPitch', 'Basketball', 'BasketballDunk', 'BenchPress'].\n```\n\nThere are 10 unique classes. For each class, there are 30 videos in the training set.\n\n## Load a model to fine-tune\n\nInstantiate a video classification model from a pretrained checkpoint and its associated image processor. The model's encoder comes with pre-trained parameters, and the classification head is randomly initialized. The image processor will come in handy when writing the preprocessing pipeline for our dataset."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 184f590e-009e-4cdc-9a0c-d5c3bdfa4270
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"184f590e-009e-4cdc-9a0c-d5c3bdfa4270\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"161d7d29-43e1-4306-bfa1-36cd04a2474d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3ed0d14d3467b190aad4d7f8c3c47f3f942dcbd36a3398099319a304d5bc33f6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e28a0280-33f2-4b85-af5b-a6798054b85a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a09cf2a703943efa4c473d3d25731756490581b0587cfb0319cb1f39ba5bdf16\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f9592bea-eb1f-4df9-8b05-1a4294b3cbd9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"940cdf82eeff2e523ce4798598006d2da23533cc6709c91b06d14dc4e4e8c253\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4120, \"end_char_idx\": 7403, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "doc_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "ref_doc_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "chroma:document": "`v_ApplyEyeMakeup_g07_c04.avi` and `v_ApplyEyeMakeup_g07_c06.avi`, for example.\n\nFor the validation and evaluation splits, you wouldn't want to have video clips from the same group / scene to prevent [data leakage](https://www.kaggle.com/code/alexisbcook/data-leakage). The subset that you are using in this tutorial takes this information into account.\n\nNext up, you will derive the set of labels present in the dataset. Also, create two dictionaries that'll be helpful when initializing the model:\n\n* `label2id`: maps the class names to integers.\n* `id2label`: maps the integers to class names. \n\n```py \n>>> class_labels = sorted({str(path).split(\"/\")[2] for path in all_video_file_paths})\n>>> label2id = {label: i for i, label in enumerate(class_labels)}\n>>> id2label = {i: label for label, i in label2id.items()}\n\n>>> print(f\"Unique classes: {list(label2id.keys())}.\")\n\n# Unique classes: ['ApplyEyeMakeup', 'ApplyLipstick', 'Archery', 'BabyCrawling', 'BalanceBeam', 'BandMarching', 'BaseballPitch', 'Basketball', 'BasketballDunk', 'BenchPress'].\n```\n\nThere are 10 unique classes. For each class, there are 30 videos in the training set.\n\n## Load a model to fine-tune\n\nInstantiate a video classification model from a pretrained checkpoint and its associated image processor. The model's encoder comes with pre-trained parameters, and the classification head is randomly initialized. The image processor will come in handy when writing the preprocessing pipeline for our dataset.\n\n```py \n>>> from transformers import VideoMAEImageProcessor, VideoMAEForVideoClassification\n\n>>> model_ckpt = \"MCG-NJU/videomae-base\"\n>>> image_processor = VideoMAEImageProcessor.from_pretrained(model_ckpt)\n>>> model = VideoMAEForVideoClassification.from_pretrained(\n...     model_ckpt,\n...     label2id=label2id,\n...     id2label=id2label,\n...     ignore_mismatched_sizes=True,  # provide this in case you're planning to fine-tune an already fine-tuned checkpoint\n... )\n```\n\nWhile the model is loading, you might notice the following warning:\n\n```bash\nSome weights of the model checkpoint at MCG-NJU/videomae-base were not used when initializing VideoMAEForVideoClassification: [..., 'decoder.decoder_layers.1.attention.output.dense.bias', 'decoder.decoder_layers.2.attention.attention.key.weight']\n- This IS expected if you are initializing VideoMAEForVideoClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n- This IS NOT expected if you are initializing VideoMAEForVideoClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\nSome weights of VideoMAEForVideoClassification were not initialized from the model checkpoint at MCG-NJU/videomae-base and are newly initialized: ['classifier.bias', 'classifier.weight']\nYou should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n```\n\nThe warning is telling us we are throwing away some weights (e.g. the weights and bias of the `classifier` layer) and randomly initializing some others (the weights and bias of a new `classifier` layer)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f9592bea-eb1f-4df9-8b05-1a4294b3cbd9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f9592bea-eb1f-4df9-8b05-1a4294b3cbd9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"161d7d29-43e1-4306-bfa1-36cd04a2474d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3ed0d14d3467b190aad4d7f8c3c47f3f942dcbd36a3398099319a304d5bc33f6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"184f590e-009e-4cdc-9a0c-d5c3bdfa4270\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2b5449a235509cd22fee385fd695487cedc004a606b724f28d123e3255612676\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"83458088-32f0-4a91-9093-135015753db2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5344fafcb0b703d191b1fa7053eaaed9e13102a67c39371145655edda82ca724\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5603, \"end_char_idx\": 9122, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "doc_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "ref_doc_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "chroma:document": "```py \n>>> from transformers import VideoMAEImageProcessor, VideoMAEForVideoClassification\n\n>>> model_ckpt = \"MCG-NJU/videomae-base\"\n>>> image_processor = VideoMAEImageProcessor.from_pretrained(model_ckpt)\n>>> model = VideoMAEForVideoClassification.from_pretrained(\n...     model_ckpt,\n...     label2id=label2id,\n...     id2label=id2label,\n...     ignore_mismatched_sizes=True,  # provide this in case you're planning to fine-tune an already fine-tuned checkpoint\n... )\n```\n\nWhile the model is loading, you might notice the following warning:\n\n```bash\nSome weights of the model checkpoint at MCG-NJU/videomae-base were not used when initializing VideoMAEForVideoClassification: [..., 'decoder.decoder_layers.1.attention.output.dense.bias', 'decoder.decoder_layers.2.attention.attention.key.weight']\n- This IS expected if you are initializing VideoMAEForVideoClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n- This IS NOT expected if you are initializing VideoMAEForVideoClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\nSome weights of VideoMAEForVideoClassification were not initialized from the model checkpoint at MCG-NJU/videomae-base and are newly initialized: ['classifier.bias', 'classifier.weight']\nYou should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n```\n\nThe warning is telling us we are throwing away some weights (e.g. the weights and bias of the `classifier` layer) and randomly initializing some others (the weights and bias of a new `classifier` layer). This is expected in this case, because we are adding a new head for which we don't have pretrained weights, so the library warns us we should fine-tune this model before using it for inference, which is exactly what we are going to do.\n\n**Note** that [this checkpoint](https://huggingface.co/MCG-NJU/videomae-base-finetuned-kinetics) leads to better performance on this task as the checkpoint was obtained fine-tuning on a similar downstream task having considerable domain overlap. You can check out [this checkpoint](https://huggingface.co/sayakpaul/videomae-base-finetuned-kinetics-finetuned-ucf101-subset) which was obtained by fine-tuning `MCG-NJU/videomae-base-finetuned-kinetics`.  \n\n## Prepare the datasets for training\n\nFor preprocessing the videos, you will leverage the [PyTorchVideo library](https://pytorchvideo.org/). Start by importing the dependencies we need. \n\n```py \n>>> import pytorchvideo.data\n\n>>> from pytorchvideo.transforms import (\n...     ApplyTransformToKey,\n...     Normalize,\n...     RandomShortSideScale,\n...     RemoveKey,\n...     ShortSideScale,\n...     UniformTemporalSubsample,\n... )\n\n>>> from torchvision.transforms import (\n...     Compose,\n...     Lambda,\n...     RandomCrop,\n...     RandomHorizontalFlip,\n...     Resize,\n... )\n```\n\nFor the training dataset transformations, use a combination of uniform temporal subsampling, pixel normalization, random cropping, and random horizontal flipping. For the validation and evaluation dataset transformations, keep the same transformation chain except for random cropping and horizontal flipping. To learn more about the details of these transformations check out the [official documentation of PyTorchVideo](https://pytorchvideo.org)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 83458088-32f0-4a91-9093-135015753db2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"83458088-32f0-4a91-9093-135015753db2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"161d7d29-43e1-4306-bfa1-36cd04a2474d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3ed0d14d3467b190aad4d7f8c3c47f3f942dcbd36a3398099319a304d5bc33f6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f9592bea-eb1f-4df9-8b05-1a4294b3cbd9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1ab65c8e0a06b6411fcdbaffd84d464f35ad1a8a94b966ce5370398774ab347a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7b900600-5651-4114-82bc-526c2c42e296\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3f9ef1f7d715aba10251a8edbfbaa7931a1ef7028a46beeb7b97e7a8c12ae087\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7404, \"end_char_idx\": 9975, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "doc_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "ref_doc_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "chroma:document": "This is expected in this case, because we are adding a new head for which we don't have pretrained weights, so the library warns us we should fine-tune this model before using it for inference, which is exactly what we are going to do.\n\n**Note** that [this checkpoint](https://huggingface.co/MCG-NJU/videomae-base-finetuned-kinetics) leads to better performance on this task as the checkpoint was obtained fine-tuning on a similar downstream task having considerable domain overlap. You can check out [this checkpoint](https://huggingface.co/sayakpaul/videomae-base-finetuned-kinetics-finetuned-ucf101-subset) which was obtained by fine-tuning `MCG-NJU/videomae-base-finetuned-kinetics`.  \n\n## Prepare the datasets for training\n\nFor preprocessing the videos, you will leverage the [PyTorchVideo library](https://pytorchvideo.org/). Start by importing the dependencies we need. \n\n```py \n>>> import pytorchvideo.data\n\n>>> from pytorchvideo.transforms import (\n...     ApplyTransformToKey,\n...     Normalize,\n...     RandomShortSideScale,\n...     RemoveKey,\n...     ShortSideScale,\n...     UniformTemporalSubsample,\n... )\n\n>>> from torchvision.transforms import (\n...     Compose,\n...     Lambda,\n...     RandomCrop,\n...     RandomHorizontalFlip,\n...     Resize,\n... )\n```\n\nFor the training dataset transformations, use a combination of uniform temporal subsampling, pixel normalization, random cropping, and random horizontal flipping. For the validation and evaluation dataset transformations, keep the same transformation chain except for random cropping and horizontal flipping. To learn more about the details of these transformations check out the [official documentation of PyTorchVideo](https://pytorchvideo.org).  \n\nUse the `image_processor` associated with the pre-trained model to obtain the following information:\n\n* Image mean and standard deviation with which the video frame pixels will be normalized.\n* Spatial resolution to which the video frames will be resized.\n\nStart by defining some constants.\n\n```py\n>>> mean = image_processor.image_mean\n>>> std = image_processor.image_std\n>>> if \"shortest_edge\" in image_processor.size:\n...     height = width = image_processor.size[\"shortest_edge\"]\n>>> else:\n...     height = image_processor.size[\"height\"]\n...     width = image_processor.size[\"width\"]\n>>> resize_to = (height, width)\n\n>>> num_frames_to_sample = model.config.num_frames\n>>> sample_rate = 4\n>>> fps = 30\n>>> clip_duration = num_frames_to_sample * sample_rate / fps\n```\n\nNow, define the dataset-specific transformations and the datasets respectively."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 7b900600-5651-4114-82bc-526c2c42e296
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7b900600-5651-4114-82bc-526c2c42e296\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"161d7d29-43e1-4306-bfa1-36cd04a2474d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3ed0d14d3467b190aad4d7f8c3c47f3f942dcbd36a3398099319a304d5bc33f6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"83458088-32f0-4a91-9093-135015753db2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d02a707f359c1bc4aed8762981ebd84445ef29eb26e6a3765550541006f12d50\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1635f8f4-3fb7-4313-89da-97fd519809ea\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1f87221d24048b78cc5c3ea9943f339a06d74fd0edaeb65faff8a10f005e3a9d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8095, \"end_char_idx\": 12062, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "doc_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "ref_doc_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "chroma:document": "## Prepare the datasets for training\n\nFor preprocessing the videos, you will leverage the [PyTorchVideo library](https://pytorchvideo.org/). Start by importing the dependencies we need. \n\n```py \n>>> import pytorchvideo.data\n\n>>> from pytorchvideo.transforms import (\n...     ApplyTransformToKey,\n...     Normalize,\n...     RandomShortSideScale,\n...     RemoveKey,\n...     ShortSideScale,\n...     UniformTemporalSubsample,\n... )\n\n>>> from torchvision.transforms import (\n...     Compose,\n...     Lambda,\n...     RandomCrop,\n...     RandomHorizontalFlip,\n...     Resize,\n... )\n```\n\nFor the training dataset transformations, use a combination of uniform temporal subsampling, pixel normalization, random cropping, and random horizontal flipping. For the validation and evaluation dataset transformations, keep the same transformation chain except for random cropping and horizontal flipping. To learn more about the details of these transformations check out the [official documentation of PyTorchVideo](https://pytorchvideo.org).  \n\nUse the `image_processor` associated with the pre-trained model to obtain the following information:\n\n* Image mean and standard deviation with which the video frame pixels will be normalized.\n* Spatial resolution to which the video frames will be resized.\n\nStart by defining some constants.\n\n```py\n>>> mean = image_processor.image_mean\n>>> std = image_processor.image_std\n>>> if \"shortest_edge\" in image_processor.size:\n...     height = width = image_processor.size[\"shortest_edge\"]\n>>> else:\n...     height = image_processor.size[\"height\"]\n...     width = image_processor.size[\"width\"]\n>>> resize_to = (height, width)\n\n>>> num_frames_to_sample = model.config.num_frames\n>>> sample_rate = 4\n>>> fps = 30\n>>> clip_duration = num_frames_to_sample * sample_rate / fps\n```\n\nNow, define the dataset-specific transformations and the datasets respectively. Starting with the training set: \n\n```py \n>>> train_transform = Compose(\n...     [\n...         ApplyTransformToKey(\n...             key=\"video\",\n...             transform=Compose(\n...                 [\n...                     UniformTemporalSubsample(num_frames_to_sample),\n...                     Lambda(lambda x: x / 255.0),\n...                     Normalize(mean, std),\n...                     RandomShortSideScale(min_size=256, max_size=320),\n...                     RandomCrop(resize_to),\n...                     RandomHorizontalFlip(p=0.5),\n...                 ]\n...             ),\n...         ),\n...     ]\n... )\n\n>>> train_dataset = pytorchvideo.data.Ucf101(\n...     data_path=os.path.join(dataset_root_path, \"train\"),\n...     clip_sampler=pytorchvideo.data.make_clip_sampler(\"random\", clip_duration),\n...     decode_audio=False,\n...     transform=train_transform,\n... )\n```\n\nThe same sequence of workflow can be applied to the validation and evaluation sets: \n\n```py \n>>> val_transform = Compose(\n...     [\n...         ApplyTransformToKey(\n...             key=\"video\",\n...             transform=Compose(\n...                 [\n...                     UniformTemporalSubsample(num_frames_to_sample),\n...                     Lambda(lambda x: x / 255.0),\n...                     Normalize(mean, std),\n...                     Resize(resize_to),\n...                 ]\n...             ),\n...         ),\n...     ]\n... )\n\n>>> val_dataset = pytorchvideo.data.Ucf101(\n...     data_path=os.path.join(dataset_root_path, \"val\"),\n...     clip_sampler=pytorchvideo.data.make_clip_sampler(\"uniform\", clip_duration),\n...     decode_audio=False,\n...     transform=val_transform,\n... )\n\n>>> test_dataset = pytorchvideo.data.Ucf101(\n...     data_path=os.path.join(dataset_root_path, \"test\"),\n...     clip_sampler=pytorchvideo.data.make_clip_sampler(\"uniform\", clip_duration),\n...     decode_audio=False,\n...     transform=val_transform,\n... )\n```\n\n**Note**: The above dataset pipelines are taken from the [official PyTorchVideo example](https://pytorchvideo.org/docs/tutorial_classification#dataset)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 1635f8f4-3fb7-4313-89da-97fd519809ea
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1635f8f4-3fb7-4313-89da-97fd519809ea\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"161d7d29-43e1-4306-bfa1-36cd04a2474d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3ed0d14d3467b190aad4d7f8c3c47f3f942dcbd36a3398099319a304d5bc33f6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7b900600-5651-4114-82bc-526c2c42e296\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e7459a97d625e3e888d0b90fde07ac534fbf7e17179eb0b820eba35a196cff34\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"672fabf0-66c2-414a-ae6d-833360ba779a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"252a9171a10cb583b6e3fae6ac2f660fb6b8ea53fee1097430f10738bcc90211\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12063, \"end_char_idx\": 15210, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "doc_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "ref_doc_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "chroma:document": "We're using the [`pytorchvideo.data.Ucf101()`](https://pytorchvideo.readthedocs.io/en/latest/api/data/data.html#pytorchvideo.data.Ucf101) function because it's tailored for the UCF-101 dataset. Under the hood, it returns a [`pytorchvideo.data.labeled_video_dataset.LabeledVideoDataset`](https://pytorchvideo.readthedocs.io/en/latest/api/data/data.html#pytorchvideo.data.LabeledVideoDataset) object. `LabeledVideoDataset` class is the base class for all things video in the PyTorchVideo dataset. So, if you want to use a custom dataset not supported off-the-shelf by PyTorchVideo, you can extend the `LabeledVideoDataset` class accordingly. Refer to the `data` API [documentation to](https://pytorchvideo.readthedocs.io/en/latest/api/data/data.html) learn more. Also, if your dataset follows a similar structure (as shown above), then using the `pytorchvideo.data.Ucf101()` should work just fine. \n\nYou can access the `num_videos` argument to know the number of videos in the dataset.\n\n```py\n>>> print(train_dataset.num_videos, val_dataset.num_videos, test_dataset.num_videos)\n# (300, 30, 75)\n```\n\n## Visualize the preprocessed video for better debugging \n\n```py \n>>> import imageio\n>>> import numpy as np\n>>> from IPython.display import Image\n\n>>> def unnormalize_img(img):\n...     \"\"\"Un-normalizes the image pixels.\"\"\"\n...     img = (img * std) + mean\n...     img = (img * 255).astype(\"uint8\")\n...     return img.clip(0, 255)\n\n>>> def create_gif(video_tensor, filename=\"sample.gif\"):\n...     \"\"\"Prepares a GIF from a video tensor.\n...     \n...     The video tensor is expected to have the following shape:\n...     (num_frames, num_channels, height, width).\n...     \"\"\"\n...     frames = []\n...     for video_frame in video_tensor:\n...         frame_unnormalized = unnormalize_img(video_frame.permute(1, 2, 0).numpy())\n...         frames.append(frame_unnormalized)\n...     kargs = {\"duration\": 0.25}\n...     imageio.mimsave(filename, frames, \"GIF\", **kargs)\n...     return filename\n\n>>> def display_gif(video_tensor, gif_name=\"sample.gif\"):\n...     \"\"\"Prepares and displays a GIF from a video tensor.\"\"\"\n...     video_tensor = video_tensor.permute(1, 0, 2, 3)\n...     gif_filename = create_gif(video_tensor, gif_name)\n...     return Image(filename=gif_filename)\n\n>>> sample_video = next(iter(train_dataset))\n>>> video_tensor = sample_video[\"video\"]\n>>> display_gif(video_tensor)\n```\n\n<div class=\"flex justify-center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/sample_gif.gif\" alt=\"Person playing basketball\"/>\n</div>\n\n## Train the model \n\nLeverage [`Trainer`](https://huggingface.co/docs/transformers/main_classes/trainer) from  \ud83e\udd17 Transformers for training the model. To instantiate a `Trainer`, you need to define the training configuration and an evaluation metric. The most important is the [`TrainingArguments`](https://huggingface.co/transformers/main_classes/trainer.html#transformers.TrainingArguments), which is a class that contains all the attributes to configure the training. It requires an output folder name, which will be used to save the checkpoints of the model."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 672fabf0-66c2-414a-ae6d-833360ba779a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"672fabf0-66c2-414a-ae6d-833360ba779a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"161d7d29-43e1-4306-bfa1-36cd04a2474d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3ed0d14d3467b190aad4d7f8c3c47f3f942dcbd36a3398099319a304d5bc33f6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1635f8f4-3fb7-4313-89da-97fd519809ea\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"af42fbaafbe2d9d7781e3a88518c512e39e3c6650872d16f5c957bcf7f1caf3f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fe895121-9907-4f64-b243-b99bb0b97c2b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"261cbc62abd204d47716f98d7d8cc4aa5c70275ca87ab7b1d02b41ed5749f22b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13595, \"end_char_idx\": 15793, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "doc_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "ref_doc_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "chroma:document": "...     \n...     The video tensor is expected to have the following shape:\n...     (num_frames, num_channels, height, width).\n...     \"\"\"\n...     frames = []\n...     for video_frame in video_tensor:\n...         frame_unnormalized = unnormalize_img(video_frame.permute(1, 2, 0).numpy())\n...         frames.append(frame_unnormalized)\n...     kargs = {\"duration\": 0.25}\n...     imageio.mimsave(filename, frames, \"GIF\", **kargs)\n...     return filename\n\n>>> def display_gif(video_tensor, gif_name=\"sample.gif\"):\n...     \"\"\"Prepares and displays a GIF from a video tensor.\"\"\"\n...     video_tensor = video_tensor.permute(1, 0, 2, 3)\n...     gif_filename = create_gif(video_tensor, gif_name)\n...     return Image(filename=gif_filename)\n\n>>> sample_video = next(iter(train_dataset))\n>>> video_tensor = sample_video[\"video\"]\n>>> display_gif(video_tensor)\n```\n\n<div class=\"flex justify-center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/sample_gif.gif\" alt=\"Person playing basketball\"/>\n</div>\n\n## Train the model \n\nLeverage [`Trainer`](https://huggingface.co/docs/transformers/main_classes/trainer) from  \ud83e\udd17 Transformers for training the model. To instantiate a `Trainer`, you need to define the training configuration and an evaluation metric. The most important is the [`TrainingArguments`](https://huggingface.co/transformers/main_classes/trainer.html#transformers.TrainingArguments), which is a class that contains all the attributes to configure the training. It requires an output folder name, which will be used to save the checkpoints of the model. It also helps sync all the information in the model repository on \ud83e\udd17 Hub.\n\nMost of the training arguments are self-explanatory, but one that is quite important here is `remove_unused_columns=False`. This one will drop any features not used by the model's call function. By default it's `True` because usually it's ideal to drop unused feature columns, making it easier to unpack inputs into the model's call function. But, in this case, you need the unused features ('video' in particular) in order to create `pixel_values` (which is a mandatory key our model expects in its inputs)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: fe895121-9907-4f64-b243-b99bb0b97c2b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"fe895121-9907-4f64-b243-b99bb0b97c2b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"161d7d29-43e1-4306-bfa1-36cd04a2474d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3ed0d14d3467b190aad4d7f8c3c47f3f942dcbd36a3398099319a304d5bc33f6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"672fabf0-66c2-414a-ae6d-833360ba779a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a2c6aabfbe051f8cad3d8a3a35c8debb7ac42993fa85cf245e702f0cdadf4404\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"89ef923f-c142-4602-9140-3b4989b05bee\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"66a10dfc39813344cb7a40227ed7c5059f2dd0fe19763a9c886a0709dc94bc6c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14166, \"end_char_idx\": 17483, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "doc_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "ref_doc_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "chroma:document": "...     video_tensor = video_tensor.permute(1, 0, 2, 3)\n...     gif_filename = create_gif(video_tensor, gif_name)\n...     return Image(filename=gif_filename)\n\n>>> sample_video = next(iter(train_dataset))\n>>> video_tensor = sample_video[\"video\"]\n>>> display_gif(video_tensor)\n```\n\n<div class=\"flex justify-center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/sample_gif.gif\" alt=\"Person playing basketball\"/>\n</div>\n\n## Train the model \n\nLeverage [`Trainer`](https://huggingface.co/docs/transformers/main_classes/trainer) from  \ud83e\udd17 Transformers for training the model. To instantiate a `Trainer`, you need to define the training configuration and an evaluation metric. The most important is the [`TrainingArguments`](https://huggingface.co/transformers/main_classes/trainer.html#transformers.TrainingArguments), which is a class that contains all the attributes to configure the training. It requires an output folder name, which will be used to save the checkpoints of the model. It also helps sync all the information in the model repository on \ud83e\udd17 Hub.\n\nMost of the training arguments are self-explanatory, but one that is quite important here is `remove_unused_columns=False`. This one will drop any features not used by the model's call function. By default it's `True` because usually it's ideal to drop unused feature columns, making it easier to unpack inputs into the model's call function. But, in this case, you need the unused features ('video' in particular) in order to create `pixel_values` (which is a mandatory key our model expects in its inputs).\n\n\n```py \n>>> from transformers import TrainingArguments, Trainer\n\n>>> model_name = model_ckpt.split(\"/\")[-1]\n>>> new_model_name = f\"{model_name}-finetuned-ucf101-subset\"\n>>> num_epochs = 4\n\n>>> args = TrainingArguments(\n...     new_model_name,\n...     remove_unused_columns=False,\n...     eval_strategy=\"epoch\",\n...     save_strategy=\"epoch\",\n...     learning_rate=5e-5,\n...     per_device_train_batch_size=batch_size,\n...     per_device_eval_batch_size=batch_size,\n...     warmup_ratio=0.1,\n...     logging_steps=10,\n...     load_best_model_at_end=True,\n...     metric_for_best_model=\"accuracy\",\n...     push_to_hub=True,\n...     max_steps=(train_dataset.num_videos // batch_size) * num_epochs,\n... )\n```\n\nThe dataset returned by `pytorchvideo.data.Ucf101()` doesn't implement the `__len__` method. As such, we must define `max_steps` when instantiating `TrainingArguments`. \n\nNext, you need to define a function to compute the metrics from the predictions, which will use the `metric` you'll load now. The only preprocessing you have to do is to take the argmax of our predicted logits:\n\n```py\nimport evaluate\n\nmetric = evaluate.load(\"accuracy\")\n\n\ndef compute_metrics(eval_pred):\n    predictions = np.argmax(eval_pred.predictions, axis=1)\n    return metric.compute(predictions=predictions, references=eval_pred.label_ids)\n```\n\n**A note on evaluation**:\n\nIn the [VideoMAE paper](https://arxiv.org/abs/2203.12602), the authors use the following evaluation strategy. They evaluate the model on several clips from test videos and apply different crops to those clips and report the aggregate score. However, in the interest of simplicity and brevity, we don't consider that in this tutorial."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 89ef923f-c142-4602-9140-3b4989b05bee
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"89ef923f-c142-4602-9140-3b4989b05bee\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"161d7d29-43e1-4306-bfa1-36cd04a2474d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3ed0d14d3467b190aad4d7f8c3c47f3f942dcbd36a3398099319a304d5bc33f6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"fe895121-9907-4f64-b243-b99bb0b97c2b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"68949190028592aa342d9aa3c44142620efdc22b435085465ec9894637e43398\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"de68dfdb-5a9b-4372-bb67-e3fc77808157\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"936d7c3bbcc39ed382a03c177501fef63f6af3c7d3bf51a0f5872c4fcdd0573b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 16944, \"end_char_idx\": 19437, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "doc_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "ref_doc_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "chroma:document": "def compute_metrics(eval_pred):\n    predictions = np.argmax(eval_pred.predictions, axis=1)\n    return metric.compute(predictions=predictions, references=eval_pred.label_ids)\n```\n\n**A note on evaluation**:\n\nIn the [VideoMAE paper](https://arxiv.org/abs/2203.12602), the authors use the following evaluation strategy. They evaluate the model on several clips from test videos and apply different crops to those clips and report the aggregate score. However, in the interest of simplicity and brevity, we don't consider that in this tutorial.\n\nAlso, define a `collate_fn`, which will be used to batch examples together. Each batch consists of 2 keys, namely `pixel_values` and `labels`.\n\n```py \n>>> def collate_fn(examples):\n...     # permute to (num_frames, num_channels, height, width)\n...     pixel_values = torch.stack(\n...         [example[\"video\"].permute(1, 0, 2, 3) for example in examples]\n...     )\n...     labels = torch.tensor([example[\"label\"] for example in examples])\n...     return {\"pixel_values\": pixel_values, \"labels\": labels}\n```\n\nThen you just pass all of this along with the datasets to `Trainer`:\n\n```py \n>>> trainer = Trainer(\n...     model,\n...     args,\n...     train_dataset=train_dataset,\n...     eval_dataset=val_dataset,\n...     tokenizer=image_processor,\n...     compute_metrics=compute_metrics,\n...     data_collator=collate_fn,\n... )\n```\n\nYou might wonder why you passed along the `image_processor` as a tokenizer when you preprocessed the data already. This is only to make sure the image processor configuration file (stored as JSON) will also be uploaded to the repo on the Hub.\n\nNow fine-tune our model by calling the `train` method:\n\n```py \n>>> train_results = trainer.train()\n```\n\nOnce training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:\n\n```py\n>>> trainer.push_to_hub()\n```\n\n## Inference\n\nGreat, now that you have fine-tuned a model, you can use it for inference!\n\nLoad a video for inference:\n\n```py \n>>> sample_test_video = next(iter(test_dataset))\n```\n\n<div class=\"flex justify-center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/sample_gif_two.gif\" alt=\"Teams playing basketball\"/>\n</div>\n\nThe simplest way to try out your fine-tuned model for inference is to use it in a [`pipeline`](https://huggingface.co/docs/transformers/main/en/main_classes/pipelines#transformers.VideoClassificationPipeline)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: de68dfdb-5a9b-4372-bb67-e3fc77808157
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"de68dfdb-5a9b-4372-bb67-e3fc77808157\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"161d7d29-43e1-4306-bfa1-36cd04a2474d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3ed0d14d3467b190aad4d7f8c3c47f3f942dcbd36a3398099319a304d5bc33f6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"89ef923f-c142-4602-9140-3b4989b05bee\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/video_classification\", \"title\": \"Video classification\", \"tokens\": 5166, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"aed5cbdcc2cfcc5ca11dde644ba7ba7f18393946511dc1e9a8ec327403d2d321\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 18429, \"end_char_idx\": 21219, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "doc_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "ref_doc_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "chroma:document": "This is only to make sure the image processor configuration file (stored as JSON) will also be uploaded to the repo on the Hub.\n\nNow fine-tune our model by calling the `train` method:\n\n```py \n>>> train_results = trainer.train()\n```\n\nOnce training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:\n\n```py\n>>> trainer.push_to_hub()\n```\n\n## Inference\n\nGreat, now that you have fine-tuned a model, you can use it for inference!\n\nLoad a video for inference:\n\n```py \n>>> sample_test_video = next(iter(test_dataset))\n```\n\n<div class=\"flex justify-center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/sample_gif_two.gif\" alt=\"Teams playing basketball\"/>\n</div>\n\nThe simplest way to try out your fine-tuned model for inference is to use it in a [`pipeline`](https://huggingface.co/docs/transformers/main/en/main_classes/pipelines#transformers.VideoClassificationPipeline). Instantiate a `pipeline` for video classification with your model, and pass your video to it:\n\n```py\n>>> from transformers import pipeline\n\n>>> video_cls = pipeline(model=\"my_awesome_video_cls_model\")\n>>> video_cls(\"https://huggingface.co/datasets/sayakpaul/ucf101-subset/resolve/main/v_BasketballDunk_g14_c06.avi\")\n[{'score': 0.9272987842559814, 'label': 'BasketballDunk'},\n {'score': 0.017777055501937866, 'label': 'BabyCrawling'},\n {'score': 0.01663011871278286, 'label': 'BalanceBeam'},\n {'score': 0.009560945443809032, 'label': 'BandMarching'},\n {'score': 0.0068979403004050255, 'label': 'BaseballPitch'}]\n```\n\nYou can also manually replicate the results of the `pipeline` if you'd like.\n\n\n```py\n>>> def run_inference(model, video):\n...     # (num_frames, num_channels, height, width)\n...     perumuted_sample_test_video = video.permute(1, 0, 2, 3)\n...     inputs = {\n...         \"pixel_values\": perumuted_sample_test_video.unsqueeze(0),\n...         \"labels\": torch.tensor(\n...             [sample_test_video[\"label\"]]\n...         ),  # this can be skipped if you don't have labels available.\n...     }\n\n...     device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n...     inputs = {k: v.to(device) for k, v in inputs.items()}\n...     model = model.to(device)\n\n...     # forward pass\n...     with torch.no_grad():\n...         outputs = model(**inputs)\n...         logits = outputs.logits\n\n...     return logits\n```\n\nNow, pass your input to the model and return the `logits`:\n\n```py\n>>> logits = run_inference(trained_model, sample_test_video[\"video\"])\n```\n\nDecoding the `logits`, we get: \n\n```py \n>>> predicted_class_idx = logits.argmax(-1).item()\n>>> print(\"Predicted class:\", model.config.id2label[predicted_class_idx])\n# Predicted class: BasketballDunk\n```"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: c8496c81-0527-4a6a-8252-468e1f595e96
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c8496c81-0527-4a6a-8252-468e1f595e96\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/text-to-speech\", \"title\": \"Text to speech\", \"tokens\": 5485, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/text-to-speech\", \"title\": \"Text to speech\", \"tokens\": 5485, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7fef769416a8a51ead2ae01a92eef274cf124d5c7c49a1c12621ab470154a9e4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"579e961f-454b-4147-bf6d-88941c373943\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8e2262c58b77ad0d91f432ac8815c9faac51f95c8265e1f89a64bb2e923e9168\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2936, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "doc_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "ref_doc_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "chroma:document": "# Text to speech\n\n[[open-in-colab]]\n\nText-to-speech (TTS) is the task of creating natural-sounding speech from text, where the speech can be generated in multiple \nlanguages and for multiple speakers. Several text-to-speech models are currently available in \ud83e\udd17 Transformers, such as \n[Bark](../model_doc/bark), [MMS](../model_doc/mms), [VITS](../model_doc/vits) and [SpeechT5](../model_doc/speecht5). \n\nYou can easily generate audio using the `\"text-to-audio\"` pipeline (or its alias - `\"text-to-speech\"`). Some models, like Bark, \ncan also be conditioned to generate non-verbal communications such as laughing, sighing and crying, or even add music.\nHere's an example of how you would use the `\"text-to-speech\"` pipeline with Bark: \n\n```py\n>>> from transformers import pipeline\n\n>>> pipe = pipeline(\"text-to-speech\", model=\"suno/bark-small\")\n>>> text = \"[clears throat] This is a test ... and I just took a long pause.\"\n>>> output = pipe(text)\n```\n\nHere's a code snippet you can use to listen to the resulting audio in a notebook: \n\n```python\n>>> from IPython.display import Audio\n>>> Audio(output[\"audio\"], rate=output[\"sampling_rate\"])\n```\n\nFor more examples on what Bark and other pretrained TTS models can do, refer to our \n[Audio course](https://huggingface.co/learn/audio-course/chapter6/pre-trained_models). \n\nIf you are looking to fine-tune a TTS model, the only text-to-speech models currently available in \ud83e\udd17 Transformers \nare [SpeechT5](model_doc/speecht5) and [FastSpeech2Conformer](model_doc/fastspeech2_conformer), though more will be added in the future. SpeechT5 is pre-trained on a combination of speech-to-text and text-to-speech data, allowing it to learn a unified space of hidden representations shared by both text and speech. This means that the same pre-trained model can be fine-tuned for different tasks. Furthermore, SpeechT5 supports multiple speakers through x-vector speaker embeddings. \n\nThe remainder of this guide illustrates how to:\n\n1. Fine-tune [SpeechT5](../model_doc/speecht5) that was originally trained on English speech on the Dutch (`nl`) language subset of the [VoxPopuli](https://huggingface.co/datasets/facebook/voxpopuli) dataset.\n2. Use your refined model for inference in one of two ways: using a pipeline or directly.\n\nBefore you begin, make sure you have all the necessary libraries installed:\n\n```bash\npip install datasets soundfile speechbrain accelerate\n```\n\nInstall \ud83e\udd17Transformers from source as not all the SpeechT5 features have been merged into an official release yet:\n\n```bash\npip install git+https://github.com/huggingface/transformers.git\n```\n\n<Tip>\n\nTo follow this guide you will need a GPU. If you're working in a notebook, run the following line to check if a GPU is available: \n\n```bash\n!nvidia-smi\n```\n\nor alternatively for AMD GPUs:\n\n```bash\n!rocm-smi\n```\n\n</Tip>\n\nWe encourage you to log in to your Hugging Face account to upload and share your model with the community."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 579e961f-454b-4147-bf6d-88941c373943
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"579e961f-454b-4147-bf6d-88941c373943\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/text-to-speech\", \"title\": \"Text to speech\", \"tokens\": 5485, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/text-to-speech\", \"title\": \"Text to speech\", \"tokens\": 5485, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7fef769416a8a51ead2ae01a92eef274cf124d5c7c49a1c12621ab470154a9e4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c8496c81-0527-4a6a-8252-468e1f595e96\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/text-to-speech\", \"title\": \"Text to speech\", \"tokens\": 5485, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ec0e882a1f110d71111eff27b46fbc18f8fa6879102004bc3dd0186734422c34\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dfe03b92-2654-4108-a98b-5f4ca21e24fc\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"45e60029c3e31b5a8cbc1c5d8dcc76ca82d1ab1a77f6ff782100bd7218221430\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1317, \"end_char_idx\": 4586, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "doc_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "ref_doc_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "chroma:document": "If you are looking to fine-tune a TTS model, the only text-to-speech models currently available in \ud83e\udd17 Transformers \nare [SpeechT5](model_doc/speecht5) and [FastSpeech2Conformer](model_doc/fastspeech2_conformer), though more will be added in the future. SpeechT5 is pre-trained on a combination of speech-to-text and text-to-speech data, allowing it to learn a unified space of hidden representations shared by both text and speech. This means that the same pre-trained model can be fine-tuned for different tasks. Furthermore, SpeechT5 supports multiple speakers through x-vector speaker embeddings. \n\nThe remainder of this guide illustrates how to:\n\n1. Fine-tune [SpeechT5](../model_doc/speecht5) that was originally trained on English speech on the Dutch (`nl`) language subset of the [VoxPopuli](https://huggingface.co/datasets/facebook/voxpopuli) dataset.\n2. Use your refined model for inference in one of two ways: using a pipeline or directly.\n\nBefore you begin, make sure you have all the necessary libraries installed:\n\n```bash\npip install datasets soundfile speechbrain accelerate\n```\n\nInstall \ud83e\udd17Transformers from source as not all the SpeechT5 features have been merged into an official release yet:\n\n```bash\npip install git+https://github.com/huggingface/transformers.git\n```\n\n<Tip>\n\nTo follow this guide you will need a GPU. If you're working in a notebook, run the following line to check if a GPU is available: \n\n```bash\n!nvidia-smi\n```\n\nor alternatively for AMD GPUs:\n\n```bash\n!rocm-smi\n```\n\n</Tip>\n\nWe encourage you to log in to your Hugging Face account to upload and share your model with the community. When prompted, enter your token to log in:\n\n```py\n>>> from huggingface_hub import notebook_login\n\n>>> notebook_login()\n```\n\n## Load the dataset\n\n[VoxPopuli](https://huggingface.co/datasets/facebook/voxpopuli) is a large-scale multilingual speech corpus consisting of \ndata sourced from 2009-2020 European Parliament event recordings. It contains labelled audio-transcription data for 15 \nEuropean languages. In this guide, we are using the Dutch language subset, feel free to pick another subset. \n\nNote that VoxPopuli or any other automated speech recognition (ASR) dataset may not be the most suitable \noption for training TTS models. The features that make it beneficial for ASR, such as excessive background noise, are \ntypically undesirable in TTS. However, finding top-quality, multilingual, and multi-speaker TTS datasets can be quite \nchallenging.\n\nLet's load the data:\n\n```py\n>>> from datasets import load_dataset, Audio\n\n>>> dataset = load_dataset(\"facebook/voxpopuli\", \"nl\", split=\"train\")\n>>> len(dataset)\n20968\n```\n\n20968 examples should be sufficient for fine-tuning. SpeechT5 expects audio data to have a sampling rate of 16 kHz, so \nmake sure the examples in the dataset meet this requirement:\n\n```py\ndataset = dataset.cast_column(\"audio\", Audio(sampling_rate=16000))\n```\n\n## Preprocess the data\n\nLet's begin by defining the model checkpoint to use and loading the appropriate processor: \n\n```py\n>>> from transformers import SpeechT5Processor\n\n>>> checkpoint = \"microsoft/speecht5_tts\"\n>>> processor = SpeechT5Processor.from_pretrained(checkpoint)\n```\n\n### Text cleanup for SpeechT5 tokenization \n\nStart by cleaning up the text data."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: dfe03b92-2654-4108-a98b-5f4ca21e24fc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"dfe03b92-2654-4108-a98b-5f4ca21e24fc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/text-to-speech\", \"title\": \"Text to speech\", \"tokens\": 5485, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/text-to-speech\", \"title\": \"Text to speech\", \"tokens\": 5485, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7fef769416a8a51ead2ae01a92eef274cf124d5c7c49a1c12621ab470154a9e4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"579e961f-454b-4147-bf6d-88941c373943\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/text-to-speech\", \"title\": \"Text to speech\", \"tokens\": 5485, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"687ca46cfa1332a77d430b8c7a6f2766eb42ca9da61e68bd619b8bde0b5e102e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cdca21eb-9442-497d-9d8e-5ab3537321b2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6ebb7bbfb08bd865b19de6fe4749c8d678be5cacd0af1a65b8c1b696cbb225dd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2937, \"end_char_idx\": 6065, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "doc_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "ref_doc_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "chroma:document": "When prompted, enter your token to log in:\n\n```py\n>>> from huggingface_hub import notebook_login\n\n>>> notebook_login()\n```\n\n## Load the dataset\n\n[VoxPopuli](https://huggingface.co/datasets/facebook/voxpopuli) is a large-scale multilingual speech corpus consisting of \ndata sourced from 2009-2020 European Parliament event recordings. It contains labelled audio-transcription data for 15 \nEuropean languages. In this guide, we are using the Dutch language subset, feel free to pick another subset. \n\nNote that VoxPopuli or any other automated speech recognition (ASR) dataset may not be the most suitable \noption for training TTS models. The features that make it beneficial for ASR, such as excessive background noise, are \ntypically undesirable in TTS. However, finding top-quality, multilingual, and multi-speaker TTS datasets can be quite \nchallenging.\n\nLet's load the data:\n\n```py\n>>> from datasets import load_dataset, Audio\n\n>>> dataset = load_dataset(\"facebook/voxpopuli\", \"nl\", split=\"train\")\n>>> len(dataset)\n20968\n```\n\n20968 examples should be sufficient for fine-tuning. SpeechT5 expects audio data to have a sampling rate of 16 kHz, so \nmake sure the examples in the dataset meet this requirement:\n\n```py\ndataset = dataset.cast_column(\"audio\", Audio(sampling_rate=16000))\n```\n\n## Preprocess the data\n\nLet's begin by defining the model checkpoint to use and loading the appropriate processor: \n\n```py\n>>> from transformers import SpeechT5Processor\n\n>>> checkpoint = \"microsoft/speecht5_tts\"\n>>> processor = SpeechT5Processor.from_pretrained(checkpoint)\n```\n\n### Text cleanup for SpeechT5 tokenization \n\nStart by cleaning up the text data. You'll need the tokenizer part of the processor to process the text:\n\n```py\n>>> tokenizer = processor.tokenizer\n```\n\nThe dataset examples contain `raw_text` and `normalized_text` features. When deciding which feature to use as the text input, \nconsider that the SpeechT5 tokenizer doesn't have any tokens for numbers. In `normalized_text` the numbers are written \nout as text. Thus, it is a better fit, and we recommend using    `normalized_text` as input text.\n\nBecause SpeechT5 was trained on the English language, it may not recognize certain characters in the Dutch dataset. If \nleft as is, these characters will be converted to `<unk>` tokens. However, in Dutch, certain characters like `\u00e0` are \nused to stress syllables. In order to preserve the meaning of the text, we can replace this character with a regular `a`.\n\nTo identify unsupported tokens, extract all unique characters in the dataset using the `SpeechT5Tokenizer` which \nworks with characters as tokens. To do this, write the `extract_all_chars` mapping function that concatenates \nthe transcriptions from all examples into one string and converts it to a set of characters. \nMake sure to set `batched=True` and `batch_size=-1` in `dataset.map()` so that all transcriptions are available at once for \nthe mapping function.\n\n```py\n>>> def extract_all_chars(batch):\n...     all_text = \" \".join(batch[\"normalized_text\"])\n...     vocab = list(set(all_text))\n...     return {\"vocab\": [vocab], \"all_text\": [all_text]}"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: cdca21eb-9442-497d-9d8e-5ab3537321b2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"cdca21eb-9442-497d-9d8e-5ab3537321b2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/text-to-speech\", \"title\": \"Text to speech\", \"tokens\": 5485, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/text-to-speech\", \"title\": \"Text to speech\", \"tokens\": 5485, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7fef769416a8a51ead2ae01a92eef274cf124d5c7c49a1c12621ab470154a9e4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dfe03b92-2654-4108-a98b-5f4ca21e24fc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/text-to-speech\", \"title\": \"Text to speech\", \"tokens\": 5485, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"88428cf5e78b65c092bce9f6411e82d068d95008713be59396339d5d83efd785\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bad2910b-1e5c-4352-92ff-ff885cc5ccc4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"df5c17246e6d4b73f5930a815a076e38794c9cb847de788868e2055f2c964460\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4587, \"end_char_idx\": 7394, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "doc_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "ref_doc_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "chroma:document": "You'll need the tokenizer part of the processor to process the text:\n\n```py\n>>> tokenizer = processor.tokenizer\n```\n\nThe dataset examples contain `raw_text` and `normalized_text` features. When deciding which feature to use as the text input, \nconsider that the SpeechT5 tokenizer doesn't have any tokens for numbers. In `normalized_text` the numbers are written \nout as text. Thus, it is a better fit, and we recommend using    `normalized_text` as input text.\n\nBecause SpeechT5 was trained on the English language, it may not recognize certain characters in the Dutch dataset. If \nleft as is, these characters will be converted to `<unk>` tokens. However, in Dutch, certain characters like `\u00e0` are \nused to stress syllables. In order to preserve the meaning of the text, we can replace this character with a regular `a`.\n\nTo identify unsupported tokens, extract all unique characters in the dataset using the `SpeechT5Tokenizer` which \nworks with characters as tokens. To do this, write the `extract_all_chars` mapping function that concatenates \nthe transcriptions from all examples into one string and converts it to a set of characters. \nMake sure to set `batched=True` and `batch_size=-1` in `dataset.map()` so that all transcriptions are available at once for \nthe mapping function.\n\n```py\n>>> def extract_all_chars(batch):\n...     all_text = \" \".join(batch[\"normalized_text\"])\n...     vocab = list(set(all_text))\n...     return {\"vocab\": [vocab], \"all_text\": [all_text]}\n\n\n>>> vocabs = dataset.map(\n...     extract_all_chars,\n...     batched=True,\n...     batch_size=-1,\n...     keep_in_memory=True,\n...     remove_columns=dataset.column_names,\n... )\n\n>>> dataset_vocab = set(vocabs[\"vocab\"][0])\n>>> tokenizer_vocab = {k for k, _ in tokenizer.get_vocab().items()}\n```\n\nNow you have two sets of characters: one with the vocabulary from the dataset and one with the vocabulary from the tokenizer. \nTo identify any unsupported characters in the dataset, you can take the difference between these two sets. The resulting \nset will contain the characters that are in the dataset but not in the tokenizer.\n\n```py\n>>> dataset_vocab - tokenizer_vocab\n{' ', '\u00e0', '\u00e7', '\u00e8', '\u00eb', '\u00ed', '\u00ef', '\u00f6', '\u00fc'}\n```\n\nTo handle the unsupported characters identified in the previous step, define a function that maps these characters to \nvalid tokens. Note that spaces are already replaced by `\u2581` in the tokenizer and don't need to be handled separately.\n\n```py\n>>> replacements = [\n...     (\"\u00e0\", \"a\"),\n...     (\"\u00e7\", \"c\"),\n...     (\"\u00e8\", \"e\"),\n...     (\"\u00eb\", \"e\"),\n...     (\"\u00ed\", \"i\"),\n...     (\"\u00ef\", \"i\"),\n...     (\"\u00f6\", \"o\"),\n...     (\"\u00fc\", \"u\"),\n... ]\n\n\n>>> def cleanup_text(inputs):\n...     for src, dst in replacements:\n...         inputs[\"normalized_text\"] = inputs[\"normalized_text\"].replace(src, dst)\n...     return inputs"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: bad2910b-1e5c-4352-92ff-ff885cc5ccc4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"bad2910b-1e5c-4352-92ff-ff885cc5ccc4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/text-to-speech\", \"title\": \"Text to speech\", \"tokens\": 5485, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/text-to-speech\", \"title\": \"Text to speech\", \"tokens\": 5485, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7fef769416a8a51ead2ae01a92eef274cf124d5c7c49a1c12621ab470154a9e4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cdca21eb-9442-497d-9d8e-5ab3537321b2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/text-to-speech\", \"title\": \"Text to speech\", \"tokens\": 5485, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"15c61a2b000ffa1ba426ff610012551e8bb868363f578af5f6d937bcf354cf3c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"83684c47-7456-4491-a4b6-ff64bb63559c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b849a9bd0545b38b87205d48571d0ccd2ff607c4e57c73a19465e6a35ccf97d2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5878, \"end_char_idx\": 9018, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "doc_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "ref_doc_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "chroma:document": "```py\n>>> def extract_all_chars(batch):\n...     all_text = \" \".join(batch[\"normalized_text\"])\n...     vocab = list(set(all_text))\n...     return {\"vocab\": [vocab], \"all_text\": [all_text]}\n\n\n>>> vocabs = dataset.map(\n...     extract_all_chars,\n...     batched=True,\n...     batch_size=-1,\n...     keep_in_memory=True,\n...     remove_columns=dataset.column_names,\n... )\n\n>>> dataset_vocab = set(vocabs[\"vocab\"][0])\n>>> tokenizer_vocab = {k for k, _ in tokenizer.get_vocab().items()}\n```\n\nNow you have two sets of characters: one with the vocabulary from the dataset and one with the vocabulary from the tokenizer. \nTo identify any unsupported characters in the dataset, you can take the difference between these two sets. The resulting \nset will contain the characters that are in the dataset but not in the tokenizer.\n\n```py\n>>> dataset_vocab - tokenizer_vocab\n{' ', '\u00e0', '\u00e7', '\u00e8', '\u00eb', '\u00ed', '\u00ef', '\u00f6', '\u00fc'}\n```\n\nTo handle the unsupported characters identified in the previous step, define a function that maps these characters to \nvalid tokens. Note that spaces are already replaced by `\u2581` in the tokenizer and don't need to be handled separately.\n\n```py\n>>> replacements = [\n...     (\"\u00e0\", \"a\"),\n...     (\"\u00e7\", \"c\"),\n...     (\"\u00e8\", \"e\"),\n...     (\"\u00eb\", \"e\"),\n...     (\"\u00ed\", \"i\"),\n...     (\"\u00ef\", \"i\"),\n...     (\"\u00f6\", \"o\"),\n...     (\"\u00fc\", \"u\"),\n... ]\n\n\n>>> def cleanup_text(inputs):\n...     for src, dst in replacements:\n...         inputs[\"normalized_text\"] = inputs[\"normalized_text\"].replace(src, dst)\n...     return inputs\n\n\n>>> dataset = dataset.map(cleanup_text)\n```\n\nNow that you have dealt with special characters in the text, it's time to shift focus to the audio data.\n\n### Speakers\n\nThe VoxPopuli dataset includes speech from multiple speakers, but how many speakers are represented in the dataset? To \ndetermine this, we can count the number of unique speakers and the number of examples each speaker contributes to the dataset. \nWith a total of 20,968 examples in the dataset, this information will give us a better understanding of the distribution of \nspeakers and examples in the data.\n\n```py\n>>> from collections import defaultdict\n\n>>> speaker_counts = defaultdict(int)\n\n>>> for speaker_id in dataset[\"speaker_id\"]:\n...     speaker_counts[speaker_id] += 1\n```\n\nBy plotting a histogram you can get a sense of how much data there is for each speaker.\n\n```py\n>>> import matplotlib.pyplot as plt\n\n>>> plt.figure()\n>>> plt.hist(speaker_counts.values(), bins=20)\n>>> plt.ylabel(\"Speakers\")\n>>> plt.xlabel(\"Examples\")\n>>> plt.show()\n```\n\n<div class=\"flex justify-center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/tts_speakers_histogram.png\" alt=\"Speakers histogram\"/>\n</div>\n\nThe histogram reveals that approximately one-third of the speakers in the dataset have fewer than 100 examples, while \naround ten speakers have more than 500 examples. To improve training efficiency and balance the dataset, we can limit \nthe data to speakers with between 100 and 400 examples. \n\n```py\n>>> def select_speaker(speaker_id):\n...     return 100 <= speaker_counts[speaker_id] <= 400"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 83684c47-7456-4491-a4b6-ff64bb63559c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"83684c47-7456-4491-a4b6-ff64bb63559c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/text-to-speech\", \"title\": \"Text to speech\", \"tokens\": 5485, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/text-to-speech\", \"title\": \"Text to speech\", \"tokens\": 5485, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7fef769416a8a51ead2ae01a92eef274cf124d5c7c49a1c12621ab470154a9e4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bad2910b-1e5c-4352-92ff-ff885cc5ccc4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/text-to-speech\", \"title\": \"Text to speech\", \"tokens\": 5485, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3e9ec4f60e4c45577a53748098d3df766ad5a75a4d5e739f5233a3483466ab11\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cb8e7ec0-d00e-411b-83da-f2893736d467\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"38a1167684bc6a98f1a6372a0d6d33d4f097430ab06153c627b7c2a6a4d3ac68\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7221, \"end_char_idx\": 10739, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "doc_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "ref_doc_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "chroma:document": ">>> def cleanup_text(inputs):\n...     for src, dst in replacements:\n...         inputs[\"normalized_text\"] = inputs[\"normalized_text\"].replace(src, dst)\n...     return inputs\n\n\n>>> dataset = dataset.map(cleanup_text)\n```\n\nNow that you have dealt with special characters in the text, it's time to shift focus to the audio data.\n\n### Speakers\n\nThe VoxPopuli dataset includes speech from multiple speakers, but how many speakers are represented in the dataset? To \ndetermine this, we can count the number of unique speakers and the number of examples each speaker contributes to the dataset. \nWith a total of 20,968 examples in the dataset, this information will give us a better understanding of the distribution of \nspeakers and examples in the data.\n\n```py\n>>> from collections import defaultdict\n\n>>> speaker_counts = defaultdict(int)\n\n>>> for speaker_id in dataset[\"speaker_id\"]:\n...     speaker_counts[speaker_id] += 1\n```\n\nBy plotting a histogram you can get a sense of how much data there is for each speaker.\n\n```py\n>>> import matplotlib.pyplot as plt\n\n>>> plt.figure()\n>>> plt.hist(speaker_counts.values(), bins=20)\n>>> plt.ylabel(\"Speakers\")\n>>> plt.xlabel(\"Examples\")\n>>> plt.show()\n```\n\n<div class=\"flex justify-center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/tts_speakers_histogram.png\" alt=\"Speakers histogram\"/>\n</div>\n\nThe histogram reveals that approximately one-third of the speakers in the dataset have fewer than 100 examples, while \naround ten speakers have more than 500 examples. To improve training efficiency and balance the dataset, we can limit \nthe data to speakers with between 100 and 400 examples. \n\n```py\n>>> def select_speaker(speaker_id):\n...     return 100 <= speaker_counts[speaker_id] <= 400\n\n\n>>> dataset = dataset.filter(select_speaker, input_columns=[\"speaker_id\"])\n```\n\nLet's check how many speakers remain: \n\n```py\n>>> len(set(dataset[\"speaker_id\"]))\n42\n```\n\nLet's see how many examples are left: \n\n```py\n>>> len(dataset)\n9973\n```\n\nYou are left with just under 10,000 examples from approximately 40 unique speakers, which should be sufficient.\n\nNote that some speakers with few examples may actually have more audio available if the examples are long. However, \ndetermining the total amount of audio for each speaker requires scanning through the entire dataset, which is a \ntime-consuming process that involves loading and decoding each audio file. As such, we have chosen to skip this step here.\n\n### Speaker embeddings\n\nTo enable the TTS model to differentiate between multiple speakers, you'll need to create a speaker embedding for each example. \nThe speaker embedding is an additional input into the model that captures a particular speaker's voice characteristics.\nTo generate these speaker embeddings, use the pre-trained [spkrec-xvect-voxceleb](https://huggingface.co/speechbrain/spkrec-xvect-voxceleb) \nmodel from SpeechBrain. \n\nCreate a function `create_speaker_embedding()` that takes an input audio waveform and outputs a 512-element vector \ncontaining the corresponding speaker embedding.\n\n```py\n>>> import os\n>>> import torch\n>>> from speechbrain.inference.classifiers import EncoderClassifier\n\n>>> spk_model_name = \"speechbrain/spkrec-xvect-voxceleb\"\n\n>>> device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n>>> speaker_model = EncoderClassifier.from_hparams(\n...     source=spk_model_name,\n...     run_opts={\"device\": device},\n...     savedir=os.path.join(\"/tmp\", spk_model_name),\n... )"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: cb8e7ec0-d00e-411b-83da-f2893736d467
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"cb8e7ec0-d00e-411b-83da-f2893736d467\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/text-to-speech\", \"title\": \"Text to speech\", \"tokens\": 5485, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/text-to-speech\", \"title\": \"Text to speech\", \"tokens\": 5485, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7fef769416a8a51ead2ae01a92eef274cf124d5c7c49a1c12621ab470154a9e4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"83684c47-7456-4491-a4b6-ff64bb63559c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/text-to-speech\", \"title\": \"Text to speech\", \"tokens\": 5485, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e6b7cbcc51065520bc0c2063ca7a90ee4e6e73dc8bda40f686b05378a17fa495\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"79178a0a-9f15-46fa-8756-4e219353d2c2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6d7add132ba9f351d722e730eb5f831fe0f00af5b91003ee0ddb261a63dc5dd2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9021, \"end_char_idx\": 14224, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "doc_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "ref_doc_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "chroma:document": ">>> dataset = dataset.filter(select_speaker, input_columns=[\"speaker_id\"])\n```\n\nLet's check how many speakers remain: \n\n```py\n>>> len(set(dataset[\"speaker_id\"]))\n42\n```\n\nLet's see how many examples are left: \n\n```py\n>>> len(dataset)\n9973\n```\n\nYou are left with just under 10,000 examples from approximately 40 unique speakers, which should be sufficient.\n\nNote that some speakers with few examples may actually have more audio available if the examples are long. However, \ndetermining the total amount of audio for each speaker requires scanning through the entire dataset, which is a \ntime-consuming process that involves loading and decoding each audio file. As such, we have chosen to skip this step here.\n\n### Speaker embeddings\n\nTo enable the TTS model to differentiate between multiple speakers, you'll need to create a speaker embedding for each example. \nThe speaker embedding is an additional input into the model that captures a particular speaker's voice characteristics.\nTo generate these speaker embeddings, use the pre-trained [spkrec-xvect-voxceleb](https://huggingface.co/speechbrain/spkrec-xvect-voxceleb) \nmodel from SpeechBrain. \n\nCreate a function `create_speaker_embedding()` that takes an input audio waveform and outputs a 512-element vector \ncontaining the corresponding speaker embedding.\n\n```py\n>>> import os\n>>> import torch\n>>> from speechbrain.inference.classifiers import EncoderClassifier\n\n>>> spk_model_name = \"speechbrain/spkrec-xvect-voxceleb\"\n\n>>> device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n>>> speaker_model = EncoderClassifier.from_hparams(\n...     source=spk_model_name,\n...     run_opts={\"device\": device},\n...     savedir=os.path.join(\"/tmp\", spk_model_name),\n... )\n\n\n>>> def create_speaker_embedding(waveform):\n...     with torch.no_grad():\n...         speaker_embeddings = speaker_model.encode_batch(torch.tensor(waveform))\n...         speaker_embeddings = torch.nn.functional.normalize(speaker_embeddings, dim=2)\n...         speaker_embeddings = speaker_embeddings.squeeze().cpu().numpy()\n...     return speaker_embeddings\n```\n\nIt's important to note that the `speechbrain/spkrec-xvect-voxceleb` model was trained on English speech from the VoxCeleb \ndataset, whereas the training examples in this guide are in Dutch. While we believe that this model will still generate \nreasonable speaker embeddings for our Dutch dataset, this assumption may not hold true in all cases.\n\nFor optimal results, we recommend training an X-vector model on the target speech first. This will ensure that the model \nis better able to capture the unique voice characteristics present in the Dutch language.\n\n### Processing the dataset\n\nFinally, let's process the data into the format the model expects. Create a `prepare_dataset` function that takes in a \nsingle example and uses the `SpeechT5Processor` object to tokenize the input text and load the target audio into a log-mel spectrogram. \nIt should also add the speaker embeddings as an additional input.\n\n```py\n>>> def prepare_dataset(example):\n...     audio = example[\"audio\"]\n\n...     example = processor(\n...         text=example[\"normalized_text\"],\n...         audio_target=audio[\"array\"],\n...         sampling_rate=audio[\"sampling_rate\"],\n...         return_attention_mask=False,\n...     )\n\n...     # strip off the batch dimension\n...     example[\"labels\"] = example[\"labels\"][0]\n\n...     # use SpeechBrain to obtain x-vector\n...     example[\"speaker_embeddings\"] = create_speaker_embedding(audio[\"array\"])\n\n...     return example\n```\n\nVerify the processing is correct by looking at a single example:\n\n```py\n>>> processed_example = prepare_dataset(dataset[0])\n>>> list(processed_example.keys())\n['input_ids', 'labels', 'stop_labels', 'speaker_embeddings']\n```\n\nSpeaker embeddings should be a 512-element vector:\n\n```py\n>>> processed_example[\"speaker_embeddings\"].shape\n(512,)\n```\n\nThe labels should be a log-mel spectrogram with 80 mel bins.\n\n```py\n>>> import matplotlib.pyplot as plt\n\n>>> plt.figure()\n>>> plt.imshow(processed_example[\"labels\"].T)\n>>> plt.show()\n```\n\n<div class=\"flex justify-center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/tts_logmelspectrogram_1.png\" alt=\"Log-mel spectrogram with 80 mel bins\"/>\n</div>\n\nSide note: If you find this spectrogram confusing, it may be due to your familiarity with the convention of placing low frequencies \nat the bottom and high frequencies at the top of a plot. However, when plotting spectrograms as an image using the matplotlib library, \nthe y-axis is flipped and the spectrograms appear upside down.\n\nNow apply the processing function to the entire dataset. This will take between 5 and 10 minutes.\n\n```py\n>>> dataset = dataset.map(prepare_dataset, remove_columns=dataset.column_names)\n```\n\nYou'll see a warning saying that some examples in the dataset are longer than the maximum input length the model can handle (600 tokens). \nRemove those examples from the dataset. Here we go even further and to allow for larger batch sizes we remove anything over 200 tokens.\n\n```py\n>>> def is_not_too_long(input_ids):\n...     input_length = len(input_ids)\n...     return input_length < 200"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 79178a0a-9f15-46fa-8756-4e219353d2c2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"79178a0a-9f15-46fa-8756-4e219353d2c2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/text-to-speech\", \"title\": \"Text to speech\", \"tokens\": 5485, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/text-to-speech\", \"title\": \"Text to speech\", \"tokens\": 5485, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7fef769416a8a51ead2ae01a92eef274cf124d5c7c49a1c12621ab470154a9e4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cb8e7ec0-d00e-411b-83da-f2893736d467\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/text-to-speech\", \"title\": \"Text to speech\", \"tokens\": 5485, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6e9f00a82ee112b84c656d1c6a790f931b586c2f2d0fc186522eba9194411834\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ecf177ec-4e67-43dd-bd5e-6fae91e745fb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c714b49e06cad6574270544a8f89ccec3835287262f9c07bdc2670db40de3f8e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14227, \"end_char_idx\": 17470, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "doc_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "ref_doc_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "chroma:document": ">>> dataset = dataset.filter(is_not_too_long, input_columns=[\"input_ids\"])\n>>> len(dataset)\n8259\n```\n\nNext, create a basic train/test split: \n\n```py\n>>> dataset = dataset.train_test_split(test_size=0.1)\n```\n\n### Data collator\n\nIn order to combine multiple examples into a batch, you need to define a custom data collator. This collator will pad shorter sequences with padding \ntokens, ensuring that all examples have the same length. For the spectrogram labels, the padded portions are replaced with the special value `-100`. This special value \ninstructs the model to ignore that part of the spectrogram when calculating the spectrogram loss.\n\n```py\n>>> from dataclasses import dataclass\n>>> from typing import Any, Dict, List, Union\n\n\n>>> @dataclass\n... class TTSDataCollatorWithPadding:\n...     processor: Any\n\n...     def __call__(self, features: List[Dict[str, Union[List[int], torch.Tensor]]]) -> Dict[str, torch.Tensor]:\n...         input_ids = [{\"input_ids\": feature[\"input_ids\"]} for feature in features]\n...         label_features = [{\"input_values\": feature[\"labels\"]} for feature in features]\n...         speaker_features = [feature[\"speaker_embeddings\"] for feature in features]\n\n...         # collate the inputs and targets into a batch\n...         batch = processor.pad(input_ids=input_ids, labels=label_features, return_tensors=\"pt\")\n\n...         # replace padding with -100 to ignore loss correctly\n...         batch[\"labels\"] = batch[\"labels\"].masked_fill(batch.decoder_attention_mask.unsqueeze(-1).ne(1), -100)\n\n...         # not used during fine-tuning\n...         del batch[\"decoder_attention_mask\"]\n\n...         # round down target lengths to multiple of reduction factor\n...         if model.config.reduction_factor > 1:\n...             target_lengths = torch.tensor([len(feature[\"input_values\"]) for feature in label_features])\n...             target_lengths = target_lengths.new(\n...                 [length - length % model.config.reduction_factor for length in target_lengths]\n...             )\n...             max_length = max(target_lengths)\n...             batch[\"labels\"] = batch[\"labels\"][:, :max_length]\n\n...         # also add in the speaker embeddings\n...         batch[\"speaker_embeddings\"] = torch.tensor(speaker_features)\n\n...         return batch\n```\n\nIn SpeechT5, the input to the decoder part of the model is reduced by a factor 2. In other words, it throws away every \nother timestep from the target sequence. The decoder then predicts a sequence that is twice as long. Since the original \ntarget sequence length may be odd, the data collator makes sure to round the maximum length of the batch down to be a \nmultiple of 2.\n\n```py \n>>> data_collator = TTSDataCollatorWithPadding(processor=processor)\n```\n\n## Train the model\n\nLoad the pre-trained model from the same checkpoint as you used for loading the processor: \n\n```py\n>>> from transformers import SpeechT5ForTextToSpeech\n\n>>> model = SpeechT5ForTextToSpeech.from_pretrained(checkpoint)\n```\n\nThe `use_cache=True` option is incompatible with gradient checkpointing. Disable it for training.\n\n```py \n>>> model.config.use_cache = False\n```\n\nDefine the training arguments. Here we are not computing any evaluation metrics during the training process."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: ecf177ec-4e67-43dd-bd5e-6fae91e745fb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ecf177ec-4e67-43dd-bd5e-6fae91e745fb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/text-to-speech\", \"title\": \"Text to speech\", \"tokens\": 5485, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/text-to-speech\", \"title\": \"Text to speech\", \"tokens\": 5485, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7fef769416a8a51ead2ae01a92eef274cf124d5c7c49a1c12621ab470154a9e4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"79178a0a-9f15-46fa-8756-4e219353d2c2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/text-to-speech\", \"title\": \"Text to speech\", \"tokens\": 5485, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"218b91c79768474f2e20cedc693321cac6573e563880a83c2b520a085d4b7701\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"35786f47-44d4-4221-b79c-779d39eaeebe\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"da446ccbb7397f67d7d8f81ed1881539faf70f049a624183443dc9fd16dcc40c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 16600, \"end_char_idx\": 19737, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "doc_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "ref_doc_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "chroma:document": "In other words, it throws away every \nother timestep from the target sequence. The decoder then predicts a sequence that is twice as long. Since the original \ntarget sequence length may be odd, the data collator makes sure to round the maximum length of the batch down to be a \nmultiple of 2.\n\n```py \n>>> data_collator = TTSDataCollatorWithPadding(processor=processor)\n```\n\n## Train the model\n\nLoad the pre-trained model from the same checkpoint as you used for loading the processor: \n\n```py\n>>> from transformers import SpeechT5ForTextToSpeech\n\n>>> model = SpeechT5ForTextToSpeech.from_pretrained(checkpoint)\n```\n\nThe `use_cache=True` option is incompatible with gradient checkpointing. Disable it for training.\n\n```py \n>>> model.config.use_cache = False\n```\n\nDefine the training arguments. Here we are not computing any evaluation metrics during the training process. Instead, we'll \nonly look at the loss:\n\n```python\n>>> from transformers import Seq2SeqTrainingArguments\n\n>>> training_args = Seq2SeqTrainingArguments(\n...     output_dir=\"speecht5_finetuned_voxpopuli_nl\",  # change to a repo name of your choice\n...     per_device_train_batch_size=4,\n...     gradient_accumulation_steps=8,\n...     learning_rate=1e-5,\n...     warmup_steps=500,\n...     max_steps=4000,\n...     gradient_checkpointing=True,\n...     fp16=True,\n...     eval_strategy=\"steps\",\n...     per_device_eval_batch_size=2,\n...     save_steps=1000,\n...     eval_steps=1000,\n...     logging_steps=25,\n...     report_to=[\"tensorboard\"],\n...     load_best_model_at_end=True,\n...     greater_is_better=False,\n...     label_names=[\"labels\"],\n...     push_to_hub=True,\n... )\n```\n\nInstantiate the `Trainer` object  and pass the model, dataset, and data collator to it.\n\n```py\n>>> from transformers import Seq2SeqTrainer\n\n>>> trainer = Seq2SeqTrainer(\n...     args=training_args,\n...     model=model,\n...     train_dataset=dataset[\"train\"],\n...     eval_dataset=dataset[\"test\"],\n...     data_collator=data_collator,\n...     tokenizer=processor,\n... )\n```\n\nAnd with that, you're ready to start training! Training will take several hours. Depending on your GPU, \nit is possible that you will encounter a CUDA \"out-of-memory\" error when you start training. In this case, you can reduce \nthe `per_device_train_batch_size` incrementally by factors of 2 and increase `gradient_accumulation_steps` by 2x to compensate.\n\n```py\n>>> trainer.train()\n```\n\nTo be able to use your checkpoint with a pipeline, make sure to save the processor with the checkpoint: \n\n```py\n>>> processor.save_pretrained(\"YOUR_ACCOUNT_NAME/speecht5_finetuned_voxpopuli_nl\")\n```\n\nPush the final model to the \ud83e\udd17 Hub:\n\n```py\n>>> trainer.push_to_hub()\n```\n\n## Inference\n\n### Inference with a pipeline\n\nGreat, now that you've fine-tuned a model, you can use it for inference!\nFirst, let's see how you can use it with a corresponding pipeline. Let's create a `\"text-to-speech\"` pipeline with your \ncheckpoint: \n\n```py\n>>> from transformers import pipeline\n\n>>> pipe = pipeline(\"text-to-speech\", model=\"YOUR_ACCOUNT_NAME/speecht5_finetuned_voxpopuli_nl\")\n```\n\nPick a piece of text in Dutch you'd like narrated, e.g."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 35786f47-44d4-4221-b79c-779d39eaeebe
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"35786f47-44d4-4221-b79c-779d39eaeebe\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/text-to-speech\", \"title\": \"Text to speech\", \"tokens\": 5485, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/text-to-speech\", \"title\": \"Text to speech\", \"tokens\": 5485, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7fef769416a8a51ead2ae01a92eef274cf124d5c7c49a1c12621ab470154a9e4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ecf177ec-4e67-43dd-bd5e-6fae91e745fb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/text-to-speech\", \"title\": \"Text to speech\", \"tokens\": 5485, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4eef41cc8002cbd1890e4aa417c64447ee5778220c2870fcaa9b4e221dbe5f7f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"440b5a6b-a4ca-4606-8ca0-42cf02ce7b24\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2da01f93b02a5634f12f7ec1a450faf61ee8d8c3bc9baa9f3cc8b1506004a58c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 18336, \"end_char_idx\": 21232, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "doc_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "ref_doc_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "chroma:document": "```py\n>>> from transformers import Seq2SeqTrainer\n\n>>> trainer = Seq2SeqTrainer(\n...     args=training_args,\n...     model=model,\n...     train_dataset=dataset[\"train\"],\n...     eval_dataset=dataset[\"test\"],\n...     data_collator=data_collator,\n...     tokenizer=processor,\n... )\n```\n\nAnd with that, you're ready to start training! Training will take several hours. Depending on your GPU, \nit is possible that you will encounter a CUDA \"out-of-memory\" error when you start training. In this case, you can reduce \nthe `per_device_train_batch_size` incrementally by factors of 2 and increase `gradient_accumulation_steps` by 2x to compensate.\n\n```py\n>>> trainer.train()\n```\n\nTo be able to use your checkpoint with a pipeline, make sure to save the processor with the checkpoint: \n\n```py\n>>> processor.save_pretrained(\"YOUR_ACCOUNT_NAME/speecht5_finetuned_voxpopuli_nl\")\n```\n\nPush the final model to the \ud83e\udd17 Hub:\n\n```py\n>>> trainer.push_to_hub()\n```\n\n## Inference\n\n### Inference with a pipeline\n\nGreat, now that you've fine-tuned a model, you can use it for inference!\nFirst, let's see how you can use it with a corresponding pipeline. Let's create a `\"text-to-speech\"` pipeline with your \ncheckpoint: \n\n```py\n>>> from transformers import pipeline\n\n>>> pipe = pipeline(\"text-to-speech\", model=\"YOUR_ACCOUNT_NAME/speecht5_finetuned_voxpopuli_nl\")\n```\n\nPick a piece of text in Dutch you'd like narrated, e.g.:\n\n```py\n>>> text = \"hallo allemaal, ik praat nederlands. groetjes aan iedereen!\"\n```\n\nTo use SpeechT5 with the pipeline, you'll need a speaker embedding. Let's get it from an example in the test dataset: \n\n```py\n>>> example = dataset[\"test\"][304]\n>>> speaker_embeddings = torch.tensor(example[\"speaker_embeddings\"]).unsqueeze(0)\n```\n\nNow you can pass the text and speaker embeddings to the pipeline, and it will take care of the rest: \n\n```py\n>>> forward_params = {\"speaker_embeddings\": speaker_embeddings}\n>>> output = pipe(text, forward_params=forward_params)\n>>> output\n{'audio': array([-6.82714235e-05, -4.26525949e-04,  1.06134125e-04, ...,\n        -1.22392643e-03, -7.76011671e-04,  3.29112721e-04], dtype=float32),\n 'sampling_rate': 16000}\n```\n\nYou can then listen to the result:\n\n```py\n>>> from IPython.display import Audio\n>>> Audio(output['audio'], rate=output['sampling_rate']) \n```\n\n### Run inference manually\n\nYou can achieve the same inference results without using the pipeline, however, more steps will be required. \n\nLoad the model from the \ud83e\udd17 Hub: \n\n```py\n>>> model = SpeechT5ForTextToSpeech.from_pretrained(\"YOUR_ACCOUNT/speecht5_finetuned_voxpopuli_nl\")\n```\n\nPick an example from the test dataset obtain a speaker embedding. \n\n```py \n>>> example = dataset[\"test\"][304]\n>>> speaker_embeddings = torch.tensor(example[\"speaker_embeddings\"]).unsqueeze(0)\n```\n\nDefine the input text and tokenize it.\n\n```py \n>>> text = \"hallo allemaal, ik praat nederlands. groetjes aan iedereen!\""}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 440b5a6b-a4ca-4606-8ca0-42cf02ce7b24
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"440b5a6b-a4ca-4606-8ca0-42cf02ce7b24\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/text-to-speech\", \"title\": \"Text to speech\", \"tokens\": 5485, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/text-to-speech\", \"title\": \"Text to speech\", \"tokens\": 5485, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7fef769416a8a51ead2ae01a92eef274cf124d5c7c49a1c12621ab470154a9e4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"35786f47-44d4-4221-b79c-779d39eaeebe\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/text-to-speech\", \"title\": \"Text to speech\", \"tokens\": 5485, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"dab92c96aabf87d48593079560229fbcb0180c167a65f1c6aa109126ef40da7c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 19795, \"end_char_idx\": 23025, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "doc_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "ref_doc_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "chroma:document": "groetjes aan iedereen!\"\n```\n\nTo use SpeechT5 with the pipeline, you'll need a speaker embedding. Let's get it from an example in the test dataset: \n\n```py\n>>> example = dataset[\"test\"][304]\n>>> speaker_embeddings = torch.tensor(example[\"speaker_embeddings\"]).unsqueeze(0)\n```\n\nNow you can pass the text and speaker embeddings to the pipeline, and it will take care of the rest: \n\n```py\n>>> forward_params = {\"speaker_embeddings\": speaker_embeddings}\n>>> output = pipe(text, forward_params=forward_params)\n>>> output\n{'audio': array([-6.82714235e-05, -4.26525949e-04,  1.06134125e-04, ...,\n        -1.22392643e-03, -7.76011671e-04,  3.29112721e-04], dtype=float32),\n 'sampling_rate': 16000}\n```\n\nYou can then listen to the result:\n\n```py\n>>> from IPython.display import Audio\n>>> Audio(output['audio'], rate=output['sampling_rate']) \n```\n\n### Run inference manually\n\nYou can achieve the same inference results without using the pipeline, however, more steps will be required. \n\nLoad the model from the \ud83e\udd17 Hub: \n\n```py\n>>> model = SpeechT5ForTextToSpeech.from_pretrained(\"YOUR_ACCOUNT/speecht5_finetuned_voxpopuli_nl\")\n```\n\nPick an example from the test dataset obtain a speaker embedding. \n\n```py \n>>> example = dataset[\"test\"][304]\n>>> speaker_embeddings = torch.tensor(example[\"speaker_embeddings\"]).unsqueeze(0)\n```\n\nDefine the input text and tokenize it.\n\n```py \n>>> text = \"hallo allemaal, ik praat nederlands. groetjes aan iedereen!\"\n>>> inputs = processor(text=text, return_tensors=\"pt\")\n```\n\nCreate a spectrogram with your model: \n\n```py\n>>> spectrogram = model.generate_speech(inputs[\"input_ids\"], speaker_embeddings)\n```\n\nVisualize the spectrogram, if you'd like to: \n\n```py\n>>> plt.figure()\n>>> plt.imshow(spectrogram.T)\n>>> plt.show()\n```\n\n<div class=\"flex justify-center\">\n    <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/tts_logmelspectrogram_2.png\" alt=\"Generated log-mel spectrogram\"/>\n</div>\n\nFinally, use the vocoder to turn the spectrogram into sound.\n\n```py\n>>> with torch.no_grad():\n...     speech = vocoder(spectrogram)\n\n>>> from IPython.display import Audio\n\n>>> Audio(speech.numpy(), rate=16000)\n```\n\nIn our experience, obtaining satisfactory results from this model can be challenging. The quality of the speaker \nembeddings appears to be a significant factor. Since SpeechT5 was pre-trained with English x-vectors, it performs best \nwhen using English speaker embeddings. If the synthesized speech sounds poor, try using a different speaker embedding.\n\nIncreasing the training duration is also likely to enhance the quality of the results. Even so, the speech clearly is Dutch instead of English, and it does \ncapture the voice characteristics of the speaker (compare to the original audio in the example).\nAnother thing to experiment with is the model's configuration. For example, try using `config.reduction_factor = 1` to \nsee if this improves the results.\n\nFinally, it is essential to consider ethical considerations. Although TTS technology has numerous useful applications, it \nmay also be used for malicious purposes, such as impersonating someone's voice without their knowledge or consent. Please \nuse TTS judiciously and responsibly."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6a06cac1-23aa-4b68-9b70-68cf66ee9114
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/audio_classification", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6a06cac1-23aa-4b68-9b70-68cf66ee9114\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/audio_classification\", \"title\": \"Audio classification\", \"tokens\": 3086, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9bf9e9b-6893-461b-bf0e-fc7387e1155c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/audio_classification\", \"title\": \"Audio classification\", \"tokens\": 3086, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ba7d015d1f001e84999554903bbcaa4d8191bf47f199a4ab3f4bfff5b10daec4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3a895197-5dcd-4588-bddf-2fb54247d648\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e46b03a0acd26943c7b5ee99ae79944766ced380d41b1ea730447865ce28ebb6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3010, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9bf9e9b-6893-461b-bf0e-fc7387e1155c", "doc_id": "d9bf9e9b-6893-461b-bf0e-fc7387e1155c", "ref_doc_id": "d9bf9e9b-6893-461b-bf0e-fc7387e1155c", "chroma:document": "# Audio classification\n\n[[open-in-colab]]\n\n<Youtube id=\"KWwzcmG98Ds\"/>\n\nAudio classification - just like with text - assigns a class label output from the input data. The only difference is instead of text inputs, you have raw audio waveforms. Some practical applications of audio classification include identifying speaker intent, language classification, and even animal species by their sounds.\n\nThis guide will show you how to:\n\n1. Finetune [Wav2Vec2](https://huggingface.co/facebook/wav2vec2-base) on the [MInDS-14](https://huggingface.co/datasets/PolyAI/minds14) dataset to classify speaker intent.\n2. Use your finetuned model for inference.\n\n<Tip>\n\nTo see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/audio-classification)\n\n</Tip>\n\nBefore you begin, make sure you have all the necessary libraries installed:\n\n```bash\npip install transformers datasets evaluate\n```\n\nWe encourage you to login to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to login:\n\n```py\n>>> from huggingface_hub import notebook_login\n\n>>> notebook_login()\n```\n\n## Load MInDS-14 dataset\n\nStart by loading the MInDS-14 dataset from the \ud83e\udd17 Datasets library:\n\n```py\n>>> from datasets import load_dataset, Audio\n\n>>> minds = load_dataset(\"PolyAI/minds14\", name=\"en-US\", split=\"train\")\n```\n\nSplit the dataset's `train` split into a smaller train and test set with the [`~datasets.Dataset.train_test_split`] method. This'll give you a chance to experiment and make sure everything works before spending more time on the full dataset.\n\n```py\n>>> minds = minds.train_test_split(test_size=0.2)\n```\n\nThen take a look at the dataset:\n\n```py\n>>> minds\nDatasetDict({\n    train: Dataset({\n        features: ['path', 'audio', 'transcription', 'english_transcription', 'intent_class', 'lang_id'],\n        num_rows: 450\n    })\n    test: Dataset({\n        features: ['path', 'audio', 'transcription', 'english_transcription', 'intent_class', 'lang_id'],\n        num_rows: 113\n    })\n})\n```\n\nWhile the dataset contains a lot of useful information, like `lang_id` and `english_transcription`, you'll focus on the `audio` and `intent_class` in this guide. Remove the other columns with the [`~datasets.Dataset.remove_columns`] method:\n\n```py\n>>> minds = minds.remove_columns([\"path\", \"transcription\", \"english_transcription\", \"lang_id\"])\n```\n\nTake a look at an example now:\n\n```py\n>>> minds[\"train\"][0]\n{'audio': {'array': array([ 0.        ,  0.        ,  0.        , ..., -0.00048828,\n         -0.00024414, -0.00024414], dtype=float32),\n  'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~APP_ERROR/602b9a5fbb1e6d0fbce91f52.wav',\n  'sampling_rate': 8000},\n 'intent_class': 2}\n```\n\nThere are two fields:\n\n- `audio`: a 1-dimensional `array` of the speech signal that must be called to load and resample the audio file."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 3a895197-5dcd-4588-bddf-2fb54247d648
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/audio_classification", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3a895197-5dcd-4588-bddf-2fb54247d648\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/audio_classification\", \"title\": \"Audio classification\", \"tokens\": 3086, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9bf9e9b-6893-461b-bf0e-fc7387e1155c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/audio_classification\", \"title\": \"Audio classification\", \"tokens\": 3086, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ba7d015d1f001e84999554903bbcaa4d8191bf47f199a4ab3f4bfff5b10daec4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6a06cac1-23aa-4b68-9b70-68cf66ee9114\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/audio_classification\", \"title\": \"Audio classification\", \"tokens\": 3086, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9012919719e644386bfcdb103b81761976f5e0adf10467d55af8d79db92e51a3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3958c7bf-8966-4308-b289-19b53e5f06eb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1d65f8041fce8b6124dfeb4287efad2a57fe5b8e88527f66f8d2509ca1d50682\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1658, \"end_char_idx\": 3078, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9bf9e9b-6893-461b-bf0e-fc7387e1155c", "doc_id": "d9bf9e9b-6893-461b-bf0e-fc7387e1155c", "ref_doc_id": "d9bf9e9b-6893-461b-bf0e-fc7387e1155c", "chroma:document": "```py\n>>> minds = minds.train_test_split(test_size=0.2)\n```\n\nThen take a look at the dataset:\n\n```py\n>>> minds\nDatasetDict({\n    train: Dataset({\n        features: ['path', 'audio', 'transcription', 'english_transcription', 'intent_class', 'lang_id'],\n        num_rows: 450\n    })\n    test: Dataset({\n        features: ['path', 'audio', 'transcription', 'english_transcription', 'intent_class', 'lang_id'],\n        num_rows: 113\n    })\n})\n```\n\nWhile the dataset contains a lot of useful information, like `lang_id` and `english_transcription`, you'll focus on the `audio` and `intent_class` in this guide. Remove the other columns with the [`~datasets.Dataset.remove_columns`] method:\n\n```py\n>>> minds = minds.remove_columns([\"path\", \"transcription\", \"english_transcription\", \"lang_id\"])\n```\n\nTake a look at an example now:\n\n```py\n>>> minds[\"train\"][0]\n{'audio': {'array': array([ 0.        ,  0.        ,  0.        , ..., -0.00048828,\n         -0.00024414, -0.00024414], dtype=float32),\n  'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~APP_ERROR/602b9a5fbb1e6d0fbce91f52.wav',\n  'sampling_rate': 8000},\n 'intent_class': 2}\n```\n\nThere are two fields:\n\n- `audio`: a 1-dimensional `array` of the speech signal that must be called to load and resample the audio file. \n- `intent_class`: represents the class id of the speaker's intent."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 3958c7bf-8966-4308-b289-19b53e5f06eb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/audio_classification", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3958c7bf-8966-4308-b289-19b53e5f06eb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/audio_classification\", \"title\": \"Audio classification\", \"tokens\": 3086, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9bf9e9b-6893-461b-bf0e-fc7387e1155c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/audio_classification\", \"title\": \"Audio classification\", \"tokens\": 3086, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ba7d015d1f001e84999554903bbcaa4d8191bf47f199a4ab3f4bfff5b10daec4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3a895197-5dcd-4588-bddf-2fb54247d648\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/audio_classification\", \"title\": \"Audio classification\", \"tokens\": 3086, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a284fdbe04ef2afa46509f4815762a7e5dadd2ce37430838cf0df89de9a19a1f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2ce9b706-d38f-4b7f-961e-35493c694121\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"232f4c0308f321a3cfb0f11a06934bda667361a969991ac8c85f04d165c5f287\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2264, \"end_char_idx\": 4766, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9bf9e9b-6893-461b-bf0e-fc7387e1155c", "doc_id": "d9bf9e9b-6893-461b-bf0e-fc7387e1155c", "ref_doc_id": "d9bf9e9b-6893-461b-bf0e-fc7387e1155c", "chroma:document": "Remove the other columns with the [`~datasets.Dataset.remove_columns`] method:\n\n```py\n>>> minds = minds.remove_columns([\"path\", \"transcription\", \"english_transcription\", \"lang_id\"])\n```\n\nTake a look at an example now:\n\n```py\n>>> minds[\"train\"][0]\n{'audio': {'array': array([ 0.        ,  0.        ,  0.        , ..., -0.00048828,\n         -0.00024414, -0.00024414], dtype=float32),\n  'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~APP_ERROR/602b9a5fbb1e6d0fbce91f52.wav',\n  'sampling_rate': 8000},\n 'intent_class': 2}\n```\n\nThere are two fields:\n\n- `audio`: a 1-dimensional `array` of the speech signal that must be called to load and resample the audio file. \n- `intent_class`: represents the class id of the speaker's intent. \n\nTo make it easier for the model to get the label name from the label id, create a dictionary that maps the label name to an integer and vice versa:\n\n```py\n>>> labels = minds[\"train\"].features[\"intent_class\"].names\n>>> label2id, id2label = dict(), dict()\n>>> for i, label in enumerate(labels):\n...     label2id[label] = str(i)\n...     id2label[str(i)] = label\n```\n\nNow you can convert the label id to a label name:\n\n```py\n>>> id2label[str(2)]\n'app_error'\n```\n\n## Preprocess\n\nThe next step is to load a Wav2Vec2 feature extractor to process the audio signal:\n\n```py\n>>> from transformers import AutoFeatureExtractor\n\n>>> feature_extractor = AutoFeatureExtractor.from_pretrained(\"facebook/wav2vec2-base\")\n```\n\nThe MInDS-14 dataset has a sampling rate of 8000khz (you can find this information in it's [dataset card](https://huggingface.co/datasets/PolyAI/minds14)), which means you'll need to resample the dataset to 16000kHz to use the pretrained Wav2Vec2 model:\n\n```py\n>>> minds = minds.cast_column(\"audio\", Audio(sampling_rate=16_000))\n>>> minds[\"train\"][0]\n{'audio': {'array': array([ 2.2098757e-05,  4.6582241e-05, -2.2803260e-05, ...,\n         -2.8419291e-04, -2.3305941e-04, -1.1425107e-04], dtype=float32),\n  'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~APP_ERROR/602b9a5fbb1e6d0fbce91f52.wav',\n  'sampling_rate': 16000},\n 'intent_class': 2}\n```\n\nNow create a preprocessing function that:\n\n1. Calls the `audio` column to load, and if necessary, resample the audio file.\n2. Checks if the sampling rate of the audio file matches the sampling rate of the audio data a model was pretrained with."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 2ce9b706-d38f-4b7f-961e-35493c694121
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/audio_classification", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2ce9b706-d38f-4b7f-961e-35493c694121\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/audio_classification\", \"title\": \"Audio classification\", \"tokens\": 3086, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9bf9e9b-6893-461b-bf0e-fc7387e1155c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/audio_classification\", \"title\": \"Audio classification\", \"tokens\": 3086, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ba7d015d1f001e84999554903bbcaa4d8191bf47f199a4ab3f4bfff5b10daec4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3958c7bf-8966-4308-b289-19b53e5f06eb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/audio_classification\", \"title\": \"Audio classification\", \"tokens\": 3086, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4b30dc3357cb80641981e368f2e7882333e621b4c48297bbeae7b550d6b88fc7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a2205ebb-3ca1-4161-a2f3-1af1bfabe826\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2363388941310bed7533ab41df23e616f83761ce5ee0c6254ac038e9a9852316\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4568, \"end_char_idx\": 6438, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9bf9e9b-6893-461b-bf0e-fc7387e1155c", "doc_id": "d9bf9e9b-6893-461b-bf0e-fc7387e1155c", "ref_doc_id": "d9bf9e9b-6893-461b-bf0e-fc7387e1155c", "chroma:document": "Calls the `audio` column to load, and if necessary, resample the audio file.\n2. Checks if the sampling rate of the audio file matches the sampling rate of the audio data a model was pretrained with. You can find this information in the Wav2Vec2 [model card](https://huggingface.co/facebook/wav2vec2-base).\n3. Set a maximum input length to batch longer inputs without truncating them.\n\n```py\n>>> def preprocess_function(examples):\n...     audio_arrays = [x[\"array\"] for x in examples[\"audio\"]]\n...     inputs = feature_extractor(\n...         audio_arrays, sampling_rate=feature_extractor.sampling_rate, max_length=16000, truncation=True\n...     )\n...     return inputs\n```\n\nTo apply the preprocessing function over the entire dataset, use \ud83e\udd17 Datasets [`~datasets.Dataset.map`] function. You can speed up `map` by setting `batched=True` to process multiple elements of the dataset at once. Remove the columns you don't need, and rename `intent_class` to `label` because that's the name the model expects:\n\n```py\n>>> encoded_minds = minds.map(preprocess_function, remove_columns=\"audio\", batched=True)\n>>> encoded_minds = encoded_minds.rename_column(\"intent_class\", \"label\")\n```\n\n## Evaluate\n\nIncluding a metric during training is often helpful for evaluating your model's performance. You can quickly load a evaluation method with the \ud83e\udd17 [Evaluate](https://huggingface.co/docs/evaluate/index) library. For this task, load the [accuracy](https://huggingface.co/spaces/evaluate-metric/accuracy) metric (see the \ud83e\udd17 Evaluate [quick tour](https://huggingface.co/docs/evaluate/a_quick_tour) to learn more about how to load and compute a metric):\n\n```py\n>>> import evaluate\n\n>>> accuracy = evaluate.load(\"accuracy\")\n```\n\nThen create a function that passes your predictions and labels to [`~evaluate.EvaluationModule.compute`] to calculate the accuracy:\n\n```py\n>>> import numpy as np"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a2205ebb-3ca1-4161-a2f3-1af1bfabe826
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/audio_classification", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a2205ebb-3ca1-4161-a2f3-1af1bfabe826\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/audio_classification\", \"title\": \"Audio classification\", \"tokens\": 3086, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9bf9e9b-6893-461b-bf0e-fc7387e1155c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/audio_classification\", \"title\": \"Audio classification\", \"tokens\": 3086, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ba7d015d1f001e84999554903bbcaa4d8191bf47f199a4ab3f4bfff5b10daec4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2ce9b706-d38f-4b7f-961e-35493c694121\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/audio_classification\", \"title\": \"Audio classification\", \"tokens\": 3086, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"10c6d7f4673224e0508ab5d3d4132be2698cb18aee15d2b5a6e8de3622eae081\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"831257b6-668e-49ef-ba40-6bdc951e997a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"35f1521d6daf246591a96ca24d3cf27487e477f83566550ee1204ac88bd03e4b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4767, \"end_char_idx\": 8014, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9bf9e9b-6893-461b-bf0e-fc7387e1155c", "doc_id": "d9bf9e9b-6893-461b-bf0e-fc7387e1155c", "ref_doc_id": "d9bf9e9b-6893-461b-bf0e-fc7387e1155c", "chroma:document": "You can find this information in the Wav2Vec2 [model card](https://huggingface.co/facebook/wav2vec2-base).\n3. Set a maximum input length to batch longer inputs without truncating them.\n\n```py\n>>> def preprocess_function(examples):\n...     audio_arrays = [x[\"array\"] for x in examples[\"audio\"]]\n...     inputs = feature_extractor(\n...         audio_arrays, sampling_rate=feature_extractor.sampling_rate, max_length=16000, truncation=True\n...     )\n...     return inputs\n```\n\nTo apply the preprocessing function over the entire dataset, use \ud83e\udd17 Datasets [`~datasets.Dataset.map`] function. You can speed up `map` by setting `batched=True` to process multiple elements of the dataset at once. Remove the columns you don't need, and rename `intent_class` to `label` because that's the name the model expects:\n\n```py\n>>> encoded_minds = minds.map(preprocess_function, remove_columns=\"audio\", batched=True)\n>>> encoded_minds = encoded_minds.rename_column(\"intent_class\", \"label\")\n```\n\n## Evaluate\n\nIncluding a metric during training is often helpful for evaluating your model's performance. You can quickly load a evaluation method with the \ud83e\udd17 [Evaluate](https://huggingface.co/docs/evaluate/index) library. For this task, load the [accuracy](https://huggingface.co/spaces/evaluate-metric/accuracy) metric (see the \ud83e\udd17 Evaluate [quick tour](https://huggingface.co/docs/evaluate/a_quick_tour) to learn more about how to load and compute a metric):\n\n```py\n>>> import evaluate\n\n>>> accuracy = evaluate.load(\"accuracy\")\n```\n\nThen create a function that passes your predictions and labels to [`~evaluate.EvaluationModule.compute`] to calculate the accuracy:\n\n```py\n>>> import numpy as np\n\n\n>>> def compute_metrics(eval_pred):\n...     predictions = np.argmax(eval_pred.predictions, axis=1)\n...     return accuracy.compute(predictions=predictions, references=eval_pred.label_ids)\n```\n\nYour `compute_metrics` function is ready to go now, and you'll return to it when you setup your training.\n\n## Train\n\n<frameworkcontent>\n<pt>\n<Tip>\n\nIf you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!\n\n</Tip>\n\nYou're ready to start training your model now! Load Wav2Vec2 with [`AutoModelForAudioClassification`] along with the number of expected labels, and the label mappings:\n\n```py\n>>> from transformers import AutoModelForAudioClassification, TrainingArguments, Trainer\n\n>>> num_labels = len(id2label)\n>>> model = AutoModelForAudioClassification.from_pretrained(\n...     \"facebook/wav2vec2-base\", num_labels=num_labels, label2id=label2id, id2label=id2label\n... )\n```\n\nAt this point, only three steps remain:\n\n1. Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the accuracy and save the training checkpoint.\n2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.\n3. Call [`~Trainer.train`] to finetune your model."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 831257b6-668e-49ef-ba40-6bdc951e997a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/audio_classification", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"831257b6-668e-49ef-ba40-6bdc951e997a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/audio_classification\", \"title\": \"Audio classification\", \"tokens\": 3086, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9bf9e9b-6893-461b-bf0e-fc7387e1155c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/audio_classification\", \"title\": \"Audio classification\", \"tokens\": 3086, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ba7d015d1f001e84999554903bbcaa4d8191bf47f199a4ab3f4bfff5b10daec4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a2205ebb-3ca1-4161-a2f3-1af1bfabe826\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/audio_classification\", \"title\": \"Audio classification\", \"tokens\": 3086, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"71d58706f76f9abec53ec540ef2eaee4817d53aa54909893297f1c649acb0ccc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"717330e7-975d-4440-a583-1a9fcb725526\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1dd51449c610acb4d99403f94fdf1307011c0cce91d4c77c8cd14e6d63842e60\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6441, \"end_char_idx\": 9495, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9bf9e9b-6893-461b-bf0e-fc7387e1155c", "doc_id": "d9bf9e9b-6893-461b-bf0e-fc7387e1155c", "ref_doc_id": "d9bf9e9b-6893-461b-bf0e-fc7387e1155c", "chroma:document": ">>> def compute_metrics(eval_pred):\n...     predictions = np.argmax(eval_pred.predictions, axis=1)\n...     return accuracy.compute(predictions=predictions, references=eval_pred.label_ids)\n```\n\nYour `compute_metrics` function is ready to go now, and you'll return to it when you setup your training.\n\n## Train\n\n<frameworkcontent>\n<pt>\n<Tip>\n\nIf you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!\n\n</Tip>\n\nYou're ready to start training your model now! Load Wav2Vec2 with [`AutoModelForAudioClassification`] along with the number of expected labels, and the label mappings:\n\n```py\n>>> from transformers import AutoModelForAudioClassification, TrainingArguments, Trainer\n\n>>> num_labels = len(id2label)\n>>> model = AutoModelForAudioClassification.from_pretrained(\n...     \"facebook/wav2vec2-base\", num_labels=num_labels, label2id=label2id, id2label=id2label\n... )\n```\n\nAt this point, only three steps remain:\n\n1. Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the accuracy and save the training checkpoint.\n2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.\n3. Call [`~Trainer.train`] to finetune your model.\n\n\n```py\n>>> training_args = TrainingArguments(\n...     output_dir=\"my_awesome_mind_model\",\n...     eval_strategy=\"epoch\",\n...     save_strategy=\"epoch\",\n...     learning_rate=3e-5,\n...     per_device_train_batch_size=32,\n...     gradient_accumulation_steps=4,\n...     per_device_eval_batch_size=32,\n...     num_train_epochs=10,\n...     warmup_ratio=0.1,\n...     logging_steps=10,\n...     load_best_model_at_end=True,\n...     metric_for_best_model=\"accuracy\",\n...     push_to_hub=True,\n... )\n\n>>> trainer = Trainer(\n...     model=model,\n...     args=training_args,\n...     train_dataset=encoded_minds[\"train\"],\n...     eval_dataset=encoded_minds[\"test\"],\n...     tokenizer=feature_extractor,\n...     compute_metrics=compute_metrics,\n... )\n\n>>> trainer.train()\n```\n\nOnce training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:\n\n```py\n>>> trainer.push_to_hub()\n```\n</pt>\n</frameworkcontent>\n\n<Tip>\n\nFor a more in-depth example of how to finetune a model for audio classification, take a look at the corresponding [PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/audio_classification.ipynb).\n\n</Tip>\n\n## Inference\n\nGreat, now that you've finetuned a model, you can use it for inference!\n\nLoad an audio file you'd like to run inference on. Remember to resample the sampling rate of the audio file to match the sampling rate of the model if you need to!"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 717330e7-975d-4440-a583-1a9fcb725526
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/audio_classification", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"717330e7-975d-4440-a583-1a9fcb725526\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/audio_classification\", \"title\": \"Audio classification\", \"tokens\": 3086, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9bf9e9b-6893-461b-bf0e-fc7387e1155c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/audio_classification\", \"title\": \"Audio classification\", \"tokens\": 3086, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ba7d015d1f001e84999554903bbcaa4d8191bf47f199a4ab3f4bfff5b10daec4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"831257b6-668e-49ef-ba40-6bdc951e997a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/audio_classification\", \"title\": \"Audio classification\", \"tokens\": 3086, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4f67c6786fdf8c125355169c5a79e42b15d64254340495db87a7c114d5a04a2a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"aa9fe628-9d39-466e-b2c0-f93ef1896f91\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"86aa687b5bf1bc2ef0f15b47aa01a8a2708715e386f54f2bf04fc5db8cc2ed4f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8017, \"end_char_idx\": 9898, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9bf9e9b-6893-461b-bf0e-fc7387e1155c", "doc_id": "d9bf9e9b-6893-461b-bf0e-fc7387e1155c", "ref_doc_id": "d9bf9e9b-6893-461b-bf0e-fc7387e1155c", "chroma:document": "```py\n>>> training_args = TrainingArguments(\n...     output_dir=\"my_awesome_mind_model\",\n...     eval_strategy=\"epoch\",\n...     save_strategy=\"epoch\",\n...     learning_rate=3e-5,\n...     per_device_train_batch_size=32,\n...     gradient_accumulation_steps=4,\n...     per_device_eval_batch_size=32,\n...     num_train_epochs=10,\n...     warmup_ratio=0.1,\n...     logging_steps=10,\n...     load_best_model_at_end=True,\n...     metric_for_best_model=\"accuracy\",\n...     push_to_hub=True,\n... )\n\n>>> trainer = Trainer(\n...     model=model,\n...     args=training_args,\n...     train_dataset=encoded_minds[\"train\"],\n...     eval_dataset=encoded_minds[\"test\"],\n...     tokenizer=feature_extractor,\n...     compute_metrics=compute_metrics,\n... )\n\n>>> trainer.train()\n```\n\nOnce training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:\n\n```py\n>>> trainer.push_to_hub()\n```\n</pt>\n</frameworkcontent>\n\n<Tip>\n\nFor a more in-depth example of how to finetune a model for audio classification, take a look at the corresponding [PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/audio_classification.ipynb).\n\n</Tip>\n\n## Inference\n\nGreat, now that you've finetuned a model, you can use it for inference!\n\nLoad an audio file you'd like to run inference on. Remember to resample the sampling rate of the audio file to match the sampling rate of the model if you need to!\n\n```py\n>>> from datasets import load_dataset, Audio\n\n>>> dataset = load_dataset(\"PolyAI/minds14\", name=\"en-US\", split=\"train\")\n>>> dataset = dataset.cast_column(\"audio\", Audio(sampling_rate=16000))\n>>> sampling_rate = dataset.features[\"audio\"].sampling_rate\n>>> audio_file = dataset[0][\"audio\"][\"path\"]\n```\n\nThe simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: aa9fe628-9d39-466e-b2c0-f93ef1896f91
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/audio_classification", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"aa9fe628-9d39-466e-b2c0-f93ef1896f91\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/audio_classification\", \"title\": \"Audio classification\", \"tokens\": 3086, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9bf9e9b-6893-461b-bf0e-fc7387e1155c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/audio_classification\", \"title\": \"Audio classification\", \"tokens\": 3086, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ba7d015d1f001e84999554903bbcaa4d8191bf47f199a4ab3f4bfff5b10daec4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"717330e7-975d-4440-a583-1a9fcb725526\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/audio_classification\", \"title\": \"Audio classification\", \"tokens\": 3086, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1a12ef90d0cdbd5d9ea2b8e03529877017739aa32b6e293ce97714e8ec5df484\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9237, \"end_char_idx\": 11536, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9bf9e9b-6893-461b-bf0e-fc7387e1155c", "doc_id": "d9bf9e9b-6893-461b-bf0e-fc7387e1155c", "ref_doc_id": "d9bf9e9b-6893-461b-bf0e-fc7387e1155c", "chroma:document": "</Tip>\n\n## Inference\n\nGreat, now that you've finetuned a model, you can use it for inference!\n\nLoad an audio file you'd like to run inference on. Remember to resample the sampling rate of the audio file to match the sampling rate of the model if you need to!\n\n```py\n>>> from datasets import load_dataset, Audio\n\n>>> dataset = load_dataset(\"PolyAI/minds14\", name=\"en-US\", split=\"train\")\n>>> dataset = dataset.cast_column(\"audio\", Audio(sampling_rate=16000))\n>>> sampling_rate = dataset.features[\"audio\"].sampling_rate\n>>> audio_file = dataset[0][\"audio\"][\"path\"]\n```\n\nThe simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for audio classification with your model, and pass your audio file to it:\n\n```py\n>>> from transformers import pipeline\n\n>>> classifier = pipeline(\"audio-classification\", model=\"stevhliu/my_awesome_minds_model\")\n>>> classifier(audio_file)\n[\n    {'score': 0.09766869246959686, 'label': 'cash_deposit'},\n    {'score': 0.07998877018690109, 'label': 'app_error'},\n    {'score': 0.0781070664525032, 'label': 'joint_account'},\n    {'score': 0.07667109370231628, 'label': 'pay_bill'},\n    {'score': 0.0755252093076706, 'label': 'balance'}\n]\n```\n\nYou can also manually replicate the results of the `pipeline` if you'd like:\n\n<frameworkcontent>\n<pt>\nLoad a feature extractor to preprocess the audio file and return the `input` as PyTorch tensors:\n\n```py\n>>> from transformers import AutoFeatureExtractor\n\n>>> feature_extractor = AutoFeatureExtractor.from_pretrained(\"stevhliu/my_awesome_minds_model\")\n>>> inputs = feature_extractor(dataset[0][\"audio\"][\"array\"], sampling_rate=sampling_rate, return_tensors=\"pt\")\n```\n\nPass your inputs to the model and return the logits:\n\n```py\n>>> from transformers import AutoModelForAudioClassification\n\n>>> model = AutoModelForAudioClassification.from_pretrained(\"stevhliu/my_awesome_minds_model\")\n>>> with torch.no_grad():\n...     logits = model(**inputs).logits\n```\n\nGet the class with the highest probability, and use the model's `id2label` mapping to convert it to a label:\n\n```py\n>>> import torch\n\n>>> predicted_class_ids = torch.argmax(logits).item()\n>>> predicted_label = model.config.id2label[predicted_class_ids]\n>>> predicted_label\n'cash_deposit'\n```\n</pt>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: bce545eb-d48e-4260-a7c9-dc6d9764590b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/asr", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"bce545eb-d48e-4260-a7c9-dc6d9764590b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/asr\", \"title\": \"Automatic speech recognition\", \"tokens\": 3709, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"703c7679-54d0-40de-ae8b-5eb2c5753369\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/asr\", \"title\": \"Automatic speech recognition\", \"tokens\": 3709, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9b81a527b304d685e80ec5838d0e7b1a16f807e377c4b561c33af2ae14d2511a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"614974d0-aa78-451f-a2fc-61d0a24d5eb0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e7c1e3e61568da0728550fd1a57b7d0893c28f06f29b24dcf530460b4b9da5a1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2336, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "703c7679-54d0-40de-ae8b-5eb2c5753369", "doc_id": "703c7679-54d0-40de-ae8b-5eb2c5753369", "ref_doc_id": "703c7679-54d0-40de-ae8b-5eb2c5753369", "chroma:document": "# Automatic speech recognition\n\n[[open-in-colab]]\n\n<Youtube id=\"TksaY_FDgnk\"/>\n\nAutomatic speech recognition (ASR) converts a speech signal to text, mapping a sequence of audio inputs to text outputs. Virtual assistants like Siri and Alexa use ASR models to help users everyday, and there are many other useful user-facing applications like live captioning and note-taking during meetings.\n\nThis guide will show you how to:\n\n1. Finetune [Wav2Vec2](https://huggingface.co/facebook/wav2vec2-base) on the [MInDS-14](https://huggingface.co/datasets/PolyAI/minds14) dataset to transcribe audio to text.\n2. Use your finetuned model for inference.\n\n<Tip>\n\nTo see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/automatic-speech-recognition)\n\n</Tip>\n\nBefore you begin, make sure you have all the necessary libraries installed:\n\n```bash\npip install transformers datasets evaluate jiwer\n```\n\nWe encourage you to login to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to login:\n\n```py\n>>> from huggingface_hub import notebook_login\n\n>>> notebook_login()\n```\n\n## Load MInDS-14 dataset\n\nStart by loading a smaller subset of the [MInDS-14](https://huggingface.co/datasets/PolyAI/minds14) dataset from the \ud83e\udd17 Datasets library. This'll give you a chance to experiment and make sure everything works before spending more time training on the full dataset.\n\n```py\n>>> from datasets import load_dataset, Audio\n\n>>> minds = load_dataset(\"PolyAI/minds14\", name=\"en-US\", split=\"train[:100]\")\n```\n\nSplit the dataset's `train` split into a train and test set with the [`~Dataset.train_test_split`] method:\n\n```py\n>>> minds = minds.train_test_split(test_size=0.2)\n```\n\nThen take a look at the dataset:\n\n```py\n>>> minds\nDatasetDict({\n    train: Dataset({\n        features: ['path', 'audio', 'transcription', 'english_transcription', 'intent_class', 'lang_id'],\n        num_rows: 16\n    })\n    test: Dataset({\n        features: ['path', 'audio', 'transcription', 'english_transcription', 'intent_class', 'lang_id'],\n        num_rows: 4\n    })\n})\n```\n\nWhile the dataset contains a lot of useful information, like `lang_id` and `english_transcription`, you'll focus on the `audio` and `transcription` in this guide."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 614974d0-aa78-451f-a2fc-61d0a24d5eb0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/asr", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"614974d0-aa78-451f-a2fc-61d0a24d5eb0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/asr\", \"title\": \"Automatic speech recognition\", \"tokens\": 3709, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"703c7679-54d0-40de-ae8b-5eb2c5753369\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/asr\", \"title\": \"Automatic speech recognition\", \"tokens\": 3709, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9b81a527b304d685e80ec5838d0e7b1a16f807e377c4b561c33af2ae14d2511a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bce545eb-d48e-4260-a7c9-dc6d9764590b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/asr\", \"title\": \"Automatic speech recognition\", \"tokens\": 3709, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ec9e215b8493fcacc9feb652a4319b5e63dfeb86750f6892ce63d139c6f3dde8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b15dbf7c-83fc-4694-98bb-a5e9e9c3296a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"89a4d8fa95f76450a573f182b251964f92ba429790df045628917c517b962071\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1076, \"end_char_idx\": 3411, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "703c7679-54d0-40de-ae8b-5eb2c5753369", "doc_id": "703c7679-54d0-40de-ae8b-5eb2c5753369", "ref_doc_id": "703c7679-54d0-40de-ae8b-5eb2c5753369", "chroma:document": "When prompted, enter your token to login:\n\n```py\n>>> from huggingface_hub import notebook_login\n\n>>> notebook_login()\n```\n\n## Load MInDS-14 dataset\n\nStart by loading a smaller subset of the [MInDS-14](https://huggingface.co/datasets/PolyAI/minds14) dataset from the \ud83e\udd17 Datasets library. This'll give you a chance to experiment and make sure everything works before spending more time training on the full dataset.\n\n```py\n>>> from datasets import load_dataset, Audio\n\n>>> minds = load_dataset(\"PolyAI/minds14\", name=\"en-US\", split=\"train[:100]\")\n```\n\nSplit the dataset's `train` split into a train and test set with the [`~Dataset.train_test_split`] method:\n\n```py\n>>> minds = minds.train_test_split(test_size=0.2)\n```\n\nThen take a look at the dataset:\n\n```py\n>>> minds\nDatasetDict({\n    train: Dataset({\n        features: ['path', 'audio', 'transcription', 'english_transcription', 'intent_class', 'lang_id'],\n        num_rows: 16\n    })\n    test: Dataset({\n        features: ['path', 'audio', 'transcription', 'english_transcription', 'intent_class', 'lang_id'],\n        num_rows: 4\n    })\n})\n```\n\nWhile the dataset contains a lot of useful information, like `lang_id` and `english_transcription`, you'll focus on the `audio` and `transcription` in this guide. Remove the other columns with the [`~datasets.Dataset.remove_columns`] method:\n\n```py\n>>> minds = minds.remove_columns([\"english_transcription\", \"intent_class\", \"lang_id\"])\n```\n\nTake a look at the example again:\n\n```py\n>>> minds[\"train\"][0]\n{'audio': {'array': array([-0.00024414,  0.        ,  0.        , ...,  0.00024414,\n          0.00024414,  0.00024414], dtype=float32),\n  'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~APP_ERROR/602ba9e2963e11ccd901cd4f.wav',\n  'sampling_rate': 8000},\n 'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~APP_ERROR/602ba9e2963e11ccd901cd4f.wav',\n 'transcription': \"hi I'm trying to use the banking app on my phone and currently my checking and savings account balance is not refreshing\"}\n```\n\nThere are two fields:\n\n- `audio`: a 1-dimensional `array` of the speech signal that must be called to load and resample the audio file.\n- `transcription`: the target text."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: b15dbf7c-83fc-4694-98bb-a5e9e9c3296a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/asr", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b15dbf7c-83fc-4694-98bb-a5e9e9c3296a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/asr\", \"title\": \"Automatic speech recognition\", \"tokens\": 3709, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"703c7679-54d0-40de-ae8b-5eb2c5753369\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/asr\", \"title\": \"Automatic speech recognition\", \"tokens\": 3709, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9b81a527b304d685e80ec5838d0e7b1a16f807e377c4b561c33af2ae14d2511a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"614974d0-aa78-451f-a2fc-61d0a24d5eb0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/asr\", \"title\": \"Automatic speech recognition\", \"tokens\": 3709, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8cef1a99ad4ce9e5ae4b54b9b9a2f22de9dc47f5028f43c49f491ddb334b3046\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e7f2aab3-4258-4e9c-92df-3be7bf2f7f61\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bf6f46d1a3e333cf23c787f0d8f171a04441907d7cdb51ca46ac1043a9043700\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2337, \"end_char_idx\": 4771, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "703c7679-54d0-40de-ae8b-5eb2c5753369", "doc_id": "703c7679-54d0-40de-ae8b-5eb2c5753369", "ref_doc_id": "703c7679-54d0-40de-ae8b-5eb2c5753369", "chroma:document": "Remove the other columns with the [`~datasets.Dataset.remove_columns`] method:\n\n```py\n>>> minds = minds.remove_columns([\"english_transcription\", \"intent_class\", \"lang_id\"])\n```\n\nTake a look at the example again:\n\n```py\n>>> minds[\"train\"][0]\n{'audio': {'array': array([-0.00024414,  0.        ,  0.        , ...,  0.00024414,\n          0.00024414,  0.00024414], dtype=float32),\n  'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~APP_ERROR/602ba9e2963e11ccd901cd4f.wav',\n  'sampling_rate': 8000},\n 'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~APP_ERROR/602ba9e2963e11ccd901cd4f.wav',\n 'transcription': \"hi I'm trying to use the banking app on my phone and currently my checking and savings account balance is not refreshing\"}\n```\n\nThere are two fields:\n\n- `audio`: a 1-dimensional `array` of the speech signal that must be called to load and resample the audio file.\n- `transcription`: the target text.\n\n## Preprocess\n\nThe next step is to load a Wav2Vec2 processor to process the audio signal:\n\n```py\n>>> from transformers import AutoProcessor\n\n>>> processor = AutoProcessor.from_pretrained(\"facebook/wav2vec2-base\")\n```\n\nThe MInDS-14 dataset has a sampling rate of 8000kHz (you can find this information in its [dataset card](https://huggingface.co/datasets/PolyAI/minds14)), which means you'll need to resample the dataset to 16000kHz to use the pretrained Wav2Vec2 model:\n\n```py\n>>> minds = minds.cast_column(\"audio\", Audio(sampling_rate=16_000))\n>>> minds[\"train\"][0]\n{'audio': {'array': array([-2.38064706e-04, -1.58618059e-04, -5.43987835e-06, ...,\n          2.78103951e-04,  2.38446111e-04,  1.18740834e-04], dtype=float32),\n  'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~APP_ERROR/602ba9e2963e11ccd901cd4f.wav',\n  'sampling_rate': 16000},\n 'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~APP_ERROR/602ba9e2963e11ccd901cd4f.wav',\n 'transcription': \"hi I'm trying to use the banking app on my phone and currently my checking and savings account balance is not refreshing\"}\n```\n\nAs you can see in the `transcription` above, the text contains a mix of upper and lowercase characters."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e7f2aab3-4258-4e9c-92df-3be7bf2f7f61
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/asr", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e7f2aab3-4258-4e9c-92df-3be7bf2f7f61\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/asr\", \"title\": \"Automatic speech recognition\", \"tokens\": 3709, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"703c7679-54d0-40de-ae8b-5eb2c5753369\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/asr\", \"title\": \"Automatic speech recognition\", \"tokens\": 3709, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9b81a527b304d685e80ec5838d0e7b1a16f807e377c4b561c33af2ae14d2511a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b15dbf7c-83fc-4694-98bb-a5e9e9c3296a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/asr\", \"title\": \"Automatic speech recognition\", \"tokens\": 3709, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ff7ad6654f87432bd33b52401634650199b7a32738873cff083ee4f1a2de458b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"50dc72e3-1d06-433b-bb63-74c04b33487b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8e0c90daed7ad60e4ab66ba8f62a516c42f87c277e79b7bb1c6c9e9e2de4e733\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4772, \"end_char_idx\": 6628, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "703c7679-54d0-40de-ae8b-5eb2c5753369", "doc_id": "703c7679-54d0-40de-ae8b-5eb2c5753369", "ref_doc_id": "703c7679-54d0-40de-ae8b-5eb2c5753369", "chroma:document": "The Wav2Vec2 tokenizer is only trained on uppercase characters so you'll need to make sure the text matches the tokenizer's vocabulary:\n\n```py\n>>> def uppercase(example):\n...     return {\"transcription\": example[\"transcription\"].upper()}\n\n\n>>> minds = minds.map(uppercase)\n```\n\nNow create a preprocessing function that:\n\n1. Calls the `audio` column to load and resample the audio file.\n2. Extracts the `input_values` from the audio file and tokenize the `transcription` column with the processor.\n\n```py\n>>> def prepare_dataset(batch):\n...     audio = batch[\"audio\"]\n...     batch = processor(audio[\"array\"], sampling_rate=audio[\"sampling_rate\"], text=batch[\"transcription\"])\n...     batch[\"input_length\"] = len(batch[\"input_values\"][0])\n...     return batch\n```\n\nTo apply the preprocessing function over the entire dataset, use \ud83e\udd17 Datasets [`~datasets.Dataset.map`] function. You can speed up `map` by increasing the number of processes with the `num_proc` parameter. Remove the columns you don't need with the [`~datasets.Dataset.remove_columns`] method:\n\n```py\n>>> encoded_minds = minds.map(prepare_dataset, remove_columns=minds.column_names[\"train\"], num_proc=4)\n```\n\n\ud83e\udd17 Transformers doesn't have a data collator for ASR, so you'll need to adapt the [`DataCollatorWithPadding`] to create a batch of examples. It'll also dynamically pad your text and labels to the length of the longest element in its batch (instead of the entire dataset) so they are a uniform length. While it is possible to pad your text in the `tokenizer` function by setting `padding=True`, dynamic padding is more efficient.\n\nUnlike other data collators, this specific data collator needs to apply a different padding method to `input_values` and `labels`:\n\n```py\n>>> import torch\n\n>>> from dataclasses import dataclass, field\n>>> from typing import Any, Dict, List, Optional, Union"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 50dc72e3-1d06-433b-bb63-74c04b33487b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/asr", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"50dc72e3-1d06-433b-bb63-74c04b33487b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/asr\", \"title\": \"Automatic speech recognition\", \"tokens\": 3709, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"703c7679-54d0-40de-ae8b-5eb2c5753369\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/asr\", \"title\": \"Automatic speech recognition\", \"tokens\": 3709, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9b81a527b304d685e80ec5838d0e7b1a16f807e377c4b561c33af2ae14d2511a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e7f2aab3-4258-4e9c-92df-3be7bf2f7f61\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/asr\", \"title\": \"Automatic speech recognition\", \"tokens\": 3709, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"14b9337583825b7db8adf6d1e8da7197557cf78f1b9da28a6ad41397aa7c0660\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6ae0b745-d0ce-408e-b737-c41a820055f9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"37743a80c704723b1384deea748f086bf401348d80219d43ac4071072a678f90\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5012, \"end_char_idx\": 8483, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "703c7679-54d0-40de-ae8b-5eb2c5753369", "doc_id": "703c7679-54d0-40de-ae8b-5eb2c5753369", "ref_doc_id": "703c7679-54d0-40de-ae8b-5eb2c5753369", "chroma:document": ">>> minds = minds.map(uppercase)\n```\n\nNow create a preprocessing function that:\n\n1. Calls the `audio` column to load and resample the audio file.\n2. Extracts the `input_values` from the audio file and tokenize the `transcription` column with the processor.\n\n```py\n>>> def prepare_dataset(batch):\n...     audio = batch[\"audio\"]\n...     batch = processor(audio[\"array\"], sampling_rate=audio[\"sampling_rate\"], text=batch[\"transcription\"])\n...     batch[\"input_length\"] = len(batch[\"input_values\"][0])\n...     return batch\n```\n\nTo apply the preprocessing function over the entire dataset, use \ud83e\udd17 Datasets [`~datasets.Dataset.map`] function. You can speed up `map` by increasing the number of processes with the `num_proc` parameter. Remove the columns you don't need with the [`~datasets.Dataset.remove_columns`] method:\n\n```py\n>>> encoded_minds = minds.map(prepare_dataset, remove_columns=minds.column_names[\"train\"], num_proc=4)\n```\n\n\ud83e\udd17 Transformers doesn't have a data collator for ASR, so you'll need to adapt the [`DataCollatorWithPadding`] to create a batch of examples. It'll also dynamically pad your text and labels to the length of the longest element in its batch (instead of the entire dataset) so they are a uniform length. While it is possible to pad your text in the `tokenizer` function by setting `padding=True`, dynamic padding is more efficient.\n\nUnlike other data collators, this specific data collator needs to apply a different padding method to `input_values` and `labels`:\n\n```py\n>>> import torch\n\n>>> from dataclasses import dataclass, field\n>>> from typing import Any, Dict, List, Optional, Union\n\n\n>>> @dataclass\n... class DataCollatorCTCWithPadding:\n...     processor: AutoProcessor\n...     padding: Union[bool, str] = \"longest\"\n\n...     def __call__(self, features: List[Dict[str, Union[List[int], torch.Tensor]]]) -> Dict[str, torch.Tensor]:\n...         # split inputs and labels since they have to be of different lengths and need\n...         # different padding methods\n...         input_features = [{\"input_values\": feature[\"input_values\"][0]} for feature in features]\n...         label_features = [{\"input_ids\": feature[\"labels\"]} for feature in features]\n\n...         batch = self.processor.pad(input_features, padding=self.padding, return_tensors=\"pt\")\n\n...         labels_batch = self.processor.pad(labels=label_features, padding=self.padding, return_tensors=\"pt\")\n\n...         # replace padding with -100 to ignore loss correctly\n...         labels = labels_batch[\"input_ids\"].masked_fill(labels_batch.attention_mask.ne(1), -100)\n\n...         batch[\"labels\"] = labels\n\n...         return batch\n```\n\nNow instantiate your `DataCollatorForCTCWithPadding`:\n\n```py\n>>> data_collator = DataCollatorCTCWithPadding(processor=processor, padding=\"longest\")\n```\n\n## Evaluate\n\nIncluding a metric during training is often helpful for evaluating your model's performance. You can quickly load a evaluation method with the \ud83e\udd17 [Evaluate](https://huggingface.co/docs/evaluate/index) library. For this task, load the [word error rate](https://huggingface.co/spaces/evaluate-metric/wer) (WER) metric (see the \ud83e\udd17 Evaluate [quick tour](https://huggingface.co/docs/evaluate/a_quick_tour) to learn more about how to load and compute a metric):\n\n```py\n>>> import evaluate\n\n>>> wer = evaluate.load(\"wer\")\n```\n\nThen create a function that passes your predictions and labels to [`~evaluate.EvaluationModule.compute`] to calculate the WER:\n\n```py\n>>> import numpy as np"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6ae0b745-d0ce-408e-b737-c41a820055f9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/asr", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6ae0b745-d0ce-408e-b737-c41a820055f9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/asr\", \"title\": \"Automatic speech recognition\", \"tokens\": 3709, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"703c7679-54d0-40de-ae8b-5eb2c5753369\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/asr\", \"title\": \"Automatic speech recognition\", \"tokens\": 3709, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9b81a527b304d685e80ec5838d0e7b1a16f807e377c4b561c33af2ae14d2511a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"50dc72e3-1d06-433b-bb63-74c04b33487b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/asr\", \"title\": \"Automatic speech recognition\", \"tokens\": 3709, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fdfa0145076ffe84a12cc12156589ff20c46828cd2f172bdeb70ae22f6eb89c0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"07f8f36f-164a-4576-a13f-cb2989d6ecdc\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d68f279720a591c69093189a95e8d14ae4bba946208c081bb32e621ea114fa19\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8486, \"end_char_idx\": 10332, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "703c7679-54d0-40de-ae8b-5eb2c5753369", "doc_id": "703c7679-54d0-40de-ae8b-5eb2c5753369", "ref_doc_id": "703c7679-54d0-40de-ae8b-5eb2c5753369", "chroma:document": ">>> def compute_metrics(pred):\n...     pred_logits = pred.predictions\n...     pred_ids = np.argmax(pred_logits, axis=-1)\n\n...     pred.label_ids[pred.label_ids == -100] = processor.tokenizer.pad_token_id\n\n...     pred_str = processor.batch_decode(pred_ids)\n...     label_str = processor.batch_decode(pred.label_ids, group_tokens=False)\n\n...     wer = wer.compute(predictions=pred_str, references=label_str)\n\n...     return {\"wer\": wer}\n```\n\nYour `compute_metrics` function is ready to go now, and you'll return to it when you setup your training.\n\n## Train\n\n<frameworkcontent>\n<pt>\n<Tip>\n\nIf you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!\n\n</Tip>\n\nYou're ready to start training your model now! Load Wav2Vec2 with [`AutoModelForCTC`]. Specify the reduction to apply with the `ctc_loss_reduction` parameter. It is often better to use the average instead of the default summation:\n\n```py\n>>> from transformers import AutoModelForCTC, TrainingArguments, Trainer\n\n>>> model = AutoModelForCTC.from_pretrained(\n...     \"facebook/wav2vec2-base\",\n...     ctc_loss_reduction=\"mean\",\n...     pad_token_id=processor.tokenizer.pad_token_id,\n... )\n```\n\nAt this point, only three steps remain:\n\n1. Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the WER and save the training checkpoint.\n2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.\n3. Call [`~Trainer.train`] to finetune your model."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 07f8f36f-164a-4576-a13f-cb2989d6ecdc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/asr", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"07f8f36f-164a-4576-a13f-cb2989d6ecdc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/asr\", \"title\": \"Automatic speech recognition\", \"tokens\": 3709, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"703c7679-54d0-40de-ae8b-5eb2c5753369\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/asr\", \"title\": \"Automatic speech recognition\", \"tokens\": 3709, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9b81a527b304d685e80ec5838d0e7b1a16f807e377c4b561c33af2ae14d2511a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6ae0b745-d0ce-408e-b737-c41a820055f9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/asr\", \"title\": \"Automatic speech recognition\", \"tokens\": 3709, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"027e886a7a71540baee70e2e866de0336442677f8c5a05b2151e67369414c710\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d51960a2-2dc4-40f2-8fa1-bc70e5345a08\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"340b985da6e92e10e3219dbb60d41f5c1cd778a5bb6c4cf97cb7df5a6a23df25\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9034, \"end_char_idx\": 12011, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "703c7679-54d0-40de-ae8b-5eb2c5753369", "doc_id": "703c7679-54d0-40de-ae8b-5eb2c5753369", "ref_doc_id": "703c7679-54d0-40de-ae8b-5eb2c5753369", "chroma:document": "## Train\n\n<frameworkcontent>\n<pt>\n<Tip>\n\nIf you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!\n\n</Tip>\n\nYou're ready to start training your model now! Load Wav2Vec2 with [`AutoModelForCTC`]. Specify the reduction to apply with the `ctc_loss_reduction` parameter. It is often better to use the average instead of the default summation:\n\n```py\n>>> from transformers import AutoModelForCTC, TrainingArguments, Trainer\n\n>>> model = AutoModelForCTC.from_pretrained(\n...     \"facebook/wav2vec2-base\",\n...     ctc_loss_reduction=\"mean\",\n...     pad_token_id=processor.tokenizer.pad_token_id,\n... )\n```\n\nAt this point, only three steps remain:\n\n1. Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the WER and save the training checkpoint.\n2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.\n3. Call [`~Trainer.train`] to finetune your model.\n\n```py\n>>> training_args = TrainingArguments(\n...     output_dir=\"my_awesome_asr_mind_model\",\n...     per_device_train_batch_size=8,\n...     gradient_accumulation_steps=2,\n...     learning_rate=1e-5,\n...     warmup_steps=500,\n...     max_steps=2000,\n...     gradient_checkpointing=True,\n...     fp16=True,\n...     group_by_length=True,\n...     eval_strategy=\"steps\",\n...     per_device_eval_batch_size=8,\n...     save_steps=1000,\n...     eval_steps=1000,\n...     logging_steps=25,\n...     load_best_model_at_end=True,\n...     metric_for_best_model=\"wer\",\n...     greater_is_better=False,\n...     push_to_hub=True,\n... )\n\n>>> trainer = Trainer(\n...     model=model,\n...     args=training_args,\n...     train_dataset=encoded_minds[\"train\"],\n...     eval_dataset=encoded_minds[\"test\"],\n...     tokenizer=processor,\n...     data_collator=data_collator,\n...     compute_metrics=compute_metrics,\n... )\n\n>>> trainer.train()\n```\n\nOnce training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:\n\n```py\n>>> trainer.push_to_hub()\n```\n</pt>\n</frameworkcontent>\n\n<Tip>\n\nFor a more in-depth example of how to finetune a model for automatic speech recognition, take a look at this blog [post](https://huggingface.co/blog/fine-tune-wav2vec2-english) for English ASR and this [post](https://huggingface.co/blog/fine-tune-xlsr-wav2vec2) for multilingual ASR.\n\n</Tip>\n\n## Inference\n\nGreat, now that you've finetuned a model, you can use it for inference!\n\nLoad an audio file you'd like to run inference on. Remember to resample the sampling rate of the audio file to match the sampling rate of the model if you need to!"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: d51960a2-2dc4-40f2-8fa1-bc70e5345a08
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/asr", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d51960a2-2dc4-40f2-8fa1-bc70e5345a08\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/asr\", \"title\": \"Automatic speech recognition\", \"tokens\": 3709, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"703c7679-54d0-40de-ae8b-5eb2c5753369\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/asr\", \"title\": \"Automatic speech recognition\", \"tokens\": 3709, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9b81a527b304d685e80ec5838d0e7b1a16f807e377c4b561c33af2ae14d2511a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"07f8f36f-164a-4576-a13f-cb2989d6ecdc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/asr\", \"title\": \"Automatic speech recognition\", \"tokens\": 3709, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ca21b2001796f10f7d99caf9f4fc627f6a4fab25468e5a761c67b517e6507a26\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11753, \"end_char_idx\": 13976, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "703c7679-54d0-40de-ae8b-5eb2c5753369", "doc_id": "703c7679-54d0-40de-ae8b-5eb2c5753369", "ref_doc_id": "703c7679-54d0-40de-ae8b-5eb2c5753369", "chroma:document": "</Tip>\n\n## Inference\n\nGreat, now that you've finetuned a model, you can use it for inference!\n\nLoad an audio file you'd like to run inference on. Remember to resample the sampling rate of the audio file to match the sampling rate of the model if you need to!\n\n```py\n>>> from datasets import load_dataset, Audio\n\n>>> dataset = load_dataset(\"PolyAI/minds14\", \"en-US\", split=\"train\")\n>>> dataset = dataset.cast_column(\"audio\", Audio(sampling_rate=16000))\n>>> sampling_rate = dataset.features[\"audio\"].sampling_rate\n>>> audio_file = dataset[0][\"audio\"][\"path\"]\n```\n\nThe simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for automatic speech recognition with your model, and pass your audio file to it:\n\n```py\n>>> from transformers import pipeline\n\n>>> transcriber = pipeline(\"automatic-speech-recognition\", model=\"stevhliu/my_awesome_asr_minds_model\")\n>>> transcriber(audio_file)\n{'text': 'I WOUD LIKE O SET UP JOINT ACOUNT WTH Y PARTNER'}\n```\n\n<Tip>\n\nThe transcription is decent, but it could be better! Try finetuning your model on more examples to get even better results!\n\n</Tip>\n\nYou can also manually replicate the results of the `pipeline` if you'd like:\n\n<frameworkcontent>\n<pt>\nLoad a processor to preprocess the audio file and transcription and return the `input` as PyTorch tensors:\n\n```py\n>>> from transformers import AutoProcessor\n\n>>> processor = AutoProcessor.from_pretrained(\"stevhliu/my_awesome_asr_mind_model\")\n>>> inputs = processor(dataset[0][\"audio\"][\"array\"], sampling_rate=sampling_rate, return_tensors=\"pt\")\n```\n\nPass your inputs to the model and return the logits:\n\n```py\n>>> from transformers import AutoModelForCTC\n\n>>> model = AutoModelForCTC.from_pretrained(\"stevhliu/my_awesome_asr_mind_model\")\n>>> with torch.no_grad():\n...     logits = model(**inputs).logits\n```\n\nGet the predicted `input_ids` with the highest probability, and use the processor to decode the predicted `input_ids` back into text:\n\n```py\n>>> import torch\n\n>>> predicted_ids = torch.argmax(logits, dim=-1)\n>>> transcription = processor.batch_decode(predicted_ids)\n>>> transcription\n['I WOUL LIKE O SET UP JOINT ACOUNT WTH Y PARTNER']\n```\n</pt>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0a73ddb2-93e7-4a94-936d-7b8d4ea2fb4c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0a73ddb2-93e7-4a94-936d-7b8d4ea2fb4c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/language_modeling\", \"title\": \"Causal language modeling\", \"tokens\": 4356, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/language_modeling\", \"title\": \"Causal language modeling\", \"tokens\": 4356, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"326c9c538adbd0a701b7618bacf6e60d06fa9b09a486befaf8143c411e1560e8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1d2e96b5-1bca-40b2-84d0-feb7afd5e04c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7733f907b8600bdc0ea7c332460b88a5192f4933d5b80079932f7055ba69b2e3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3113, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "doc_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "ref_doc_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "chroma:document": "# Causal language modeling\n\n[[open-in-colab]]\n\nThere are two types of language modeling, causal and masked. This guide illustrates causal language modeling.\nCausal language models are frequently used for text generation. You can use these models for creative applications like\nchoosing your own text adventure or an intelligent coding assistant like Copilot or CodeParrot.\n\n<Youtube id=\"Vpjb1lu0MDk\"/>\n\nCausal language modeling predicts the next token in a sequence of tokens, and the model can only attend to tokens on\nthe left. This means the model cannot see future tokens. GPT-2 is an example of a causal language model.\n\nThis guide will show you how to:\n\n1. Finetune [DistilGPT2](https://huggingface.co/distilbert/distilgpt2) on the [r/askscience](https://www.reddit.com/r/askscience/) subset of the [ELI5](https://huggingface.co/datasets/eli5) dataset.\n2. Use your finetuned model for inference.\n\n<Tip>\n\nTo see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/text-generation)\n\n</Tip>\n\nBefore you begin, make sure you have all the necessary libraries installed:\n\n```bash\npip install transformers datasets evaluate\n```\n\nWe encourage you to log in to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to log in:\n\n```py\n>>> from huggingface_hub import notebook_login\n\n>>> notebook_login()\n```\n\n## Load ELI5 dataset\n\nStart by loading the first 5000 examples from the [ELI5-Category](https://huggingface.co/datasets/eli5_category) dataset with the \ud83e\udd17 Datasets library. This'll give you a chance to experiment and make sure everything works before spending more time training on the full dataset.\n\n```py\n>>> from datasets import load_dataset\n\n>>> eli5 = load_dataset(\"eli5_category\", split=\"train[:5000]\")\n```\n\nSplit the dataset's `train` split into a train and test set with the [`~datasets.Dataset.train_test_split`] method:\n\n```py\n>>> eli5 = eli5.train_test_split(test_size=0.2)\n```\n\nThen take a look at an example:\n\n```py\n>>> eli5[\"train\"][0]\n{'q_id': '7h191n',\n 'title': 'What does the tax bill that was passed today mean? How will it affect Americans in each tax bracket?',\n 'selftext': '',\n 'category': 'Economics',\n 'subreddit': 'explainlikeimfive',\n 'answers': {'a_id': ['dqnds8l', 'dqnd1jl', 'dqng3i1', 'dqnku5x'],\n  'text': [\"The tax bill is 500 pages long and there were a lot of changes still going on right to the end. It's not just an adjustment to the income tax brackets, it's a whole bunch of changes. As such there is no good answer to your question. The big take aways are: - Big reduction in corporate income tax rate will make large companies very happy. - Pass through rate change will make certain styles of business (law firms, hedge funds) extremely happy - Income tax changes are moderate, and are set to expire (though it's the kind of thing that might just always get re-applied without being made permanent) - People in high tax states (California, New York) lose out, and many of them will end up with their taxes raised.\",\n   'None yet."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 1d2e96b5-1bca-40b2-84d0-feb7afd5e04c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1d2e96b5-1bca-40b2-84d0-feb7afd5e04c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/language_modeling\", \"title\": \"Causal language modeling\", \"tokens\": 4356, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/language_modeling\", \"title\": \"Causal language modeling\", \"tokens\": 4356, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"326c9c538adbd0a701b7618bacf6e60d06fa9b09a486befaf8143c411e1560e8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0a73ddb2-93e7-4a94-936d-7b8d4ea2fb4c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/language_modeling\", \"title\": \"Causal language modeling\", \"tokens\": 4356, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b1b1634cf1ee551b3d7d985715eb4f4b1d78f64a412f6e10bd766fd494912d55\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"34390285-d521-4e02-b786-77e05e7cacb7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8ee1073c72965596d03a49fb92a3bc923905d5c8f3b23a97e27b797a31d18236\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1615, \"end_char_idx\": 4536, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "doc_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "ref_doc_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "chroma:document": "This'll give you a chance to experiment and make sure everything works before spending more time training on the full dataset.\n\n```py\n>>> from datasets import load_dataset\n\n>>> eli5 = load_dataset(\"eli5_category\", split=\"train[:5000]\")\n```\n\nSplit the dataset's `train` split into a train and test set with the [`~datasets.Dataset.train_test_split`] method:\n\n```py\n>>> eli5 = eli5.train_test_split(test_size=0.2)\n```\n\nThen take a look at an example:\n\n```py\n>>> eli5[\"train\"][0]\n{'q_id': '7h191n',\n 'title': 'What does the tax bill that was passed today mean? How will it affect Americans in each tax bracket?',\n 'selftext': '',\n 'category': 'Economics',\n 'subreddit': 'explainlikeimfive',\n 'answers': {'a_id': ['dqnds8l', 'dqnd1jl', 'dqng3i1', 'dqnku5x'],\n  'text': [\"The tax bill is 500 pages long and there were a lot of changes still going on right to the end. It's not just an adjustment to the income tax brackets, it's a whole bunch of changes. As such there is no good answer to your question. The big take aways are: - Big reduction in corporate income tax rate will make large companies very happy. - Pass through rate change will make certain styles of business (law firms, hedge funds) extremely happy - Income tax changes are moderate, and are set to expire (though it's the kind of thing that might just always get re-applied without being made permanent) - People in high tax states (California, New York) lose out, and many of them will end up with their taxes raised.\",\n   'None yet. It has to be reconciled with a vastly different house bill and then passed again.',\n   'Also: does this apply to 2017 taxes? Or does it start with 2018 taxes?',\n   'This article explains both the House and senate bills, including the proposed changes to your income taxes based on your income level. URL_0'],\n  'score': [21, 19, 5, 3],\n  'text_urls': [[],\n   [],\n   [],\n   ['https://www.investopedia.com/news/trumps-tax-reform-what-can-be-done/']]},\n 'title_urls': ['url'],\n 'selftext_urls': ['url']}\n```\n\nWhile this may look like a lot, you're only really interested in the `text` field. What's cool about language modeling\ntasks is you don't need labels (also known as an unsupervised task) because the next word *is* the label.\n\n## Preprocess\n\n<Youtube id=\"ma1TrR7gE7I\"/>\n\nThe next step is to load a DistilGPT2 tokenizer to process the `text` subfield:\n\n```py\n>>> from transformers import AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"distilbert/distilgpt2\")\n```\n\nYou'll notice from the example above, the `text` field is actually nested inside `answers`. This means you'll need to\nextract the `text` subfield from its nested structure with the [`flatten`](https://huggingface.co/docs/datasets/process#flatten) method:\n\n```py\n>>> eli5 = eli5.flatten()\n>>> eli5[\"train\"][0]\n{'q_id': '7h191n',\n 'title': 'What does the tax bill that was passed today mean? How will it affect Americans in each tax bracket?"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 34390285-d521-4e02-b786-77e05e7cacb7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"34390285-d521-4e02-b786-77e05e7cacb7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/language_modeling\", \"title\": \"Causal language modeling\", \"tokens\": 4356, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/language_modeling\", \"title\": \"Causal language modeling\", \"tokens\": 4356, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"326c9c538adbd0a701b7618bacf6e60d06fa9b09a486befaf8143c411e1560e8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1d2e96b5-1bca-40b2-84d0-feb7afd5e04c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/language_modeling\", \"title\": \"Causal language modeling\", \"tokens\": 4356, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"540d187167a880ff0ed7626520f912705aae7a9e53183d9f5e73fec53dc18946\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9b148aa9-9b52-45e4-9768-d4cb99cabbb9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"60ead5054e0da33219d7c8bed13b92301992a131cc258bfe42ccb382dfe9c2f5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3097, \"end_char_idx\": 5727, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "doc_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "ref_doc_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "chroma:document": "\",\n   'None yet. It has to be reconciled with a vastly different house bill and then passed again.',\n   'Also: does this apply to 2017 taxes? Or does it start with 2018 taxes?',\n   'This article explains both the House and senate bills, including the proposed changes to your income taxes based on your income level. URL_0'],\n  'score': [21, 19, 5, 3],\n  'text_urls': [[],\n   [],\n   [],\n   ['https://www.investopedia.com/news/trumps-tax-reform-what-can-be-done/']]},\n 'title_urls': ['url'],\n 'selftext_urls': ['url']}\n```\n\nWhile this may look like a lot, you're only really interested in the `text` field. What's cool about language modeling\ntasks is you don't need labels (also known as an unsupervised task) because the next word *is* the label.\n\n## Preprocess\n\n<Youtube id=\"ma1TrR7gE7I\"/>\n\nThe next step is to load a DistilGPT2 tokenizer to process the `text` subfield:\n\n```py\n>>> from transformers import AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"distilbert/distilgpt2\")\n```\n\nYou'll notice from the example above, the `text` field is actually nested inside `answers`. This means you'll need to\nextract the `text` subfield from its nested structure with the [`flatten`](https://huggingface.co/docs/datasets/process#flatten) method:\n\n```py\n>>> eli5 = eli5.flatten()\n>>> eli5[\"train\"][0]\n{'q_id': '7h191n',\n 'title': 'What does the tax bill that was passed today mean? How will it affect Americans in each tax bracket?',\n 'selftext': '',\n 'category': 'Economics',\n 'subreddit': 'explainlikeimfive',\n 'answers.a_id': ['dqnds8l', 'dqnd1jl', 'dqng3i1', 'dqnku5x'],\n 'answers.text': [\"The tax bill is 500 pages long and there were a lot of changes still going on right to the end. It's not just an adjustment to the income tax brackets, it's a whole bunch of changes. As such there is no good answer to your question. The big take aways are: - Big reduction in corporate income tax rate will make large companies very happy. - Pass through rate change will make certain styles of business (law firms, hedge funds) extremely happy - Income tax changes are moderate, and are set to expire (though it's the kind of thing that might just always get re-applied without being made permanent) - People in high tax states (California, New York) lose out, and many of them will end up with their taxes raised.\",\n  'None yet. It has to be reconciled with a vastly different house bill and then passed again.',\n  'Also: does this apply to 2017 taxes? Or does it start with 2018 taxes?',\n  'This article explains both the House and senate bills, including the proposed changes to your income taxes based on your income level."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 9b148aa9-9b52-45e4-9768-d4cb99cabbb9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9b148aa9-9b52-45e4-9768-d4cb99cabbb9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/language_modeling\", \"title\": \"Causal language modeling\", \"tokens\": 4356, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/language_modeling\", \"title\": \"Causal language modeling\", \"tokens\": 4356, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"326c9c538adbd0a701b7618bacf6e60d06fa9b09a486befaf8143c411e1560e8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"34390285-d521-4e02-b786-77e05e7cacb7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/language_modeling\", \"title\": \"Causal language modeling\", \"tokens\": 4356, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c93659434cb8ff9ddf926b3ed7536987c6dd827a5b470b6f3c9659d436f7b53a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9885de43-d86a-4b20-abcd-a29250cbfbc2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1ea3a628a8690de0cc7fcf1cc4a76cf98dfe790e6ba0b2ce26bc629dac64bb5e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4189, \"end_char_idx\": 7298, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "doc_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "ref_doc_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "chroma:document": "This means you'll need to\nextract the `text` subfield from its nested structure with the [`flatten`](https://huggingface.co/docs/datasets/process#flatten) method:\n\n```py\n>>> eli5 = eli5.flatten()\n>>> eli5[\"train\"][0]\n{'q_id': '7h191n',\n 'title': 'What does the tax bill that was passed today mean? How will it affect Americans in each tax bracket?',\n 'selftext': '',\n 'category': 'Economics',\n 'subreddit': 'explainlikeimfive',\n 'answers.a_id': ['dqnds8l', 'dqnd1jl', 'dqng3i1', 'dqnku5x'],\n 'answers.text': [\"The tax bill is 500 pages long and there were a lot of changes still going on right to the end. It's not just an adjustment to the income tax brackets, it's a whole bunch of changes. As such there is no good answer to your question. The big take aways are: - Big reduction in corporate income tax rate will make large companies very happy. - Pass through rate change will make certain styles of business (law firms, hedge funds) extremely happy - Income tax changes are moderate, and are set to expire (though it's the kind of thing that might just always get re-applied without being made permanent) - People in high tax states (California, New York) lose out, and many of them will end up with their taxes raised.\",\n  'None yet. It has to be reconciled with a vastly different house bill and then passed again.',\n  'Also: does this apply to 2017 taxes? Or does it start with 2018 taxes?',\n  'This article explains both the House and senate bills, including the proposed changes to your income taxes based on your income level. URL_0'],\n 'answers.score': [21, 19, 5, 3],\n 'answers.text_urls': [[],\n  [],\n  [],\n  ['https://www.investopedia.com/news/trumps-tax-reform-what-can-be-done/']],\n 'title_urls': ['url'],\n 'selftext_urls': ['url']}\n```\n\nEach subfield is now a separate column as indicated by the `answers` prefix, and the `text` field is a list now. Instead\nof tokenizing each sentence separately, convert the list to a string so you can jointly tokenize them.\n\nHere is a first preprocessing function to join the list of strings for each example and tokenize the result:\n\n```py\n>>> def preprocess_function(examples):\n...     return tokenizer([\" \".join(x) for x in examples[\"answers.text\"]])\n```\n\nTo apply this preprocessing function over the entire dataset, use the \ud83e\udd17 Datasets [`~datasets.Dataset.map`] method. You can speed up the `map` function by setting `batched=True` to process multiple elements of the dataset at once, and increasing the number of processes with `num_proc`. Remove any columns you don't need:\n\n```py\n>>> tokenized_eli5 = eli5.map(\n...     preprocess_function,\n...     batched=True,\n...     num_proc=4,\n...     remove_columns=eli5[\"train\"].column_names,\n... )\n```\n\nThis dataset contains the token sequences, but some of these are longer than the maximum input length for the model.\n\nYou can now use a second preprocessing function to\n\n- concatenate all the sequences\n- split the concatenated sequences into shorter chunks defined by `block_size`, which should be both shorter than the maximum input length and short enough for your GPU RAM.\n\n```py\n>>> block_size = 128"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 9885de43-d86a-4b20-abcd-a29250cbfbc2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9885de43-d86a-4b20-abcd-a29250cbfbc2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/language_modeling\", \"title\": \"Causal language modeling\", \"tokens\": 4356, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/language_modeling\", \"title\": \"Causal language modeling\", \"tokens\": 4356, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"326c9c538adbd0a701b7618bacf6e60d06fa9b09a486befaf8143c411e1560e8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9b148aa9-9b52-45e4-9768-d4cb99cabbb9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/language_modeling\", \"title\": \"Causal language modeling\", \"tokens\": 4356, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6db1a56e39fd7830a7dd560fcd7f86aa717b42c748ae8c6c7cc8b885bd86d7e5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cf65ee5e-6aaf-411f-bae8-d803e275759b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"580433daf2582c45b8b8c46fda6daafb27cdf7b1d3f9d8a86d5f313535a0e3e1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5728, \"end_char_idx\": 9198, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "doc_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "ref_doc_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "chroma:document": "URL_0'],\n 'answers.score': [21, 19, 5, 3],\n 'answers.text_urls': [[],\n  [],\n  [],\n  ['https://www.investopedia.com/news/trumps-tax-reform-what-can-be-done/']],\n 'title_urls': ['url'],\n 'selftext_urls': ['url']}\n```\n\nEach subfield is now a separate column as indicated by the `answers` prefix, and the `text` field is a list now. Instead\nof tokenizing each sentence separately, convert the list to a string so you can jointly tokenize them.\n\nHere is a first preprocessing function to join the list of strings for each example and tokenize the result:\n\n```py\n>>> def preprocess_function(examples):\n...     return tokenizer([\" \".join(x) for x in examples[\"answers.text\"]])\n```\n\nTo apply this preprocessing function over the entire dataset, use the \ud83e\udd17 Datasets [`~datasets.Dataset.map`] method. You can speed up the `map` function by setting `batched=True` to process multiple elements of the dataset at once, and increasing the number of processes with `num_proc`. Remove any columns you don't need:\n\n```py\n>>> tokenized_eli5 = eli5.map(\n...     preprocess_function,\n...     batched=True,\n...     num_proc=4,\n...     remove_columns=eli5[\"train\"].column_names,\n... )\n```\n\nThis dataset contains the token sequences, but some of these are longer than the maximum input length for the model.\n\nYou can now use a second preprocessing function to\n\n- concatenate all the sequences\n- split the concatenated sequences into shorter chunks defined by `block_size`, which should be both shorter than the maximum input length and short enough for your GPU RAM.\n\n```py\n>>> block_size = 128\n\n\n>>> def group_texts(examples):\n...     # Concatenate all texts.\n...     concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}\n...     total_length = len(concatenated_examples[list(examples.keys())[0]])\n...     # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can\n...     # customize this part to your needs.\n...     if total_length >= block_size:\n...         total_length = (total_length // block_size) * block_size\n...     # Split by chunks of block_size.\n...     result = {\n...         k: [t[i : i + block_size] for i in range(0, total_length, block_size)]\n...         for k, t in concatenated_examples.items()\n...     }\n...     result[\"labels\"] = result[\"input_ids\"].copy()\n...     return result\n```\n\nApply the `group_texts` function over the entire dataset:\n\n```py\n>>> lm_dataset = tokenized_eli5.map(group_texts, batched=True, num_proc=4)\n```\n\nNow create a batch of examples using [`DataCollatorForLanguageModeling`]. It's more efficient to *dynamically pad* the\nsentences to the longest length in a batch during collation, instead of padding the whole dataset to the maximum length.\n\n<frameworkcontent>\n<pt>\nUse the end-of-sequence token as the padding token and set `mlm=False`. This will use the inputs as labels shifted to the right by one element:\n\n```py\n>>> from transformers import DataCollatorForLanguageModeling\n\n>>> tokenizer.pad_token = tokenizer.eos_token\n>>> data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)\n```\n\n</pt>\n<tf>\nUse the end-of-sequence token as the padding token and set `mlm=False`. This will use the inputs as labels shifted to the right by one element:\n\n```py\n>>> from transformers import DataCollatorForLanguageModeling\n\n>>> data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False, return_tensors=\"tf\")\n```\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: cf65ee5e-6aaf-411f-bae8-d803e275759b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"cf65ee5e-6aaf-411f-bae8-d803e275759b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/language_modeling\", \"title\": \"Causal language modeling\", \"tokens\": 4356, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/language_modeling\", \"title\": \"Causal language modeling\", \"tokens\": 4356, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"326c9c538adbd0a701b7618bacf6e60d06fa9b09a486befaf8143c411e1560e8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9885de43-d86a-4b20-abcd-a29250cbfbc2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/language_modeling\", \"title\": \"Causal language modeling\", \"tokens\": 4356, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3b9731b020923dd41c73b8ec88f649673af7e91903bc4bb737261e1e25b6b9e6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b3d5def7-06ef-441a-bc5c-cc5b2e254203\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6e259563c812d45c702b4fbe8d10ec8160479a7f6a985dd4e36224fd4b845f2e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9201, \"end_char_idx\": 12221, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "doc_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "ref_doc_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "chroma:document": "## Train\n\n<frameworkcontent>\n<pt>\n<Tip>\n\nIf you aren't familiar with finetuning a model with the [`Trainer`], take a look at the [basic tutorial](../training#train-with-pytorch-trainer)!\n\n</Tip>\n\nYou're ready to start training your model now! Load DistilGPT2 with [`AutoModelForCausalLM`]:\n\n```py\n>>> from transformers import AutoModelForCausalLM, TrainingArguments, Trainer\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"distilbert/distilgpt2\")\n```\n\nAt this point, only three steps remain:\n\n1. Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model).\n2. Pass the training arguments to [`Trainer`] along with the model, datasets, and data collator.\n3. Call [`~Trainer.train`] to finetune your model.\n\n```py\n>>> training_args = TrainingArguments(\n...     output_dir=\"my_awesome_eli5_clm-model\",\n...     eval_strategy=\"epoch\",\n...     learning_rate=2e-5,\n...     weight_decay=0.01,\n...     push_to_hub=True,\n... )\n\n>>> trainer = Trainer(\n...     model=model,\n...     args=training_args,\n...     train_dataset=lm_dataset[\"train\"],\n...     eval_dataset=lm_dataset[\"test\"],\n...     data_collator=data_collator,\n... )\n\n>>> trainer.train()\n```\n\nOnce training is completed, use the [`~transformers.Trainer.evaluate`] method to evaluate your model and get its perplexity:\n\n```py\n>>> import math\n\n>>> eval_results = trainer.evaluate()\n>>> print(f\"Perplexity: {math.exp(eval_results['eval_loss']):.2f}\")\nPerplexity: 49.61\n```\n\nThen share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:\n\n```py\n>>> trainer.push_to_hub()\n```\n</pt>\n<tf>\n<Tip>\n\nIf you aren't familiar with finetuning a model with Keras, take a look at the [basic tutorial](../training#train-a-tensorflow-model-with-keras)!\n\n</Tip>\nTo finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:\n\n```py\n>>> from transformers import create_optimizer, AdamWeightDecay\n\n>>> optimizer = AdamWeightDecay(learning_rate=2e-5, weight_decay_rate=0.01)\n```\n\nThen you can load DistilGPT2 with [`TFAutoModelForCausalLM`]:\n\n```py\n>>> from transformers import TFAutoModelForCausalLM\n\n>>> model = TFAutoModelForCausalLM.from_pretrained(\"distilbert/distilgpt2\")\n```\n\nConvert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:\n\n```py\n>>> tf_train_set = model.prepare_tf_dataset(\n...     lm_dataset[\"train\"],\n...     shuffle=True,\n...     batch_size=16,\n...     collate_fn=data_collator,\n... )\n\n>>> tf_test_set = model.prepare_tf_dataset(\n...     lm_dataset[\"test\"],\n...     shuffle=False,\n...     batch_size=16,\n...     collate_fn=data_collator,\n... )\n```\n\nConfigure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: b3d5def7-06ef-441a-bc5c-cc5b2e254203
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b3d5def7-06ef-441a-bc5c-cc5b2e254203\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/language_modeling\", \"title\": \"Causal language modeling\", \"tokens\": 4356, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/language_modeling\", \"title\": \"Causal language modeling\", \"tokens\": 4356, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"326c9c538adbd0a701b7618bacf6e60d06fa9b09a486befaf8143c411e1560e8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cf65ee5e-6aaf-411f-bae8-d803e275759b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/language_modeling\", \"title\": \"Causal language modeling\", \"tokens\": 4356, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d4cfb4de79c09731d5529160527ad4bc247a696919c98bae5853386725baceb2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"792f192f-5f1c-4033-9d66-7d4c35c517a1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5524ba32f3c72e3290214d6a1bdc7e1eb019fe9c6eea66c43cc1c26f769ddba0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11162, \"end_char_idx\": 14440, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "doc_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "ref_doc_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "chroma:document": "</Tip>\nTo finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:\n\n```py\n>>> from transformers import create_optimizer, AdamWeightDecay\n\n>>> optimizer = AdamWeightDecay(learning_rate=2e-5, weight_decay_rate=0.01)\n```\n\nThen you can load DistilGPT2 with [`TFAutoModelForCausalLM`]:\n\n```py\n>>> from transformers import TFAutoModelForCausalLM\n\n>>> model = TFAutoModelForCausalLM.from_pretrained(\"distilbert/distilgpt2\")\n```\n\nConvert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:\n\n```py\n>>> tf_train_set = model.prepare_tf_dataset(\n...     lm_dataset[\"train\"],\n...     shuffle=True,\n...     batch_size=16,\n...     collate_fn=data_collator,\n... )\n\n>>> tf_test_set = model.prepare_tf_dataset(\n...     lm_dataset[\"test\"],\n...     shuffle=False,\n...     batch_size=16,\n...     collate_fn=data_collator,\n... )\n```\n\nConfigure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method). Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:\n\n```py\n>>> import tensorflow as tf\n\n>>> model.compile(optimizer=optimizer)  # No loss argument!\n```\n\nThis can be done by specifying where to push your model and tokenizer in the [`~transformers.PushToHubCallback`]:\n\n```py\n>>> from transformers.keras_callbacks import PushToHubCallback\n\n>>> callback = PushToHubCallback(\n...     output_dir=\"my_awesome_eli5_clm-model\",\n...     tokenizer=tokenizer,\n... )\n```\n\nFinally, you're ready to start training your model! Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callback to finetune the model:\n\n```py\n>>> model.fit(x=tf_train_set, validation_data=tf_test_set, epochs=3, callbacks=[callback])\n```\n\nOnce training is completed, your model is automatically uploaded to the Hub so everyone can use it!\n</tf>\n</frameworkcontent>\n\n<Tip>\n\nFor a more in-depth example of how to finetune a model for causal language modeling, take a look at the corresponding\n[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb)\nor [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n\n</Tip>\n\n## Inference\n\nGreat, now that you've finetuned a model, you can use it for inference!\n\nCome up with a prompt you'd like to generate text from:\n\n```py\n>>> prompt = \"Somatic hypermutation allows the immune system to\"\n```\n\nThe simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for text generation with your model, and pass your text to it:\n\n```py\n>>> from transformers import pipeline\n\n>>> generator = pipeline(\"text-generation\", model=\"username/my_awesome_eli5_clm-model\")\n>>> generator(prompt)\n[{'generated_text': \"Somatic hypermutation allows the immune system to be able to effectively reverse the damage caused by an infection.\\n\\n\\nThe damage caused by an infection is caused by the immune system's ability to perform its own self-correcting tasks.\"}]"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 792f192f-5f1c-4033-9d66-7d4c35c517a1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"792f192f-5f1c-4033-9d66-7d4c35c517a1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/language_modeling\", \"title\": \"Causal language modeling\", \"tokens\": 4356, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/language_modeling\", \"title\": \"Causal language modeling\", \"tokens\": 4356, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"326c9c538adbd0a701b7618bacf6e60d06fa9b09a486befaf8143c411e1560e8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b3d5def7-06ef-441a-bc5c-cc5b2e254203\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/language_modeling\", \"title\": \"Causal language modeling\", \"tokens\": 4356, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4930bf51777760f5698a713f09aa331216982123b384361caabbbf02412f2f6f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"37fbec03-0428-487e-a268-657967d0b71e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"706cbd17ce0f9dc5d8ab0095a276d702b241e00b9088aac983b7600e5a70ae81\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12813, \"end_char_idx\": 15868, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "doc_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "ref_doc_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "chroma:document": "Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callback to finetune the model:\n\n```py\n>>> model.fit(x=tf_train_set, validation_data=tf_test_set, epochs=3, callbacks=[callback])\n```\n\nOnce training is completed, your model is automatically uploaded to the Hub so everyone can use it!\n</tf>\n</frameworkcontent>\n\n<Tip>\n\nFor a more in-depth example of how to finetune a model for causal language modeling, take a look at the corresponding\n[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb)\nor [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).\n\n</Tip>\n\n## Inference\n\nGreat, now that you've finetuned a model, you can use it for inference!\n\nCome up with a prompt you'd like to generate text from:\n\n```py\n>>> prompt = \"Somatic hypermutation allows the immune system to\"\n```\n\nThe simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for text generation with your model, and pass your text to it:\n\n```py\n>>> from transformers import pipeline\n\n>>> generator = pipeline(\"text-generation\", model=\"username/my_awesome_eli5_clm-model\")\n>>> generator(prompt)\n[{'generated_text': \"Somatic hypermutation allows the immune system to be able to effectively reverse the damage caused by an infection.\\n\\n\\nThe damage caused by an infection is caused by the immune system's ability to perform its own self-correcting tasks.\"}]\n```\n\n<frameworkcontent>\n<pt>\nTokenize the text and return the `input_ids` as PyTorch tensors:\n\n```py\n>>> from transformers import AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"username/my_awesome_eli5_clm-model\")\n>>> inputs = tokenizer(prompt, return_tensors=\"pt\").input_ids\n```\n\nUse the [`~generation.GenerationMixin.generate`] method to generate text.\nFor more details about the different text generation strategies and parameters for controlling generation, check out the [Text generation strategies](../generation_strategies) page.\n\n```py\n>>> from transformers import AutoModelForCausalLM\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"username/my_awesome_eli5_clm-model\")\n>>> outputs = model.generate(inputs, max_new_tokens=100, do_sample=True, top_k=50, top_p=0.95)\n```\n\nDecode the generated token ids back into text:\n\n```py\n>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)\n[\"Somatic hypermutation allows the immune system to react to drugs with the ability to adapt to a different environmental situation. In other words, a system of 'hypermutation' can help the immune system to adapt to a different environmental situation or in some cases even a single life. In contrast, researchers at the University of Massachusetts-Boston have found that 'hypermutation' is much stronger in mice than in humans but can be found in humans, and that it's not completely unknown to the immune system."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 37fbec03-0428-487e-a268-657967d0b71e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"37fbec03-0428-487e-a268-657967d0b71e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/language_modeling\", \"title\": \"Causal language modeling\", \"tokens\": 4356, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/language_modeling\", \"title\": \"Causal language modeling\", \"tokens\": 4356, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"326c9c538adbd0a701b7618bacf6e60d06fa9b09a486befaf8143c411e1560e8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"792f192f-5f1c-4033-9d66-7d4c35c517a1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/language_modeling\", \"title\": \"Causal language modeling\", \"tokens\": 4356, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b8466ba1126345346404343b8e9c471bdc41e3799f85941d8cc305d942e9fcc5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14441, \"end_char_idx\": 17489, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "doc_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "ref_doc_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "chroma:document": "```\n\n<frameworkcontent>\n<pt>\nTokenize the text and return the `input_ids` as PyTorch tensors:\n\n```py\n>>> from transformers import AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"username/my_awesome_eli5_clm-model\")\n>>> inputs = tokenizer(prompt, return_tensors=\"pt\").input_ids\n```\n\nUse the [`~generation.GenerationMixin.generate`] method to generate text.\nFor more details about the different text generation strategies and parameters for controlling generation, check out the [Text generation strategies](../generation_strategies) page.\n\n```py\n>>> from transformers import AutoModelForCausalLM\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"username/my_awesome_eli5_clm-model\")\n>>> outputs = model.generate(inputs, max_new_tokens=100, do_sample=True, top_k=50, top_p=0.95)\n```\n\nDecode the generated token ids back into text:\n\n```py\n>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)\n[\"Somatic hypermutation allows the immune system to react to drugs with the ability to adapt to a different environmental situation. In other words, a system of 'hypermutation' can help the immune system to adapt to a different environmental situation or in some cases even a single life. In contrast, researchers at the University of Massachusetts-Boston have found that 'hypermutation' is much stronger in mice than in humans but can be found in humans, and that it's not completely unknown to the immune system. A study on how the immune system\"]\n```\n</pt>\n<tf>\nTokenize the text and return the `input_ids` as TensorFlow tensors:\n\n```py\n>>> from transformers import AutoTokenizer\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"username/my_awesome_eli5_clm-model\")\n>>> inputs = tokenizer(prompt, return_tensors=\"tf\").input_ids\n```\n\nUse the [`~transformers.generation_tf_utils.TFGenerationMixin.generate`] method to create the summarization. For more details about the different text generation strategies and parameters for controlling generation, check out the [Text generation strategies](../generation_strategies) page.\n\n```py\n>>> from transformers import TFAutoModelForCausalLM\n\n>>> model = TFAutoModelForCausalLM.from_pretrained(\"username/my_awesome_eli5_clm-model\")\n>>> outputs = model.generate(input_ids=inputs, max_new_tokens=100, do_sample=True, top_k=50, top_p=0.95)\n```\n\nDecode the generated token ids back into text:\n\n```py\n>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)\n['Somatic hypermutation allows the immune system to detect the presence of other viruses as they become more prevalent. Therefore, researchers have identified a high proportion of human viruses. The proportion of virus-associated viruses in our study increases with age. Therefore, we propose a simple algorithm to detect the presence of these new viruses in our samples as a sign of improved immunity. A first study based on this algorithm, which will be published in Science on Friday, aims to show that this finding could translate into the development of a better vaccine that is more effective for']\n```\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 17880d83-56f5-4b6d-ad62-a12837eee84f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"17880d83-56f5-4b6d-ad62-a12837eee84f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1de2f80f-f53c-43e8-b5f3-49b3b915c9f7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"87094aa86af139e913445f44b4ef5b8586c1ce3c46f8f7ba6b5f094bd133cd43\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6779fcd0-f13a-403a-89c4-94628c393fc2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"656c0869a016a237b5020712d12db0967f1a86ebe21a4d51676f7326d4975deb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3069, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "doc_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "ref_doc_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "chroma:document": "# LLM prompting guide\n\n[[open-in-colab]]\n\nLarge Language Models such as Falcon, LLaMA, etc. are pretrained transformer models initially trained to predict the \nnext token given some input text. They typically have billions of parameters and have been trained on trillions of \ntokens for an extended period of time. As a result, these models become quite powerful and versatile, and you can use \nthem to solve multiple NLP tasks out of the box by instructing the models with natural language prompts.\n\nDesigning such prompts to ensure the optimal output is often called \"prompt engineering\". Prompt engineering is an \niterative process that requires a fair amount of experimentation. Natural languages are much more flexible and expressive \nthan programming languages, however, they can also introduce some ambiguity. At the same time, prompts in natural language \nare quite sensitive to changes. Even minor modifications in prompts can lead to wildly different outputs.\n\nWhile there is no exact recipe for creating prompts to match all cases, researchers have worked out a number of best \npractices that help to achieve optimal results more consistently. \n\nThis guide covers the prompt engineering best practices to help you craft better LLM prompts and solve various NLP tasks. \nYou'll learn:\n\n- [Basics of prompting](#basics-of-prompting)\n- [Best practices of LLM prompting](#best-practices-of-llm-prompting)\n- [Advanced prompting techniques: few-shot prompting and chain-of-thought](#advanced-prompting-techniques)\n- [When to fine-tune instead of prompting](#prompting-vs-fine-tuning)\n\n<Tip>\n\nPrompt engineering is only a part of the LLM output optimization process. Another essential component is choosing the \noptimal text generation strategy. You can customize how your LLM selects each of the subsequent tokens when generating \nthe text without modifying any of the trainable parameters. By tweaking the text generation parameters, you can reduce \nrepetition in the generated text and make it more coherent and human-sounding. \nText generation strategies and parameters are out of scope for this guide, but you can learn more about these topics in \nthe following guides: \n \n* [Generation with LLMs](../llm_tutorial)\n* [Text generation strategies](../generation_strategies)\n\n</Tip>\n\n## Basics of prompting\n\n### Types of models \n\nThe majority of modern LLMs are decoder-only transformers. Some examples include: [LLaMA](../model_doc/llama), \n[Llama2](../model_doc/llama2), [Falcon](../model_doc/falcon), [GPT2](../model_doc/gpt2). However, you may encounter\nencoder-decoder transformer LLMs as well, for instance, [Flan-T5](../model_doc/flan-t5) and [BART](../model_doc/bart).\n\nEncoder-decoder-style models are typically used in generative tasks where the output **heavily** relies on the input, for \nexample, in translation and summarization. The decoder-only models are used for all other types of generative tasks.\n\nWhen using a pipeline to generate text with an LLM, it's important to know what type of LLM you are using, because \nthey use different pipelines."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6779fcd0-f13a-403a-89c4-94628c393fc2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6779fcd0-f13a-403a-89c4-94628c393fc2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1de2f80f-f53c-43e8-b5f3-49b3b915c9f7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"87094aa86af139e913445f44b4ef5b8586c1ce3c46f8f7ba6b5f094bd133cd43\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"17880d83-56f5-4b6d-ad62-a12837eee84f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d65d612da17f442df4d403ead652e280ad2d3ce9651429e38f1623f95b7bfd94\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8fdf6f14-5d91-406a-929a-cb001f19d26d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"63581c22a359f713342e19976c753b25b474ebd67578855a8973c42e79663fde\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1670, \"end_char_idx\": 4865, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "doc_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "ref_doc_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "chroma:document": "Another essential component is choosing the \noptimal text generation strategy. You can customize how your LLM selects each of the subsequent tokens when generating \nthe text without modifying any of the trainable parameters. By tweaking the text generation parameters, you can reduce \nrepetition in the generated text and make it more coherent and human-sounding. \nText generation strategies and parameters are out of scope for this guide, but you can learn more about these topics in \nthe following guides: \n \n* [Generation with LLMs](../llm_tutorial)\n* [Text generation strategies](../generation_strategies)\n\n</Tip>\n\n## Basics of prompting\n\n### Types of models \n\nThe majority of modern LLMs are decoder-only transformers. Some examples include: [LLaMA](../model_doc/llama), \n[Llama2](../model_doc/llama2), [Falcon](../model_doc/falcon), [GPT2](../model_doc/gpt2). However, you may encounter\nencoder-decoder transformer LLMs as well, for instance, [Flan-T5](../model_doc/flan-t5) and [BART](../model_doc/bart).\n\nEncoder-decoder-style models are typically used in generative tasks where the output **heavily** relies on the input, for \nexample, in translation and summarization. The decoder-only models are used for all other types of generative tasks.\n\nWhen using a pipeline to generate text with an LLM, it's important to know what type of LLM you are using, because \nthey use different pipelines. \n\nRun inference with decoder-only models with the `text-generation` pipeline:\n\n```python\n>>> from transformers import pipeline\n>>> import torch\n\n>>> torch.manual_seed(0) # doctest: +IGNORE_RESULT\n\n>>> generator = pipeline('text-generation', model = 'openai-community/gpt2')\n>>> prompt = \"Hello, I'm a language model\"\n\n>>> generator(prompt, max_length = 30)\n[{'generated_text': \"Hello, I'm a language model programmer so you can use some of my stuff. But you also need some sort of a C program to run.\"}]\n```\n\nTo run inference with an encoder-decoder, use the `text2text-generation` pipeline:\n\n```python\n>>> text2text_generator = pipeline(\"text2text-generation\", model = 'google/flan-t5-base')\n>>> prompt = \"Translate from English to French: I'm very happy to see you\"\n\n>>> text2text_generator(prompt)\n[{'generated_text': 'Je suis tr\u00e8s heureuse de vous rencontrer.'}]\n```\n\n### Base vs instruct/chat models\n\nMost of the recent LLM checkpoints available on \ud83e\udd17 Hub come in two versions: base and instruct (or chat). For example, \n[`tiiuae/falcon-7b`](https://huggingface.co/tiiuae/falcon-7b) and [`tiiuae/falcon-7b-instruct`](https://huggingface.co/tiiuae/falcon-7b-instruct).\n\nBase models are excellent at completing the text when given an initial prompt, however, they are not ideal for NLP tasks \nwhere they need to follow instructions, or for conversational use. This is where the instruct (chat) versions come in. \nThese checkpoints are the result of further fine-tuning of the pre-trained base versions on instructions and conversational data. \nThis additional fine-tuning makes them a better choice for many NLP tasks.  \n\nLet's illustrate some simple prompts that you can use with [`tiiuae/falcon-7b-instruct`](https://huggingface.co/tiiuae/falcon-7b-instruct) \nto solve some common NLP tasks."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 8fdf6f14-5d91-406a-929a-cb001f19d26d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8fdf6f14-5d91-406a-929a-cb001f19d26d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1de2f80f-f53c-43e8-b5f3-49b3b915c9f7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"87094aa86af139e913445f44b4ef5b8586c1ce3c46f8f7ba6b5f094bd133cd43\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6779fcd0-f13a-403a-89c4-94628c393fc2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a857a8d2bbb513bcd95c283fb6e009c883a4234dcdab3b3053e642075ce03b62\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dec1d932-6c77-49a5-87c7-08313ba1ccd7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c3a31a8dd216c705a8499d4502e35785011d93f7a74ba4135ee68e9fee63196d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3520, \"end_char_idx\": 6568, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "doc_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "ref_doc_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "chroma:document": "But you also need some sort of a C program to run.\"}]\n```\n\nTo run inference with an encoder-decoder, use the `text2text-generation` pipeline:\n\n```python\n>>> text2text_generator = pipeline(\"text2text-generation\", model = 'google/flan-t5-base')\n>>> prompt = \"Translate from English to French: I'm very happy to see you\"\n\n>>> text2text_generator(prompt)\n[{'generated_text': 'Je suis tr\u00e8s heureuse de vous rencontrer.'}]\n```\n\n### Base vs instruct/chat models\n\nMost of the recent LLM checkpoints available on \ud83e\udd17 Hub come in two versions: base and instruct (or chat). For example, \n[`tiiuae/falcon-7b`](https://huggingface.co/tiiuae/falcon-7b) and [`tiiuae/falcon-7b-instruct`](https://huggingface.co/tiiuae/falcon-7b-instruct).\n\nBase models are excellent at completing the text when given an initial prompt, however, they are not ideal for NLP tasks \nwhere they need to follow instructions, or for conversational use. This is where the instruct (chat) versions come in. \nThese checkpoints are the result of further fine-tuning of the pre-trained base versions on instructions and conversational data. \nThis additional fine-tuning makes them a better choice for many NLP tasks.  \n\nLet's illustrate some simple prompts that you can use with [`tiiuae/falcon-7b-instruct`](https://huggingface.co/tiiuae/falcon-7b-instruct) \nto solve some common NLP tasks.\n\n### NLP tasks \n\nFirst, let's set up the environment: \n\n```bash\npip install -q transformers accelerate\n```\n\nNext, let's load the model with the appropriate pipeline (`\"text-generation\"`): \n\n```python\n>>> from transformers import pipeline, AutoTokenizer\n>>> import torch\n\n>>> torch.manual_seed(0) # doctest: +IGNORE_RESULT\n>>> model = \"tiiuae/falcon-7b-instruct\"\n\n>>> tokenizer = AutoTokenizer.from_pretrained(model)\n>>> pipe = pipeline(\n...     \"text-generation\",\n...     model=model,\n...     tokenizer=tokenizer,\n...     torch_dtype=torch.bfloat16,\n...     device_map=\"auto\",\n... )\n```\n\n<Tip>\n\nNote that Falcon models were trained using the `bfloat16` datatype, so we recommend you use the same. This requires a recent \nversion of CUDA and works best on modern cards.\n\n</Tip>\n\nNow that we have the model loaded via the pipeline, let's explore how you can use prompts to solve NLP tasks.\n\n#### Text classification\n\nOne of the most common forms of text classification is sentiment analysis, which assigns a label like \"positive\", \"negative\", \nor \"neutral\" to a sequence of text. Let's write a prompt that instructs the model to classify a given text (a movie review). \nWe'll start by giving the instruction, and then specifying the text to classify. Note that instead of leaving it at that, we're \nalso adding the beginning of the response - `\"Sentiment: \"`:\n\n```python\n>>> torch.manual_seed(0) # doctest: +IGNORE_RESULT\n>>> prompt = \"\"\"Classify the text into neutral, negative or positive. \n... Text: This movie is definitely one of my favorite movies of its kind. The interaction between respectable and morally strong characters is an ode to chivalry and the honor code amongst thieves and policemen."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: dec1d932-6c77-49a5-87c7-08313ba1ccd7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"dec1d932-6c77-49a5-87c7-08313ba1ccd7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1de2f80f-f53c-43e8-b5f3-49b3b915c9f7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"87094aa86af139e913445f44b4ef5b8586c1ce3c46f8f7ba6b5f094bd133cd43\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8fdf6f14-5d91-406a-929a-cb001f19d26d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"834a333b360517b1a2f0a685352cfb0f17be893f33295b8797130ddf1ca536dc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"31da29e1-ba61-4572-8798-8d059cb5f036\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0767139c1b1a02a31e4eb8f57639e7a004174333d919b7509921baa7cac2659d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4867, \"end_char_idx\": 8037, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "doc_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "ref_doc_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "chroma:document": "### NLP tasks \n\nFirst, let's set up the environment: \n\n```bash\npip install -q transformers accelerate\n```\n\nNext, let's load the model with the appropriate pipeline (`\"text-generation\"`): \n\n```python\n>>> from transformers import pipeline, AutoTokenizer\n>>> import torch\n\n>>> torch.manual_seed(0) # doctest: +IGNORE_RESULT\n>>> model = \"tiiuae/falcon-7b-instruct\"\n\n>>> tokenizer = AutoTokenizer.from_pretrained(model)\n>>> pipe = pipeline(\n...     \"text-generation\",\n...     model=model,\n...     tokenizer=tokenizer,\n...     torch_dtype=torch.bfloat16,\n...     device_map=\"auto\",\n... )\n```\n\n<Tip>\n\nNote that Falcon models were trained using the `bfloat16` datatype, so we recommend you use the same. This requires a recent \nversion of CUDA and works best on modern cards.\n\n</Tip>\n\nNow that we have the model loaded via the pipeline, let's explore how you can use prompts to solve NLP tasks.\n\n#### Text classification\n\nOne of the most common forms of text classification is sentiment analysis, which assigns a label like \"positive\", \"negative\", \nor \"neutral\" to a sequence of text. Let's write a prompt that instructs the model to classify a given text (a movie review). \nWe'll start by giving the instruction, and then specifying the text to classify. Note that instead of leaving it at that, we're \nalso adding the beginning of the response - `\"Sentiment: \"`:\n\n```python\n>>> torch.manual_seed(0) # doctest: +IGNORE_RESULT\n>>> prompt = \"\"\"Classify the text into neutral, negative or positive. \n... Text: This movie is definitely one of my favorite movies of its kind. The interaction between respectable and morally strong characters is an ode to chivalry and the honor code amongst thieves and policemen.\n... Sentiment:\n... \"\"\"\n\n>>> sequences = pipe(\n...     prompt,\n...     max_new_tokens=10,\n... )\n\n>>> for seq in sequences:\n...     print(f\"Result: {seq['generated_text']}\")\nResult: Classify the text into neutral, negative or positive. \nText: This movie is definitely one of my favorite movies of its kind. The interaction between respectable and morally strong characters is an ode to chivalry and the honor code amongst thieves and policemen.\nSentiment:\nPositive\n```\n\nAs a result, the output contains a classification label from the list we have provided in the instructions, and it is a correct one!\n\n<Tip>\n\nYou may notice that in addition to the prompt, we pass a `max_new_tokens` parameter. It controls the number of tokens the \nmodel shall generate, and it is one of the many text generation parameters that you can learn about \nin [Text generation strategies](../generation_strategies) guide.\n\n</Tip>\n\n#### Named Entity Recognition\n\nNamed Entity Recognition (NER) is a task of finding named entities in a piece of text, such as a person, location, or organization.\nLet's modify the instructions in the prompt to make the LLM perform this task. Here, let's also set `return_full_text = False` \nso that output doesn't contain the prompt:\n\n```python\n>>> torch.manual_seed(1) # doctest: +IGNORE_RESULT\n>>> prompt = \"\"\"Return a list of named entities in the text.\n... Text: The Golden State Warriors are an American professional basketball team based in San Francisco."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 31da29e1-ba61-4572-8798-8d059cb5f036
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"31da29e1-ba61-4572-8798-8d059cb5f036\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1de2f80f-f53c-43e8-b5f3-49b3b915c9f7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"87094aa86af139e913445f44b4ef5b8586c1ce3c46f8f7ba6b5f094bd133cd43\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dec1d932-6c77-49a5-87c7-08313ba1ccd7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e6c8fb24860ffbfd8cb5f7f2e36ff24879b9c94cbfcc1435766318c2d5db766e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a1fae959-44d9-4531-b20b-ec51834eaa3f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a48562b15cccd099d04321985244c4b7d302f5b2119c34af32d1d255531a06b8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6357, \"end_char_idx\": 9810, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "doc_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "ref_doc_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "chroma:document": "... Text: This movie is definitely one of my favorite movies of its kind. The interaction between respectable and morally strong characters is an ode to chivalry and the honor code amongst thieves and policemen.\n... Sentiment:\n... \"\"\"\n\n>>> sequences = pipe(\n...     prompt,\n...     max_new_tokens=10,\n... )\n\n>>> for seq in sequences:\n...     print(f\"Result: {seq['generated_text']}\")\nResult: Classify the text into neutral, negative or positive. \nText: This movie is definitely one of my favorite movies of its kind. The interaction between respectable and morally strong characters is an ode to chivalry and the honor code amongst thieves and policemen.\nSentiment:\nPositive\n```\n\nAs a result, the output contains a classification label from the list we have provided in the instructions, and it is a correct one!\n\n<Tip>\n\nYou may notice that in addition to the prompt, we pass a `max_new_tokens` parameter. It controls the number of tokens the \nmodel shall generate, and it is one of the many text generation parameters that you can learn about \nin [Text generation strategies](../generation_strategies) guide.\n\n</Tip>\n\n#### Named Entity Recognition\n\nNamed Entity Recognition (NER) is a task of finding named entities in a piece of text, such as a person, location, or organization.\nLet's modify the instructions in the prompt to make the LLM perform this task. Here, let's also set `return_full_text = False` \nso that output doesn't contain the prompt:\n\n```python\n>>> torch.manual_seed(1) # doctest: +IGNORE_RESULT\n>>> prompt = \"\"\"Return a list of named entities in the text.\n... Text: The Golden State Warriors are an American professional basketball team based in San Francisco.\n... Named entities:\n... \"\"\"\n\n>>> sequences = pipe(\n...     prompt,\n...     max_new_tokens=15,\n...     return_full_text = False,    \n... )\n\n>>> for seq in sequences:\n...     print(f\"{seq['generated_text']}\")\n- Golden State Warriors\n- San Francisco\n```\n\nAs you can see, the model correctly identified two named entities from the given text.\n\n#### Translation\n\nAnother task LLMs can perform is translation. You can choose to use encoder-decoder models for this task, however, here,\nfor the simplicity of the examples, we'll keep using Falcon-7b-instruct, which does a decent job. Once again, here's how \nyou can write a basic prompt to instruct a model to translate a piece of text from English to Italian: \n\n```python\n>>> torch.manual_seed(2) # doctest: +IGNORE_RESULT\n>>> prompt = \"\"\"Translate the English text to Italian.\n... Text: Sometimes, I've believed as many as six impossible things before breakfast.\n... Translation:\n... \"\"\"\n\n>>> sequences = pipe(\n...     prompt,\n...     max_new_tokens=20,\n...     do_sample=True,\n...     top_k=10,\n...     return_full_text = False,\n... )\n\n>>> for seq in sequences:\n...     print(f\"{seq['generated_text']}\")\nA volte, ho creduto a sei impossibili cose prima di colazione.\n```\n\nHere we've added a `do_sample=True` and `top_k=10` to allow the model to be a bit more flexible when generating output.\n\n#### Text summarization\n\nSimilar to the translation, text summarization is another generative task where the output **heavily** relies on the input, \nand encoder-decoder models can be a better choice. However, decoder-style models can be used for this task as well.\nPreviously, we have placed the instructions at the very beginning of the prompt. However, the very end of the prompt can \nalso be a suitable location for instructions."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a1fae959-44d9-4531-b20b-ec51834eaa3f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a1fae959-44d9-4531-b20b-ec51834eaa3f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1de2f80f-f53c-43e8-b5f3-49b3b915c9f7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"87094aa86af139e913445f44b4ef5b8586c1ce3c46f8f7ba6b5f094bd133cd43\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"31da29e1-ba61-4572-8798-8d059cb5f036\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"35f6b7690dce355c700fb0184a5748ab05e332c2274615c356b58ac5e72355f3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6b3b6ea1-5d47-47b7-9f6d-fb4f09bc2f42\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"da625d01de5300012b1895629a71e7dce3d00229a908353241e951748d7d2387\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8378, \"end_char_idx\": 11937, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "doc_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "ref_doc_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "chroma:document": "#### Translation\n\nAnother task LLMs can perform is translation. You can choose to use encoder-decoder models for this task, however, here,\nfor the simplicity of the examples, we'll keep using Falcon-7b-instruct, which does a decent job. Once again, here's how \nyou can write a basic prompt to instruct a model to translate a piece of text from English to Italian: \n\n```python\n>>> torch.manual_seed(2) # doctest: +IGNORE_RESULT\n>>> prompt = \"\"\"Translate the English text to Italian.\n... Text: Sometimes, I've believed as many as six impossible things before breakfast.\n... Translation:\n... \"\"\"\n\n>>> sequences = pipe(\n...     prompt,\n...     max_new_tokens=20,\n...     do_sample=True,\n...     top_k=10,\n...     return_full_text = False,\n... )\n\n>>> for seq in sequences:\n...     print(f\"{seq['generated_text']}\")\nA volte, ho creduto a sei impossibili cose prima di colazione.\n```\n\nHere we've added a `do_sample=True` and `top_k=10` to allow the model to be a bit more flexible when generating output.\n\n#### Text summarization\n\nSimilar to the translation, text summarization is another generative task where the output **heavily** relies on the input, \nand encoder-decoder models can be a better choice. However, decoder-style models can be used for this task as well.\nPreviously, we have placed the instructions at the very beginning of the prompt. However, the very end of the prompt can \nalso be a suitable location for instructions. Typically, it's better to place the instruction on one of the extreme ends.  \n\n```python\n>>> torch.manual_seed(3) # doctest: +IGNORE_RESULT\n>>> prompt = \"\"\"Permaculture is a design process mimicking the diversity, functionality and resilience of natural ecosystems. The principles and practices are drawn from traditional ecological knowledge of indigenous cultures combined with modern scientific understanding and technological innovations. Permaculture design provides a framework helping individuals and communities develop innovative, creative and effective strategies for meeting basic needs while preparing for and mitigating the projected impacts of climate change.\n... Write a summary of the above text.\n... Summary:\n... \"\"\"\n\n>>> sequences = pipe(\n...     prompt,\n...     max_new_tokens=30,\n...     do_sample=True,\n...     top_k=10,\n...     return_full_text = False,\n... )\n\n>>> for seq in sequences:\n...     print(f\"{seq['generated_text']}\")\nPermaculture is an ecological design mimicking natural ecosystems to meet basic needs and prepare for climate change. It is based on traditional knowledge and scientific understanding.\n```\n\n#### Question answering\n\nFor question answering task we can structure the prompt into the following logical components: instructions, context, question, and \nthe leading word or phrase (`\"Answer:\"`) to nudge the model to start generating the answer:\n\n```python\n>>> torch.manual_seed(4) # doctest: +IGNORE_RESULT\n>>> prompt = \"\"\"Answer the question using the context below.\n... Context: Gazpacho is a cold soup and drink made of raw, blended vegetables. Most gazpacho includes stale bread, tomato, cucumbers, onion, bell peppers, garlic, olive oil, wine vinegar, water, and salt. Northern recipes often include cumin and/or piment\u00f3n (smoked sweet paprika). Traditionally, gazpacho was made by pounding the vegetables in a mortar with a pestle; this more laborious method is still sometimes used as it helps keep the gazpacho cool and avoids the foam and silky consistency of smoothie versions made in blenders or food processors.\n... Question: What modern tool is used to make gazpacho?"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6b3b6ea1-5d47-47b7-9f6d-fb4f09bc2f42
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6b3b6ea1-5d47-47b7-9f6d-fb4f09bc2f42\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1de2f80f-f53c-43e8-b5f3-49b3b915c9f7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"87094aa86af139e913445f44b4ef5b8586c1ce3c46f8f7ba6b5f094bd133cd43\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a1fae959-44d9-4531-b20b-ec51834eaa3f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/tasks/prompting\", \"title\": \"LLM prompting guide\", \"tokens\": 4688, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4ca6fed20f5dc1192a37a97eaef919d33fbf61d094bf97f47f1b0cfb4d09795a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dbfd220e-e4f0-4327-8759-1679e6d108b2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4965024648f8fdcedce6901e2caf15795f090780c617b1cb2d57af77a2df1b36\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10077, \"end_char_idx\": 13381, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "doc_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "ref_doc_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "chroma:document": "The principles and practices are drawn from traditional ecological knowledge of indigenous cultures combined with modern scientific understanding and technological innovations. Permaculture design provides a framework helping individuals and communities develop innovative, creative and effective strategies for meeting basic needs while preparing for and mitigating the projected impacts of climate change.\n... Write a summary of the above text.\n... Summary:\n... \"\"\"\n\n>>> sequences = pipe(\n...     prompt,\n...     max_new_tokens=30,\n...     do_sample=True,\n...     top_k=10,\n...     return_full_text = False,\n... )\n\n>>> for seq in sequences:\n...     print(f\"{seq['generated_text']}\")\nPermaculture is an ecological design mimicking natural ecosystems to meet basic needs and prepare for climate change. It is based on traditional knowledge and scientific understanding.\n```\n\n#### Question answering\n\nFor question answering task we can structure the prompt into the following logical components: instructions, context, question, and \nthe leading word or phrase (`\"Answer:\"`) to nudge the model to start generating the answer:\n\n```python\n>>> torch.manual_seed(4) # doctest: +IGNORE_RESULT\n>>> prompt = \"\"\"Answer the question using the context below.\n... Context: Gazpacho is a cold soup and drink made of raw, blended vegetables. Most gazpacho includes stale bread, tomato, cucumbers, onion, bell peppers, garlic, olive oil, wine vinegar, water, and salt. Northern recipes often include cumin and/or piment\u00f3n (smoked sweet paprika). Traditionally, gazpacho was made by pounding the vegetables in a mortar with a pestle; this more laborious method is still sometimes used as it helps keep the gazpacho cool and avoids the foam and silky consistency of smoothie versions made in blenders or food processors.\n... Question: What modern tool is used to make gazpacho?\n... Answer:\n... \"\"\"\n\n>>> sequences = pipe(\n...     prompt,\n...     max_new_tokens=10,\n...     do_sample=True,\n...     top_k=10,\n...     return_full_text = False,\n... )\n\n>>> for seq in sequences:\n...     print(f\"Result: {seq['generated_text']}\")\nResult: Modern tools often used to make gazpacho include\n```\n\n#### Reasoning\n\nReasoning is one of the most difficult tasks for LLMs, and achieving good results often requires applying advanced prompting techniques, like \n[Chain-of-though](#chain-of-thought).\n\nLet's try if we can make a model reason about a simple arithmetics task with a basic prompt: \n\n```python\n>>> torch.manual_seed(5) # doctest: +IGNORE_RESULT\n>>> prompt = \"\"\"There are 5 groups of students in the class. Each group has 4 students. How many students are there in the class?\"\"\"\n\n>>> sequences = pipe(\n...     prompt,\n...     max_new_tokens=30,\n...     do_sample=True,\n...     top_k=10,\n...     return_full_text = False,\n... )\n\n>>> for seq in sequences:\n...     print(f\"Result: {seq['generated_text']}\")\nResult: \nThere are a total of 5 groups, so there are 5 x 4=20 students in the class.\n```\n\nCorrect! Let's increase the complexity a little and see if we can still get away with a basic prompt:\n\n```python\n>>> torch.manual_seed(6) # doctest: +IGNORE_RESULT\n>>> prompt = \"\"\"I baked 15 muffins. I ate 2 muffins and gave 5 muffins to a neighbor. My partner then bought 6 more muffins and ate 2. How many muffins do we now have?\"\"\""}


