# embeddings_queue

created_at: 2025-02-05 09:13:39
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/language-modeling/run_plm.py:228-574
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 574, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/language-modeling/run_plm.py", "name": "main", "start_line": 228, "type": "function", "chroma:document": "CODE:\n\ndef main():\n    # See all possible arguments in src/transformers/training_args.py\n    # or by passing the --help flag to this script.\n    # We now keep distinct sets of args, for a cleaner separation of concerns.\n\n    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TrainingArguments))\n    if len(sys.argv) == 2 and sys.argv[1].endswith(\".json\"):\n        # If we pass only one argument to the script and it's the path to a json file,\n        # let's parse it to get our arguments.\n        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))\n    else:\n        model_args, data_args, training_args = parser.parse_args_into_dataclasses()\n\n    # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The\n    # information sent is the one passed as arguments along with your Python/PyTorch versions.\n    send_example_telemetry(\"run_plm\", model_args, data_args)\n\n    # Setup logging\n    logging.basicConfig(\n        format=\"%(asctime)s - %(levelname)s - %(name)s - %(message)s\",\n        datefmt=\"%m/%d/%Y %H:%M:%S\",\n        handlers=[logging.StreamHandler(sys.stdout)],\n    )\n\n    if training_args.should_log:\n        # The default of training_args.log_level is passive, so we set log level at info here to have that default.\n        transformers.utils.logging.set_verbosity_info()\n\n    log_level = training_args.get_process_log_level()\n    logger.setLevel(log_level)\n    datasets.utils.logging.set_verbosity(log_level)\n    transformers.utils.logging.set_verbosity(log_level)\n    transformers.utils.logging.enable_default_handler()\n    transformers.utils.logging.enable_explicit_format()\n\n    # Log on each process the small summary:\n    logger.warning(\n        f\"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}, \"\n        + f\"distributed training: {training_args.parallel_mode.value == 'distributed'}, 16-bits training: {training_args.fp16}\"\n    )\n    logger.info(f\"Training/evaluation parameters {training_args}\")\n\n    # Detecting last checkpoint.\n    last_checkpoint = None\n    if os.path.isdir(training_args.output_dir) and training_args.do_train and not training_args.overwrite_output_dir:\n        last_checkpoint = get_last_checkpoint(training_args.output_dir)\n        if last_checkpoint is None and len(os.listdir(training_args.output_dir)) > 0:\n            raise ValueError(\n                f\"Output directory ({training_args.output_dir}) already exists and is not empty. \"\n                \"Use --overwrite_output_dir to overcome.\"\n            )\n        elif last_checkpoint is not None and training_args.resume_from_checkpoint is None:\n            logger.info(\n                f\"Checkpoint detected, resuming training at {last_checkpoint}. To avoid this behavior, change \"\n                \"the `--output_dir` or add `--overwrite_output_dir` to train from scratch.\"\n            )\n\n    # Set seed before initializing model.\n    set_seed(training_args.seed)\n\n    # Get the datasets: you can either provide your own CSV/JSON/TXT training and evaluation files (see below)\n    # or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/\n    # (the dataset will be downloaded automatically from the datasets Hub).\n    #\n    # For CSV/JSON files, this script will use the column called 'text' or the first column if no column called\n    # 'text' is found. You can easily tweak this behavior (see below).\n    #\n    # In distributed training, the load_dataset function guarantee that only one local process can concurrently\n    # download the dataset.\n    if data_args.dataset_name is not None:\n        # Downloading and loading a dataset from the hub.\n        raw_datasets = load_dataset(\n            data_args.dataset_name,\n            data_args.dataset_config_name,\n            cache_dir=model_args.cache_dir,\n            token=model_args.token,\n            trust_remote_code=data_args.trust_remote_code,\n        )\n        if \"validation\" not in raw_datasets.keys():\n            raw_datasets[\"validation\"] = load_dataset(\n                data_args.dataset_name,\n                data_args.dataset_config_name,\n                split=f\"train[:{data_args.validation_split_percentage}%]\",\n                cache_dir=model_args.cache_dir,\n                token=model_args.token,\n                trust_remote_code=data_args.trust_remote_code,\n            )\n            raw_datasets[\"train\"] = load_dataset(\n                data_args.dataset_name,\n                data_args.dataset_config_name,\n                split=f\"train[{data_args.validation_split_percentage}%:]\",\n                cache_dir=model_args.cache_dir,\n                token=model_args.token,\n                trust_remote_code=data_args.trust_remote_code,\n            )\n    else:\n        data_files = {}\n        if data_args.train_file is not None:\n            data_files[\"train\"] = data_args.train_file\n            extension = data_args.train_file.split(\".\")[-1]\n        if data_args.validation_file is not None:\n            data_files[\"validation\"] = data_args.validation_file\n            extension = data_args.validation_file.split(\".\")[-1]\n        if extension == \"txt\":\n            extension = \"text\"\n        raw_datasets = load_dataset(extension, data_files=data_files, cache_dir=model_args.cache_dir)\n        # If no validation data is there, validation_split_percentage will be used to divide the dataset.\n        if \"validation\" not in raw_datasets.keys():\n            raw_datasets[\"validation\"] = load_dataset(\n                extension,\n                data_files=data_files,\n                split=f\"train[:{data_args.validation_split_percentage}%]\",\n                cache_dir=model_args.cache_dir,\n                token=model_args.token,\n            )\n            raw_datasets[\"train\"] = load_dataset(\n                extension,\n                data_files=data_files,\n                split=f\"train[{data_args.validation_split_percentage}%:]\",\n                cache_dir=model_args.cache_dir,\n                token=model_args.token,\n            )\n\n    # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at\n    # https://huggingface.co/docs/datasets/loading_datasets.\n\n    # Load pretrained model and tokenizer\n    #\n    # Distributed training:\n    # The .from_pretrained methods guarantee that only one local process can concurrently\n    # download model & vocab.\n    config_kwargs = {\n        \"cache_dir\": model_args.cache_dir,\n        \"revision\": model_args.model_revision,\n        \"token\": model_args.token,\n    }\n    if model_args.config_name:\n        config = AutoConfig.from_pretrained(model_args.config_name, **config_kwargs)\n    elif model_args.model_name_or_path:\n        config = AutoConfig.from_pretrained(model_args.model_name_or_path, **config_kwargs)\n    else:\n        config = XLNetConfig()\n        logger.warning(\"You are instantiating a new config instance from scratch.\")\n        if model_args.config_overrides is not None:\n            logger.info(f\"Overriding config: {model_args.config_overrides}\")\n            config.update_from_string(model_args.config_overrides)\n            logger.info(f\"New config: {config}\")\n\n    tokenizer_kwargs = {\n        \"cache_dir\": model_args.cache_dir,\n        \"use_fast\": model_args.use_fast_tokenizer,\n        \"revision\": model_args.model_revision,\n        \"token\": model_args.token,\n    }\n    if model_args.tokenizer_name:\n        tokenizer = AutoTokenizer.from_pretrained(model_args.tokenizer_name, **tokenizer_kwargs)\n    elif model_args.model_name_or_path:\n        tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path, **tokenizer_kwargs)\n    else:\n        raise ValueError(\n            \"You are instantiating a new tokenizer from scratch. This is not supported by this script. \"\n            \"You can do it from another script, save it, and load it from here, using --tokenizer_name.\"\n        )\n\n    if model_args.model_name_or_path:\n        model = XLNetLMHeadModel.from_pretrained(\n            model_args.model_name_or_path,\n            from_tf=bool(\".ckpt\" in model_args.model_name_or_path),\n            config=config,\n            cache_dir=model_args.cache_dir,\n            revision=model_args.model_revision,\n            token=model_args.token,\n            low_cpu_mem_usage=model_args.low_cpu_mem_usage,\n        )\n    else:\n        logger.info(\"Training new model from scratch\")\n        model = XLNetLMHeadModel(config)\n\n    # We resize the embeddings only when necessary to avoid index errors. If you are creating a model from scratch\n    # on a small vocab and want a smaller embedding size, remove this test.\n    embedding_size = model.get_input_embeddings().weight.shape[0]\n    if len(tokenizer) > embedding_size:\n        model.resize_token_embeddings(len(tokenizer))\n\n    # Preprocessing the datasets.\n    # First we tokenize all the texts.\n    if training_args.do_train:\n        column_names = raw_datasets[\"train\"].column_names\n    else:\n        column_names = raw_datasets[\"validation\"].column_names\n    text_column_name = \"text\" if \"text\" in column_names else column_names[0]\n\n    if data_args.max_seq_length > tokenizer.model_max_length:\n        logger.warning(\n            f\"The max_seq_length passed ({data_args.max_seq_length}) is larger than the maximum length for the \"\n            f\"model ({tokenizer.model_max_length}). Using max_seq_length={tokenizer.model_max_length}.\"\n        )\n    max_seq_length = min(data_args.max_seq_length, tokenizer.model_max_length)\n\n    if data_args.line_by_line:\n        # When using line_by_line, we just tokenize each nonempty line.\n        padding = \"max_length\" if data_args.pad_to_max_length else False\n\n        def tokenize_function(examples):\n            # Remove empty lines\n            examples[\"text\"] = [line for line in examples[\"text\"] if len(line) > 0 and not line.isspace()]\n            return tokenizer(examples[\"text\"], padding=padding, truncation=True, max_length=max_seq_length)\n\n        with training_args.main_process_first(desc=\"dataset map tokenization\"):\n            tokenized_datasets = raw_datasets.map(\n                tokenize_function,\n                batched=True,\n                num_proc=data_args.preprocessing_num_workers,\n                remove_columns=[text_column_name],\n                load_from_cache_file=not data_args.overwrite_cache,\n                desc=\"Running tokenizer on dataset line_by_line\",\n            )\n    else:\n        # Otherwise, we tokenize every text, then concatenate them together before splitting them in smaller parts.\n        def tokenize_function(examples):\n            return tokenizer(examples[text_column_name])\n\n        with training_args.main_process_first(desc=\"dataset map tokenization\"):\n            tokenized_datasets = raw_datasets.map(\n                tokenize_function,\n                batched=True,\n                num_proc=data_args.preprocessing_num_workers,\n                remove_columns=column_names,\n                load_from_cache_file=not data_args.overwrite_cache,\n                desc=\"Running tokenizer on every text in dataset\",\n            )\n\n        # Main data processing function that will concatenate all texts from our dataset and generate chunks of\n        # max_seq_length.\n        def group_texts(examples):\n            # Concatenate all texts.\n            concatenated_examples = {k: list(chain(*examples[k])) for k in examples.keys()}\n            total_length = len(concatenated_examples[list(examples.keys())[0]])\n            # We drop the small remainder, and if the total_length < max_seq_length  we exclude this batch and return an empty dict.\n            # We could add padding if the model supported it instead of this drop, you can customize this part to your needs.\n            total_length = (total_length // max_seq_length) * max_seq_length\n            # Split by chunks of max_len.\n            result = {\n                k: [t[i : i + max_seq_length] for i in range(0, total_length, max_seq_length)]\n                for k, t in concatenated_examples.items()\n            }\n            return result\n\n        # Note that with `batched=True`, this map processes 1,000 texts together, so group_texts throws away a\n        # remainder for each of those groups of 1,000 texts. You can adjust that batch_size here but a higher value\n        # might be slower to preprocess.\n        #\n        # To speed up this part, we use multiprocessing. See the documentation of the map method for more information:\n        # https://huggingface.co/docs/datasets/process#map\n\n        with training_args.main_process_first(desc=\"grouping texts together\"):\n            tokenized_datasets = tokenized_datasets.map(\n                group_texts,\n                batched=True,\n                num_proc=data_args.preprocessing_num_workers,\n                load_from_cache_file=not data_args.overwrite_cache,\n                desc=f\"Grouping texts in chunks of {max_seq_length}\",\n            )\n\n    if training_args.do_train:\n        if \"train\" not in tokenized_datasets:\n            raise ValueError(\"--do_train requires a train dataset\")\n        train_dataset = tokenized_datasets[\"train\"]\n        if data_args.max_train_samples is not None:\n            max_train_samples = min(len(train_dataset), data_args.max_train_samples)\n            train_dataset = train_dataset.select(range(max_train_samples))\n\n    if training_args.do_eval:\n        if \"validation\" not in tokenized_datasets:\n            raise ValueError(\"--do_eval requires a validation dataset\")\n        eval_dataset = tokenized_datasets[\"validation\"]\n        if data_args.max_eval_samples is not None:\n            max_eval_samples = min(len(eval_dataset), data_args.max_eval_samples)\n            eval_dataset = eval_dataset.select(range(max_eval_samples))\n\n    # Data collator\n    data_collator = DataCollatorForPermutationLanguageModeling(\n        tokenizer=tokenizer,\n        plm_probability=data_args.plm_probability,\n        max_span_length=data_args.max_span_length,\n    )\n\n    # Initialize our Trainer\n    trainer = Trainer(\n        model=model,\n        args=training_args,\n        train_dataset=train_dataset if training_args.do_train else None,\n        eval_dataset=eval_dataset if training_args.do_eval else None,\n        processing_classtokenizer=tokenizer,\n        data_collator=data_collator,\n    )\n\n    # Training\n    if training_args.do_train:\n        checkpoint = None\n        if training_args.resume_from_checkpoint is not None:\n            checkpoint = training_args.resume_from_checkpoint\n        elif last_checkpoint is not None:\n            checkpoint = last_checkpoint\n        train_result = trainer.train(resume_from_checkpoint=checkpoint)\n        trainer.save_model()  # Saves the tokenizer too for easy upload\n        metrics = train_result.metrics\n\n        max_train_samples = (\n            data_args.max_train_samples if data_args.max_train_samples is not None else len(train_dataset)\n        )\n        metrics[\"train_samples\"] = min(max_train_samples, len(train_dataset))\n\n        trainer.log_metrics(\"train\", metrics)\n        trainer.save_metrics(\"train\", metrics)\n        trainer.save_state()\n\n    # Evaluation\n    if training_args.do_eval:\n        logger.info(\"*** Evaluate ***\")\n\n        metrics = trainer.evaluate()\n\n        max_eval_samples = data_args.max_eval_samples if data_args.max_eval_samples is not None else len(eval_dataset)\n        metrics[\"eval_samples\"] = min(max_eval_samples, len(eval_dataset))\n        try:\n            perplexity = math.exp(metrics[\"eval_loss\"])\n        except OverflowError:\n            perplexity = float(\"inf\")\n        metrics[\"perplexity\"] = perplexity\n\n        trainer.log_metrics(\"eval\", metrics)\n        trainer.save_metrics(\"eval\", metrics)\n\n    kwargs = {\"finetuned_from\": model_args.model_name_or_path, \"tasks\": \"language-modeling\"}\n    if data_args.dataset_name is not None:\n        kwargs[\"dataset_tags\"] = data_args.dataset_name\n        if data_args.dataset_config_name is not None:\n            kwargs[\"dataset_args\"] = data_args.dataset_config_name\n            kwargs[\"dataset\"] = f\"{data_args.dataset_name} {data_args.dataset_config_name}\"\n        else:\n            kwargs[\"dataset\"] = data_args.dataset_name\n\n    if training_args.push_to_hub:\n        trainer.push_to_hub(**kwargs)\n    else:\n\n\nSUMMARY:\nThis is a PyTorch code snippet that defines a script for training and evaluating a pre-trained model, likely on the Hugging Face Transformers library. The script:\n\n1. Initializes a dataset from a specified data source (e.g., text file) and collates it into batches of permutated language inputs.\n2. Setups a `Trainer` object with the trained model, training arguments, and evaluation datasets.\n3. Trains the model using the trainer, saving checkpoints as needed.\n4. Evaluates the model on the validation dataset, logging metrics.\n\nSome high-level points:\n\n* The script is designed to work with Hugging Face's Transformers library, which provides a simple framework for text processing and machine learning tasks.\n* The `Trainer` object is used to manage training and evaluation of the model, including saving checkpoints and metrics.\n* The script uses PyTorch's built-in tensors and data structures to process and store input data.\n* The final script saves output models, logs train and eval metrics, and optionally pushes the trained model to Hugging Face's Hub.\n\nExample use case:\n\nTo use this code as a starting point for fine-tuning a pre-trained language translation model on a custom dataset, you would:\n\n1. Replace the `model_args` dictionary with your own training arguments (e.g., maximum sequence length).\n2. Update the `data_collator` to accommodate any specific requirements of your model.\n3. Specify your own datasets in the script (e.g., using the `Dataset.from_iterable()` constructor in Hugging Face's library).\n\nSome key functions used in this code:\n\n* `Trainer.train()`: Trains a model on the specified dataset and evaluation metrics.\n* `Trainer.evaluate()`: Evaluates a model on the validation dataset, returning loss values and other metrics.\n* `DataCollatorForPermutationLanguageModeling` is used to collate permutated input data in batches.\n\nPlease note that some variables are set to default or specific values (e.g., `max_train_samples`, `eval_loss`) which you can change according to your data size and optimization requirements."}


created_at: 2025-02-05 09:13:39
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/language-modeling/run_plm.py:577-579
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 579, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/language-modeling/run_plm.py", "name": "_mp_fn", "start_line": 577, "type": "function", "chroma:document": "CODE:\n\ndef _mp_fn(index):\n    # For xla_spawn (TPUs)\n\n\nSUMMARY:\n**Summary:**\n\n`_mp_fn(index)` appears to be a function wrapper for distributing computation across multiple computing resources, likely relevant for TPUs (Tensor Processing Units). The main purpose is to execute the `main()` function in an multi-processing context. \n\n**Key Details:**\n\n- Uses `_mp_fn(index)`, suggesting potential multi-threading or process management.\n- Calls `main()`, indicating that this function serves as a wrapper around the primary computation logic.\n\n**Notable Dependencies/Requirements:**\n\n- Relies on XLA Spawn (TPUs), a technology introduced by Google to simplify machine learning inference on TPUs."}


created_at: 2025-02-05 09:13:39
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/language-modeling/run_mlm.py:248-681
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 681, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/language-modeling/run_mlm.py", "name": "main", "start_line": 248, "type": "function", "chroma:document": "CODE:\n\ndef main():\n    # See all possible arguments in src/transformers/training_args.py\n    # or by passing the --help flag to this script.\n    # We now keep distinct sets of args, for a cleaner separation of concerns.\n\n    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TrainingArguments))\n    if len(sys.argv) == 2 and sys.argv[1].endswith(\".json\"):\n        # If we pass only one argument to the script and it's the path to a json file,\n        # let's parse it to get our arguments.\n        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))\n    else:\n        model_args, data_args, training_args = parser.parse_args_into_dataclasses()\n\n    # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The\n    # information sent is the one passed as arguments along with your Python/PyTorch versions.\n    send_example_telemetry(\"run_mlm\", model_args, data_args)\n\n    # Setup logging\n    logging.basicConfig(\n        format=\"%(asctime)s - %(levelname)s - %(name)s - %(message)s\",\n        datefmt=\"%m/%d/%Y %H:%M:%S\",\n        handlers=[logging.StreamHandler(sys.stdout)],\n    )\n\n    if training_args.should_log:\n        # The default of training_args.log_level is passive, so we set log level at info here to have that default.\n        transformers.utils.logging.set_verbosity_info()\n\n    log_level = training_args.get_process_log_level()\n    logger.setLevel(log_level)\n    datasets.utils.logging.set_verbosity(log_level)\n    transformers.utils.logging.set_verbosity(log_level)\n    transformers.utils.logging.enable_default_handler()\n    transformers.utils.logging.enable_explicit_format()\n\n    # Log on each process the small summary:\n    logger.warning(\n        f\"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}, \"\n        + f\"distributed training: {training_args.parallel_mode.value == 'distributed'}, 16-bits training: {training_args.fp16}\"\n    )\n    # Set the verbosity to info of the Transformers logger (on main process only):\n    logger.info(f\"Training/evaluation parameters {training_args}\")\n\n    # Detecting last checkpoint.\n    last_checkpoint = None\n    if os.path.isdir(training_args.output_dir) and training_args.do_train and not training_args.overwrite_output_dir:\n        last_checkpoint = get_last_checkpoint(training_args.output_dir)\n        if last_checkpoint is None and len(os.listdir(training_args.output_dir)) > 0:\n            raise ValueError(\n                f\"Output directory ({training_args.output_dir}) already exists and is not empty. \"\n                \"Use --overwrite_output_dir to overcome.\"\n            )\n        elif last_checkpoint is not None and training_args.resume_from_checkpoint is None:\n            logger.info(\n                f\"Checkpoint detected, resuming training at {last_checkpoint}. To avoid this behavior, change \"\n                \"the `--output_dir` or add `--overwrite_output_dir` to train from scratch.\"\n            )\n\n    # Set seed before initializing model.\n    set_seed(training_args.seed)\n\n    # Get the datasets: you can either provide your own CSV/JSON/TXT training and evaluation files (see below)\n    # or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/\n    # (the dataset will be downloaded automatically from the datasets Hub\n    #\n    # For CSV/JSON files, this script will use the column called 'text' or the first column. You can easily tweak this\n    # behavior (see below)\n    #\n    # In distributed training, the load_dataset function guarantee that only one local process can concurrently\n    # download the dataset.\n    if data_args.dataset_name is not None:\n        # Downloading and loading a dataset from the hub.\n        raw_datasets = load_dataset(\n            data_args.dataset_name,\n            data_args.dataset_config_name,\n            cache_dir=model_args.cache_dir,\n            token=model_args.token,\n            streaming=data_args.streaming,\n            trust_remote_code=model_args.trust_remote_code,\n        )\n        if \"validation\" not in raw_datasets.keys():\n            raw_datasets[\"validation\"] = load_dataset(\n                data_args.dataset_name,\n                data_args.dataset_config_name,\n                split=f\"train[:{data_args.validation_split_percentage}%]\",\n                cache_dir=model_args.cache_dir,\n                token=model_args.token,\n                streaming=data_args.streaming,\n                trust_remote_code=model_args.trust_remote_code,\n            )\n            raw_datasets[\"train\"] = load_dataset(\n                data_args.dataset_name,\n                data_args.dataset_config_name,\n                split=f\"train[{data_args.validation_split_percentage}%:]\",\n                cache_dir=model_args.cache_dir,\n                token=model_args.token,\n                streaming=data_args.streaming,\n                trust_remote_code=model_args.trust_remote_code,\n            )\n    else:\n        data_files = {}\n        if data_args.train_file is not None:\n            data_files[\"train\"] = data_args.train_file\n            extension = data_args.train_file.split(\".\")[-1]\n        if data_args.validation_file is not None:\n            data_files[\"validation\"] = data_args.validation_file\n            extension = data_args.validation_file.split(\".\")[-1]\n        if extension == \"txt\":\n            extension = \"text\"\n        raw_datasets = load_dataset(\n            extension,\n            data_files=data_files,\n            cache_dir=model_args.cache_dir,\n            token=model_args.token,\n        )\n\n        # If no validation data is there, validation_split_percentage will be used to divide the dataset.\n        if \"validation\" not in raw_datasets.keys():\n            raw_datasets[\"validation\"] = load_dataset(\n                extension,\n                data_files=data_files,\n                split=f\"train[:{data_args.validation_split_percentage}%]\",\n                cache_dir=model_args.cache_dir,\n                token=model_args.token,\n            )\n            raw_datasets[\"train\"] = load_dataset(\n                extension,\n                data_files=data_files,\n                split=f\"train[{data_args.validation_split_percentage}%:]\",\n                cache_dir=model_args.cache_dir,\n                token=model_args.token,\n            )\n\n    # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at\n    # https://huggingface.co/docs/datasets/loading_datasets.\n\n    # Load pretrained model and tokenizer\n    #\n    # Distributed training:\n    # The .from_pretrained methods guarantee that only one local process can concurrently\n    # download model & vocab.\n    config_kwargs = {\n        \"cache_dir\": model_args.cache_dir,\n        \"revision\": model_args.model_revision,\n        \"token\": model_args.token,\n        \"trust_remote_code\": model_args.trust_remote_code,\n    }\n    if model_args.config_name:\n        config = AutoConfig.from_pretrained(model_args.config_name, **config_kwargs)\n    elif model_args.model_name_or_path:\n        config = AutoConfig.from_pretrained(model_args.model_name_or_path, **config_kwargs)\n    else:\n        config = CONFIG_MAPPING[model_args.model_type]()\n        logger.warning(\"You are instantiating a new config instance from scratch.\")\n        if model_args.config_overrides is not None:\n            logger.info(f\"Overriding config: {model_args.config_overrides}\")\n            config.update_from_string(model_args.config_overrides)\n            logger.info(f\"New config: {config}\")\n\n    tokenizer_kwargs = {\n        \"cache_dir\": model_args.cache_dir,\n        \"use_fast\": model_args.use_fast_tokenizer,\n        \"revision\": model_args.model_revision,\n        \"token\": model_args.token,\n        \"trust_remote_code\": model_args.trust_remote_code,\n    }\n    if model_args.tokenizer_name:\n        tokenizer = AutoTokenizer.from_pretrained(model_args.tokenizer_name, **tokenizer_kwargs)\n    elif model_args.model_name_or_path:\n        tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path, **tokenizer_kwargs)\n    else:\n        raise ValueError(\n            \"You are instantiating a new tokenizer from scratch. This is not supported by this script. \"\n            \"You can do it from another script, save it, and load it from here, using --tokenizer_name.\"\n        )\n\n    if model_args.model_name_or_path:\n        torch_dtype = (\n            model_args.torch_dtype\n            if model_args.torch_dtype in [\"auto\", None]\n            else getattr(torch, model_args.torch_dtype)\n        )\n        model = AutoModelForMaskedLM.from_pretrained(\n            model_args.model_name_or_path,\n            from_tf=bool(\".ckpt\" in model_args.model_name_or_path),\n            config=config,\n            cache_dir=model_args.cache_dir,\n            revision=model_args.model_revision,\n            token=model_args.token,\n            trust_remote_code=model_args.trust_remote_code,\n            torch_dtype=torch_dtype,\n            low_cpu_mem_usage=model_args.low_cpu_mem_usage,\n        )\n    else:\n        logger.info(\"Training new model from scratch\")\n        model = AutoModelForMaskedLM.from_config(config, trust_remote_code=model_args.trust_remote_code)\n\n    # We resize the embeddings only when necessary to avoid index errors. If you are creating a model from scratch\n    # on a small vocab and want a smaller embedding size, remove this test.\n    embedding_size = model.get_input_embeddings().weight.shape[0]\n    if len(tokenizer) > embedding_size:\n        model.resize_token_embeddings(len(tokenizer))\n\n    # Preprocessing the datasets.\n    # First we tokenize all the texts.\n    if training_args.do_train:\n        column_names = list(raw_datasets[\"train\"].features)\n    else:\n        column_names = list(raw_datasets[\"validation\"].features)\n    text_column_name = \"text\" if \"text\" in column_names else column_names[0]\n\n    if data_args.max_seq_length is None:\n        max_seq_length = tokenizer.model_max_length\n        if max_seq_length > 1024:\n            logger.warning(\n                \"The chosen tokenizer supports a `model_max_length` that is longer than the default `block_size` value\"\n                \" of 1024. If you would like to use a longer `block_size` up to `tokenizer.model_max_length` you can\"\n                \" override this default with `--block_size xxx`.\"\n            )\n            max_seq_length = 1024\n    else:\n        if data_args.max_seq_length > tokenizer.model_max_length:\n            logger.warning(\n                f\"The max_seq_length passed ({data_args.max_seq_length}) is larger than the maximum length for the \"\n                f\"model ({tokenizer.model_max_length}). Using max_seq_length={tokenizer.model_max_length}.\"\n            )\n        max_seq_length = min(data_args.max_seq_length, tokenizer.model_max_length)\n\n    if data_args.line_by_line:\n        # When using line_by_line, we just tokenize each nonempty line.\n        padding = \"max_length\" if data_args.pad_to_max_length else False\n\n        def tokenize_function(examples):\n            # Remove empty lines\n            examples[text_column_name] = [\n                line for line in examples[text_column_name] if len(line) > 0 and not line.isspace()\n            ]\n            return tokenizer(\n                examples[text_column_name],\n                padding=padding,\n                truncation=True,\n                max_length=max_seq_length,\n                # We use this option because DataCollatorForLanguageModeling (see below) is more efficient when it\n                # receives the `special_tokens_mask`.\n                return_special_tokens_mask=True,\n            )\n\n        with training_args.main_process_first(desc=\"dataset map tokenization\"):\n            if not data_args.streaming:\n                tokenized_datasets = raw_datasets.map(\n                    tokenize_function,\n                    batched=True,\n                    num_proc=data_args.preprocessing_num_workers,\n                    remove_columns=[text_column_name],\n                    load_from_cache_file=not data_args.overwrite_cache,\n                    desc=\"Running tokenizer on dataset line_by_line\",\n                )\n            else:\n                tokenized_datasets = raw_datasets.map(\n                    tokenize_function,\n                    batched=True,\n                    remove_columns=[text_column_name],\n                )\n    else:\n        # Otherwise, we tokenize every text, then concatenate them together before splitting them in smaller parts.\n        # We use `return_special_tokens_mask=True` because DataCollatorForLanguageModeling (see below) is more\n        # efficient when it receives the `special_tokens_mask`.\n        def tokenize_function(examples):\n            return tokenizer(examples[text_column_name], return_special_tokens_mask=True)\n\n        with training_args.main_process_first(desc=\"dataset map tokenization\"):\n            if not data_args.streaming:\n                tokenized_datasets = raw_datasets.map(\n                    tokenize_function,\n                    batched=True,\n                    num_proc=data_args.preprocessing_num_workers,\n                    remove_columns=column_names,\n                    load_from_cache_file=not data_args.overwrite_cache,\n                    desc=\"Running tokenizer on every text in dataset\",\n                )\n            else:\n                tokenized_datasets = raw_datasets.map(\n                    tokenize_function,\n                    batched=True,\n                    remove_columns=column_names,\n                )\n\n        # Main data processing function that will concatenate all texts from our dataset and generate chunks of\n        # max_seq_length.\n        def group_texts(examples):\n            # Concatenate all texts.\n            concatenated_examples = {k: list(chain(*examples[k])) for k in examples.keys()}\n            total_length = len(concatenated_examples[list(examples.keys())[0]])\n            # We drop the small remainder, and if the total_length < max_seq_length  we exclude this batch and return an empty dict.\n            # We could add padding if the model supported it instead of this drop, you can customize this part to your needs.\n            total_length = (total_length // max_seq_length) * max_seq_length\n            # Split by chunks of max_len.\n            result = {\n                k: [t[i : i + max_seq_length] for i in range(0, total_length, max_seq_length)]\n                for k, t in concatenated_examples.items()\n            }\n            return result\n\n        # Note that with `batched=True`, this map processes 1,000 texts together, so group_texts throws away a\n        # remainder for each of those groups of 1,000 texts. You can adjust that batch_size here but a higher value\n        # might be slower to preprocess.\n        #\n        # To speed up this part, we use multiprocessing. See the documentation of the map method for more information:\n        # https://huggingface.co/docs/datasets/process#map\n\n        with training_args.main_process_first(desc=\"grouping texts together\"):\n            if not data_args.streaming:\n                tokenized_datasets = tokenized_datasets.map(\n                    group_texts,\n                    batched=True,\n                    num_proc=data_args.preprocessing_num_workers,\n                    load_from_cache_file=not data_args.overwrite_cache,\n                    desc=f\"Grouping texts in chunks of {max_seq_length}\",\n                )\n            else:\n                tokenized_datasets = tokenized_datasets.map(\n                    group_texts,\n                    batched=True,\n                )\n\n    if training_args.do_train:\n        if \"train\" not in tokenized_datasets:\n            raise ValueError(\"--do_train requires a train dataset\")\n        train_dataset = tokenized_datasets[\"train\"]\n        if data_args.max_train_samples is not None:\n            max_train_samples = min(len(train_dataset), data_args.max_train_samples)\n            train_dataset = train_dataset.select(range(max_train_samples))\n\n    if training_args.do_eval:\n        if \"validation\" not in tokenized_datasets:\n            raise ValueError(\"--do_eval requires a validation dataset\")\n        eval_dataset = tokenized_datasets[\"validation\"]\n        if data_args.max_eval_samples is not None:\n            max_eval_samples = min(len(eval_dataset), data_args.max_eval_samples)\n            eval_dataset = eval_dataset.select(range(max_eval_samples))\n\n        def preprocess_logits_for_metrics(logits, labels):\n            if isinstance(logits, tuple):\n                # Depending on the model and config, logits may contain extra tensors,\n                # like past_key_values, but logits always come first\n                logits = logits[0]\n            return logits.argmax(dim=-1)\n\n        metric = evaluate.load(\"accuracy\", cache_dir=model_args.cache_dir)\n\n        def compute_metrics(eval_preds):\n            preds, labels = eval_preds\n            # preds have the same shape as the labels, after the argmax(-1) has been calculated\n            # by preprocess_logits_for_metrics\n            labels = labels.reshape(-1)\n            preds = preds.reshape(-1)\n            mask = labels != -100\n            labels = labels[mask]\n            preds = preds[mask]\n            return metric.compute(predictions=preds, references=labels)\n\n    # Data collator\n    # This one will take care of randomly masking the tokens.\n    pad_to_multiple_of_8 = data_args.line_by_line and training_args.fp16 and not data_args.pad_to_max_length\n    data_collator = DataCollatorForLanguageModeling(\n        tokenizer=tokenizer,\n        mlm_probability=data_args.mlm_probability,\n        pad_to_multiple_of=8 if pad_to_multiple_of_8 else None,\n    )\n\n    # Initialize our Trainer\n    trainer = Trainer(\n        model=model,\n        args=training_args,\n        train_dataset=train_dataset if training_args.do_train else None,\n        eval_dataset=eval_dataset if training_args.do_eval else None,\n        processing_class=tokenizer,\n        data_collator=data_collator,\n        compute_metrics=compute_metrics if training_args.do_eval and not is_torch_xla_available() else None,\n        preprocess_logits_for_metrics=preprocess_logits_for_metrics\n        if training_args.do_eval and not is_torch_xla_available()\n        else None,\n    )\n\n    # Training\n    if training_args.do_train:\n        checkpoint = None\n        if training_args.resume_from_checkpoint is not None:\n            checkpoint = training_args.resume_from_checkpoint\n        elif last_checkpoint is not None:\n            checkpoint = last_checkpoint\n        train_result = trainer.train(resume_from_checkpoint=checkpoint)\n        trainer.save_model()  # Saves the tokenizer too for easy upload\n        metrics = train_result.metrics\n\n        max_train_samples = (\n            data_args.max_train_samples if data_args.max_train_samples is not None else len(train_dataset)\n        )\n        metrics[\"train_samples\"] = min(max_train_samples, len(train_dataset))\n\n        trainer.log_metrics(\"train\", metrics)\n        trainer.save_metrics(\"train\", metrics)\n        trainer.save_state()\n\n    # Evaluation\n    if training_args.do_eval:\n        logger.info(\"*** Evaluate ***\")\n\n        metrics = trainer.evaluate()\n\n        max_eval_samples = data_args.max_eval_samples if data_args.max_eval_samples is not None else len(eval_dataset)\n        metrics[\"eval_samples\"] = min(max_eval_samples, len(eval_dataset))\n        try:\n            perplexity = math.exp(metrics[\"eval_loss\"])\n        except OverflowError:\n            perplexity = float(\"inf\")\n        metrics[\"perplexity\"] = perplexity\n\n        trainer.log_metrics(\"eval\", metrics)\n        trainer.save_metrics(\"eval\", metrics)\n\n    kwargs = {\"finetuned_from\": model_args.model_name_or_path, \"tasks\": \"fill-mask\"}\n    if data_args.dataset_name is not None:\n        kwargs[\"dataset_tags\"] = data_args.dataset_name\n        if data_args.dataset_config_name is not None:\n            kwargs[\"dataset_args\"] = data_args.dataset_config_name\n            kwargs[\"dataset\"] = f\"{data_args.dataset_name} {data_args.dataset_config_name}\"\n        else:\n            kwargs[\"dataset\"] = data_args.dataset_name\n\n    if training_args.push_to_hub:\n        trainer.push_to_hub(**kwargs)\n    else:\n\n\nSUMMARY:\nThis is a code snippet for fine-tuning a pre-trained language model (such as BERT) on the fill-mask task using Hugging Face's Trainer API. The goal is to create a custom trained model that can be used to solve the fill-mask problem from a given dataset.\n\nHere are the key points:\n\n1. Loading and preparing datasets\n2. Creating and configuring the model and args\n3. Defining data collators, preprocessors, and metrics\n4. Initializing the Trainer\n5. Training and evaluation\n\nThe code includes various sections for different parts of the process, such as loading datasets and models, defining hyperparameters and data collators, initializing the Trainer, training, and evaluating the model.\n\nNote that this is a complex piece of code that assumes knowledge of Hugging Face's API and Python programming. It may not be suitable for reading or understanding without prior experience with these concepts."}


created_at: 2025-02-05 09:13:39
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/language-modeling/run_mlm.py:684-686
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 686, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/language-modeling/run_mlm.py", "name": "_mp_fn", "start_line": 684, "type": "function", "chroma:document": "CODE:\n\ndef _mp_fn(index):\n    # For xla_spawn (TPUs)\n\n\nSUMMARY:\n**Summary:**\n\nThe `_mp_fn` function is part of a multi-processing framework and serves as an entry point for executing tasks in parallel. It calls the `main` function when operating on Google TPU's (`xla_spawn`). The precise purpose is:\n\n*   **Primary Functionality:** Execute task in parallel using multiple processes (TPUs).\n*   **Key Implementation Detail:** Calls `main()` as a fallback for xla_spawn environment.\n*   **Notable Dependencies/Requirements:**\n    *   `xla_spawn` (Google TPU)\n   "}


created_at: 2025-02-05 09:13:40
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/language-modeling/run_clm.py:237-647
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 647, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/language-modeling/run_clm.py", "name": "main", "start_line": 237, "type": "function", "chroma:document": "CODE:\n\ndef main():\n    # See all possible arguments in src/transformers/training_args.py\n    # or by passing the --help flag to this script.\n    # We now keep distinct sets of args, for a cleaner separation of concerns.\n\n    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TrainingArguments))\n    if len(sys.argv) == 2 and sys.argv[1].endswith(\".json\"):\n        # If we pass only one argument to the script and it's the path to a json file,\n        # let's parse it to get our arguments.\n        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))\n    else:\n        model_args, data_args, training_args = parser.parse_args_into_dataclasses()\n\n    # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The\n    # information sent is the one passed as arguments along with your Python/PyTorch versions.\n    send_example_telemetry(\"run_clm\", model_args, data_args)\n\n    # Setup logging\n    logging.basicConfig(\n        format=\"%(asctime)s - %(levelname)s - %(name)s - %(message)s\",\n        datefmt=\"%m/%d/%Y %H:%M:%S\",\n        handlers=[logging.StreamHandler(sys.stdout)],\n    )\n\n    if training_args.should_log:\n        # The default of training_args.log_level is passive, so we set log level at info here to have that default.\n        transformers.utils.logging.set_verbosity_info()\n\n    log_level = training_args.get_process_log_level()\n    logger.setLevel(log_level)\n    datasets.utils.logging.set_verbosity(log_level)\n    transformers.utils.logging.set_verbosity(log_level)\n    transformers.utils.logging.enable_default_handler()\n    transformers.utils.logging.enable_explicit_format()\n\n    # Log on each process the small summary:\n    logger.warning(\n        f\"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}, \"\n        + f\"distributed training: {training_args.parallel_mode.value == 'distributed'}, 16-bits training: {training_args.fp16}\"\n    )\n    logger.info(f\"Training/evaluation parameters {training_args}\")\n\n    # Detecting last checkpoint.\n    last_checkpoint = None\n    if os.path.isdir(training_args.output_dir) and training_args.do_train and not training_args.overwrite_output_dir:\n        last_checkpoint = get_last_checkpoint(training_args.output_dir)\n        if last_checkpoint is None and len(os.listdir(training_args.output_dir)) > 0:\n            raise ValueError(\n                f\"Output directory ({training_args.output_dir}) already exists and is not empty. \"\n                \"Use --overwrite_output_dir to overcome.\"\n            )\n        elif last_checkpoint is not None and training_args.resume_from_checkpoint is None:\n            logger.info(\n                f\"Checkpoint detected, resuming training at {last_checkpoint}. To avoid this behavior, change \"\n                \"the `--output_dir` or add `--overwrite_output_dir` to train from scratch.\"\n            )\n\n    # Set seed before initializing model.\n    set_seed(training_args.seed)\n\n    # Get the datasets: you can either provide your own CSV/JSON/TXT training and evaluation files (see below)\n    # or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/\n    # (the dataset will be downloaded automatically from the datasets Hub).\n    #\n    # For CSV/JSON files, this script will use the column called 'text' or the first column if no column called\n    # 'text' is found. You can easily tweak this behavior (see below).\n    #\n    # In distributed training, the load_dataset function guarantee that only one local process can concurrently\n    # download the dataset.\n    if data_args.dataset_name is not None:\n        # Downloading and loading a dataset from the hub.\n        raw_datasets = load_dataset(\n            data_args.dataset_name,\n            data_args.dataset_config_name,\n            cache_dir=model_args.cache_dir,\n            token=model_args.token,\n            streaming=data_args.streaming,\n            trust_remote_code=model_args.trust_remote_code,\n        )\n        if \"validation\" not in raw_datasets.keys():\n            raw_datasets[\"validation\"] = load_dataset(\n                data_args.dataset_name,\n                data_args.dataset_config_name,\n                split=f\"train[:{data_args.validation_split_percentage}%]\",\n                cache_dir=model_args.cache_dir,\n                token=model_args.token,\n                streaming=data_args.streaming,\n                trust_remote_code=model_args.trust_remote_code,\n            )\n            raw_datasets[\"train\"] = load_dataset(\n                data_args.dataset_name,\n                data_args.dataset_config_name,\n                split=f\"train[{data_args.validation_split_percentage}%:]\",\n                cache_dir=model_args.cache_dir,\n                token=model_args.token,\n                streaming=data_args.streaming,\n                trust_remote_code=model_args.trust_remote_code,\n            )\n    else:\n        data_files = {}\n        dataset_args = {}\n        if data_args.train_file is not None:\n            data_files[\"train\"] = data_args.train_file\n        if data_args.validation_file is not None:\n            data_files[\"validation\"] = data_args.validation_file\n        extension = (\n            data_args.train_file.split(\".\")[-1]\n            if data_args.train_file is not None\n            else data_args.validation_file.split(\".\")[-1]\n        )\n        if extension == \"txt\":\n            extension = \"text\"\n            dataset_args[\"keep_linebreaks\"] = data_args.keep_linebreaks\n        raw_datasets = load_dataset(\n            extension,\n            data_files=data_files,\n            cache_dir=model_args.cache_dir,\n            token=model_args.token,\n            **dataset_args,\n        )\n        # If no validation data is there, validation_split_percentage will be used to divide the dataset.\n        if \"validation\" not in raw_datasets.keys():\n            raw_datasets[\"validation\"] = load_dataset(\n                extension,\n                data_files=data_files,\n                split=f\"train[:{data_args.validation_split_percentage}%]\",\n                cache_dir=model_args.cache_dir,\n                token=model_args.token,\n                **dataset_args,\n            )\n            raw_datasets[\"train\"] = load_dataset(\n                extension,\n                data_files=data_files,\n                split=f\"train[{data_args.validation_split_percentage}%:]\",\n                cache_dir=model_args.cache_dir,\n                token=model_args.token,\n                **dataset_args,\n            )\n\n    # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at\n    # https://huggingface.co/docs/datasets/loading_datasets.\n\n    # Load pretrained model and tokenizer\n    #\n    # Distributed training:\n    # The .from_pretrained methods guarantee that only one local process can concurrently\n    # download model & vocab.\n\n    config_kwargs = {\n        \"cache_dir\": model_args.cache_dir,\n        \"revision\": model_args.model_revision,\n        \"token\": model_args.token,\n        \"trust_remote_code\": model_args.trust_remote_code,\n    }\n    if model_args.config_name:\n        config = AutoConfig.from_pretrained(model_args.config_name, **config_kwargs)\n    elif model_args.model_name_or_path:\n        config = AutoConfig.from_pretrained(model_args.model_name_or_path, **config_kwargs)\n    else:\n        config = CONFIG_MAPPING[model_args.model_type]()\n        logger.warning(\"You are instantiating a new config instance from scratch.\")\n        if model_args.config_overrides is not None:\n            logger.info(f\"Overriding config: {model_args.config_overrides}\")\n            config.update_from_string(model_args.config_overrides)\n            logger.info(f\"New config: {config}\")\n\n    tokenizer_kwargs = {\n        \"cache_dir\": model_args.cache_dir,\n        \"use_fast\": model_args.use_fast_tokenizer,\n        \"revision\": model_args.model_revision,\n        \"token\": model_args.token,\n        \"trust_remote_code\": model_args.trust_remote_code,\n    }\n    if model_args.tokenizer_name:\n        tokenizer = AutoTokenizer.from_pretrained(model_args.tokenizer_name, **tokenizer_kwargs)\n    elif model_args.model_name_or_path:\n        tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path, **tokenizer_kwargs)\n    else:\n        raise ValueError(\n            \"You are instantiating a new tokenizer from scratch. This is not supported by this script. \"\n            \"You can do it from another script, save it, and load it from here, using --tokenizer_name.\"\n        )\n\n    if model_args.model_name_or_path:\n        torch_dtype = (\n            model_args.torch_dtype\n            if model_args.torch_dtype in [\"auto\", None]\n            else getattr(torch, model_args.torch_dtype)\n        )\n        model = AutoModelForCausalLM.from_pretrained(\n            model_args.model_name_or_path,\n            from_tf=bool(\".ckpt\" in model_args.model_name_or_path),\n            config=config,\n            cache_dir=model_args.cache_dir,\n            revision=model_args.model_revision,\n            token=model_args.token,\n            trust_remote_code=model_args.trust_remote_code,\n            torch_dtype=torch_dtype,\n            low_cpu_mem_usage=model_args.low_cpu_mem_usage,\n        )\n    else:\n        model = AutoModelForCausalLM.from_config(config, trust_remote_code=model_args.trust_remote_code)\n        n_params = sum({p.data_ptr(): p.numel() for p in model.parameters()}.values())\n        logger.info(f\"Training new model from scratch - Total size={n_params/2**20:.2f}M params\")\n\n    # We resize the embeddings only when necessary to avoid index errors. If you are creating a model from scratch\n    # on a small vocab and want a smaller embedding size, remove this test.\n    embedding_size = model.get_input_embeddings().weight.shape[0]\n    if len(tokenizer) > embedding_size:\n        model.resize_token_embeddings(len(tokenizer))\n\n    # Preprocessing the datasets.\n    # First we tokenize all the texts.\n    if training_args.do_train:\n        column_names = list(raw_datasets[\"train\"].features)\n    else:\n        column_names = list(raw_datasets[\"validation\"].features)\n    text_column_name = \"text\" if \"text\" in column_names else column_names[0]\n\n    # since this will be pickled to avoid _LazyModule error in Hasher force logger loading before tokenize_function\n    tok_logger = transformers.utils.logging.get_logger(\"transformers.tokenization_utils_base\")\n\n    def tokenize_function(examples):\n        with CaptureLogger(tok_logger) as cl:\n            output = tokenizer(examples[text_column_name])\n        # clm input could be much much longer than block_size\n        if \"Token indices sequence length is longer than the\" in cl.out:\n            tok_logger.warning(\n                \"^^^^^^^^^^^^^^^^ Please ignore the warning above - this long input will be chunked into smaller bits\"\n                \" before being passed to the model.\"\n            )\n        return output\n\n    with training_args.main_process_first(desc=\"dataset map tokenization\"):\n        if not data_args.streaming:\n            tokenized_datasets = raw_datasets.map(\n                tokenize_function,\n                batched=True,\n                num_proc=data_args.preprocessing_num_workers,\n                remove_columns=column_names,\n                load_from_cache_file=not data_args.overwrite_cache,\n                desc=\"Running tokenizer on dataset\",\n            )\n        else:\n            tokenized_datasets = raw_datasets.map(\n                tokenize_function,\n                batched=True,\n                remove_columns=column_names,\n            )\n    if hasattr(config, \"max_position_embeddings\"):\n        max_pos_embeddings = config.max_position_embeddings\n    else:\n        # Define a default value if the attribute is missing in the config.\n        max_pos_embeddings = 1024\n\n    if data_args.block_size is None:\n        block_size = tokenizer.model_max_length\n        if block_size > max_pos_embeddings:\n            logger.warning(\n                f\"The tokenizer picked seems to have a very large `model_max_length` ({tokenizer.model_max_length}). \"\n                f\"Using block_size={min(1024, max_pos_embeddings)} instead. You can change that default value by passing --block_size xxx.\"\n            )\n            if max_pos_embeddings > 0:\n                block_size = min(1024, max_pos_embeddings)\n            else:\n                block_size = 1024\n    else:\n        if data_args.block_size > tokenizer.model_max_length:\n            logger.warning(\n                f\"The block_size passed ({data_args.block_size}) is larger than the maximum length for the model \"\n                f\"({tokenizer.model_max_length}). Using block_size={tokenizer.model_max_length}.\"\n            )\n        block_size = min(data_args.block_size, tokenizer.model_max_length)\n\n    # Main data processing function that will concatenate all texts from our dataset and generate chunks of block_size.\n    def group_texts(examples):\n        # Concatenate all texts.\n        concatenated_examples = {k: list(chain(*examples[k])) for k in examples.keys()}\n        total_length = len(concatenated_examples[list(examples.keys())[0]])\n        # We drop the small remainder, and if the total_length < block_size  we exclude this batch and return an empty dict.\n        # We could add padding if the model supported it instead of this drop, you can customize this part to your needs.\n        total_length = (total_length // block_size) * block_size\n        # Split by chunks of max_len.\n        result = {\n            k: [t[i : i + block_size] for i in range(0, total_length, block_size)]\n            for k, t in concatenated_examples.items()\n        }\n        result[\"labels\"] = result[\"input_ids\"].copy()\n        return result\n\n    # Note that with `batched=True`, this map processes 1,000 texts together, so group_texts throws away a remainder\n    # for each of those groups of 1,000 texts. You can adjust that batch_size here but a higher value might be slower\n    # to preprocess.\n    #\n    # To speed up this part, we use multiprocessing. See the documentation of the map method for more information:\n    # https://huggingface.co/docs/datasets/process#map\n\n    with training_args.main_process_first(desc=\"grouping texts together\"):\n        if not data_args.streaming:\n            lm_datasets = tokenized_datasets.map(\n                group_texts,\n                batched=True,\n                num_proc=data_args.preprocessing_num_workers,\n                load_from_cache_file=not data_args.overwrite_cache,\n                desc=f\"Grouping texts in chunks of {block_size}\",\n            )\n        else:\n            lm_datasets = tokenized_datasets.map(\n                group_texts,\n                batched=True,\n            )\n\n    if training_args.do_train:\n        if \"train\" not in tokenized_datasets:\n            raise ValueError(\"--do_train requires a train dataset\")\n        train_dataset = lm_datasets[\"train\"]\n        if data_args.max_train_samples is not None:\n            max_train_samples = min(len(train_dataset), data_args.max_train_samples)\n            train_dataset = train_dataset.select(range(max_train_samples))\n\n    if training_args.do_eval:\n        if \"validation\" not in tokenized_datasets:\n            raise ValueError(\"--do_eval requires a validation dataset\")\n        eval_dataset = lm_datasets[\"validation\"]\n        if data_args.max_eval_samples is not None:\n            max_eval_samples = min(len(eval_dataset), data_args.max_eval_samples)\n            eval_dataset = eval_dataset.select(range(max_eval_samples))\n\n        def preprocess_logits_for_metrics(logits, labels):\n            if isinstance(logits, tuple):\n                # Depending on the model and config, logits may contain extra tensors,\n                # like past_key_values, but logits always come first\n                logits = logits[0]\n            return logits.argmax(dim=-1)\n\n        metric = evaluate.load(\"accuracy\", cache_dir=model_args.cache_dir)\n\n        def compute_metrics(eval_preds):\n            preds, labels = eval_preds\n            # preds have the same shape as the labels, after the argmax(-1) has been calculated\n            # by preprocess_logits_for_metrics but we need to shift the labels\n            labels = labels[:, 1:].reshape(-1)\n            preds = preds[:, :-1].reshape(-1)\n            return metric.compute(predictions=preds, references=labels)\n\n    # Initialize our Trainer\n    trainer = Trainer(\n        model=model,\n        args=training_args,\n        train_dataset=train_dataset if training_args.do_train else None,\n        eval_dataset=eval_dataset if training_args.do_eval else None,\n        processing_class=tokenizer,\n        # Data collator will default to DataCollatorWithPadding, so we change it.\n        data_collator=default_data_collator,\n        compute_metrics=compute_metrics if training_args.do_eval and not is_torch_xla_available() else None,\n        preprocess_logits_for_metrics=preprocess_logits_for_metrics\n        if training_args.do_eval and not is_torch_xla_available()\n        else None,\n    )\n\n    # Training\n    if training_args.do_train:\n        checkpoint = None\n        if training_args.resume_from_checkpoint is not None:\n            checkpoint = training_args.resume_from_checkpoint\n        elif last_checkpoint is not None:\n            checkpoint = last_checkpoint\n        train_result = trainer.train(resume_from_checkpoint=checkpoint)\n        trainer.save_model()  # Saves the tokenizer too for easy upload\n\n        metrics = train_result.metrics\n\n        max_train_samples = (\n            data_args.max_train_samples if data_args.max_train_samples is not None else len(train_dataset)\n        )\n        metrics[\"train_samples\"] = min(max_train_samples, len(train_dataset))\n\n        trainer.log_metrics(\"train\", metrics)\n        trainer.save_metrics(\"train\", metrics)\n        trainer.save_state()\n\n    # Evaluation\n    if training_args.do_eval:\n        logger.info(\"*** Evaluate ***\")\n\n        metrics = trainer.evaluate()\n\n        max_eval_samples = data_args.max_eval_samples if data_args.max_eval_samples is not None else len(eval_dataset)\n        metrics[\"eval_samples\"] = min(max_eval_samples, len(eval_dataset))\n        try:\n            perplexity = math.exp(metrics[\"eval_loss\"])\n        except OverflowError:\n            perplexity = float(\"inf\")\n        metrics[\"perplexity\"] = perplexity\n\n        trainer.log_metrics(\"eval\", metrics)\n        trainer.save_metrics(\"eval\", metrics)\n\n    kwargs = {\"finetuned_from\": model_args.model_name_or_path, \"tasks\": \"text-generation\"}\n    if data_args.dataset_name is not None:\n        kwargs[\"dataset_tags\"] = data_args.dataset_name\n        if data_args.dataset_config_name is not None:\n            kwargs[\"dataset_args\"] = data_args.dataset_config_name\n            kwargs[\"dataset\"] = f\"{data_args.dataset_name} {data_args.dataset_config_name}\"\n        else:\n            kwargs[\"dataset\"] = data_args.dataset_name\n\n    if training_args.push_to_hub:\n        trainer.push_to_hub(**kwargs)\n    else:\n\n\nSUMMARY:\nThis code is used to train and evaluate a transformer-based language generation model using the Hugging Face Python library. Here's a high-level overview of what it does:\n\n1. Loads pre-trained tokenizer and model from specific paths or configurations.\n2. Downloads and preprocesses text data for training and evaluation (e.g., dataset names, file paths).\n3. Initializes a `Trainer` object with the trained model, hyperparameters, and necessary datasets (training, evaluation) if specified.\n4. Trains the model using the `train` method with optional resumption from a checkpoint (`resume_from_checkpoint`) or last saved training state (`last_checkpoint`).\n5. Evaluates the model on the evaluation dataset after training is completed.\n6. If requested, fine-tunes the model to match certain specific tasks or configurations (if applicable) and publishes results to Hub.\n\nOverall, this code is an example of how to use Hugging Face for fine-tuning pre-trained language models for text generation tasks."}


created_at: 2025-02-05 09:13:40
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/language-modeling/run_clm.py:650-652
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 652, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/language-modeling/run_clm.py", "name": "_mp_fn", "start_line": 650, "type": "function", "chroma:document": "CODE:\n\ndef _mp_fn(index):\n    # For xla_spawn (TPUs)\n\n\nSUMMARY:\n**Function Summary: `_mp_fn`**\n\n* Primary purpose: The `_mp_fn` function appears to be part of an implementation supporting Google Cloud Tensor Processing Units (TPUs) execution. It likely serves as the main entry point for spawning multi-threading or parallel processing on TPUs.\n* Key implementation details:\n\t+ Calls the `main()` function, which is not defined in this code snippet.\n\t+ Notable dependency: Uses `xla_spawn` library, specifically designed for TPU execution.\n* Notable dependencies and requirements:\n\t+ Inherently uses external libraries (`xla_spawn`) related to TPU execution."}


created_at: 2025-02-05 09:13:40
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/language-modeling/run_fim.py:304-854
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 854, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/language-modeling/run_fim.py", "name": "main", "start_line": 304, "type": "function", "chroma:document": "CODE:\n\ndef main():\n    # See all possible arguments in src/transformers/training_args.py\n    # or by passing the --help flag to this script.\n    # We now keep distinct sets of args, for a cleaner separation of concerns.\n\n    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TrainingArguments))\n    if len(sys.argv) == 2 and sys.argv[1].endswith(\".json\"):\n        # If we pass only one argument to the script and it's the path to a json file,\n        # let's parse it to get our arguments.\n        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))\n    else:\n        model_args, data_args, training_args = parser.parse_args_into_dataclasses()\n\n    # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The\n    # information sent is the one passed as arguments along with your Python/PyTorch versions.\n    send_example_telemetry(\"run_fim\", model_args, data_args)\n\n    # Setup logging\n    logging.basicConfig(\n        format=\"%(asctime)s - %(levelname)s - %(name)s - %(message)s\",\n        datefmt=\"%m/%d/%Y %H:%M:%S\",\n        handlers=[logging.StreamHandler(sys.stdout)],\n    )\n\n    if training_args.should_log:\n        # The default of training_args.log_level is passive, so we set log level at info here to have that default.\n        transformers.utils.logging.set_verbosity_info()\n\n    log_level = training_args.get_process_log_level()\n    logger.setLevel(log_level)\n    datasets.utils.logging.set_verbosity(log_level)\n    transformers.utils.logging.set_verbosity(log_level)\n    transformers.utils.logging.enable_default_handler()\n    transformers.utils.logging.enable_explicit_format()\n\n    # Log on each process the small summary:\n    logger.warning(\n        f\"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}, \"\n        + f\"distributed training: {training_args.parallel_mode.value == 'distributed'}, 16-bits training: {training_args.fp16}\"\n    )\n    logger.info(f\"Training/evaluation parameters {training_args}\")\n\n    # Detecting last checkpoint.\n    last_checkpoint = None\n    if os.path.isdir(training_args.output_dir) and training_args.do_train and not training_args.overwrite_output_dir:\n        last_checkpoint = get_last_checkpoint(training_args.output_dir)\n        if last_checkpoint is None and len(os.listdir(training_args.output_dir)) > 0:\n            raise ValueError(\n                f\"Output directory ({training_args.output_dir}) already exists and is not empty. \"\n                \"Use --overwrite_output_dir to overcome.\"\n            )\n        elif last_checkpoint is not None and training_args.resume_from_checkpoint is None:\n            logger.info(\n                f\"Checkpoint detected, resuming training at {last_checkpoint}. To avoid this behavior, change \"\n                \"the `--output_dir` or add `--overwrite_output_dir` to train from scratch.\"\n            )\n\n    # Set seed before initializing model.\n    set_seed(training_args.seed)\n\n    # Set a numpy random state for FIM transformations\n    np_rng = np.random.RandomState(seed=training_args.seed)\n\n    # Get the datasets: you can either provide your own CSV/JSON/TXT training and evaluation files (see below)\n    # or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/\n    # (the dataset will be downloaded automatically from the datasets Hub).\n    #\n    # For CSV/JSON files, this script will use the column called 'text' or the first column if no column called\n    # 'text' is found. You can easily tweak this behavior (see below).\n    #\n    # In distributed training, the load_dataset function guarantee that only one local process can concurrently\n    # download the dataset.\n    if data_args.dataset_name is not None:\n        # Downloading and loading a dataset from the hub.\n        raw_datasets = load_dataset(\n            data_args.dataset_name,\n            data_args.dataset_config_name,\n            cache_dir=model_args.cache_dir,\n            token=model_args.token,\n            streaming=data_args.streaming,\n            trust_remote_code=model_args.trust_remote_code,\n        )\n        if \"validation\" not in raw_datasets.keys():\n            raw_datasets[\"validation\"] = load_dataset(\n                data_args.dataset_name,\n                data_args.dataset_config_name,\n                split=f\"train[:{data_args.validation_split_percentage}%]\",\n                cache_dir=model_args.cache_dir,\n                token=model_args.token,\n                streaming=data_args.streaming,\n                trust_remote_code=model_args.trust_remote_code,\n            )\n            raw_datasets[\"train\"] = load_dataset(\n                data_args.dataset_name,\n                data_args.dataset_config_name,\n                split=f\"train[{data_args.validation_split_percentage}%:]\",\n                cache_dir=model_args.cache_dir,\n                token=model_args.token,\n                streaming=data_args.streaming,\n                trust_remote_code=model_args.trust_remote_code,\n            )\n    else:\n        data_files = {}\n        dataset_args = {}\n        if data_args.train_file is not None:\n            data_files[\"train\"] = data_args.train_file\n        if data_args.validation_file is not None:\n            data_files[\"validation\"] = data_args.validation_file\n        extension = (\n            data_args.train_file.split(\".\")[-1]\n            if data_args.train_file is not None\n            else data_args.validation_file.split(\".\")[-1]\n        )\n        if extension == \"txt\":\n            extension = \"text\"\n            dataset_args[\"keep_linebreaks\"] = data_args.keep_linebreaks\n        raw_datasets = load_dataset(\n            extension,\n            data_files=data_files,\n            cache_dir=model_args.cache_dir,\n            token=model_args.token,\n            **dataset_args,\n        )\n        # If no validation data is there, validation_split_percentage will be used to divide the dataset.\n        if \"validation\" not in raw_datasets.keys():\n            raw_datasets[\"validation\"] = load_dataset(\n                extension,\n                data_files=data_files,\n                split=f\"train[:{data_args.validation_split_percentage}%]\",\n                cache_dir=model_args.cache_dir,\n                token=model_args.token,\n                **dataset_args,\n            )\n            raw_datasets[\"train\"] = load_dataset(\n                extension,\n                data_files=data_files,\n                split=f\"train[{data_args.validation_split_percentage}%:]\",\n                cache_dir=model_args.cache_dir,\n                token=model_args.token,\n                **dataset_args,\n            )\n\n    # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at\n    # https://huggingface.co/docs/datasets/loading_datasets.html.\n\n    # Load pretrained model and tokenizer\n    #\n    # Distributed training:\n    # The .from_pretrained methods guarantee that only one local process can concurrently\n    # download model & vocab.\n\n    config_kwargs = {\n        \"cache_dir\": model_args.cache_dir,\n        \"revision\": model_args.model_revision,\n        \"token\": model_args.token,\n        \"trust_remote_code\": model_args.trust_remote_code,\n    }\n    if model_args.config_name:\n        config = AutoConfig.from_pretrained(model_args.config_name, **config_kwargs)\n    elif model_args.model_name_or_path:\n        config = AutoConfig.from_pretrained(model_args.model_name_or_path, **config_kwargs)\n    else:\n        config = CONFIG_MAPPING[model_args.model_type]()\n        logger.warning(\"You are instantiating a new config instance from scratch.\")\n        if model_args.config_overrides is not None:\n            logger.info(f\"Overriding config: {model_args.config_overrides}\")\n            config.update_from_string(model_args.config_overrides)\n            logger.info(f\"New config: {config}\")\n\n    tokenizer_kwargs = {\n        \"cache_dir\": model_args.cache_dir,\n        \"use_fast\": model_args.use_fast_tokenizer,\n        \"revision\": model_args.model_revision,\n        \"token\": model_args.token,\n        \"trust_remote_code\": model_args.trust_remote_code,\n    }\n    if model_args.tokenizer_name:\n        tokenizer = AutoTokenizer.from_pretrained(model_args.tokenizer_name, **tokenizer_kwargs)\n    elif model_args.model_name_or_path:\n        tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path, **tokenizer_kwargs)\n    else:\n        raise ValueError(\n            \"You are instantiating a new tokenizer from scratch. This is not supported by this script. \"\n            \"You can do it from another script, save it, and load it from here, using --tokenizer_name.\"\n        )\n\n    if model_args.model_name_or_path:\n        torch_dtype = (\n            model_args.torch_dtype\n            if model_args.torch_dtype in [\"auto\", None]\n            else getattr(torch, model_args.torch_dtype)\n        )\n        model = AutoModelForCausalLM.from_pretrained(\n            model_args.model_name_or_path,\n            from_tf=bool(\".ckpt\" in model_args.model_name_or_path),\n            config=config,\n            cache_dir=model_args.cache_dir,\n            revision=model_args.model_revision,\n            token=model_args.token,\n            trust_remote_code=model_args.trust_remote_code,\n            torch_dtype=torch_dtype,\n            low_cpu_mem_usage=model_args.low_cpu_mem_usage,\n            attn_implementation=model_args.attn_implementation,\n        )\n\n    else:\n        model = AutoModelForCausalLM.from_config(\n            config,\n            trust_remote_code=model_args.trust_remote_code,\n            attn_implementation=model_args.attn_implementation,\n        )\n        n_params = sum({p.data_ptr(): p.numel() for p in model.parameters()}.values())\n        logger.info(f\"Training new model from scratch - Total size={n_params/2**20:.2f}M params\")\n\n    # Add the new FIM tokens to the tokenizer and resize model's vocab embeddings\n    special_tokens = [data_args.fim_prefix_token, data_args.fim_middle_token, data_args.fim_suffix_token]\n    if data_args.truncate_or_pad:\n        special_tokens.append(data_args.pad_token)\n\n    # Get the factor by which the embedding layer should be padded based on the device\n    pad_factor = 1\n    if torch.cuda.is_availble():\n        pad_factor = 8\n\n    elif is_torch_tpu_available():\n        pad_factor = 128\n\n    # Add the new tokens to the tokenizer\n    tokenizer.add_tokens(special_tokens)\n    original_embeddings = model.get_input_embeddings()\n\n    if is_deepspeed_zero3_enabled():\n        import deepspeed\n\n        with deepspeed.zero.GatheredParameters(original_embeddings.weight, modifier_rank=0):\n            # Get the pre-expansion embeddings of the model and resize the embedding layer\n            model.resize_token_embeddings(len(tokenizer), pad_to_multiple_of=pad_factor)\n            embeddings = model.get_input_embeddings()\n\n            # Sample the embeddings for the new tokens from a multivariate normal distribution\n            # We do this so that the new embeddings are close to the original embeddings and not necessarily zero\n            # More on this: https://nlp.stanford.edu/~johnhew/vocab-expansion.html\n            mean = original_embeddings.mean(dim=0)\n            n = original_embeddings.size()[0]\n            sigma = ((original_embeddings - mean).T @ (original_embeddings - mean)) / n\n            dist = torch.distributions.multivariate_normal.MultivariateNormal(\n                mean,\n                covariance_matrix=1e-5 * sigma,\n            )\n            new_token_embeddings = torch.stack(\n                tuple((dist.sample() for _ in range(len(special_tokens)))),\n                dim=0,\n            )\n    else:\n        original_embeddings = model.get_input_embeddings()\n        # Get the pre-expansion embeddings of the model and resize the embedding layer\n        model.resize_token_embeddings(len(tokenizer), pad_to_multiple_of=pad_factor)\n        embeddings = model.get_input_embeddings()\n\n        # Sample the embeddings for the new tokens from a multivariate normal distribution\n        # We do this so that the new embeddings are close to the original embeddings and not necessarily zero\n        # More on this: https://nlp.stanford.edu/~johnhew/vocab-expansion.html\n        mean = original_embeddings.mean(dim=0)\n        n = original_embeddings.size()[0]\n        sigma = ((original_embeddings - mean).T @ (original_embeddings - mean)) / n\n        dist = torch.distributions.multivariate_normal.MultivariateNormal(\n            mean,\n            covariance_matrix=1e-5 * sigma,\n        )\n        new_token_embeddings = torch.stack(\n            tuple((dist.sample() for _ in range(len(special_tokens)))),\n            dim=0,\n        )\n\n    if is_deepspeed_zero3_enabled():\n        import deepspeed\n\n        with deepspeed.zero.GatheredParameters(embeddings.weight, modifier_rank=0):\n            # Set the new tokens' embeddings to the newly sampled embeddings\n            embeddings.weight.data[-len(special_tokens) :] = new_token_embeddings\n    else:\n        # Set the new tokens' embeddings to the newly sampled embeddings\n        embeddings.weight.data[-len(special_tokens) :] = new_token_embeddings\n\n    # Update the model's embeddings with the new embeddings\n    model.set_input_embeddings(embeddings)\n\n    logger.info(\"Added special tokens to the tokenizer and resized model's embedding layer\")\n\n    # Preprocessing the datasets.\n    # First we tokenize all the texts.\n    if training_args.do_train:\n        column_names = list(raw_datasets[\"train\"].features)\n    else:\n        column_names = list(raw_datasets[\"validation\"].features)\n    text_column_name = \"text\" if \"text\" in column_names else column_names[0]\n\n    # since this will be pickled to avoid _LazyModule error in Hasher force logger loading before tokenize_function\n    tok_logger = transformers.utils.logging.get_logger(\"transformers.tokenization_utils_base\")\n\n    def tokenize_function(examples):\n        with CaptureLogger(tok_logger) as cl:\n            output = tokenizer(examples[text_column_name])\n        # clm-fim input could be much much longer than block_size\n        if \"Token indices sequence length is longer than the\" in cl.out:\n            tok_logger.warning(\n                \"^^^^^^^^^^^^^^^^ Please ignore the warning above - this long input will be chunked into smaller bits\"\n                \" before being passed to the model.\"\n            )\n        return output\n\n    with training_args.main_process_first(desc=\"dataset map tokenization\"):\n        if not data_args.streaming:\n            tokenized_datasets = raw_datasets.map(\n                tokenize_function,\n                batched=True,\n                num_proc=data_args.preprocessing_num_workers,\n                remove_columns=column_names,\n                load_from_cache_file=not data_args.overwrite_cache,\n                desc=\"Running tokenizer on dataset\",\n            )\n        else:\n            tokenized_datasets = raw_datasets.map(\n                tokenize_function,\n                batched=True,\n                remove_columns=column_names,\n            )\n\n    if data_args.block_size is None:\n        block_size = tokenizer.model_max_length\n        if block_size > config.max_position_embeddings:\n            logger.warning(\n                f\"The tokenizer picked seems to have a very large `model_max_length` ({tokenizer.model_max_length}). \"\n                f\"Using block_size={min(1024, config.max_position_embeddings)} instead. You can change that default value by passing --block_size xxx.\"\n            )\n            block_size = min(1024, config.max_position_embeddings)\n    else:\n        if data_args.block_size > tokenizer.model_max_length:\n            logger.warning(\n                f\"The block_size passed ({data_args.block_size}) is larger than the maximum length for the model \"\n                f\"({tokenizer.model_max_length}). Using block_size={tokenizer.model_max_length}.\"\n            )\n        block_size = min(data_args.block_size, tokenizer.model_max_length)\n\n    # Data processing function that will concatenate all texts from our dataset and generate chunks of block_size.\n    def group_texts(examples):\n        # Concatenate all texts.\n        concatenated_examples = {k: list(chain(*examples[k])) for k in examples.keys()}\n        total_length = len(concatenated_examples[list(examples.keys())[0]])\n        # We drop the small remainder, and if the total_length < block_size  we exclude this batch and return an empty dict.\n        # We could add padding if the model supported it instead of this drop, you can customize this part to your needs.\n        total_length = (total_length // block_size) * block_size\n        # Split by chunks of max_len.\n        result = {\n            k: [t[i : i + block_size] for i in range(0, total_length, block_size)]\n            for k, t in concatenated_examples.items()\n        }\n        result[\"labels\"] = result[\"input_ids\"].copy()\n        return result\n\n    # Get the FIM-specific token ids\n    prefix_tok_id = tokenizer.convert_tokens_to_ids(data_args.fim_prefix_token)\n    middle_tok_id = tokenizer.convert_tokens_to_ids(data_args.fim_middle_token)\n    suffix_tok_id = tokenizer.convert_tokens_to_ids(data_args.fim_suffix_token)\n    pad_tok_id = None\n\n    # If truncate_or_pad is on, also get pad token id\n    if data_args.truncate_or_pad:\n        pad_tok_id = tokenizer.convert_tokens_to_ids(data_args.pad_token)\n\n    # The two functions below perform the FIM transformation on the data (either PSM or SPM or PSM+SPM)\n    # Don't call fim_transform directly in .map()\n    # Adapted from https://github.com/loubnabnl/santacoder-finetuning/blob/main/fim.py#L22C13-L83\n    def fim_transform(example):\n        \"\"\"\n        This function performs FIM transformation on a single example (list of tokens)\n        \"\"\"\n        if np_rng.binomial(1, data_args.fim_rate):\n            boundaries = sorted(np_rng.randint(low=0, high=len(example) + 1, size=2))\n\n            prefix = example[: boundaries[0]]\n            middle = example[boundaries[0] : boundaries[1]]\n            suffix = example[boundaries[1] :]\n\n            if data_args.truncate_or_pad:\n                total_length = len(prefix) + len(middle) + len(suffix) + 3\n                diff = total_length - len(example)\n                if diff > 0:\n                    suffix = suffix[: max(0, len(suffix) - diff)]\n                elif diff < 0:\n                    suffix.extend([pad_tok_id] * (-diff))\n\n            if np_rng.binomial(1, data_args.fim_spm_rate):\n                # Apply Suffix-Prefix-Middle (SPM) transformation\n                transformed_example = [prefix_tok_id, suffix_tok_id] + suffix + [middle_tok_id] + prefix + middle\n            else:\n                # Apply Prefix-Suffix-Middle (PSM) transformation\n                transformed_example = [prefix_tok_id] + prefix + [suffix_tok_id] + suffix + [middle_tok_id] + middle\n        else:\n            transformed_example = example\n\n        return transformed_example\n\n    # Below function is the one you are supposed to call in the .map() function\n    def apply_fim(examples):\n        \"\"\"\n        Apply FIM transformation to a batch of examples\n        \"\"\"\n        fim_transform_ids = [fim_transform(ids) for ids in examples[\"input_ids\"]]\n        examples[\"input_ids\"] = fim_transform_ids\n        examples[\"labels\"] = fim_transform_ids\n        # If your application requires custom attention mask, please adjust this function's below line.\n        # Since FIM transformation increases the number of tokens in input_ids and labels\n        # but leaves the number of tokens unchanged in attention_masks which would cause problems\n        examples[\"attention_mask\"] = [[1] * len(mask) for mask in examples[\"input_ids\"]]\n        return examples\n\n    # Note that with `batched=True`, this map processes 1,000 texts together, so group_texts throws away a remainder\n    # for each of those groups of 1,000 texts. You can adjust that batch_size here but a higher value might be slower\n    # to preprocess.\n    #\n    # To speed up this part, we use multiprocessing. See the documentation of the map method for more information:\n    # https://huggingface.co/docs/datasets/process#map\n\n    # FIM transformations are only supposed to be applied before group_texts processing otherwise some sentences will\n    # have 3-4 more tokens than others due to probabilistic addition of FIM-specific tokens which will raise errors\n    with training_args.main_process_first(desc=\"processing texts together\"):\n        if not data_args.streaming:\n            fim_datasets = tokenized_datasets.map(\n                apply_fim,\n                batched=True,\n                num_proc=data_args.preprocessing_num_workers,\n                load_from_cache_file=not data_args.overwrite_cache,\n                desc=\"Performing FIM transformation\",\n            )\n            lm_datasets = fim_datasets.map(\n                group_texts,\n                batched=True,\n                num_proc=data_args.preprocessing_num_workers,\n                load_from_cache_file=not data_args.overwrite_cache,\n                desc=f\"Grouping texts in chunks of {block_size}\",\n            )\n        else:\n            fim_datasets = tokenized_datasets.map(\n                apply_fim,\n                batched=True,\n            )\n            lm_datasets = fim_datasets.map(\n                group_texts,\n                batched=True,\n            )\n\n    if training_args.do_train:\n        if \"train\" not in tokenized_datasets:\n            raise ValueError(\"--do_train requires a train dataset\")\n        train_dataset = lm_datasets[\"train\"]\n        if data_args.max_train_samples is not None:\n            max_train_samples = min(len(train_dataset), data_args.max_train_samples)\n            train_dataset = train_dataset.select(range(max_train_samples))\n\n    if training_args.do_eval:\n        if \"validation\" not in tokenized_datasets:\n            raise ValueError(\"--do_eval requires a validation dataset\")\n        eval_dataset = lm_datasets[\"validation\"]\n        if data_args.max_eval_samples is not None:\n            max_eval_samples = min(len(eval_dataset), data_args.max_eval_samples)\n            eval_dataset = eval_dataset.select(range(max_eval_samples))\n\n        def preprocess_logits_for_metrics(logits, labels):\n            if isinstance(logits, tuple):\n                # Depending on the model and config, logits may contain extra tensors,\n                # like past_key_values, but logits always come first\n                logits = logits[0]\n            return logits.argmax(dim=-1)\n\n        metric = evaluate.load(\"accuracy\")\n\n        def compute_metrics(eval_preds):\n            preds, labels = eval_preds\n            # preds have the same shape as the labels, after the argmax(-1) has been calculated\n            # by preprocess_logits_for_metrics but we need to shift the labels\n            labels = labels[:, 1:].reshape(-1)\n            preds = preds[:, :-1].reshape(-1)\n            return metric.compute(predictions=preds, references=labels)\n\n    # Initialize our Trainer\n    trainer = Trainer(\n        model=model,\n        args=training_args,\n        train_dataset=train_dataset if training_args.do_train else None,\n        eval_dataset=eval_dataset if training_args.do_eval else None,\n        processing_class=tokenizer,\n        # Data collator will default to DataCollatorWithPadding, so we change it.\n        data_collator=default_data_collator,\n        compute_metrics=compute_metrics if training_args.do_eval and not is_torch_tpu_available() else None,\n        preprocess_logits_for_metrics=(\n            preprocess_logits_for_metrics if training_args.do_eval and not is_torch_tpu_available() else None\n        ),\n    )\n\n    # Training\n    if training_args.do_train:\n        checkpoint = None\n        if training_args.resume_from_checkpoint is not None:\n            checkpoint = training_args.resume_from_checkpoint\n        elif last_checkpoint is not None:\n            checkpoint = last_checkpoint\n        train_result = trainer.train(resume_from_checkpoint=checkpoint)\n        trainer.save_model()  # Saves the tokenizer too for easy upload\n\n        metrics = train_result.metrics\n\n        max_train_samples = (\n            data_args.max_train_samples if data_args.max_train_samples is not None else len(train_dataset)\n        )\n        metrics[\"train_samples\"] = min(max_train_samples, len(train_dataset))\n\n        trainer.log_metrics(\"train\", metrics)\n        trainer.save_metrics(\"train\", metrics)\n        trainer.save_state()\n\n    # Evaluation\n    if training_args.do_eval:\n        logger.info(\"*** Evaluate ***\")\n\n        metrics = trainer.evaluate()\n\n        max_eval_samples = data_args.max_eval_samples if data_args.max_eval_samples is not None else len(eval_dataset)\n        metrics[\"eval_samples\"] = min(max_eval_samples, len(eval_dataset))\n        try:\n            perplexity = math.exp(metrics[\"eval_loss\"])\n        except OverflowError:\n            perplexity = float(\"inf\")\n        metrics[\"perplexity\"] = perplexity\n\n        trainer.log_metrics(\"eval\", metrics)\n        trainer.save_metrics(\"eval\", metrics)\n\n    kwargs = {\"finetuned_from\": model_args.model_name_or_path, \"tasks\": \"text-generation\"}\n    if data_args.dataset_name is not None:\n        kwargs[\"dataset_tags\"] = data_args.dataset_name\n        if data_args.dataset_config_name is not None:\n            kwargs[\"dataset_args\"] = data_args.dataset_config_name\n            kwargs[\"dataset\"] = f\"{data_args.dataset_name} {data_args.dataset_config_name}\"\n        else:\n            kwargs[\"dataset\"] = data_args.dataset_name\n\n    if training_args.push_to_hub:\n        trainer.push_to_hub(**kwargs)\n    else:\n\n\nSUMMARY:\nThis is the code for a Hugging Face Transformers training script. It's used to train and evaluate a text generation model on a dataset of your choice.\n\n**Key Features:**\n\n*   Supports both fine-tuning and training from scratch.\n*   Handles streaming datasets with a progress bar and metric logging.\n*   Trains models on different devices (GPU/TPU) if available and optimizes for performance, memory, or mixed-mode training.\n*   Automatically saves checkpoints and logs metrics to help track the training progress.\n\n**Preprocessing Steps:**\n\n1.  **Load Data**: The script loads a dataset using `tokenized_datasets.load()`.\n2.  **Perform Fine-tuning or Training from Scratch**: Before starting the actual training process, it fine-tunes or trains your model using `Trainer.train()` if you already have data. Otherwise, it starts with empty datasets and begins preprocessing your data.\n3.  **Fine-tune or Train Model**: Depending on whether you want to continue or start anew, it processes your dataset before calling `Trainer.train()`.\n4.  **Logging of Metrics**: As part of the training process, metrics like accuracy, perplexity, train loss, and validation metrics are logged.\n\n**Customization Options:**\n\nYou can customize several aspects of your experiment, including:\n\n*   The task type (e.g., \"text-generation\")\n*   Model configuration\n*   Data collator (by specifying `default_data_collator` as you can if you desire)\n*   Training settings (`do_train`, `do_eval`, `max Train samples`, `max eval samples`)\n*   Model configuration\n\n**Example Code:**\n\n```python\n# Load pre-trained model and tokenizer\nfrom transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n\nmodel = AutoModelForSeq2SeqLM.from_pretrained('t5-small')\ntokenizer = AutoTokenizer.from_pretrained('t5-small')\n\n# Create a custom dataset class for your specific task\nclass MyDataset(torch.utils.data.Dataset):\n    def __init__(self, inputs, labels):\n        self.inputs = inputs\n        self.labels = labels\n\n    def __getitem__(self, idx):\n        item = {key: torch.tensor(val[idx]) for key, val in enumerate(reversed(self.labels)) if key not in ['labels']}\n        item['labels'] = torch.tensor(self.labels[idx])\n        \n        input_seq = tokenizer.encode(['generate'] + self.inputs[idx], return_tensors='pt').squeeze(0)\n        labels = torch.tensor(self.labels[idx])  \n        \n        item.update({'input_ids': input_seq, 'attention_mask': (input_seq != 0).unsqueeze(-1)})\n        return item\n\n    def __len__(self):\n        return len(self.labels)\n\n# Perform preprocessing and setup for training\n\ndef create_dataset(model_name_or_path):\n    # Load pre-trained model and tokenizer\n    from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, AutoTrainingArguments\n    \n    global dataset_args\n    dataset_args = TrainingArguments(\n         # Input args for the dataset\n        output_dir='training outputs',\n        seed=42,\n        num_train_epochs=3.5,\n        per_device_train_batch_size=16 ,\n        per_device_eval_batch_size=64,\n        \n        # Training arguments\n        per_device_eval_iter_size=32, warmup_steps=500 , warmup_ratio=0.1\n    )\n    \n    args = AutoTrainingArguments(model_name_or_path=model_name_or_path, \n                                output_dir=output_dir, \n                                seed=42, \n                                num_train_epochs=num_train_epochs, \n                                per_device_train_batch_size=per_device_train_batch_size,\n                                per_device_eval_batch_size=per_device_eval_batch_size,\n                                warmup_steps=warmup_steps, warmup_ratio=warmup_ratio\n                            )\n    \n    training_args = args\n    \n    tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)\n    dataset = load_dataset('your_dataset')\n    if dataset is None:\n        return\n    dataset = dataset.map MyDataset(tokenizer).\n         batched(train=True)\n     # Evaluation set (if applicable):\n    eval_dataset  = dataset['validation'].\n            batched(train=False)\n\n    return training_args\n\ntraining_args = create_dataset(\"path_to_your_pretrained_model\")\n# Training \ndef main():\n   trainer = Trainer(\n                        model=model,\n                        args=training_args,  \n                        train_dataset=dataset,    \n                     )\n    trainer.train()\n    # Save the results\n    trainer.save_results()\nif __name__ == \"__main__\":\n    main()\n\n```\n\nThis script provides an example for performing the training with a basic Hugging Face text generation model."}


created_at: 2025-02-05 09:13:40
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/language-modeling/run_fim.py:857-859
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 859, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/language-modeling/run_fim.py", "name": "_mp_fn", "start_line": 857, "type": "function", "chroma:document": "CODE:\n\ndef _mp_fn(index):\n    # For xla_spawn (TPUs)\n\n\nSUMMARY:\n**Summary**\n\n`_mp_fn(index)` is a function that initializes and executes the multi-threading process. It calls the `main()` function, which appears to be the primary entry point of the program.\n\n**Key Details**\n\n* Purpose: Initialize multi-threading process\n* Implementation: Calls `main()`\n* Parameters: `index` (not used)\n\nNote: The exact purpose and implementation of the `main()` function are not specified in this code chunk."}


created_at: 2025-02-05 09:13:40
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/language-modeling/run_clm_no_trainer.py:69-258

metadata:
{"docstring": "", "end_line": 258, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/language-modeling/run_clm_no_trainer.py", "name": "parse_args", "start_line": 69, "type": "function", "chroma:document": "CODE:\n\ndef parse_args():\n    parser = argparse.ArgumentParser(description=\"Finetune a transformers model on a causal language modeling task\")\n    parser.add_argument(\n        \"--dataset_name\",\n        type=str,\n        default=None,\n        help=\"The name of the dataset to use (via the datasets library).\",\n    )\n    parser.add_argument(\n        \"--dataset_config_name\",\n        type=str,\n        default=None,\n        help=\"The configuration name of the dataset to use (via the datasets library).\",\n    )\n    parser.add_argument(\n        \"--train_file\", type=str, default=None, help=\"A csv, txt or a json file containing the training data.\"\n    )\n    parser.add_argument(\n        \"--validation_file\", type=str, default=None, help=\"A csv, txt or a json file containing the validation data.\"\n    )\n    parser.add_argument(\n        \"--validation_split_percentage\",\n        default=5,\n        help=\"The percentage of the train set used as validation set in case there's no validation split\",\n    )\n    parser.add_argument(\n        \"--model_name_or_path\",\n        type=str,\n        help=\"Path to pretrained model or model identifier from huggingface.co/models.\",\n        required=False,\n    )\n    parser.add_argument(\n        \"--config_name\",\n        type=str,\n        default=None,\n        help=\"Pretrained config name or path if not the same as model_name\",\n    )\n    parser.add_argument(\n        \"--tokenizer_name\",\n        type=str,\n        default=None,\n        help=\"Pretrained tokenizer name or path if not the same as model_name\",\n    )\n    parser.add_argument(\n        \"--use_slow_tokenizer\",\n        action=\"store_true\",\n        help=\"If passed, will use a slow tokenizer (not backed by the \ud83e\udd17 Tokenizers library).\",\n    )\n    parser.add_argument(\n        \"--per_device_train_batch_size\",\n        type=int,\n        default=8,\n        help=\"Batch size (per device) for the training dataloader.\",\n    )\n    parser.add_argument(\n        \"--per_device_eval_batch_size\",\n        type=int,\n        default=8,\n        help=\"Batch size (per device) for the evaluation dataloader.\",\n    )\n    parser.add_argument(\n        \"--learning_rate\",\n        type=float,\n        default=5e-5,\n        help=\"Initial learning rate (after the potential warmup period) to use.\",\n    )\n    parser.add_argument(\"--weight_decay\", type=float, default=0.0, help=\"Weight decay to use.\")\n    parser.add_argument(\"--num_train_epochs\", type=int, default=3, help=\"Total number of training epochs to perform.\")\n    parser.add_argument(\n        \"--max_train_steps\",\n        type=int,\n        default=None,\n        help=\"Total number of training steps to perform. If provided, overrides num_train_epochs.\",\n    )\n    parser.add_argument(\n        \"--gradient_accumulation_steps\",\n        type=int,\n        default=1,\n        help=\"Number of updates steps to accumulate before performing a backward/update pass.\",\n    )\n    parser.add_argument(\n        \"--lr_scheduler_type\",\n        type=SchedulerType,\n        default=\"linear\",\n        help=\"The scheduler type to use.\",\n        choices=[\"linear\", \"cosine\", \"cosine_with_restarts\", \"polynomial\", \"constant\", \"constant_with_warmup\"],\n    )\n    parser.add_argument(\n        \"--num_warmup_steps\", type=int, default=0, help=\"Number of steps for the warmup in the lr scheduler.\"\n    )\n    parser.add_argument(\"--output_dir\", type=str, default=None, help=\"Where to store the final model.\")\n    parser.add_argument(\"--seed\", type=int, default=None, help=\"A seed for reproducible training.\")\n    parser.add_argument(\n        \"--model_type\",\n        type=str,\n        default=None,\n        help=\"Model type to use if training from scratch.\",\n        choices=MODEL_TYPES,\n    )\n    parser.add_argument(\n        \"--block_size\",\n        type=int,\n        default=None,\n        help=(\n            \"Optional input sequence length after tokenization. The training dataset will be truncated in block of\"\n            \" this size for training. Default to the model max input length for single sentence inputs (take into\"\n            \" account special tokens).\"\n        ),\n    )\n    parser.add_argument(\n        \"--preprocessing_num_workers\",\n        type=int,\n        default=None,\n        help=\"The number of processes to use for the preprocessing.\",\n    )\n    parser.add_argument(\n        \"--overwrite_cache\", action=\"store_true\", help=\"Overwrite the cached training and evaluation sets\"\n    )\n    parser.add_argument(\n        \"--no_keep_linebreaks\", action=\"store_true\", help=\"Do not keep line breaks when using TXT files.\"\n    )\n    parser.add_argument(\"--push_to_hub\", action=\"store_true\", help=\"Whether or not to push the model to the Hub.\")\n    parser.add_argument(\n        \"--hub_model_id\", type=str, help=\"The name of the repository to keep in sync with the local `output_dir`.\"\n    )\n    parser.add_argument(\"--hub_token\", type=str, help=\"The token to use to push to the Model Hub.\")\n    parser.add_argument(\n        \"--trust_remote_code\",\n        action=\"store_true\",\n        help=(\n            \"Whether to trust the execution of code from datasets/models defined on the Hub.\"\n            \" This option should only be set to `True` for repositories you trust and in which you have read the\"\n            \" code, as it will execute code present on the Hub on your local machine.\"\n        ),\n    )\n    parser.add_argument(\n        \"--checkpointing_steps\",\n        type=str,\n        default=None,\n        help=\"Whether the various states should be saved at the end of every n steps, or 'epoch' for each epoch.\",\n    )\n    parser.add_argument(\n        \"--resume_from_checkpoint\",\n        type=str,\n        default=None,\n        help=\"If the training should continue from a checkpoint folder.\",\n    )\n    parser.add_argument(\n        \"--with_tracking\",\n        action=\"store_true\",\n        help=\"Whether to enable experiment trackers for logging.\",\n    )\n    parser.add_argument(\n        \"--report_to\",\n        type=str,\n        default=\"all\",\n        help=(\n            'The integration to report the results and logs to. Supported platforms are `\"tensorboard\"`,'\n            ' `\"wandb\"`, `\"comet_ml\"` and `\"clearml\"`. Use `\"all\"` (default) to report to all integrations. '\n            \"Only applicable when `--with_tracking` is passed.\"\n        ),\n    )\n    parser.add_argument(\n        \"--low_cpu_mem_usage\",\n        action=\"store_true\",\n        help=(\n            \"It is an option to create the model as an empty shell, then only materialize its parameters when the pretrained weights are loaded. \"\n            \"If passed, LLM loading time and RAM consumption will be benefited.\"\n        ),\n    )\n    args = parser.parse_args()\n\n    # Sanity checks\n    if args.dataset_name is None and args.train_file is None and args.validation_file is None:\n        raise ValueError(\"Need either a dataset name or a training/validation file.\")\n    else:\n        if args.train_file is not None:\n            extension = args.train_file.split(\".\")[-1]\n            if extension not in [\"csv\", \"json\", \"txt\"]:\n                raise ValueError(\"`train_file` should be a csv, json or txt file.\")\n        if args.validation_file is not None:\n            extension = args.validation_file.split(\".\")[-1]\n            if extension not in [\"csv\", \"json\", \"txt\"]:\n                raise ValueError(\"`validation_file` should be a csv, json or txt file.\")\n\n    if args.push_to_hub:\n        if args.output_dir is None:\n            raise ValueError(\"Need an `output_dir` to create a repo when `--push_to_hub` is passed.\")\n\n\n\nSUMMARY:\n**Summary**\n\nThis function uses the `argparse` library to define a set of command-line arguments for fine-tuning a transformer model on a causal language modeling task.\n\n### Main Purpose and Functionality\n\nThe function `parse_args()` parses the command-line arguments, allowing users to configure various hyperparameters, such as:\n\n* Model architectures (e.g., `model_name_or_path`)\n* Training parameters (e.g., `num_train_epochs`, `batch_size`, `learning_rate`)\n* Data loading options (e.g., `train_file`, `validation_file`)\n* Output directory and push-to-Hub configuration\n\n### Key Implementation Details\n\nThe function uses the following key algorithms, patterns, or design choices:\n\n* Args parsing using `argparse`\n* Sanity checks to ensure valid input values for user-provided files\n* Hyperparameter validation for training parameters (e.g., batch sizes, learning rates)\n\n### Notable Dependencies and Requirements\n\nThis code relies on the following external dependencies and interfaces:\n\n* `datasets` library for loading datasets\n* `huggingface.co/models` for accessing pre-trained models\n* `argparse` library for defining command-line arguments"}


created_at: 2025-02-05 09:13:40
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/language-modeling/run_clm_no_trainer.py:261-720

metadata:
{"docstring": "", "end_line": 720, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/language-modeling/run_clm_no_trainer.py", "name": "main", "start_line": 261, "type": "function", "chroma:document": "CODE:\n\ndef main():\n    args = parse_args()\n\n    # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The\n    # information sent is the one passed as arguments along with your Python/PyTorch versions.\n    send_example_telemetry(\"run_clm_no_trainer\", args)\n\n    # Initialize the accelerator. We will let the accelerator handle device placement for us in this example.\n    # If we're using tracking, we also need to initialize it here and it will by default pick up all supported trackers\n    # in the environment\n    accelerator_log_kwargs = {}\n\n    if args.with_tracking:\n        accelerator_log_kwargs[\"log_with\"] = args.report_to\n        accelerator_log_kwargs[\"project_dir\"] = args.output_dir\n\n    accelerator = Accelerator(gradient_accumulation_steps=args.gradient_accumulation_steps, **accelerator_log_kwargs)\n\n    # Make one log on every process with the configuration for debugging.\n    logging.basicConfig(\n        format=\"%(asctime)s - %(levelname)s - %(name)s - %(message)s\",\n        datefmt=\"%m/%d/%Y %H:%M:%S\",\n        level=logging.INFO,\n    )\n    logger.info(accelerator.state, main_process_only=False)\n    if accelerator.is_local_main_process:\n        datasets.utils.logging.set_verbosity_warning()\n        transformers.utils.logging.set_verbosity_info()\n    else:\n        datasets.utils.logging.set_verbosity_error()\n        transformers.utils.logging.set_verbosity_error()\n\n    # If passed along, set the training seed now.\n    if args.seed is not None:\n        set_seed(args.seed)\n\n    # Handle the repository creation\n    if accelerator.is_main_process:\n        if args.push_to_hub:\n            # Retrieve of infer repo_name\n            repo_name = args.hub_model_id\n            if repo_name is None:\n                repo_name = Path(args.output_dir).absolute().name\n            # Create repo and retrieve repo_id\n            api = HfApi()\n            repo_id = api.create_repo(repo_name, exist_ok=True, token=args.hub_token).repo_id\n\n            with open(os.path.join(args.output_dir, \".gitignore\"), \"w+\") as gitignore:\n                if \"step_*\" not in gitignore:\n                    gitignore.write(\"step_*\\n\")\n                if \"epoch_*\" not in gitignore:\n                    gitignore.write(\"epoch_*\\n\")\n        elif args.output_dir is not None:\n            os.makedirs(args.output_dir, exist_ok=True)\n    accelerator.wait_for_everyone()\n\n    # Get the datasets: you can either provide your own CSV/JSON/TXT training and evaluation files (see below)\n    # or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/\n    # (the dataset will be downloaded automatically from the datasets Hub).\n    #\n    # For CSV/JSON files, this script will use the column called 'text' or the first column if no column called\n    # 'text' is found. You can easily tweak this behavior (see below).\n    #\n    # In distributed training, the load_dataset function guarantee that only one local process can concurrently\n    # download the dataset.\n    if args.dataset_name is not None:\n        # Downloading and loading a dataset from the hub.\n        raw_datasets = load_dataset(\n            args.dataset_name, args.dataset_config_name, trust_remote_code=args.trust_remote_code\n        )\n        if \"validation\" not in raw_datasets.keys():\n            raw_datasets[\"validation\"] = load_dataset(\n                args.dataset_name,\n                args.dataset_config_name,\n                split=f\"train[:{args.validation_split_percentage}%]\",\n                trust_remote_code=args.trust_remote_code,\n            )\n            raw_datasets[\"train\"] = load_dataset(\n                args.dataset_name,\n                args.dataset_config_name,\n                split=f\"train[{args.validation_split_percentage}%:]\",\n                trust_remote_code=args.trust_remote_code,\n            )\n    else:\n        data_files = {}\n        dataset_args = {}\n        if args.train_file is not None:\n            data_files[\"train\"] = args.train_file\n            extension = args.train_file.split(\".\")[-1]\n        if args.validation_file is not None:\n            data_files[\"validation\"] = args.validation_file\n            extension = args.validation_file.split(\".\")[-1]\n        if extension == \"txt\":\n            extension = \"text\"\n            dataset_args[\"keep_linebreaks\"] = not args.no_keep_linebreaks\n        raw_datasets = load_dataset(extension, data_files=data_files, **dataset_args)\n        # If no validation data is there, validation_split_percentage will be used to divide the dataset.\n        if \"validation\" not in raw_datasets.keys():\n            raw_datasets[\"validation\"] = load_dataset(\n                extension,\n                data_files=data_files,\n                split=f\"train[:{args.validation_split_percentage}%]\",\n                **dataset_args,\n            )\n            raw_datasets[\"train\"] = load_dataset(\n                extension,\n                data_files=data_files,\n                split=f\"train[{args.validation_split_percentage}%:]\",\n                **dataset_args,\n            )\n\n    # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at\n    # https://huggingface.co/docs/datasets/loading_datasets.\n\n    # Load pretrained model and tokenizer\n    #\n    # In distributed training, the .from_pretrained methods guarantee that only one local process can concurrently\n    # download model & vocab.\n    if args.config_name:\n        config = AutoConfig.from_pretrained(\n            args.config_name,\n            trust_remote_code=args.trust_remote_code,\n        )\n    elif args.model_name_or_path:\n        config = AutoConfig.from_pretrained(\n            args.model_name_or_path,\n            trust_remote_code=args.trust_remote_code,\n        )\n    else:\n        config = CONFIG_MAPPING[args.model_type]()\n        logger.warning(\"You are instantiating a new config instance from scratch.\")\n\n    if args.tokenizer_name:\n        tokenizer = AutoTokenizer.from_pretrained(\n            args.tokenizer_name, use_fast=not args.use_slow_tokenizer, trust_remote_code=args.trust_remote_code\n        )\n    elif args.model_name_or_path:\n        tokenizer = AutoTokenizer.from_pretrained(\n            args.model_name_or_path, use_fast=not args.use_slow_tokenizer, trust_remote_code=args.trust_remote_code\n        )\n    else:\n        raise ValueError(\n            \"You are instantiating a new tokenizer from scratch. This is not supported by this script. \"\n            \"You can do it from another script, save it, and load it from here, using --tokenizer_name.\"\n        )\n\n    if args.model_name_or_path:\n        model = AutoModelForCausalLM.from_pretrained(\n            args.model_name_or_path,\n            from_tf=bool(\".ckpt\" in args.model_name_or_path),\n            config=config,\n            low_cpu_mem_usage=args.low_cpu_mem_usage,\n            trust_remote_code=args.trust_remote_code,\n        )\n    else:\n        logger.info(\"Training new model from scratch\")\n        model = AutoModelForCausalLM.from_config(config, trust_remote_code=args.trust_remote_code)\n\n    # We resize the embeddings only when necessary to avoid index errors. If you are creating a model from scratch\n    # on a small vocab and want a smaller embedding size, remove this test.\n    embedding_size = model.get_input_embeddings().weight.shape[0]\n    if len(tokenizer) > embedding_size:\n        model.resize_token_embeddings(len(tokenizer))\n\n    # Preprocessing the datasets.\n    # First we tokenize all the texts.\n    column_names = raw_datasets[\"train\"].column_names\n    text_column_name = \"text\" if \"text\" in column_names else column_names[0]\n\n    def tokenize_function(examples):\n        return tokenizer(examples[text_column_name])\n\n    with accelerator.main_process_first():\n        tokenized_datasets = raw_datasets.map(\n            tokenize_function,\n            batched=True,\n            num_proc=args.preprocessing_num_workers,\n            remove_columns=column_names,\n            load_from_cache_file=not args.overwrite_cache,\n            desc=\"Running tokenizer on dataset\",\n        )\n\n    if args.block_size is None:\n        block_size = tokenizer.model_max_length\n        if block_size > config.max_position_embeddings:\n            logger.warning(\n                f\"The tokenizer picked seems to have a very large `model_max_length` ({tokenizer.model_max_length}). \"\n                f\"Using block_size={min(1024, config.max_position_embeddings)} instead. You can change that default value by passing --block_size xxx.\"\n            )\n            block_size = min(1024, config.max_position_embeddings)\n    else:\n        if args.block_size > tokenizer.model_max_length:\n            logger.warning(\n                f\"The block_size passed ({args.block_size}) is larger than the maximum length for the model \"\n                f\"({tokenizer.model_max_length}). Using block_size={tokenizer.model_max_length}.\"\n            )\n        block_size = min(args.block_size, tokenizer.model_max_length)\n\n    # Main data processing function that will concatenate all texts from our dataset and generate chunks of block_size.\n    def group_texts(examples):\n        # Concatenate all texts.\n        concatenated_examples = {k: list(chain(*examples[k])) for k in examples.keys()}\n        total_length = len(concatenated_examples[list(examples.keys())[0]])\n        # We drop the small remainder, and if the total_length < block_size  we exclude this batch and return an empty dict.\n        # We could add padding if the model supported it instead of this drop, you can customize this part to your needs.\n        total_length = (total_length // block_size) * block_size\n        # Split by chunks of max_len.\n        result = {\n            k: [t[i : i + block_size] for i in range(0, total_length, block_size)]\n            for k, t in concatenated_examples.items()\n        }\n        result[\"labels\"] = result[\"input_ids\"].copy()\n        return result\n\n    # Note that with `batched=True`, this map processes 1,000 texts together, so group_texts throws away a remainder\n    # for each of those groups of 1,000 texts. You can adjust that batch_size here but a higher value might be slower\n    # to preprocess.\n    #\n    # To speed up this part, we use multiprocessing. See the documentation of the map method for more information:\n    # https://huggingface.co/docs/datasets/process#map\n\n    with accelerator.main_process_first():\n        lm_datasets = tokenized_datasets.map(\n            group_texts,\n            batched=True,\n            num_proc=args.preprocessing_num_workers,\n            load_from_cache_file=not args.overwrite_cache,\n            desc=f\"Grouping texts in chunks of {block_size}\",\n        )\n\n    train_dataset = lm_datasets[\"train\"]\n    eval_dataset = lm_datasets[\"validation\"]\n\n    # Log a few random samples from the training set:\n    for index in random.sample(range(len(train_dataset)), 3):\n        logger.info(f\"Sample {index} of the training set: {train_dataset[index]}.\")\n\n    # DataLoaders creation:\n    train_dataloader = DataLoader(\n        train_dataset, shuffle=True, collate_fn=default_data_collator, batch_size=args.per_device_train_batch_size\n    )\n    eval_dataloader = DataLoader(\n        eval_dataset, collate_fn=default_data_collator, batch_size=args.per_device_eval_batch_size\n    )\n\n    # Optimizer\n    # Split weights in two groups, one with weight decay and the other not.\n    no_decay = [\"bias\", \"layer_norm.weight\"]\n    optimizer_grouped_parameters = [\n        {\n            \"params\": [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)],\n            \"weight_decay\": args.weight_decay,\n        },\n        {\n            \"params\": [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)],\n            \"weight_decay\": 0.0,\n        },\n    ]\n    optimizer = torch.optim.AdamW(optimizer_grouped_parameters, lr=args.learning_rate)\n\n    # Scheduler and math around the number of training steps.\n    overrode_max_train_steps = False\n    num_update_steps_per_epoch = math.ceil(len(train_dataloader) / args.gradient_accumulation_steps)\n    if args.max_train_steps is None:\n        args.max_train_steps = args.num_train_epochs * num_update_steps_per_epoch\n        overrode_max_train_steps = True\n\n    lr_scheduler = get_scheduler(\n        name=args.lr_scheduler_type,\n        optimizer=optimizer,\n        num_warmup_steps=args.num_warmup_steps * accelerator.num_processes,\n        num_training_steps=args.max_train_steps\n        if overrode_max_train_steps\n        else args.max_train_steps * accelerator.num_processes,\n    )\n\n    # Prepare everything with our `accelerator`.\n    model, optimizer, train_dataloader, eval_dataloader, lr_scheduler = accelerator.prepare(\n        model, optimizer, train_dataloader, eval_dataloader, lr_scheduler\n    )\n\n    # On TPU, the tie weights in our model have been disconnected, so we need to restore the ties.\n    if accelerator.distributed_type == DistributedType.TPU:\n        model.tie_weights()\n\n    # We need to recalculate our total training steps as the size of the training dataloader may have changed.\n    num_update_steps_per_epoch = math.ceil(len(train_dataloader) / args.gradient_accumulation_steps)\n    if overrode_max_train_steps:\n        args.max_train_steps = args.num_train_epochs * num_update_steps_per_epoch\n    # Afterwards we recalculate our number of training epochs\n    args.num_train_epochs = math.ceil(args.max_train_steps / num_update_steps_per_epoch)\n\n    # Figure out how many steps we should save the Accelerator states\n    checkpointing_steps = args.checkpointing_steps\n    if checkpointing_steps is not None and checkpointing_steps.isdigit():\n        checkpointing_steps = int(checkpointing_steps)\n\n    # We need to initialize the trackers we use, and also store our configuration.\n    # The trackers initializes automatically on the main process.\n    if args.with_tracking:\n        experiment_config = vars(args)\n        # TensorBoard cannot log Enums, need the raw value\n        experiment_config[\"lr_scheduler_type\"] = experiment_config[\"lr_scheduler_type\"].value\n        accelerator.init_trackers(\"clm_no_trainer\", experiment_config)\n\n    # Train!\n    total_batch_size = args.per_device_train_batch_size * accelerator.num_processes * args.gradient_accumulation_steps\n\n    logger.info(\"***** Running training *****\")\n    logger.info(f\"  Num examples = {len(train_dataset)}\")\n    logger.info(f\"  Num Epochs = {args.num_train_epochs}\")\n    logger.info(f\"  Instantaneous batch size per device = {args.per_device_train_batch_size}\")\n    logger.info(f\"  Total train batch size (w. parallel, distributed & accumulation) = {total_batch_size}\")\n    logger.info(f\"  Gradient Accumulation steps = {args.gradient_accumulation_steps}\")\n    logger.info(f\"  Total optimization steps = {args.max_train_steps}\")\n    # Only show the progress bar once on each machine.\n    progress_bar = tqdm(range(args.max_train_steps), disable=not accelerator.is_local_main_process)\n    completed_steps = 0\n    starting_epoch = 0\n\n    # Potentially load in the weights and states from a previous save\n    if args.resume_from_checkpoint:\n        if args.resume_from_checkpoint is not None or args.resume_from_checkpoint != \"\":\n            checkpoint_path = args.resume_from_checkpoint\n            path = os.path.basename(args.resume_from_checkpoint)\n        else:\n            # Get the most recent checkpoint\n            dirs = [f.name for f in os.scandir(os.getcwd()) if f.is_dir()]\n            dirs.sort(key=os.path.getctime)\n            path = dirs[-1]  # Sorts folders by date modified, most recent checkpoint is the last\n            checkpoint_path = path\n            path = os.path.basename(checkpoint_path)\n\n        accelerator.print(f\"Resumed from checkpoint: {checkpoint_path}\")\n        accelerator.load_state(checkpoint_path)\n        # Extract `epoch_{i}` or `step_{i}`\n        training_difference = os.path.splitext(path)[0]\n\n        if \"epoch\" in training_difference:\n            starting_epoch = int(training_difference.replace(\"epoch_\", \"\")) + 1\n            resume_step = None\n            completed_steps = starting_epoch * num_update_steps_per_epoch\n        else:\n            # need to multiply `gradient_accumulation_steps` to reflect real steps\n            resume_step = int(training_difference.replace(\"step_\", \"\")) * args.gradient_accumulation_steps\n            starting_epoch = resume_step // len(train_dataloader)\n            completed_steps = resume_step // args.gradient_accumulation_steps\n            resume_step -= starting_epoch * len(train_dataloader)\n\n    # update the progress_bar if load from checkpoint\n    progress_bar.update(completed_steps)\n\n    for epoch in range(starting_epoch, args.num_train_epochs):\n        model.train()\n        if args.with_tracking:\n            total_loss = 0\n        if args.resume_from_checkpoint and epoch == starting_epoch and resume_step is not None:\n            # We skip the first `n` batches in the dataloader when resuming from a checkpoint\n            active_dataloader = accelerator.skip_first_batches(train_dataloader, resume_step)\n        else:\n            active_dataloader = train_dataloader\n        for step, batch in enumerate(active_dataloader):\n            with accelerator.accumulate(model):\n                outputs = model(**batch)\n                loss = outputs.loss\n                # We keep track of the loss at each epoch\n                if args.with_tracking:\n                    total_loss += loss.detach().float()\n                accelerator.backward(loss)\n                optimizer.step()\n                lr_scheduler.step()\n                optimizer.zero_grad()\n\n            # Checks if the accelerator has performed an optimization step behind the scenes\n            if accelerator.sync_gradients:\n                progress_bar.update(1)\n                completed_steps += 1\n\n            if isinstance(checkpointing_steps, int):\n                if completed_steps % checkpointing_steps == 0 and accelerator.sync_gradients:\n                    output_dir = f\"step_{completed_steps}\"\n                    if args.output_dir is not None:\n                        output_dir = os.path.join(args.output_dir, output_dir)\n                    accelerator.save_state(output_dir)\n            if completed_steps >= args.max_train_steps:\n                break\n\n        model.eval()\n        losses = []\n        for step, batch in enumerate(eval_dataloader):\n            with torch.no_grad():\n                outputs = model(**batch)\n\n            loss = outputs.loss\n            losses.append(accelerator.gather_for_metrics(loss.repeat(args.per_device_eval_batch_size)))\n\n        losses = torch.cat(losses)\n        try:\n            eval_loss = torch.mean(losses)\n            perplexity = math.exp(eval_loss)\n        except OverflowError:\n            perplexity = float(\"inf\")\n\n        logger.info(f\"epoch {epoch}: perplexity: {perplexity} eval_loss: {eval_loss}\")\n\n        if args.with_tracking:\n            accelerator.log(\n                {\n                    \"perplexity\": perplexity,\n                    \"eval_loss\": eval_loss,\n                    \"train_loss\": total_loss.item() / len(train_dataloader),\n                    \"epoch\": epoch,\n                    \"step\": completed_steps,\n                },\n                step=completed_steps,\n            )\n\n        if args.push_to_hub and epoch < args.num_train_epochs - 1:\n            accelerator.wait_for_everyone()\n            unwrapped_model = accelerator.unwrap_model(model)\n            unwrapped_model.save_pretrained(\n                args.output_dir, is_main_process=accelerator.is_main_process, save_function=accelerator.save\n            )\n            if accelerator.is_main_process:\n                tokenizer.save_pretrained(args.output_dir)\n                api.upload_folder(\n                    commit_message=f\"Training in progress epoch {epoch}\",\n                    folder_path=args.output_dir,\n                    repo_id=repo_id,\n                    repo_type=\"model\",\n                    token=args.hub_token,\n                )\n\n        if args.checkpointing_steps == \"epoch\":\n            output_dir = f\"epoch_{epoch}\"\n            if args.output_dir is not None:\n                output_dir = os.path.join(args.output_dir, output_dir)\n            accelerator.save_state(output_dir)\n\n    if args.with_tracking:\n        accelerator.end_training()\n\n    if args.output_dir is not None:\n        accelerator.wait_for_everyone()\n        unwrapped_model = accelerator.unwrap_model(model)\n        unwrapped_model.save_pretrained(\n            args.output_dir, is_main_process=accelerator.is_main_process, save_function=accelerator.save\n        )\n        if accelerator.is_main_process:\n            tokenizer.save_pretrained(args.output_dir)\n            if args.push_to_hub:\n                api.upload_folder(\n                    commit_message=\"End of training\",\n                    folder_path=args.output_dir,\n                    repo_id=repo_id,\n                    repo_type=\"model\",\n                    token=args.hub_token,\n                )\n            with open(os.path.join(args.output_dir, \"all_results.json\"), \"w\") as f:\n\n\nSUMMARY:\nThis code is for training a PyTorch model using the Transformer architecture. The model is trained on an entire corpus of text data and produces a sequence-to-sequence model that can translate or generate new text based on the input prompt.\n\nHere are some key features and behaviors:\n\n1. **Training loop**: The model trains iteratively, updating its weights using backpropagation to minimize the loss function.\n2. **Batching and parallelization**: The code uses PyTorch's `DataLoader` to load batches of data in parallel, utilizing multiple GPUs or CPU cores for faster training.\n3. **Loss and optimization**: The model measures its performance on a per-epoch basis using the perplexity score, which is calculated at each step of the training process.\n4. **Regularization and checkpointing**: The code saves the model's state every `checkpoint_steps` steps, ensuring that the model can be restored from a previous epoch in case of sudden termination of training.\n5. **Evaluation and metrics**: After completing an epoch, the model evaluates its performance on a held-out validation set, tracking the loss, perplexity, and other relevant metrics.\n\nHowever, there are also some areas for potential improvement:\n\n1. **Data preprocessing**: The code assumes that the input data is already preprocessed and split into batches, but in practice, this step may require significant effort.\n2. **Hyperparameter tuning**: While the code includes a basic optimizer (e.g., Adam), more extensive hyperparameter tuning would be beneficial to explore optimal settings for training performance.\n3. **Early stopping and patience**: The code does not implement early stopping or a patience mechanism, which can significantly improve the stability of the training process.\n\nTo make this code more concise and readable, here are some suggestions:\n\n1. Extract the training loop into a separate function to simplify and reuse the logic.\n2. Consider using `torchvision`'s `DataLoaders` instead of creating batches manually for easier management.\n3. Define constants or variables at the top to improve readability and minimize global variable usage.\n\nHere is an updated version of the code following these suggestions:\n\n```python\nimport torch\n\n# Set model architecture, training parameters, and data loading path\nmodel = TransformerModel()\noptimizer = Adam(model.parameters(), lr=0.001)\ndata_path = \"path/to/data\"\n\ndef train_model(data_loader, epoch):\n    # Model evaluation metrics will be tracked\n    total_loss = 0\n    \n    for batch in data_loader:\n        model.train()\n        optimizer.zero_grad()\n        \n        # Generate output and loss\n        \n        outputs = model(**batch)\n        loss = outputs.loss\n        loss.backward()\n        optimizer.step()\n\n        # Update total loss across batches\n        total_loss += loss.item() * len(batch[\"input_ids\"])\n    \n    # Return total loss for evaluation at the end of each epoch\n    return total_loss / (len(data_loader) * len(batch[\"input_ids\"]))\n```"}


created_at: 2025-02-05 09:13:40
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/image-classification/run_image_classification_no_trainer.py:58-218

metadata:
{"docstring": "", "end_line": 218, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/image-classification/run_image_classification_no_trainer.py", "name": "parse_args", "start_line": 58, "type": "function", "chroma:document": "CODE:\n\ndef parse_args():\n    parser = argparse.ArgumentParser(description=\"Fine-tune a Transformers model on an image classification dataset\")\n    parser.add_argument(\n        \"--dataset_name\",\n        type=str,\n        default=\"cifar10\",\n        help=(\n            \"The name of the Dataset (from the HuggingFace hub) to train on (could be your own, possibly private,\"\n            \" dataset).\"\n        ),\n    )\n    parser.add_argument(\"--train_dir\", type=str, default=None, help=\"A folder containing the training data.\")\n    parser.add_argument(\"--validation_dir\", type=str, default=None, help=\"A folder containing the validation data.\")\n    parser.add_argument(\n        \"--max_train_samples\",\n        type=int,\n        default=None,\n        help=(\n            \"For debugging purposes or quicker training, truncate the number of training examples to this \"\n            \"value if set.\"\n        ),\n    )\n    parser.add_argument(\n        \"--max_eval_samples\",\n        type=int,\n        default=None,\n        help=(\n            \"For debugging purposes or quicker training, truncate the number of evaluation examples to this \"\n            \"value if set.\"\n        ),\n    )\n    parser.add_argument(\n        \"--train_val_split\",\n        type=float,\n        default=0.15,\n        help=\"Percent to split off of train for validation\",\n    )\n    parser.add_argument(\n        \"--model_name_or_path\",\n        type=str,\n        help=\"Path to pretrained model or model identifier from huggingface.co/models.\",\n        default=\"google/vit-base-patch16-224-in21k\",\n    )\n    parser.add_argument(\n        \"--per_device_train_batch_size\",\n        type=int,\n        default=8,\n        help=\"Batch size (per device) for the training dataloader.\",\n    )\n    parser.add_argument(\n        \"--per_device_eval_batch_size\",\n        type=int,\n        default=8,\n        help=\"Batch size (per device) for the evaluation dataloader.\",\n    )\n    parser.add_argument(\n        \"--learning_rate\",\n        type=float,\n        default=5e-5,\n        help=\"Initial learning rate (after the potential warmup period) to use.\",\n    )\n    parser.add_argument(\"--weight_decay\", type=float, default=0.0, help=\"Weight decay to use.\")\n    parser.add_argument(\"--num_train_epochs\", type=int, default=3, help=\"Total number of training epochs to perform.\")\n    parser.add_argument(\n        \"--max_train_steps\",\n        type=int,\n        default=None,\n        help=\"Total number of training steps to perform. If provided, overrides num_train_epochs.\",\n    )\n    parser.add_argument(\n        \"--gradient_accumulation_steps\",\n        type=int,\n        default=1,\n        help=\"Number of updates steps to accumulate before performing a backward/update pass.\",\n    )\n    parser.add_argument(\n        \"--lr_scheduler_type\",\n        type=SchedulerType,\n        default=\"linear\",\n        help=\"The scheduler type to use.\",\n        choices=[\"linear\", \"cosine\", \"cosine_with_restarts\", \"polynomial\", \"constant\", \"constant_with_warmup\"],\n    )\n    parser.add_argument(\n        \"--num_warmup_steps\", type=int, default=0, help=\"Number of steps for the warmup in the lr scheduler.\"\n    )\n    parser.add_argument(\"--output_dir\", type=str, default=None, help=\"Where to store the final model.\")\n    parser.add_argument(\"--seed\", type=int, default=None, help=\"A seed for reproducible training.\")\n    parser.add_argument(\"--push_to_hub\", action=\"store_true\", help=\"Whether or not to push the model to the Hub.\")\n    parser.add_argument(\n        \"--hub_model_id\", type=str, help=\"The name of the repository to keep in sync with the local `output_dir`.\"\n    )\n    parser.add_argument(\"--hub_token\", type=str, help=\"The token to use to push to the Model Hub.\")\n    parser.add_argument(\n        \"--trust_remote_code\",\n        action=\"store_true\",\n        help=(\n            \"Whether to trust the execution of code from datasets/models defined on the Hub.\"\n            \" This option should only be set to `True` for repositories you trust and in which you have read the\"\n            \" code, as it will execute code present on the Hub on your local machine.\"\n        ),\n    )\n    parser.add_argument(\n        \"--checkpointing_steps\",\n        type=str,\n        default=None,\n        help=\"Whether the various states should be saved at the end of every n steps, or 'epoch' for each epoch.\",\n    )\n    parser.add_argument(\n        \"--resume_from_checkpoint\",\n        type=str,\n        default=None,\n        help=\"If the training should continue from a checkpoint folder.\",\n    )\n    parser.add_argument(\n        \"--with_tracking\",\n        action=\"store_true\",\n        help=\"Whether to enable experiment trackers for logging.\",\n    )\n    parser.add_argument(\n        \"--report_to\",\n        type=str,\n        default=\"all\",\n        help=(\n            'The integration to report the results and logs to. Supported platforms are `\"tensorboard\"`,'\n            ' `\"wandb\"`, `\"comet_ml\"` and `\"clearml\"`. Use `\"all\"` (default) to report to all integrations. '\n            \"Only applicable when `--with_tracking` is passed.\"\n        ),\n    )\n    parser.add_argument(\n        \"--ignore_mismatched_sizes\",\n        action=\"store_true\",\n        help=\"Whether or not to enable to load a pretrained model whose head dimensions are different.\",\n    )\n    parser.add_argument(\n        \"--image_column_name\",\n        type=str,\n        default=\"image\",\n        help=\"The name of the dataset column containing the image data. Defaults to 'image'.\",\n    )\n    parser.add_argument(\n        \"--label_column_name\",\n        type=str,\n        default=\"label\",\n        help=\"The name of the dataset column containing the labels. Defaults to 'label'.\",\n    )\n    args = parser.parse_args()\n\n    # Sanity checks\n    if args.dataset_name is None and args.train_dir is None and args.validation_dir is None:\n        raise ValueError(\"Need either a dataset name or a training/validation folder.\")\n\n    if args.push_to_hub or args.with_tracking:\n        if args.output_dir is None:\n            raise ValueError(\n                \"Need an `output_dir` to create a repo when `--push_to_hub` or `with_tracking` is specified.\"\n            )\n\n    if args.output_dir is not None:\n        os.makedirs(args.output_dir, exist_ok=True)\n\n\n\nSUMMARY:\n**Function Summary:**\n\nThis function generates and parses command-line arguments for fine-tuning a pre-trained Transformers model on an image classification dataset. It defines various hyperparameters, paths to training/validation datasets and checkpoints, and tracking options.\n\n**Key Features:**\n\n* Generates command-line arguments using `argparse`\n* Defines over 30 hyperparameters with default values\n* Supports loading pre-trained models from HuggingFace Hub or own repositories\n* Enforces sanity checks for required inputs\n\n**Function Signature:**\n```python\ndef parse_args():\n    parser = ...  # construct parser object and define arguments\n    args = parser.parse_args()\n    return args\n```\n**Notable Functions/Variables:**\n\n* `argparse.ArgumentParser`: creates an argument parser object\n* `SchedulerType` (not defined in this code snippet): represents different learning rate schedules\n* `output_dir`, `train_dir`, `validation_dir`: paths to output, training, and validation directories respectively"}


created_at: 2025-02-05 09:13:40
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/image-classification/run_image_classification_no_trainer.py:221-641

metadata:
{"docstring": "", "end_line": 641, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/image-classification/run_image_classification_no_trainer.py", "name": "main", "start_line": 221, "type": "function", "chroma:document": "CODE:\n\ndef main():\n    args = parse_args()\n\n    # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The\n    # information sent is the one passed as arguments along with your Python/PyTorch versions.\n    send_example_telemetry(\"run_image_classification_no_trainer\", args)\n\n    # Initialize the accelerator. We will let the accelerator handle device placement for us in this example.\n    # If we're using tracking, we also need to initialize it here and it will by default pick up all supported trackers\n    # in the environment\n    accelerator_log_kwargs = {}\n\n    if args.with_tracking:\n        accelerator_log_kwargs[\"log_with\"] = args.report_to\n        accelerator_log_kwargs[\"project_dir\"] = args.output_dir\n\n    accelerator = Accelerator(gradient_accumulation_steps=args.gradient_accumulation_steps, **accelerator_log_kwargs)\n\n    logger.info(accelerator.state)\n    # Make one log on every process with the configuration for debugging.\n    logging.basicConfig(\n        format=\"%(asctime)s - %(levelname)s - %(name)s - %(message)s\",\n        datefmt=\"%m/%d/%Y %H:%M:%S\",\n        level=logging.INFO,\n    )\n    logger.info(accelerator.state, main_process_only=False)\n    if accelerator.is_local_main_process:\n        datasets.utils.logging.set_verbosity_warning()\n        transformers.utils.logging.set_verbosity_info()\n    else:\n        datasets.utils.logging.set_verbosity_error()\n        transformers.utils.logging.set_verbosity_error()\n\n    # If passed along, set the training seed now.\n    if args.seed is not None:\n        set_seed(args.seed)\n\n    # Handle the repository creation\n    if accelerator.is_main_process:\n        if args.push_to_hub:\n            # Retrieve of infer repo_name\n            repo_name = args.hub_model_id\n            if repo_name is None:\n                repo_name = Path(args.output_dir).absolute().name\n            # Create repo and retrieve repo_id\n            api = HfApi()\n            repo_id = api.create_repo(repo_name, exist_ok=True, token=args.hub_token).repo_id\n\n            with open(os.path.join(args.output_dir, \".gitignore\"), \"w+\") as gitignore:\n                if \"step_*\" not in gitignore:\n                    gitignore.write(\"step_*\\n\")\n                if \"epoch_*\" not in gitignore:\n                    gitignore.write(\"epoch_*\\n\")\n        elif args.output_dir is not None:\n            os.makedirs(args.output_dir, exist_ok=True)\n    accelerator.wait_for_everyone()\n\n    # Get the datasets: you can either provide your own training and evaluation files (see below)\n    # or specify a Dataset from the hub (the dataset will be downloaded automatically from the datasets Hub).\n\n    # In distributed training, the load_dataset function guarantees that only one local process can concurrently\n    # download the dataset.\n    if args.dataset_name is not None:\n        # Downloading and loading a dataset from the hub.\n        dataset = load_dataset(args.dataset_name, trust_remote_code=args.trust_remote_code)\n    else:\n        data_files = {}\n        if args.train_dir is not None:\n            data_files[\"train\"] = os.path.join(args.train_dir, \"**\")\n        if args.validation_dir is not None:\n            data_files[\"validation\"] = os.path.join(args.validation_dir, \"**\")\n        dataset = load_dataset(\n            \"imagefolder\",\n            data_files=data_files,\n        )\n        # See more about loading custom images at\n        # https://huggingface.co/docs/datasets/v2.0.0/en/image_process#imagefolder.\n\n    dataset_column_names = dataset[\"train\"].column_names if \"train\" in dataset else dataset[\"validation\"].column_names\n    if args.image_column_name not in dataset_column_names:\n        raise ValueError(\n            f\"--image_column_name {args.image_column_name} not found in dataset '{args.dataset_name}'. \"\n            \"Make sure to set `--image_column_name` to the correct audio column - one of \"\n            f\"{', '.join(dataset_column_names)}.\"\n        )\n    if args.label_column_name not in dataset_column_names:\n        raise ValueError(\n            f\"--label_column_name {args.label_column_name} not found in dataset '{args.dataset_name}'. \"\n            \"Make sure to set `--label_column_name` to the correct text column - one of \"\n            f\"{', '.join(dataset_column_names)}.\"\n        )\n\n    # If we don't have a validation split, split off a percentage of train as validation.\n    args.train_val_split = None if \"validation\" in dataset.keys() else args.train_val_split\n    if isinstance(args.train_val_split, float) and args.train_val_split > 0.0:\n        split = dataset[\"train\"].train_test_split(args.train_val_split)\n        dataset[\"train\"] = split[\"train\"]\n        dataset[\"validation\"] = split[\"test\"]\n\n    # Prepare label mappings.\n    # We'll include these in the model's config to get human readable labels in the Inference API.\n    labels = dataset[\"train\"].features[args.label_column_name].names\n    label2id = {label: str(i) for i, label in enumerate(labels)}\n    id2label = {str(i): label for i, label in enumerate(labels)}\n\n    # Load pretrained model and image processor\n    #\n    # In distributed training, the .from_pretrained methods guarantee that only one local process can concurrently\n    # download model & vocab.\n    config = AutoConfig.from_pretrained(\n        args.model_name_or_path,\n        num_labels=len(labels),\n        id2label=id2label,\n        label2id=label2id,\n        finetuning_task=\"image-classification\",\n        trust_remote_code=args.trust_remote_code,\n    )\n    image_processor = AutoImageProcessor.from_pretrained(\n        args.model_name_or_path,\n        trust_remote_code=args.trust_remote_code,\n    )\n    model = AutoModelForImageClassification.from_pretrained(\n        args.model_name_or_path,\n        from_tf=bool(\".ckpt\" in args.model_name_or_path),\n        config=config,\n        ignore_mismatched_sizes=args.ignore_mismatched_sizes,\n        trust_remote_code=args.trust_remote_code,\n    )\n\n    # Preprocessing the datasets\n\n    # Define torchvision transforms to be applied to each image.\n    if \"shortest_edge\" in image_processor.size:\n        size = image_processor.size[\"shortest_edge\"]\n    else:\n        size = (image_processor.size[\"height\"], image_processor.size[\"width\"])\n    normalize = (\n        Normalize(mean=image_processor.image_mean, std=image_processor.image_std)\n        if hasattr(image_processor, \"image_mean\") and hasattr(image_processor, \"image_std\")\n        else Lambda(lambda x: x)\n    )\n    train_transforms = Compose(\n        [\n            RandomResizedCrop(size),\n            RandomHorizontalFlip(),\n            ToTensor(),\n            normalize,\n        ]\n    )\n    val_transforms = Compose(\n        [\n            Resize(size),\n            CenterCrop(size),\n            ToTensor(),\n            normalize,\n        ]\n    )\n\n    def preprocess_train(example_batch):\n        \"\"\"Apply _train_transforms across a batch.\"\"\"\n        example_batch[\"pixel_values\"] = [\n            train_transforms(image.convert(\"RGB\")) for image in example_batch[args.image_column_name]\n        ]\n        return example_batch\n\n    def preprocess_val(example_batch):\n        \"\"\"Apply _val_transforms across a batch.\"\"\"\n        example_batch[\"pixel_values\"] = [\n            val_transforms(image.convert(\"RGB\")) for image in example_batch[args.image_column_name]\n        ]\n        return example_batch\n\n    with accelerator.main_process_first():\n        if args.max_train_samples is not None:\n            dataset[\"train\"] = dataset[\"train\"].shuffle(seed=args.seed).select(range(args.max_train_samples))\n        # Set the training transforms\n        train_dataset = dataset[\"train\"].with_transform(preprocess_train)\n        if args.max_eval_samples is not None:\n            dataset[\"validation\"] = dataset[\"validation\"].shuffle(seed=args.seed).select(range(args.max_eval_samples))\n        # Set the validation transforms\n        eval_dataset = dataset[\"validation\"].with_transform(preprocess_val)\n\n    # DataLoaders creation:\n    def collate_fn(examples):\n        pixel_values = torch.stack([example[\"pixel_values\"] for example in examples])\n        labels = torch.tensor([example[args.label_column_name] for example in examples])\n        return {\"pixel_values\": pixel_values, \"labels\": labels}\n\n    train_dataloader = DataLoader(\n        train_dataset, shuffle=True, collate_fn=collate_fn, batch_size=args.per_device_train_batch_size\n    )\n    eval_dataloader = DataLoader(eval_dataset, collate_fn=collate_fn, batch_size=args.per_device_eval_batch_size)\n\n    # Optimizer\n    # Split weights in two groups, one with weight decay and the other not.\n    no_decay = [\"bias\", \"LayerNorm.weight\"]\n    optimizer_grouped_parameters = [\n        {\n            \"params\": [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)],\n            \"weight_decay\": args.weight_decay,\n        },\n        {\n            \"params\": [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)],\n            \"weight_decay\": 0.0,\n        },\n    ]\n    optimizer = torch.optim.AdamW(optimizer_grouped_parameters, lr=args.learning_rate)\n\n    # Scheduler and math around the number of training steps.\n    overrode_max_train_steps = False\n    num_update_steps_per_epoch = math.ceil(len(train_dataloader) / args.gradient_accumulation_steps)\n    if args.max_train_steps is None:\n        args.max_train_steps = args.num_train_epochs * num_update_steps_per_epoch\n        overrode_max_train_steps = True\n\n    lr_scheduler = get_scheduler(\n        name=args.lr_scheduler_type,\n        optimizer=optimizer,\n        num_warmup_steps=args.num_warmup_steps * accelerator.num_processes,\n        num_training_steps=args.max_train_steps\n        if overrode_max_train_steps\n        else args.max_train_steps * accelerator.num_processes,\n    )\n\n    # Prepare everything with our `accelerator`.\n    model, optimizer, train_dataloader, eval_dataloader, lr_scheduler = accelerator.prepare(\n        model, optimizer, train_dataloader, eval_dataloader, lr_scheduler\n    )\n\n    # We need to recalculate our total training steps as the size of the training dataloader may have changed.\n    num_update_steps_per_epoch = math.ceil(len(train_dataloader) / args.gradient_accumulation_steps)\n    if overrode_max_train_steps:\n        args.max_train_steps = args.num_train_epochs * num_update_steps_per_epoch\n    # Afterwards we recalculate our number of training epochs\n    args.num_train_epochs = math.ceil(args.max_train_steps / num_update_steps_per_epoch)\n\n    # Figure out how many steps we should save the Accelerator states\n    checkpointing_steps = args.checkpointing_steps\n    if checkpointing_steps is not None and checkpointing_steps.isdigit():\n        checkpointing_steps = int(checkpointing_steps)\n\n    # We need to initialize the trackers we use, and also store our configuration.\n    # The trackers initializes automatically on the main process.\n    if args.with_tracking:\n        experiment_config = vars(args)\n        # TensorBoard cannot log Enums, need the raw value\n        experiment_config[\"lr_scheduler_type\"] = experiment_config[\"lr_scheduler_type\"].value\n        accelerator.init_trackers(\"image_classification_no_trainer\", experiment_config)\n\n    # Get the metric function\n    metric = evaluate.load(\"accuracy\")\n\n    # Train!\n    total_batch_size = args.per_device_train_batch_size * accelerator.num_processes * args.gradient_accumulation_steps\n\n    logger.info(\"***** Running training *****\")\n    logger.info(f\"  Num examples = {len(train_dataset)}\")\n    logger.info(f\"  Num Epochs = {args.num_train_epochs}\")\n    logger.info(f\"  Instantaneous batch size per device = {args.per_device_train_batch_size}\")\n    logger.info(f\"  Total train batch size (w. parallel, distributed & accumulation) = {total_batch_size}\")\n    logger.info(f\"  Gradient Accumulation steps = {args.gradient_accumulation_steps}\")\n    logger.info(f\"  Total optimization steps = {args.max_train_steps}\")\n    # Only show the progress bar once on each machine.\n    progress_bar = tqdm(range(args.max_train_steps), disable=not accelerator.is_local_main_process)\n    completed_steps = 0\n    starting_epoch = 0\n    # Potentially load in the weights and states from a previous save\n    if args.resume_from_checkpoint:\n        if args.resume_from_checkpoint is not None or args.resume_from_checkpoint != \"\":\n            checkpoint_path = args.resume_from_checkpoint\n            path = os.path.basename(args.resume_from_checkpoint)\n        else:\n            # Get the most recent checkpoint\n            dirs = [f.name for f in os.scandir(os.getcwd()) if f.is_dir()]\n            dirs.sort(key=os.path.getctime)\n            path = dirs[-1]  # Sorts folders by date modified, most recent checkpoint is the last\n            checkpoint_path = path\n            path = os.path.basename(checkpoint_path)\n\n        accelerator.print(f\"Resumed from checkpoint: {checkpoint_path}\")\n        accelerator.load_state(checkpoint_path)\n        # Extract `epoch_{i}` or `step_{i}`\n        training_difference = os.path.splitext(path)[0]\n\n        if \"epoch\" in training_difference:\n            starting_epoch = int(training_difference.replace(\"epoch_\", \"\")) + 1\n            resume_step = None\n            completed_steps = starting_epoch * num_update_steps_per_epoch\n        else:\n            # need to multiply `gradient_accumulation_steps` to reflect real steps\n            resume_step = int(training_difference.replace(\"step_\", \"\")) * args.gradient_accumulation_steps\n            starting_epoch = resume_step // len(train_dataloader)\n            completed_steps = resume_step // args.gradient_accumulation_steps\n            resume_step -= starting_epoch * len(train_dataloader)\n\n    # update the progress_bar if load from checkpoint\n    progress_bar.update(completed_steps)\n\n    for epoch in range(starting_epoch, args.num_train_epochs):\n        model.train()\n        if args.with_tracking:\n            total_loss = 0\n        if args.resume_from_checkpoint and epoch == starting_epoch and resume_step is not None:\n            # We skip the first `n` batches in the dataloader when resuming from a checkpoint\n            active_dataloader = accelerator.skip_first_batches(train_dataloader, resume_step)\n        else:\n            active_dataloader = train_dataloader\n        for step, batch in enumerate(active_dataloader):\n            with accelerator.accumulate(model):\n                outputs = model(**batch)\n                loss = outputs.loss\n                # We keep track of the loss at each epoch\n                if args.with_tracking:\n                    total_loss += loss.detach().float()\n                accelerator.backward(loss)\n                optimizer.step()\n                lr_scheduler.step()\n                optimizer.zero_grad()\n\n            # Checks if the accelerator has performed an optimization step behind the scenes\n            if accelerator.sync_gradients:\n                progress_bar.update(1)\n                completed_steps += 1\n\n            if isinstance(checkpointing_steps, int):\n                if completed_steps % checkpointing_steps == 0 and accelerator.sync_gradients:\n                    output_dir = f\"step_{completed_steps}\"\n                    if args.output_dir is not None:\n                        output_dir = os.path.join(args.output_dir, output_dir)\n                    accelerator.save_state(output_dir)\n\n                    if args.push_to_hub and epoch < args.num_train_epochs - 1:\n                        accelerator.wait_for_everyone()\n                        unwrapped_model = accelerator.unwrap_model(model)\n                        unwrapped_model.save_pretrained(\n                            args.output_dir,\n                            is_main_process=accelerator.is_main_process,\n                            save_function=accelerator.save,\n                        )\n                        if accelerator.is_main_process:\n                            image_processor.save_pretrained(args.output_dir)\n                            api.upload_folder(\n                                commit_message=f\"Training in progress epoch {epoch}\",\n                                folder_path=args.output_dir,\n                                repo_id=repo_id,\n                                repo_type=\"model\",\n                                token=args.hub_token,\n                            )\n\n            if completed_steps >= args.max_train_steps:\n                break\n\n        model.eval()\n        for step, batch in enumerate(eval_dataloader):\n            with torch.no_grad():\n                outputs = model(**batch)\n            predictions = outputs.logits.argmax(dim=-1)\n            predictions, references = accelerator.gather_for_metrics((predictions, batch[\"labels\"]))\n            metric.add_batch(\n                predictions=predictions,\n                references=references,\n            )\n\n        eval_metric = metric.compute()\n        logger.info(f\"epoch {epoch}: {eval_metric}\")\n\n        if args.with_tracking:\n            accelerator.log(\n                {\n                    \"accuracy\": eval_metric,\n                    \"train_loss\": total_loss.item() / len(train_dataloader),\n                    \"epoch\": epoch,\n                    \"step\": completed_steps,\n                },\n                step=completed_steps,\n            )\n\n        if args.push_to_hub and epoch < args.num_train_epochs - 1:\n            accelerator.wait_for_everyone()\n            unwrapped_model = accelerator.unwrap_model(model)\n            unwrapped_model.save_pretrained(\n                args.output_dir, is_main_process=accelerator.is_main_process, save_function=accelerator.save\n            )\n            if accelerator.is_main_process:\n                image_processor.save_pretrained(args.output_dir)\n                api.upload_folder(\n                    commit_message=f\"Training in progress epoch {epoch}\",\n                    folder_path=args.output_dir,\n                    repo_id=repo_id,\n                    repo_type=\"model\",\n                    token=args.hub_token,\n                )\n\n        if args.checkpointing_steps == \"epoch\":\n            output_dir = f\"epoch_{epoch}\"\n            if args.output_dir is not None:\n                output_dir = os.path.join(args.output_dir, output_dir)\n            accelerator.save_state(output_dir)\n\n    if args.with_tracking:\n        accelerator.end_training()\n\n    if args.output_dir is not None:\n        accelerator.wait_for_everyone()\n        unwrapped_model = accelerator.unwrap_model(model)\n        unwrapped_model.save_pretrained(\n            args.output_dir, is_main_process=accelerator.is_main_process, save_function=accelerator.save\n        )\n        if accelerator.is_main_process:\n            image_processor.save_pretrained(args.output_dir)\n            if args.push_to_hub:\n                api.upload_folder(\n                    commit_message=\"End of training\",\n                    folder_path=args.output_dir,\n                    repo_id=repo_id,\n                    repo_type=\"model\",\n                    token=args.hub_token,\n                )\n            all_results = {f\"eval_{k}\": v for k, v in eval_metric.items()}\n            with open(os.path.join(args.output_dir, \"all_results.json\"), \"w\") as f:\n\n\nSUMMARY:\nThis code is used to train a model and evaluate its performance on a dataset. Here's a brief summary:\n\n**Training**\n\n* The model is trained for multiple epochs\n* During each epoch, the model processes batches of data from a training dataset\n* The loss is calculated for each batch and backpropagated through the network\n* The optimizer updates the model parameters based on the gradient\n\n**Evaluation**\n\n* After each epoch, the model evaluates its performance on a validation dataset\n* The accuracy, loss, epoch number, and step size are logged to a CSV file if tracking is enabled\n* If pushing to Hub is enabled, the model is uploaded to Hugging Face with metadata\n\n**Saving**\n\n* The trained model is saved to a directory at specified output path when training finishes\n* The model can be saved to JSON format using `json.dump` and loaded later for further use\n\nThe code uses PyTorch and its Accelerator library to parallelize the training process across multiple GPUs, which can significantly speed up the training time."}


created_at: 2025-02-05 09:13:40
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/image-classification/run_image_classification.py:67-70

metadata:
{"docstring": "", "end_line": 70, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/image-classification/run_image_classification.py", "name": "pil_loader", "start_line": 67, "type": "function", "chroma:document": "CODE:\n\ndef pil_loader(path: str):\n    with open(path, \"rb\") as f:\n        im = Image.open(f)\n\n\nSUMMARY:\n**pil_loader Function Summary**\n\n### Purpose and Functionality\nThe `pil_loader` function loads an image from a given path and converts it to RGB color mode.\n\n### Key Implementation Details\n- Opens the specified file in binary read mode (`\"rb\"`).\n- Uses the Python Imaging Library (PIL) to open the image, represented by the `im` variable.\n- Converts the loaded image to the RGB color space using the `convert(\"RGB\")` method.\n\n### Notable Dependencies or Requirements\n\n* **Python Imaging Library (PIL):** Import in the original codebase is required to utilize this function.\n* **Image.open() :** Utilizes a PIL class (`<module.pil.Image>`) to create an image object."}


created_at: 2025-02-05 09:13:40
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/image-classification/run_image_classification.py:179-437

metadata:
{"docstring": "", "end_line": 437, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/image-classification/run_image_classification.py", "name": "main", "start_line": 179, "type": "function", "chroma:document": "CODE:\n\ndef main():\n    # See all possible arguments in src/transformers/training_args.py\n    # or by passing the --help flag to this script.\n    # We now keep distinct sets of args, for a cleaner separation of concerns.\n\n    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TrainingArguments))\n    if len(sys.argv) == 2 and sys.argv[1].endswith(\".json\"):\n        # If we pass only one argument to the script and it's the path to a json file,\n        # let's parse it to get our arguments.\n        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))\n    else:\n        model_args, data_args, training_args = parser.parse_args_into_dataclasses()\n\n    # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The\n    # information sent is the one passed as arguments along with your Python/PyTorch versions.\n    send_example_telemetry(\"run_image_classification\", model_args, data_args)\n\n    # Setup logging\n    logging.basicConfig(\n        format=\"%(asctime)s - %(levelname)s - %(name)s - %(message)s\",\n        datefmt=\"%m/%d/%Y %H:%M:%S\",\n        handlers=[logging.StreamHandler(sys.stdout)],\n    )\n\n    if training_args.should_log:\n        # The default of training_args.log_level is passive, so we set log level at info here to have that default.\n        transformers.utils.logging.set_verbosity_info()\n\n    log_level = training_args.get_process_log_level()\n    logger.setLevel(log_level)\n    transformers.utils.logging.set_verbosity(log_level)\n    transformers.utils.logging.enable_default_handler()\n    transformers.utils.logging.enable_explicit_format()\n\n    # Log on each process the small summary:\n    logger.warning(\n        f\"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}, \"\n        + f\"distributed training: {training_args.parallel_mode.value == 'distributed'}, 16-bits training: {training_args.fp16}\"\n    )\n    logger.info(f\"Training/evaluation parameters {training_args}\")\n\n    # Detecting last checkpoint.\n    last_checkpoint = None\n    if os.path.isdir(training_args.output_dir) and training_args.do_train and not training_args.overwrite_output_dir:\n        last_checkpoint = get_last_checkpoint(training_args.output_dir)\n        if last_checkpoint is None and len(os.listdir(training_args.output_dir)) > 0:\n            raise ValueError(\n                f\"Output directory ({training_args.output_dir}) already exists and is not empty. \"\n                \"Use --overwrite_output_dir to overcome.\"\n            )\n        elif last_checkpoint is not None and training_args.resume_from_checkpoint is None:\n            logger.info(\n                f\"Checkpoint detected, resuming training at {last_checkpoint}. To avoid this behavior, change \"\n                \"the `--output_dir` or add `--overwrite_output_dir` to train from scratch.\"\n            )\n\n    # Set seed before initializing model.\n    set_seed(training_args.seed)\n\n    # Initialize our dataset and prepare it for the 'image-classification' task.\n    if data_args.dataset_name is not None:\n        dataset = load_dataset(\n            data_args.dataset_name,\n            data_args.dataset_config_name,\n            cache_dir=model_args.cache_dir,\n            token=model_args.token,\n            trust_remote_code=model_args.trust_remote_code,\n        )\n    else:\n        data_files = {}\n        if data_args.train_dir is not None:\n            data_files[\"train\"] = os.path.join(data_args.train_dir, \"**\")\n        if data_args.validation_dir is not None:\n            data_files[\"validation\"] = os.path.join(data_args.validation_dir, \"**\")\n        dataset = load_dataset(\n            \"imagefolder\",\n            data_files=data_files,\n            cache_dir=model_args.cache_dir,\n        )\n\n    dataset_column_names = dataset[\"train\"].column_names if \"train\" in dataset else dataset[\"validation\"].column_names\n    if data_args.image_column_name not in dataset_column_names:\n        raise ValueError(\n            f\"--image_column_name {data_args.image_column_name} not found in dataset '{data_args.dataset_name}'. \"\n            \"Make sure to set `--image_column_name` to the correct audio column - one of \"\n            f\"{', '.join(dataset_column_names)}.\"\n        )\n    if data_args.label_column_name not in dataset_column_names:\n        raise ValueError(\n            f\"--label_column_name {data_args.label_column_name} not found in dataset '{data_args.dataset_name}'. \"\n            \"Make sure to set `--label_column_name` to the correct text column - one of \"\n            f\"{', '.join(dataset_column_names)}.\"\n        )\n\n    def collate_fn(examples):\n        pixel_values = torch.stack([example[\"pixel_values\"] for example in examples])\n        labels = torch.tensor([example[data_args.label_column_name] for example in examples])\n        return {\"pixel_values\": pixel_values, \"labels\": labels}\n\n    # If we don't have a validation split, split off a percentage of train as validation.\n    data_args.train_val_split = None if \"validation\" in dataset.keys() else data_args.train_val_split\n    if isinstance(data_args.train_val_split, float) and data_args.train_val_split > 0.0:\n        split = dataset[\"train\"].train_test_split(data_args.train_val_split)\n        dataset[\"train\"] = split[\"train\"]\n        dataset[\"validation\"] = split[\"test\"]\n\n    # Prepare label mappings.\n    # We'll include these in the model's config to get human readable labels in the Inference API.\n    labels = dataset[\"train\"].features[data_args.label_column_name].names\n    label2id, id2label = {}, {}\n    for i, label in enumerate(labels):\n        label2id[label] = str(i)\n        id2label[str(i)] = label\n\n    # Load the accuracy metric from the datasets package\n    metric = evaluate.load(\"accuracy\", cache_dir=model_args.cache_dir)\n\n    # Define our compute_metrics function. It takes an `EvalPrediction` object (a namedtuple with a\n    # predictions and label_ids field) and has to return a dictionary string to float.\n    def compute_metrics(p):\n        \"\"\"Computes accuracy on a batch of predictions\"\"\"\n        return metric.compute(predictions=np.argmax(p.predictions, axis=1), references=p.label_ids)\n\n    config = AutoConfig.from_pretrained(\n        model_args.config_name or model_args.model_name_or_path,\n        num_labels=len(labels),\n        label2id=label2id,\n        id2label=id2label,\n        finetuning_task=\"image-classification\",\n        cache_dir=model_args.cache_dir,\n        revision=model_args.model_revision,\n        token=model_args.token,\n        trust_remote_code=model_args.trust_remote_code,\n    )\n    model = AutoModelForImageClassification.from_pretrained(\n        model_args.model_name_or_path,\n        from_tf=bool(\".ckpt\" in model_args.model_name_or_path),\n        config=config,\n        cache_dir=model_args.cache_dir,\n        revision=model_args.model_revision,\n        token=model_args.token,\n        trust_remote_code=model_args.trust_remote_code,\n        ignore_mismatched_sizes=model_args.ignore_mismatched_sizes,\n    )\n    image_processor = AutoImageProcessor.from_pretrained(\n        model_args.image_processor_name or model_args.model_name_or_path,\n        cache_dir=model_args.cache_dir,\n        revision=model_args.model_revision,\n        token=model_args.token,\n        trust_remote_code=model_args.trust_remote_code,\n    )\n\n    # Define torchvision transforms to be applied to each image.\n    if isinstance(image_processor, TimmWrapperImageProcessor):\n        _train_transforms = image_processor.train_transforms\n        _val_transforms = image_processor.val_transforms\n    else:\n        if \"shortest_edge\" in image_processor.size:\n            size = image_processor.size[\"shortest_edge\"]\n        else:\n            size = (image_processor.size[\"height\"], image_processor.size[\"width\"])\n\n        # Create normalization transform\n        if hasattr(image_processor, \"image_mean\") and hasattr(image_processor, \"image_std\"):\n            normalize = Normalize(mean=image_processor.image_mean, std=image_processor.image_std)\n        else:\n            normalize = Lambda(lambda x: x)\n        _train_transforms = Compose(\n            [\n                RandomResizedCrop(size),\n                RandomHorizontalFlip(),\n                ToTensor(),\n                normalize,\n            ]\n        )\n        _val_transforms = Compose(\n            [\n                Resize(size),\n                CenterCrop(size),\n                ToTensor(),\n                normalize,\n            ]\n        )\n\n    def train_transforms(example_batch):\n        \"\"\"Apply _train_transforms across a batch.\"\"\"\n        example_batch[\"pixel_values\"] = [\n            _train_transforms(pil_img.convert(\"RGB\")) for pil_img in example_batch[data_args.image_column_name]\n        ]\n        return example_batch\n\n    def val_transforms(example_batch):\n        \"\"\"Apply _val_transforms across a batch.\"\"\"\n        example_batch[\"pixel_values\"] = [\n            _val_transforms(pil_img.convert(\"RGB\")) for pil_img in example_batch[data_args.image_column_name]\n        ]\n        return example_batch\n\n    if training_args.do_train:\n        if \"train\" not in dataset:\n            raise ValueError(\"--do_train requires a train dataset\")\n        if data_args.max_train_samples is not None:\n            dataset[\"train\"] = (\n                dataset[\"train\"].shuffle(seed=training_args.seed).select(range(data_args.max_train_samples))\n            )\n        # Set the training transforms\n        dataset[\"train\"].set_transform(train_transforms)\n\n    if training_args.do_eval:\n        if \"validation\" not in dataset:\n            raise ValueError(\"--do_eval requires a validation dataset\")\n        if data_args.max_eval_samples is not None:\n            dataset[\"validation\"] = (\n                dataset[\"validation\"].shuffle(seed=training_args.seed).select(range(data_args.max_eval_samples))\n            )\n        # Set the validation transforms\n        dataset[\"validation\"].set_transform(val_transforms)\n\n    # Initialize our trainer\n    trainer = Trainer(\n        model=model,\n        args=training_args,\n        train_dataset=dataset[\"train\"] if training_args.do_train else None,\n        eval_dataset=dataset[\"validation\"] if training_args.do_eval else None,\n        compute_metrics=compute_metrics,\n        processing_class=image_processor,\n        data_collator=collate_fn,\n    )\n\n    # Training\n    if training_args.do_train:\n        checkpoint = None\n        if training_args.resume_from_checkpoint is not None:\n            checkpoint = training_args.resume_from_checkpoint\n        elif last_checkpoint is not None:\n            checkpoint = last_checkpoint\n        train_result = trainer.train(resume_from_checkpoint=checkpoint)\n        trainer.save_model()\n        trainer.log_metrics(\"train\", train_result.metrics)\n        trainer.save_metrics(\"train\", train_result.metrics)\n        trainer.save_state()\n\n    # Evaluation\n    if training_args.do_eval:\n        metrics = trainer.evaluate()\n        trainer.log_metrics(\"eval\", metrics)\n        trainer.save_metrics(\"eval\", metrics)\n\n    # Write model card and (optionally) push to hub\n    kwargs = {\n        \"finetuned_from\": model_args.model_name_or_path,\n        \"tasks\": \"image-classification\",\n        \"dataset\": data_args.dataset_name,\n        \"tags\": [\"image-classification\", \"vision\"],\n    }\n    if training_args.push_to_hub:\n        trainer.push_to_hub(**kwargs)\n    else:\n\n\nSUMMARY:\n**Trainer Class in Hugging Face Transformers**\n\nBelow is a concise version of the Trainer class defined in the Hugging Face Transformers library.\n\n```python\nfrom transformers import AutoModelForImageClassification, AutoImageProcessor, Compose, RandomResizedCrop, Resize, CenterCrop, Lambda, Normalize, TimmWrapperImageProcessor\n\nclass Trainer:\n    \"\"\"\n        The :class:`~transformers.Trainer` class is used for training or fine-tuning a model.\n        \n        Args:\n            **args:  \n                See :func:`transformers.TrainerArgs`\n            \n            model (Model): Model to be trained\n            args (TrainingArguments): Training arguments\n            \n        Classes:\n            :: :class:`~transformers.data.Datasets`: Pre-computed datasets.\n           (`train_dataset`: The train data)\n        \"\"\"\n    def __init__(self, \n                 model: AutoModelForImageClassification,\n                 args: TrainingArguments = TrainingArguments(),\n                 \n                 # Dataset\n                train_dataset: Datasets = None,\n                eval_dataset: Optional[Datasets] = None,\n\n                 # Data processing\n                data_collator: Callable[[Sequence], Dict[str, Any]] = None,\n                \n                compute_metrics: Callable[[DataCollatorWithBatch], Metrics] = None,\n\n                # Training related\n                ignore_mismatched_sizes: bool = True,\n                \n                gradient_accumulation_steps: int = 1,\n\n                per_device_train_batch_size: int = 8,\n                per_device_eval_batch_size: Optional[int] = None,\n                per_device_predict batch_size: Optional[int] = None,\n                \n                max_grad_norm: float = 1.0,\n                \n                # Gradient Clipping\n                gradient_clip_value: float = None,  # default is set to infinity\n        \n        training_args=TrainingArguments(\n            save_on_each_node=True,   \n            load_best_model_at_end=False,\n            metric_for_best_model=\"_accuracy\",\n            greater_is_better=True,\n            num_train_epochs=int(args.train_args.num_train_epochs), \n            per_device_train_batch_size = args.per_device_train_batch_size if 'per_device_train_batch_size' in str(args) else 8,\n            per_device_eval_batch_size =   args.per_device_eval_batch_size if 'per_device_eval_batch_size' in str(args) else None,\n            per_device_predict_batch_size = 128, \n            total_train_steps = None,  \n            optimization_impl='AdamW',  \n                no_cuda = not has_cuda(),\n                seed= random.randint(42, 212),  \n        \n            save_total_limit=2,\n            warmup_steps=0 if args.dry_run else int(args.warmup_percentage * args.train_args.num_train_epochs),\n            weight_decay=args.weight_decay,\n            load_best_and_tray=False,\n        ):\n            \n        self.model = model\n        self.args = training_args\n        # Model configuration data\n        \n        self.image_processor= AutoImageProcessor.from_pretrained(\n                        training_args.image_processor_name or training_args.model_name_or_path,\n                         cache_dir=training_args.cache_dir, \n                          revision=training_args.model_revision\n \n            )\n        \n                 # Train data transformer\n        \n\n        if not \"train_dataset\" with train_dataset:\n          raise ValueError(\"--train_dataset is required\")\n        \n        self.train_dataset=train_dataset\n     \n        \n        self.data_collator = data_collator or Compose([\n                RandomResizedCrop(max_resolution=(224, 224)),\n                RandomHorizontalFlip(),\n                ToTensor()\n            ])\n    \n    \n            if data_args.max_train_samples is not None:\n              train_dataset=tuple([e[0]for e in \n                                   train_dataset[\n                                       \"train\"].shuffle(seed=training_args.seed)\n                                   .select(range(\n                                       min(\n                                           training_args.max_train_samples,\n                                       len(\n                                           dataset[\"train\"])\n                                       )\n                                       ))])\n\n        # Set the validation dataset and validation data processor\n        if eval_dataset is not None:\n            assert isinstance(eval_dataset, Datasets), 'Training arguments must contain \"train_dataset\", but received {type(eval_dataset)}'\n            \n            self.eval_dataset=eval_dataset\n            self.eval_data_collator=data_collator or Compose([\n                Resize((224, 224)),\n                CenterCrop(224),\n                ToTensor()\n            ])\n\n\n        # Initialize the Trainer instance\n\nif training_args.do_train\n```"}


created_at: 2025-02-05 09:13:40
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/speech-recognition/run_speech_recognition_ctc_adapter.py:63-64

metadata:
{"docstring": "", "end_line": 64, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/speech-recognition/run_speech_recognition_ctc_adapter.py", "name": "list_field", "start_line": 63, "type": "function", "chroma:document": "CODE:\n\ndef list_field(default=None, metadata=None):\n\n\nSUMMARY:\n**Function Summary:** `\tlist_field`\n\n**Primary Purpose:**\nCreates and returns a Django admin field, leveraging the `field` function to initialize it with a default value and specified metadata.\n\n**Key Implementation Details:**\n\n*   Uses `default_factory=lambda: default` to set an attribute on the field instance.\n*   Passes `metadata=metadata` as a keyword argument to customize the field's metadata.\n\n**Notable Dependencies/Requirements:**\nNone explicitly stated, but it is assumed the `django.admin_fields` module or equivalent is imported."}


created_at: 2025-02-05 09:13:40
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/speech-recognition/run_speech_recognition_ctc_adapter.py:332-371

metadata:
{"docstring": "", "end_line": 371, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/speech-recognition/run_speech_recognition_ctc_adapter.py", "name": "create_vocabulary_from_data", "start_line": 332, "type": "function", "chroma:document": "CODE:\n\ndef create_vocabulary_from_data(\n    datasets: DatasetDict,\n    word_delimiter_token: Optional[str] = None,\n    unk_token: Optional[str] = None,\n    pad_token: Optional[str] = None,\n):\n    # Given training and test labels create vocabulary\n    def extract_all_chars(batch):\n        all_text = \" \".join(batch[\"target_text\"])\n        vocab = list(set(all_text))\n        return {\"vocab\": [vocab], \"all_text\": [all_text]}\n\n    vocabs = datasets.map(\n        extract_all_chars,\n        batched=True,\n        batch_size=-1,\n        keep_in_memory=True,\n        remove_columns=datasets[\"train\"].column_names,\n    )\n\n    # take union of all unique characters in each dataset\n    vocab_set = functools.reduce(\n        lambda vocab_1, vocab_2: set(vocab_1[\"vocab\"][0]) | set(vocab_2[\"vocab\"][0]), vocabs.values()\n    )\n\n    vocab_dict = {v: k for k, v in enumerate(sorted(vocab_set))}\n\n    # replace white space with delimiter token\n    if word_delimiter_token is not None:\n        vocab_dict[word_delimiter_token] = vocab_dict[\" \"]\n        del vocab_dict[\" \"]\n\n    # add unk and pad token\n    if unk_token is not None:\n        vocab_dict[unk_token] = len(vocab_dict)\n\n    if pad_token is not None:\n        vocab_dict[pad_token] = len(vocab_dict)\n\n\n\nSUMMARY:\n**Concise Summary**\n\nCreates a vocabulary dictionary (`vocab_dict`) from the provided `datasets`. Maps each batch of dataset to extract unique characters in \"target_text\" and aggregates these unions. The resulting dictionary is populated with:\n\n* Unique character set (merged across all datasets)\n* Replaced whit space with delimiter token (`word_delimiter_token`)\n* Added unk token (`unk_token`)\n* Added pad token (`pad_token`) at the end\n\n**Key Implementation Details**\n\n* Maps datasets to extract unique character sets\n* Aggregates these character sets using functools.reduce\n* Populates vocabulary dictionary with sorted, indexed unique characters\n* Handles special tokens:\n  - Replaces whit space with `word_delimiter_token` if specified\n  - Adds `unk_token` and `pad_token` to the end of the dictionary if specified\n\n**Notable Dependencies**\n\n* ` datasets`: A DatasetDict object containing training and test labels\n* External Dependency: `fun\u03c4\u03bf\u03c2.reduce`\n\nNote: This summary provides a high-level overview of the function's purpose, implementation details, and dependencies. It aims to help developers quickly understand the functionality and key aspects of this code chunk in a RAG system."}


created_at: 2025-02-05 09:13:40
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/speech-recognition/run_speech_recognition_ctc_adapter.py:374-818

metadata:
{"docstring": "", "end_line": 818, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/speech-recognition/run_speech_recognition_ctc_adapter.py", "name": "main", "start_line": 374, "type": "function", "chroma:document": "CODE:\n\ndef main():\n    # See all possible arguments in src/transformers/training_args.py\n    # or by passing the --help flag to this script.\n    # We now keep distinct sets of args, for a cleaner separation of concerns.\n\n    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TrainingArguments))\n    if len(sys.argv) == 2 and sys.argv[1].endswith(\".json\"):\n        # If we pass only one argument to the script and it's the path to a json file,\n        # let's parse it to get our arguments.\n        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))\n    else:\n        model_args, data_args, training_args = parser.parse_args_into_dataclasses()\n\n    # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The\n    # information sent is the one passed as arguments along with your Python/PyTorch versions.\n    send_example_telemetry(\"run_speech_recognition_ctc_adapter\", model_args, data_args)\n\n    # Detecting last checkpoint.\n    last_checkpoint = None\n    if os.path.isdir(training_args.output_dir) and training_args.do_train and not training_args.overwrite_output_dir:\n        last_checkpoint = get_last_checkpoint(training_args.output_dir)\n        if last_checkpoint is None and len(os.listdir(training_args.output_dir)) > 0:\n            raise ValueError(\n                f\"Output directory ({training_args.output_dir}) already exists and is not empty. \"\n                \"Use --overwrite_output_dir to overcome.\"\n            )\n        elif last_checkpoint is not None:\n            logger.info(\n                f\"Checkpoint detected, resuming training at {last_checkpoint}. To avoid this behavior, change \"\n                \"the `--output_dir` or add `--overwrite_output_dir` to train from scratch.\"\n            )\n\n    # Setup logging\n    logging.basicConfig(\n        format=\"%(asctime)s - %(levelname)s - %(name)s - %(message)s\",\n        datefmt=\"%m/%d/%Y %H:%M:%S\",\n        handlers=[logging.StreamHandler(sys.stdout)],\n    )\n    logger.setLevel(logging.INFO if is_main_process(training_args.local_rank) else logging.WARN)\n\n    # Log on each process the small summary:\n    logger.warning(\n        f\"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}, \"\n        f\"distributed training: {training_args.parallel_mode.value == 'distributed'}, 16-bits training: {training_args.fp16}\"\n    )\n    # Set the verbosity to info of the Transformers logger (on main process only):\n    if is_main_process(training_args.local_rank):\n        transformers.utils.logging.set_verbosity_info()\n    logger.info(\"Training/evaluation parameters %s\", training_args)\n\n    # Set seed before initializing model.\n    set_seed(training_args.seed)\n\n    # 1. First, let's load the dataset\n    raw_datasets = DatasetDict()\n\n    if training_args.do_train:\n        raw_datasets[\"train\"] = load_dataset(\n            data_args.dataset_name,\n            data_args.dataset_config_name,\n            split=data_args.train_split_name,\n            token=data_args.token,\n            trust_remote_code=data_args.trust_remote_code,\n        )\n\n        if data_args.audio_column_name not in raw_datasets[\"train\"].column_names:\n            raise ValueError(\n                f\"--audio_column_name '{data_args.audio_column_name}' not found in dataset '{data_args.dataset_name}'.\"\n                \" Make sure to set `--audio_column_name` to the correct audio column - one of\"\n                f\" {', '.join(raw_datasets['train'].column_names)}.\"\n            )\n\n        if data_args.text_column_name not in raw_datasets[\"train\"].column_names:\n            raise ValueError(\n                f\"--text_column_name {data_args.text_column_name} not found in dataset '{data_args.dataset_name}'. \"\n                \"Make sure to set `--text_column_name` to the correct text column - one of \"\n                f\"{', '.join(raw_datasets['train'].column_names)}.\"\n            )\n\n        if data_args.max_train_samples is not None:\n            raw_datasets[\"train\"] = raw_datasets[\"train\"].select(range(data_args.max_train_samples))\n\n    if training_args.do_eval:\n        raw_datasets[\"eval\"] = load_dataset(\n            data_args.dataset_name,\n            data_args.dataset_config_name,\n            split=data_args.eval_split_name,\n            token=data_args.token,\n            trust_remote_code=data_args.trust_remote_code,\n        )\n\n        if data_args.max_eval_samples is not None:\n            raw_datasets[\"eval\"] = raw_datasets[\"eval\"].select(range(data_args.max_eval_samples))\n\n    # 2. We remove some special characters from the datasets\n    # that make training complicated and do not help in transcribing the speech\n    # E.g. characters, such as `,` and `.` do not really have an acoustic characteristic\n    # that could be easily picked up by the model\n    chars_to_ignore_regex = (\n        f'[{\"\".join(data_args.chars_to_ignore)}]' if data_args.chars_to_ignore is not None else None\n    )\n    text_column_name = data_args.text_column_name\n\n    def remove_special_characters(batch):\n        if chars_to_ignore_regex is not None:\n            batch[\"target_text\"] = re.sub(chars_to_ignore_regex, \"\", batch[text_column_name]).lower() + \" \"\n        else:\n            batch[\"target_text\"] = batch[text_column_name].lower() + \" \"\n        return batch\n\n    with training_args.main_process_first(desc=\"dataset map special characters removal\"):\n        raw_datasets = raw_datasets.map(\n            remove_special_characters,\n            remove_columns=[text_column_name],\n            desc=\"remove special characters from datasets\",\n        )\n\n    # save special tokens for tokenizer\n    word_delimiter_token = data_args.word_delimiter_token\n    unk_token = data_args.unk_token\n    pad_token = data_args.pad_token\n\n    # 3. Next, let's load the config as we might need it to create\n    # the tokenizer\n    # load config\n    config = AutoConfig.from_pretrained(\n        model_args.model_name_or_path,\n        cache_dir=model_args.cache_dir,\n        token=data_args.token,\n        trust_remote_code=data_args.trust_remote_code,\n    )\n\n    # 4. Next, if no tokenizer file is defined,\n    # we create the vocabulary of the model by extracting all unique characters from\n    # the training and evaluation datasets\n    # We need to make sure that only first rank saves vocabulary\n    # make sure all processes wait until vocab is created\n    tokenizer_name_or_path = model_args.tokenizer_name_or_path\n    tokenizer_kwargs = {}\n\n    vocab_dict = {}\n    if tokenizer_name_or_path is not None:\n        # load vocabulary of other adapter languages so that new language can be appended\n        tokenizer = AutoTokenizer.from_pretrained(\n            tokenizer_name_or_path,\n            token=data_args.token,\n            trust_remote_code=data_args.trust_remote_code,\n        )\n        vocab_dict = tokenizer.vocab.copy()\n        if tokenizer.target_lang is None:\n            raise ValueError(\"Make sure to load a multi-lingual tokenizer with a set target language.\")\n\n        if data_args.target_language in tokenizer.vocab and not data_args.overwrite_lang_vocab:\n            logger.info(\n                \"Adapter language already exists.\"\n                \" Skipping vocabulary creating. If you want to create a new vocabulary\"\n                f\" for {data_args.target_language} make sure to add '--overwrite_lang_vocab'\"\n            )\n        else:\n            tokenizer_name_or_path = None\n\n    if tokenizer_name_or_path is None:\n        # save vocab in training output dir\n        tokenizer_name_or_path = training_args.output_dir\n\n        vocab_file = os.path.join(tokenizer_name_or_path, \"vocab.json\")\n\n        with training_args.main_process_first():\n            if training_args.overwrite_output_dir and os.path.isfile(vocab_file):\n                try:\n                    os.remove(vocab_file)\n                except OSError:\n                    # in shared file-systems it might be the case that\n                    # two processes try to delete the vocab file at the some time\n                    pass\n\n        with training_args.main_process_first(desc=\"dataset map vocabulary creation\"):\n            if not os.path.isfile(vocab_file):\n                os.makedirs(tokenizer_name_or_path, exist_ok=True)\n                lang_dict = create_vocabulary_from_data(\n                    raw_datasets,\n                    word_delimiter_token=word_delimiter_token,\n                    unk_token=unk_token,\n                    pad_token=pad_token,\n                )\n\n                # if we doing adapter language training, save\n                # vocab with adpter language\n                if data_args.target_language is not None:\n                    vocab_dict[data_args.target_language] = lang_dict\n\n                # save vocab dict to be loaded into tokenizer\n                with open(vocab_file, \"w\") as file:\n                    json.dump(vocab_dict, file)\n\n        # if tokenizer has just been created\n        # it is defined by `tokenizer_class` if present in config else by `model_type`\n        tokenizer_kwargs = {\n            \"config\": config if config.tokenizer_class is not None else None,\n            \"tokenizer_type\": config.model_type if config.tokenizer_class is None else None,\n            \"unk_token\": unk_token,\n            \"pad_token\": pad_token,\n            \"word_delimiter_token\": word_delimiter_token,\n            \"target_lang\": data_args.target_language,\n        }\n\n    # 5. Now we can instantiate the feature extractor, tokenizer and model\n    # Note for distributed training, the .from_pretrained methods guarantee that only\n    # one local process can concurrently download model & vocab.\n\n    # load feature_extractor and tokenizer\n    tokenizer = AutoTokenizer.from_pretrained(\n        tokenizer_name_or_path,\n        token=data_args.token,\n        trust_remote_code=data_args.trust_remote_code,\n        **tokenizer_kwargs,\n    )\n    feature_extractor = AutoFeatureExtractor.from_pretrained(\n        model_args.model_name_or_path,\n        cache_dir=model_args.cache_dir,\n        token=data_args.token,\n        trust_remote_code=data_args.trust_remote_code,\n    )\n\n    # adapt config\n    config.update(\n        {\n            \"final_dropout\": model_args.final_dropout,\n            \"mask_time_prob\": model_args.mask_time_prob,\n            \"mask_time_length\": model_args.mask_time_length,\n            \"mask_feature_prob\": model_args.mask_feature_prob,\n            \"mask_feature_length\": model_args.mask_feature_length,\n            \"gradient_checkpointing\": training_args.gradient_checkpointing,\n            \"layerdrop\": model_args.layerdrop,\n            \"ctc_loss_reduction\": model_args.ctc_loss_reduction,\n            \"pad_token_id\": tokenizer.pad_token_id,\n            \"vocab_size\": len(tokenizer),\n            \"adapter_attn_dim\": model_args.adapter_attn_dim,\n        }\n    )\n\n    # create model\n    model = AutoModelForCTC.from_pretrained(\n        model_args.model_name_or_path,\n        cache_dir=model_args.cache_dir,\n        config=config,\n        token=data_args.token,\n        trust_remote_code=data_args.trust_remote_code,\n        ignore_mismatched_sizes=True,\n    )\n\n    # if attn adapter is defined, freeze all non-adapter weights\n    if model.config.adapter_attn_dim is not None:\n        model.init_adapter_layers()\n        # first we freeze the whole base model\n        model.freeze_base_model()\n\n        # next we unfreeze all adapter layers\n        adapter_weights = model._get_adapters()\n        for param in adapter_weights.values():\n            param.requires_grad = True\n\n    # 6. Now we preprocess the datasets including loading the audio, resampling and normalization\n    # Thankfully, `datasets` takes care of automatically loading and resampling the audio,\n    # so that we just need to set the correct target sampling rate and normalize the input\n    # via the `feature_extractor`\n\n    # make sure that dataset decodes audio with correct sampling rate\n    dataset_sampling_rate = next(iter(raw_datasets.values())).features[data_args.audio_column_name].sampling_rate\n    if dataset_sampling_rate != feature_extractor.sampling_rate:\n        raw_datasets = raw_datasets.cast_column(\n            data_args.audio_column_name, datasets.features.Audio(sampling_rate=feature_extractor.sampling_rate)\n        )\n\n    # derive max & min input length for sample rate & max duration\n    max_input_length = data_args.max_duration_in_seconds * feature_extractor.sampling_rate\n    min_input_length = data_args.min_duration_in_seconds * feature_extractor.sampling_rate\n    audio_column_name = data_args.audio_column_name\n    num_workers = data_args.preprocessing_num_workers\n\n    # Preprocessing the datasets.\n    # We need to read the audio files as arrays and tokenize the targets.\n    def prepare_dataset(batch):\n        # load audio\n        sample = batch[audio_column_name]\n\n        inputs = feature_extractor(sample[\"array\"], sampling_rate=sample[\"sampling_rate\"])\n        batch[\"input_values\"] = inputs.input_values[0]\n        batch[\"input_length\"] = len(batch[\"input_values\"])\n\n        # encode targets\n        batch[\"labels\"] = tokenizer(batch[\"target_text\"]).input_ids\n        return batch\n\n    with training_args.main_process_first(desc=\"dataset map preprocessing\"):\n        vectorized_datasets = raw_datasets.map(\n            prepare_dataset,\n            remove_columns=next(iter(raw_datasets.values())).column_names,\n            num_proc=num_workers,\n            desc=\"preprocess datasets\",\n        )\n\n        def is_audio_in_length_range(length):\n            return length > min_input_length and length < max_input_length\n\n        # filter data that is shorter than min_input_length\n        vectorized_datasets = vectorized_datasets.filter(\n            is_audio_in_length_range,\n            num_proc=num_workers,\n            input_columns=[\"input_length\"],\n        )\n\n    # 7. Next, we can prepare the training.\n    # Let's use word error rate (WER) as our evaluation metric,\n    # instantiate a data collator and the trainer\n\n    # Define evaluation metrics during training, *i.e.* word error rate, character error rate\n    eval_metrics = {metric: evaluate.load(metric, cache_dir=model_args.cache_dir) for metric in data_args.eval_metrics}\n\n    # for large datasets it is advised to run the preprocessing on a\n    # single machine first with ``args.preprocessing_only`` since there will mostly likely\n    # be a timeout when running the script in distributed mode.\n    # In a second step ``args.preprocessing_only`` can then be set to `False` to load the\n    # cached dataset\n    if data_args.preprocessing_only:\n        logger.info(f\"Data preprocessing finished. Files cached at {vectorized_datasets.cache_files}\")\n        return\n\n    def compute_metrics(pred):\n        pred_logits = pred.predictions\n        pred_ids = np.argmax(pred_logits, axis=-1)\n\n        pred.label_ids[pred.label_ids == -100] = tokenizer.pad_token_id\n\n        pred_str = tokenizer.batch_decode(pred_ids)\n        # we do not want to group tokens when computing the metrics\n        label_str = tokenizer.batch_decode(pred.label_ids, group_tokens=False)\n\n        metrics = {k: v.compute(predictions=pred_str, references=label_str) for k, v in eval_metrics.items()}\n\n        return metrics\n\n    # Now save everything to be able to create a single processor later\n    # make sure all processes wait until data is saved\n    with training_args.main_process_first():\n        # only the main process saves them\n        if is_main_process(training_args.local_rank):\n            # save feature extractor, tokenizer and config\n            feature_extractor.save_pretrained(training_args.output_dir)\n            tokenizer.save_pretrained(training_args.output_dir)\n            config.save_pretrained(training_args.output_dir)\n\n    try:\n        processor = AutoProcessor.from_pretrained(training_args.output_dir)\n    except (OSError, KeyError):\n        warnings.warn(\n            \"Loading a processor from a feature extractor config that does not\"\n            \" include a `processor_class` attribute is deprecated and will be removed in v5. Please add the following \"\n            \" attribute to your `preprocessor_config.json` file to suppress this warning: \"\n            \" `'processor_class': 'Wav2Vec2Processor'`\",\n            FutureWarning,\n        )\n        processor = Wav2Vec2Processor.from_pretrained(training_args.output_dir)\n\n    # Instantiate custom data collator\n    data_collator = DataCollatorCTCWithPadding(processor=processor)\n\n    # Initialize Trainer\n    trainer = Trainer(\n        model=model,\n        data_collator=data_collator,\n        args=training_args,\n        compute_metrics=compute_metrics,\n        train_dataset=vectorized_datasets[\"train\"] if training_args.do_train else None,\n        eval_dataset=vectorized_datasets[\"eval\"] if training_args.do_eval else None,\n        processing_class=processor,\n    )\n\n    # 8. Finally, we can start training\n\n    # Training\n    if training_args.do_train:\n        # use last checkpoint if exist\n        if last_checkpoint is not None:\n            checkpoint = last_checkpoint\n        elif os.path.isdir(model_args.model_name_or_path):\n            checkpoint = model_args.model_name_or_path\n        else:\n            checkpoint = None\n\n        train_result = trainer.train(resume_from_checkpoint=checkpoint)\n        trainer.save_model()\n\n        metrics = train_result.metrics\n        max_train_samples = (\n            data_args.max_train_samples\n            if data_args.max_train_samples is not None\n            else len(vectorized_datasets[\"train\"])\n        )\n        metrics[\"train_samples\"] = min(max_train_samples, len(vectorized_datasets[\"train\"]))\n\n        trainer.log_metrics(\"train\", metrics)\n        trainer.save_metrics(\"train\", metrics)\n        trainer.save_state()\n\n    # Evaluation\n    results = {}\n    if training_args.do_eval:\n        logger.info(\"*** Evaluate ***\")\n        metrics = trainer.evaluate()\n        max_eval_samples = (\n            data_args.max_eval_samples if data_args.max_eval_samples is not None else len(vectorized_datasets[\"eval\"])\n        )\n        metrics[\"eval_samples\"] = min(max_eval_samples, len(vectorized_datasets[\"eval\"]))\n\n        trainer.log_metrics(\"eval\", metrics)\n        trainer.save_metrics(\"eval\", metrics)\n\n    # Write model card and (optionally) push to hub\n    config_name = data_args.dataset_config_name if data_args.dataset_config_name is not None else \"na\"\n    kwargs = {\n        \"finetuned_from\": model_args.model_name_or_path,\n        \"tasks\": \"automatic-speech-recognition\",\n        \"tags\": [\"automatic-speech-recognition\", data_args.dataset_name, \"mms\"],\n        \"dataset_args\": (\n            f\"Config: {config_name}, Training split: {data_args.train_split_name}, Eval split:\"\n            f\" {data_args.eval_split_name}\"\n        ),\n        \"dataset\": f\"{data_args.dataset_name.upper()} - {config_name.upper()}\",\n    }\n    if \"common_voice\" in data_args.dataset_name:\n        kwargs[\"language\"] = config_name\n\n    # make sure that adapter weights are saved seperately\n    adapter_file = WAV2VEC2_ADAPTER_SAFE_FILE.format(data_args.target_language)\n    adapter_file = os.path.join(training_args.output_dir, adapter_file)\n    logger.info(f\"Saving adapter weights under {adapter_file}...\")\n    safe_save_file(model._get_adapters(), adapter_file, metadata={\"format\": \"pt\"})\n\n    if training_args.push_to_hub:\n        trainer.push_to_hub(**kwargs)\n    else:\n        trainer.create_model_card(**kwargs)\n\n\n\nSUMMARY:\nHere is a concise summary of the code:\n\nThis is a script for training and evaluating a Wav2Vec 2 model using the Hugging Face Transformers library. The script performs the following tasks:\n\n1. Loads the dataset, tokenizer, and processor configurations\n2. Creates a Trainer object with customizable arguments (e.g., data collator, model name)\n3. Trains the model on the training dataset if `do_train` is set to True\n4. Evaluates the model on the evaluation dataset if `do_eval` is set to True\n5. Saves the trained model and metrics\n6. Optionally pushes the trained model to Hugging Face Hub\n7. Creates a model card with metadata (e.g., dataset name, language)\n\nThe script uses various libraries and modules from Hugging Face and PyTorch, including `Trainer`, `DataCollatorCTCWithPadding`, `Processor`, and `Transformers`."}


created_at: 2025-02-05 09:13:40
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/speech-recognition/run_speech_recognition_seq2seq.py:276-629

metadata:
{"docstring": "", "end_line": 629, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/speech-recognition/run_speech_recognition_seq2seq.py", "name": "main", "start_line": 276, "type": "function", "chroma:document": "CODE:\n\ndef main():\n    # 1. Parse input arguments\n    # See all possible arguments in src/transformers/training_args.py\n    # or by passing the --help flag to this script.\n    # We now keep distinct sets of args, for a cleaner separation of concerns.\n    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, Seq2SeqTrainingArguments))\n\n    if len(sys.argv) == 2 and sys.argv[1].endswith(\".json\"):\n        # If we pass only one argument to the script and it's the path to a json file,\n        # let's parse it to get our arguments.\n        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))\n    else:\n        model_args, data_args, training_args = parser.parse_args_into_dataclasses()\n\n    # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The\n    # information sent is the one passed as arguments along with your Python/PyTorch versions.\n    send_example_telemetry(\"run_speech_recognition_seq2seq\", model_args, data_args)\n\n    # 2. Setup logging\n    logging.basicConfig(\n        format=\"%(asctime)s - %(levelname)s - %(name)s - %(message)s\",\n        datefmt=\"%m/%d/%Y %H:%M:%S\",\n        handlers=[logging.StreamHandler(sys.stdout)],\n    )\n    log_level = training_args.get_process_log_level()\n    logger.setLevel(log_level)\n    datasets.utils.logging.set_verbosity(log_level)\n    transformers.utils.logging.set_verbosity(log_level)\n    transformers.utils.logging.enable_default_handler()\n    transformers.utils.logging.enable_explicit_format()\n\n    logger.setLevel(logging.INFO if is_main_process(training_args.local_rank) else logging.WARN)\n\n    # Log on each process the small summary:\n    logger.warning(\n        f\"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}, \"\n        f\"distributed training: {training_args.parallel_mode.value == 'distributed'}, 16-bits training: {training_args.fp16}\"\n    )\n    logger.info(f\"Training/evaluation parameters {training_args}\")\n\n    # Set the verbosity to info of the Transformers logger (on main process only):\n    if is_main_process(training_args.local_rank):\n        transformers.utils.logging.set_verbosity_info()\n    logger.info(\"Training/evaluation parameters %s\", training_args)\n\n    # 3. Detecting last checkpoint and eventually continue from last checkpoint\n    last_checkpoint = None\n    if os.path.isdir(training_args.output_dir) and training_args.do_train and not training_args.overwrite_output_dir:\n        last_checkpoint = get_last_checkpoint(training_args.output_dir)\n        if last_checkpoint is None and len(os.listdir(training_args.output_dir)) > 0:\n            raise ValueError(\n                f\"Output directory ({training_args.output_dir}) already exists and is not empty. \"\n                \"Use --overwrite_output_dir to overcome.\"\n            )\n        elif last_checkpoint is not None and training_args.resume_from_checkpoint is None:\n            logger.info(\n                f\"Checkpoint detected, resuming training at {last_checkpoint}. To avoid this behavior, change \"\n                \"the `--output_dir` or add `--overwrite_output_dir` to train from scratch.\"\n            )\n\n    # Set seed before initializing model.\n    set_seed(training_args.seed)\n\n    # 4. Load dataset\n    raw_datasets = DatasetDict()\n\n    if training_args.do_train:\n        raw_datasets[\"train\"] = load_dataset(\n            data_args.dataset_name,\n            data_args.dataset_config_name,\n            split=data_args.train_split_name,\n            cache_dir=model_args.cache_dir,\n            token=model_args.token,\n            trust_remote_code=model_args.trust_remote_code,\n        )\n\n    if training_args.do_eval:\n        raw_datasets[\"eval\"] = load_dataset(\n            data_args.dataset_name,\n            data_args.dataset_config_name,\n            split=data_args.eval_split_name,\n            cache_dir=model_args.cache_dir,\n            token=model_args.token,\n            trust_remote_code=model_args.trust_remote_code,\n        )\n\n    if data_args.audio_column_name not in next(iter(raw_datasets.values())).column_names:\n        raise ValueError(\n            f\"--audio_column_name '{data_args.audio_column_name}' not found in dataset '{data_args.dataset_name}'. \"\n            \"Make sure to set `--audio_column_name` to the correct audio column - one of \"\n            f\"{', '.join(next(iter(raw_datasets.values())).column_names)}.\"\n        )\n\n    if data_args.text_column_name not in next(iter(raw_datasets.values())).column_names:\n        raise ValueError(\n            f\"--text_column_name {data_args.text_column_name} not found in dataset '{data_args.dataset_name}'. \"\n            \"Make sure to set `--text_column_name` to the correct text column - one of \"\n            f\"{', '.join(next(iter(raw_datasets.values())).column_names)}.\"\n        )\n\n    # 5. Load pretrained model, tokenizer, and feature extractor\n    #\n    # Distributed training:\n    # The .from_pretrained methods guarantee that only one local process can concurrently\n    config = AutoConfig.from_pretrained(\n        model_args.config_name if model_args.config_name else model_args.model_name_or_path,\n        cache_dir=model_args.cache_dir,\n        revision=model_args.model_revision,\n        token=model_args.token,\n        trust_remote_code=model_args.trust_remote_code,\n    )\n\n    # SpecAugment for whisper models\n    if getattr(config, \"model_type\", None) == \"whisper\":\n        config.update({\"apply_spec_augment\": model_args.apply_spec_augment})\n\n    feature_extractor = AutoFeatureExtractor.from_pretrained(\n        model_args.feature_extractor_name if model_args.feature_extractor_name else model_args.model_name_or_path,\n        cache_dir=model_args.cache_dir,\n        revision=model_args.model_revision,\n        token=model_args.token,\n        trust_remote_code=model_args.trust_remote_code,\n    )\n    tokenizer = AutoTokenizer.from_pretrained(\n        model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path,\n        cache_dir=model_args.cache_dir,\n        use_fast=model_args.use_fast_tokenizer,\n        revision=model_args.model_revision,\n        token=model_args.token,\n        trust_remote_code=model_args.trust_remote_code,\n    )\n    model = AutoModelForSpeechSeq2Seq.from_pretrained(\n        model_args.model_name_or_path,\n        config=config,\n        cache_dir=model_args.cache_dir,\n        revision=model_args.model_revision,\n        token=model_args.token,\n        trust_remote_code=model_args.trust_remote_code,\n    )\n\n    if model.config.decoder_start_token_id is None:\n        raise ValueError(\"Make sure that `config.decoder_start_token_id` is correctly defined\")\n\n    if model_args.freeze_feature_encoder:\n        model.freeze_feature_encoder()\n\n    if model_args.freeze_encoder:\n        model.freeze_encoder()\n        model.model.encoder.gradient_checkpointing = False\n\n    if hasattr(model.generation_config, \"is_multilingual\") and model.generation_config.is_multilingual:\n        # We only need to set the language and task ids in a multilingual setting\n        tokenizer.set_prefix_tokens(language=data_args.language, task=data_args.task)\n        model.generation_config.language = data_args.language\n        model.generation_config.task = data_args.task\n    elif data_args.language is not None:\n        raise ValueError(\n            \"Setting language token for an English-only checkpoint is not permitted. The language argument should \"\n            \"only be set for multilingual checkpoints.\"\n        )\n\n    # TODO (Sanchit): deprecate these arguments in v4.41\n    if model_args.forced_decoder_ids is not None:\n        logger.warning(\n            \"The use of `forced_decoder_ids` is deprecated and will be removed in v4.41.\"\n            \"Please use the `language` and `task` arguments instead\"\n        )\n        model.generation_config.forced_decoder_ids = model_args.forced_decoder_ids\n    else:\n        model.generation_config.forced_decoder_ids = None\n        model.config.forced_decoder_ids = None\n\n    if model_args.suppress_tokens is not None:\n        logger.warning(\n            \"The use of `suppress_tokens` is deprecated and will be removed in v4.41.\"\n            \"Should you need `suppress_tokens`, please manually set them in the fine-tuning script.\"\n        )\n        model.generation_config.suppress_tokens = model_args.suppress_tokens\n\n    # 6. Resample speech dataset if necessary\n    dataset_sampling_rate = next(iter(raw_datasets.values())).features[data_args.audio_column_name].sampling_rate\n    if dataset_sampling_rate != feature_extractor.sampling_rate:\n        raw_datasets = raw_datasets.cast_column(\n            data_args.audio_column_name, datasets.features.Audio(sampling_rate=feature_extractor.sampling_rate)\n        )\n\n    # 7. Preprocessing the datasets.\n    # We need to read the audio files as arrays and tokenize the targets.\n    max_input_length = data_args.max_duration_in_seconds * feature_extractor.sampling_rate\n    min_input_length = data_args.min_duration_in_seconds * feature_extractor.sampling_rate\n    audio_column_name = data_args.audio_column_name\n    num_workers = data_args.preprocessing_num_workers\n    text_column_name = data_args.text_column_name\n    model_input_name = feature_extractor.model_input_names[0]\n    do_lower_case = data_args.do_lower_case\n    # if SpecAugment is used for whisper models, return attention_mask to guide the mask along time axis\n    forward_attention_mask = (\n        getattr(config, \"model_type\", None) == \"whisper\"\n        and getattr(config, \"apply_spec_augment\", False)\n        and getattr(config, \"mask_time_prob\", 0) > 0\n    )\n\n    if data_args.max_train_samples is not None:\n        raw_datasets[\"train\"] = raw_datasets[\"train\"].select(range(data_args.max_train_samples))\n\n    if data_args.max_eval_samples is not None:\n        raw_datasets[\"eval\"] = raw_datasets[\"eval\"].select(range(data_args.max_eval_samples))\n\n    def prepare_dataset(batch):\n        # process audio\n        sample = batch[audio_column_name]\n        inputs = feature_extractor(\n            sample[\"array\"], sampling_rate=sample[\"sampling_rate\"], return_attention_mask=forward_attention_mask\n        )\n        # process audio length\n        batch[model_input_name] = inputs.get(model_input_name)[0]\n        batch[\"input_length\"] = len(sample[\"array\"])\n        if forward_attention_mask:\n            batch[\"attention_mask\"] = inputs.get(\"attention_mask\")[0]\n\n        # process targets\n        input_str = batch[text_column_name].lower() if do_lower_case else batch[text_column_name]\n        batch[\"labels\"] = tokenizer(input_str).input_ids\n        return batch\n\n    with training_args.main_process_first(desc=\"dataset map pre-processing\"):\n        vectorized_datasets = raw_datasets.map(\n            prepare_dataset,\n            remove_columns=next(iter(raw_datasets.values())).column_names,\n            num_proc=data_args.preprocessing_num_workers,\n            desc=\"preprocess train dataset\",\n        )\n\n    # filter data that is shorter than min_input_length or longer than\n    # max_input_length\n    def is_audio_in_length_range(length):\n        return length > min_input_length and length < max_input_length\n\n    vectorized_datasets = vectorized_datasets.filter(\n        is_audio_in_length_range,\n        num_proc=num_workers,\n        input_columns=[\"input_length\"],\n    )\n\n    # for large datasets it is advised to run the preprocessing on a\n    # single machine first with `args.preprocessing_only` since there will mostly likely\n    # be a timeout when running the script in distributed mode.\n    # In a second step `args.preprocessing_only` can then be set to `False` to load the\n    # cached dataset\n    if data_args.preprocessing_only:\n        cache = {k: v.cache_files for k, v in vectorized_datasets.items()}\n        logger.info(f\"Data preprocessing finished. Files cached at {cache}.\")\n        return\n\n    # 8. Load Metric\n    metric = evaluate.load(\"wer\", cache_dir=model_args.cache_dir)\n\n    def compute_metrics(pred):\n        pred_ids = pred.predictions\n\n        pred.label_ids[pred.label_ids == -100] = tokenizer.pad_token_id\n\n        pred_str = tokenizer.batch_decode(pred_ids, skip_special_tokens=True)\n        # we do not want to group tokens when computing the metrics\n        label_str = tokenizer.batch_decode(pred.label_ids, skip_special_tokens=True)\n\n        wer = metric.compute(predictions=pred_str, references=label_str)\n\n        return {\"wer\": wer}\n\n    # 9. Create a single speech processor\n    # make sure all processes wait until data is saved\n    with training_args.main_process_first():\n        # only the main process saves them\n        if is_main_process(training_args.local_rank):\n            # save feature extractor, tokenizer and config\n            feature_extractor.save_pretrained(training_args.output_dir)\n            tokenizer.save_pretrained(training_args.output_dir)\n            config.save_pretrained(training_args.output_dir)\n\n    processor = AutoProcessor.from_pretrained(training_args.output_dir)\n\n    # 10. Define data collator\n    data_collator = DataCollatorSpeechSeq2SeqWithPadding(\n        processor=processor,\n        decoder_start_token_id=model.config.decoder_start_token_id,\n        forward_attention_mask=forward_attention_mask,\n    )\n\n    # 11. Initialize Trainer\n    trainer = Seq2SeqTrainer(\n        model=model,\n        args=training_args,\n        train_dataset=vectorized_datasets[\"train\"] if training_args.do_train else None,\n        eval_dataset=vectorized_datasets[\"eval\"] if training_args.do_eval else None,\n        processing_class=feature_extractor,\n        data_collator=data_collator,\n        compute_metrics=compute_metrics if training_args.predict_with_generate else None,\n    )\n\n    # 12. Training\n    if training_args.do_train:\n        checkpoint = None\n        if training_args.resume_from_checkpoint is not None:\n            checkpoint = training_args.resume_from_checkpoint\n        elif last_checkpoint is not None:\n            checkpoint = last_checkpoint\n        train_result = trainer.train(resume_from_checkpoint=checkpoint)\n        trainer.save_model()  # Saves the feature extractor too for easy upload\n\n        metrics = train_result.metrics\n        max_train_samples = (\n            data_args.max_train_samples\n            if data_args.max_train_samples is not None\n            else len(vectorized_datasets[\"train\"])\n        )\n        metrics[\"train_samples\"] = min(max_train_samples, len(vectorized_datasets[\"train\"]))\n        trainer.log_metrics(\"train\", metrics)\n        trainer.save_metrics(\"train\", metrics)\n        trainer.save_state()\n\n    # 13. Evaluation\n    results = {}\n    if training_args.do_eval:\n        logger.info(\"*** Evaluate ***\")\n        metrics = trainer.evaluate(\n            metric_key_prefix=\"eval\",\n            max_length=training_args.generation_max_length,\n            num_beams=training_args.generation_num_beams,\n        )\n        max_eval_samples = (\n            data_args.max_eval_samples if data_args.max_eval_samples is not None else len(vectorized_datasets[\"eval\"])\n        )\n        metrics[\"eval_samples\"] = min(max_eval_samples, len(vectorized_datasets[\"eval\"]))\n\n        trainer.log_metrics(\"eval\", metrics)\n        trainer.save_metrics(\"eval\", metrics)\n\n    # 14. Write Training Stats\n    kwargs = {\"finetuned_from\": model_args.model_name_or_path, \"tasks\": \"automatic-speech-recognition\"}\n    if data_args.dataset_name is not None:\n        kwargs[\"dataset_tags\"] = data_args.dataset_name\n        if data_args.dataset_config_name is not None:\n            kwargs[\"dataset_args\"] = data_args.dataset_config_name\n            kwargs[\"dataset\"] = f\"{data_args.dataset_name} {data_args.dataset_config_name}\"\n        else:\n            kwargs[\"dataset\"] = data_args.dataset_name\n\n    if training_args.push_to_hub:\n        trainer.push_to_hub(**kwargs)\n    else:\n        trainer.create_model_card(**kwargs)\n\n\n\nSUMMARY:\nThis is the code for a speech recognition model, specifically an automatic speech recognition (ASR) system. The code is written in Python and uses the Hugging Face Transformers library.\n\nThe code consists of three main parts:\n\n1. **Data Preparation**: This section includes data loading, preprocessing, and loading a metric (WER, Word Error Rate) for evaluation.\n2. **Model Training**: This section trains the speech recognition model using the `Seq2SeqTrainer` class from Hugging Face's Transformers library. The training process involves:\n\t* Loading pre-trained models and tokens\n\t* Defining the custom data collator and processor\n\t* Initializing the trainer\n\t* Triggering the training loop\n3. **Evaluation**: This section runs the evaluation of the trained model on a test dataset using the previously loaded metric.\n4. **Model Saving and Storage**: The code saves the trained model, tokenizer, and configuration to a specified output directory.\n\nKey parameters and hyperparameters used in this code include:\n\n* `model_args`: training arguments that include input data, output directory, etc.\n* `training_args`: trainer-specific arguments that control the training process, such as device acceleration, gradient accumulation, etc.\n* `data_args`: dataset-related arguments, including dataset name, config name, and other metadata.\n\nKey libraries used in this code include:\n\n* Hugging Face's Transformers library for natural language processing tasks\n* PyTorch for deep learning and neural network operations\n\nOverall, this code is designed to train a speech recognition model on a specific dataset (e.g., LibriSpeech) using a combination of pre-trained models and custom data preprocessing. The resulting trained model can then be used for real-world speech recognition applications."}


created_at: 2025-02-05 09:13:40
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py:60-61

metadata:
{"docstring": "", "end_line": 61, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py", "name": "list_field", "start_line": 60, "type": "function", "chroma:document": "CODE:\n\ndef list_field(default=None, metadata=None):\n\n\nSUMMARY:\n**Summary**\n\nThis is a Python function that creates a database model field with optional `default` and `metadata`. It uses Django's `field` function to create the field.\n\n**Key Details**\n\n* Primary purpose: Create a database model field\n* Implementation details:\n\t+ Uses Django's `field` function\n\t+ Sets a default value using a lambda function (`default_factory=lambda: default`)\n\t+ Passes metadata to the created field (`metadata=metadata`)\n* Notable dependencies:\n\t+ Django (the `field` function is part of its ORM)\n\n**RAG System Classification**\n\n* Tag: `ORM`\n* Category: `Model Field Creation`\n* Priority: H\n* Severity: M"}


created_at: 2025-02-05 09:13:40
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py:352-391

metadata:
{"docstring": "", "end_line": 391, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py", "name": "create_vocabulary_from_data", "start_line": 352, "type": "function", "chroma:document": "CODE:\n\ndef create_vocabulary_from_data(\n    datasets: DatasetDict,\n    word_delimiter_token: Optional[str] = None,\n    unk_token: Optional[str] = None,\n    pad_token: Optional[str] = None,\n):\n    # Given training and test labels create vocabulary\n    def extract_all_chars(batch):\n        all_text = \" \".join(batch[\"target_text\"])\n        vocab = list(set(all_text))\n        return {\"vocab\": [vocab], \"all_text\": [all_text]}\n\n    vocabs = datasets.map(\n        extract_all_chars,\n        batched=True,\n        batch_size=-1,\n        keep_in_memory=True,\n        remove_columns=datasets[\"train\"].column_names,\n    )\n\n    # take union of all unique characters in each dataset\n    vocab_set = functools.reduce(\n        lambda vocab_1, vocab_2: set(vocab_1[\"vocab\"][0]) | set(vocab_2[\"vocab\"][0]), vocabs.values()\n    )\n\n    vocab_dict = {v: k for k, v in enumerate(sorted(vocab_set))}\n\n    # replace white space with delimiter token\n    if word_delimiter_token is not None:\n        vocab_dict[word_delimiter_token] = vocab_dict[\" \"]\n        del vocab_dict[\" \"]\n\n    # add unk and pad token\n    if unk_token is not None:\n        vocab_dict[unk_token] = len(vocab_dict)\n\n    if pad_token is not None:\n        vocab_dict[pad_token] = len(vocab_dict)\n\n\n\nSUMMARY:\nHere's a concise summary of the code:\n\n**Function Purpose:** Creates a vocabulary dictionary from a collection of datasets, allowing for flexible tokenization and padding.\n\n**Main Functionality:**\n\n1. Extracts unique characters from training and test data using `extract_all_chars` function.\n2. Applies filter to remove irrelevant columns from dataset.\n3. Combines character sets into a single set using `functools.reduce`.\n4. Replaces white space with delimiter token if specified, removes empty entry.\n5. Assigns special tokens (UNK and PAD) values based on usage.\n\n**Key Insights:**\n\n* Utilizes `DatasetDict` as input data source.\n* Incorporates string manipulation (`join`, `set`) for vocabulary generation.\n* Introduces custom token mapping through conditional statements.\n\n**Notable Dependencies/Requirements:**\n\n* `DatasetDict`\n* `functools.reduce`\n* String manipulation libraries (e.g., `str.join`)"}


created_at: 2025-02-05 09:13:40
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py:394-817

metadata:
{"docstring": "", "end_line": 817, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py", "name": "main", "start_line": 394, "type": "function", "chroma:document": "CODE:\n\ndef main():\n    # See all possible arguments in src/transformers/training_args.py\n    # or by passing the --help flag to this script.\n    # We now keep distinct sets of args, for a cleaner separation of concerns.\n\n    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TrainingArguments))\n    if len(sys.argv) == 2 and sys.argv[1].endswith(\".json\"):\n        # If we pass only one argument to the script and it's the path to a json file,\n        # let's parse it to get our arguments.\n        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))\n    else:\n        model_args, data_args, training_args = parser.parse_args_into_dataclasses()\n\n    # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The\n    # information sent is the one passed as arguments along with your Python/PyTorch versions.\n    send_example_telemetry(\"run_speech_recognition_ctc\", model_args, data_args)\n\n    # Detecting last checkpoint.\n    last_checkpoint = None\n    if os.path.isdir(training_args.output_dir) and training_args.do_train and not training_args.overwrite_output_dir:\n        last_checkpoint = get_last_checkpoint(training_args.output_dir)\n        if last_checkpoint is None and len(os.listdir(training_args.output_dir)) > 0:\n            raise ValueError(\n                f\"Output directory ({training_args.output_dir}) already exists and is not empty. \"\n                \"Use --overwrite_output_dir to overcome.\"\n            )\n        elif last_checkpoint is not None:\n            logger.info(\n                f\"Checkpoint detected, resuming training at {last_checkpoint}. To avoid this behavior, change \"\n                \"the `--output_dir` or add `--overwrite_output_dir` to train from scratch.\"\n            )\n\n    # Setup logging\n    logging.basicConfig(\n        format=\"%(asctime)s - %(levelname)s - %(name)s - %(message)s\",\n        datefmt=\"%m/%d/%Y %H:%M:%S\",\n        handlers=[logging.StreamHandler(sys.stdout)],\n    )\n    logger.setLevel(logging.INFO if is_main_process(training_args.local_rank) else logging.WARN)\n\n    # Log on each process the small summary:\n    logger.warning(\n        f\"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}, \"\n        f\"distributed training: {training_args.parallel_mode.value == 'distributed'}, 16-bits training: {training_args.fp16}\"\n    )\n    # Set the verbosity to info of the Transformers logger (on main process only):\n    if is_main_process(training_args.local_rank):\n        transformers.utils.logging.set_verbosity_info()\n    logger.info(\"Training/evaluation parameters %s\", training_args)\n\n    # Set seed before initializing model.\n    set_seed(training_args.seed)\n\n    # 1. First, let's load the dataset\n    raw_datasets = DatasetDict()\n\n    if training_args.do_train:\n        raw_datasets[\"train\"] = load_dataset(\n            data_args.dataset_name,\n            data_args.dataset_config_name,\n            split=data_args.train_split_name,\n            token=data_args.token,\n            trust_remote_code=data_args.trust_remote_code,\n        )\n\n        if data_args.audio_column_name not in raw_datasets[\"train\"].column_names:\n            raise ValueError(\n                f\"--audio_column_name '{data_args.audio_column_name}' not found in dataset '{data_args.dataset_name}'.\"\n                \" Make sure to set `--audio_column_name` to the correct audio column - one of\"\n                f\" {', '.join(raw_datasets['train'].column_names)}.\"\n            )\n\n        if data_args.text_column_name not in raw_datasets[\"train\"].column_names:\n            raise ValueError(\n                f\"--text_column_name {data_args.text_column_name} not found in dataset '{data_args.dataset_name}'. \"\n                \"Make sure to set `--text_column_name` to the correct text column - one of \"\n                f\"{', '.join(raw_datasets['train'].column_names)}.\"\n            )\n\n        if data_args.max_train_samples is not None:\n            raw_datasets[\"train\"] = raw_datasets[\"train\"].select(range(data_args.max_train_samples))\n\n    if training_args.do_eval:\n        raw_datasets[\"eval\"] = load_dataset(\n            data_args.dataset_name,\n            data_args.dataset_config_name,\n            split=data_args.eval_split_name,\n            token=data_args.token,\n            trust_remote_code=data_args.trust_remote_code,\n        )\n\n        if data_args.max_eval_samples is not None:\n            raw_datasets[\"eval\"] = raw_datasets[\"eval\"].select(range(data_args.max_eval_samples))\n\n    # 2. We remove some special characters from the datasets\n    # that make training complicated and do not help in transcribing the speech\n    # E.g. characters, such as `,` and `.` do not really have an acoustic characteristic\n    # that could be easily picked up by the model\n    chars_to_ignore_regex = (\n        f'[{\"\".join(data_args.chars_to_ignore)}]' if data_args.chars_to_ignore is not None else None\n    )\n    text_column_name = data_args.text_column_name\n\n    def remove_special_characters(batch):\n        if chars_to_ignore_regex is not None:\n            batch[\"target_text\"] = re.sub(chars_to_ignore_regex, \"\", batch[text_column_name]).lower() + \" \"\n        else:\n            batch[\"target_text\"] = batch[text_column_name].lower() + \" \"\n        return batch\n\n    with training_args.main_process_first(desc=\"dataset map special characters removal\"):\n        raw_datasets = raw_datasets.map(\n            remove_special_characters,\n            remove_columns=[text_column_name],\n            desc=\"remove special characters from datasets\",\n        )\n\n    # save special tokens for tokenizer\n    word_delimiter_token = data_args.word_delimiter_token\n    unk_token = data_args.unk_token\n    pad_token = data_args.pad_token\n\n    # 3. Next, let's load the config as we might need it to create\n    # the tokenizer\n    # load config\n    config = AutoConfig.from_pretrained(\n        model_args.model_name_or_path,\n        cache_dir=model_args.cache_dir,\n        token=data_args.token,\n        trust_remote_code=data_args.trust_remote_code,\n    )\n\n    # 4. Next, if no tokenizer file is defined,\n    # we create the vocabulary of the model by extracting all unique characters from\n    # the training and evaluation datasets\n    # We need to make sure that only first rank saves vocabulary\n    # make sure all processes wait until vocab is created\n    tokenizer_name_or_path = model_args.tokenizer_name_or_path\n    tokenizer_kwargs = {}\n    if tokenizer_name_or_path is None:\n        # save vocab in training output dir\n        tokenizer_name_or_path = training_args.output_dir\n\n        vocab_file = os.path.join(tokenizer_name_or_path, \"vocab.json\")\n\n        with training_args.main_process_first():\n            if training_args.overwrite_output_dir and os.path.isfile(vocab_file):\n                try:\n                    os.remove(vocab_file)\n                except OSError:\n                    # in shared file-systems it might be the case that\n                    # two processes try to delete the vocab file at the some time\n                    pass\n\n        with training_args.main_process_first(desc=\"dataset map vocabulary creation\"):\n            if not os.path.isfile(vocab_file):\n                os.makedirs(tokenizer_name_or_path, exist_ok=True)\n                vocab_dict = create_vocabulary_from_data(\n                    raw_datasets,\n                    word_delimiter_token=word_delimiter_token,\n                    unk_token=unk_token,\n                    pad_token=pad_token,\n                )\n\n                # save vocab dict to be loaded into tokenizer\n                with open(vocab_file, \"w\") as file:\n                    json.dump(vocab_dict, file)\n\n        # if tokenizer has just been created\n        # it is defined by `tokenizer_class` if present in config else by `model_type`\n        tokenizer_kwargs = {\n            \"config\": config if config.tokenizer_class is not None else None,\n            \"tokenizer_type\": config.model_type if config.tokenizer_class is None else None,\n            \"unk_token\": unk_token,\n            \"pad_token\": pad_token,\n            \"word_delimiter_token\": word_delimiter_token,\n        }\n\n    # 5. Now we can instantiate the feature extractor, tokenizer and model\n    # Note for distributed training, the .from_pretrained methods guarantee that only\n    # one local process can concurrently download model & vocab.\n\n    # load feature_extractor and tokenizer\n    tokenizer = AutoTokenizer.from_pretrained(\n        tokenizer_name_or_path,\n        token=data_args.token,\n        trust_remote_code=data_args.trust_remote_code,\n        **tokenizer_kwargs,\n    )\n    feature_extractor = AutoFeatureExtractor.from_pretrained(\n        model_args.model_name_or_path,\n        cache_dir=model_args.cache_dir,\n        token=data_args.token,\n        trust_remote_code=data_args.trust_remote_code,\n    )\n\n    # adapt config\n    config.update(\n        {\n            \"feat_proj_dropout\": model_args.feat_proj_dropout,\n            \"attention_dropout\": model_args.attention_dropout,\n            \"hidden_dropout\": model_args.hidden_dropout,\n            \"final_dropout\": model_args.final_dropout,\n            \"mask_time_prob\": model_args.mask_time_prob,\n            \"mask_time_length\": model_args.mask_time_length,\n            \"mask_feature_prob\": model_args.mask_feature_prob,\n            \"mask_feature_length\": model_args.mask_feature_length,\n            \"gradient_checkpointing\": training_args.gradient_checkpointing,\n            \"layerdrop\": model_args.layerdrop,\n            \"ctc_loss_reduction\": model_args.ctc_loss_reduction,\n            \"ctc_zero_infinity\": model_args.ctc_zero_infinity,\n            \"pad_token_id\": tokenizer.pad_token_id,\n            \"vocab_size\": len(tokenizer),\n            \"activation_dropout\": model_args.activation_dropout,\n            \"add_adapter\": model_args.add_adapter,\n        }\n    )\n\n    # create model\n    model = AutoModelForCTC.from_pretrained(\n        model_args.model_name_or_path,\n        cache_dir=model_args.cache_dir,\n        config=config,\n        token=data_args.token,\n        trust_remote_code=data_args.trust_remote_code,\n    )\n\n    # freeze encoder\n    if model_args.freeze_feature_encoder:\n        model.freeze_feature_encoder()\n\n    # 6. Now we preprocess the datasets including loading the audio, resampling and normalization\n    # Thankfully, `datasets` takes care of automatically loading and resampling the audio,\n    # so that we just need to set the correct target sampling rate and normalize the input\n    # via the `feature_extractor`\n\n    # make sure that dataset decodes audio with correct sampling rate\n    dataset_sampling_rate = next(iter(raw_datasets.values())).features[data_args.audio_column_name].sampling_rate\n    if dataset_sampling_rate != feature_extractor.sampling_rate:\n        raw_datasets = raw_datasets.cast_column(\n            data_args.audio_column_name, datasets.features.Audio(sampling_rate=feature_extractor.sampling_rate)\n        )\n\n    # derive max & min input length for sample rate & max duration\n    max_input_length = data_args.max_duration_in_seconds * feature_extractor.sampling_rate\n    min_input_length = data_args.min_duration_in_seconds * feature_extractor.sampling_rate\n    audio_column_name = data_args.audio_column_name\n    num_workers = data_args.preprocessing_num_workers\n    feature_extractor_input_name = feature_extractor.model_input_names[0]\n\n    # `phoneme_language` is only relevant if the model is fine-tuned on phoneme classification\n    phoneme_language = data_args.phoneme_language\n\n    # Preprocessing the datasets.\n    # We need to read the audio files as arrays and tokenize the targets.\n    def prepare_dataset(batch):\n        # load audio\n        sample = batch[audio_column_name]\n\n        inputs = feature_extractor(sample[\"array\"], sampling_rate=sample[\"sampling_rate\"])\n        batch[feature_extractor_input_name] = getattr(inputs, feature_extractor_input_name)[0]\n        # take length of raw audio waveform\n        batch[\"input_length\"] = len(sample[\"array\"].squeeze())\n\n        # encode targets\n        additional_kwargs = {}\n        if phoneme_language is not None:\n            additional_kwargs[\"phonemizer_lang\"] = phoneme_language\n\n        batch[\"labels\"] = tokenizer(batch[\"target_text\"], **additional_kwargs).input_ids\n        return batch\n\n    with training_args.main_process_first(desc=\"dataset map preprocessing\"):\n        vectorized_datasets = raw_datasets.map(\n            prepare_dataset,\n            remove_columns=next(iter(raw_datasets.values())).column_names,\n            num_proc=num_workers,\n            desc=\"preprocess datasets\",\n        )\n\n        def is_audio_in_length_range(length):\n            return length > min_input_length and length < max_input_length\n\n        # filter data that is shorter than min_input_length\n        vectorized_datasets = vectorized_datasets.filter(\n            is_audio_in_length_range,\n            num_proc=num_workers,\n            input_columns=[\"input_length\"],\n        )\n\n    # 7. Next, we can prepare the training.\n    # Let's use word error rate (WER) as our evaluation metric,\n    # instantiate a data collator and the trainer\n\n    # Define evaluation metrics during training, *i.e.* word error rate, character error rate\n    eval_metrics = {metric: evaluate.load(metric, cache_dir=model_args.cache_dir) for metric in data_args.eval_metrics}\n\n    # for large datasets it is advised to run the preprocessing on a\n    # single machine first with ``args.preprocessing_only`` since there will mostly likely\n    # be a timeout when running the script in distributed mode.\n    # In a second step ``args.preprocessing_only`` can then be set to `False` to load the\n    # cached dataset\n    if data_args.preprocessing_only:\n        logger.info(f\"Data preprocessing finished. Files cached at {vectorized_datasets.cache_files}\")\n        return\n\n    # For languages like Chinese with large vocabulary size, we need to discard logits\n    # and only keep the argmax, otherwise we run out of memory during evaluation.\n    def preprocess_logits_for_metrics(logits, labels):\n        pred_ids = torch.argmax(logits, dim=-1)\n        return pred_ids, labels\n\n    def compute_metrics(pred):\n        pred_ids = pred.predictions[0]\n        pred.label_ids[pred.label_ids == -100] = tokenizer.pad_token_id\n\n        pred_str = tokenizer.batch_decode(pred_ids)\n        # we do not want to group tokens when computing the metrics\n        label_str = tokenizer.batch_decode(pred.label_ids, group_tokens=False)\n\n        metrics = {k: v.compute(predictions=pred_str, references=label_str) for k, v in eval_metrics.items()}\n\n        return metrics\n\n    # Now save everything to be able to create a single processor later\n    # make sure all processes wait until data is saved\n    with training_args.main_process_first():\n        # only the main process saves them\n        if is_main_process(training_args.local_rank):\n            # save feature extractor, tokenizer and config\n            feature_extractor.save_pretrained(training_args.output_dir)\n            tokenizer.save_pretrained(training_args.output_dir)\n            config.save_pretrained(training_args.output_dir)\n\n    try:\n        processor = AutoProcessor.from_pretrained(training_args.output_dir)\n    except (OSError, KeyError):\n        warnings.warn(\n            \"Loading a processor from a feature extractor config that does not\"\n            \" include a `processor_class` attribute is deprecated and will be removed in v5. Please add the following \"\n            \" attribute to your `preprocessor_config.json` file to suppress this warning: \"\n            \" `'processor_class': 'Wav2Vec2Processor'`\",\n            FutureWarning,\n        )\n        processor = Wav2Vec2Processor.from_pretrained(training_args.output_dir)\n\n    # Instantiate custom data collator\n    data_collator = DataCollatorCTCWithPadding(\n        processor=processor, feature_extractor_input_name=feature_extractor_input_name\n    )\n\n    # Initialize Trainer\n    trainer = Trainer(\n        model=model,\n        data_collator=data_collator,\n        args=training_args,\n        compute_metrics=compute_metrics,\n        train_dataset=vectorized_datasets[\"train\"] if training_args.do_train else None,\n        eval_dataset=vectorized_datasets[\"eval\"] if training_args.do_eval else None,\n        processing_class=processor,\n        preprocess_logits_for_metrics=preprocess_logits_for_metrics,\n    )\n\n    # 8. Finally, we can start training\n\n    # Training\n    if training_args.do_train:\n        # use last checkpoint if exist\n        if last_checkpoint is not None:\n            checkpoint = last_checkpoint\n        elif os.path.isdir(model_args.model_name_or_path):\n            checkpoint = model_args.model_name_or_path\n        else:\n            checkpoint = None\n\n        train_result = trainer.train(resume_from_checkpoint=checkpoint)\n        trainer.save_model()\n\n        metrics = train_result.metrics\n        max_train_samples = (\n            data_args.max_train_samples\n            if data_args.max_train_samples is not None\n            else len(vectorized_datasets[\"train\"])\n        )\n        metrics[\"train_samples\"] = min(max_train_samples, len(vectorized_datasets[\"train\"]))\n\n        trainer.log_metrics(\"train\", metrics)\n        trainer.save_metrics(\"train\", metrics)\n        trainer.save_state()\n\n    # Evaluation\n    results = {}\n    if training_args.do_eval:\n        logger.info(\"*** Evaluate ***\")\n        metrics = trainer.evaluate()\n        max_eval_samples = (\n            data_args.max_eval_samples if data_args.max_eval_samples is not None else len(vectorized_datasets[\"eval\"])\n        )\n        metrics[\"eval_samples\"] = min(max_eval_samples, len(vectorized_datasets[\"eval\"]))\n\n        trainer.log_metrics(\"eval\", metrics)\n        trainer.save_metrics(\"eval\", metrics)\n\n    # Write model card and (optionally) push to hub\n    config_name = data_args.dataset_config_name if data_args.dataset_config_name is not None else \"na\"\n    kwargs = {\n        \"finetuned_from\": model_args.model_name_or_path,\n        \"tasks\": \"automatic-speech-recognition\",\n        \"tags\": [\"automatic-speech-recognition\", data_args.dataset_name],\n        \"dataset_args\": (\n            f\"Config: {config_name}, Training split: {data_args.train_split_name}, Eval split:\"\n            f\" {data_args.eval_split_name}\"\n        ),\n        \"dataset\": f\"{data_args.dataset_name.upper()} - {config_name.upper()}\",\n    }\n    if \"common_voice\" in data_args.dataset_name:\n        kwargs[\"language\"] = config_name\n\n    if training_args.push_to_hub:\n        trainer.push_to_hub(**kwargs)\n    else:\n        trainer.create_model_card(**kwargs)\n\n\n\nSUMMARY:\nThis code is for fine-tuning the Wav2Vec2 model on an automatic speech recognition task using the Hugging Face Transformers library. It consists of the following steps:\n\n1. Data loading and preprocessing\n2. Model initialization and configuration\n3. Training the model with a custom data collator\n4. Evaluation of the trained model\n5. Saving the model and its metrics, as well as pushing it to the HUGGINGFACE MODEL Hub.\n\nHere is a concise summary of the code:\n\n- **Main function**: This Python script uses the Transformers library to fine-tune the Wav2Vec2 model on an ASC-related data set, in order to develop a model for automatic speech recognition.\n- **Input files**: The input files required are a file named `data.json` and either a checkpoint filename or directory name specified as `model_name_or_path`, also containing the output path to be used for saving training artifacts. The code will load preprocessed audio data from an H5AR file, along with labels.\n- **Output**: The function returns the metrics produced during evaluation of the model in question. \n\n**Note:** This Python script does not take user input arguments. It uses default parameters specified within its environment (e.g., `data_args` dictionary), to configure training specifics such as the model architecture, hyperparameters and datasets used.\n\n```\nif __name__ == \"__main__\":\n    # ... function call with default parameters\n```\n\nThe output is also not provided explicitly in this script. However it's worth noting that you would typically log or display these results after computation."}


created_at: 2025-02-05 09:13:40
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modular_my_new_model2.py:5-25

metadata:
{"docstring": "# Example where we only want to only modify the docstring", "end_line": 25, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modular_my_new_model2.py", "name": "MyNewModel2Config", "start_line": 5, "type": "class", "chroma:document": "CODE:\n# Example where we only want to only modify the docstring\nclass MyNewModel2Config(LlamaConfig):\n    r\"\"\"\n    This is the configuration class to store the configuration of a [`GemmaModel`]. It is used to instantiate an Gemma\n    model according to the specified arguments, defining the model architecture. Instantiating a configuration with the\n    defaults will yield a similar configuration to that of the Gemma-7B.\n    e.g. [google/gemma-7b](https://huggingface.co/google/gemma-7b)\n    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the\n    documentation from [`PretrainedConfig`] for more information.\n    Args:\n        vocab_size (`int`, *optional*, defaults to 256000):\n            Vocabulary size of the Gemma model. Defines the number of different tokens that can be represented by the\n            `inputs_ids` passed when calling [`GemmaModel`]\n    ```python\n    >>> from transformers import GemmaModel, GemmaConfig\n    >>> # Initializing a Gemma gemma-7b style configuration\n    >>> configuration = GemmaConfig()\n    >>> # Initializing a model from the gemma-7b style configuration\n    >>> model = GemmaModel(configuration)\n    >>> # Accessing the model configuration\n    >>> configuration = model.config\n\n\nSUMMARY:\n**Summary**\n\n`MyNewModel2Config` is a configuration class for `GemmaModel`, inheriting from `LlamaConfig`. It defines the model architecture and vocabulary size (default: 256000). The primary purpose of this class is to control the model outputs.\n\n\n**Key Implementation Details**\n\n* Vocabulary size (`vocab_size`): an integer parameter that defines the number of different tokens represented by `inputs_ids`.\n* Inheritance from `PretrainedConfig`: Configuration objects can be used to control model outputs.\n* Supported methods and interfaces:\n\t+ Conforming to `LlamaConfig`\n\t+ Extending `PretrainedConfig`\n\n**Notable Dependencies or Requirements**\n\n* The model is inspired by the Gemma-7B, reference to a specific pre-trained transformer model on Hugging Face.\n\nNote: I have preserved the exact class/function names and followed the technical terms used in the original code."}


created_at: 2025-02-05 09:13:40
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modular_my_new_model2.py:29-30

metadata:
{"docstring": "# Example where alllllll the dependencies are fetched to just copy the entire class", "end_line": 30, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modular_my_new_model2.py", "name": "MyNewModel2ForSequenceClassification", "start_line": 29, "type": "class", "chroma:document": "CODE:\n# Example where alllllll the dependencies are fetched to just copy the entire class\nclass MyNewModel2ForSequenceClassification(GemmaForSequenceClassification):\n\n\nSUMMARY:\n**Summary**\n\n`MyNewModel2ForSequenceClassification` is a subclass of `GemmaForSequenceClassification`, extending its functionality for sequence classification tasks.\n\n**Key Implementation Details**\n\n* Exposes no additional methods or attributes, passing through to the parent class (`GemmaForSequenceClassification`).\n\n**Notable Dependencies and Requirements**\n\n* Inherited from: `GemmaForSequenceClassification`\n* No external dependencies mentioned.\n           "}


created_at: 2025-02-05 09:13:40
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modular_multimodal1.py:4-5

metadata:
{"docstring": "# Check that we can correctly change the prefix (here add Text part at the end of the name)", "end_line": 5, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modular_multimodal1.py", "name": "Multimodal1TextModel", "start_line": 4, "type": "class", "chroma:document": "CODE:\n# Check that we can correctly change the prefix (here add Text part at the end of the name)\nclass Multimodal1TextModel(LlamaModel):\n\n\nSUMMARY:\n**Summary**\n\nThis code defines a new subclass `Multimodal1TextModel` which inherits from `LlamaModel`. The class has no additional implementation, suggesting it may be intended as a placeholder or base model for text-specific multimodal applications.\n\n**Key Implementation Details**\n\n* Inherited interface: `LlamaModel`\n* No distinct methods or attributes implemented\n\n**Notable Dependencies or Requirements**\n\n* `LlamaModel`: The superclass from which this class inherits, indicating that the `Multimodal1TextModel` relies on the properties and behavior of its parent class.\n* External Dependencies: Not identified in this snippet.\n\n**Primary Purpose**\n\nTo establish a textual extension of an existing model framework (`LlamaModel`), potentially serving as a foundation for text-modality specific functionality or applications."}


created_at: 2025-02-05 09:13:40
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modular_my_new_model.py:5-14

metadata:
{"docstring": "# Example where we only want to only add a new config argument and new arg doc\n# here there is no `ARG` so we are gonna take parent doc", "end_line": 14, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modular_my_new_model.py", "name": "MyNewModelConfig", "start_line": 5, "type": "class", "chroma:document": "CODE:\n# here there is no `ARG` so we are gonna take parent doc\nclass MyNewModelConfig(LlamaConfig):\n    r\"\"\"\n    new_param (`int`, *optional*, defaults to `False`):\n        A fun new parameter\n    \"\"\"\n\n    def __init__(self, mlp_bias=True, new_param=0, **super_kwargs):\n        super().__init__(self, **super_kwargs)\n        self.mlp_bias = mlp_bias\n\n\nSUMMARY:\n**Summary**\n\n*Class: MyNewModelConfig*\n---------------------\n\nThe `MyNewModelConfig` class inherits from `LlamaConfig`. It introduces two primary parameters:\n\t+ `new_param`: An optional integer parameter with default value `False`.\n\t+ `mlp_bias`: A boolean parameter controlling the bias for multi-layer perceptrons.\n\n**Implementation Details**\n------------------------\n\n* Initializes `MyNewModelConfig` with `self` as super class reference, accepting keyword arguments through `super_kwargs`.\n\n**Notable Dependencies and Requirements**\n-----------------------------------------\n\n* The class uses inheritance from `LlamaConfig`, implying its requirement to have an environment that supports this base class.\n* External dependency: `LlamaConfig`.\n \nNo additional dependencies are assumed."}


created_at: 2025-02-05 09:13:40
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modular_new_model.py:5-34

metadata:
{"docstring": "", "end_line": 34, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modular_new_model.py", "name": "NewModelConfig", "start_line": 5, "type": "class", "chroma:document": "CODE:\n\nclass NewModelConfig(GemmaConfig):\n    def __init__(\n        self,\n        vocab_size=256030,\n        hidden_size=64,\n        intermediate_size=90,\n        num_hidden_layers=28,\n        num_attention_heads=16,\n        num_key_value_heads=16,\n        head_dim=256,\n        hidden_act=\"gelu_pytorch_tanh\",\n        hidden_activation=None,\n        max_position_embeddings=1500,\n        initializer_range=0.02,\n        rms_norm_eps=1e-6,\n        use_cache=True,\n        pad_token_id=0,\n        eos_token_id=1,\n        bos_token_id=2,\n        tie_word_embeddings=True,\n        rope_theta=10000.0,\n        attention_bias=False,\n        attention_dropout=0.0,\n        **kwargs,\n    ):\n        super().__init__(self, **kwargs)\n\n    @property\n    def num_heads(self):\n\n\nSUMMARY:\n**Class Summary: `NewModelConfig`**\n\n*   **Purpose:** Configures the architecture of a new model using GemmaConfig as its base class.\n\n**Primary Functionality:**\nThe primary purpose of this class is to define and configure various hyperparameters for building a machine learning model, specifically a transformer-based architecture. These parameters include vocabulary size, hidden layers, attention heads, intermediate layers, activation functions, embedding dimensions, and more.\n\n**Implementation Details:**\n\n*   **Hyperparameter Tuning:** The configuration includes various tunable hyperparameters such as `vocab_size`, `hidden_size`, `intermediate_size`, `num_hidden_layers`, and others.\n*   **Attention Mechanism:** Key components like attention heads (`num_attention_heads`), intermediate layers, and embedding dimensions are defined to enable the model to focus on specific parts of the input data.\n\n**Notable Dependencies/Requirements:**\n\n*   Inherited from GemmaConfig: `super().__init__(self, **kwargs)`.\n*   Not explicitly listed but might be assumed or inherit from related classes/functions:\n    *   `vocab_size` might require access to a vocabulary or dictionary class.\n    *   `hidden_act` and `hidden_activation` could interact with related methods or functions for activation computations.\n\n**Key Methods:**\n\n*   `num_heads`: Returns the number of attention heads, which is stored in `self.num_attention_heads`.\n*   Other configuration values can be accessed as instance attributes (e.g., `self.vocab_size`, `self.hidden_size`)."}


created_at: 2025-02-05 09:13:40
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/image_processing_new_imgproc_model.py:36-286

metadata:
{"docstring": "", "end_line": 286, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/image_processing_new_imgproc_model.py", "name": "ImgprocModelImageProcessor", "start_line": 36, "type": "class", "chroma:document": "CODE:\n\nclass ImgprocModelImageProcessor(BaseImageProcessor):\n    r\"\"\"\n    Constructs a IMGPROC_MODEL image processor.\n\n    Args:\n        do_resize (`bool`, *optional*, defaults to `True`):\n            Whether to resize the image's (height, width) dimensions to the specified `size`. Can be overridden by the\n            `do_resize` parameter in the `preprocess` method.\n        size (`dict`, *optional*, defaults to `{\"height\": 384, \"width\": 384}`):\n            Size of the output image after resizing. Can be overridden by the `size` parameter in the `preprocess`\n            method.\n        resample (`PILImageResampling`, *optional*, defaults to `Resampling.BICUBIC`):\n            Resampling filter to use if resizing the image. Only has an effect if `do_resize` is set to `True`. Can be\n            overridden by the `resample` parameter in the `preprocess` method.\n        do_rescale (`bool`, *optional*, defaults to `True`):\n            Whether to rescale the image by the specified scale `rescale_factor`. Can be overridden by the\n            `do_rescale` parameter in the `preprocess` method.\n        rescale_factor (`int` or `float`, *optional*, defaults to `1/255`):\n            Scale factor to use if rescaling the image. Only has an effect if `do_rescale` is set to `True`. Can be\n            overridden by the `rescale_factor` parameter in the `preprocess` method.\n        do_normalize (`bool`, *optional*, defaults to `True`):\n            Whether to normalize the image. Can be overridden by the `do_normalize` parameter in the `preprocess`\n            method. Can be overridden by the `do_normalize` parameter in the `preprocess` method.\n        image_mean (`float` or `List[float]`, *optional*, defaults to `IMAGENET_STANDARD_MEAN`):\n            Mean to use if normalizing the image. This is a float or list of floats the length of the number of\n            channels in the image. Can be overridden by the `image_mean` parameter in the `preprocess` method. Can be\n            overridden by the `image_mean` parameter in the `preprocess` method.\n        image_std (`float` or `List[float]`, *optional*, defaults to `IMAGENET_STANDARD_STD`):\n            Standard deviation to use if normalizing the image. This is a float or list of floats the length of the\n            number of channels in the image. Can be overridden by the `image_std` parameter in the `preprocess` method.\n            Can be overridden by the `image_std` parameter in the `preprocess` method.\n        do_convert_rgb (`bool`, *optional*, defaults to `True`):\n            Whether to convert the image to RGB.\n    \"\"\"\n\n    model_input_names = [\"pixel_values\"]\n\n    def __init__(\n        self,\n        do_resize: bool = True,\n        size: Dict[str, int] = None,\n        resample: PILImageResampling = PILImageResampling.BICUBIC,\n        do_rescale: bool = True,\n        rescale_factor: Union[int, float] = 1 / 255,\n        do_normalize: bool = True,\n        image_mean: Optional[Union[float, List[float]]] = None,\n        image_std: Optional[Union[float, List[float]]] = None,\n        do_convert_rgb: bool = True,\n        **kwargs,\n    ) -> None:\n        super().__init__(**kwargs)\n        size = size if size is not None else {\"height\": 384, \"width\": 384}\n        size = get_size_dict(size, default_to_square=True)\n\n        self.do_resize = do_resize\n        self.size = size\n        self.resample = resample\n        self.do_rescale = do_rescale\n        self.rescale_factor = rescale_factor\n        self.do_normalize = do_normalize\n        self.image_mean = image_mean if image_mean is not None else OPENAI_CLIP_MEAN\n        self.image_std = image_std if image_std is not None else OPENAI_CLIP_STD\n        self.do_convert_rgb = do_convert_rgb\n\n    def resize(\n        self,\n        image: np.ndarray,\n        size: Dict[str, int],\n        resample: PILImageResampling = PILImageResampling.BICUBIC,\n        data_format: Optional[Union[str, ChannelDimension]] = None,\n        input_data_format: Optional[Union[str, ChannelDimension]] = None,\n        **kwargs,\n    ) -> np.ndarray:\n        \"\"\"\n        Resize an image to `(size[\"height\"], size[\"width\"])`.\n\n        Args:\n            image (`np.ndarray`):\n                Image to resize.\n            size (`Dict[str, int]`):\n                Dictionary in the format `{\"height\": int, \"width\": int}` specifying the size of the output image.\n            resample (`PILImageResampling`, *optional*, defaults to `PILImageResampling.BICUBIC`):\n                `PILImageResampling` filter to use when resizing the image e.g. `PILImageResampling.BICUBIC`.\n            data_format (`ChannelDimension` or `str`, *optional*):\n                The channel dimension format for the output image. If unset, the channel dimension format of the input\n                image is used. Can be one of:\n                - `\"channels_first\"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format.\n                - `\"channels_last\"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format.\n                - `\"none\"` or `ChannelDimension.NONE`: image in (height, width) format.\n            input_data_format (`ChannelDimension` or `str`, *optional*):\n                The channel dimension format for the input image. If unset, the channel dimension format is inferred\n                from the input image. Can be one of:\n                - `\"channels_first\"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format.\n                - `\"channels_last\"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format.\n                - `\"none\"` or `ChannelDimension.NONE`: image in (height, width) format.\n\n        Returns:\n            `np.ndarray`: The resized image.\n        \"\"\"\n        size = get_size_dict(size)\n        if \"height\" not in size or \"width\" not in size:\n            raise ValueError(f\"The `size` dictionary must contain the keys `height` and `width`. Got {size.keys()}\")\n        output_size = (size[\"height\"], size[\"width\"])\n        return resize(\n            image,\n            size=output_size,\n            resample=resample,\n            data_format=data_format,\n            input_data_format=input_data_format,\n            **kwargs,\n        )\n\n    @filter_out_non_signature_kwargs()\n    def preprocess(\n        self,\n        images: ImageInput,\n        do_resize: Optional[bool] = None,\n        size: Optional[Dict[str, int]] = None,\n        resample: PILImageResampling = None,\n        do_rescale: Optional[bool] = None,\n        rescale_factor: Optional[float] = None,\n        do_normalize: Optional[bool] = None,\n        image_mean: Optional[Union[float, List[float]]] = None,\n        image_std: Optional[Union[float, List[float]]] = None,\n        return_tensors: Optional[Union[str, TensorType]] = None,\n        do_convert_rgb: bool = None,\n        data_format: ChannelDimension = ChannelDimension.FIRST,\n        input_data_format: Optional[Union[str, ChannelDimension]] = None,\n    ) -> PIL.Image.Image:\n        \"\"\"\n        Preprocess an image or batch of images.\n\n        Args:\n            images (`ImageInput`):\n                Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If\n                passing in images with pixel values between 0 and 1, set `do_rescale=False`.\n            do_resize (`bool`, *optional*, defaults to `self.do_resize`):\n                Whether to resize the image.\n            size (`Dict[str, int]`, *optional*, defaults to `self.size`):\n                Controls the size of the image after `resize`. The shortest edge of the image is resized to\n                `size[\"shortest_edge\"]` whilst preserving the aspect ratio. If the longest edge of this resized image\n                is > `int(size[\"shortest_edge\"] * (1333 / 800))`, then the image is resized again to make the longest\n                edge equal to `int(size[\"shortest_edge\"] * (1333 / 800))`.\n            resample (`PILImageResampling`, *optional*, defaults to `self.resample`):\n                Resampling filter to use if resizing the image. Only has an effect if `do_resize` is set to `True`.\n            do_rescale (`bool`, *optional*, defaults to `self.do_rescale`):\n                Whether to rescale the image values between [0 - 1].\n            rescale_factor (`float`, *optional*, defaults to `self.rescale_factor`):\n                Rescale factor to rescale the image by if `do_rescale` is set to `True`.\n            do_normalize (`bool`, *optional*, defaults to `self.do_normalize`):\n                Whether to normalize the image.\n            image_mean (`float` or `List[float]`, *optional*, defaults to `self.image_mean`):\n                Image mean to normalize the image by if `do_normalize` is set to `True`.\n            image_std (`float` or `List[float]`, *optional*, defaults to `self.image_std`):\n                Image standard deviation to normalize the image by if `do_normalize` is set to `True`.\n            do_convert_rgb (`bool`, *optional*, defaults to `self.do_convert_rgb`):\n                Whether to convert the image to RGB.\n            return_tensors (`str` or `TensorType`, *optional*):\n                The type of tensors to return. Can be one of:\n                    - Unset: Return a list of `np.ndarray`.\n                    - `TensorType.TENSORFLOW` or `'tf'`: Return a batch of type `tf.Tensor`.\n                    - `TensorType.PYTORCH` or `'pt'`: Return a batch of type `torch.Tensor`.\n                    - `TensorType.NUMPY` or `'np'`: Return a batch of type `np.ndarray`.\n                    - `TensorType.JAX` or `'jax'`: Return a batch of type `jax.numpy.ndarray`.\n            data_format (`ChannelDimension` or `str`, *optional*, defaults to `ChannelDimension.FIRST`):\n                The channel dimension format for the output image. Can be one of:\n                - `\"channels_first\"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format.\n                - `\"channels_last\"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format.\n                - Unset: Use the channel dimension format of the input image.\n            input_data_format (`ChannelDimension` or `str`, *optional*):\n                The channel dimension format for the input image. If unset, the channel dimension format is inferred\n                from the input image. Can be one of:\n                - `\"channels_first\"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format.\n                - `\"channels_last\"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format.\n                - `\"none\"` or `ChannelDimension.NONE`: image in (height, width) format.\n        \"\"\"\n        do_resize = do_resize if do_resize is not None else self.do_resize\n        resample = resample if resample is not None else self.resample\n        do_rescale = do_rescale if do_rescale is not None else self.do_rescale\n        rescale_factor = rescale_factor if rescale_factor is not None else self.rescale_factor\n        do_normalize = do_normalize if do_normalize is not None else self.do_normalize\n        image_mean = image_mean if image_mean is not None else self.image_mean\n        image_std = image_std if image_std is not None else self.image_std\n        do_convert_rgb = do_convert_rgb if do_convert_rgb is not None else self.do_convert_rgb\n\n        size = size if size is not None else self.size\n        size = get_size_dict(size, default_to_square=False)\n\n        images = make_list_of_images(images)\n\n        if not valid_images(images):\n            raise ValueError(\n                \"Invalid image type. Must be of type PIL.Image.Image, numpy.ndarray, \"\n                \"torch.Tensor, tf.Tensor or jax.ndarray.\"\n            )\n\n        validate_preprocess_arguments(\n            do_rescale=do_rescale,\n            rescale_factor=rescale_factor,\n            do_normalize=do_normalize,\n            image_mean=image_mean,\n            image_std=image_std,\n            do_resize=do_resize,\n            size=size,\n            resample=resample,\n        )\n        # PIL RGBA images are converted to RGB\n        if do_convert_rgb:\n            images = [convert_to_rgb(image) for image in images]\n\n        # All transformations expect numpy arrays.\n        images = [to_numpy_array(image) for image in images]\n\n        if do_rescale and is_scaled_image(images[0]):\n            logger.warning_once(\n                \"It looks like you are trying to rescale already rescaled images. If the input\"\n                \" images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again.\"\n            )\n\n        if input_data_format is None:\n            # We assume that all images have the same channel dimension format.\n            input_data_format = infer_channel_dimension_format(images[0])\n\n        if do_resize:\n            images = [\n                self.resize(image=image, size=size, resample=resample, input_data_format=input_data_format)\n                for image in images\n            ]\n\n        if do_rescale:\n            images = [\n                self.rescale(image=image, scale=rescale_factor, input_data_format=input_data_format)\n                for image in images\n            ]\n\n        if do_normalize:\n            images = [\n                self.normalize(image=image, mean=image_mean, std=image_std, input_data_format=input_data_format)\n                for image in images\n            ]\n\n        images = [\n            to_channel_dimension_format(image, data_format, input_channel_dim=input_data_format) for image in images\n        ]\n\n        encoded_outputs = BatchFeature(data={\"pixel_values\": images}, tensor_type=return_tensors)\n\n        return encoded_outputs\n\n    def new_image_processing_method(self, pixel_values: torch.FloatTensor):\n\n\nSUMMARY:\nThis is a PyTorch-based image processing class. The class seems to be designed for preprocessing and enhancing images in various ways, such as resizing, scaling, normalizing, and converting them to RGB format.\n\nHere's a high-level overview of the class:\n\n- The class has several attributes that determine how images are processed:\n  - `do_rescale`: Whether or not to rescale the pixel values.\n  - `rescale_factor`: The factor by which the pixel values should be scaled if rescaling is needed.\n  - `do_normalize`: Whether or not to normalize the pixel values.\n  - `image_mean` and `image_std`: Mean and standard deviation used for normalizing.\n  - `size`: The size at which images are resized (for downsampling).\n  - `resample`: The method used for resizing (e.g., nearest-neighbor, bicubic).\n- It also attributes and methods that handle errors and edge cases:\n  - `make_list_of_images` takes a collection of images as input.\n  - `validate_preprocess_arguments` checks if all preprocessings arguments are valid.\n  - It checks for invalid image type (i.e., PIL Image, np.ndarray, torch.Tensor, tf.Tensor or jax.ndarray).\n- The class processes preprocessed images by:\n  - Converting them to numpy arrays\n  - Rescaling and normalizing these arrays according to their respective settings.\n  - After preprocessing, it encodes the output in a format that can be consumed by a certain model.\n\nHere's a summary of what this code does:\n\n* Preprocesses input image(s) based on specified transformation settings\n* May rescale pixel values if necessary\n* Normalize pixel values with mean and standard deviation (if applicable)\n* Apply transformations like resizing or resampling if required\n\nPlease provide more context about where you'd like to improve this implementation, to give a clearer idea of what changes are acceptable and relevant."}


created_at: 2025-02-05 09:13:40
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy.py:26-43
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 43, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy.py", "name": "DummyRMSNorm", "start_line": 26, "type": "class", "chroma:document": "CODE:\n\nclass DummyRMSNorm(nn.Module):\n    def __init__(self, hidden_size, eps=1e-6):\n        \"\"\"\n        DummyRMSNorm is equivalent to T5LayerNorm\n        \"\"\"\n        super().__init__()\n        self.weight = nn.Parameter(torch.ones(hidden_size))\n        self.variance_epsilon = eps\n\n    def forward(self, hidden_states):\n        input_dtype = hidden_states.dtype\n        hidden_states = hidden_states.to(torch.float32)\n        variance = hidden_states.pow(2).mean(-1, keepdim=True)\n        hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)\n        return self.weight * hidden_states.to(input_dtype)\n\n    def extra_repr(self):\n\n\nSUMMARY:\n**Summary: DummyRMSNorm Module**\n\nThe `DummyRMSNorm` class is a PyTorch module equivalent to the T5LayerNorm. It normalizes input tensors with a scaling factor determined by the variance of the input.\n\n**Key Implementation Details:**\n\n* Scales input tensor by square root of mean-variance, modified by epsilon value.\n* Uses self-weight parameter to scale output.\n* Employs `pow` and `mean` operations over input tensor with shape `(batch_size, seq_len, hidden_size)`.\n\n**Notable Dependencies/Requirements:**\n\n* PyTorch library (`torch`, `nn.Module`)\n* TensorFlow-compatible data types (`float32`)"}


created_at: 2025-02-05 09:13:41
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy.py:46-108
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 108, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy.py", "name": "DummyRotaryEmbedding", "start_line": 46, "type": "class", "chroma:document": "CODE:\n\nclass DummyRotaryEmbedding(nn.Module):\n    def __init__(\n        self,\n        config: DummyConfig,\n        device=None,\n    ):\n        super().__init__()\n        self.rope_kwargs = {}\n        # BC: \"rope_type\" was originally \"type\"\n        if hasattr(config, \"rope_scaling\") and config.rope_scaling is not None:\n            self.rope_type = config.rope_scaling.get(\"rope_type\", config.rope_scaling.get(\"type\"))\n        else:\n            self.rope_type = \"default\"\n        self.max_seq_len_cached = config.max_position_embeddings\n        self.original_max_seq_len = config.max_position_embeddings\n\n        self.config = config\n        self.rope_init_fn = ROPE_INIT_FUNCTIONS[self.rope_type]\n\n        inv_freq, self.attention_scaling = self.rope_init_fn(self.config, device, **self.rope_kwargs)\n        self.register_buffer(\"inv_freq\", inv_freq, persistent=False)\n        self.original_inv_freq = self.inv_freq\n\n    def _dynamic_frequency_update(self, position_ids, device):\n        \"\"\"\n        dynamic RoPE layers should recompute `inv_freq` in the following situations:\n        1 - growing beyond the cached sequence length (allow scaling)\n        2 - the current sequence length is in the original scale (avoid losing precision with small sequences)\n        \"\"\"\n        seq_len = torch.max(position_ids) + 1\n        if seq_len > self.max_seq_len_cached:  # growth\n            inv_freq, self.attention_scaling = self.rope_init_fn(\n                self.config, device, seq_len=seq_len, **self.rope_kwargs\n            )\n            self.register_buffer(\"inv_freq\", inv_freq, persistent=False)  # TODO joao: may break with compilation\n            self.max_seq_len_cached = seq_len\n\n        if seq_len < self.original_max_seq_len and self.max_seq_len_cached > self.original_max_seq_len:  # reset\n            self.register_buffer(\"inv_freq\", self.original_inv_freq, persistent=False)\n            self.max_seq_len_cached = self.original_max_seq_len\n\n    @torch.no_grad()\n    def forward(self, x, position_ids):\n        if \"dynamic\" in self.rope_type:\n            self._dynamic_frequency_update(position_ids, device=x.device)\n\n        # Core RoPE block\n        inv_freq_expanded = self.inv_freq[None, :, None].float().expand(position_ids.shape[0], -1, 1)\n        position_ids_expanded = position_ids[:, None, :].float()\n        # Force float32 (see https://github.com/huggingface/transformers/pull/29285)\n        device_type = x.device.type\n        device_type = device_type if isinstance(device_type, str) and device_type != \"mps\" else \"cpu\"\n        with torch.autocast(device_type=device_type, enabled=False):\n            freqs = (inv_freq_expanded.float() @ position_ids_expanded.float()).transpose(1, 2)\n            emb = torch.cat((freqs, freqs), dim=-1)\n            cos = emb.cos()\n            sin = emb.sin()\n\n        # Advanced RoPE types (e.g. yarn) apply a post-processing scaling factor, equivalent to scaling attention\n        cos = cos * self.attention_scaling\n        sin = sin * self.attention_scaling\n\n\n\nSUMMARY:\n**Summary:**\n\nThe `DummyRotaryEmbedding` class is a PyTorch module implementing RoPE (Rotation Embeddings) layers. It adapts to dynamic sequence lengths and applies attention scaling.\n\n**Main Purpose and Functionality:**\n\nThis class is designed to compute frequency-based embeddings for input sequences, allowing the model to handle varying sequence lengths efficiently. The main functionality includes:\n\n* Adaptive initialization of `inv_freq` based on sequence length\n* Dynamic re-computation of `inv_freq` when sequence length grows or shrinks\n* Application of attention scaling factor to post-process embeddings\n\n**Key Implementation Details:**\n\n* Use of `torch.autocast` for dynamic torch device type management\n* Custom RoPE initialization function selection based on `rope_type`\n* Usage of `position_ids` for frequency-based embedding computation\n* Attention scaling factor application for post-processing embeddings\n\n**Notable Dependencies and Requirements:**\n\n* PyTorch module (`nn.Module`)\n* External dependencies:\n\t+ `DummyConfig` class (config passing)\n\t+ `ROPE_INIT_FUNCTIONS` dictionary (rope type initialization function selection)\n\n**Additional Notes:**\n\nThe RoPE implementation utilizes frequency-based embeddings for input sequences, enabling the model to adapt to varying sequence lengths while maintaining efficient computational complexity."}


created_at: 2025-02-05 09:13:41
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy.py:111-124

metadata:
{"docstring": "", "end_line": 124, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy.py", "name": "DummyMLP", "start_line": 111, "type": "class", "chroma:document": "CODE:\n\nclass DummyMLP(nn.Module):\n    def __init__(self, config):\n        super().__init__()\n        self.config = config\n        self.hidden_size = config.hidden_size\n        self.intermediate_size = config.intermediate_size\n        self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=config.mlp_bias)\n        self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=config.mlp_bias)\n        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=config.mlp_bias)\n        self.act_fn = ACT2FN[config.hidden_act]\n\n    def forward(self, x):\n        down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))\n\n\nSUMMARY:\n**Class Summary: `DummyMLP`**\n\n**Main Purpose and Functionality:** \nThe `DummyMLP` class implements a basic multi-layer perceptron (MLP) architecture. It takes in input data `x`, applies actuarial function, combines with two projections (`up_proj` and `down_proj`) and outputs the result.\n\n**Key Implementation Details:**\n\n* Uses feed-forward neural network (`nn.Module`) for model construction.\n* Inclodes multiple linear transformations:\n\t+ `gate_proj`: projects input onto hidden layer\n\t+ `up_proj`: up-projects output from gateProj\n  - `down_proj`: Down-projects and combines with act_fn(x)\n* Utilizes an acturial function (`ACT2FN[config.hidden_act]`) to apply activation mapping.\n\n**Notable Dependencies or Requirements:**\n\n* Uses PyTorch's neural network module (`nn.Module`)\n* References external dictionary (`ACT2FN`)\n\nPlease let me know if you want me to clarify any details."}


created_at: 2025-02-05 09:13:41
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy.py:127-131

metadata:
{"docstring": "", "end_line": 131, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy.py", "name": "rotate_half", "start_line": 127, "type": "function", "chroma:document": "CODE:\n\ndef rotate_half(x):\n    \"\"\"Rotates half the hidden dims of the input.\"\"\"\n    x1 = x[..., : x.shape[-1] // 4]\n    x2 = x[..., x.shape[-1] // 4 :]\n\n\nSUMMARY:\n**Summary**\n\nRotates the specified half of the hidden dimensions in input `x`. Specifically, it:\n\n* Splits the input into two halves using index division by 4\n* Rotates the second half (`x2`) to the beginning and keeps the first half (`x1`)\n* Concatenates the rotated halves with opposite signs using `torch cat`\n\n**Implementation Details**\n\n* Uses PyTorch tensor indexing and concatenation\n\n**Dependencies/Requirements**\n\n* Requires `torch` library for tensor operations."}


created_at: 2025-02-05 09:13:41
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy.py:134-158

metadata:
{"docstring": "", "end_line": 158, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy.py", "name": "apply_rotary_pos_emb", "start_line": 134, "type": "function", "chroma:document": "CODE:\n\ndef apply_rotary_pos_emb(q, k, cos, sin, position_ids=None, unsqueeze_dim=1):\n    \"\"\"Applies Rotary Position Embedding to the query and key tensors.\n\n    Args:\n        q (`torch.Tensor`): The query tensor.\n        k (`torch.Tensor`): The key tensor.\n        cos (`torch.Tensor`): The cosine part of the rotary embedding.\n        sin (`torch.Tensor`): The sine part of the rotary embedding.\n        position_ids (`torch.Tensor`, *optional*):\n            Deprecated and unused.\n        unsqueeze_dim (`int`, *optional*, defaults to 1):\n            The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and\n            sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note\n            that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and\n            k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes\n            cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have\n            the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2.\n    Returns:\n        `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding.\n    \"\"\"\n    cos = cos.unsqueeze(unsqueeze_dim)\n    sin = sin.unsqueeze(unsqueeze_dim)\n    q_embed = (q * cos) + (rotate_half(q) * sin)\n    k_embed = (k * cos) + (rotate_half(k) * sin)\n\n\nSUMMARY:\n**Rotary Position Embedding Function Summary**\n\n### Purpose and Functionality\nApplies Rotary Position Embedding to query and key tensors.\n\n#### Key Implementation Details\n\n* Applies Rotatory Position Embedding using `cos(x) + rotate_half(sin(x))`\n* Uses cosine (`cos`) and sine (`sin`) components of the embedding as specified arguments.\n* Unsqueeze dimensions of `cos` and `sin` tensors for proper broadcasting with query (`q`) and key (`k`) tensors.\n\n#### Notable Dependencies\n\n* Inputs:\n\t+ `q`: Query tensor (torch.Tensor)\n\t+ `k`: Key tensor (torch.Tensor)\n\t+ `cos`, `sin`: Cosine and sine components of the embedding (torch.Tensor)\n\t+ `position_ids` (Deprecated, Unused): Optional position ID tensor\n\t+ `unsqueeze_dim`: Dimension to unsqueeze `cos[position_ids]` and `sin[position_ids]` for broadcasting\n* Returns: Tuples of rotated query (`q_embed`) and key (`k_embed`) tensors.\n\n#### Dependencies and Interfaces\n\n* No external dependencies.\n* Implies use of `torch.Tensor` type and vector operations."}


created_at: 2025-02-05 09:13:41
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy.py:161-170

metadata:
{"docstring": "", "end_line": 170, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy.py", "name": "repeat_kv", "start_line": 161, "type": "function", "chroma:document": "CODE:\n\ndef repeat_kv(hidden_states: torch.Tensor, n_rep: int) -> torch.Tensor:\n    \"\"\"\n    This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch,\n    num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim)\n    \"\"\"\n    batch, num_key_value_heads, slen, head_dim = hidden_states.shape\n    if n_rep == 1:\n        return hidden_states\n    hidden_states = hidden_states[:, :, None, :, :].expand(batch, num_key_value_heads, n_rep, slen, head_dim)\n\n\nSUMMARY:\n**Summary:**\n\nThe `repeat_kv` function is equivalent to `torch.repeat_interleave`, repeating the `hidden_states` tensor along the key-value heads dimension (`dim=1`) with `n_rep` repetitions. It transforms the input from `(batch, num_key_value_heads, seqlen, head_dim)` to `(batch, num_attention_heads, seqlen, head_dim)`. The function is designed to handle multiple attention heads by repeating each head `n_rep` times.\n\n**Key details:**\n\n* Transform: Repeats along key-value heads dimension (`dim=1`)\n* Repeats: `n_rep` repetitions\n* Output shape: `(batch, num_attention_heads, seqlen, head_dim)`\n* Equivalent to: `torch.repeat_interleave(x, dim=1, repeats=n_rep)`\n\n**Notable dependencies/requirements:**\n\n* `torch`: Required for tensor operations and repeat functionality."}


created_at: 2025-02-05 09:13:41
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy.py:173-196

metadata:
{"docstring": "", "end_line": 196, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy.py", "name": "eager_attention_forward", "start_line": 173, "type": "function", "chroma:document": "CODE:\n\ndef eager_attention_forward(\n    module: nn.Module,\n    query: torch.Tensor,\n    key: torch.Tensor,\n    value: torch.Tensor,\n    attention_mask: Optional[torch.Tensor],\n    scaling: float,\n    dropout: float = 0.0,\n    **kwargs,\n):\n    key_states = repeat_kv(key, module.num_key_value_groups)\n    value_states = repeat_kv(value, module.num_key_value_groups)\n\n    attn_weights = torch.matmul(query, key_states.transpose(2, 3)) * scaling\n    if attention_mask is not None:\n        causal_mask = attention_mask[:, :, :, : key_states.shape[-2]]\n        attn_weights = attn_weights + causal_mask\n\n    attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)\n    attn_weights = nn.functional.dropout(attn_weights, p=dropout, training=module.training)\n    attn_output = torch.matmul(attn_weights, value_states)\n    attn_output = attn_output.transpose(1, 2).contiguous()\n\n\n\nSUMMARY:\n**Concise Summary**\n\nThe `eager_attention_forward` function implements multi-head attention mechanism using eager attention. It takes in query, key, value tensors and an optional attention mask. The function:\n\n1. Scales the key-value pairs by a factor of 2^module.num_key_value_groups.\n2. Computes attention weights through matrix multiplication between query and scaled keys (dim = -1).\n3. Applies causal masking if provided.\n4. Softmax normalization and dropout are applied to the attention weights.\n5. Computes output by multiplying scaled weights with value tensor.\n6. Returns the output and scaled weights.\n\n**Key Implementation Details**\n\n* `torch.matmul` for matrix multiplication\n* `nn.functional.softmax` for softmax normalization\n* `nn.functional.dropout` for scaling weights\n\n**Dependences and Notes**\n\n* Inheritance: no external modules, only PyTorch's `nn.Module`\n* Interfaces: implements Multi-Head Attention (uses repeated key-value groups)\n* Scaling factor can be adjusted to the `module.num_key_value_groups`"}


created_at: 2025-02-05 09:13:41
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy.py:199-272

metadata:
{"docstring": "", "end_line": 272, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy.py", "name": "DummyAttention", "start_line": 199, "type": "class", "chroma:document": "CODE:\n\nclass DummyAttention(nn.Module):\n    \"\"\"Multi-headed attention from 'Attention Is All You Need' paper\"\"\"\n\n    def __init__(self, config: DummyConfig, layer_idx: int):\n        super().__init__()\n        self.config = config\n        self.layer_idx = layer_idx\n        self.head_dim = getattr(config, \"head_dim\", config.hidden_size // config.num_attention_heads)\n        self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads\n        self.scaling = self.head_dim**-0.5\n        self.attention_dropout = config.attention_dropout\n        self.is_causal = True\n\n        self.q_proj = nn.Linear(\n            config.hidden_size, config.num_attention_heads * self.head_dim, bias=config.attention_bias\n        )\n        self.k_proj = nn.Linear(\n            config.hidden_size, config.num_key_value_heads * self.head_dim, bias=config.attention_bias\n        )\n        self.v_proj = nn.Linear(\n            config.hidden_size, config.num_key_value_heads * self.head_dim, bias=config.attention_bias\n        )\n        self.o_proj = nn.Linear(\n            config.num_attention_heads * self.head_dim, config.hidden_size, bias=config.attention_bias\n        )\n\n    def forward(\n        self,\n        hidden_states: torch.Tensor,\n        position_embeddings: Tuple[torch.Tensor, torch.Tensor],\n        attention_mask: Optional[torch.Tensor],\n        past_key_value: Optional[Cache] = None,\n        cache_position: Optional[torch.LongTensor] = None,\n        **kwargs: Unpack[FlashAttentionKwargs],\n    ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:\n        input_shape = hidden_states.shape[:-1]\n        hidden_shape = (*input_shape, -1, self.head_dim)\n\n        query_states = self.q_proj(hidden_states).view(hidden_shape).transpose(1, 2)\n        key_states = self.k_proj(hidden_states).view(hidden_shape).transpose(1, 2)\n        value_states = self.v_proj(hidden_states).view(hidden_shape).transpose(1, 2)\n\n        cos, sin = position_embeddings\n        query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)\n\n        if past_key_value is not None:\n            # sin and cos are specific to RoPE models; cache_position needed for the static cache\n            cache_kwargs = {\"sin\": sin, \"cos\": cos, \"cache_position\": cache_position}\n            key_states, value_states = past_key_value.update(key_states, value_states, self.layer_idx, cache_kwargs)\n\n        attention_interface: Callable = eager_attention_forward\n        if self.config._attn_implementation != \"eager\":\n            if self.config._attn_implementation == \"sdpa\" and kwargs.get(\"output_attentions\", False):\n                logger.warning_once(\n                    \"`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to \"\n                    'eager attention. This warning can be removed using the argument `attn_implementation=\"eager\"` when loading the model.'\n                )\n            else:\n                attention_interface = ALL_ATTENTION_FUNCTIONS[self.config._attn_implementation]\n\n        attn_output, attn_weights = attention_interface(\n            self,\n            query_states,\n            key_states,\n            value_states,\n            attention_mask,\n            dropout=0.0 if not self.training else self.attention_dropout,\n            scaling=self.scaling,\n            **kwargs,\n        )\n\n        attn_output = attn_output.reshape(*input_shape, -1).contiguous()\n        attn_output = self.o_proj(attn_output)\n\n\nSUMMARY:\n**Summary:**\n\nThe `DummyAttention` class implements multi-headed attention from the \"Attention Is All You Need\" paper. It extends PyTorch's `nn.Module` and is used in transformer architectures.\n\n**Key Features:**\n\n* Takes a configuration object, layer index, and input states (hidden_states) as inputs.\n* Projects hidden_states to query, key, and value vectors using linear layers.\n* Applies position embeddings for cosine and sine values.\n\n**Implementation Details:**\n\n* Uses the `eager` attention implementation by default or based on the `_attn_implementation` configuration.\n* Incorporates a past_key_value cache for efficient updates when caching positional information.\n* Projects outputs through an output projection layer (`o_proj`) to obtain attention scores.\n\n**Notable Dependencies and Requirements:**\n\n* PyTorch's `nn.Module`\n* `dummy_config` class (config object)\n* `torch.nn.Linear`, `torch.utils.data.Tensor`"}


created_at: 2025-02-05 09:13:41
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy.py:275-326

metadata:
{"docstring": "", "end_line": 326, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy.py", "name": "DummyDecoderLayer", "start_line": 275, "type": "class", "chroma:document": "CODE:\n\nclass DummyDecoderLayer(nn.Module):\n    def __init__(self, config: DummyConfig, layer_idx: int):\n        super().__init__()\n        self.hidden_size = config.hidden_size\n\n        self.self_attn = DummyAttention(config=config, layer_idx=layer_idx)\n\n        self.mlp = DummyMLP(config)\n        self.input_layernorm = DummyRMSNorm(config.hidden_size, eps=config.rms_norm_eps)\n        self.post_attention_layernorm = DummyRMSNorm(config.hidden_size, eps=config.rms_norm_eps)\n\n    def forward(\n        self,\n        hidden_states: torch.Tensor,\n        attention_mask: Optional[torch.Tensor] = None,\n        position_ids: Optional[torch.LongTensor] = None,\n        past_key_value: Optional[Cache] = None,\n        output_attentions: Optional[bool] = False,\n        use_cache: Optional[bool] = False,\n        cache_position: Optional[torch.LongTensor] = None,\n        position_embeddings: Optional[Tuple[torch.Tensor, torch.Tensor]] = None,  # necessary, but kept here for BC\n        **kwargs: Unpack[FlashAttentionKwargs],\n    ) -> Tuple[torch.FloatTensor, Optional[Tuple[torch.FloatTensor, torch.FloatTensor]]]:\n        residual = hidden_states\n\n        hidden_states = self.input_layernorm(hidden_states)\n\n        # Self Attention\n        hidden_states, self_attn_weights = self.self_attn(\n            hidden_states=hidden_states,\n            attention_mask=attention_mask,\n            position_ids=position_ids,\n            past_key_value=past_key_value,\n            output_attentions=output_attentions,\n            use_cache=use_cache,\n            cache_position=cache_position,\n            position_embeddings=position_embeddings,\n            **kwargs,\n        )\n        hidden_states = residual + hidden_states\n\n        # Fully Connected\n        residual = hidden_states\n        hidden_states = self.post_attention_layernorm(hidden_states)\n        hidden_states = self.mlp(hidden_states)\n        hidden_states = residual + hidden_states\n\n        outputs = (hidden_states,)\n        if output_attentions:\n            outputs += (self_attn_weights,)\n\n\n\nSUMMARY:\n**DummyDecoderLayer Class Summary**\n\n### Purpose and Functionality\nThe `DummyDecoderLayer` class implements a layer in a transformer decoder architecture. Its primary purpose is to perform self-attention and fully connected linear transformations on input embeddings.\n\n### Key Implementation Details\n\n* Performs self-attention using the `self_attn` method, which takes input embeddings, attention mask, and other optional arguments.\n\t+ Method Signature: `self.self_attn(hidden_states=..., attention_mask=..., ...)`\n\t+ Returns: `(hidden_states, self_attn_weights)`\n* Applies linear transformation using the `mlp` method, which takes input embeddings after self-attention.\n\t+ Method Signature: `self.mlp(hidden_states=...)`\n\t+ Returns: `(hidden_states)`\n* Normalizes inputs with layer normalization (`input_layernorm`) and post-attention normalization (`post_attention_layernorm`).\n\n### Notable Dependencies and Requirements\n\n* `config`: an instance of `DummyConfig`, which contains hyperparameters for the layer.\n* External dependencies:\n\t+ `torch`\n\t+ `nn.Module` (from PyTorch)\n\t+ `torch.Tensor`\n\t+ `Optional`\n\t+ `torch.LongTensor`\n\n### Inherited or Implemented Interfaces\n\n* Implements the `nn.Module` interface from PyTorch.\n\nNote: This summary focuses on the key implementation details and dependencies specific to the `DummyDecoderLayer` class, without providing unnecessary or generic information."}


created_at: 2025-02-05 09:13:41
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modular_new_imgproc_model.py:6-8

metadata:
{"docstring": "", "end_line": 8, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modular_new_imgproc_model.py", "name": "ImgprocModelImageProcessor", "start_line": 6, "type": "class", "chroma:document": "CODE:\n\nclass ImgprocModelImageProcessor(BlipImageProcessor):\n    def new_image_processing_method(self, pixel_values: torch.FloatTensor):\n\n\nSUMMARY:\n**Summary:**\n\nThis class, `ImgprocModelImageProcessor`, inherits from `BlipImageProcessor`. It defines a method `new_image_processing_method` which takes a `torch.FloatTensor` input `pixel_values` and returns the pixel values divided by 2. \n\n**Key Details:**\n\n- Algorithm: Simple Linear Scaling\n- Method Signature: `torch.FloatTensor -> torch.FloatTensor`\n- Interfaces: Inherited from `BlipImageProcessor`\n\n**Notable Dependencies/Requirements:** \n- `torch` tensor library"}


created_at: 2025-02-05 09:13:41
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/configuration_my_new_model.py:11-205

metadata:
{"docstring": "", "end_line": 205, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/configuration_my_new_model.py", "name": "MyNewModelConfig", "start_line": 11, "type": "class", "chroma:document": "CODE:\n\nclass MyNewModelConfig(PretrainedConfig):\n    r\"\"\"\n    This is the configuration class to store the configuration of a [`MyNewModelModel`]. It is used to instantiate an MyNewModel\n    model according to the specified arguments, defining the model architecture. Instantiating a configuration with the\n    defaults will yield a similar configuration to that of the MyNewModel-7B.\n\n    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the\n    documentation from [`PretrainedConfig`] for more information.\n\n\n    Args:\n        vocab_size (`int`, *optional*, defaults to 32000):\n            Vocabulary size of the MyNewModel model. Defines the number of different tokens that can be represented by the\n            `inputs_ids` passed when calling [`MyNewModelModel`]\n        hidden_size (`int`, *optional*, defaults to 4096):\n            Dimension of the hidden representations.\n        intermediate_size (`int`, *optional*, defaults to 11008):\n            Dimension of the MLP representations.\n        num_hidden_layers (`int`, *optional*, defaults to 32):\n            Number of hidden layers in the Transformer decoder.\n        num_attention_heads (`int`, *optional*, defaults to 32):\n            Number of attention heads for each attention layer in the Transformer decoder.\n        num_key_value_heads (`int`, *optional*):\n            This is the number of key_value heads that should be used to implement Grouped Query Attention. If\n            `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if\n            `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When\n            converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed\n            by meanpooling all the original heads within that group. For more details checkout [this\n            paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to\n            `num_attention_heads`.\n        hidden_act (`str` or `function`, *optional*, defaults to `\"silu\"`):\n            The non-linear activation function (function or string) in the decoder.\n        max_position_embeddings (`int`, *optional*, defaults to 2048):\n            The maximum sequence length that this model might ever be used with. MyNewModel 1 supports up to 2048 tokens,\n            MyNewModel 2 up to 4096, CodeLlama up to 16384.\n        initializer_range (`float`, *optional*, defaults to 0.02):\n            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.\n        rms_norm_eps (`float`, *optional*, defaults to 1e-06):\n            The epsilon used by the rms normalization layers.\n        use_cache (`bool`, *optional*, defaults to `True`):\n            Whether or not the model should return the last key/values attentions (not used by all models). Only\n            relevant if `config.is_decoder=True`.\n        pad_token_id (`int`, *optional*):\n            Padding token id.\n        bos_token_id (`int`, *optional*, defaults to 1):\n            Beginning of stream token id.\n        eos_token_id (`int`, *optional*, defaults to 2):\n            End of stream token id.\n        pretraining_tp (`int`, *optional*, defaults to 1):\n            Experimental feature. Tensor parallelism rank used during pretraining. Please refer to [this\n            document](https://huggingface.co/docs/transformers/main/perf_train_gpu_many#tensor-parallelism) to\n            understand more about it. This value is necessary to ensure exact reproducibility of the pretraining\n            results. Please refer to [this issue](https://github.com/pytorch/pytorch/issues/76232).\n        tie_word_embeddings (`bool`, *optional*, defaults to `False`):\n            Whether to tie weight embeddings\n        rope_theta (`float`, *optional*, defaults to 10000.0):\n            The base period of the RoPE embeddings.\n        rope_scaling (`Dict`, *optional*):\n            Dictionary containing the scaling configuration for the RoPE embeddings. NOTE: if you apply new rope type\n            and you expect the model to work on longer `max_position_embeddings`, we recommend you to update this value\n            accordingly.\n            Expected contents:\n                `rope_type` (`str`):\n                    The sub-variant of RoPE to use. Can be one of ['default', 'linear', 'dynamic', 'yarn', 'longrope',\n                    'my_new_model3'], with 'default' being the original RoPE implementation.\n                `factor` (`float`, *optional*):\n                    Used with all rope types except 'default'. The scaling factor to apply to the RoPE embeddings. In\n                    most scaling types, a `factor` of x will enable the model to handle sequences of length x *\n                    original maximum pre-trained length.\n                `original_max_position_embeddings` (`int`, *optional*):\n                    Used with 'dynamic', 'longrope' and 'my_new_model3'. The original max position embeddings used during\n                    pretraining.\n                `attention_factor` (`float`, *optional*):\n                    Used with 'yarn' and 'longrope'. The scaling factor to be applied on the attention\n                    computation. If unspecified, it defaults to value recommended by the implementation, using the\n                    `factor` field to infer the suggested value.\n                `beta_fast` (`float`, *optional*):\n                    Only used with 'yarn'. Parameter to set the boundary for extrapolation (only) in the linear\n                    ramp function. If unspecified, it defaults to 32.\n                `beta_slow` (`float`, *optional*):\n                    Only used with 'yarn'. Parameter to set the boundary for interpolation (only) in the linear\n                    ramp function. If unspecified, it defaults to 1.\n                `short_factor` (`List[float]`, *optional*):\n                    Only used with 'longrope'. The scaling factor to be applied to short contexts (<\n                    `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden\n                    size divided by the number of attention heads divided by 2\n                `long_factor` (`List[float]`, *optional*):\n                    Only used with 'longrope'. The scaling factor to be applied to long contexts (<\n                    `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden\n                    size divided by the number of attention heads divided by 2\n                `low_freq_factor` (`float`, *optional*):\n                    Only used with 'my_new_model3'. Scaling factor applied to low frequency components of the RoPE\n                `high_freq_factor` (`float`, *optional*):\n                    Only used with 'my_new_model3'. Scaling factor applied to high frequency components of the RoPE\n        attention_bias (`bool`, *optional*, defaults to `False`):\n            Whether to use a bias in the query, key, value and output projection layers during self-attention.\n        attention_dropout (`float`, *optional*, defaults to 0.0):\n            The dropout ratio for the attention probabilities.\n        mlp_bias (`bool`, *optional*, defaults to `False`):\n            Whether to use a bias in up_proj, down_proj and gate_proj layers in the MLP layers.\n        head_dim (`int`, *optional*):\n            The attention head dimension. If None, it will default to hidden_size // num_attention_heads\n\n    ```python\n    >>> from transformers import MyNewModelModel, MyNewModelConfig\n\n    >>> # Initializing a MyNewModel my_new_model-7b style configuration\n    >>> configuration = MyNewModelConfig()\n\n    >>> # Initializing a model from the my_new_model-7b style configuration\n    >>> model = MyNewModelModel(configuration)\n\n    >>> # Accessing the model configuration\n    >>> configuration = model.config\n    ```\n        new_param (`int`, *optional*, defaults to `False`):\n            A fun new parameter\n    \"\"\"\n\n    model_type = \"my_new_model\"\n    keys_to_ignore_at_inference = [\"past_key_values\"]\n    # Default tensor parallel plan for base model `MyNewModelModel`\n    base_model_tp_plan = {\n        \"layers.*.self_attn.q_proj\": \"colwise\",\n        \"layers.*.self_attn.k_proj\": \"colwise\",\n        \"layers.*.self_attn.v_proj\": \"colwise\",\n        \"layers.*.self_attn.o_proj\": \"rowwise\",\n        \"layers.*.mlp.gate_proj\": \"colwise\",\n        \"layers.*.mlp.up_proj\": \"colwise\",\n        \"layers.*.mlp.down_proj\": \"rowwise\",\n    }\n\n    def __init__(\n        self,\n        vocab_size=32000,\n        hidden_size=4096,\n        intermediate_size=11008,\n        num_hidden_layers=32,\n        num_attention_heads=32,\n        num_key_value_heads=None,\n        hidden_act=\"silu\",\n        max_position_embeddings=2048,\n        initializer_range=0.02,\n        rms_norm_eps=1e-6,\n        use_cache=True,\n        pad_token_id=None,\n        bos_token_id=1,\n        eos_token_id=2,\n        pretraining_tp=1,\n        tie_word_embeddings=False,\n        rope_theta=10000.0,\n        rope_scaling=None,\n        attention_bias=False,\n        attention_dropout=0.0,\n        mlp_bias=True,\n        head_dim=None,\n        new_param=0,\n        **kwargs,\n    ):\n        super().__init__(\n            pad_token_id=pad_token_id,\n            bos_token_id=bos_token_id,\n            eos_token_id=eos_token_id,\n            tie_word_embeddings=tie_word_embeddings,\n            **kwargs,\n        )\n        self.vocab_size = vocab_size\n        self.max_position_embeddings = max_position_embeddings\n        self.hidden_size = hidden_size\n        self.intermediate_size = intermediate_size\n        self.num_hidden_layers = num_hidden_layers\n        self.num_attention_heads = num_attention_heads\n\n        # for backward compatibility\n        if num_key_value_heads is None:\n            num_key_value_heads = num_attention_heads\n\n        self.num_key_value_heads = num_key_value_heads\n        self.hidden_act = hidden_act\n        self.initializer_range = initializer_range\n        self.rms_norm_eps = rms_norm_eps\n        self.pretraining_tp = pretraining_tp\n        self.use_cache = use_cache\n        self.rope_theta = rope_theta\n        self.rope_scaling = rope_scaling\n        self.attention_bias = attention_bias\n        self.attention_dropout = attention_dropout\n        self.mlp_bias = mlp_bias\n        self.head_dim = head_dim if head_dim is not None else self.hidden_size // self.num_attention_heads\n        # Validate the correctness of rotary position embeddings parameters\n        # BC: if there is a 'type' field, copy it it to 'rope_type'.\n        if self.rope_scaling is not None and \"type\" in self.rope_scaling:\n            self.rope_scaling[\"rope_type\"] = self.rope_scaling[\"type\"]\n        rope_config_validation(self)\n\n\nSUMMARY:\nThis code defines the configuration for a custom transformer model called `MyNewModel`. It consists of various hyperparameters and settings, such as:\n\n* Model architecture (number of hidden layers, attention heads, intermediate size)\n* Training settings (vocab size, max position embeddings, initial optimizer range)\n* Hyperparameters (hidden act, beta test parameter)\n* Performance metrics and validation scheme\n* Output model type, key fields to ignore at inference, default tensor parallel plan\n\nThe code is intended for use with a transformer-based machine learning model, where the configuration defines how the model should be trained, processed, and evaluated. \n\nIt's often used in deep learning models such as those in NLP tasks like language translation or question-answering, although it is just a placeholder example of many more transformer based architectures that are built with the Hugging Face framework.\n\nOverall this code provides an object-oriented definition for how transformers should be constructed and configured to handle many aspects of their training and usage."}


created_at: 2025-02-05 09:13:41
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy_bert.py:35-96

metadata:
{"docstring": "", "end_line": 96, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy_bert.py", "name": "DummyBertEmbeddings", "start_line": 35, "type": "class", "chroma:document": "CODE:\n\nclass DummyBertEmbeddings(nn.Module):\n    \"\"\"Construct the embeddings from word, position and token_type embeddings.\"\"\"\n\n    def __init__(self, config):\n        super().__init__()\n        self.word_embeddings = nn.Embedding(config.vocab_size, config.hidden_size, padding_idx=config.pad_token_id)\n        self.position_embeddings = nn.Embedding(config.max_position_embeddings, config.hidden_size)\n        self.token_type_embeddings = nn.Embedding(config.type_vocab_size, config.hidden_size)\n\n        # self.LayerNorm is not snake-cased to stick with TensorFlow model variable name and be able to load\n        # any TensorFlow checkpoint file\n        self.LayerNorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)\n        self.dropout = nn.Dropout(config.hidden_dropout_prob)\n        # position_ids (1, len position emb) is contiguous in memory and exported when serialized\n        self.position_embedding_type = getattr(config, \"position_embedding_type\", \"absolute\")\n        self.register_buffer(\n            \"position_ids\", torch.arange(config.max_position_embeddings).expand((1, -1)), persistent=False\n        )\n        self.register_buffer(\n            \"token_type_ids\", torch.zeros(self.position_ids.size(), dtype=torch.long), persistent=False\n        )\n\n    def forward(\n        self,\n        input_ids: Optional[torch.LongTensor] = None,\n        token_type_ids: Optional[torch.LongTensor] = None,\n        position_ids: Optional[torch.LongTensor] = None,\n        inputs_embeds: Optional[torch.FloatTensor] = None,\n        past_key_values_length: int = 0,\n    ) -> torch.Tensor:\n        if input_ids is not None:\n            input_shape = input_ids.size()\n        else:\n            input_shape = inputs_embeds.size()[:-1]\n\n        seq_length = input_shape[1]\n\n        if position_ids is None:\n            position_ids = self.position_ids[:, past_key_values_length : seq_length + past_key_values_length]\n\n        # Setting the token_type_ids to the registered buffer in constructor where it is all zeros, which usually occurs\n        # when its auto-generated, registered buffer helps users when tracing the model without passing token_type_ids, solves\n        # issue #5664\n        if token_type_ids is None:\n            if hasattr(self, \"token_type_ids\"):\n                buffered_token_type_ids = self.token_type_ids[:, :seq_length]\n                buffered_token_type_ids_expanded = buffered_token_type_ids.expand(input_shape[0], seq_length)\n                token_type_ids = buffered_token_type_ids_expanded\n            else:\n                token_type_ids = torch.zeros(input_shape, dtype=torch.long, device=self.position_ids.device)\n\n        if inputs_embeds is None:\n            inputs_embeds = self.word_embeddings(input_ids)\n        token_type_embeddings = self.token_type_embeddings(token_type_ids)\n\n        embeddings = inputs_embeds + token_type_embeddings\n        if self.position_embedding_type == \"absolute\":\n            position_embeddings = self.position_embeddings(position_ids)\n            embeddings += position_embeddings\n        embeddings = self.LayerNorm(embeddings)\n        embeddings = self.dropout(embeddings)\n\n\nSUMMARY:\n**Summary:**\nThe `DummyBertEmbeddings` class is a PyTorch module responsible for constructing BERT-style embeddings from word, position, and token type inputs. It extends the PyTorch `nn.Module` class.\n\n**Purpose:** The main purpose of this class is to compute contextualized character-level representations by combining word embeddings with position and token type information.\n\n**Key Implementations:**\n* Embeds words using a learned embedding matrix (`word_embeddings`)\n* Apps `position_embeddings` from word embeddings based on `position_ids`\n* Applies `token_type_embeddings` using an embedding layer for token types\n* Employs multi-head attention mechanisms through the `LayerNorm` and `dropout` layers\n\n**Notable Dependencies:**\n* PyTorch library (specifically, `torch.nn`)\n* The Hugging Face Transformers library is referenced indirectly through its usage of `config` objects"}


created_at: 2025-02-05 09:13:41
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy_bert.py:99-230

metadata:
{"docstring": "", "end_line": 230, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy_bert.py", "name": "DummyBertSelfAttention", "start_line": 99, "type": "class", "chroma:document": "CODE:\n\nclass DummyBertSelfAttention(nn.Module):\n    def __init__(self, config, position_embedding_type=None):\n        super().__init__()\n        if config.hidden_size % config.num_attention_heads != 0 and not hasattr(config, \"embedding_size\"):\n            raise ValueError(\n                f\"The hidden size ({config.hidden_size}) is not a multiple of the number of attention \"\n                f\"heads ({config.num_attention_heads})\"\n            )\n\n        self.num_attention_heads = config.num_attention_heads\n        self.attention_head_size = int(config.hidden_size / config.num_attention_heads)\n        self.all_head_size = self.num_attention_heads * self.attention_head_size\n\n        self.query = nn.Linear(config.hidden_size, self.all_head_size)\n        self.key = nn.Linear(config.hidden_size, self.all_head_size)\n        self.value = nn.Linear(config.hidden_size, self.all_head_size)\n\n        self.dropout = nn.Dropout(config.attention_probs_dropout_prob)\n        self.position_embedding_type = position_embedding_type or getattr(\n            config, \"position_embedding_type\", \"absolute\"\n        )\n        if self.position_embedding_type == \"relative_key\" or self.position_embedding_type == \"relative_key_query\":\n            self.max_position_embeddings = config.max_position_embeddings\n            self.distance_embedding = nn.Embedding(2 * config.max_position_embeddings - 1, self.attention_head_size)\n\n        self.is_decoder = config.is_decoder\n\n    def transpose_for_scores(self, x: torch.Tensor) -> torch.Tensor:\n        new_x_shape = x.size()[:-1] + (self.num_attention_heads, self.attention_head_size)\n        x = x.view(new_x_shape)\n        return x.permute(0, 2, 1, 3)\n\n    def forward(\n        self,\n        hidden_states: torch.Tensor,\n        attention_mask: Optional[torch.FloatTensor] = None,\n        head_mask: Optional[torch.FloatTensor] = None,\n        encoder_hidden_states: Optional[torch.FloatTensor] = None,\n        encoder_attention_mask: Optional[torch.FloatTensor] = None,\n        past_key_value: Optional[Tuple[Tuple[torch.FloatTensor]]] = None,\n        output_attentions: Optional[bool] = False,\n    ) -> Tuple[torch.Tensor]:\n        mixed_query_layer = self.query(hidden_states)\n\n        # If this is instantiated as a cross-attention module, the keys\n        # and values come from an encoder; the attention mask needs to be\n        # such that the encoder's padding tokens are not attended to.\n        is_cross_attention = encoder_hidden_states is not None\n\n        if is_cross_attention and past_key_value is not None:\n            # reuse k,v, cross_attentions\n            key_layer = past_key_value[0]\n            value_layer = past_key_value[1]\n            attention_mask = encoder_attention_mask\n        elif is_cross_attention:\n            key_layer = self.transpose_for_scores(self.key(encoder_hidden_states))\n            value_layer = self.transpose_for_scores(self.value(encoder_hidden_states))\n            attention_mask = encoder_attention_mask\n        elif past_key_value is not None:\n            key_layer = self.transpose_for_scores(self.key(hidden_states))\n            value_layer = self.transpose_for_scores(self.value(hidden_states))\n            key_layer = torch.cat([past_key_value[0], key_layer], dim=2)\n            value_layer = torch.cat([past_key_value[1], value_layer], dim=2)\n        else:\n            key_layer = self.transpose_for_scores(self.key(hidden_states))\n            value_layer = self.transpose_for_scores(self.value(hidden_states))\n\n        query_layer = self.transpose_for_scores(mixed_query_layer)\n\n        use_cache = past_key_value is not None\n        if self.is_decoder:\n            # if cross_attention save Tuple(torch.Tensor, torch.Tensor) of all cross attention key/value_states.\n            # Further calls to cross_attention layer can then reuse all cross-attention\n            # key/value_states (first \"if\" case)\n            # if uni-directional self-attention (decoder) save Tuple(torch.Tensor, torch.Tensor) of\n            # all previous decoder key/value_states. Further calls to uni-directional self-attention\n            # can concat previous decoder key/value_states to current projected key/value_states (third \"elif\" case)\n            # if encoder bi-directional self-attention `past_key_value` is always `None`\n            past_key_value = (key_layer, value_layer)\n\n        # Take the dot product between \"query\" and \"key\" to get the raw attention scores.\n        attention_scores = torch.matmul(query_layer, key_layer.transpose(-1, -2))\n\n        if self.position_embedding_type == \"relative_key\" or self.position_embedding_type == \"relative_key_query\":\n            query_length, key_length = query_layer.shape[2], key_layer.shape[2]\n            if use_cache:\n                position_ids_l = torch.tensor(key_length - 1, dtype=torch.long, device=hidden_states.device).view(\n                    -1, 1\n                )\n            else:\n                position_ids_l = torch.arange(query_length, dtype=torch.long, device=hidden_states.device).view(-1, 1)\n            position_ids_r = torch.arange(key_length, dtype=torch.long, device=hidden_states.device).view(1, -1)\n            distance = position_ids_l - position_ids_r\n\n            positional_embedding = self.distance_embedding(distance + self.max_position_embeddings - 1)\n            positional_embedding = positional_embedding.to(dtype=query_layer.dtype)  # fp16 compatibility\n\n            if self.position_embedding_type == \"relative_key\":\n                relative_position_scores = torch.einsum(\"bhld,lrd->bhlr\", query_layer, positional_embedding)\n                attention_scores = attention_scores + relative_position_scores\n            elif self.position_embedding_type == \"relative_key_query\":\n                relative_position_scores_query = torch.einsum(\"bhld,lrd->bhlr\", query_layer, positional_embedding)\n                relative_position_scores_key = torch.einsum(\"bhrd,lrd->bhlr\", key_layer, positional_embedding)\n                attention_scores = attention_scores + relative_position_scores_query + relative_position_scores_key\n\n        attention_scores = attention_scores / math.sqrt(self.attention_head_size)\n        if attention_mask is not None:\n            # Apply the attention mask is (precomputed for all layers in DummyBertModel forward() function)\n            attention_scores = attention_scores + attention_mask\n\n        # Normalize the attention scores to probabilities.\n        attention_probs = nn.functional.softmax(attention_scores, dim=-1)\n\n        # This is actually dropping out entire tokens to attend to, which might\n        # seem a bit unusual, but is taken from the original Transformer paper.\n        attention_probs = self.dropout(attention_probs)\n\n        # Mask heads if we want to\n        if head_mask is not None:\n            attention_probs = attention_probs * head_mask\n\n        context_layer = torch.matmul(attention_probs, value_layer)\n\n        context_layer = context_layer.permute(0, 2, 1, 3).contiguous()\n        new_context_layer_shape = context_layer.size()[:-2] + (self.all_head_size,)\n        context_layer = context_layer.view(new_context_layer_shape)\n\n        outputs = (context_layer, attention_probs) if output_attentions else (context_layer,)\n\n        if self.is_decoder:\n            outputs = outputs + (past_key_value,)\n\n\nSUMMARY:\n**Class Summary**\n\n`DummyBertSelfAttention`\n======================\n\nA PyTorch implementation of the multi-head self-attention mechanism from the BERT model architecture.\n\n**Primary Purpose**\n-------------------\n\nThis class computes the attention scores and context layer for self-supervised learning or next sentence prediction tasks.\n\n**Key Implementation Details**\n-----------------------------\n\n* Uses a multihead decoder attention mechanism with dimensionality reduction (dot product).\n* Applies position embeddings for relative position encoding.\n* Supports different types of position embeddings: `absolute`, `relative_key`, and `relative_key_query`.\n* Masks heads using a provided head mask tensor, if applicable.\n\n**Notable Dependencies or Requirements**\n------------------------------------------\n\n* `torch`: PyTorch library with neural network module (`nn`).\n* `math.sqrt()`: Mathematical operations to compute square roots.\n* `nn.functional.softmax()` : Softmax activation function for normalizing attention scores.\n* `nn.Dropout()`: Dropout probability layer to prevent overfitting.\n\n**Method Summary**\n------------------\n\n### `__init__(config, position_embedding_type=None)` init\n\n Initializes the module with configuration and optional position embedding type.\n\n### `transpose_for_scores(x: torch.Tensor) -> torch.Tensor` transpose for scores\n\n Transposes input tensor for multihead self-attention mechanism computation.\n\n### `forward(hidden_states: torch.Tensor, ...)` forward pass\n\n Computes attention scores and context layer for self-supervised learning or next sentence prediction tasks.\n \n Returns a tuple of `(context_layer, attention_probs)` if `output_attentions` is True; otherwise returns only the first element."}


created_at: 2025-02-05 09:13:41
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy_bert.py:233-331

metadata:
{"docstring": "", "end_line": 331, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy_bert.py", "name": "DummyBertSdpaSelfAttention", "start_line": 233, "type": "class", "chroma:document": "CODE:\n\nclass DummyBertSdpaSelfAttention(DummyBertSelfAttention):\n    def __init__(self, config, position_embedding_type=None):\n        super().__init__(config, position_embedding_type=position_embedding_type)\n        self.dropout_prob = config.attention_probs_dropout_prob\n        self.require_contiguous_qkv = version.parse(get_torch_version()) < version.parse(\"2.2.0\")\n\n    # Adapted from DummyBertSelfAttention\n    def forward(\n        self,\n        hidden_states: torch.Tensor,\n        attention_mask: Optional[torch.Tensor] = None,\n        head_mask: Optional[torch.FloatTensor] = None,\n        encoder_hidden_states: Optional[torch.FloatTensor] = None,\n        encoder_attention_mask: Optional[torch.FloatTensor] = None,\n        past_key_value: Optional[Tuple[Tuple[torch.FloatTensor]]] = None,\n        output_attentions: Optional[bool] = False,\n    ) -> Tuple[torch.Tensor]:\n        if self.position_embedding_type != \"absolute\" or output_attentions or head_mask is not None:\n            # TODO: Improve this warning with e.g. `model.config._attn_implementation = \"manual\"` once implemented.\n            logger.warning_once(\n                \"DummyBertSdpaSelfAttention is used but `torch.nn.functional.scaled_dot_product_attention` does not support \"\n                \"non-absolute `position_embedding_type` or `output_attentions=True` or `head_mask`. Falling back to \"\n                \"the manual attention implementation, but specifying the manual implementation will be required from \"\n                \"Transformers version v5.0.0 onwards. This warning can be removed using the argument \"\n                '`attn_implementation=\"eager\"` when loading the model.'\n            )\n            return super().forward(\n                hidden_states,\n                attention_mask,\n                head_mask,\n                encoder_hidden_states,\n                encoder_attention_mask,\n                past_key_value,\n                output_attentions,\n            )\n\n        bsz, tgt_len, _ = hidden_states.size()\n\n        query_layer = self.transpose_for_scores(self.query(hidden_states))\n\n        # If this is instantiated as a cross-attention module, the keys and values come from an encoder; the attention\n        # mask needs to be such that the encoder's padding tokens are not attended to.\n        is_cross_attention = encoder_hidden_states is not None\n\n        current_states = encoder_hidden_states if is_cross_attention else hidden_states\n        attention_mask = encoder_attention_mask if is_cross_attention else attention_mask\n\n        # Check `seq_length` of `past_key_value` == `len(current_states)` to support prefix tuning\n        if is_cross_attention and past_key_value and past_key_value[0].shape[2] == current_states.shape[1]:\n            key_layer, value_layer = past_key_value\n        else:\n            key_layer = self.transpose_for_scores(self.key(current_states))\n            value_layer = self.transpose_for_scores(self.value(current_states))\n            if past_key_value is not None and not is_cross_attention:\n                key_layer = torch.cat([past_key_value[0], key_layer], dim=2)\n                value_layer = torch.cat([past_key_value[1], value_layer], dim=2)\n\n        if self.is_decoder:\n            # if cross_attention save Tuple(torch.Tensor, torch.Tensor) of all cross attention key/value_states.\n            # Further calls to cross_attention layer can then reuse all cross-attention\n            # key/value_states (first \"if\" case)\n            # if uni-directional self-attention (decoder) save Tuple(torch.Tensor, torch.Tensor) of\n            # all previous decoder key/value_states. Further calls to uni-directional self-attention\n            # can concat previous decoder key/value_states to current projected key/value_states (third \"elif\" case)\n            # if encoder bi-directional self-attention `past_key_value` is always `None`\n            past_key_value = (key_layer, value_layer)\n\n        # SDPA with memory-efficient backend is broken in torch==2.1.2 when using non-contiguous inputs and a custom\n        # attn_mask, so we need to call `.contiguous()` here. This was fixed in torch==2.2.0.\n        # Reference: https://github.com/pytorch/pytorch/issues/112577\n        if self.require_contiguous_qkv and query_layer.device.type == \"cuda\" and attention_mask is not None:\n            query_layer = query_layer.contiguous()\n            key_layer = key_layer.contiguous()\n            value_layer = value_layer.contiguous()\n\n        # We dispatch to SDPA's Flash Attention or Efficient kernels via this `is_causal` if statement instead of an inline conditional assignment\n        # in SDPA to support both torch.compile's dynamic shapes and full graph options. An inline conditional prevents dynamic shapes from compiling.\n        # The tgt_len > 1 is necessary to match with AttentionMaskConverter.to_causal_4d that does not create\n        # a causal mask in case tgt_len == 1.\n        is_causal = (\n            True if self.is_decoder and not is_cross_attention and attention_mask is None and tgt_len > 1 else False\n        )\n\n        attn_output = torch.nn.functional.scaled_dot_product_attention(\n            query_layer,\n            key_layer,\n            value_layer,\n            attn_mask=attention_mask,\n            dropout_p=self.dropout_prob if self.training else 0.0,\n            is_causal=is_causal,\n        )\n\n        attn_output = attn_output.transpose(1, 2)\n        attn_output = attn_output.reshape(bsz, tgt_len, self.all_head_size)\n\n        outputs = (attn_output,)\n        if self.is_decoder:\n            outputs = outputs + (past_key_value,)\n\n\nSUMMARY:\n**Class Summary: DummyBertSdpaSelfAttention**\n\nThe `DummyBertSdpaSelfAttention` class extends the `DummyBertSelfAttention` class, inheriting its functionality. It is used to simulate the behavior of the transformers library's `scaled_dot_product_attention` function.\n\n**Primary Purpose**\n\nThe primary purpose of this class is to handle non-absolute position embeddings and support torch.compile's dynamic shapes for fast inference.\n\n**Key Implementation Details**\n\n*   Uses `torch.nn.functional.scaled_dot_product_attention` to compute attention outputs.\n*   Dispatches to SDPA's Flash Attention or Efficient kernels via an `is_causal` if statement.\n*   Applies contouring operations for efficient computation on non-contiguous inputs.\n*   Handles cross-attention and prefix tuning support.\n\n**Notable Dependencies and Requirements**\n\n*   Depends on the `DummyBertSelfAttention` class and uses its `forward` method as a base implementation.\n*   Requires torch 2.2.0 or later due to fixes in SDPA's Flash Attention backend.\n\n**(External References)**\n\n-   `torch.nn.functional.scaled_dot_product_attention`\n-   ` DummyBertSelfAttention` class\n-   `config.attention_probs_dropout_prob`\n\nNote: The exact reference to the corresponding Transformers library function mentioned above can be found at :https://github.com/huggingfacetransformers/Transformers"}


created_at: 2025-02-05 09:13:41
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy_bert.py:334-345

metadata:
{"docstring": "", "end_line": 345, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy_bert.py", "name": "DummyBertSelfOutput", "start_line": 334, "type": "class", "chroma:document": "CODE:\n\nclass DummyBertSelfOutput(nn.Module):\n    def __init__(self, config):\n        super().__init__()\n        self.dense = nn.Linear(config.hidden_size, config.hidden_size)\n        self.LayerNorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)\n        self.dropout = nn.Dropout(config.hidden_dropout_prob)\n\n    def forward(self, hidden_states: torch.Tensor, input_tensor: torch.Tensor) -> torch.Tensor:\n        hidden_states = self.dense(hidden_states)\n        hidden_states = self.dropout(hidden_states)\n        hidden_states = self.LayerNorm(hidden_states + input_tensor)\n\n\nSUMMARY:\n**Summary**\n\nThis is a PyTorch implementation of a self-output transformation module for BERT, inspired by the Transformer architecture. The `DummyBertSelfOutput` class transforms the input-hidden-states tensor using a linear layer (`self.dense`), dropout regularization (`self.dropout`), and layer normalization (`self.LayerNorm`). The primary purpose is to normalize and conditionally scale the hidden states before outputting them.\n\n**Key Implementation Details**\n\n1. Linear transformation: `self.dense(nn.Linear(config.hidden_size, config.hidden_size))`\n2. Dropout regularization: `self.dropout(config.hidden_dropout_prob)`\n3. Layer Normalization: `self.LayerNorm(config.layer_norm_eps)`\n\n**Notable Dependencies and Requirements**\n\n* PyTorch (`torch`)\n* Hugging Face Transformer library\n* BERT configuration (`config`)"}


created_at: 2025-02-05 09:13:41
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy_bert.py:354-402

metadata:
{"docstring": "", "end_line": 402, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy_bert.py", "name": "DummyBertAttention", "start_line": 354, "type": "class", "chroma:document": "CODE:\n\nclass DummyBertAttention(nn.Module):\n    def __init__(self, config, position_embedding_type=None):\n        super().__init__()\n        self.self = DUMMY_BERT_SELF_ATTENTION_CLASSES[config._attn_implementation](\n            config, position_embedding_type=position_embedding_type\n        )\n        self.output = DummyBertSelfOutput(config)\n        self.pruned_heads = set()\n\n    def prune_heads(self, heads):\n        if len(heads) == 0:\n            return\n        heads, index = find_pruneable_heads_and_indices(\n            heads, self.self.num_attention_heads, self.self.attention_head_size, self.pruned_heads\n        )\n\n        # Prune linear layers\n        self.self.query = prune_linear_layer(self.self.query, index)\n        self.self.key = prune_linear_layer(self.self.key, index)\n        self.self.value = prune_linear_layer(self.self.value, index)\n        self.output.dense = prune_linear_layer(self.output.dense, index, dim=1)\n\n        # Update hyper params and store pruned heads\n        self.self.num_attention_heads = self.self.num_attention_heads - len(heads)\n        self.self.all_head_size = self.self.attention_head_size * self.self.num_attention_heads\n        self.pruned_heads = self.pruned_heads.union(heads)\n\n    def forward(\n        self,\n        hidden_states: torch.Tensor,\n        attention_mask: Optional[torch.FloatTensor] = None,\n        head_mask: Optional[torch.FloatTensor] = None,\n        encoder_hidden_states: Optional[torch.FloatTensor] = None,\n        encoder_attention_mask: Optional[torch.FloatTensor] = None,\n        past_key_value: Optional[Tuple[Tuple[torch.FloatTensor]]] = None,\n        output_attentions: Optional[bool] = False,\n    ) -> Tuple[torch.Tensor]:\n        self_outputs = self.self(\n            hidden_states,\n            attention_mask,\n            head_mask,\n            encoder_hidden_states,\n            encoder_attention_mask,\n            past_key_value,\n            output_attentions,\n        )\n        attention_output = self.output(self_outputs[0], hidden_states)\n        outputs = (attention_output,) + self_outputs[1:]  # add attentions if we output them\n\n\nSUMMARY:\n**Concise Summary:**\n\nThe `DummyBertAttention` class implements the BERT attention mechanism. It uses a `SelfAttention` module to compute attention weights and a `SelfOutput` module to apply the learned embedding.\n\n**Key Implementation Details:**\n\n*   **Attention Mechanism:** The `prune_heads` method applies attention head pruning based on the provided heads, updating hyperparameters accordingly.\n*   **Forward Method:** The `forward` method first computes attention outputs using `self.self` and then passes them through `self.output`.\n*   **Notable Dependencies:**\n\n    *   `DummyBertSelfAttention_CLASSES`: A mapping class for BERT attention classes.\n    *   `DummyBertSelfOutput`: A module for self-output operation.\n\n**External Dependencies:**\n\n*   `torch` library\n*   `find_pruneable_heads_and_indices`, `prune_linear_layer`, and other utility functions in the codebase."}


created_at: 2025-02-05 09:13:41
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy_bert.py:405-417

metadata:
{"docstring": "", "end_line": 417, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy_bert.py", "name": "DummyBertIntermediate", "start_line": 405, "type": "class", "chroma:document": "CODE:\n\nclass DummyBertIntermediate(nn.Module):\n    def __init__(self, config):\n        super().__init__()\n        self.dense = nn.Linear(config.hidden_size, config.intermediate_size)\n        if isinstance(config.hidden_act, str):\n            self.intermediate_act_fn = ACT2FN[config.hidden_act]\n        else:\n            self.intermediate_act_fn = config.hidden_act\n\n    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:\n        hidden_states = self.dense(hidden_states)\n        hidden_states = self.intermediate_act_fn(hidden_states)\n\n\nSUMMARY:\n**Summary**\n\nThe `DummyBertIntermediate` class inherits from PyTorch's `nn.Module`. It implements the intermediate layer of BERT, comprising:\n\n* A linear dense layer (`self.dense`) with an input size of `config.hidden_size` and output size of `config.intermediate_size`.\n* If `config.hidden_act` is a string, it calls the corresponding activation function from the `ACT2FN` dictionary. Otherwise, it uses `config.hidden_act` as the activation function (`self.intermediate_act_fn`).\n* The forward pass applies self.dense to input `hidden_states`, followed by the application of `self.intermediate_act_fn`.\n\n**Key Details**\n\n* Input size: `config.hidden_size`\n* Output size: `config(intermediate_size)`\n* Activation function:\n\t+ If string, uses dictionary lookup\n\t+ Otherwise, uses `config.hidden_act`\n\n**Dependencies & Requirements**\n\nNone explicitly mentioned. However, it's implied that `config` object contains the necessary configuration parameters (`hidden_size`, `intermediate_size`, and `hidden_act`). Additionally, PyTorch's neural network module (`nn.Module`) is required."}


created_at: 2025-02-05 09:13:41
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy_bert.py:420-431

metadata:
{"docstring": "", "end_line": 431, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy_bert.py", "name": "DummyBertOutput", "start_line": 420, "type": "class", "chroma:document": "CODE:\n\nclass DummyBertOutput(nn.Module):\n    def __init__(self, config):\n        super().__init__()\n        self.dense = nn.Linear(config.intermediate_size, config.hidden_size)\n        self.LayerNorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)\n        self.dropout = nn.Dropout(config.hidden_dropout_prob)\n\n    def forward(self, hidden_states: torch.Tensor, input_tensor: torch.Tensor) -> torch.Tensor:\n        hidden_states = self.dense(hidden_states)\n        hidden_states = self.dropout(hidden_states)\n        hidden_states = self.LayerNorm(hidden_states + input_tensor)\n\n\nSUMMARY:\n**Summary**\n\nThe `DummyBertOutput` class is an implementation of a PyTorch neural network module (`nn.Module`) designed to mimic the output layer of BERT (Bidirectional Encoder Representations from Transformers). It takes two inputs: `hidden_states` and `input_tensor`.\n\n**Primary Purpose**\nThe main purpose of this class is to process `hidden_states` using a linear transformation, followed by a dropout regularization, and finally applying layer normalization with an additional input adjustment.\n\n**Key Implementation Details**\n\n*   Linear Transformation (`self.dense`)\n    *   Input: `hidden_states` (torch.Tensor)\n    *   Output: transformed tensor\n*   Dropout Regularization (`self.dropout`)\n    *   Input: transformed tensor from linear transformation\n    *   Output: regularized tensor after dropout\n*   Layer Normalization (`self.LayerNorm`)\n    *   Input: combination of output from linear transformation, dropout regularization, and input_tensor\n    *   Output: normalized tensor\n\n**Notable Dependencies**\n\n*   `nn.Module` (PyTorch neural network module)\n*   `torch.Tensor`\n*   `nn.Linear`, `nn.Dropout`, and `nn.LayerNorm` modules ( PyTorch linear and normalization layers)\n\n**No external dependencies listed.**"}


created_at: 2025-02-05 09:13:41
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy_bert.py:434-517

metadata:
{"docstring": "", "end_line": 517, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy_bert.py", "name": "DummyBertLayer", "start_line": 434, "type": "class", "chroma:document": "CODE:\n\nclass DummyBertLayer(nn.Module):\n    def __init__(self, config):\n        super().__init__()\n        self.chunk_size_feed_forward = config.chunk_size_feed_forward\n        self.seq_len_dim = 1\n        self.attention = DummyBertAttention(config)\n        self.is_decoder = config.is_decoder\n        self.add_cross_attention = config.add_cross_attention\n        if self.add_cross_attention:\n            if not self.is_decoder:\n                raise ValueError(f\"{self} should be used as a decoder model if cross attention is added\")\n            self.crossattention = DummyBertAttention(config, position_embedding_type=\"absolute\")\n        self.intermediate = DummyBertIntermediate(config)\n        self.output = DummyBertOutput(config)\n\n    def forward(\n        self,\n        hidden_states: torch.Tensor,\n        attention_mask: Optional[torch.FloatTensor] = None,\n        head_mask: Optional[torch.FloatTensor] = None,\n        encoder_hidden_states: Optional[torch.FloatTensor] = None,\n        encoder_attention_mask: Optional[torch.FloatTensor] = None,\n        past_key_value: Optional[Tuple[Tuple[torch.FloatTensor]]] = None,\n        output_attentions: Optional[bool] = False,\n    ) -> Tuple[torch.Tensor]:\n        # decoder uni-directional self-attention cached key/values tuple is at positions 1,2\n        self_attn_past_key_value = past_key_value[:2] if past_key_value is not None else None\n        self_attention_outputs = self.attention(\n            hidden_states,\n            attention_mask,\n            head_mask,\n            output_attentions=output_attentions,\n            past_key_value=self_attn_past_key_value,\n        )\n        attention_output = self_attention_outputs[0]\n\n        # if decoder, the last output is tuple of self-attn cache\n        if self.is_decoder:\n            outputs = self_attention_outputs[1:-1]\n            present_key_value = self_attention_outputs[-1]\n        else:\n            outputs = self_attention_outputs[1:]  # add self attentions if we output attention weights\n\n        cross_attn_present_key_value = None\n        if self.is_decoder and encoder_hidden_states is not None:\n            if not hasattr(self, \"crossattention\"):\n                raise ValueError(\n                    f\"If `encoder_hidden_states` are passed, {self} has to be instantiated with cross-attention layers\"\n                    \" by setting `config.add_cross_attention=True`\"\n                )\n\n            # cross_attn cached key/values tuple is at positions 3,4 of past_key_value tuple\n            cross_attn_past_key_value = past_key_value[-2:] if past_key_value is not None else None\n            cross_attention_outputs = self.crossattention(\n                attention_output,\n                attention_mask,\n                head_mask,\n                encoder_hidden_states,\n                encoder_attention_mask,\n                cross_attn_past_key_value,\n                output_attentions,\n            )\n            attention_output = cross_attention_outputs[0]\n            outputs = outputs + cross_attention_outputs[1:-1]  # add cross attentions if we output attention weights\n\n            # add cross-attn cache to positions 3,4 of present_key_value tuple\n            cross_attn_present_key_value = cross_attention_outputs[-1]\n            present_key_value = present_key_value + cross_attn_present_key_value\n\n        layer_output = apply_chunking_to_forward(\n            self.feed_forward_chunk, self.chunk_size_feed_forward, self.seq_len_dim, attention_output\n        )\n        outputs = (layer_output,) + outputs\n\n        # if decoder, return the attn key/values as the last output\n        if self.is_decoder:\n            outputs = outputs + (present_key_value,)\n\n        return outputs\n\n    def feed_forward_chunk(self, attention_output):\n        intermediate_output = self.intermediate(attention_output)\n        layer_output = self.output(intermediate_output, attention_output)\n\n\nSUMMARY:\n**Class Summary: DummyBertLayer**\n\nThe `DummyBertLayer` class represents a single layer in the BERT architecture. It is a PyTorch module that utilizes multi-head self-attention and feed-forward networks.\n\n**Primary Purpose:** To process input data through self-attention and feed-forward networks, producing output features.\n\n**Key Implementation Details:**\n\n*   The `forward` method processes input tensors for self-attention and cross-attention. It returns a tuple of output features.\n*   Two critical components are the `DummyBertAttention` and `DummyBertIntermediate` modules, which implement attention mechanisms and feed-forward networks, respectively.\n\n**Notable Dependencies or Requirements:**\n\n*   Uses `torch.Tensor`, `Optional[ torch.FloatTensor]`, and other PyTorch-related classes/types for tensor operations.\n*   Incorporates dependency on `DummyBertAttention` and `DummyBertIntermediate` modules (not shown in this code snippet)."}


created_at: 2025-02-05 09:13:41
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy_bert.py:520-610

metadata:
{"docstring": "", "end_line": 610, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy_bert.py", "name": "DummyBertEncoder", "start_line": 520, "type": "class", "chroma:document": "CODE:\n\nclass DummyBertEncoder(nn.Module):\n    def __init__(self, config):\n        super().__init__()\n        self.config = config\n        self.layer = nn.ModuleList([DummyBertLayer(config) for _ in range(config.num_hidden_layers)])\n        self.gradient_checkpointing = False\n\n    def forward(\n        self,\n        hidden_states: torch.Tensor,\n        attention_mask: Optional[torch.FloatTensor] = None,\n        head_mask: Optional[torch.FloatTensor] = None,\n        encoder_hidden_states: Optional[torch.FloatTensor] = None,\n        encoder_attention_mask: Optional[torch.FloatTensor] = None,\n        past_key_values: Optional[Tuple[Tuple[torch.FloatTensor]]] = None,\n        use_cache: Optional[bool] = None,\n        output_attentions: Optional[bool] = False,\n        output_hidden_states: Optional[bool] = False,\n        return_dict: Optional[bool] = True,\n    ) -> Union[Tuple[torch.Tensor], BaseModelOutputWithPastAndCrossAttentions]:\n        all_hidden_states = () if output_hidden_states else None\n        all_self_attentions = () if output_attentions else None\n        all_cross_attentions = () if output_attentions and self.config.add_cross_attention else None\n\n        if self.gradient_checkpointing and self.training:\n            if use_cache:\n                logger.warning_once(\n                    \"`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...\"\n                )\n                use_cache = False\n\n        next_decoder_cache = () if use_cache else None\n        for i, layer_module in enumerate(self.layer):\n            if output_hidden_states:\n                all_hidden_states = all_hidden_states + (hidden_states,)\n\n            layer_head_mask = head_mask[i] if head_mask is not None else None\n            past_key_value = past_key_values[i] if past_key_values is not None else None\n\n            if self.gradient_checkpointing and self.training:\n                layer_outputs = self._gradient_checkpointing_func(\n                    layer_module.__call__,\n                    hidden_states,\n                    attention_mask,\n                    layer_head_mask,\n                    encoder_hidden_states,\n                    encoder_attention_mask,\n                    past_key_value,\n                    output_attentions,\n                )\n            else:\n                layer_outputs = layer_module(\n                    hidden_states,\n                    attention_mask,\n                    layer_head_mask,\n                    encoder_hidden_states,\n                    encoder_attention_mask,\n                    past_key_value,\n                    output_attentions,\n                )\n\n            hidden_states = layer_outputs[0]\n            if use_cache:\n                next_decoder_cache += (layer_outputs[-1],)\n            if output_attentions:\n                all_self_attentions = all_self_attentions + (layer_outputs[1],)\n                if self.config.add_cross_attention:\n                    all_cross_attentions = all_cross_attentions + (layer_outputs[2],)\n\n        if output_hidden_states:\n            all_hidden_states = all_hidden_states + (hidden_states,)\n\n        if not return_dict:\n            return tuple(\n                v\n                for v in [\n                    hidden_states,\n                    next_decoder_cache,\n                    all_hidden_states,\n                    all_self_attentions,\n                    all_cross_attentions,\n                ]\n                if v is not None\n            )\n        return BaseModelOutputWithPastAndCrossAttentions(\n            last_hidden_state=hidden_states,\n            past_key_values=next_decoder_cache,\n            hidden_states=all_hidden_states,\n            attentions=all_self_attentions,\n            cross_attentions=all_cross_attentions,\n\n\nSUMMARY:\n**Concise Summary:**\n\nThe `DummyBertEncoder` class is an encoder layer for the BERT model, implementing the `nn.Module` interface. It takes in various input tensors (`hidden_states`, `attention_mask`, etc.) and transforms them through self-attention mechanisms.\n\n**Key Implementation Details:**\n\n* The encoder consists of multiple identical `DummyBertLayer` instances wrapped within a `ModuleList`.\n* Each layer runs through a forward pass, applying self-attention to the input `hidden_states`. If gradient checkpointing is enabled, it falls back to a fallback implementation.\n* After each layer's forward pass, it updates the cache and attentions outputs.\n\n**Notable Dependencies and Requirements:**\n\n* The class depends on `nn.Module`, `torch.Tensor`, and various PyTorch modules (e.g., `nn.ModuleList`, `Optional`).\n* It uses `logger.warning_once` for logging warnings related to gradient checkpointing.\n* The class implements the `BaseModelOutputWithPastAndCrossAttentions` interface, which suggests an integration with larger PyTorch-based models.\n* External dependencies are implicitly required through the use of PyTorch modules and classes."}


created_at: 2025-02-05 09:13:41
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy_bert.py:613-625

metadata:
{"docstring": "", "end_line": 625, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy_bert.py", "name": "DummyBertPooler", "start_line": 613, "type": "class", "chroma:document": "CODE:\n\nclass DummyBertPooler(nn.Module):\n    def __init__(self, config):\n        super().__init__()\n        self.dense = nn.Linear(config.hidden_size, config.hidden_size)\n        self.activation = nn.Tanh()\n\n    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:\n        # We \"pool\" the model by simply taking the hidden state corresponding\n        # to the first token.\n        first_token_tensor = hidden_states[:, 0]\n        pooled_output = self.dense(first_token_tensor)\n        pooled_output = self.activation(pooled_output)\n\n\nSUMMARY:\n**Summary:**\n\n`DummyBertPooler` is a PyTorch neural network module implementing a simplified BERT pooler. It takes in `hidden_states` and outputs the pooled representation of the first token using a linear transformation (`self.dense`) followed by the Tanh activation function (`self.activation`). \n\n**Key Details:**\n- Takes `config.hidden_size` as input and output.\n- Utilizes `nn.Linear` for linear transformation and `nn.Tanh` for activation function.\n\n**Dependencies:** \n- References `torch.Tensor`\n- Inherited interfaces: `nn.Module`, `nn.Linear`\n\n**Notable Requirements:**\n- Inherits from PyTorch's neural network module (`nn.Module`)\n- Import of specific activation functions (`nn.Tanh`) and linear transformation (`nn.Linear`)."}


created_at: 2025-02-05 09:13:41
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy_bert.py:628-698

metadata:
{"docstring": "", "end_line": 698, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy_bert.py", "name": "load_tf_weights_in_dummy_bert", "start_line": 628, "type": "function", "chroma:document": "CODE:\n\ndef load_tf_weights_in_dummy_bert(model, config, tf_checkpoint_path):\n    \"\"\"Load tf checkpoints in a pytorch model.\"\"\"\n    try:\n        import re\n\n        import numpy as np\n        import tensorflow as tf\n    except ImportError:\n        logger.error(\n            \"Loading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see \"\n            \"https://www.tensorflow.org/install/ for installation instructions.\"\n        )\n        raise\n    tf_path = os.path.abspath(tf_checkpoint_path)\n    logger.info(f\"Converting TensorFlow checkpoint from {tf_path}\")\n    # Load weights from TF model\n    init_vars = tf.train.list_variables(tf_path)\n    names = []\n    arrays = []\n    for name, shape in init_vars:\n        logger.info(f\"Loading TF weight {name} with shape {shape}\")\n        array = tf.train.load_variable(tf_path, name)\n        names.append(name)\n        arrays.append(array)\n\n    for name, array in zip(names, arrays):\n        name = name.split(\"/\")\n        # adam_v and adam_m are variables used in AdamWeightDecayOptimizer to calculated m and v\n        # which are not required for using pretrained model\n        if any(\n            n in [\"adam_v\", \"adam_m\", \"AdamWeightDecayOptimizer\", \"AdamWeightDecayOptimizer_1\", \"global_step\"]\n            for n in name\n        ):\n            logger.info(f\"Skipping {'/'.join(name)}\")\n            continue\n        pointer = model\n        for m_name in name:\n            if re.fullmatch(r\"[A-Za-z]+_\\d+\", m_name):\n                scope_names = re.split(r\"_(\\d+)\", m_name)\n            else:\n                scope_names = [m_name]\n            if scope_names[0] == \"kernel\" or scope_names[0] == \"gamma\":\n                pointer = getattr(pointer, \"weight\")\n            elif scope_names[0] == \"output_bias\" or scope_names[0] == \"beta\":\n                pointer = getattr(pointer, \"bias\")\n            elif scope_names[0] == \"output_weights\":\n                pointer = getattr(pointer, \"weight\")\n            elif scope_names[0] == \"squad\":\n                pointer = getattr(pointer, \"classifier\")\n            else:\n                try:\n                    pointer = getattr(pointer, scope_names[0])\n                except AttributeError:\n                    logger.info(f\"Skipping {'/'.join(name)}\")\n                    continue\n            if len(scope_names) >= 2:\n                num = int(scope_names[1])\n                pointer = pointer[num]\n        if m_name[-11:] == \"_embeddings\":\n            pointer = getattr(pointer, \"weight\")\n        elif m_name == \"kernel\":\n            array = np.transpose(array)\n        try:\n            if pointer.shape != array.shape:\n                raise ValueError(f\"Pointer shape {pointer.shape} and array shape {array.shape} mismatched\")\n        except ValueError as e:\n            e.args += (pointer.shape, array.shape)\n            raise\n        logger.info(f\"Initialize PyTorch weight {name}\")\n        pointer.data = torch.from_numpy(array)\n\n\nSUMMARY:\n**Summary: Loading TensorFlow Weights in PyTorch Model**\n\n```python\ndef load_tf_weights_in_dummy_bert(model, config, tf_checkpoint_path):\n    \"\"\"\n    Load pre-trained TensorFlow weights into a PyTorch model.\n\n    Args:\n    - model (PyTorch model): The model to load weights into.\n    - config: (not used)\n    - tf_checkpoint_path: Path to the TensorFlow checkpoint file.\n\n    Returns:\n    - The loaded PyTorch model with pre-trained weights.\n    \"\"\"\n\n    **Implementation Details**\n\n    1. Loads TensorFlow checkpoints using `tf.train.list_variables` and `tf.train.load_variable`.\n    2. Skips certain variables (e.g., `adam_v`, `adam_m`) that are not required for using the pre-trained model.\n    3. Uses NumPy to transpose array `array` when its name ends with `_embeddings`.\n    4. Checks shape mismatch between `pointer` and `array` using `ValueError`.\n\n**Design Choice:**\n\n* The function uses a pointer-based approach to navigate the PyTorch model's namespace.\n* This design choice enables efficient loading of TensorFlow weights into the PyTorch model.\n\n**Dependencies:**\n\n* TensorFlow (`tf`)\n* NumPy (`np`)\n* PyTorch (`torch`)\n\n**Interfaces:**\n\n* `model`: The PyTorch model to load weights into (instance-specific).\n* `logger`: A logging interface for reporting progress and errors (not shown in the code snippet)."}


created_at: 2025-02-05 09:13:41
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy_bert.py:701-727

metadata:
{"docstring": "", "end_line": 727, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_dummy_bert.py", "name": "DummyBertPreTrainedModel", "start_line": 701, "type": "class", "chroma:document": "CODE:\n\nclass DummyBertPreTrainedModel(PreTrainedModel):\n    \"\"\"\n    An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained\n    models.\n    \"\"\"\n\n    config_class = DummyBertConfig\n    load_tf_weights = load_tf_weights_in_dummy_bert\n    base_model_prefix = \"dummy_bert\"\n    supports_gradient_checkpointing = True\n    _supports_sdpa = True\n\n    def _init_weights(self, module):\n        \"\"\"Initialize the weights\"\"\"\n        if isinstance(module, nn.Linear):\n            # Slightly different from the TF version which uses truncated_normal for initialization\n            # cf https://github.com/pytorch/pytorch/pull/5617\n            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)\n            if module.bias is not None:\n                module.bias.data.zero_()\n        elif isinstance(module, nn.Embedding):\n            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)\n            if module.padding_idx is not None:\n                module.weight.data[module.padding_idx].zero_()\n        elif isinstance(module, nn.LayerNorm):\n            module.bias.data.zero_()\n\n\nSUMMARY:\n**Summary:**\n\n`DummyBertPreTrainedModel`: An abstract class implementing a simplified interface for downloading and loading pre-trained models.\n\n**Main Purpose and Functionality:**\nThis class provides a basic framework for handling weights initialization, allowing users to download and load pre-trained models.\n\n**Key Implementation Details:**\n\n* Initializes model weights using a normal distribution with mean 0.0 and standard deviation `config.initializer_range`.\n* Handles initialization for linear layers, embedding tables, and layer normalization modules.\n* Copies behavior from the PyTorch `load_tf_weights_in_dummy_bert` function.\n\n**Notable Dependencies or Requirements:**\n\n* `DummyBertConfig`: Config class for this model (not shown in this snippet).\n* `nn.Linear`, `nn.Embedding`, `nn.LayerNorm`: PyTorch neural network module classes.\n* `load_tf_weights_in_dummy_bert`: Function imported from an external library, used for loading TensorFlow weights.\n\n**Interface Inheritance:**\nInherits from `PreTrainedModel`, providing a simplified interface for downloading and loading pre-trained models."}


created_at: 2025-02-05 09:13:41
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/configuration_new_model.py:11-132

metadata:
{"docstring": "", "end_line": 132, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/configuration_new_model.py", "name": "NewModelConfig", "start_line": 11, "type": "class", "chroma:document": "CODE:\n\nclass NewModelConfig(PretrainedConfig):\n    r\"\"\"\n    This is the configuration class to store the configuration of a [`NewModelModel`]. It is used to instantiate an NewModel\n    model according to the specified arguments, defining the model architecture. Instantiating a configuration with the\n    defaults will yield a similar configuration to that of the NewModel-7B.\n    e.g. [google/new_model-7b](https://huggingface.co/google/new_model-7b)\n    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the\n    documentation from [`PretrainedConfig`] for more information.\n    Args:\n        vocab_size (`int`, *optional*, defaults to 256000):\n            Vocabulary size of the NewModel model. Defines the number of different tokens that can be represented by the\n            `inputs_ids` passed when calling [`NewModelModel`]\n        hidden_size (`int`, *optional*, defaults to 3072):\n            Dimension of the hidden representations.\n        intermediate_size (`int`, *optional*, defaults to 24576):\n            Dimension of the MLP representations.\n        num_hidden_layers (`int`, *optional*, defaults to 28):\n            Number of hidden layers in the Transformer decoder.\n        num_attention_heads (`int`, *optional*, defaults to 16):\n            Number of attention heads for each attention layer in the Transformer decoder.\n        num_key_value_heads (`int`, *optional*, defaults to 16):\n            This is the number of key_value heads that should be used to implement Grouped Query Attention. If\n            `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if\n            `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When\n            converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed\n            by meanpooling all the original heads within that group. For more details checkout [this\n            paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to\n            `num_attention_heads`.\n        head_dim (`int`, *optional*, defaults to 256):\n            The attention head dimension.\n        hidden_act (`str` or `function`, *optional*, defaults to `\"gelu_pytorch_tanh\"`):\n            The legacy activation function. It is overwritten by the `hidden_activation`.\n        hidden_activation (`str` or `function`, *optional*):\n            The non-linear activation function (function or string) in the decoder. Will default to `\"gelu_pytorch_tanh\"`\n            if not specified. `\"gelu_pytorch_tanh\"` uses an approximation of the `\"gelu\"` activation function.\n        max_position_embeddings (`int`, *optional*, defaults to 8192):\n            The maximum sequence length that this model might ever be used with.\n        initializer_range (`float`, *optional*, defaults to 0.02):\n            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.\n        rms_norm_eps (`float`, *optional*, defaults to 1e-06):\n            The epsilon used by the rms normalization layers.\n        use_cache (`bool`, *optional*, defaults to `True`):\n            Whether or not the model should return the last key/values attentions (not used by all models). Only\n            relevant if `config.is_decoder=True`.\n        pad_token_id (`int`, *optional*, defaults to 0):\n            Padding token id.\n        eos_token_id (`int`, *optional*, defaults to 1):\n            End of stream token id.\n        bos_token_id (`int`, *optional*, defaults to 2):\n            Beginning of stream token id.\n        tie_word_embeddings (`bool`, *optional*, defaults to `True`):\n            Whether to tie weight embeddings\n        rope_theta (`float`, *optional*, defaults to 10000.0):\n            The base period of the RoPE embeddings.\n        attention_bias (`bool`, defaults to `False`, *optional*, defaults to `False`):\n            Whether to use a bias in the query, key, value and output projection layers during self-attention.\n        attention_dropout (`float`, *optional*, defaults to 0.0):\n            The dropout ratio for the attention probabilities.\n    ```python\n    >>> from transformers import NewModelModel, NewModelConfig\n    >>> # Initializing a NewModel new_model-7b style configuration\n    >>> configuration = NewModelConfig()\n    >>> # Initializing a model from the new_model-7b style configuration\n    >>> model = NewModelModel(configuration)\n    >>> # Accessing the model configuration\n    >>> configuration = model.config\n    ```\"\"\"\n\n    model_type = \"new_model\"\n    keys_to_ignore_at_inference = [\"past_key_values\"]\n\n    def __init__(\n        self,\n        vocab_size=256030,\n        hidden_size=64,\n        intermediate_size=90,\n        num_hidden_layers=28,\n        num_attention_heads=16,\n        num_key_value_heads=16,\n        head_dim=256,\n        hidden_act=\"gelu_pytorch_tanh\",\n        hidden_activation=None,\n        max_position_embeddings=1500,\n        initializer_range=0.02,\n        rms_norm_eps=1e-6,\n        use_cache=True,\n        pad_token_id=0,\n        eos_token_id=1,\n        bos_token_id=2,\n        tie_word_embeddings=True,\n        rope_theta=10000.0,\n        attention_bias=False,\n        attention_dropout=0.0,\n        **kwargs,\n    ):\n        super().__init__(\n            pad_token_id=pad_token_id,\n            bos_token_id=bos_token_id,\n            eos_token_id=eos_token_id,\n            tie_word_embeddings=tie_word_embeddings,\n            **kwargs,\n        )\n        self.vocab_size = vocab_size\n        self.max_position_embeddings = max_position_embeddings\n        self.hidden_size = hidden_size\n        self.intermediate_size = intermediate_size\n        self.num_hidden_layers = num_hidden_layers\n        self.num_attention_heads = num_attention_heads\n        self.head_dim = head_dim\n        self.num_key_value_heads = num_key_value_heads\n        self.hidden_act = hidden_act\n        self.hidden_activation = hidden_activation\n        self.initializer_range = initializer_range\n        self.rms_norm_eps = rms_norm_eps\n        self.use_cache = use_cache\n        self.rope_theta = rope_theta\n        self.attention_bias = attention_bias\n        self.attention_dropout = attention_dropout\n\n    @property\n    def num_heads(self):\n\n\nSUMMARY:\nHere is a concise and focused summary of the class code chunk:\n\n**Class:** `NewModelConfig`\n**Purpose:** The configuration class for storing parameters defining a [`NewModelModel`].\n\n**Main Functionality:**\nThis class enables instantiation of a `NewModel` model according to specified arguments, including vocabulary size, hidden dimensions, number of layers, attention heads, and other hyperparameters.\n\n**Key Implementation Details:**\n\n* **Attributes:** Vocabulary size (`vocab_size`), hidden dimension sizes (`hidden_size`, `intermediate_size`), number of layers (`num_hidden_layers`), attention head count (`num_attention_heads`), head dimension (`head_dim`), activation functions (`hidden_act`, `hidden_activation`), and other hyperparameters.\n* **Constructor:** Initializes the class with an optional dictionary of hyperparameters.\n\n**Notable Dependencies and Requirements:**\n\n* Inherits from [`PretrainedConfig`]\n* Uses external dependencies for weight initialization (e.g., truncated normal initializer)\n* Implements Grouped Query Attention and RoPE embeddings using hyperparameter settings (`num_key_value_heads`, `rope_theta`)"}


created_at: 2025-02-05 09:13:42
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modular_super.py:11-38
encoding: FLOAT32
metadata:
{"docstring": "# example where we need some deps and some functions", "end_line": 38, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modular_super.py", "name": "SuperModel", "start_line": 11, "type": "class", "chroma:document": "CODE:\n# example where we need some deps and some functions\nclass SuperModel(LlamaModel):\n    def forward(\n        self,\n        input_ids: torch.LongTensor = None,\n        attention_mask: Optional[torch.Tensor] = None,\n        position_ids: Optional[torch.LongTensor] = None,\n        past_key_values: Optional[Union[Cache, List[torch.FloatTensor]]] = None,\n        inputs_embeds: Optional[torch.FloatTensor] = None,\n        use_cache: Optional[bool] = None,\n        output_attentions: Optional[bool] = None,\n        output_hidden_states: Optional[bool] = None,\n        return_dict: Optional[bool] = None,\n        cache_position: Optional[torch.LongTensor] = None,\n    ) -> Union[Tuple, CausalLMOutputWithPast]:\n        out = super().forward(\n            input_ids,\n            attention_mask,\n            position_ids,\n            past_key_values,\n            inputs_embeds,\n            use_cache,\n            output_attentions,\n            output_hidden_states,\n            return_dict,\n            cache_position,\n        )\n        out.logits *= 2**4\n\n\nSUMMARY:\n**Class Summary: `SuperModel`**\n\n### Main Purpose and Functionality\n\nThe `SuperModel` class extends `LlamaModel` and inherits its functionality. It provides an improved forward pass that includes additional computations.\n\n### Key Implementation Details\n\n* **Forward pass**: The model's input is passed through a modified call to the parent class's `forward` method.\n* **Linear transformation**: The output logits are scaled by a factor of 2^4 (16).\n\n### Notable Dependencies and Requirements\n\nThe following dependencies and interfaces are used in this code:\n\n* `torch`: PyTorch library for tensor operations\n* `LlamaModel`: Parent class with its own `forward` method implementation\n* `Cache`, `List[torch.FloatTensor]`: Optional types in the PyTorch library\n* `CausalLMOutputWithPast`: Expected return type of the `forward` method\n\n### Interface and Inheritance\n\nThe `SuperModel` class inherits from `LlamaModel`, suggesting that it is designed to build upon or extend the parent class's functionality.\n\n### External Dependencies (none mentioned in this code snippet)"}


created_at: 2025-02-05 09:13:42
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_new_task_model.py:68-76

metadata:
{"docstring": "", "end_line": 76, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_new_task_model.py", "name": "NewTaskModelMultiModalProjector", "start_line": 68, "type": "class", "chroma:document": "CODE:\n\nclass NewTaskModelMultiModalProjector(nn.Module):\n    def __init__(self, config: NewTaskModelConfig):\n        super().__init__()\n        self.linear = nn.Linear(config.vision_config.hidden_size, config.vision_config.projection_dim, bias=True)\n\n    def forward(self, image_features):\n        hidden_states = self.linear(image_features)\n\n\n\nSUMMARY:\n**Summary**\n\n`NewTaskModelMultiModalProjector`: A multi-modal projector class extending PyTorch's `nn.Module`. This model projects input features from various modalities onto a shared high-dimensional space.\n\n**Key Details:**\n\n* **Forward Pass**: Takes in image features, applies a linear transformation, and returns the projected output.\n\t+ Input: `image_features`\n\t+ `linear` = `nn.Linear(config.vision_config.hidden_size, config.vision_config.projection_dim, bias=True)`\n* **Configured Linear Layer**: A fully connected layer with:\n\t+ Input dimension equal to `config.vision_config.hidden_size`\n\t+ Output dimension equal to `config.vision_config.projection_dim`\n\t+ Includes bias term\n* **No External Dependencies**\n\nThis summary highlights the primary purpose and key implementation details of the class, including its forward pass, configured linear layer, and lack of external dependencies."}


created_at: 2025-02-05 09:13:42
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modular_dummy.py:5-9
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 9, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modular_dummy.py", "name": "rotate_half", "start_line": 5, "type": "function", "chroma:document": "CODE:\n\ndef rotate_half(x):\n    \"\"\"Rotates half the hidden dims of the input.\"\"\"\n    x1 = x[..., : x.shape[-1] // 4]\n    x2 = x[..., x.shape[-1] // 4 :]\n\n\nSUMMARY:\n**Summary:**\n\nRotates half of the hidden dimensions in the input tensor `x` by shifting and concatenating two adjacent segments.\n\n**Key Details:**\n\n* Rotates only half of the hidden dimensions (top 4 dimensions).\n* Shifts top part to bottom and vice versa using concatenation:\n\t+ `x1 = x[..., : x.shape[-1] // 4]`\n\t+ `-x2 = -x[..., x.shape[-1] // 4 :]`\n* Uses PyTorch tensor manipulation, specifically `torch.cat`.\n\n**Notable Dependencies:**\n\n* Requires PyTorch library (`import torch`)\n* Assumes input tensor `x` has shape that supports splitting along the last dimension."}


created_at: 2025-02-05 09:13:42
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modular_dummy.py:13-14
encoding: FLOAT32
metadata:
{"docstring": "# example where we need some deps and some functions", "end_line": 14, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modular_dummy.py", "name": "DummyModel", "start_line": 13, "type": "class", "chroma:document": "CODE:\n# example where we need some deps and some functions\nclass DummyModel(LlamaModel):\n\n\nSUMMARY:\n**Summary**\n\nThis Java code defines a `DummyModel` class that extends the `LlamaModel` class. The main purpose of this class is to serve as a placeholder or example model.\n\n**Key Implementation Details:**\n\n* Extends `LlamaModel`, indicating inheritance of methods and properties.\n* An empty class (`pass`), suggesting no specific implementation or functionality added on top of the base class.\n\n**Notable Dependencies/Requirements:**\n\n* References the `LlamaModel` class, implying that it relies on some functionality provided by this parent class.\n\n**Additional Information:**\nThis code may be part of a larger model or library system where custom models are defined to interact with LlamaModel."}


created_at: 2025-02-05 09:13:42
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modular_roberta.py:5-11
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 11, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modular_roberta.py", "name": "RobertaEmbeddings", "start_line": 5, "type": "class", "chroma:document": "CODE:\n\nclass RobertaEmbeddings(BertEmbeddings):\n    def __init__(self, config):\n        super().__init__(config)\n        self.pad_token_id = config.pad_token_id\n        self.position_embeddings = nn.Embedding(\n            config.max_position_embeddings, config.hidden_size, config.pad_token_id\n\n\nSUMMARY:\n**Class Summary: RobertaEmbeddings**\n\nThe `RobertaEmbeddings` class extends the `BertEmbeddings` base class and is used for obtaining embeddings from input sequences in a Hugging Face's transformers library.\n\n**Primary Purpose:** To compute sequence embeddings using a custom embedding layer, inheriting properties from the parent `BertEmbeddings` class.\n\n**Key Implementation Details:**\n\n* Initializes the embedding layer with a specific configuration (`config.max_position_embeddings`, `config.hidden_size`, and `config.pad_token_id`)\n* Utilizes an `nn.Embedding` module to store position embeddings\n\n**Notable Dependencies/Requirements:**\n\n* Inherits from `BertEmbeddings`, utilizing its properties\n* Depends on the Hugging Face's transformers library for the embedding layer (`nn.Embedding`)"}


created_at: 2025-02-05 09:13:42
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modular_roberta.py:14-16
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 16, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modular_roberta.py", "name": "RobertaModel", "start_line": 14, "type": "class", "chroma:document": "CODE:\n\nclass RobertaModel(BertModel):\n    def __init__(self, config, add_pooling_layer=True):\n\n\nSUMMARY:\n**Summary: `RobertaModel` Class**\n\n### Purpose and Functionality:\n\nThe `RobertaModel` class is an extension of the `BertModel`, inheriting its functionality. It is designed to accommodate the Roberta language model architecture, allowing for efficient generation and prediction tasks.\n\n### Key Implementation Details:\n\n- Initializes with a `config` dictionary and optionally adds a pooling layer (`add_pooling_layer=True`)\n- Calls the parent class's (`BertModel`) initializer with the inherited parameters\n\n### Dependencies and Requirements:\n\n- Inherited from `BertModel`\n- Extends Roberta language model architecture"}


created_at: 2025-02-05 09:13:42
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modular_dummy_bert.py:9-26

metadata:
{"docstring": "", "end_line": 26, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modular_dummy_bert.py", "name": "DummyBertModel", "start_line": 9, "type": "class", "chroma:document": "CODE:\n\nclass DummyBertModel(BertModel):\n    def forward(\n        self,\n        input_ids: Optional[torch.Tensor] = None,\n        attention_mask: Optional[torch.Tensor] = None,\n        token_type_ids: Optional[torch.Tensor] = None,\n        position_ids: Optional[torch.Tensor] = None,\n        head_mask: Optional[torch.Tensor] = None,\n        inputs_embeds: Optional[torch.Tensor] = None,\n        encoder_hidden_states: Optional[torch.Tensor] = None,\n        encoder_attention_mask: Optional[torch.Tensor] = None,\n        past_key_values: Optional[List[torch.FloatTensor]] = None,\n        use_cache: Optional[bool] = None,\n        output_attentions: Optional[bool] = None,\n        output_hidden_states: Optional[bool] = None,\n        return_dict: Optional[bool] = None,\n    ) -> Union[Tuple[torch.Tensor], BaseModelOutputWithPoolingAndCrossAttentions]:\n\n\nSUMMARY:\n**Class Summary: DummyBertModel**\n\nThe `DummyBertModel` class is a subclass of `BertModel` that overrides the `forward` method to simplify its behavior.\n\n**Primary Purpose:**\nThe model's primary purpose is to forward input tensors through a basic BERT-like processing pipeline, effectively reducing its functionality by always returning `super().forward(input_ids)`.\n\n**Key Implementation Details:**\n\n*   **Method Signature:** The `forward` method accepts a variety of optional input tensors and returns either a tensor or a `BaseModelOutputWithPoolingAndCrossAttentions` object.\n*   **Inherited Behavior:** The model calls the parent class's `forward` method directly with only the `input_ids` argument, implying that all other arguments are ignored.\n\n**Notable Dependencies:**\n\n*   `BertModel`\n*   `torch`\n        *   External Dependency: `torch`\n\n**Interfaces:**\n\n*   Inherited from `BertModel`: `BaseModelOutputWithPoolingAndCrossAttentions`."}


created_at: 2025-02-05 09:13:42
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_multimodal2.py:36-137

metadata:
{"docstring": "", "end_line": 137, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_multimodal2.py", "name": "Multimodal2VisionAttention", "start_line": 36, "type": "class", "chroma:document": "CODE:\n\nclass Multimodal2VisionAttention(nn.Module):\n    \"\"\"Multi-headed attention from 'Attention Is All You Need' paper\"\"\"\n\n    def __init__(self, config):\n        super().__init__()\n        self.config = config\n        self.embed_dim = config.hidden_size\n        self.num_heads = config.num_attention_heads\n        self.head_dim = self.embed_dim // self.num_heads\n        if self.head_dim * self.num_heads != self.embed_dim:\n            raise ValueError(\n                f\"embed_dim must be divisible by num_heads (got `embed_dim`: {self.embed_dim} and `num_heads`:\"\n                f\" {self.num_heads}).\"\n            )\n        self.scale = self.head_dim**-0.5\n        self.dropout = config.attention_dropout\n\n        self.k_proj = nn.Linear(self.embed_dim, self.embed_dim)\n        self.v_proj = nn.Linear(self.embed_dim, self.embed_dim)\n        self.q_proj = nn.Linear(self.embed_dim, self.embed_dim)\n        self.out_proj = nn.Linear(self.embed_dim, self.embed_dim)\n\n    def _shape(self, tensor: torch.Tensor, seq_len: int, bsz: int):\n        return tensor.view(bsz, seq_len, self.num_heads, self.head_dim).transpose(1, 2).contiguous()\n\n    def forward(\n        self,\n        hidden_states: torch.Tensor,\n        attention_mask: Optional[torch.Tensor] = None,\n        causal_attention_mask: Optional[torch.Tensor] = None,\n        output_attentions: Optional[bool] = False,\n    ) -> Tuple[torch.Tensor, Optional[torch.Tensor]]:\n        \"\"\"Input shape: Batch x Time x Channel\"\"\"\n\n        bsz, tgt_len, embed_dim = hidden_states.size()\n\n        # get query proj\n        query_states = self.q_proj(hidden_states) * self.scale\n        key_states = self._shape(self.k_proj(hidden_states), -1, bsz)\n        value_states = self._shape(self.v_proj(hidden_states), -1, bsz)\n\n        proj_shape = (bsz * self.num_heads, -1, self.head_dim)\n        query_states = self._shape(query_states, tgt_len, bsz).view(*proj_shape)\n        key_states = key_states.view(*proj_shape)\n        value_states = value_states.view(*proj_shape)\n\n        src_len = key_states.size(1)\n        attn_weights = torch.bmm(query_states, key_states.transpose(1, 2))\n\n        if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):\n            raise ValueError(\n                f\"Attention weights should be of size {(bsz * self.num_heads, tgt_len, src_len)}, but is\"\n                f\" {attn_weights.size()}\"\n            )\n\n        # apply the causal_attention_mask first\n        if causal_attention_mask is not None:\n            if causal_attention_mask.size() != (bsz, 1, tgt_len, src_len):\n                raise ValueError(\n                    f\"Attention mask should be of size {(bsz, 1, tgt_len, src_len)}, but is\"\n                    f\" {causal_attention_mask.size()}\"\n                )\n            attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) + causal_attention_mask\n            attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)\n\n        if attention_mask is not None:\n            if attention_mask.size() != (bsz, 1, tgt_len, src_len):\n                raise ValueError(\n                    f\"Attention mask should be of size {(bsz, 1, tgt_len, src_len)}, but is {attention_mask.size()}\"\n                )\n            attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) + attention_mask\n            attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)\n\n        attn_weights = nn.functional.softmax(attn_weights, dim=-1)\n\n        if output_attentions:\n            # this operation is a bit akward, but it's required to\n            # make sure that attn_weights keeps its gradient.\n            # In order to do so, attn_weights have to reshaped\n            # twice and have to be reused in the following\n            attn_weights_reshaped = attn_weights.view(bsz, self.num_heads, tgt_len, src_len)\n            attn_weights = attn_weights_reshaped.view(bsz * self.num_heads, tgt_len, src_len)\n        else:\n            attn_weights_reshaped = None\n\n        attn_probs = nn.functional.dropout(attn_weights, p=self.dropout, training=self.training)\n\n        attn_output = torch.bmm(attn_probs, value_states)\n\n        if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):\n            raise ValueError(\n                f\"`attn_output` should be of size {(bsz, self.num_heads, tgt_len, self.head_dim)}, but is\"\n                f\" {attn_output.size()}\"\n            )\n\n        attn_output = attn_output.view(bsz, self.num_heads, tgt_len, self.head_dim)\n        attn_output = attn_output.transpose(1, 2)\n        attn_output = attn_output.reshape(bsz, tgt_len, embed_dim)\n\n        attn_output = self.out_proj(attn_output)\n\n\n\nSUMMARY:\n**Class Summary: Multimodal2VisionAttention**\n=============================================\n\n*   **Main Purpose and Functionality:** This class implements the Multi-Head Attention mechanism as described in the \"Attention Is All You Need\" paper. It takes in input `hidden_states` and optional masks (`attention_mask` and `causal_attention_mask`) to compute attention weights and outputs.\n*   **Key Implementation Details:**\n    *   Uses a linear transformation followed by a scaling factor to project queries, keys, and values.\n    *   Computes attention weights using a dot product and softmax function.\n    *   Applies dropout to the attention weights.\n    *   Transposes and reshapes the output for broadcasting.\n*   **Notable Dependencies or Requirements:**\n    *   `config`: Configuration object containing hidden size, number of heads, and attention dropouts.\n    *   `nn.Module`: Base PyTorch module class.\n    *   `torch.Tensor`: PyTorch tensor class.\n    *   `nn.functional`: PyTorch neural network functional module.\n\n**Method Signature:**\n```python\nforward(hidden_states: torch.Tensor, \n         attention_mask: Optional[torch.Tensor] = None, \n         causal_attention_mask: Optional[torch.Tensor] = None, \n         output_attentions: Optional[bool] = False) -> Tuple[torch.Tensor, Optional[torch.Tensor]]\n```\n\n**Example Use Case:**\n```python\nmultimodal2vision_attention = Multimodal2VisionAttention(config)\nhidden_states = ...\nattention_mask = ...\n\nattn_output, attn_weights = multimodal2vision_attention(hidden_states, attention_mask, None, False)\n```"}


created_at: 2025-02-05 09:13:42
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_multimodal2.py:140-210

metadata:
{"docstring": "", "end_line": 210, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/modular-transformers/modeling_multimodal2.py", "name": "Multimodal2VisionSdpaAttention", "start_line": 140, "type": "class", "chroma:document": "CODE:\n\nclass Multimodal2VisionSdpaAttention(Multimodal2VisionAttention):\n    \"\"\"\n    SDPA attention module using torch.nn.functional.scaled_dot_product_attention. This module inherits from\n    `Multimodal2VisionAttention` as the weights of the module stays untouched. The only changes are on the forward pass to adapt to\n    SDPA API.\n    \"\"\"\n\n    # Adapted from Multimodal2VisionAttention.forward\n    def forward(\n        self,\n        hidden_states: torch.Tensor,\n        attention_mask: Optional[torch.Tensor] = None,\n        causal_attention_mask: Optional[torch.Tensor] = None,\n        output_attentions: Optional[bool] = False,\n    ) -> Tuple[torch.Tensor, Optional[torch.Tensor]]:\n        if output_attentions:\n            # TODO: Improve this warning with e.g. `model.config.attn_implementation = \"manual\"` once this is implemented.\n            logger.warning_once(\n                \"Multimodal2VisionModel is using Multimodal2VisionSdpaAttention, but `torch.nn.functional.scaled_dot_product_attention` does not \"\n                \"support `output_attentions=True`. Falling back to the manual attention implementation, but specifying \"\n                \"the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can \"\n                'be removed using the argument `attn_implementation=\"eager\"` when loading the model.'\n            )\n            return super().forward(\n                hidden_states=hidden_states,\n                attention_mask=attention_mask,\n                causal_attention_mask=causal_attention_mask,\n                output_attentions=output_attentions,\n            )\n\n        # MULTIMODAL2_VISION text model uses both `causal_attention_mask` and `attention_mask`\n        if attention_mask is not None and causal_attention_mask is not None:\n            attn_mask = attention_mask + causal_attention_mask\n        elif causal_attention_mask is not None:\n            attn_mask = causal_attention_mask\n        else:\n            attn_mask = attention_mask\n\n        bsz, tgt_len, embed_dim = hidden_states.size()\n\n        query_states = self.q_proj(hidden_states)\n        key_states = self.k_proj(hidden_states)\n        value_states = self.v_proj(hidden_states)\n\n        query_states = query_states.view(bsz, -1, self.num_heads, self.head_dim).transpose(1, 2)\n        key_states = key_states.view(bsz, -1, self.num_heads, self.head_dim).transpose(1, 2)\n        value_states = value_states.view(bsz, -1, self.num_heads, self.head_dim).transpose(1, 2)\n\n        # SDPA with memory-efficient backend is currently (torch==2.1.2) bugged with non-contiguous inputs with custom attn_mask,\n        # Reference: https://github.com/pytorch/pytorch/issues/112577.\n        if not is_torch_greater_or_equal_than_2_2 and query_states.device.type == \"cuda\" and attn_mask is not None:\n            query_states = query_states.contiguous()\n            key_states = key_states.contiguous()\n            value_states = value_states.contiguous()\n\n        # MULTIMODAL2_VISION text model uses both `causal_attention_mask` and `attention_mask` sequentially.\n        attn_output = torch.nn.functional.scaled_dot_product_attention(\n            query_states,\n            key_states,\n            value_states,\n            attn_mask=attn_mask,\n            dropout_p=self.dropout if self.training else 0.0,\n            scale=self.scale,\n        )\n\n        attn_output = attn_output.transpose(1, 2)\n        attn_output = attn_output.reshape(bsz, tgt_len, embed_dim)\n\n        attn_output = self.out_proj(attn_output)\n\n\n\nSUMMARY:\n**Multimodal2VisionSdpaAttention Class Summary**\n\n**Main Purpose and Functionality:**\nThe `Multimodal2VisionSdpaAttention` class is an attention module derived from `Multimodal2VisionAttention`, utilizing the `torch.nn.functional.scaled_dot_product_attention` function. It adapts the forward pass to work with the SDPA API.\n\n**Key Implementation Details:**\n\n*   Implements scaled dot-product attention using `torch.nn.functional.scaled_dot_product_attention`.\n*   Handles both `causal_attention_mask` and `attention_mask` in the input.\n*   Performs dimension-wise transposition of query, key, and value states before passing them to the scaled dot-product attention function.\n*   Applies an optional scaling factor (`self.scale`) to the attention output.\n\n**Notable Dependencies or Requirements:**\n\n*   `torch.nn.functional.scaled_dot_product_attention`\n*   `Multimodal2VisionAttention` class (with unchanged weights)\n*   `torch` version 2.1.2+ for memory-efficient backend support\n*   `self.dropout`, `self.scale`, and other attributes from `Multimodal2VisionAttention`\n\n**Interface Summary:**\n\nThe class has the following interface:\n\n*   `forward(hidden_states, attention_mask, causal_attention_mask, output_attentions) -> (attn_output, None)`\n    *   Returns the scaled dot-product attention output.\n    *   If `output_attentions` is set to `True`, falls back to a manual attention implementation."}


