# embeddings_queue

created_at: 2025-02-05 09:13:46
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/question-answering/utils_qa.py:252-443

metadata:
{"docstring": "", "end_line": 443, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/question-answering/utils_qa.py", "name": "postprocess_qa_predictions_with_beam_search", "start_line": 252, "type": "function", "chroma:document": "CODE:\n\ndef postprocess_qa_predictions_with_beam_search(\n    examples,\n    features,\n    predictions: Tuple[np.ndarray, np.ndarray],\n    version_2_with_negative: bool = False,\n    n_best_size: int = 20,\n    max_answer_length: int = 30,\n    start_n_top: int = 5,\n    end_n_top: int = 5,\n    output_dir: Optional[str] = None,\n    prefix: Optional[str] = None,\n    log_level: Optional[int] = logging.WARNING,\n):\n    \"\"\"\n    Post-processes the predictions of a question-answering model with beam search to convert them to answers that are substrings of the\n    original contexts. This is the postprocessing functions for models that return start and end logits, indices, as well as\n    cls token predictions.\n\n    Args:\n        examples: The non-preprocessed dataset (see the main script for more information).\n        features: The processed dataset (see the main script for more information).\n        predictions (:obj:`Tuple[np.ndarray, np.ndarray]`):\n            The predictions of the model: two arrays containing the start logits and the end logits respectively. Its\n            first dimension must match the number of elements of :obj:`features`.\n        version_2_with_negative (:obj:`bool`, `optional`, defaults to :obj:`False`):\n            Whether or not the underlying dataset contains examples with no answers.\n        n_best_size (:obj:`int`, `optional`, defaults to 20):\n            The total number of n-best predictions to generate when looking for an answer.\n        max_answer_length (:obj:`int`, `optional`, defaults to 30):\n            The maximum length of an answer that can be generated. This is needed because the start and end predictions\n            are not conditioned on one another.\n        start_n_top (:obj:`int`, `optional`, defaults to 5):\n            The number of top start logits too keep when searching for the :obj:`n_best_size` predictions.\n        end_n_top (:obj:`int`, `optional`, defaults to 5):\n            The number of top end logits too keep when searching for the :obj:`n_best_size` predictions.\n        output_dir (:obj:`str`, `optional`):\n            If provided, the dictionaries of predictions, n_best predictions (with their scores and logits) and, if\n            :obj:`version_2_with_negative=True`, the dictionary of the scores differences between best and null\n            answers, are saved in `output_dir`.\n        prefix (:obj:`str`, `optional`):\n            If provided, the dictionaries mentioned above are saved with `prefix` added to their names.\n        log_level (:obj:`int`, `optional`, defaults to ``logging.WARNING``):\n            ``logging`` log level (e.g., ``logging.WARNING``)\n    \"\"\"\n    if len(predictions) != 5:\n        raise ValueError(\"`predictions` should be a tuple with five elements.\")\n    start_top_log_probs, start_top_index, end_top_log_probs, end_top_index, cls_logits = predictions\n\n    if len(predictions[0]) != len(features):\n        raise ValueError(f\"Got {len(predictions[0])} predictions and {len(features)} features.\")\n\n    # Build a map example to its corresponding features.\n    example_id_to_index = {k: i for i, k in enumerate(examples[\"id\"])}\n    features_per_example = collections.defaultdict(list)\n    for i, feature in enumerate(features):\n        features_per_example[example_id_to_index[feature[\"example_id\"]]].append(i)\n\n    # The dictionaries we have to fill.\n    all_predictions = collections.OrderedDict()\n    all_nbest_json = collections.OrderedDict()\n    scores_diff_json = collections.OrderedDict() if version_2_with_negative else None\n\n    # Logging.\n    logger.setLevel(log_level)\n    logger.info(f\"Post-processing {len(examples)} example predictions split into {len(features)} features.\")\n\n    # Let's loop over all the examples!\n    for example_index, example in enumerate(tqdm(examples)):\n        # Those are the indices of the features associated to the current example.\n        feature_indices = features_per_example[example_index]\n\n        min_null_score = None\n        prelim_predictions = []\n\n        # Looping through all the features associated to the current example.\n        for feature_index in feature_indices:\n            # We grab the predictions of the model for this feature.\n            start_log_prob = start_top_log_probs[feature_index]\n            start_indexes = start_top_index[feature_index]\n            end_log_prob = end_top_log_probs[feature_index]\n            end_indexes = end_top_index[feature_index]\n            feature_null_score = cls_logits[feature_index]\n            # This is what will allow us to map some the positions in our logits to span of texts in the original\n            # context.\n            offset_mapping = features[feature_index][\"offset_mapping\"]\n            # Optional `token_is_max_context`, if provided we will remove answers that do not have the maximum context\n            # available in the current feature.\n            token_is_max_context = features[feature_index].get(\"token_is_max_context\", None)\n\n            # Update minimum null prediction\n            if min_null_score is None or feature_null_score < min_null_score:\n                min_null_score = feature_null_score\n\n            # Go through all possibilities for the `n_start_top`/`n_end_top` greater start and end logits.\n            for i in range(start_n_top):\n                for j in range(end_n_top):\n                    start_index = int(start_indexes[i])\n                    j_index = i * end_n_top + j\n                    end_index = int(end_indexes[j_index])\n                    # Don't consider out-of-scope answers (last part of the test should be unnecessary because of the\n                    # p_mask but let's not take any risk)\n                    if (\n                        start_index >= len(offset_mapping)\n                        or end_index >= len(offset_mapping)\n                        or offset_mapping[start_index] is None\n                        or len(offset_mapping[start_index]) < 2\n                        or offset_mapping[end_index] is None\n                        or len(offset_mapping[end_index]) < 2\n                    ):\n                        continue\n\n                    # Don't consider answers with a length negative or > max_answer_length.\n                    if end_index < start_index or end_index - start_index + 1 > max_answer_length:\n                        continue\n                    # Don't consider answer that don't have the maximum context available (if such information is\n                    # provided).\n                    if token_is_max_context is not None and not token_is_max_context.get(str(start_index), False):\n                        continue\n                    prelim_predictions.append(\n                        {\n                            \"offsets\": (offset_mapping[start_index][0], offset_mapping[end_index][1]),\n                            \"score\": start_log_prob[i] + end_log_prob[j_index],\n                            \"start_log_prob\": start_log_prob[i],\n                            \"end_log_prob\": end_log_prob[j_index],\n                        }\n                    )\n\n        # Only keep the best `n_best_size` predictions.\n        predictions = sorted(prelim_predictions, key=lambda x: x[\"score\"], reverse=True)[:n_best_size]\n\n        # Use the offsets to gather the answer text in the original context.\n        context = example[\"context\"]\n        for pred in predictions:\n            offsets = pred.pop(\"offsets\")\n            pred[\"text\"] = context[offsets[0] : offsets[1]]\n\n        # In the very rare edge case we have not a single non-null prediction, we create a fake prediction to avoid\n        # failure.\n        if len(predictions) == 0:\n            # Without predictions min_null_score is going to be None and None will cause an exception later\n            min_null_score = -2e-6\n            predictions.insert(0, {\"text\": \"\", \"start_logit\": -1e-6, \"end_logit\": -1e-6, \"score\": min_null_score})\n\n        # Compute the softmax of all scores (we do it with numpy to stay independent from torch/tf in this file, using\n        # the LogSumExp trick).\n        scores = np.array([pred.pop(\"score\") for pred in predictions])\n        exp_scores = np.exp(scores - np.max(scores))\n        probs = exp_scores / exp_scores.sum()\n\n        # Include the probabilities in our predictions.\n        for prob, pred in zip(probs, predictions):\n            pred[\"probability\"] = prob\n\n        # Pick the best prediction and set the probability for the null answer.\n        all_predictions[example[\"id\"]] = predictions[0][\"text\"]\n        if version_2_with_negative:\n            scores_diff_json[example[\"id\"]] = float(min_null_score)\n\n        # Make `predictions` JSON-serializable by casting np.float back to float.\n        all_nbest_json[example[\"id\"]] = [\n            {k: (float(v) if isinstance(v, (np.float16, np.float32, np.float64)) else v) for k, v in pred.items()}\n            for pred in predictions\n        ]\n\n    # If we have an output_dir, let's save all those dicts.\n    if output_dir is not None:\n        if not os.path.isdir(output_dir):\n            raise EnvironmentError(f\"{output_dir} is not a directory.\")\n\n        prediction_file = os.path.join(\n            output_dir, \"predictions.json\" if prefix is None else f\"{prefix}_predictions.json\"\n        )\n        nbest_file = os.path.join(\n            output_dir, \"nbest_predictions.json\" if prefix is None else f\"{prefix}_nbest_predictions.json\"\n        )\n        if version_2_with_negative:\n            null_odds_file = os.path.join(\n                output_dir, \"null_odds.json\" if prefix is None else f\"{prefix}_null_odds.json\"\n            )\n\n        logger.info(f\"Saving predictions to {prediction_file}.\")\n        with open(prediction_file, \"w\") as writer:\n            writer.write(json.dumps(all_predictions, indent=4) + \"\\n\")\n        logger.info(f\"Saving nbest_preds to {nbest_file}.\")\n        with open(nbest_file, \"w\") as writer:\n            writer.write(json.dumps(all_nbest_json, indent=4) + \"\\n\")\n        if version_2_with_negative:\n            logger.info(f\"Saving null_odds to {null_odds_file}.\")\n            with open(null_odds_file, \"w\") as writer:\n                writer.write(json.dumps(scores_diff_json, indent=4) + \"\\n\")\n\n\n\nSUMMARY:\nHere is a concise summary of the provided function:\n\n**Function Name**: `compute_prediction`\n\n**Description**: This function computes and returns predictions for a given text input. It uses a scoring system to rank possible answers based on their likelihood. It outputs two dictionaries: one containing the best predicted answer (`all_predictions`), and another with null odds scores if the model supports it.\n\n**Input Parameters**:\n\n* `context`: The input text context.\n* `p_mask`: A mask indicating which parts of the input are relevant (not used in this implementation).\n* `output_dir`: An optional directory to save output files to.\n* `n_best_size`: The number of predictions to return.\n* `token_is_max_context`: A function to determine if a token has the maximum context available.\n\n**Return Values**:\n\n* `all_predictions`: A dictionary mapping input IDs to their best predicted answers.\n* `scores_diff_json` (optional): A dictionary with null odds scores for each input ID, if applicable."}


created_at: 2025-02-05 09:13:46
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/question-answering/run_qa.py:242-251

metadata:
{"docstring": "# endregion\n# region Helper classes", "end_line": 251, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/question-answering/run_qa.py", "name": "SavePretrainedCallback", "start_line": 242, "type": "class", "chroma:document": "CODE:\n# region Helper classes\nclass SavePretrainedCallback(keras.callbacks.Callback):\n    # Hugging Face models have a save_pretrained() method that saves both the weights and the necessary\n    # metadata to allow them to be loaded as a pretrained model in future. This is a simple Keras callback\n    # that saves the model with this method after each epoch.\n    def __init__(self, output_dir, **kwargs):\n        super().__init__()\n        self.output_dir = output_dir\n\n    def on_epoch_end(self, epoch, logs=None):\n\n\nSUMMARY:\n**SavePretrainedCallback Class Summary**\n\n* **Purpose**: Saves a Keras model using Hugging Face's `save_pretrained()` method after each epoch, preserving weights and necessary metadata.\n\n* **Key Implementation Details**:\n\t+ Saves the model using `self.model.save_pretrained(self.output_dir)`\n\t+ Utilizes Hugging Face's `save_pretrained()` method to save both weights and metadata\n\n* **Notable Dependencies/Requirements**:\n\t+ None explicitly listed\n\t+ Assumes the presence of `Docstring` (potentially a helper class or function)\n\n* **Code Structure**:\n\t+\u7ee7\u627f\u81ea Keras Callback (`super().__init__()`)\n\t+\u6709\u4e00\u4e2a\u8f93\u51fa\u76ee\u5f55 (`self.output_dir`) \u548c\u5176\u4ed6\u53ef\u9009\u53c2\u6570 (`**kwargs`)\n\t+\u5177\u6709\u4e00\u4e2aepoch\u7ed3\u675f\u7684\u56de\u8c03\u4e8b\u4ef6\u5904\u7406\u51fd\u6570\uff08`on_epoch_end()`)"}


created_at: 2025-02-05 09:13:46
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/question-answering/run_qa.py:257-826

metadata:
{"docstring": "# endregion", "end_line": 826, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/question-answering/run_qa.py", "name": "main", "start_line": 257, "type": "function", "chroma:document": "CODE:\n\ndef main():\n    # region Argument parsing\n    # See all possible arguments in src/transformers/training_args.py\n    # or by passing the --help flag to this script.\n    # We now keep distinct sets of args, for a cleaner separation of concerns.\n\n    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TFTrainingArguments))\n    if len(sys.argv) == 2 and sys.argv[1].endswith(\".json\"):\n        # If we pass only one argument to the script and it's the path to a json file,\n        # let's parse it to get our arguments.\n        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))\n    else:\n        model_args, data_args, training_args = parser.parse_args_into_dataclasses()\n\n    # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The\n    # information sent is the one passed as arguments along with your Python/PyTorch versions.\n    send_example_telemetry(\"run_qa\", model_args, data_args, framework=\"tensorflow\")\n\n    output_dir = Path(training_args.output_dir)\n    output_dir.mkdir(parents=True, exist_ok=True)\n    # endregion\n\n    # region Checkpoints\n    checkpoint = None\n    if len(os.listdir(training_args.output_dir)) > 0 and not training_args.overwrite_output_dir:\n        if (output_dir / CONFIG_NAME).is_file() and (output_dir / TF2_WEIGHTS_NAME).is_file():\n            checkpoint = output_dir\n            logger.info(\n                f\"Checkpoint detected, resuming training from checkpoint in {training_args.output_dir}. To avoid this\"\n                \" behavior, change the `--output_dir` or add `--overwrite_output_dir` to train from scratch.\"\n            )\n        else:\n            raise ValueError(\n                f\"Output directory ({training_args.output_dir}) already exists and is not empty. \"\n                \"Use --overwrite_output_dir to continue regardless.\"\n            )\n    # endregion\n\n    # region Logging\n    logging.basicConfig(\n        format=\"%(asctime)s - %(levelname)s - %(name)s - %(message)s\",\n        datefmt=\"%m/%d/%Y %H:%M:%S\",\n        handlers=[logging.StreamHandler(sys.stdout)],\n    )\n    logger.setLevel(logging.INFO if training_args.should_log else logging.WARN)\n\n    # Set the verbosity to info of the Transformers logger (on main process only):\n    if training_args.should_log:\n        transformers.utils.logging.set_verbosity_info()\n        transformers.utils.logging.enable_default_handler()\n        transformers.utils.logging.enable_explicit_format()\n    logger.info(f\"Training/evaluation parameters {training_args}\")\n    # endregion\n\n    # Set seed before initializing model.\n    set_seed(training_args.seed)\n\n    # region Load Data\n    # Get the datasets: you can either provide your own CSV/JSON/TXT training and evaluation files (see below)\n    # or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/\n    # (the dataset will be downloaded automatically from the datasets Hub).\n    #\n    # For CSV/JSON files, this script will use the column called 'text' or the first column if no column called\n    # 'text' is found. You can easily tweak this behavior (see below).\n    #\n    # In distributed training, the load_dataset function guarantee that only one local process can concurrently\n    # download the dataset.\n    if data_args.dataset_name is not None:\n        # Downloading and loading a dataset from the hub.\n        datasets = load_dataset(\n            data_args.dataset_name,\n            data_args.dataset_config_name,\n            cache_dir=model_args.cache_dir,\n            token=model_args.token,\n            trust_remote_code=model_args.trust_remote_code,\n        )\n    else:\n        data_files = {}\n        if data_args.train_file is not None:\n            data_files[\"train\"] = data_args.train_file\n            extension = data_args.train_file.split(\".\")[-1]\n\n        if data_args.validation_file is not None:\n            data_files[\"validation\"] = data_args.validation_file\n            extension = data_args.validation_file.split(\".\")[-1]\n        if data_args.test_file is not None:\n            data_files[\"test\"] = data_args.test_file\n            extension = data_args.test_file.split(\".\")[-1]\n        datasets = load_dataset(\n            extension,\n            data_files=data_files,\n            field=\"data\",\n            cache_dir=model_args.cache_dir,\n            token=model_args.token,\n        )\n    # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at\n    # https://huggingface.co/docs/datasets/loading_datasets.\n    # endregion\n\n    # region Load pretrained model and tokenizer\n    #\n    # Distributed training:\n    # The .from_pretrained methods guarantee that only one local process can concurrently\n    # download model & vocab.\n    config = AutoConfig.from_pretrained(\n        model_args.config_name if model_args.config_name else model_args.model_name_or_path,\n        cache_dir=model_args.cache_dir,\n        revision=model_args.model_revision,\n        token=model_args.token,\n        trust_remote_code=model_args.trust_remote_code,\n    )\n    tokenizer = AutoTokenizer.from_pretrained(\n        model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path,\n        cache_dir=model_args.cache_dir,\n        use_fast=True,\n        revision=model_args.model_revision,\n        token=model_args.token,\n        trust_remote_code=model_args.trust_remote_code,\n    )\n    # endregion\n\n    # region Tokenizer check: this script requires a fast tokenizer.\n    if not isinstance(tokenizer, PreTrainedTokenizerFast):\n        raise ValueError(\n            \"This example script only works for models that have a fast tokenizer. Checkout the big table of models at\"\n            \" https://huggingface.co/transformers/index.html#supported-frameworks to find the model types that meet\"\n            \" this requirement\"\n        )\n    # endregion\n\n    # region Preprocessing the datasets\n    # Preprocessing is slightly different for training and evaluation.\n    if training_args.do_train:\n        column_names = datasets[\"train\"].column_names\n    elif training_args.do_eval:\n        column_names = datasets[\"validation\"].column_names\n    else:\n        column_names = datasets[\"test\"].column_names\n    question_column_name = \"question\" if \"question\" in column_names else column_names[0]\n    context_column_name = \"context\" if \"context\" in column_names else column_names[1]\n    answer_column_name = \"answers\" if \"answers\" in column_names else column_names[2]\n\n    # Padding side determines if we do (question|context) or (context|question).\n    pad_on_right = tokenizer.padding_side == \"right\"\n\n    if data_args.max_seq_length > tokenizer.model_max_length:\n        logger.warning(\n            f\"The max_seq_length passed ({data_args.max_seq_length}) is larger than the maximum length for the \"\n            f\"model ({tokenizer.model_max_length}). Using max_seq_length={tokenizer.model_max_length}.\"\n        )\n    max_seq_length = min(data_args.max_seq_length, tokenizer.model_max_length)\n\n    if data_args.pad_to_max_length or isinstance(training_args.strategy, tf.distribute.TPUStrategy):\n        logger.info(\"Padding all batches to max length because argument was set or we're on TPU.\")\n        padding = \"max_length\"\n    else:\n        padding = False\n\n    # Training preprocessing\n    def prepare_train_features(examples):\n        # Some of the questions have lots of whitespace on the left, which is not useful and will make the\n        # truncation of the context fail (the tokenized question will take a lots of space). So we remove that\n        # left whitespace\n        examples[question_column_name] = [q.lstrip() for q in examples[question_column_name]]\n\n        # Tokenize our examples with truncation and maybe padding, but keep the overflows using a stride. This results\n        # in one example possible giving several features when a context is long, each of those features having a\n        # context that overlaps a bit the context of the previous feature.\n        tokenized_examples = tokenizer(\n            examples[question_column_name if pad_on_right else context_column_name],\n            examples[context_column_name if pad_on_right else question_column_name],\n            truncation=\"only_second\" if pad_on_right else \"only_first\",\n            max_length=max_seq_length,\n            stride=data_args.doc_stride,\n            return_overflowing_tokens=True,\n            return_offsets_mapping=True,\n            padding=padding,\n        )\n\n        # Since one example might give us several features if it has a long context, we need a map from a feature to\n        # its corresponding example. This key gives us just that.\n        sample_mapping = tokenized_examples.pop(\"overflow_to_sample_mapping\")\n        # The offset mappings will give us a map from token to character position in the original context. This will\n        # help us compute the start_positions and end_positions.\n        offset_mapping = tokenized_examples.pop(\"offset_mapping\")\n\n        # Let's label those examples!\n        tokenized_examples[\"start_positions\"] = []\n        tokenized_examples[\"end_positions\"] = []\n\n        for i, offsets in enumerate(offset_mapping):\n            # We will label impossible answers with the index of the CLS token.\n            input_ids = tokenized_examples[\"input_ids\"][i]\n            cls_index = input_ids.index(tokenizer.cls_token_id)\n\n            # Grab the sequence corresponding to that example (to know what is the context and what is the question).\n            sequence_ids = tokenized_examples.sequence_ids(i)\n\n            # One example can give several spans, this is the index of the example containing this span of text.\n            sample_index = sample_mapping[i]\n            answers = examples[answer_column_name][sample_index]\n            # If no answers are given, set the cls_index as answer.\n            if len(answers[\"answer_start\"]) == 0:\n                tokenized_examples[\"start_positions\"].append(cls_index)\n                tokenized_examples[\"end_positions\"].append(cls_index)\n            else:\n                # Start/end character index of the answer in the text.\n                start_char = answers[\"answer_start\"][0]\n                end_char = start_char + len(answers[\"text\"][0])\n\n                # Start token index of the current span in the text.\n                token_start_index = 0\n                while sequence_ids[token_start_index] != (1 if pad_on_right else 0):\n                    token_start_index += 1\n\n                # End token index of the current span in the text.\n                token_end_index = len(input_ids) - 1\n                while sequence_ids[token_end_index] != (1 if pad_on_right else 0):\n                    token_end_index -= 1\n\n                # Detect if the answer is out of the span (in which case this feature is labeled with the CLS index).\n                if not (offsets[token_start_index][0] <= start_char and offsets[token_end_index][1] >= end_char):\n                    tokenized_examples[\"start_positions\"].append(cls_index)\n                    tokenized_examples[\"end_positions\"].append(cls_index)\n                else:\n                    # Otherwise move the token_start_index and token_end_index to the two ends of the answer.\n                    # Note: we could go after the last offset if the answer is the last word (edge case).\n                    while token_start_index < len(offsets) and offsets[token_start_index][0] <= start_char:\n                        token_start_index += 1\n                    tokenized_examples[\"start_positions\"].append(token_start_index - 1)\n                    while offsets[token_end_index][1] >= end_char:\n                        token_end_index -= 1\n                    tokenized_examples[\"end_positions\"].append(token_end_index + 1)\n\n        return tokenized_examples\n\n    processed_datasets = {}\n    if training_args.do_train:\n        if \"train\" not in datasets:\n            raise ValueError(\"--do_train requires a train dataset\")\n        train_dataset = datasets[\"train\"]\n        if data_args.max_train_samples is not None:\n            # We will select sample from whole data if argument is specified\n            max_train_samples = min(len(train_dataset), data_args.max_train_samples)\n            train_dataset = train_dataset.select(range(max_train_samples))\n        # Create train feature from dataset\n        train_dataset = train_dataset.map(\n            prepare_train_features,\n            batched=True,\n            num_proc=data_args.preprocessing_num_workers,\n            remove_columns=column_names,\n            load_from_cache_file=not data_args.overwrite_cache,\n        )\n        if data_args.max_train_samples is not None:\n            # Number of samples might increase during Feature Creation, We select only specified max samples\n            max_train_samples = min(len(train_dataset), data_args.max_train_samples)\n            train_dataset = train_dataset.select(range(max_train_samples))\n        processed_datasets[\"train\"] = train_dataset\n\n    # Validation preprocessing\n    def prepare_validation_features(examples):\n        # Some of the questions have lots of whitespace on the left, which is not useful and will make the\n        # truncation of the context fail (the tokenized question will take a lots of space). So we remove that\n        # left whitespace\n        examples[question_column_name] = [q.lstrip() for q in examples[question_column_name]]\n\n        # Tokenize our examples with truncation and maybe padding, but keep the overflows using a stride. This results\n        # in one example possible giving several features when a context is long, each of those features having a\n        # context that overlaps a bit the context of the previous feature.\n        tokenized_examples = tokenizer(\n            examples[question_column_name if pad_on_right else context_column_name],\n            examples[context_column_name if pad_on_right else question_column_name],\n            truncation=\"only_second\" if pad_on_right else \"only_first\",\n            max_length=max_seq_length,\n            stride=data_args.doc_stride,\n            return_overflowing_tokens=True,\n            return_offsets_mapping=True,\n            padding=padding,\n        )\n\n        # Since one example might give us several features if it has a long context, we need a map from a feature to\n        # its corresponding example. This key gives us just that.\n        sample_mapping = tokenized_examples.pop(\"overflow_to_sample_mapping\")\n\n        # For evaluation, we will need to convert our predictions to substrings of the context, so we keep the\n        # corresponding example_id and we will store the offset mappings.\n        tokenized_examples[\"example_id\"] = []\n\n        for i in range(len(tokenized_examples[\"input_ids\"])):\n            # Grab the sequence corresponding to that example (to know what is the context and what is the question).\n            sequence_ids = tokenized_examples.sequence_ids(i)\n            context_index = 1 if pad_on_right else 0\n\n            # One example can give several spans, this is the index of the example containing this span of text.\n            sample_index = sample_mapping[i]\n            tokenized_examples[\"example_id\"].append(examples[\"id\"][sample_index])\n\n            # Set to None the offset_mapping that are not part of the context so it's easy to determine if a token\n            # position is part of the context or not.\n            tokenized_examples[\"offset_mapping\"][i] = [\n                (o if sequence_ids[k] == context_index else None)\n                for k, o in enumerate(tokenized_examples[\"offset_mapping\"][i])\n            ]\n\n        return tokenized_examples\n\n    if training_args.do_eval:\n        if \"validation\" not in datasets:\n            raise ValueError(\"--do_eval requires a validation dataset\")\n        eval_examples = datasets[\"validation\"]\n        if data_args.max_eval_samples is not None:\n            # We will select sample from whole data\n            max_eval_samples = min(len(eval_examples), data_args.max_eval_samples)\n            eval_examples = eval_examples.select(range(max_eval_samples))\n        # Validation Feature Creation\n        eval_dataset = eval_examples.map(\n            prepare_validation_features,\n            batched=True,\n            num_proc=data_args.preprocessing_num_workers,\n            remove_columns=column_names,\n            load_from_cache_file=not data_args.overwrite_cache,\n        )\n        if data_args.max_eval_samples is not None:\n            # During Feature creation dataset samples might increase, we will select required samples again\n            max_eval_samples = min(len(eval_dataset), data_args.max_eval_samples)\n            eval_dataset = eval_dataset.select(range(max_eval_samples))\n        processed_datasets[\"validation\"] = eval_dataset\n\n    if training_args.do_predict:\n        if \"test\" not in datasets:\n            raise ValueError(\"--do_predict requires a test dataset\")\n        predict_examples = datasets[\"test\"]\n        if data_args.max_predict_samples is not None:\n            # We will select sample from whole data\n            predict_examples = predict_examples.select(range(data_args.max_predict_samples))\n        # Predict Feature Creation\n        predict_dataset = predict_examples.map(\n            prepare_validation_features,\n            batched=True,\n            num_proc=data_args.preprocessing_num_workers,\n            remove_columns=column_names,\n            load_from_cache_file=not data_args.overwrite_cache,\n        )\n        if data_args.max_predict_samples is not None:\n            # During Feature creation dataset samples might increase, we will select required samples again\n            max_predict_samples = min(len(predict_dataset), data_args.max_predict_samples)\n            predict_dataset = predict_dataset.select(range(max_predict_samples))\n        processed_datasets[\"test\"] = predict_dataset\n    # endregion\n\n    # region Metrics and Post-processing:\n    def post_processing_function(examples, features, predictions, stage=\"eval\"):\n        # Post-processing: we match the start logits and end logits to answers in the original context.\n        predictions = postprocess_qa_predictions(\n            examples=examples,\n            features=features,\n            predictions=predictions,\n            version_2_with_negative=data_args.version_2_with_negative,\n            n_best_size=data_args.n_best_size,\n            max_answer_length=data_args.max_answer_length,\n            null_score_diff_threshold=data_args.null_score_diff_threshold,\n            output_dir=training_args.output_dir,\n            prefix=stage,\n        )\n        # Format the result to the format the metric expects.\n        if data_args.version_2_with_negative:\n            formatted_predictions = [\n                {\"id\": k, \"prediction_text\": v, \"no_answer_probability\": 0.0} for k, v in predictions.items()\n            ]\n        else:\n            formatted_predictions = [{\"id\": k, \"prediction_text\": v} for k, v in predictions.items()]\n\n        references = [{\"id\": ex[\"id\"], \"answers\": ex[answer_column_name]} for ex in examples]\n        return EvalPrediction(predictions=formatted_predictions, label_ids=references)\n\n    metric = evaluate.load(\n        \"squad_v2\" if data_args.version_2_with_negative else \"squad\", cache_dir=model_args.cache_dir\n    )\n\n    def compute_metrics(p: EvalPrediction):\n        return metric.compute(predictions=p.predictions, references=p.label_ids)\n\n    # endregion\n\n    with training_args.strategy.scope():\n        dataset_options = tf.data.Options()\n        dataset_options.experimental_distribute.auto_shard_policy = tf.data.experimental.AutoShardPolicy.OFF\n        num_replicas = training_args.strategy.num_replicas_in_sync\n\n        # region Load model and prepare datasets\n        if checkpoint is None:\n            model_path = model_args.model_name_or_path\n        else:\n            model_path = checkpoint\n        model = TFAutoModelForQuestionAnswering.from_pretrained(\n            model_path,\n            config=config,\n            cache_dir=model_args.cache_dir,\n            revision=model_args.model_revision,\n            token=model_args.token,\n            trust_remote_code=model_args.trust_remote_code,\n        )\n        if training_args.do_train:\n            training_dataset = model.prepare_tf_dataset(\n                processed_datasets[\"train\"],\n                shuffle=True,\n                batch_size=training_args.per_device_train_batch_size * num_replicas,\n                tokenizer=tokenizer,\n            )\n\n            training_dataset = training_dataset.with_options(dataset_options)\n\n            num_train_steps = len(training_dataset) * training_args.num_train_epochs\n            if training_args.warmup_steps > 0:\n                num_warmup_steps = training_args.warmup_steps\n            elif training_args.warmup_ratio > 0:\n                num_warmup_steps = int(num_train_steps * training_args.warmup_ratio)\n            else:\n                num_warmup_steps = 0\n\n            optimizer, schedule = create_optimizer(\n                init_lr=training_args.learning_rate,\n                num_train_steps=len(training_dataset) * training_args.num_train_epochs,\n                num_warmup_steps=num_warmup_steps,\n                adam_beta1=training_args.adam_beta1,\n                adam_beta2=training_args.adam_beta2,\n                adam_epsilon=training_args.adam_epsilon,\n                weight_decay_rate=training_args.weight_decay,\n                adam_global_clipnorm=training_args.max_grad_norm,\n            )\n\n            # Transformers models compute the right loss for their task by default when labels are passed, and will\n            # use this for training unless you specify your own loss function in compile().\n            model.compile(optimizer=optimizer, jit_compile=training_args.xla, metrics=[\"accuracy\"])\n\n        else:\n            # Optimizer doesn't matter as it won't be used anyway\n            model.compile(optimizer=\"sgd\", jit_compile=training_args.xla, metrics=[\"accuracy\"])\n            training_dataset = None\n\n        if training_args.do_eval:\n            eval_dataset = model.prepare_tf_dataset(\n                processed_datasets[\"validation\"],\n                shuffle=False,\n                batch_size=training_args.per_device_train_batch_size * num_replicas,\n                tokenizer=tokenizer,\n            )\n            eval_dataset = eval_dataset.with_options(dataset_options)\n        else:\n            eval_dataset = None\n\n        if training_args.do_predict:\n            predict_dataset = model.prepare_tf_dataset(\n                processed_datasets[\"test\"],\n                shuffle=False,\n                batch_size=training_args.per_device_eval_batch_size * num_replicas,\n                tokenizer=tokenizer,\n            )\n            predict_dataset = predict_dataset.with_options(dataset_options)\n        else:\n            predict_dataset = None\n\n        # endregion\n\n        # region Preparing push_to_hub and model card\n        push_to_hub_model_id = training_args.push_to_hub_model_id\n        model_name = model_args.model_name_or_path.split(\"/\")[-1]\n        if not push_to_hub_model_id:\n            if data_args.dataset_name is not None:\n                push_to_hub_model_id = f\"{model_name}-finetuned-{data_args.dataset_name}\"\n            else:\n                push_to_hub_model_id = f\"{model_name}-finetuned-question-answering\"\n\n        model_card_kwargs = {\"finetuned_from\": model_args.model_name_or_path, \"tasks\": \"question-answering\"}\n        if data_args.dataset_name is not None:\n            model_card_kwargs[\"dataset_tags\"] = data_args.dataset_name\n            if data_args.dataset_config_name is not None:\n                model_card_kwargs[\"dataset_args\"] = data_args.dataset_config_name\n                model_card_kwargs[\"dataset\"] = f\"{data_args.dataset_name} {data_args.dataset_config_name}\"\n            else:\n                model_card_kwargs[\"dataset\"] = data_args.dataset_name\n\n        if training_args.push_to_hub:\n            callbacks = [\n                PushToHubCallback(\n                    output_dir=training_args.output_dir,\n                    hub_model_id=push_to_hub_model_id,\n                    hub_token=training_args.push_to_hub_token,\n                    tokenizer=tokenizer,\n                    **model_card_kwargs,\n                )\n            ]\n        else:\n            callbacks = []\n        # endregion\n\n        # region Training and Evaluation\n\n        if training_args.do_train:\n            # Note that the validation and test datasets have been processed in a different way to the\n            # training datasets in this example, and so they don't have the same label structure.\n            # As such, we don't pass them directly to Keras, but instead get model predictions to evaluate\n            # after training.\n            model.fit(training_dataset, epochs=int(training_args.num_train_epochs), callbacks=callbacks)\n\n        if training_args.do_eval:\n            logger.info(\"*** Evaluation ***\")\n\n            # In this example, we compute advanced metrics at the end of training, but\n            # if you'd like to compute metrics every epoch that are too complex to be written as\n            # standard Keras metrics, you can use our KerasMetricCallback. See\n            # https://huggingface.co/docs/transformers/main/en/main_classes/keras_callbacks\n\n            eval_predictions = model.predict(eval_dataset)\n            if isinstance(eval_predictions.start_logits, tf.RaggedTensor):\n                # If predictions are RaggedTensor, we densify them. Since they are logits, padding with 0 is a bad idea!\n                # The reason is that a logit of 0 can often end up as quite a high probability value, sometimes even\n                # the highest probability in a sample. Instead, we use a large negative value, which ensures that the\n                # padding positions are correctly masked.\n                eval_start_logits = eval_predictions.start_logits.to_tensor(default_value=-1000).numpy()\n                eval_end_logits = eval_predictions.end_logits.to_tensor(default_value=-1000).numpy()\n            else:\n                eval_start_logits = eval_predictions.start_logits\n                eval_end_logits = eval_predictions.end_logits\n\n            post_processed_eval = post_processing_function(\n                datasets[\"validation\"],\n                processed_datasets[\"validation\"],\n                (eval_start_logits, eval_end_logits),\n            )\n            metrics = compute_metrics(post_processed_eval)\n            logging.info(\"Evaluation metrics:\")\n            for metric, value in metrics.items():\n                logging.info(f\"{metric}: {value:.3f}\")\n            if training_args.output_dir is not None:\n                output_eval_file = os.path.join(training_args.output_dir, \"all_results.json\")\n                with open(output_eval_file, \"w\") as writer:\n                    writer.write(json.dumps(metrics))\n        # endregion\n\n        # region Prediction\n        if training_args.do_predict:\n            logger.info(\"*** Predict ***\")\n\n            test_predictions = model.predict(predict_dataset)\n            if isinstance(test_predictions.start_logits, tf.RaggedTensor):\n                # If predictions are RaggedTensor, we densify them. Since they are logits, padding with 0 is a bad idea!\n                # The reason is that a logit of 0 can often end up as quite a high probability value, sometimes even\n                # the highest probability in a sample. Instead, we use a large negative value, which ensures that the\n                # padding positions are correctly masked.\n                test_start_logits = test_predictions.start_logits.to_tensor(default_value=-1000).numpy()\n                test_end_logits = test_predictions.end_logits.to_tensor(default_value=-1000).numpy()\n            else:\n                test_start_logits = test_predictions.start_logits\n                test_end_logits = test_predictions.end_logits\n            post_processed_test = post_processing_function(\n                datasets[\"test\"],\n                processed_datasets[\"test\"],\n                (test_start_logits, test_end_logits),\n            )\n            metrics = compute_metrics(post_processed_test)\n\n            logging.info(\"Test metrics:\")\n            for metric, value in metrics.items():\n                logging.info(f\"{metric}: {value:.3f}\")\n        # endregion\n\n    if training_args.output_dir is not None and not training_args.push_to_hub:\n        # If we're not pushing to hub, at least save a local copy when we're done\n\n\nSUMMARY:\nThis code is a template for a command-line interface (CLI) in PyTorch for training and fine-tuning a pre-trained Hugging Face Transformers model. \n\nThe CLI provides options to train, evaluate, and predict with the model after it has been loaded from a checkpoint file.\n\nKey Features:\n\n1. Training: The CLI allows users to specify the number of epochs, batch size, learning rate, decay factor, and other hyperparameters for training.\n2. Evaluation: Users can specify which datasets to use for evaluation during training (validation and/or test). Advanced metrics are also computed after evaluation is complete.\n3. Prediction: If desired, users can predict answers from a specific dataset using the trained model.\n\nImprovements That Could Be Suggested:\n\n1. Type Hints: The function parameters could have type hints to make the code more readable and self-documenting.\n2. Hyperparameter Tuning: A simple grid search or Random Search for hyperparameters would improve the performance of the model by finding optimal settings.\n3. Early Stopping: Implementing early stopping during training to prevent overfitting.\n4. Model Monitoring: Inbuilt mechanisms like monitoring loss, accuracy, and predictions on validation sets to track model's performance effectively.\n\nOverall, this code serves as a solid foundation for creating custom scripts using PyTorch Transformer models."}


created_at: 2025-02-05 09:13:46
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/summarization/run_summarization.py:285-745

metadata:
{"docstring": "# endregion", "end_line": 745, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/summarization/run_summarization.py", "name": "main", "start_line": 285, "type": "function", "chroma:document": "CODE:\n\ndef main():\n    # region Argument parsing\n    # See all possible arguments in src/transformers/training_args.py\n    # or by passing the --help flag to this script.\n    # We now keep distinct sets of args, for a cleaner separation of concerns.\n\n    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TFTrainingArguments))\n    if len(sys.argv) == 2 and sys.argv[1].endswith(\".json\"):\n        # If we pass only one argument to the script and it's the path to a json file,\n        # let's parse it to get our arguments.\n        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))\n    else:\n        model_args, data_args, training_args = parser.parse_args_into_dataclasses()\n\n    # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The\n    # information sent is the one passed as arguments along with your Python/PyTorch versions.\n    send_example_telemetry(\"run_summarization\", model_args, data_args, framework=\"tensorflow\")\n    # endregion\n\n    # region Logging\n    logging.basicConfig(\n        format=\"%(asctime)s - %(levelname)s - %(name)s - %(message)s\",\n        datefmt=\"%m/%d/%Y %H:%M:%S\",\n        handlers=[logging.StreamHandler(sys.stdout)],\n    )\n    logger.setLevel(logging.INFO)\n    datasets.utils.logging.set_verbosity(logging.INFO)\n    transformers.utils.logging.set_verbosity(logging.INFO)\n\n    # Log on each process the small summary:\n    logger.info(f\"Training/evaluation parameters {training_args}\")\n    # endregion\n\n    # region T5 special-casing\n    if data_args.source_prefix is None and model_args.model_name_or_path in [\n        \"google-t5/t5-small\",\n        \"google-t5/t5-base\",\n        \"google-t5/t5-large\",\n        \"google-t5/t5-3b\",\n        \"google-t5/t5-11b\",\n    ]:\n        logger.warning(\n            \"You're running a t5 model but didn't provide a source prefix, which is the expected, e.g. with \"\n            \"`--source_prefix 'summarize: ' `\"\n        )\n    # endregion\n\n    # region Detecting last checkpoint\n    last_checkpoint = None\n    if os.path.isdir(training_args.output_dir) and training_args.do_train and not training_args.overwrite_output_dir:\n        last_checkpoint = get_last_checkpoint(training_args.output_dir)\n        if last_checkpoint is None and len(os.listdir(training_args.output_dir)) > 0:\n            raise ValueError(\n                f\"Output directory ({training_args.output_dir}) already exists and is not empty. \"\n                \"Use --overwrite_output_dir to overcome.\"\n            )\n        elif last_checkpoint is not None and training_args.resume_from_checkpoint is None:\n            logger.info(\n                f\"Checkpoint detected, resuming training at {last_checkpoint}. To avoid this behavior, change \"\n                \"the `--output_dir` or add `--overwrite_output_dir` to train from scratch.\"\n            )\n    # endregion\n\n    # Set seed before initializing model.\n    set_seed(training_args.seed)\n\n    # region Load datasets\n    # Get the datasets: you can either provide your own CSV/JSON training and evaluation files (see below)\n    # or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/\n    # (the dataset will be downloaded automatically from the datasets Hub).\n    #\n    # For CSV/JSON files this script will use the first column for the full texts and the second column for the\n    # summaries (unless you specify column names for this with the `text_column` and `summary_column` arguments).\n    #\n    # In distributed training, the load_dataset function guarantee that only one local process can concurrently\n    # download the dataset.\n    if data_args.dataset_name is not None:\n        # Downloading and loading a dataset from the hub.\n        raw_datasets = load_dataset(\n            data_args.dataset_name,\n            data_args.dataset_config_name,\n            cache_dir=model_args.cache_dir,\n            token=model_args.token,\n            trust_remote_code=model_args.trust_remote_code,\n        )\n    else:\n        data_files = {}\n        if data_args.train_file is not None:\n            data_files[\"train\"] = data_args.train_file\n            extension = data_args.train_file.split(\".\")[-1]\n        if data_args.validation_file is not None:\n            data_files[\"validation\"] = data_args.validation_file\n            extension = data_args.validation_file.split(\".\")[-1]\n        if data_args.test_file is not None:\n            data_files[\"test\"] = data_args.test_file\n            extension = data_args.test_file.split(\".\")[-1]\n        raw_datasets = load_dataset(\n            extension,\n            data_files=data_files,\n            cache_dir=model_args.cache_dir,\n            token=model_args.token,\n        )\n    # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at\n    # https://huggingface.co/docs/datasets/loading_datasets.\n    # endregion\n\n    # region Load model config and tokenizer\n    #\n    # Distributed training:\n    # The .from_pretrained methods guarantee that only one local process can concurrently\n    # download model & vocab.\n\n    config = AutoConfig.from_pretrained(\n        model_args.config_name if model_args.config_name else model_args.model_name_or_path,\n        cache_dir=model_args.cache_dir,\n        revision=model_args.model_revision,\n        token=model_args.token,\n        trust_remote_code=model_args.trust_remote_code,\n    )\n    tokenizer = AutoTokenizer.from_pretrained(\n        model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path,\n        cache_dir=model_args.cache_dir,\n        use_fast=model_args.use_fast_tokenizer,\n        revision=model_args.model_revision,\n        token=model_args.token,\n        trust_remote_code=model_args.trust_remote_code,\n    )\n\n    prefix = data_args.source_prefix if data_args.source_prefix is not None else \"\"\n    # endregion\n\n    # region Dataset preprocessing\n    # We need to tokenize inputs and targets.\n    if training_args.do_train:\n        column_names = raw_datasets[\"train\"].column_names\n    elif training_args.do_eval:\n        column_names = raw_datasets[\"validation\"].column_names\n    else:\n        logger.info(\"There is nothing to do. Please pass `do_train`, and/or `do_eval`.\")\n        return\n\n    # Get the column names for input/target.\n    dataset_columns = summarization_name_mapping.get(data_args.dataset_name, None)\n    if data_args.text_column is None:\n        text_column = dataset_columns[0] if dataset_columns is not None else column_names[0]\n    else:\n        text_column = data_args.text_column\n        if text_column not in column_names:\n            raise ValueError(\n                f\"--text_column' value '{data_args.text_column}' needs to be one of: {', '.join(column_names)}\"\n            )\n    if data_args.summary_column is None:\n        summary_column = dataset_columns[1] if dataset_columns is not None else column_names[1]\n    else:\n        summary_column = data_args.summary_column\n        if summary_column not in column_names:\n            raise ValueError(\n                f\"--summary_column' value '{data_args.summary_column}' needs to be one of: {', '.join(column_names)}\"\n            )\n\n    # Temporarily set max_target_length for training.\n    max_target_length = data_args.max_target_length\n    padding = \"max_length\" if data_args.pad_to_max_length else False\n\n    def preprocess_function(examples):\n        inputs = examples[text_column]\n        targets = examples[summary_column]\n        inputs = [prefix + inp for inp in inputs]\n        model_inputs = tokenizer(inputs, max_length=data_args.max_source_length, padding=padding, truncation=True)\n\n        # Tokenize targets with the `text_target` keyword argument\n        labels = tokenizer(text_target=targets, max_length=max_target_length, padding=padding, truncation=True)\n\n        # If we are padding here, replace all tokenizer.pad_token_id in the labels by -100 when we want to ignore\n        # padding in the loss.\n        if padding == \"max_length\" and data_args.ignore_pad_token_for_loss:\n            labels[\"input_ids\"] = [\n                [(l if l != tokenizer.pad_token_id else -100) for l in label] for label in labels[\"input_ids\"]\n            ]\n\n        model_inputs[\"labels\"] = labels[\"input_ids\"]\n        return model_inputs\n\n    if training_args.do_train:\n        if \"train\" not in raw_datasets:\n            raise ValueError(\"--do_train requires a train dataset\")\n        train_dataset = raw_datasets[\"train\"]\n        if data_args.max_train_samples is not None:\n            max_train_samples = min(len(train_dataset), data_args.max_train_samples)\n            train_dataset = train_dataset.select(range(max_train_samples))\n        train_dataset = train_dataset.map(\n            preprocess_function,\n            batched=True,\n            num_proc=data_args.preprocessing_num_workers,\n            remove_columns=column_names,\n            load_from_cache_file=not data_args.overwrite_cache,\n            desc=\"Running tokenizer on train dataset\",\n        )\n    else:\n        train_dataset = None\n\n    if training_args.do_eval:\n        max_target_length = data_args.val_max_target_length\n        if \"validation\" not in raw_datasets:\n            raise ValueError(\"--do_eval requires a validation dataset\")\n        eval_dataset = raw_datasets[\"validation\"]\n        if data_args.max_eval_samples is not None:\n            max_eval_samples = min(len(eval_dataset), data_args.max_eval_samples)\n            eval_dataset = eval_dataset.select(range(max_eval_samples))\n        eval_dataset = eval_dataset.map(\n            preprocess_function,\n            batched=True,\n            num_proc=data_args.preprocessing_num_workers,\n            remove_columns=column_names,\n            load_from_cache_file=not data_args.overwrite_cache,\n            desc=\"Running tokenizer on validation dataset\",\n        )\n    else:\n        eval_dataset = None\n    # endregion\n\n    # region Text preprocessing\n    def postprocess_text(preds, labels):\n        preds = [pred.strip() for pred in preds]\n        labels = [label.strip() for label in labels]\n\n        # rougeLSum expects newline after each sentence\n        preds = [\"\\n\".join(nltk.sent_tokenize(pred)) for pred in preds]\n        labels = [\"\\n\".join(nltk.sent_tokenize(label)) for label in labels]\n\n        return preds, labels\n\n    # endregion\n\n    with training_args.strategy.scope():\n        # region Prepare model\n        model = TFAutoModelForSeq2SeqLM.from_pretrained(\n            model_args.model_name_or_path,\n            config=config,\n            cache_dir=model_args.cache_dir,\n            revision=model_args.model_revision,\n            token=model_args.token,\n            trust_remote_code=model_args.trust_remote_code,\n        )\n\n        # We resize the embeddings only when necessary to avoid index errors. If you are creating a model from scratch\n        # on a small vocab and want a smaller embedding size, remove this test.\n        embeddings = model.get_input_embeddings()\n\n        # Matt: This is a temporary workaround as we transition our models to exclusively using Keras embeddings.\n        #       As soon as the transition is complete, all embeddings should be keras.Embeddings layers, and\n        #       the weights will always be in embeddings.embeddings.\n        if hasattr(embeddings, \"embeddings\"):\n            embedding_size = embeddings.embeddings.shape[0]\n        else:\n            embedding_size = embeddings.weight.shape[0]\n        if len(tokenizer) > embedding_size:\n            model.resize_token_embeddings(len(tokenizer))\n        # endregion\n\n        # region Prepare TF Dataset objects\n        if model.config.decoder_start_token_id is None:\n            raise ValueError(\"Make sure that `config.decoder_start_token_id` is correctly defined\")\n\n        label_pad_token_id = -100 if data_args.ignore_pad_token_for_loss else tokenizer.pad_token_id\n        data_collator = DataCollatorForSeq2Seq(\n            tokenizer,\n            model=model,\n            label_pad_token_id=label_pad_token_id,\n            pad_to_multiple_of=128,  # Reduce the number of unique shapes for XLA, especially for generation\n            return_tensors=\"np\",\n        )\n\n        dataset_options = tf.data.Options()\n        dataset_options.experimental_distribute.auto_shard_policy = tf.data.experimental.AutoShardPolicy.OFF\n\n        num_replicas = training_args.strategy.num_replicas_in_sync\n        total_train_batch_size = training_args.per_device_train_batch_size * num_replicas\n        total_eval_batch_size = training_args.per_device_eval_batch_size * num_replicas\n\n        # model.prepare_tf_dataset() wraps a Hugging Face dataset in a tf.data.Dataset which is ready to use in\n        # training. This is the recommended way to use a Hugging Face dataset when training with Keras. You can also\n        # use the lower-level dataset.to_tf_dataset() method, but you will have to specify things like column names\n        # yourself if you use this method, whereas they are automatically inferred from the model input names when\n        # using model.prepare_tf_dataset()\n        # For more info see the docs:\n        # https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.TFPreTrainedModel.prepare_tf_dataset\n        # https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset.to_tf_dataset\n\n        tf_train_dataset = model.prepare_tf_dataset(\n            train_dataset,\n            collate_fn=data_collator,\n            batch_size=total_train_batch_size,\n            shuffle=True,\n        ).with_options(dataset_options)\n        tf_eval_dataset = model.prepare_tf_dataset(\n            eval_dataset,\n            collate_fn=data_collator,\n            batch_size=total_eval_batch_size,\n            shuffle=False,\n        ).with_options(dataset_options)\n        # endregion\n\n        # region Optimizer, loss and LR scheduling\n        num_train_steps = int(len(tf_train_dataset) * training_args.num_train_epochs)\n        if training_args.warmup_steps > 0:\n            num_warmup_steps = training_args.warmup_steps\n        elif training_args.warmup_ratio > 0:\n            num_warmup_steps = int(num_train_steps * training_args.warmup_ratio)\n        else:\n            num_warmup_steps = 0\n        if training_args.do_train:\n            optimizer, lr_schedule = create_optimizer(\n                init_lr=training_args.learning_rate,\n                num_train_steps=num_train_steps,\n                num_warmup_steps=num_warmup_steps,\n                adam_beta1=training_args.adam_beta1,\n                adam_beta2=training_args.adam_beta2,\n                adam_epsilon=training_args.adam_epsilon,\n                weight_decay_rate=training_args.weight_decay,\n                adam_global_clipnorm=training_args.max_grad_norm,\n            )\n        else:\n            optimizer = \"sgd\"  # Just write anything because we won't be using it\n\n        # endregion\n\n        # region Metric and KerasMetricCallback\n        if training_args.do_eval:\n            metric = evaluate.load(\"rouge\", cache_dir=model_args.cache_dir)\n\n            if data_args.val_max_target_length is None:\n                data_args.val_max_target_length = data_args.max_target_length\n\n            gen_kwargs = {\n                \"max_length\": data_args.val_max_target_length if data_args is not None else config.max_length,\n                \"num_beams\": data_args.num_beams,\n                \"no_repeat_ngram_size\": 0,  # Not supported under XLA right now, and some models set it by default\n            }\n\n            def compute_metrics(preds):\n                predictions, labels = preds\n                if isinstance(predictions, tuple):\n                    predictions = predictions[0]\n                decoded_preds = tokenizer.batch_decode(predictions, skip_special_tokens=True)\n                labels = np.where(labels != -100, labels, tokenizer.pad_token_id)\n                decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)\n                decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels)\n                metrics = metric.compute(predictions=decoded_preds, references=decoded_labels, use_stemmer=True)\n                # Only print the mid f-measures, but there are a lot of other statistics in there too!\n                metrics = {key: round(val.mid.fmeasure * 100, 4) for key, val in metrics.items()}\n                return metrics\n\n            # The KerasMetricCallback allows metrics that are too complex to write as standard Keras metrics\n            # to be computed each epoch. Any Python code can be included in the metric_fn. This is especially\n            # useful for metrics like BLEU and ROUGE that perform string comparisons on decoded model outputs.\n            # For more information, see the docs at\n            # https://huggingface.co/docs/transformers/main_classes/keras_callbacks#transformers.KerasMetricCallback\n\n            metric_callback = KerasMetricCallback(\n                metric_fn=compute_metrics,\n                eval_dataset=tf_eval_dataset,\n                predict_with_generate=True,\n                use_xla_generation=True,\n                generate_kwargs=gen_kwargs,\n            )\n            callbacks = [metric_callback]\n        else:\n            callbacks = []\n        # endregion\n\n        # region Preparing push_to_hub and model card\n        push_to_hub_model_id = training_args.push_to_hub_model_id\n        model_name = model_args.model_name_or_path.split(\"/\")[-1]\n        if not push_to_hub_model_id:\n            if data_args.dataset_name is not None:\n                push_to_hub_model_id = f\"{model_name}-finetuned-{data_args.dataset_name}\"\n            else:\n                push_to_hub_model_id = f\"{model_name}-finetuned-summarization\"\n\n        model_card_kwargs = {\"finetuned_from\": model_args.model_name_or_path, \"tasks\": \"summarization\"}\n        if data_args.dataset_name is not None:\n            model_card_kwargs[\"dataset_tags\"] = data_args.dataset_name\n            if data_args.dataset_config_name is not None:\n                model_card_kwargs[\"dataset_args\"] = data_args.dataset_config_name\n                model_card_kwargs[\"dataset\"] = f\"{data_args.dataset_name} {data_args.dataset_config_name}\"\n            else:\n                model_card_kwargs[\"dataset\"] = data_args.dataset_name\n\n        if training_args.push_to_hub:\n            # Because this training can be quite long, we save once per epoch.\n            callbacks.append(\n                PushToHubCallback(\n                    output_dir=training_args.output_dir,\n                    hub_model_id=push_to_hub_model_id,\n                    hub_token=training_args.push_to_hub_token,\n                    tokenizer=tokenizer,\n                    **model_card_kwargs,\n                )\n            )\n        # endregion\n\n        # region Training\n        # Transformers models compute the right loss for their task by default when labels are passed, and will\n        # use this for training unless you specify your own loss function in compile().\n        model.compile(optimizer=optimizer, jit_compile=training_args.xla)\n        eval_metrics = None\n        if training_args.do_train:\n            logger.info(\"***** Running training *****\")\n            logger.info(f\"  Num examples = {len(train_dataset)}\")\n            logger.info(f\"  Num Epochs = {training_args.num_train_epochs}\")\n            logger.info(f\"  Instantaneous batch size per device = {training_args.per_device_train_batch_size}\")\n            logger.info(f\"  Total train batch size = {total_train_batch_size}\")\n            logger.info(f\"  Total optimization steps = {num_train_steps}\")\n\n            if training_args.xla and not data_args.pad_to_max_length:\n                logger.warning(\n                    \"XLA training may be slow at first when --pad_to_max_length is not set \"\n                    \"until all possible shapes have been compiled.\"\n                )\n            history = model.fit(tf_train_dataset, epochs=int(training_args.num_train_epochs), callbacks=callbacks)\n            eval_metrics = {key: val[-1] for key, val in history.history.items()}\n        # endregion\n\n        # region Validation\n\n        if training_args.do_eval and not training_args.do_train:\n            # Do a standalone evaluation run\n            logger.info(\"Evaluation...\")\n\n            # Compiling generation with XLA yields enormous speedups, see https://huggingface.co/blog/tf-xla-generate\n            @tf.function(jit_compile=True)\n            def generate(**kwargs):\n                return model.generate(**kwargs)\n\n            for batch, labels in tf_eval_dataset:\n                batch.update(gen_kwargs)\n                generated_tokens = generate(**batch)\n                if isinstance(generated_tokens, tuple):\n                    generated_tokens = generated_tokens[0]\n                decoded_preds = tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)\n                labels = np.where(labels != -100, labels, tokenizer.pad_token_id)\n                decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)\n                decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels)\n\n                metric.add_batch(predictions=decoded_preds, references=decoded_labels)\n\n            eval_metrics = metric.compute(use_stemmer=True)\n\n            result = {key: round(val.mid.fmeasure * 100, 4) for key, val in eval_metrics.items()}\n            logger.info(result)\n        # endregion\n\n        if training_args.output_dir is not None and eval_metrics is not None:\n            output_eval_file = os.path.join(training_args.output_dir, \"all_results.json\")\n            with open(output_eval_file, \"w\") as writer:\n                writer.write(json.dumps(eval_metrics))\n\n        if training_args.output_dir is not None and not training_args.push_to_hub:\n            # If we're not pushing to hub, at least save a local copy when we're done\n\n\nSUMMARY:\nHere is a concise summary of the code:\n\nThe code sets up and trains a Hugging Face Transformers model for summarization. It takes in various training parameters, including dataset name, configuration, push to Hub, and output directory. It defines several callbacks for tracking progress, saving models, and logging results.\n\nKey components include:\n\n1. Model setup: importing necessary libraries, loading pre-trained model weights, and initializing model configuration.\n2. Data preparation: loading dataset, tokenizing input/outputs, and building dataloaders.\n3. Callbacks: setting up various callbacks for tracking progress (PushToHubCallback), saving models (model.save_pretrained()), logging results (KerasMetricCallback).\n4. Training loop: training the model using the Transformers `fit` method, passing in training dataset and epochs.\n5. Evaluation: running a standalone evaluation script after training, calculating evaluation metrics using the Hugging Face metric API.\n\nOverall, this code sets up a complete pipeline for training a summarization model with various customizations and logging capabilities."}


created_at: 2025-02-05 09:13:46
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/language-modeling/run_mlm.py:228-677

metadata:
{"docstring": "# endregion", "end_line": 677, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/language-modeling/run_mlm.py", "name": "main", "start_line": 228, "type": "function", "chroma:document": "CODE:\n\ndef main():\n    # region Argument Parsing\n    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TFTrainingArguments))\n    if len(sys.argv) == 2 and sys.argv[1].endswith(\".json\"):\n        # If we pass only one argument to the script and it's the path to a json file,\n        # let's parse it to get our arguments.\n        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))\n    else:\n        model_args, data_args, training_args = parser.parse_args_into_dataclasses()\n\n    # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The\n    # information sent is the one passed as arguments along with your Python/PyTorch versions.\n    send_example_telemetry(\"run_mlm\", model_args, data_args, framework=\"tensorflow\")\n\n    # Sanity checks\n    if data_args.dataset_name is None and data_args.train_file is None and data_args.validation_file is None:\n        raise ValueError(\"Need either a dataset name or a training/validation file.\")\n    else:\n        if data_args.train_file is not None:\n            extension = data_args.train_file.split(\".\")[-1]\n            assert extension in [\"csv\", \"json\", \"txt\"], \"`train_file` should be a csv, json or txt file.\"\n        if data_args.validation_file is not None:\n            extension = data_args.validation_file.split(\".\")[-1]\n            assert extension in [\"csv\", \"json\", \"txt\"], \"`validation_file` should be a csv, json or txt file.\"\n\n    if training_args.output_dir is not None:\n        training_args.output_dir = Path(training_args.output_dir)\n        os.makedirs(training_args.output_dir, exist_ok=True)\n\n    if isinstance(training_args.strategy, tf.distribute.TPUStrategy) and not data_args.pad_to_max_length:\n        logger.warning(\"We are training on TPU - forcing pad_to_max_length\")\n        data_args.pad_to_max_length = True\n    # endregion\n\n    # region Checkpoints\n    # Detecting last checkpoint.\n    checkpoint = None\n    if len(os.listdir(training_args.output_dir)) > 0 and not training_args.overwrite_output_dir:\n        config_path = training_args.output_dir / CONFIG_NAME\n        weights_path = training_args.output_dir / TF2_WEIGHTS_NAME\n        if config_path.is_file() and weights_path.is_file():\n            checkpoint = training_args.output_dir\n            logger.warning(\n                f\"Checkpoint detected, resuming training from checkpoint in {training_args.output_dir}. To avoid this\"\n                \" behavior, change the `--output_dir` or add `--overwrite_output_dir` to train from scratch.\"\n            )\n        else:\n            raise ValueError(\n                f\"Output directory ({training_args.output_dir}) already exists and is not empty. \"\n                \"Use --overwrite_output_dir to continue regardless.\"\n            )\n\n    # endregion\n\n    # region Setup logging\n    # accelerator.is_local_main_process is only True for one process per machine.\n    logger.setLevel(logging.INFO)\n    datasets.utils.logging.set_verbosity_warning()\n    transformers.utils.logging.set_verbosity_info()\n    # endregion\n\n    # If passed along, set the training seed now.\n    if training_args.seed is not None:\n        set_seed(training_args.seed)\n\n    # region Load datasets\n    # Get the datasets: you can either provide your own CSV/JSON/TXT training and evaluation files (see below)\n    # or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/\n    # (the dataset will be downloaded automatically from the datasets Hub).\n    #\n    # For CSV/JSON files, this script will use the column called 'text' or the first column if no column called\n    # 'text' is found. You can easily tweak this behavior (see below).\n    #\n    # In distributed training, the load_dataset function guarantee that only one local process can concurrently\n    # download the dataset.\n    if data_args.dataset_name is not None:\n        # Downloading and loading a dataset from the hub.\n        raw_datasets = load_dataset(\n            data_args.dataset_name,\n            data_args.dataset_config_name,\n            token=model_args.token,\n            trust_remote_code=model_args.trust_remote_code,\n        )\n        if \"validation\" not in raw_datasets.keys():\n            raw_datasets[\"validation\"] = load_dataset(\n                data_args.dataset_name,\n                data_args.dataset_config_name,\n                split=f\"train[:{data_args.validation_split_percentage}%]\",\n                token=model_args.token,\n                trust_remote_code=model_args.trust_remote_code,\n            )\n            raw_datasets[\"train\"] = load_dataset(\n                data_args.dataset_name,\n                data_args.dataset_config_name,\n                split=f\"train[{data_args.validation_split_percentage}%:]\",\n                token=model_args.token,\n                trust_remote_code=model_args.trust_remote_code,\n            )\n    else:\n        data_files = {}\n        if data_args.train_file is not None:\n            data_files[\"train\"] = data_args.train_file\n            extension = data_args.train_file.split(\".\")[-1]\n        if data_args.validation_file is not None:\n            data_files[\"validation\"] = data_args.validation_file\n            extension = data_args.validation_file.split(\".\")[-1]\n        if extension == \"txt\":\n            extension = \"text\"\n        raw_datasets = load_dataset(\n            extension,\n            data_files=data_files,\n            token=model_args.token,\n        )\n\n    # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at\n    # https://huggingface.co/docs/datasets/loading_datasets.\n    # endregion\n\n    # region Load pretrained model and tokenizer\n    #\n    # In distributed training, the .from_pretrained methods guarantee that only one local process can concurrently\n    # download model & vocab.\n    if checkpoint is not None:\n        config = AutoConfig.from_pretrained(\n            checkpoint, token=model_args.token, trust_remote_code=model_args.trust_remote_code\n        )\n    elif model_args.config_name:\n        config = AutoConfig.from_pretrained(\n            model_args.config_name, token=model_args.token, trust_remote_code=model_args.trust_remote_code\n        )\n    elif model_args.model_name_or_path:\n        config = AutoConfig.from_pretrained(\n            model_args.model_name_or_path, token=model_args.token, trust_remote_code=model_args.trust_remote_code\n        )\n    else:\n        config = CONFIG_MAPPING[model_args.model_type]()\n        logger.warning(\"You are instantiating a new config instance from scratch.\")\n\n    if model_args.tokenizer_name:\n        tokenizer = AutoTokenizer.from_pretrained(\n            model_args.tokenizer_name, token=model_args.token, trust_remote_code=model_args.trust_remote_code\n        )\n    elif model_args.model_name_or_path:\n        tokenizer = AutoTokenizer.from_pretrained(\n            model_args.model_name_or_path, token=model_args.token, trust_remote_code=model_args.trust_remote_code\n        )\n    else:\n        raise ValueError(\n            \"You are instantiating a new tokenizer from scratch. This is not supported by this script. \"\n            \"You can do it from another script, save it, and load it from here, using --tokenizer_name.\"\n        )\n    # endregion\n\n    # region Dataset preprocessing\n    # First we tokenize all the texts.\n    column_names = raw_datasets[\"train\"].column_names\n    text_column_name = \"text\" if \"text\" in column_names else column_names[0]\n\n    if data_args.max_seq_length is None:\n        max_seq_length = tokenizer.model_max_length\n        if max_seq_length > 1024:\n            logger.warning(\n                f\"The tokenizer picked seems to have a very large `model_max_length` ({tokenizer.model_max_length}). \"\n                \"Picking 1024 instead. You can reduce that default value by passing --max_seq_length xxx.\"\n            )\n            max_seq_length = 1024\n    else:\n        if data_args.max_seq_length > tokenizer.model_max_length:\n            logger.warning(\n                f\"The max_seq_length passed ({data_args.max_seq_length}) is larger than the maximum length for the \"\n                f\"model ({tokenizer.model_max_length}). Using max_seq_length={tokenizer.model_max_length}.\"\n            )\n        max_seq_length = min(data_args.max_seq_length, tokenizer.model_max_length)\n\n    if data_args.line_by_line:\n        # When using line_by_line, we just tokenize each nonempty line.\n        padding = \"max_length\" if data_args.pad_to_max_length else False\n\n        def tokenize_function(examples):\n            # Remove empty lines\n            examples[text_column_name] = [\n                line for line in examples[text_column_name] if len(line) > 0 and not line.isspace()\n            ]\n            return tokenizer(\n                examples[text_column_name],\n                padding=padding,\n                truncation=True,\n                max_length=max_seq_length,\n                # We use this option because DataCollatorForLanguageModeling (see below) is more efficient when it\n                # receives the `special_tokens_mask`.\n                return_special_tokens_mask=True,\n            )\n\n        tokenized_datasets = raw_datasets.map(\n            tokenize_function,\n            batched=True,\n            num_proc=data_args.preprocessing_num_workers,\n            remove_columns=[text_column_name],\n            load_from_cache_file=not data_args.overwrite_cache,\n            desc=\"Running tokenizer on dataset line_by_line\",\n        )\n    else:\n        # Otherwise, we tokenize every text, then concatenate them together before splitting them in smaller parts.\n        # We use `return_special_tokens_mask=True` because DataCollatorForLanguageModeling (see below) is more\n        # efficient when it receives the `special_tokens_mask`.\n        def tokenize_function(examples):\n            return tokenizer(examples[text_column_name], return_special_tokens_mask=True)\n\n        tokenized_datasets = raw_datasets.map(\n            tokenize_function,\n            batched=True,\n            num_proc=data_args.preprocessing_num_workers,\n            remove_columns=column_names,\n            load_from_cache_file=not data_args.overwrite_cache,\n            desc=\"Running tokenizer on every text in dataset\",\n        )\n\n        # Main data processing function that will concatenate all texts from our dataset and generate chunks of\n        # max_seq_length.\n        def group_texts(examples):\n            # Concatenate all texts.\n            concatenated_examples = {k: list(chain(*examples[k])) for k in examples.keys()}\n            total_length = len(concatenated_examples[list(examples.keys())[0]])\n            # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can\n            # customize this part to your needs.\n            if total_length >= max_seq_length:\n                total_length = (total_length // max_seq_length) * max_seq_length\n            # Split by chunks of max_len.\n            result = {\n                k: [t[i : i + max_seq_length] for i in range(0, total_length, max_seq_length)]\n                for k, t in concatenated_examples.items()\n            }\n            return result\n\n        # Note that with `batched=True`, this map processes 1,000 texts together, so group_texts throws away a\n        # remainder for each of those groups of 1,000 texts. You can adjust that batch_size here but a higher value\n        # might be slower to preprocess.\n        #\n        # To speed up this part, we use multiprocessing. See the documentation of the map method for more information:\n        # https://huggingface.co/docs/datasets/process#map\n\n        tokenized_datasets = tokenized_datasets.map(\n            group_texts,\n            batched=True,\n            num_proc=data_args.preprocessing_num_workers,\n            load_from_cache_file=not data_args.overwrite_cache,\n            desc=f\"Grouping texts in chunks of {max_seq_length}\",\n        )\n\n    train_dataset = tokenized_datasets[\"train\"]\n\n    if data_args.validation_file is not None:\n        eval_dataset = tokenized_datasets[\"validation\"]\n    else:\n        logger.info(\n            f\"Validation file not found: using {data_args.validation_split_percentage}% of the dataset as validation\"\n            \" as provided in data_args\"\n        )\n        train_indices, val_indices = train_test_split(\n            list(range(len(train_dataset))), test_size=data_args.validation_split_percentage / 100\n        )\n\n        eval_dataset = train_dataset.select(val_indices)\n        train_dataset = train_dataset.select(train_indices)\n\n    if data_args.max_train_samples is not None:\n        max_train_samples = min(len(train_dataset), data_args.max_train_samples)\n        train_dataset = train_dataset.select(range(max_train_samples))\n    if data_args.max_eval_samples is not None:\n        max_eval_samples = min(len(eval_dataset), data_args.max_eval_samples)\n        eval_dataset = eval_dataset.select(range(max_eval_samples))\n\n    # Log a few random samples from the training set:\n    for index in random.sample(range(len(train_dataset)), min(3, len(train_dataset))):\n        logger.info(f\"Sample {index} of the training set: {train_dataset[index]}.\")\n    # endregion\n\n    with training_args.strategy.scope():\n        # region Prepare model\n        if checkpoint is not None:\n            model = TFAutoModelForMaskedLM.from_pretrained(\n                checkpoint, config=config, token=model_args.token, trust_remote_code=model_args.trust_remote_code\n            )\n        elif model_args.model_name_or_path:\n            model = TFAutoModelForMaskedLM.from_pretrained(\n                model_args.model_name_or_path,\n                config=config,\n                token=model_args.token,\n                trust_remote_code=model_args.trust_remote_code,\n            )\n        else:\n            logger.info(\"Training new model from scratch\")\n            model = TFAutoModelForMaskedLM.from_config(\n                config, token=model_args.token, trust_remote_code=model_args.trust_remote_code\n            )\n\n        # We resize the embeddings only when necessary to avoid index errors. If you are creating a model from scratch\n        # on a small vocab and want a smaller embedding size, remove this test.\n        embeddings = model.get_input_embeddings()\n\n        # Matt: This is a temporary workaround as we transition our models to exclusively using Keras embeddings.\n        #       As soon as the transition is complete, all embeddings should be keras.Embeddings layers, and\n        #       the weights will always be in embeddings.embeddings.\n        if hasattr(embeddings, \"embeddings\"):\n            embedding_size = embeddings.embeddings.shape[0]\n        else:\n            embedding_size = embeddings.weight.shape[0]\n        if len(tokenizer) > embedding_size:\n            model.resize_token_embeddings(len(tokenizer))\n        # endregion\n\n        # region TF Dataset preparation\n        num_replicas = training_args.strategy.num_replicas_in_sync\n        data_collator = DataCollatorForLanguageModeling(\n            tokenizer=tokenizer, mlm_probability=data_args.mlm_probability, return_tensors=\"np\"\n        )\n        options = tf.data.Options()\n        options.experimental_distribute.auto_shard_policy = tf.data.experimental.AutoShardPolicy.OFF\n\n        # model.prepare_tf_dataset() wraps a Hugging Face dataset in a tf.data.Dataset which is ready to use in\n        # training. This is the recommended way to use a Hugging Face dataset when training with Keras. You can also\n        # use the lower-level dataset.to_tf_dataset() method, but you will have to specify things like column names\n        # yourself if you use this method, whereas they are automatically inferred from the model input names when\n        # using model.prepare_tf_dataset()\n        # For more info see the docs:\n        # https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.TFPreTrainedModel.prepare_tf_dataset\n        # https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset.to_tf_dataset\n\n        tf_train_dataset = model.prepare_tf_dataset(\n            train_dataset,\n            shuffle=True,\n            batch_size=num_replicas * training_args.per_device_train_batch_size,\n            collate_fn=data_collator,\n        ).with_options(options)\n\n        tf_eval_dataset = model.prepare_tf_dataset(\n            eval_dataset,\n            # labels are passed as input, as we will use the model's internal loss\n            shuffle=False,\n            batch_size=num_replicas * training_args.per_device_eval_batch_size,\n            collate_fn=data_collator,\n            drop_remainder=True,\n        ).with_options(options)\n        # endregion\n\n        # region Optimizer and loss\n        num_train_steps = len(tf_train_dataset) * int(training_args.num_train_epochs)\n        if training_args.warmup_steps > 0:\n            num_warmup_steps = training_args.warmup_steps\n        elif training_args.warmup_ratio > 0:\n            num_warmup_steps = int(num_train_steps * training_args.warmup_ratio)\n        else:\n            num_warmup_steps = 0\n\n        # Bias and layernorm weights are automatically excluded from the decay\n        optimizer, lr_schedule = create_optimizer(\n            init_lr=training_args.learning_rate,\n            num_train_steps=num_train_steps,\n            num_warmup_steps=num_warmup_steps,\n            adam_beta1=training_args.adam_beta1,\n            adam_beta2=training_args.adam_beta2,\n            adam_epsilon=training_args.adam_epsilon,\n            weight_decay_rate=training_args.weight_decay,\n            adam_global_clipnorm=training_args.max_grad_norm,\n        )\n\n        # Transformers models compute the right loss for their task by default when labels are passed, and will\n        # use this for training unless you specify your own loss function in compile().\n        model.compile(optimizer=optimizer, jit_compile=training_args.xla)\n        # endregion\n\n        # region Preparing push_to_hub and model card\n        push_to_hub_model_id = training_args.push_to_hub_model_id\n        model_name = model_args.model_name_or_path.split(\"/\")[-1]\n        if not push_to_hub_model_id:\n            if data_args.dataset_name is not None:\n                push_to_hub_model_id = f\"{model_name}-finetuned-{data_args.dataset_name}\"\n            else:\n                push_to_hub_model_id = f\"{model_name}-finetuned-mlm\"\n\n        model_card_kwargs = {\"finetuned_from\": model_args.model_name_or_path, \"tasks\": \"fill-mask\"}\n        if data_args.dataset_name is not None:\n            model_card_kwargs[\"dataset_tags\"] = data_args.dataset_name\n            if data_args.dataset_config_name is not None:\n                model_card_kwargs[\"dataset_args\"] = data_args.dataset_config_name\n                model_card_kwargs[\"dataset\"] = f\"{data_args.dataset_name} {data_args.dataset_config_name}\"\n            else:\n                model_card_kwargs[\"dataset\"] = data_args.dataset_name\n\n        if training_args.push_to_hub:\n            callbacks = [\n                PushToHubCallback(\n                    output_dir=training_args.output_dir,\n                    hub_model_id=push_to_hub_model_id,\n                    hub_token=training_args.push_to_hub_token,\n                    tokenizer=tokenizer,\n                    **model_card_kwargs,\n                )\n            ]\n        else:\n            callbacks = []\n        # endregion\n\n        # region Training and validation\n        logger.info(\"***** Running training *****\")\n        logger.info(f\"  Num examples = {len(train_dataset)}\")\n        logger.info(f\"  Num Epochs = {training_args.num_train_epochs}\")\n        logger.info(f\"  Instantaneous batch size per device = {training_args.per_device_train_batch_size}\")\n        logger.info(f\"  Total train batch size = {training_args.per_device_train_batch_size * num_replicas}\")\n\n        # For long training runs, you may wish to use the PushToHub() callback here to save intermediate checkpoints\n        # to the Hugging Face Hub rather than just pushing the finished model.\n        # See https://huggingface.co/docs/transformers/main_classes/keras_callbacks#transformers.PushToHubCallback\n\n        history = model.fit(\n            tf_train_dataset,\n            validation_data=tf_eval_dataset,\n            epochs=int(training_args.num_train_epochs),\n            callbacks=callbacks,\n        )\n        train_loss = history.history[\"loss\"][-1]\n        try:\n            train_perplexity = math.exp(train_loss)\n        except OverflowError:\n            train_perplexity = math.inf\n        logger.info(f\"  Final train loss: {train_loss:.3f}\")\n        logger.info(f\"  Final train perplexity: {train_perplexity:.3f}\")\n\n    validation_loss = history.history[\"val_loss\"][-1]\n    try:\n        validation_perplexity = math.exp(validation_loss)\n    except OverflowError:\n        validation_perplexity = math.inf\n    logger.info(f\"  Final validation loss: {validation_loss:.3f}\")\n    logger.info(f\"  Final validation perplexity: {validation_perplexity:.3f}\")\n\n    if training_args.output_dir is not None:\n        output_eval_file = os.path.join(training_args.output_dir, \"all_results.json\")\n        results_dict = {}\n        results_dict[\"train_loss\"] = train_loss\n        results_dict[\"train_perplexity\"] = train_perplexity\n        results_dict[\"eval_loss\"] = validation_loss\n        results_dict[\"eval_perplexity\"] = validation_perplexity\n        with open(output_eval_file, \"w\") as writer:\n            writer.write(json.dumps(results_dict))\n        # endregion\n\n    if training_args.output_dir is not None and not training_args.push_to_hub:\n        # If we're not pushing to hub, at least save a local copy when we're done\n\n\nSUMMARY:\nThis code is for training a masked language modeling (MLM) model using the Hugging Face Transformers library. It takes in a large dataset and fine-tunes the model on it to predict the missing words in sentences.\n\nHere's a simplified breakdown of what the code does:\n\n1. **Data Preparation**: The code loads a dataset and splits it into training and validation sets.\n2. **Model Initialization**: A new instance of the MLM model is created with the specified configuration.\n3. **Optimizer and Loss Function**: An optimizer (AdamW) and loss function are defined for the model, which will update the model's parameters to minimize the loss.\n4. **Training Loop**: The training loop iterates through the training data, updating the model's parameters using the optimizer. Every few epochs, the model also evaluates its performance on the validation set.\n5. **Model Evaluation**: After training is complete, the final accuracy, perplexity, and other metrics are calculated for both the training and validation sets.\n6. **Model Saving**: If desired, a local copy of the trained model is saved to disk.\n\nThis code can be used as a base for fine-tuning pre-trained language models on custom datasets. It provides a solid framework for experimenting with different model architectures and hyperparameters."}


created_at: 2025-02-05 09:13:46
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/language-modeling/run_clm.py:222-653

metadata:
{"docstring": "# endregion", "end_line": 653, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/language-modeling/run_clm.py", "name": "main", "start_line": 222, "type": "function", "chroma:document": "CODE:\n\ndef main():\n    # region Argument Parsing\n    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TFTrainingArguments))\n    if len(sys.argv) == 2 and sys.argv[1].endswith(\".json\"):\n        # If we pass only one argument to the script and it's the path to a json file,\n        # let's parse it to get our arguments.\n        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))\n    else:\n        model_args, data_args, training_args = parser.parse_args_into_dataclasses()\n\n    # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The\n    # information sent is the one passed as arguments along with your Python/PyTorch versions.\n    send_example_telemetry(\"run_clm\", model_args, data_args, framework=\"tensorflow\")\n\n    # Sanity checks\n    if data_args.dataset_name is None and data_args.train_file is None and data_args.validation_file is None:\n        raise ValueError(\"Need either a dataset name or a training/validation file.\")\n    else:\n        if data_args.train_file is not None:\n            extension = data_args.train_file.split(\".\")[-1]\n            assert extension in [\"csv\", \"json\", \"txt\"], \"`train_file` should be a csv, json or txt file.\"\n        if data_args.validation_file is not None:\n            extension = data_args.validation_file.split(\".\")[-1]\n            assert extension in [\"csv\", \"json\", \"txt\"], \"`validation_file` should be a csv, json or txt file.\"\n\n    if training_args.output_dir is not None:\n        training_args.output_dir = Path(training_args.output_dir)\n        os.makedirs(training_args.output_dir, exist_ok=True)\n    # endregion\n\n    # region Checkpoints\n    # Detecting last checkpoint.\n    checkpoint = None\n    if len(os.listdir(training_args.output_dir)) > 0 and not training_args.overwrite_output_dir:\n        config_path = training_args.output_dir / CONFIG_NAME\n        weights_path = training_args.output_dir / TF2_WEIGHTS_NAME\n        if config_path.is_file() and weights_path.is_file():\n            checkpoint = training_args.output_dir\n            logger.info(\n                f\"Checkpoint detected, resuming training from checkpoint in {training_args.output_dir}. To avoid this\"\n                \" behavior, change the `--output_dir` or add `--overwrite_output_dir` to train from scratch.\"\n            )\n        else:\n            raise ValueError(\n                f\"Output directory ({training_args.output_dir}) already exists and is not empty. \"\n                \"Use --overwrite_output_dir to continue regardless.\"\n            )\n\n    # endregion\n\n    # region Setup logging\n    # accelerator.is_local_main_process is only True for one process per machine.\n    logger.setLevel(logging.INFO)\n    datasets.utils.logging.set_verbosity_warning()\n    transformers.utils.logging.set_verbosity_info()\n    # endregion\n\n    # If passed along, set the training seed now.\n    if training_args.seed is not None:\n        set_seed(training_args.seed)\n\n    # region Load datasets\n    # Get the datasets: you can either provide your own CSV/JSON/TXT training and evaluation files (see below)\n    # or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/\n    # (the dataset will be downloaded automatically from the datasets Hub).\n    #\n    # For CSV/JSON files, this script will use the column called 'text' or the first column if no column called\n    # 'text' is found. You can easily tweak this behavior (see below).\n    #\n    # In distributed training, the load_dataset function guarantee that only one local process can concurrently\n    # download the dataset.\n    if data_args.dataset_name is not None:\n        # Downloading and loading a dataset from the hub.\n        raw_datasets = load_dataset(\n            data_args.dataset_name,\n            data_args.dataset_config_name,\n            cache_dir=model_args.cache_dir,\n            token=model_args.token,\n            trust_remote_code=model_args.trust_remote_code,\n        )\n        if \"validation\" not in raw_datasets.keys():\n            raw_datasets[\"validation\"] = load_dataset(\n                data_args.dataset_name,\n                data_args.dataset_config_name,\n                split=f\"train[:{data_args.validation_split_percentage}%]\",\n                cache_dir=model_args.cache_dir,\n                token=model_args.token,\n                trust_remote_code=model_args.trust_remote_code,\n            )\n            raw_datasets[\"train\"] = load_dataset(\n                data_args.dataset_name,\n                data_args.dataset_config_name,\n                split=f\"train[{data_args.validation_split_percentage}%:]\",\n                cache_dir=model_args.cache_dir,\n                token=model_args.token,\n                trust_remote_code=model_args.trust_remote_code,\n            )\n    else:\n        data_files = {}\n        dataset_args = {}\n        if data_args.train_file is not None:\n            data_files[\"train\"] = data_args.train_file\n        if data_args.validation_file is not None:\n            data_files[\"validation\"] = data_args.validation_file\n        extension = (\n            data_args.train_file.split(\".\")[-1]\n            if data_args.train_file is not None\n            else data_args.validation_file.split(\".\")[-1]\n        )\n        if extension == \"txt\":\n            extension = \"text\"\n            dataset_args[\"keep_linebreaks\"] = data_args.keep_linebreaks\n        raw_datasets = load_dataset(\n            extension,\n            data_files=data_files,\n            cache_dir=model_args.cache_dir,\n            token=model_args.token,\n            **dataset_args,\n        )\n        # If no validation data is there, validation_split_percentage will be used to divide the dataset.\n        if \"validation\" not in raw_datasets.keys():\n            raw_datasets[\"validation\"] = load_dataset(\n                extension,\n                data_files=data_files,\n                split=f\"train[:{data_args.validation_split_percentage}%]\",\n                cache_dir=model_args.cache_dir,\n                token=model_args.token,\n                **dataset_args,\n            )\n            raw_datasets[\"train\"] = load_dataset(\n                extension,\n                data_files=data_files,\n                split=f\"train[{data_args.validation_split_percentage}%:]\",\n                cache_dir=model_args.cache_dir,\n                token=model_args.token,\n                **dataset_args,\n            )\n    # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at\n    # https://huggingface.co/docs/datasets/loading_datasets.\n    # endregion\n\n    # region Load pretrained model and tokenizer\n    #\n    # In distributed training, the .from_pretrained methods guarantee that only one local process can concurrently\n    # download model & vocab.\n    if model_args.config_name:\n        config = AutoConfig.from_pretrained(\n            model_args.config_name,\n            token=model_args.token,\n            trust_remote_code=model_args.trust_remote_code,\n        )\n    elif model_args.model_name_or_path:\n        config = AutoConfig.from_pretrained(\n            model_args.model_name_or_path, token=model_args.token, trust_remote_code=model_args.trust_remote_code\n        )\n    else:\n        config = CONFIG_MAPPING[model_args.model_type]()\n        logger.warning(\"You are instantiating a new config instance from scratch.\")\n\n    if model_args.tokenizer_name:\n        tokenizer = AutoTokenizer.from_pretrained(\n            model_args.tokenizer_name, token=model_args.token, trust_remote_code=model_args.trust_remote_code\n        )\n    elif model_args.model_name_or_path:\n        tokenizer = AutoTokenizer.from_pretrained(\n            model_args.model_name_or_path, token=model_args.token, trust_remote_code=model_args.trust_remote_code\n        )\n    else:\n        raise ValueError(\n            \"You are instantiating a new tokenizer from scratch. This is not supported by this script. \"\n            \"You can do it from another script, save it, and load it from here, using --tokenizer_name.\"\n        )\n    # endregion\n\n    # region Dataset preprocessing\n    # First we tokenize all the texts.\n    column_names = raw_datasets[\"train\"].column_names\n    text_column_name = \"text\" if \"text\" in column_names else column_names[0]\n\n    def tokenize_function(examples):\n        return tokenizer(examples[text_column_name])\n\n    tokenized_datasets = raw_datasets.map(\n        tokenize_function,\n        batched=True,\n        num_proc=data_args.preprocessing_num_workers,\n        remove_columns=column_names,\n        load_from_cache_file=not data_args.overwrite_cache,\n        desc=\"Running tokenizer on dataset\",\n    )\n\n    if data_args.block_size is None:\n        block_size = tokenizer.model_max_length\n        if block_size > config.max_position_embeddings:\n            logger.warning(\n                f\"The tokenizer picked seems to have a very large `model_max_length` ({tokenizer.model_max_length}). \"\n                f\"Using block_size={min(1024, config.max_position_embeddings)} instead. You can change that default value by passing --block_size xxx.\"\n            )\n            block_size = min(1024, config.max_position_embeddings)\n    else:\n        if data_args.block_size > tokenizer.model_max_length:\n            logger.warning(\n                f\"The block_size passed ({data_args.block_size}) is larger than the maximum length for the model \"\n                f\"({tokenizer.model_max_length}). Using block_size={tokenizer.model_max_length}.\"\n            )\n        block_size = min(data_args.block_size, tokenizer.model_max_length)\n\n    # Main data processing function that will concatenate all texts from our dataset and generate chunks of block_size.\n    def group_texts(examples):\n        # Concatenate all texts.\n        concatenated_examples = {k: list(chain(*examples[k])) for k in examples.keys()}\n        total_length = len(concatenated_examples[list(examples.keys())[0]])\n        # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can\n        # customize this part to your needs.\n        if total_length >= block_size:\n            total_length = (total_length // block_size) * block_size\n        # Split by chunks of max_len.\n        result = {\n            k: [t[i : i + block_size] for i in range(0, total_length, block_size)]\n            for k, t in concatenated_examples.items()\n        }\n        result[\"labels\"] = result[\"input_ids\"].copy()\n        return result\n\n    # Note that with `batched=True`, this map processes 1,000 texts together, so group_texts throws away a remainder\n    # for each of those groups of 1,000 texts. You can adjust that batch_size here but a higher value might be slower\n    # to preprocess.\n    #\n    # To speed up this part, we use multiprocessing. See the documentation of the map method for more information:\n    # https://huggingface.co/docs/datasets/process#map\n\n    lm_datasets = tokenized_datasets.map(\n        group_texts,\n        batched=True,\n        num_proc=data_args.preprocessing_num_workers,\n        load_from_cache_file=not data_args.overwrite_cache,\n        desc=f\"Grouping texts in chunks of {block_size}\",\n    )\n\n    train_dataset = lm_datasets[\"train\"]\n    if data_args.validation_file is not None:\n        eval_dataset = lm_datasets[\"validation\"]\n    else:\n        logger.info(\n            f\"Validation file not found: using {data_args.validation_split_percentage}% of the dataset as validation\"\n            \" as provided in data_args\"\n        )\n        train_indices, val_indices = train_test_split(\n            list(range(len(train_dataset))), test_size=data_args.validation_split_percentage / 100\n        )\n\n        eval_dataset = train_dataset.select(val_indices)\n        train_dataset = train_dataset.select(train_indices)\n\n    if data_args.max_train_samples is not None:\n        max_train_samples = min(len(train_dataset), data_args.max_train_samples)\n        train_dataset = train_dataset.select(range(max_train_samples))\n    if data_args.max_eval_samples is not None:\n        max_eval_samples = min(len(eval_dataset), data_args.max_eval_samples)\n        eval_dataset = eval_dataset.select(range(max_eval_samples))\n\n    # Log a few random samples from the training set:\n    for index in random.sample(range(len(train_dataset)), min(3, len(train_dataset))):\n        logger.info(f\"Sample {index} of the training set: {train_dataset[index]}.\")\n    # endregion\n\n    with training_args.strategy.scope():\n        # region Prepare model\n        if checkpoint is not None:\n            model = TFAutoModelForCausalLM.from_pretrained(\n                checkpoint, config=config, token=model_args.token, trust_remote_code=model_args.trust_remote_code\n            )\n        elif model_args.model_name_or_path:\n            model = TFAutoModelForCausalLM.from_pretrained(\n                model_args.model_name_or_path,\n                config=config,\n                token=model_args.token,\n                trust_remote_code=model_args.trust_remote_code,\n            )\n        else:\n            logger.info(\"Training new model from scratch\")\n            model = TFAutoModelForCausalLM.from_config(\n                config, token=model_args.token, trust_remote_code=model_args.trust_remote_code\n            )\n\n        # We resize the embeddings only when necessary to avoid index errors. If you are creating a model from scratch\n        # on a small vocab and want a smaller embedding size, remove this test.\n        embeddings = model.get_input_embeddings()\n\n        # Matt: This is a temporary workaround as we transition our models to exclusively using Keras embeddings.\n        #       As soon as the transition is complete, all embeddings should be keras.Embeddings layers, and\n        #       the weights will always be in embeddings.embeddings.\n        if hasattr(embeddings, \"embeddings\"):\n            embedding_size = embeddings.embeddings.shape[0]\n        else:\n            embedding_size = embeddings.weight.shape[0]\n        if len(tokenizer) > embedding_size:\n            model.resize_token_embeddings(len(tokenizer))\n        # endregion\n\n        # region TF Dataset preparation\n        num_replicas = training_args.strategy.num_replicas_in_sync\n        options = tf.data.Options()\n        options.experimental_distribute.auto_shard_policy = tf.data.experimental.AutoShardPolicy.OFF\n\n        # model.prepare_tf_dataset() wraps a Hugging Face dataset in a tf.data.Dataset which is ready to use in\n        # training. This is the recommended way to use a Hugging Face dataset when training with Keras. You can also\n        # use the lower-level dataset.to_tf_dataset() method, but you will have to specify things like column names\n        # yourself if you use this method, whereas they are automatically inferred from the model input names when\n        # using model.prepare_tf_dataset()\n        # For more info see the docs:\n        # https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.TFPreTrainedModel.prepare_tf_dataset\n        # https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset.to_tf_dataset\n\n        tf_train_dataset = model.prepare_tf_dataset(\n            train_dataset,\n            shuffle=True,\n            batch_size=num_replicas * training_args.per_device_train_batch_size,\n        ).with_options(options)\n\n        tf_eval_dataset = model.prepare_tf_dataset(\n            eval_dataset,\n            shuffle=False,\n            batch_size=num_replicas * training_args.per_device_eval_batch_size,\n            drop_remainder=True,\n        ).with_options(options)\n        # endregion\n\n        # region Optimizer and loss\n        num_train_steps = len(tf_train_dataset) * int(training_args.num_train_epochs)\n        if training_args.warmup_steps > 0:\n            num_warmup_steps = training_args.warmup_steps\n        elif training_args.warmup_ratio > 0:\n            num_warmup_steps = int(num_train_steps * training_args.warmup_ratio)\n        else:\n            num_warmup_steps = 0\n\n        # Bias and layernorm weights are automatically excluded from the decay\n        optimizer, lr_schedule = create_optimizer(\n            init_lr=training_args.learning_rate,\n            num_train_steps=num_train_steps,\n            num_warmup_steps=num_warmup_steps,\n            adam_beta1=training_args.adam_beta1,\n            adam_beta2=training_args.adam_beta2,\n            adam_epsilon=training_args.adam_epsilon,\n            weight_decay_rate=training_args.weight_decay,\n            adam_global_clipnorm=training_args.max_grad_norm,\n        )\n\n        # Transformers models compute the right loss for their task by default when labels are passed, and will\n        # use this for training unless you specify your own loss function in compile().\n        model.compile(optimizer=optimizer, jit_compile=training_args.xla)\n        # endregion\n\n        # region Preparing push_to_hub and model card\n        push_to_hub_model_id = training_args.push_to_hub_model_id\n        model_name = model_args.model_name_or_path.split(\"/\")[-1]\n        if not push_to_hub_model_id:\n            if data_args.dataset_name is not None:\n                push_to_hub_model_id = f\"{model_name}-finetuned-{data_args.dataset_name}\"\n            else:\n                push_to_hub_model_id = f\"{model_name}-finetuned-clm\"\n\n        model_card_kwargs = {\"finetuned_from\": model_args.model_name_or_path, \"tasks\": \"text-generation\"}\n        if data_args.dataset_name is not None:\n            model_card_kwargs[\"dataset_tags\"] = data_args.dataset_name\n            if data_args.dataset_config_name is not None:\n                model_card_kwargs[\"dataset_args\"] = data_args.dataset_config_name\n                model_card_kwargs[\"dataset\"] = f\"{data_args.dataset_name} {data_args.dataset_config_name}\"\n            else:\n                model_card_kwargs[\"dataset\"] = data_args.dataset_name\n\n        if training_args.push_to_hub:\n            callbacks = [\n                PushToHubCallback(\n                    output_dir=training_args.output_dir,\n                    hub_model_id=push_to_hub_model_id,\n                    hub_token=training_args.push_to_hub_token,\n                    tokenizer=tokenizer,\n                    **model_card_kwargs,\n                )\n            ]\n        else:\n            callbacks = []\n        # endregion\n\n        # region Training and validation\n        logger.info(\"***** Running training *****\")\n        logger.info(f\"  Num examples = {len(train_dataset)}\")\n        logger.info(f\"  Num Epochs = {training_args.num_train_epochs}\")\n        logger.info(f\"  Instantaneous batch size per device = {training_args.per_device_train_batch_size}\")\n        logger.info(f\"  Total train batch size = {training_args.per_device_train_batch_size * num_replicas}\")\n\n        # For long training runs, you may wish to use the PushToHub() callback here to save intermediate checkpoints\n        # to the Hugging Face Hub rather than just pushing the finished model.\n        # See https://huggingface.co/docs/transformers/main_classes/keras_callbacks#transformers.PushToHubCallback\n\n        history = model.fit(\n            tf_train_dataset,\n            validation_data=tf_eval_dataset,\n            epochs=int(training_args.num_train_epochs),\n            callbacks=callbacks,\n        )\n        train_loss = history.history[\"loss\"][-1]\n        try:\n            train_perplexity = math.exp(train_loss)\n        except OverflowError:\n            train_perplexity = math.inf\n        logger.info(f\"  Final train loss: {train_loss:.3f}\")\n        logger.info(f\"  Final train perplexity: {train_perplexity:.3f}\")\n        validation_loss = history.history[\"val_loss\"][-1]\n        try:\n            validation_perplexity = math.exp(validation_loss)\n        except OverflowError:\n            validation_perplexity = math.inf\n        logger.info(f\"  Final validation loss: {validation_loss:.3f}\")\n        logger.info(f\"  Final validation perplexity: {validation_perplexity:.3f}\")\n\n        if training_args.output_dir is not None:\n            output_eval_file = os.path.join(training_args.output_dir, \"all_results.json\")\n            results_dict = {}\n            results_dict[\"train_loss\"] = train_loss\n            results_dict[\"train_perplexity\"] = train_perplexity\n            results_dict[\"eval_loss\"] = validation_loss\n            results_dict[\"eval_perplexity\"] = validation_perplexity\n            with open(output_eval_file, \"w\") as writer:\n                writer.write(json.dumps(results_dict))\n        # endregion\n\n    if training_args.output_dir is not None and not training_args.push_to_hub:\n        # If we're not pushing to hub, at least save a local copy when we're done\n\n\nSUMMARY:\nHere's a concise summary of the code:\n\n**Training a Text Generation Model**\n\nThis is a script that trains a text generation model using the Hugging Face Transformers library. The model takes in a set of input data and generates text based on the task provided.\n\n**Key Steps:**\n\n1. **Data Preparation**: Load the dataset into the model.\n2. **Model Initialization**: Initialize the model with the given parameters (e.g., model name, learning rate).\n3. **Loss Function and Optimizer**: Define the loss function and optimizer for training.\n4. **Model Compilation**: Compile the model with the chosen optimizer and loss function.\n5. **Training**: Train the model on the dataset using the defined hyperparameters.\n6. **Validation**: Validate the model's performance on a separate validation set.\n\n**Key Hyperparameters:**\n\n* Learning rate\n* Number of epochs\n* Batch size\n* Regularization parameters (e.g., weight decay, Adam beta-values)\n\n**Additional Features:**\n\n* Pushing the final trained model to the Hugging Face Hub for sharing.\n* Saving the evaluation results as a JSON file.\n\nOverall, this script provides a flexible framework for training and fine-tuning text generation models using Transformers."}


created_at: 2025-02-05 09:13:46
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/image-classification/run_image_classification.py:65-68

metadata:
{"docstring": "", "end_line": 68, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/image-classification/run_image_classification.py", "name": "pil_loader", "start_line": 65, "type": "function", "chroma:document": "CODE:\n\ndef pil_loader(path: str):\n    with open(path, \"rb\") as f:\n        im = Image.open(f)\n\n\nSUMMARY:\n**pil_loader Function Summary**\n\n* **Purpose:** Loads and converts an image to RGB format using the Python Imaging Library (PIL).\n\n**Function Details:**\n\n* Reads an image from the specified `path` file in binary mode (`\"rb\"`).\n* Uses PIL's `Image.open()` function to load the image.\n* Converts the loaded image to the \"RGB\" color space using the `convert()` method.\n\n**External Dependencies:**\n- Python Imaging Library (PIL)\n\nNo additional interfaces or classes are implemented or inherited by this function."}


created_at: 2025-02-05 09:13:46
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/image-classification/run_image_classification.py:185-192

metadata:
{"docstring": "", "end_line": 192, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/image-classification/run_image_classification.py", "name": "center_crop", "start_line": 185, "type": "function", "chroma:document": "CODE:\n\ndef center_crop(image, size):\n    size = (size, size) if isinstance(size, int) else size\n    orig_height, orig_width, _ = image.shape\n    crop_height, crop_width = size\n    top = (orig_height - orig_width) // 2\n    left = (orig_width - crop_width) // 2\n    image = tf.image.crop_to_bounding_box(image, top, left, crop_height, crop_width)\n\n\nSUMMARY:\n**Summary:**\n\nThis function `(center_crop)` takes an `image` and optional `size` parameters. It centers a cropped region to the specified size within the original image.\n\n**Implementation Details:**\n\n- **Crop Region Calculation**: The top-left coordinates are calculated based on the difference between the image dimensions and the crop size.\n- **Image Cropping**: The `tf.image.crop_to_bounding_box` function is used to extract the centered cropped region from the original image.\n\n**Notable Dependencies/Requirements:**\n\n- `tf` (TensorFlow) for image processing functionality\n- `(size, size)` or exact integer size as input, defaulting to square if an integer is provided"}


created_at: 2025-02-05 09:13:46
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/image-classification/run_image_classification.py:197-217

metadata:
{"docstring": "# Numpy and TensorFlow compatible version of PyTorch RandomResizedCrop. Code adapted from:\n# https://pytorch.org/vision/main/_modules/torchvision/transforms/transforms.html#RandomResizedCrop", "end_line": 217, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/image-classification/run_image_classification.py", "name": "random_crop", "start_line": 197, "type": "function", "chroma:document": "CODE:\n# https://pytorch.org/vision/main/_modules/torchvision/transforms/transforms.html#RandomResizedCrop\ndef random_crop(image, scale=(0.08, 1.0), ratio=(3.0 / 4.0, 4.0 / 3.0)):\n    height, width, _ = image.shape\n    area = height * width\n    log_ratio = np.log(ratio)\n    for _ in range(10):\n        target_area = np.random.uniform(*scale) * area\n        aspect_ratio = np.exp(np.random.uniform(*log_ratio))\n        w = int(round(np.sqrt(target_area * aspect_ratio)))\n        h = int(round(np.sqrt(target_area / aspect_ratio)))\n        if 0 < w <= width and 0 < h <= height:\n            i = np.random.randint(0, height - h + 1)\n            j = np.random.randint(0, width - w + 1)\n            return image[i : i + h, j : j + w, :]\n\n    # Fallback to central crop\n    in_ratio = float(width) / float(height)\n    w = width if in_ratio < min(ratio) else int(round(height * max(ratio)))\n    h = height if in_ratio > max(ratio) else int(round(width / min(ratio)))\n    i = (height - h) // 2\n    j = (width - w) // 2\n\n\nSUMMARY:\n**Concise Summary**\n\n```python\ndef random_crop(image, \n               scale=(0.08, 1.0), \n               ratio=(3.0 / 4.0, 4.0 / 3.0)):\n    \"\"\"\n    Returns a randomly resized crop of the input image.\n    \n    Args:\n        image: Input image tensor.\n        scale: Tuple of (min_scale, max_scale) for area scaling.\n        ratio: Tuple of (width_ratio, height_ratio) for aspect ratio.\n    \n    Returns:\n        A cropped image tensor.\n    \"\"\"\n```\n\n**Implementation Details**\n\n* The function generates a random crop by iterating 10 times through different scales and ratios to find a valid crop.\n* On the 11th iteration, if no valid crop is found, it falls back to a central crop using a median aspect ratio.\n\n**Dependencies**\n\n* `numpy` for numerical computations\n* No explicit dependencies on TensorFlow (TF) since this code snippet only defines the `random_crop` function.\n\n**Key Insights**\n\n* The function uses both area scaling and aspect ratio scaling for random cropping.\n* It employs a fallback to central crop in case no valid crop is found after 10 iterations."}


created_at: 2025-02-05 09:13:46
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/image-classification/run_image_classification.py:220-224

metadata:
{"docstring": "", "end_line": 224, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/image-classification/run_image_classification.py", "name": "random_resized_crop", "start_line": 220, "type": "function", "chroma:document": "CODE:\n\ndef random_resized_crop(image, size, scale=(0.08, 1.0), ratio=(3.0 / 4.0, 4.0 / 3.0)):\n    size = (size, size) if isinstance(size, int) else size\n    image = random_crop(image, scale, ratio)\n    image = tf.image.resize(image, size)\n\n\nSUMMARY:\n**Summary:**\n\nThis function (`random_resized_crop`) generates a randomized cropped image with a specified `size`, while maintaining its aspect ratio.\n\n**Key Details:**\n\n* Resizes an input `image` to a random size within the specified `scale` and `ratio`.\n* Utilizes TensorFlow's `tf.image.resize` to resize the image.\n\n**Implementation Parameters:**\n\n* `size`: The target size of the output image (int or tuple).\n* `scale`: A scaling factor for randomly resizing the image (tuple).\n* `ratio`: The aspect ratio constraint for the resized image (tuple).\n* `random_crop`: Utilized function from TensorFlow for cropping (not shown in this code snippet).\n\n**Notable Dependencies:**\n\n* TensorFlow (`tf`): For image processing and randomization utilities.\n* `random_crop`: A custom or external function implemented by the library."}


created_at: 2025-02-05 09:13:46
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/image-classification/run_image_classification.py:227-572

metadata:
{"docstring": "", "end_line": 572, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/image-classification/run_image_classification.py", "name": "main", "start_line": 227, "type": "function", "chroma:document": "CODE:\n\ndef main():\n    # See all possible arguments in src/transformers/training_args.py\n    # or by passing the --help flag to this script.\n    # We now keep distinct sets of args, for a cleaner separation of concerns.\n    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TFTrainingArguments))\n    if len(sys.argv) == 2 and sys.argv[1].endswith(\".json\"):\n        # If we pass only one argument to the script and it's the path to a json file,\n        # let's parse it to get our arguments.\n        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))\n    else:\n        model_args, data_args, training_args = parser.parse_args_into_dataclasses()\n\n    # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The\n    # information sent is the one passed as arguments along with your Python/TensorFlow versions.\n    send_example_telemetry(\"run_image_classification\", model_args, data_args, framework=\"tensorflow\")\n\n    # Checkpoints. Find the checkpoint the use when loading the model.\n    checkpoint = None\n    if os.path.isdir(training_args.output_dir) and training_args.do_train and not training_args.overwrite_output_dir:\n        checkpoint = get_last_checkpoint(training_args.output_dir)\n        if checkpoint is None and len(os.listdir(training_args.output_dir)) > 0:\n            raise ValueError(\n                f\"Output directory ({training_args.output_dir}) already exists and is not empty. \"\n                \"Use --overwrite_output_dir to overcome.\"\n            )\n        elif checkpoint is not None and training_args.resume_from_checkpoint is None:\n            logger.info(\n                f\"Checkpoint detected, resuming training at {checkpoint}. To avoid this behavior, change \"\n                \"the `--output_dir` or add `--overwrite_output_dir` to train from scratch.\"\n            )\n\n    # Setup logging\n    logging.basicConfig(\n        format=\"%(asctime)s - %(levelname)s - %(name)s - %(message)s\",\n        datefmt=\"%m/%d/%Y %H:%M:%S\",\n        handlers=[logging.StreamHandler(sys.stdout)],\n    )\n    log_level = training_args.get_process_log_level()\n    logger.setLevel(log_level)\n\n    # Set the verbosity to info of the Transformers logger (on main process only):\n    if is_main_process(training_args.local_rank):\n        transformers.utils.logging.set_verbosity_info()\n        transformers.utils.logging.enable_default_handler()\n        transformers.utils.logging.enable_explicit_format()\n    logger.info(f\"Training/evaluation parameters {training_args}\")\n\n    # region Dataset and labels\n    # Set seed before initializing model.\n    set_seed(training_args.seed)\n\n    # Initialize our dataset and prepare it for the 'image-classification' task.\n    if data_args.dataset_name is not None:\n        dataset = load_dataset(\n            data_args.dataset_name,\n            data_args.dataset_config_name,\n            cache_dir=model_args.cache_dir,\n            token=model_args.token,\n            trust_remote_code=model_args.trust_remote_code,\n        )\n    else:\n        data_files = {}\n        if data_args.train_dir is not None:\n            data_files[\"train\"] = os.path.join(data_args.train_dir, \"**\")\n        if data_args.validation_dir is not None:\n            data_files[\"validation\"] = os.path.join(data_args.validation_dir, \"**\")\n        dataset = load_dataset(\n            \"imagefolder\",\n            data_files=data_files,\n            cache_dir=model_args.cache_dir,\n        )\n    # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at\n    # https://huggingface.co/docs/datasets/loading_datasets.\n\n    # Prepare label mappings.\n    # We'll include these in the model's config to get human readable labels in the Inference API.\n    labels = dataset[\"train\"].features[\"labels\"].names\n    label2id, id2label = {}, {}\n    for i, label in enumerate(labels):\n        label2id[label] = str(i)\n        id2label[str(i)] = label\n\n    # Load model image processor and configuration\n    config = AutoConfig.from_pretrained(\n        model_args.config_name or model_args.model_name_or_path,\n        num_labels=len(labels),\n        label2id=label2id,\n        id2label=id2label,\n        finetuning_task=\"image-classification\",\n        cache_dir=model_args.cache_dir,\n        revision=model_args.model_revision,\n        token=model_args.token,\n        trust_remote_code=model_args.trust_remote_code,\n    )\n    image_processor = AutoImageProcessor.from_pretrained(\n        model_args.image_processor_name or model_args.model_name_or_path,\n        cache_dir=model_args.cache_dir,\n        revision=model_args.model_revision,\n        token=model_args.token,\n        trust_remote_code=model_args.trust_remote_code,\n    )\n\n    # If we don't have a validation split, split off a percentage of train as validation.\n    data_args.train_val_split = None if \"validation\" in dataset.keys() else data_args.train_val_split\n    if isinstance(data_args.train_val_split, float) and data_args.train_val_split > 0.0:\n        split = dataset[\"train\"].train_test_split(data_args.train_val_split)\n        dataset[\"train\"] = split[\"train\"]\n        dataset[\"validation\"] = split[\"test\"]\n\n    # Define our data preprocessing function. It takes an image file path as input and returns\n    # Write a note describing the resizing behaviour.\n    if \"shortest_edge\" in image_processor.size:\n        # We instead set the target size as (shortest_edge, shortest_edge) to here to ensure all images are batchable.\n        image_size = (image_processor.size[\"shortest_edge\"], image_processor.size[\"shortest_edge\"])\n    else:\n        image_size = (image_processor.size[\"height\"], image_processor.size[\"width\"])\n\n    def _train_transforms(image):\n        img_size = image_size\n        image = keras.utils.img_to_array(image)\n        image = random_resized_crop(image, size=img_size)\n        image = tf.image.random_flip_left_right(image)\n        image /= 255.0\n        image = (image - image_processor.image_mean) / image_processor.image_std\n        image = tf.transpose(image, perm=[2, 0, 1])\n        return image\n\n    def _val_transforms(image):\n        image = keras.utils.img_to_array(image)\n        image = tf.image.resize(image, size=image_size)\n        # image = np.array(image) # FIXME - use tf.image function\n        image = center_crop(image, size=image_size)\n        image /= 255.0\n        image = (image - image_processor.image_mean) / image_processor.image_std\n        image = tf.transpose(image, perm=[2, 0, 1])\n        return image\n\n    def train_transforms(example_batch):\n        \"\"\"Apply _train_transforms across a batch.\"\"\"\n        example_batch[\"pixel_values\"] = [\n            _train_transforms(pil_img.convert(\"RGB\")) for pil_img in example_batch[\"image\"]\n        ]\n        return example_batch\n\n    def val_transforms(example_batch):\n        \"\"\"Apply _val_transforms across a batch.\"\"\"\n        example_batch[\"pixel_values\"] = [_val_transforms(pil_img.convert(\"RGB\")) for pil_img in example_batch[\"image\"]]\n        return example_batch\n\n    train_dataset = None\n    if training_args.do_train:\n        if \"train\" not in dataset:\n            raise ValueError(\"--do_train requires a train dataset\")\n        train_dataset = dataset[\"train\"]\n        if data_args.max_train_samples is not None:\n            train_dataset = train_dataset.shuffle(seed=training_args.seed).select(range(data_args.max_train_samples))\n        train_dataset = train_dataset.map(\n            train_transforms,\n            batched=True,\n            num_proc=data_args.preprocessing_num_workers,\n            load_from_cache_file=not data_args.overwrite_cache,\n        )\n\n    eval_dataset = None\n    if training_args.do_eval:\n        if \"validation\" not in dataset:\n            raise ValueError(\"--do_eval requires a validation dataset\")\n        eval_dataset = dataset[\"validation\"]\n        if data_args.max_eval_samples is not None:\n            eval_dataset = eval_dataset.select(range(data_args.max_eval_samples))\n        # Set the validation transforms\n        eval_dataset = eval_dataset.map(\n            val_transforms,\n            batched=True,\n            num_proc=data_args.preprocessing_num_workers,\n            load_from_cache_file=not data_args.overwrite_cache,\n        )\n\n    predict_dataset = None\n    if training_args.do_predict:\n        if \"test\" not in dataset:\n            raise ValueError(\"--do_predict requires a test dataset\")\n        predict_dataset = dataset[\"test\"]\n        if data_args.max_predict_samples is not None:\n            predict_dataset = predict_dataset.select(range(data_args.max_predict_samples))\n        # Set the test transforms\n        predict_dataset = predict_dataset.map(\n            val_transforms,\n            batched=True,\n            num_proc=data_args.preprocessing_num_workers,\n            load_from_cache_file=not data_args.overwrite_cache,\n        )\n\n    collate_fn = DefaultDataCollator(return_tensors=\"np\")\n\n    # Load the accuracy metric from the datasets package\n    metric = evaluate.load(\"accuracy\", cache_dir=model_args.cache_dir)\n\n    # Define our compute_metrics function. It takes an `EvalPrediction` object (a namedtuple with a\n    # predictions and label_ids field) and has to return a dictionary string to float.\n    def compute_metrics(p):\n        \"\"\"Computes accuracy on a batch of predictions\"\"\"\n        logits, label_ids = p\n        predictions = np.argmax(logits, axis=-1)\n        metrics = metric.compute(predictions=predictions, references=label_ids)\n        return metrics\n\n    with training_args.strategy.scope():\n        if checkpoint is None:\n            model_path = model_args.model_name_or_path\n        else:\n            model_path = checkpoint\n\n        model = TFAutoModelForImageClassification.from_pretrained(\n            model_path,\n            config=config,\n            from_pt=bool(\".bin\" in model_path),\n            cache_dir=model_args.cache_dir,\n            revision=model_args.model_revision,\n            token=model_args.token,\n            trust_remote_code=model_args.trust_remote_code,\n            ignore_mismatched_sizes=model_args.ignore_mismatched_sizes,\n        )\n        num_replicas = training_args.strategy.num_replicas_in_sync\n        total_train_batch_size = training_args.per_device_train_batch_size * num_replicas\n        total_eval_batch_size = training_args.per_device_eval_batch_size * num_replicas\n\n        dataset_options = tf.data.Options()\n        dataset_options.experimental_distribute.auto_shard_policy = tf.data.experimental.AutoShardPolicy.OFF\n\n        if training_args.do_train:\n            num_train_steps = int(len(train_dataset) * training_args.num_train_epochs)\n            if training_args.warmup_steps > 0:\n                num_warmpup_steps = int(training_args.warmup_steps)\n            elif training_args.warmup_ratio > 0:\n                num_warmpup_steps = int(training_args.warmup_ratio * num_train_steps)\n            else:\n                num_warmpup_steps = 0\n\n            optimizer, _ = create_optimizer(\n                init_lr=training_args.learning_rate,\n                num_train_steps=num_train_steps,\n                num_warmup_steps=num_warmpup_steps,\n                adam_beta1=training_args.adam_beta1,\n                adam_beta2=training_args.adam_beta2,\n                adam_epsilon=training_args.adam_epsilon,\n                weight_decay_rate=training_args.weight_decay,\n                adam_global_clipnorm=training_args.max_grad_norm,\n            )\n            # model.prepare_tf_dataset() wraps a Hugging Face dataset in a tf.data.Dataset which is ready to use in\n            # training. This is the recommended way to use a Hugging Face dataset when training with Keras. You can also\n            # use the lower-level dataset.to_tf_dataset() method, but you will have to specify things like column names\n            # yourself if you use this method, whereas they are automatically inferred from the model input names when\n            # using model.prepare_tf_dataset()\n            # For more info see the docs:\n            # https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.TFPreTrainedModel.prepare_tf_dataset\n            # https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset.to_tf_dataset\n            train_dataset = model.prepare_tf_dataset(\n                train_dataset,\n                shuffle=True,\n                batch_size=total_train_batch_size,\n                collate_fn=collate_fn,\n            ).with_options(dataset_options)\n        else:\n            optimizer = \"sgd\"  # Just write anything because we won't be using it\n\n        if training_args.do_eval:\n            eval_dataset = model.prepare_tf_dataset(\n                eval_dataset,\n                shuffle=False,\n                batch_size=total_eval_batch_size,\n                collate_fn=collate_fn,\n            ).with_options(dataset_options)\n\n        if training_args.do_predict:\n            predict_dataset = model.prepare_tf_dataset(\n                predict_dataset,\n                shuffle=False,\n                batch_size=total_eval_batch_size,\n                collate_fn=collate_fn,\n            ).with_options(dataset_options)\n\n        # Transformers models compute the right loss for their task by default when labels are passed, and will\n        # use this for training unless you specify your own loss function in compile().\n        model.compile(optimizer=optimizer, jit_compile=training_args.xla, metrics=[\"accuracy\"])\n\n        push_to_hub_model_id = training_args.push_to_hub_model_id\n        if not push_to_hub_model_id:\n            model_name = model_args.model_name_or_path.split(\"/\")[-1]\n            push_to_hub_model_id = f\"{model_name}-finetuned-image-classification\"\n\n        model_card_kwargs = {\n            \"finetuned_from\": model_args.model_name_or_path,\n            \"tasks\": \"image-classification\",\n            \"dataset\": data_args.dataset_name,\n            \"tags\": [\"image-classification\", \"tensorflow\", \"vision\"],\n        }\n\n        callbacks = []\n        if eval_dataset is not None:\n            callbacks.append(KerasMetricCallback(metric_fn=compute_metrics, eval_dataset=eval_dataset))\n        if training_args.push_to_hub:\n            callbacks.append(\n                PushToHubCallback(\n                    output_dir=training_args.output_dir,\n                    hub_model_id=push_to_hub_model_id,\n                    hub_token=training_args.push_to_hub_token,\n                    tokenizer=image_processor,\n                    **model_card_kwargs,\n                )\n            )\n\n        if training_args.do_train:\n            model.fit(\n                train_dataset,\n                validation_data=eval_dataset,\n                epochs=int(training_args.num_train_epochs),\n                callbacks=callbacks,\n            )\n\n        if training_args.do_eval:\n            n_eval_batches = len(eval_dataset)\n            eval_predictions = model.predict(eval_dataset, steps=n_eval_batches)\n            eval_labels = dataset[\"validation\"][\"labels\"][: n_eval_batches * total_eval_batch_size]\n            eval_metrics = compute_metrics((eval_predictions.logits, eval_labels))\n            logging.info(\"Eval metrics:\")\n            for metric_name, value in eval_metrics.items():\n                logging.info(f\"{metric_name}: {value:.3f}\")\n\n        if training_args.output_dir is not None:\n            os.makedirs(training_args.output_dir, exist_ok=True)\n            with open(os.path.join(training_args.output_dir, \"all_results.json\"), \"w\") as f:\n                f.write(json.dumps(eval_metrics))\n\n        if training_args.do_predict:\n            n_predict_batches = len(predict_dataset)\n            test_predictions = model.predict(predict_dataset, steps=n_predict_batches)\n            test_labels = dataset[\"validation\"][\"labels\"][: n_predict_batches * total_eval_batch_size]\n            test_metrics = compute_metrics((test_predictions.logits, test_labels))\n            logging.info(\"Test metrics:\")\n            for metric_name, value in test_metrics.items():\n                logging.info(f\"{metric_name}: {value:.3f}\")\n\n        if training_args.output_dir is not None and not training_args.push_to_hub:\n            # If we're not pushing to hub, at least save a local copy when we're done\n\n\nSUMMARY:\nThis is a Python script that trains a Hugging Face Transformers model on an image classification task. Here's a concise summary:\n\n1. The script sets up the training environment, including the model, optimizer, and hyperparameters.\n2. It prepares the datasets for training and evaluation using `model.prepare_tf_dataset()`.\n3. The model is compiled with the specified loss function, optimizer, and metrics (accuracy).\n4. The model is trained on the training dataset, evaluated on the validation dataset, and predicts on the test dataset.\n5. After training, the model's performance is logged, and results are saved to a JSON file if desired.\n\nKey features:\n\n* Uses Hugging Face Transformers for image classification\n* Supports optional evaluation, prediction, and saving modes\n* Allows for hyperparameter tuning and output directory customization\n\nNote that this script assumes you have a basic understanding of PyTorch and Keras, as well as the Hugging Face Transformers library."}


created_at: 2025-02-05 09:13:46
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/benchmarking/run_benchmark_tf.py:21-43

metadata:
{"docstring": "", "end_line": 43, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/benchmarking/run_benchmark_tf.py", "name": "main", "start_line": 21, "type": "function", "chroma:document": "CODE:\n\ndef main():\n    parser = HfArgumentParser(TensorFlowBenchmarkArguments)\n    benchmark_args = parser.parse_args_into_dataclasses()[0]\n    benchmark = TensorFlowBenchmark(args=benchmark_args)\n    try:\n        benchmark_args = parser.parse_args_into_dataclasses()[0]\n    except ValueError as e:\n        arg_error_msg = \"Arg --no_{0} is no longer used, please use --no-{0} instead.\"\n        begin_error_msg = \" \".join(str(e).split(\" \")[:-1])\n        full_error_msg = \"\"\n        depreciated_args = eval(str(e).split(\" \")[-1])\n        wrong_args = []\n        for arg in depreciated_args:\n            # arg[2:] removes '--'\n            if arg[2:] in TensorFlowBenchmark.deprecated_args:\n                # arg[5:] removes '--no_'\n                full_error_msg += arg_error_msg.format(arg[5:])\n            else:\n                wrong_args.append(arg)\n        if len(wrong_args) > 0:\n            full_error_msg = full_error_msg + begin_error_msg + str(wrong_args)\n        raise ValueError(full_error_msg)\n\n\nSUMMARY:\n**Function Summary**\n\n*   **Purpose**: The main function `main()` parses command-line arguments, creates a TensorFlowBenchmark instance, and runs the benchmark.\n*   **Functionality**: It utilizes an `HfArgumentParser` to parse arguments into dataclasses, handles deprecated argument warnings, and executes the benchmark using the `TensorFlowBenchmark` class.\n\n**Key Implementation Details**\n\n*   Uses an `HfArgumentParser` with `TensorFlowBenchmarkArguments` to parse command-line arguments.\n*   Handles deprecated arguments raised by Python's `eval()` function on a string of deprecated options.\n*   Includes error handling for invalid argument values, raising a `ValueError` with a formatted error message.\n\n**Notable Dependencies and Requirements**\n\n*   `HfArgumentParser`: A class from the Hugging Face Transformers library used to parse command-line arguments into dataclasses.\n*   `TensorFlowBenchmarkArguments`: A custom class or interface extended from `arguments.BaseArguments`.\n*   `TensorFlowBenchmark`: The benchmarking class executed after parsing arguments.\n*   `python's eval() function`: Used for parsing deprecated argument names.\n\n**Notes**: This code assumes the availability of specific classes and interfaces (`HfArgumentParser`, `TensorFlowBenchmarkArguments`, `TensorflowBenchmark`) in the project, which may require external libraries or custom definitions."}


created_at: 2025-02-05 09:13:46
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/benchmarking/plot_csv_file.py:26-27

metadata:
{"docstring": "", "end_line": 27, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/benchmarking/plot_csv_file.py", "name": "list_field", "start_line": 26, "type": "function", "chroma:document": "CODE:\n\ndef list_field(default=None, metadata=None):\n\n\nSUMMARY:\n**Summary**\n\nConcats `field` function with factory method creating an instance of the same type as `default`, injecting `metadata`. This function acts as a shortcut to create and configure custom fields.\n\n**Key Implementation Details:**\n\n- Utilizes `field` function provided by `django-felix` library.\n- Employs `factory_method` to dynamically create an instance based on the `default` value.\n- Passes `metadata` directly to the parent `field()` method for configuration.\n\n**Notable Dependencies/Requirements:**\n\n- Requires `django-felix` library, referencing `Field`."}


created_at: 2025-02-05 09:13:46
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/benchmarking/plot_csv_file.py:66-71

metadata:
{"docstring": "", "end_line": 71, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/benchmarking/plot_csv_file.py", "name": "can_convert_to_int", "start_line": 66, "type": "function", "chroma:document": "CODE:\n\ndef can_convert_to_int(string):\n    try:\n        int(string)\n        return True\n    except ValueError:\n\n\nSUMMARY:\n**Summary:**\n\nThe `can_convert_to_int` function takes a string as input and attempts to convert it to an integer using the built-in `int()` function. If successful, it returns `True`. Otherwise, it catches the `ValueError` exception and returns `False`.\n\n* Main purpose: Determine if a given string can be converted to an integer.\n* Key implementation detail: Conversion with try-except block for handling potential `ValueError`.\n* Notable dependency/interface: Built-in Python data type `int()`.\n\n**RAG Code Summary:** \n\n`can_convert_to_int(str string) : bool`\n\n`def can_convert_to_int(string):`\n  `try: int(string)`\n`except ValueError: return False`\n`return True`"}


created_at: 2025-02-05 09:13:46
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/benchmarking/plot_csv_file.py:74-79

metadata:
{"docstring": "", "end_line": 79, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/benchmarking/plot_csv_file.py", "name": "can_convert_to_float", "start_line": 74, "type": "function", "chroma:document": "CODE:\n\ndef can_convert_to_float(string):\n    try:\n        float(string)\n        return True\n    except ValueError:\n\n\nSUMMARY:\n**Summary:**\n\n```markdown\n**Function:** `can_convert_to_float`\n**Purpose:** Checks if a given string can be converted to a Float.\n\n* Attempts to convert the input string to a float using `float()`.\n* Returns True (representing successful conversion) or False on failure.\n\n## Key Implementation Details\n\n* Uses `try-except` block to handle `ValueError` exceptions raised during floating-point conversion.\n\n## Notable Dependencies/Requirements**\n\n* None. This function does not import any external modules or classes.\n```\n\n**RAG (Required Action Guide) Tags:**\n\n* Primary Purpose: Convert input string to float\n* Key Implementation Details: Floating-point conversion attempt and `try-except` block handling\n* Notable Dependencies: None"}


created_at: 2025-02-05 09:13:46
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/benchmarking/plot_csv_file.py:82-166

metadata:
{"docstring": "", "end_line": 166, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/benchmarking/plot_csv_file.py", "name": "Plot", "start_line": 82, "type": "class", "chroma:document": "CODE:\n\nclass Plot:\n    def __init__(self, args):\n        self.args = args\n        self.result_dict = defaultdict(lambda: {\"bsz\": [], \"seq_len\": [], \"result\": {}})\n\n        with open(self.args.csv_file, newline=\"\") as csv_file:\n            reader = csv.DictReader(csv_file)\n            for row in reader:\n                model_name = row[\"model\"]\n                self.result_dict[model_name][\"bsz\"].append(int(row[\"batch_size\"]))\n                self.result_dict[model_name][\"seq_len\"].append(int(row[\"sequence_length\"]))\n                if can_convert_to_int(row[\"result\"]):\n                    # value is not None\n                    self.result_dict[model_name][\"result\"][(int(row[\"batch_size\"]), int(row[\"sequence_length\"]))] = (\n                        int(row[\"result\"])\n                    )\n                elif can_convert_to_float(row[\"result\"]):\n                    # value is not None\n                    self.result_dict[model_name][\"result\"][(int(row[\"batch_size\"]), int(row[\"sequence_length\"]))] = (\n                        float(row[\"result\"])\n                    )\n\n    def plot(self):\n        fig, ax = plt.subplots()\n        title_str = \"Time usage\" if self.args.is_time else \"Memory usage\"\n        title_str = title_str + \" for training\" if self.args.is_train else title_str + \" for inference\"\n\n        if not self.args.no_log_scale:\n            # set logarithm scales\n            ax.set_xscale(\"log\")\n            ax.set_yscale(\"log\")\n\n        for axis in [ax.xaxis, ax.yaxis]:\n            axis.set_major_formatter(ScalarFormatter())\n\n        for model_name_idx, model_name in enumerate(self.result_dict.keys()):\n            batch_sizes = sorted(set(self.result_dict[model_name][\"bsz\"]))\n            sequence_lengths = sorted(set(self.result_dict[model_name][\"seq_len\"]))\n            results = self.result_dict[model_name][\"result\"]\n\n            (x_axis_array, inner_loop_array) = (\n                (batch_sizes, sequence_lengths) if self.args.plot_along_batch else (sequence_lengths, batch_sizes)\n            )\n\n            label_model_name = (\n                model_name if self.args.short_model_names is None else self.args.short_model_names[model_name_idx]\n            )\n\n            for inner_loop_value in inner_loop_array:\n                if self.args.plot_along_batch:\n                    y_axis_array = np.asarray(\n                        [results[(x, inner_loop_value)] for x in x_axis_array if (x, inner_loop_value) in results],\n                        dtype=int,\n                    )\n                else:\n                    y_axis_array = np.asarray(\n                        [results[(inner_loop_value, x)] for x in x_axis_array if (inner_loop_value, x) in results],\n                        dtype=np.float32,\n                    )\n\n                (x_axis_label, inner_loop_label) = (\n                    (\"batch_size\", \"len\") if self.args.plot_along_batch else (\"in #tokens\", \"bsz\")\n                )\n\n                x_axis_array = np.asarray(x_axis_array, int)[: len(y_axis_array)]\n                plt.scatter(\n                    x_axis_array, y_axis_array, label=f\"{label_model_name} - {inner_loop_label}: {inner_loop_value}\"\n                )\n                plt.plot(x_axis_array, y_axis_array, \"--\")\n\n            title_str += f\" {label_model_name} vs.\"\n\n        title_str = title_str[:-4]\n        y_axis_label = \"Time in s\" if self.args.is_time else \"Memory in MB\"\n\n        # plot\n        plt.title(title_str)\n        plt.xlabel(x_axis_label)\n        plt.ylabel(y_axis_label)\n        plt.legend()\n\n        if self.args.figure_png_file is not None:\n            plt.savefig(self.args.figure_png_file)\n        else:\n\n\nSUMMARY:\n**Summary:**\n\nThe `Plot` class is designed to visualize training and inference time(memory) usage patterns of different models. It takes a CSV file containing model information as input.\n\n**Main Purpose and Functionality:**\n\n1. Load the specified CSV file, which contains model names and their corresponding training/inference statistics (e.g., batch size, sequence length, result values).\n2. Plot the result values for each model in a scatter plot, with x-axis representing one of the statistics (batch size or sequence length) and y-axis representing the other statistic and its result value.\n\n**Key Implementation Details:**\n\n1. The `plot` method creates two subplots if plotting along batch vs sequence length, and otherwise plots results as functions of one statistic.\n2. The scatter plot is generated using the seaborn `scatter` function, with line plots connecting each point for consistency.\n3. The plot includes labels, titles, and legends to explain the data points.\n\n**Notable Dependencies and Requirements:**\n\n1. The class uses the `defaultdict` from the `collections` module to efficiently group results by model name.\n2. It relies on the pandas library through CSV file reading (`csv.DictReader`).\n3. The plot generation utilizes the matplotlib library.\n4. Optional dependencies include seaborn for the scatter plot function and the plt.show() function for displaying plots, as well as PIL for saving PNG files.\n\n**Interfaces Implemented:**\n\n1. The `Plot` class adheres to the standard plotting APIs in matplotlib (e.g., `fig`, `ax`, `plt.scatter`, etc.) and supports additional options via the matplotlib rcParams (e.g., `'no_log_scale'`)"}


created_at: 2025-02-05 09:13:46
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
encoding: FLOAT32
id:
/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/benchmarking/plot_csv_file.py:169-173

metadata:
{"docstring": "", "end_line": 173, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/benchmarking/plot_csv_file.py", "name": "main", "start_line": 169, "type": "function", "chroma:document": "CODE:\n\ndef main():\n    parser = HfArgumentParser(PlotArguments)\n    plot_args = parser.parse_args_into_dataclasses()[0]\n    plot = Plot(args=plot_args)\n\n\nSUMMARY:\n**Summary:**\n\n`main()` function initializes and executes a plot generation pipeline. It uses an `HfArgumentParser` to parse command-line arguments, creates a `Plot` object with the parsed data, and calls its `plot()` method.\n\n**Key Details:**\n\n*   Main entry point for the application\n*   Uses `HfArgumentParser` to parse command-line arguments of type `PlotArguments`\n*   Creates and initializes a `Plot` object (`plot`) with a reference to the parsed argument data.\n*   Calls the `plot()` method on the `Plot` object.\n\n**Notable Dependencies/Requirements:**\n\n*   `HfArgumentParser`: A class for parsing command-line arguments\n*   `PlotArguments` : The type of arguments to be parsed by `HfArgumentParser`\n*   `Plot`: A class used for generating plots, with a method `plot()` that implements the actual plotting functionality."}


created_at: 2025-02-05 09:13:46
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_dummies.py:75-91
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 91, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_dummies.py", "name": "find_backend", "start_line": 75, "type": "function", "chroma:document": "CODE:\n\ndef find_backend(line: str) -> Optional[str]:\n    \"\"\"\n    Find one (or multiple) backend in a code line of the init.\n\n    Args:\n        line (`str`): A code line in an init file.\n\n    Returns:\n        Optional[`str`]: If one (or several) backend is found, returns it. In the case of multiple backends (the line\n        contains `if is_xxx_available() and `is_yyy_available()`) returns all backends joined on `_and_` (so\n        `xxx_and_yyy` for instance).\n    \"\"\"\n    if _re_test_backend.search(line) is None:\n        return None\n    backends = [b[0] for b in _re_backend.findall(line)]\n    backends.sort()\n\n\nSUMMARY:\n**Summary**\n\nThis function, `find_backend`, extracts and aggregates backend names (`b[0]`) from a given code line. It uses regular expressions to search for matching patterns and then returns the found backends in sorted order, optionally joined with `_and_`.\n\n**Key Implementation Details**\n\n* Uses `_re_test_backend` and `_re_backend` regular expression objects to search and find backend names.\n* Returns `None` if no match is found.\n\n**Dependencies**\n\n* `_re_test_backend`, `_re_backend`\n* `Optional[str]`\n\n**Return Value**\n\n* A string containing one or more backend names, separated by `_and_`. \n\n**Primary Functionality**\nFind backend names from a given code line in an init file."}


created_at: 2025-02-05 09:13:46
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_dummies.py:94-136
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 136, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_dummies.py", "name": "read_init", "start_line": 94, "type": "function", "chroma:document": "CODE:\n\ndef read_init() -> Dict[str, List[str]]:\n    \"\"\"\n    Read the init and extract backend-specific objects.\n\n    Returns:\n        Dict[str, List[str]]: A dictionary mapping backend name to the list of object names requiring that backend.\n    \"\"\"\n    with open(os.path.join(PATH_TO_TRANSFORMERS, \"__init__.py\"), \"r\", encoding=\"utf-8\", newline=\"\\n\") as f:\n        lines = f.readlines()\n\n    # Get to the point we do the actual imports for type checking\n    line_index = 0\n    while not lines[line_index].startswith(\"if TYPE_CHECKING\"):\n        line_index += 1\n\n    backend_specific_objects = {}\n    # Go through the end of the file\n    while line_index < len(lines):\n        # If the line is an if is_backend_available, we grab all objects associated.\n        backend = find_backend(lines[line_index])\n        if backend is not None:\n            while not lines[line_index].startswith(\"    else:\"):\n                line_index += 1\n            line_index += 1\n\n            objects = []\n            # Until we unindent, add backend objects to the list\n            while len(lines[line_index]) <= 1 or lines[line_index].startswith(\" \" * 8):\n                line = lines[line_index]\n                single_line_import_search = _re_single_line_import.search(line)\n                if single_line_import_search is not None:\n                    # Single-line imports\n                    objects.extend(single_line_import_search.groups()[0].split(\", \"))\n                elif line.startswith(\" \" * 12):\n                    # Multiple-line imports (with 3 indent level)\n                    objects.append(line[12:-2])\n                line_index += 1\n\n            backend_specific_objects[backend] = objects\n        else:\n            line_index += 1\n\n\n\nSUMMARY:\n**Function Summary: `read_init`**\n\n* **Purpose and Functionality**: Extracts backend-specific objects from the `__init__.py` file of a Hugging Face transformer model, mapping each backend to its associated object names.\n* **Key Implementation Details**:\n\t+ Reads and parses the entire `__init__.py` file.\n\t+ Searches for `if is_backend_available` statements to identify backend-specific imports.\n\t+ Parses multiple lines using regular expressions to extract object names from import statements.\n* **Dependencies and Requirements**:\n\t+ Hugging Face transformer models (specifically, `PATH_TO_TRANSFORMERS`)\n\t+ `_re_single_line_import` regex pattern (internal function)\n\t+ `find_backend` function (imported but not explicitly shown in this code snippet)\n* **Interfaces/Inherited:** This function inherits from no specific interface.\n\n**Notes:**\n\n* The function returns a dictionary with the format `{backend_name: [object_names]}`.\n* The `_re_single_line_import` regex pattern is used to extract object names from single-line import statements."}


created_at: 2025-02-05 09:13:46
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_dummies.py:139-155
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 155, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_dummies.py", "name": "create_dummy_object", "start_line": 139, "type": "function", "chroma:document": "CODE:\n\ndef create_dummy_object(name: str, backend_name: str) -> str:\n    \"\"\"\n    Create the code for a dummy object.\n\n    Args:\n        name (`str`): The name of the object.\n        backend_name (`str`): The name of the backend required for that object.\n\n    Returns:\n        `str`: The code of the dummy object.\n    \"\"\"\n    if name.isupper():\n        return DUMMY_CONSTANT.format(name)\n    elif name.islower():\n        return DUMMY_FUNCTION.format(name, backend_name)\n    else:\n\n\nSUMMARY:\n**Function Summary**\n\n`create_dummy_object`: Returns code for a dummy object of specified type based on input `name`. The type is determined by the casing of `name`.\n\n* **Purpose**: Generate code for a dummy object with given name and backend requirement.\n* **Implementation Details**:\n\t+ Uses format strings (`DUMMY_CONSTANT`, `DUMMY_FUNCTION`, `DUMMY_CLASS`) to generate code.\n\t+ Includes `backend_name` in generated code when `name` is not all upper case or lower case.\n* **Notable Dependencies**: Uses string formatting (`format`) and likely inherited from a parent class/inheritance hierarchy containing `DUMMY_CONSTANT`, `DUMMY_FUNCTION`, and `DUMMY_CLASS`. \n\n**Key Interfaces/Methods:**\n\n- `DUMMY_CONSTANT`\n- `DUMMY FUNCTION`\n- `DUMMY CLASS`\n\n**External Dependencies:** None"}


created_at: 2025-02-05 09:13:46
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_dummies.py:158-182
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 182, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_dummies.py", "name": "create_dummy_files", "start_line": 158, "type": "function", "chroma:document": "CODE:\n\ndef create_dummy_files(backend_specific_objects: Optional[Dict[str, List[str]]] = None) -> Dict[str, str]:\n    \"\"\"\n    Create the content of the dummy files.\n\n    Args:\n        backend_specific_objects (`Dict[str, List[str]]`, *optional*):\n            The mapping backend name to list of backend-specific objects. If not passed, will be obtained by calling\n            `read_init()`.\n\n    Returns:\n        `Dict[str, str]`: A dictionary mapping backend name to code of the corresponding backend file.\n    \"\"\"\n    if backend_specific_objects is None:\n        backend_specific_objects = read_init()\n\n    dummy_files = {}\n\n    for backend, objects in backend_specific_objects.items():\n        backend_name = \"[\" + \", \".join(f'\"{b}\"' for b in backend.split(\"_and_\")) + \"]\"\n        dummy_file = \"# This file is autogenerated by the command `make fix-copies`, do not edit.\\n\"\n        dummy_file += \"from ..utils import DummyObject, requires_backends\\n\\n\"\n        dummy_file += \"\\n\".join([create_dummy_object(o, backend_name) for o in objects])\n        dummy_files[backend] = dummy_file\n\n\n\nSUMMARY:\n**Function Summary: create_dummy_files**\n\n**Purpose**: Creates the content of dummy files based on backend-specific objects.\n\n**Functionality**\n\n* Takes an optional dictionary `backend_specific_objects` mapping backend names to lists of backend-specific objects.\n* If none provided, reads them from `read_init()`.\n* Creates a dictionary `dummy_files` with backend names as keys and their corresponding code as values.\n* For each backend:\n\t+ Constructs the code by concatenating object imports and calls to `create_dummy_object` method.\n\t+ Prefers formatted output for list of backend objects.\n\n**Key Implementation Details**\n\n* Uses `read_init()` to retrieve backend-specific objects when not provided.\n* Employed string formatting to nicely display list of backend objects.\n\n**Notable Dependencies**\n\n* `read_init()`\n* `create_dummy_object()\n* `DummyObject` class from `..utils` module\n* External dependencies: requires `make` command and directory structure"}


created_at: 2025-02-05 09:13:47
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_dummies.py:185-228
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 228, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_dummies.py", "name": "check_dummies", "start_line": 185, "type": "function", "chroma:document": "CODE:\n\ndef check_dummies(overwrite: bool = False):\n    \"\"\"\n    Check if the dummy files are up to date and maybe `overwrite` with the right content.\n\n    Args:\n        overwrite (`bool`, *optional*, default to `False`):\n            Whether or not to overwrite the content of the dummy files. Will raise an error if they are not up to date\n            when `overwrite=False`.\n    \"\"\"\n    dummy_files = create_dummy_files()\n    # For special correspondence backend name to shortcut as used in utils/dummy_xxx_objects.py\n    short_names = {\"torch\": \"pt\"}\n\n    # Locate actual dummy modules and read their content.\n    path = os.path.join(PATH_TO_TRANSFORMERS, \"utils\")\n    dummy_file_paths = {\n        backend: os.path.join(path, f\"dummy_{short_names.get(backend, backend)}_objects.py\")\n        for backend in dummy_files.keys()\n    }\n\n    actual_dummies = {}\n    for backend, file_path in dummy_file_paths.items():\n        if os.path.isfile(file_path):\n            with open(file_path, \"r\", encoding=\"utf-8\", newline=\"\\n\") as f:\n                actual_dummies[backend] = f.read()\n        else:\n            actual_dummies[backend] = \"\"\n\n    # Compare actual with what they should be.\n    for backend in dummy_files.keys():\n        if dummy_files[backend] != actual_dummies[backend]:\n            if overwrite:\n                print(\n                    f\"Updating transformers.utils.dummy_{short_names.get(backend, backend)}_objects.py as the main \"\n                    \"__init__ has new objects.\"\n                )\n                with open(dummy_file_paths[backend], \"w\", encoding=\"utf-8\", newline=\"\\n\") as f:\n                    f.write(dummy_files[backend])\n            else:\n                raise ValueError(\n                    \"The main __init__ has objects that are not present in \"\n                    f\"transformers.utils.dummy_{short_names.get(backend, backend)}_objects.py. Run `make fix-copies` \"\n                    \"to fix this.\"\n\n\nSUMMARY:\n**Function Summary: check_dummies**\n\n**Purpose:** Verify the integrity of dummy files and optionally update their content.\n\n**Functionality:**\n\n1. Checks if dummy file paths exist and reads their contents.\n2. Compares actual contents with expected ones. If updated, updates the file.\n3. Raises `ValueError` if dummy files are outdated (non-present) when `overwrite=False`.\n\n**Key Implementation Details:**\n\n* Uses `create_dummy_files()` to generate dictionary of dummy files.\n* Uses a mapping (`short_names`) for shortcutting backend names.\n\n**Notable Dependencies and Requirements:**\n\n* `utils/dummy_xxx_objects.py` file\n* `os` module (for interacting with the file system)\n* `pathlib` (not explicitly mentioned, but implied by `PATH_TO_TRANSFORMERS`)\n* `make` command (to run `fix-copies` to update outdated dummy files)"}


created_at: 2025-02-05 09:13:47
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py:46-54
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 54, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py", "name": "get_module_source_from_name", "start_line": 46, "type": "function", "chroma:document": "CODE:\n\ndef get_module_source_from_name(module_name: str) -> str:\n    # Extract the source code from the module name\n    spec = importlib.util.find_spec(module_name)\n    if spec is None or spec.origin is None:\n        raise ValueError(f\"Cannot open file associated with {module_name} module.\")\n\n    with open(spec.origin, \"r\", encoding=\"utf-8\") as file:\n        source_code = file.read()\n\n\nSUMMARY:\n**Summary**\n\n`get_module_source_from_name`: Extracts(source code) from a given Python module name (`module_name`). \n\n### Key Details\n\n* Uses `importlib.util.find_spec` to locate the module spec.\n* Returns the read contents of the module's associated file.\n\n### Implementation Notes\n\n* Checks for existence and validity of the module spec before proceeding.\n* Raises a `ValueError` if the file associated with the module is not found or cannot be opened.\n* reads the source code from the file using UTF-8 encoding.\n\n### Dependencies and Interfaces\n\n* `importlib.util`: Module to locate Python modules.\n* `python files/utf-8 encoding (`encoding=\"utf-8\"`): Specifies character encoding for reading the file."}


created_at: 2025-02-05 09:13:47
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py:57-76
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 76, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py", "name": "preserve_case_replace", "start_line": 57, "type": "function", "chroma:document": "CODE:\n\ndef preserve_case_replace(text, patterns: dict, default_name: str):\n    # Create a regex pattern to match all variations\n    regex_pattern = \"|\".join(re.escape(key) for key in patterns.keys())\n    compiled_regex = re.compile(f\"(?<![a-z0-9])({regex_pattern})(.|$)\", re.IGNORECASE | re.DOTALL)\n\n    def replace(match):\n        matched_pattern = match.group(1)\n        next_char = match.group(2)\n        new_pattern = patterns.get(matched_pattern, default_name)\n\n        # In this case, the cased old model did not respect CamelCase and was all UPPERCASE, so we need to rely on next char\n        # The heuristic is: if next char is not a letter, then it is not part of a model name and result should be `new_name`.upper()\n        if len(patterns) == 2 and matched_pattern.isupper():\n            if not next_char.isalpha():\n                # `new_name.upper()` is just the other entry for `matched_pattern.lower()`, uppercased\n                new_pattern = patterns[matched_pattern.lower()].upper()\n\n        return new_pattern + next_char\n\n\n\nSUMMARY:\n**Function Summary: preserve_case_replace**\n\nPreserves sentence case in a given `text` by replacing matched patterns with corresponding names from the `patterns` dictionary. The function uses a regex pattern to match all variations of model names.\n\n**Key Functionality:**\n\n*   Preserves original text while replacing matched patterns.\n*   Replaces patterns based on the order and key-value pairs in `patterns`.\n*   Uses case-insensitive matching for non-alphanumeric characters.\n\n**Important Implementation Details:**\n\n*   Regex pattern `(?<![a-z0-9])({regex_pattern})(.|$)` matches all variations of model names while preserving sentence case.\n*   The `replace` function uses the matched pattern, next character, and corresponding name from `patterns` to replace the original text.\n\n**Notable Dependencies:**\n\n*   `re` module for regex operations.\n*   Python's built-in string methods (e.g., `isalpha()`, `upper()`).\n\n**External Dependencies:**\n\n*   None specified in the provided code chunk. However, it can be assumed that any library or framework being used by the target project may also require this function to function appropriately."}


created_at: 2025-02-05 09:13:47
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py:79-84
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 84, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py", "name": "get_cased_name", "start_line": 79, "type": "function", "chroma:document": "CODE:\n\ndef get_cased_name(lowercase_name: str) -> str:\n    \"\"\"From a model name in lowercase in the format `my_model`, return the cased name in the format `MyModel`.\"\"\"\n    if lowercase_name in CONFIG_MAPPING_NAMES:\n        return CONFIG_MAPPING_NAMES[lowercase_name].replace(\"Config\", \"\")\n    else:\n\n\nSUMMARY:\n**Summary:**\n\nThis function, `get_cased_name`, takes a lowercase model name as input and returns the cased version. If the input matches a predefined mapping, it replaces \"Config\" with an empty string; otherwise, it applies title case to the split and joined components.\n\n**Key Implementation Details:**\n\n* Replaces \"Config\" for matched mappings\n* Applies title case using `str.title()` for unmatched inputs\n\n**Dependencies/Interfaces:**\n\n* `CONFIG_MAPPING_NAMES`: a dictionary-like data structure (not defined in this code snippet)\n\n**Notable Method Signatures:**\n\n* `get_cased_name(lowercase_name: str) -> str`"}


created_at: 2025-02-05 09:13:47
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py:87-93
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 93, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py", "name": "get_lowercase_name", "start_line": 87, "type": "function", "chroma:document": "CODE:\n\ndef get_lowercase_name(cased_name: str) -> str:\n    \"\"\"From a model name in Camelcase in the format `MyModel`, return the lowercase name in the format `my_model`.\"\"\"\n    inverse_mapping = {value: key for key, value in CONFIG_MAPPING_NAMES.items()}\n    if cased_name + \"Config\" in inverse_mapping:\n        return inverse_mapping[cased_name + \"Config\"]\n    else:\n\n\nSUMMARY:\n**Function Summary:**\n\n`get_lowercase_name`\n\n* **Purpose:** Convert a Camelcase name to lowercase and format according to the ` CONFIG_MAPPING_NAMES`.\n* **Functionality:** \n\t+ Checks if a Config variant exists in the inverse mapping for the given Camelcase name.\n\t+ If not, returns the lowered Camelcase name by splitting into words and joining with underscores.\n\n**Key Implementation Details:**\n\n* Uses regular expressions (`re.findall`) to extract words from the Camelcase input string.\n* Applies `str.lower()` to each word before rejoining them with underscores.\n\n**Notable Dependencies or Requirements:**\n\n* `CONFIG_MAPPING_NAMES` (injected as a constant)\n* `re` (regular expression module)"}


created_at: 2025-02-05 09:13:47
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py:96-146
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 146, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py", "name": "ReplaceNameTransformer", "start_line": 96, "type": "class", "chroma:document": "CODE:\n\nclass ReplaceNameTransformer(m.MatcherDecoratableTransformer):\n    \"\"\"A transformer that replaces `old_name` with `new_name` in comments, string and any references.\n    It should take into account name like `MyNewModel`, or `my_new_model`. Without using the AUTO_MAPPING.\n    Supported renaming patterns:\n        - llama -> my_new_model     and     my_new_model    -> llama\n        - Llama -> MyNewModel       and     MyNewModel      -> Llama\n        - LLAMA -> MY_NEW_MODEL     and     MY_NEW_MODEL    -> LLAMA\n        - LLaMa -> MyNewModel       abd     MyNewModel      -> Llama\n    \"\"\"\n\n    def __init__(self, old_name: str, new_name: str, original_new_model_name: str = \"\", only_doc: bool = False):\n        super().__init__()\n        self.old_name = old_name\n        self.new_name = new_name\n        self.cased_new_name = get_cased_name(self.new_name)\n        self.cased_old_name = get_cased_name(self.old_name)\n        self.patterns = {\n            old_name: new_name,\n            old_name.upper(): new_name.upper(),\n            # For some old models, `self.cased_old_name` == `old_name.upper()` in which case this overwrite previous entry\n            self.cased_old_name: self.cased_new_name,\n        }\n        # In case new_name is a prefix alias, and not the original new model name\n        self.original_new_model_name = original_new_model_name\n        self.only_doc = only_doc\n\n    def _replace_name(self, original_node, updated_node):\n        if re.findall(r\"# Copied from\", updated_node.value):\n            return cst.RemoveFromParent()\n        update = preserve_case_replace(updated_node.value, self.patterns, self.cased_new_name)\n        return updated_node.with_changes(value=update)\n\n    @m.leave(m.SimpleString() | m.Comment())\n    def replace_name(self, original_node, updated_node):\n        return self._replace_name(original_node, updated_node)\n\n    def leave_Name(self, original_node, updated_node):\n        if not self.only_doc:\n            return self._replace_name(original_node, updated_node)\n        return updated_node\n\n    def leave_ImportFrom(self, original_node, updated_node):\n        \"\"\"The imports from other file types (configuration, processing etc) should use original model name.\"\"\"\n        if self.original_new_model_name != self.new_name and m.matches(updated_node.module, m.Name()):\n            patterns = \"|\".join(ALL_FILE_TYPES)\n            regex = rf\"({patterns})_{self.new_name}\"\n            new_source = re.sub(\n                regex, lambda m: f\"{m.group(1)}_{self.original_new_model_name}\", updated_node.module.value\n            )\n            updated_node = updated_node.with_changes(module=updated_node.module.with_changes(value=new_source))\n\n\nSUMMARY:\n**Summary**\n\nThe `ReplaceNameTransformer` class is designed to replace specific names in comments, strings, and references. It handles various naming patterns, considering case sensitivity and prefixes/qualifiers.\n\n**Key Functionality:**\n\n1.  Replaces `old_name` with `new_name` in:\n    *   Comments\n    *   Strings\n    *   References\n\n2.  Supports different renaming patterns:\n\n    *   `llama -> my_new_model`\n    *   `Llama -> MyNewModel`\n    *   `LLAMA -> MY_NEW_MODEL`\n    *   `(prefix)_LLaMa_(suffix)`\n\n**Implementation Details:**\n\n*   `_replace_name` method applies the replacement rules to an updated node.\n\n*   The transformer considers the case of the original name and uses a `patterns` dictionary for replacements.\n*   It preserves the original meaning by allowing certain suffixes (like `.py`) to remain unchanged.\n\n**Dependencies/Notable Requirements:**\n\n*   Uses Python's `re` module for regular expressions\n*   Leverages the `cst` library for parsing nodes\n\nHere's a concise summary of key classes and functions:\n\n*   `class ReplaceNameTransformer(m.MatcherDecoratableTransformer):`\n\n    *   Initializes with `old_name`, `new_name`, optional `original_new_model_name`, and `only_doc`.\n    *   Provides `replace_name`, `leave_Name` and `leave_ImportFrom` methods to apply the transformer transformations\n\nThe full implementation showcases how this transformer can be used to standardize naming conventions within a larger system."}


created_at: 2025-02-05 09:13:47
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py:161-162
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 162, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py", "name": "SUPER_CALL_NODE", "start_line": 161, "type": "function", "chroma:document": "CODE:\n\ndef SUPER_CALL_NODE(func_name):\n\n\nSUMMARY:\n**Summary:**\n\nThe `SUPER_CALL_NODE` function generates an abstract syntax tree (AST) node for the `super()` call in Python.\n\n**Key Implementation Details:**\n\n* The `m.Call` node represents the call operation with its attributes.\n* The `attr` attribute within the `Call` node specifies the `func_name` parameter, which is passed from the caller function.\n* A nested `m.Call` node represents the built-in `super()` function and has a specific structure, calling another inner `m.Name` node named `\"super\"`.\n* Both nodes rely on another higher-level node `m.Attribute`, indicating method calls.\n\n**Notable Dependencies/Requirements:**\n\n* The function uses the `ast` module (implied but not shown) to create an abstract syntax tree.\n* It relies on a meta-programming system, possibly a Python parser generator like Parsec or PyParsing."}


created_at: 2025-02-05 09:13:47
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py:165-168
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 168, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py", "name": "is_call_to_super", "start_line": 165, "type": "function", "chroma:document": "CODE:\n\ndef is_call_to_super(node, func_name):\n    return m.matches(\n        node, m.SimpleStatementLine(body=[m.Return(SUPER_CALL_NODE(func_name)) | m.Expr(SUPER_CALL_NODE(func_name))])\n\n\nSUMMARY:\n**Summary:**\n\n`is_call_to_super` function checks if the given `node` represents a call to the superclass (__Super__) from a method named `func_name`. \n\n**Key Implementation Details:**\n- Uses regular expression (`m.matches`) to match a specific pattern in the node.\n- Expects a match with either a return statement calling `SUPER_CALL_NODE(func_name)` or an expression also calling `SUPER_CALL_NODE(func_name)`.\n\n**Notable Dependencies/Requirements:**\n\n- `SuperCallNode` is assumed to be a previously defined class or function.\n- External dependency on regular expression engine (`m`) using `m.matches`. \n\nNote: Without additional context, the exact nature of `SUPER_CALL_NODE(func_name)`, `m`, and related classes/functions cannot be precisely determined."}


created_at: 2025-02-05 09:13:47
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py:171-189
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 189, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py", "name": "get_full_attribute_name", "start_line": 171, "type": "function", "chroma:document": "CODE:\n\ndef get_full_attribute_name(node: cst.Attribute | cst.Name) -> str | None:\n    \"\"\"Get the full name of an Attribute or Name node (e.g. `\"nn.Module\"` for an Attribute representing it). If the\n    successive value of an Attribute are not Name nodes, return `None`.\"\"\"\n    if m.matches(node, m.Name()):\n        return node.value\n    elif m.matches(node, m.Attribute()):\n        if not m.matches(node.attr, m.Name()):\n            return None\n        name = node.attr.value\n        new_node = node.value\n        while m.matches(new_node, m.Attribute()):\n            if not m.matches(new_node.attr, m.Name()):\n                return None\n            name = new_node.attr.value + \".\" + name\n            new_node = new_node.value\n        if not m.matches(new_node, m.Name()):\n            return None\n        return new_node.value + \".\" + name\n\n\nSUMMARY:\n**Summary**\n\n`get_full_attribute_name` function extracts the full attribute name from an AST node (either `Attribute` or `Name`). It traverses attributes and checks if successive values are `Name` nodes. If successful, it returns the fully qualified name.\n\n**Key Details**\n\n* Function takes an `AST` node (`cst.Attribute | cst.Name`) as input\n* Returns a string representing the full attribute name or `None` if incomplete\n* Uses a regular expression (m) to match node types\n\n**Implementation Choices**\n\n* Uses recursive descent approach for traversing attributes\n* Employed a try-and-contract approach to handle cases with incomplete successive values"}


created_at: 2025-02-05 09:13:47
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py:193-243
encoding: FLOAT32
metadata:
{"docstring": "# Transformer class to replace ClassB.call_to_method and ClassB().call_to_method with super().call_to_method", "end_line": 243, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py", "name": "ReplaceMethodCallTransformer", "start_line": 193, "type": "class", "chroma:document": "CODE:\n# Transformer class to replace ClassB.call_to_method and ClassB().call_to_method with super().call_to_method\nclass ReplaceMethodCallTransformer(cst.CSTTransformer):\n    def __init__(self, all_bases: Set[str]):\n        self.all_bases = all_bases\n\n    def leave_Attribute(self, original_node: cst.Attribute, updated_node: cst.Attribute) -> cst.CSTNode:\n        # Handle ClassB.call_to_method or module.classB.call_to_method\n        if (\n            m.matches(original_node.value, m.Name() | m.Attribute())\n            and get_full_attribute_name(original_node.value) in self.all_bases\n            and m.matches(original_node.attr, m.Name())\n        ):\n            # Replace with super().call_to_method\n            return updated_node.with_changes(\n                value=cst.Call(cst.Name(\"super\")),\n            )\n        # Handle ClassB().call_to_method or module.ClassB().call_to_method\n        elif (\n            m.matches(original_node.value, m.Call())\n            and m.matches(original_node.value.func, m.Name() | m.Attribute())\n            and get_full_attribute_name(original_node.value.func) in self.all_bases\n            and m.matches(original_node.attr, m.Name())\n        ):\n            # Replace with super().call_to_method\n            return updated_node.with_changes(value=cst.Call(cst.Name(\"super\")))\n        return updated_node\n\n    def leave_Call(self, original_node: cst.Call, updated_node: cst.Call) -> cst.CSTNode:\n        # Check if the function being called is of the form ClassB().func_a or ClassB.func_a\n        if m.matches(original_node.func, m.Attribute()) and (\n            # Match ClassB().func_a(...) or module\n            (\n                m.matches(original_node.func.value, m.Call())\n                and m.matches(original_node.func.value.func, m.Name() | m.Attribute())\n                and get_full_attribute_name(original_node.func.value.func) in self.all_bases\n            )\n            or\n            # Match ClassB.func_a(...)\n            (\n                m.matches(original_node.func.value, m.Name() | m.Attribute())\n                and get_full_attribute_name(original_node.func.value) in self.all_bases\n            )\n        ):\n            # Check if the first argument is 'self', and remove it\n            if len(original_node.args) > 0 and m.matches(original_node.args[0].value, m.Name(\"self\")):\n                # Create the new argument list without 'self'\n                new_args = updated_node.args[1:]\n            else:\n                new_args = updated_node.args\n\n            return updated_node.with_changes(args=new_args)\n\n\nSUMMARY:\n**Class Summary**\n\n`ReplaceMethodCallTransformer` is a CST transformer that replaces `ClassB.call_to_method` and `ClassB().call_to_method` with `super().call_to_method`.\n\n**Primary Purpose**\n\nThe main purpose of this class is to transform Python code by replacing method calls in the specified formats with the equivalent Super class call.\n\n**Key Implementation Details**\n\n1. **leave_Attribute**: This function handles case where attribute name is present as a string (`ClassB.call_to_method`). It checks if the attribute name is in `all_bases` and replaces it with `super().call_to_method`.\n2. **leave_Call**: This function handles cases where the method call contains an instance of `(module)` or only the class method call (e.g. `ClassB.func_a`). It checks for the presence of 'self' as an argument, removes it if present and updates the arguments of the function call.\n\n**Notable Dependencies/Requirements**\n\n* `cst` module for working with Python abstract syntax trees (ASTs)\n* `Set[str]` for storing class names\n* `get_full_attribute_name` and `m.matches` functions appear to be custom utility functions not explicitly included in the provided code."}


created_at: 2025-02-05 09:13:47
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py:246-252
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 252, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py", "name": "get_docstring_indent", "start_line": 246, "type": "function", "chroma:document": "CODE:\n\ndef get_docstring_indent(docstring):\n    # Match the first line after the opening triple quotes\n    match = re.search(r'(?:\"\"\"|\\'\\'\\'|```)\\n(\\s+)', docstring)\n    if match:\n        # Return the indentation spaces captured\n        return len(match.group(1))\n\n\nSUMMARY:\n**Function Summary**\n\n```\nget_docstring_indent function\n=============================\n\nReturns the number of indentation spaces preceding the first line after triple quotes in a docstring.\n\n### Purpose\n\nExtracts and returns the indentation level from the matched opening docstring triplet.\n\n### Implementation Details\n\n- Uses regular expression to match the opening triplet and capture the following whitespace content.\n- Returns the length of the captured whitespace as the indentation distance.\n\n### Dependency\n\nNone\n```"}


created_at: 2025-02-05 09:13:47
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py:255-281
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 281, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py", "name": "merge_docstrings", "start_line": 255, "type": "function", "chroma:document": "CODE:\n\ndef merge_docstrings(original_docstring, updated_docstring):\n    # indent_level = get_docstring_indent(updated_docstring)\n    original_level = get_docstring_indent(original_docstring)\n    if not re.findall(r\"\\n\\s*Args:\\n\", updated_docstring):\n        # Split the docstring at the example section, assuming `\"\"\"` is used to define the docstring\n        parts = original_docstring.split(\"```\")\n        if \"```\" in updated_docstring and len(parts) > 1:\n            updated_docstring = updated_docstring.lstrip('r\"')\n            new_parts = updated_docstring.split(\"```\")\n            if len(new_parts) != 3:\n                raise ValueError(\"There should only be one example, and it should have opening and closing '```'\")\n            parts[1] = new_parts[1]\n            updated_docstring = \"\".join(\n                [\n                    parts[0].rstrip(\" \\n\") + new_parts[0],\n                    f\"\\n{original_level*' '}```\",\n                    parts[1],\n                    \"```\",\n                    parts[2],\n                ]\n            )\n        elif updated_docstring not in original_docstring:\n            # add tabulation if we are at the lowest level.\n            if re.search(r\"\\n\\s*.*\\(.*\\)\\:\\n\\s*\\w\", updated_docstring):\n                updated_docstring = updated_docstring.replace(\"\\n    \", \"\\n        \")\n            updated_docstring = original_docstring.rstrip('\"') + \"\\n\" + updated_docstring.lstrip('r\"\\n')\n\n\nSUMMARY:\n**Summary**\n\nThe `merge_docstrings` function merges two docstrings, aiming to blend the style of both into a single cohesive doc string. Specifically:\n\n### Main Purpose and Functionality\n\n* Merges two doc strings (`original_docstring` and `updated_docstring`) into one.\n* Preserves indentation structure from `original_docstring`.\n* Updates `Args:` section with new content from `updated_docstring`.\n\n### Key Implementation Details\n\n* Uses regular expression to detect and replace `Args:` format in `updated_docstring`.\n* Corrects inconsistent indentation by tabulation.\n* Handles different cases of doc string formatting, including single-step and multi-step examples.\n\n### Notable Dependencies or Requirements\n\n* Utilizes string manipulation functions (e.g., `rstrip`, `lstrip`, `split`) and regular expressions (e.g., `re.findall`, `re.search`).\n* Assumes the use of triple quoting (`\"\"\"`) for defining docstrings.\n* Expected input: Two doc strings, with an optional third example section using triple quotes."}


created_at: 2025-02-05 09:13:47
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py:284-427
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 427, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py", "name": "SuperTransformer", "start_line": 284, "type": "class", "chroma:document": "CODE:\n\nclass SuperTransformer(cst.CSTTransformer):\n    METADATA_DEPENDENCIES = (ParentNodeProvider,)\n\n    def __init__(self, python_module: cst.Module, original_methods, updated_methods, all_bases=None):\n        self.python_module = python_module\n        self.original_methods = original_methods\n        self.updated_methods = updated_methods\n        self.all_assign_target = {}\n        self.deleted_targets = {}  # child node can delete some arguments\n        self.all_bases = all_bases or []\n        self.transformer = ReplaceMethodCallTransformer(set(self.all_bases))\n\n    def update_body(self, existing_body, new_statements):\n        \"\"\"\n        Helper method to update the body by removing duplicates before adding new statements.\n        `existing_body` is the body of the original method, the parent class\n        `new_statements` are the additional statements\n        \"\"\"\n        deduplicated_new_body = []\n        existing_nodes = set()\n        for node in new_statements:\n            if m.matches(node, m.SimpleStatementLine(body=[m.Assign()])):\n                target = self.python_module.code_for_node(node.body[0].targets[0].target)\n                self.all_assign_target[target] = node\n            if m.matches(node, m.SimpleStatementLine(body=[m.Del()])):\n                target = self.python_module.code_for_node(node.body[0].target)\n                self.deleted_targets[target] = node\n\n        for stmt in existing_body:\n            if m.matches(stmt, m.SimpleStatementLine(body=[m.Assign()])):\n                target = self.python_module.code_for_node(stmt.body[0].targets[0].target)\n                if target in self.deleted_targets:\n                    continue\n                if target in self.all_assign_target:\n                    stmt = self.all_assign_target[target]\n            # Skip the docstring (will be added later on, at the beginning)\n            elif m.matches(stmt, DOCSTRING_NODE):\n                continue\n            comment_less_code = re.sub(r\"#.*\", \"\", self.python_module.code_for_node(stmt)).strip()\n            comment_less_code = re.sub(r\"\\ *\\n\", \"\\n\", comment_less_code).strip()\n            deduplicated_new_body.append(stmt)\n            existing_nodes.add(comment_less_code)\n\n        for node in new_statements:\n            code = self.python_module.code_for_node(node)\n            comment_less_code = re.sub(r\"#.*\", \"\", code).strip()\n            comment_less_code = re.sub(r\"\\ *\\n\", \"\\n\", comment_less_code).strip()\n            if node not in deduplicated_new_body and comment_less_code not in existing_nodes:\n                if not m.matches(node, m.SimpleStatementLine(body=[m.Del()])):\n                    deduplicated_new_body.append(node)\n                    existing_nodes.add(comment_less_code)\n\n        deduplicated_new_body = self._fix_post_init_location(deduplicated_new_body)\n\n        return deduplicated_new_body\n\n    def _fix_post_init_location(self, new_body: list[cst.CSTNode]):\n        \"\"\"Fix the location of the `post_init()` in the new body, if we added statements after the call to\n        `super()` (it needs to be the very last statement called)\"\"\"\n        # Fix the post_init() that has to be last\n        for i, node in enumerate(new_body):\n            code = self.python_module.code_for_node(node)\n            comment_less_code = re.sub(r\"#.*\", \"\", code).strip()\n            comment_less_code = re.sub(r\"\\ *\\n\", \"\\n\", comment_less_code).strip()\n            if \"self.post_init(\" in comment_less_code and i < len(new_body) - 1:\n                # Remove it and add it again at the end\n                new_body.pop(i)\n                new_body.append(node)\n                break\n        return new_body\n\n    def _fix_init_location(self, new_body):\n        \"\"\"Fix the location of the `super().__init__()` in the new body, if we had new statements before it.\"\"\"\n        start_index = 0\n        for i, node in enumerate(new_body):\n            if m.matches(node, DOCSTRING_NODE) and i == start_index:\n                start_index += 1\n                continue\n            code = self.python_module.code_for_node(node)\n            comment_less_code = re.sub(r\"#.*\", \"\", code).strip()\n            comment_less_code = re.sub(r\"\\ *\\n\", \"\\n\", comment_less_code).strip()\n            if \"super().__init__\" in comment_less_code and i > start_index:\n                # Remove it and add it again at the top after the docstrings\n                node = new_body.pop(i)\n                new_body = new_body[:start_index] + [node] + new_body[start_index:]\n                break\n        return new_body\n\n    def replace_super_calls(self, node: cst.IndentedBlock, func_name: str) -> cst.CSTNode:\n        \"\"\"Updates the body of the input `node`'s `func_name` function by replacing calls\n        to super().func_name() with the source code of the parent class' `func_name`.\n        It keeps everything that is defined before `super().func_name()`.\n        \"\"\"\n        self.has_docstring = False\n        parent_has_docstring = False\n        if func_name in self.original_methods:\n            parent_has_docstring = m.matches(self.original_methods[func_name].body.body[0], DOCSTRING_NODE)\n        new_body = []\n        has_super_call = False\n\n        for i, expr in enumerate(node.body):\n            if is_call_to_super(expr, func_name):\n                has_super_call = True\n                new_body.extend(self.update_body(self.original_methods[func_name].body.body, node.body[i + 1 :]))\n                new_body = self._fix_init_location(new_body)\n            else:\n                expr = expr.visit(self.transformer)\n            if m.matches(expr, DOCSTRING_NODE):\n                self.has_docstring = True\n                if parent_has_docstring:  # actually here we ought to de-duplicate?\n                    original_docstring = self.original_methods[func_name].body.body[0].body[0].value.value\n                    updated_docstring = expr.body[0].value.value\n                    merged_doc = merge_docstrings(original_docstring, updated_docstring)\n                    new_node = [expr.with_changes(body=[cst.Expr(value=cst.SimpleString(value=merged_doc))])]\n                else:\n                    new_node = [expr]\n                new_body.extend(new_node)\n            elif not m.matches(expr, m.SimpleStatementLine(body=[m.Del()])) and not has_super_call:\n                new_body.append(expr)\n        if not self.has_docstring and parent_has_docstring:\n            new_body = [self.original_methods[func_name].body.body[0]] + new_body\n        return node.with_changes(body=new_body)\n\n    def leave_FunctionDef(self, original_node: cst.Call, updated_node: cst.Call) -> cst.CSTNode:\n        if updated_node.name.value in self.updated_methods:\n            name = updated_node.name.value\n            new_body = self.replace_super_calls(updated_node.body, name)\n            return updated_node.with_changes(body=new_body, params=updated_node.params)\n        return updated_node\n\n    def leave_Return(self, original_node: cst.Return, updated_node: cst.Return) -> cst.CSTNode:\n        \"\"\" \"When a return statement is reached, it is replaced with the unrolled super code\"\"\"\n        if m.matches(updated_node.value, m.Call(func=m.Attribute(attr=m.Name(\"super\")))):\n            func_def = self.get_metadata(ParentNodeProvider, original_node)\n            if m.matched(func_def, m.FunctionDef()) and func_def.name.value in self.original_methods:\n                updated_return_value = updated_node.value.with_changes(\n                    args=[\n                        cst.Arg(\n                            value=cst.Call(func=cst.Name(\"super\"), args=[cst.Arg(value=cst.Name(func_def.name.value))])\n                        )\n                    ]\n                )\n                return updated_node.with_changes(value=updated_return_value)\n\n\nSUMMARY:\n**Summary:**\n\nThis class `SuperTransformer` is designed to transform Python code that references `super()` calls. It achieves this by:\n\n*   **Replacing `super()` calls**: Replaces calls to `super()` with the source code of the parent class's method implementation.\n*   **De-duplicating docstrings**: Merges overlapping docstrings from different sources in a unified way.\n\n**Key Implementation Details:**\n\n1.  **Main Functionality**: `replace_super_calls` function\n    *   Updates the body of a function node by replacing calls to `super()` with the source code of the parent class's method implementation.\n2.  **Docstring Handling**\n    *   Uses separate functions `_fix_post_init_location` and `_fix_init_location` to handle docstrings in different locations (at the end, or at the top).\n3.  **Visitor Pattern**: Utilizes a visitor pattern (`visit`) to transform AST nodes.\n\n**Notable Dependencies:**\n\n1.  `ParentNodeProvider`\n2.  `m` (a module for parsing and manipulating code)\n3.  `DOCSTRING_NODE` (a node representing docstrings in the AST)"}


created_at: 2025-02-05 09:13:47
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py:430-514
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 514, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py", "name": "find_all_dependencies", "start_line": 430, "type": "function", "chroma:document": "CODE:\n\ndef find_all_dependencies(\n    dependency_mapping: Dict[str, set],\n    start_entity: str | None = None,\n    initial_dependencies: set | None = None,\n    initial_checked_dependencies: set | None = None,\n    return_parent: bool = False,\n) -> list | set:\n    \"\"\"Return all the dependencies of the given `start_entity` or `initial_dependencies`. This is basically some kind of\n    BFS traversal algorithm. It can either start from `start_entity`, or `initial_dependencies`.\n\n    Args:\n        dependency_mapping (`Dict[str, set]`):\n            A mapping from entities (usually function/assignment names), to immediate dependencies. That is, for function names,\n            a mapping {\"foo\": {\"bar\", \"test\"}} would indicate that functions `bar` and `test` are immediately called\n            in `foo`'s definition.\n        start_entity (str | None, *optional*):\n            A key of `dependency_mapping`, indicating from which entity to start the search.\n        initial_dependencies (set | None, *optional*):\n            If `start_entity` is not provided, this can be used as an alternative. In this case, the search will continue\n            from all the entities in `initial_dependencies`, if they are in `dependency_mapping`.\n        initial_checked_dependencies (set | None, *optional*):\n            If provided, entities already present in `initial_checked_dependencies` will not be part of the returned dependencies.\n        return_parent (bool, *optional*):\n            If `True`, will return a list consisting of tuples (dependency, parent) instead of a simple set of dependencies. Note\n            that the order of the items in the list reflects the traversal order. Thus, no parent can ever appear before childs.\n    Returns:\n        A set of all the dependencies, or a list of tuples `(dependency, parent)` if `return_parent=True`.\n\n    Example:\n    Given the following structure in the `modular_xxx.py` file:\n    ```\n    def foo1():\n        pass\n\n    def foo2():\n        pass\n\n    def bar():\n        foo1()\n\n    def foobar():\n        bar()\n        foo2()\n\n    class MyLayer(SomeOtherModelLayer):\n        def forward(...):\n            foobar()\n    ```\n    and the `dependency_mapping` created when visiting the `modular_xxx.py` file, we get:\n    ```\n    dependency_mapping = {'bar': {'foo1'}, 'foobar': {'bar', 'foo2'}}\n    find_all_dependencies(dependency_mapping, start_entity='foobar', return_parent=True)\n    >>> [('bar', 'foobar'), ('foo2', 'foobar'), ('foo1', 'bar')]\n    ```\n    That is, all the functions needed (and potentially their immediate parent) so that the function to be added\n    in MyLayer (`foobar`) can work correctly.\n    \"\"\"\n    if initial_dependencies is None and start_entity is not None:\n        initial_dependencies = dependency_mapping[start_entity]\n    if initial_checked_dependencies is None:\n        initial_checked_dependencies = set()\n\n    dependency_queue = deque(initial_dependencies)\n    all_dependencies = set()\n    all_dependencies_with_parent = []\n    checked_dependencies = set(initial_checked_dependencies)\n    parents = {initial_dep: start_entity for initial_dep in initial_dependencies}\n    while len(dependency_queue) > 0:\n        # Pick element to visit\n        current = dependency_queue.popleft()\n        if current not in checked_dependencies:\n            # Add the dependencies\n            all_dependencies.add(current)\n            all_dependencies_with_parent += [(current, parents[current])]\n            if current in dependency_mapping.keys():\n                # Update dependency queue\n                dependency_queue.extend(dependency_mapping[current])\n                parents.update({dep: current for dep in dependency_mapping[current]})\n            # add visited node to the list\n            checked_dependencies.add(current)\n\n    if not return_parent:\n        return all_dependencies\n    # no child can ever appear before its parent thanks to the queue (needed to add them at the correct location in the body later)\n\n\nSUMMARY:\nHere is a concise summary of the code:\n\n**Function Purpose and Functionality**\n\nThis function performs a breadth-first traversal (BFS) on a dependency graph, where each entity represents a function or assignment name. It finds all dependencies for a given `start_entity` or a set of initial dependencies.\n\n**Main Implementation Details**\n\n* The function uses a queue to efficiently traverse the graph.\n* It keeps track of visited entities and their dependencies using sets.\n* If `return_parent=True`, it returns a list of tuples (dependency, parent) instead of just a set of dependencies.\n\n**Key Method Signatures and Interfaces**\n\n* `find_all_dependencies(Dict[str, set], str | None = None, ..., return_parent: bool = False -> list | set)`\n* `deque()`\n* `set()`\n\n**Notable Dependencies or Requirements**\n\n* `dependency_mapping`: a dictionary mapping entities to their dependencies.\n* `start_entity`, `initial_dependencies`, and `return_parent` are optional parameters used to customize the traversal.\n\n**Example Use Case**\n\nGiven a `modular_xxx.py` file with function definitions, this function returns all dependencies needed for a given entity or initial set of dependencies."}


created_at: 2025-02-05 09:13:47
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py:523-545
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 545, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py", "name": "ClassDependencyMapper", "start_line": 523, "type": "class", "chroma:document": "CODE:\n\nclass ClassDependencyMapper(CSTVisitor):\n    \"\"\"A visitor which is designed to analyze a single class node to get all its dependencies that are shared with the set of\n    `global_names`.\n    \"\"\"\n\n    def __init__(\n        self, class_name: str, global_names: set[str], objects_imported_from_modeling: set[str] | None = None\n    ):\n        super().__init__()\n        self.class_name = class_name\n        self.dependencies = set()\n        self.global_names = global_names\n        self.objects_imported_from_modeling = (\n            set() if objects_imported_from_modeling is None else objects_imported_from_modeling\n        )\n\n    def visit_Name(self, node):\n        if (\n            node.value != self.class_name\n            and node.value in self.global_names\n            and node.value not in self.objects_imported_from_modeling\n        ):\n\n\nSUMMARY:\n**ClassDependencyMapper Summary**\n\nThe `ClassDependencyMapper` class is a custom visitor for analyzing class nodes to identify shared dependencies with `global_names`. It primarily maps shared node values to set of dependencies.\n\n**Key Implementation Details:**\n\n* Visits each name node in the class definition.\n* Checks if name value matches global names (excluding imported from modeling) and adds it to the dependencies set.\n\n**Notable Dependencies or Requirements:**\n\n* `global_names`: a set of global names used for dependency analysis.\n* `objects_imported_from_modeling`: an optional set of objects imported from modeling, defaulting to an empty set if not provided.\n* `_CSTVisitor`: the base visitor class inherited by this class.\n\nNote that external dependencies are not explicitly mentioned in this code snippet."}


created_at: 2025-02-05 09:13:47
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py:548-553
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 553, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py", "name": "dependencies_for_class_node", "start_line": 548, "type": "function", "chroma:document": "CODE:\n\ndef dependencies_for_class_node(node: cst.ClassDef, global_names: set[str]) -> set:\n    \"\"\"Create immediate dependencies for a class node based on the `global_names`.\"\"\"\n    temp_module = cst.Module(body=[node])\n    visitor = ClassDependencyMapper(node.name.value, global_names)\n    temp_module.visit(visitor)\n\n\nSUMMARY:\n**Summary:**\n\nCreates immediate dependencies for a class node based on `global_names`.\n\n**Main Purpose and Functionality:** \nThe function `dependencies_for_class_node` takes a `ClassDef` node and a set of global names, and returns the dependencies required by that class.\n\n**Key Implementation Details:**\n\n- Creates a temporary module with the input class node\n- Instantiates a `ClassDependencyMapper` visitor with the class name and global names\n- Visits the temporary module, allowing the visitor to inspect dependencies\n\n**Notable Dependencies or Requirements:**\n\n- `cst` (Abstract Syntax Tree) module for parsing\n- `ClassDependencyMapper` visitor for dependency resolution\n- `global_names` set of imported names for dependency filtering"}


created_at: 2025-02-05 09:13:47
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py:556-565
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 565, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py", "name": "augmented_dependencies_for_class_node", "start_line": 556, "type": "function", "chroma:document": "CODE:\n\ndef augmented_dependencies_for_class_node(\n    node: cst.ClassDef, mapper: \"ModuleMapper\", objects_imported_from_modeling: set[str] | None = None\n) -> set:\n    \"\"\"Create augmented dependencies for a class node based on a `mapper`.\n    Augmented dependencies means immediate dependencies + recursive function and assignments dependencies.\n    \"\"\"\n    temp_module = cst.Module(body=[node])\n    visitor = ClassDependencyMapper(node.name.value, set(mapper.global_nodes.keys()), objects_imported_from_modeling)\n    temp_module.visit(visitor)\n\n\nSUMMARY:\n**Summary:**\n\n`augmented_dependencies_for_class_node` function:\n Creates augmented dependencies for a class node by traversing its members and recursively visiting dependent modules.\n\n**Key Highlights:**\n\n* Takes in `cst.ClassDef`, `ModuleMapper`, and `objects_imported_from_modeling` parameters.\n* Visits the class node using a custom `ClassDependencyMapper` visitor, which collects dependencies.\n* Returns augmented dependencies via the ` augment_dependencies` method of the `ModuleMapper`.\n\n**Implementation Details:**\n\n* Creates a temporary module with the class node as its body.\n* Uses the `ClassDependencyMapper` to recursively traverse dependent modules.\n* Augments the collected dependencies using the `augment_dependencies` method."}


created_at: 2025-02-05 09:13:47
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py:579-733
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 733, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py", "name": "ModuleMapper", "start_line": 579, "type": "class", "chroma:document": "CODE:\n\nclass ModuleMapper(CSTVisitor, ABC):\n    \"\"\"An abstract visitor class which analyses a module, creating a mapping of dependencies for classes, functions and assignments.\n    Class dependencies are computed with `compute_class_dependencies()`, while function and assignment dependencies are stored in\n    `self.object_recursive_dependency_mapping` (can be computed by `_compute_recursive_object_dependencies()`).\n    It defines common visiting patterns (i.e. common visit_xxx/leave_xxx functions) between the modular file and the\n    modeling files that will be visited.\n    \"\"\"\n\n    METADATA_DEPENDENCIES = (ParentNodeProvider, PositionProvider)\n\n    def __init__(self, python_module: cst.Module):\n        # fmt: off\n        self.python_module: cst.Module = python_module             # original cst.Module being visited\n        self.classes: Dict[str, cst.ClassDef] = {}                 # mapping from class names to Nodes (it will be ordered by default!!)\n        self.imports = []                                          # stores all import statements\n        self.functions: Dict[str, cst.FunctionDef] = {}            # mapping of global scope function names to Nodes\n        self.object_dependency_mapping = defaultdict(set)          # immediate function/assignment dependency mapping (i.e. dependencies immediately in the function/assignment definition)\n        self.assignments: Dict[str, cst.SimpleStatementLine] = {}  # mapping of global assignments names to Nodes\n        self.current_function = None                               # this keeps track of the current module-scope function\n        self.current_assignment = None                             # this keeps track of the current module-scope assignment\n        # this keeps track of objects imported from modeling files (`from .configuration import Config`) -> `Config` should not be a dependency\n        self.objects_imported_from_modeling = set()\n        # regex pattern joining every possible file type\n        self.match_patterns = \"|\".join(ALL_FILE_TYPES)\n        # fmt: on\n\n    def visit_ImportFrom(self, node):\n        \"\"\"This keeps track of objects imported from neighbor modeling files (e.g. in `modeling_xxx.py, we have\n        `from .configuration_xxx import Config`, then `Config` should be recorded as it is not a dependency that needs\n        to be added (because it will be part of the imports)\"\"\"\n        import_module = self.python_module.code_for_node(node.module)\n        import_statement = \".\" * len(node.relative) + import_module\n        if re.search(rf\"^\\.({self.match_patterns})_.*\", import_statement):\n            for imported_object in node.names:\n                # If an alias is present, we record it and not the original name\n                if imported_object.evaluated_alias is not None:\n                    self.objects_imported_from_modeling.add(imported_object.evaluated_alias)\n                else:\n                    self.objects_imported_from_modeling.add(imported_object.evaluated_name)\n\n    def visit_SimpleStatementLine(self, node):\n        \"\"\"\n        Global Assigns like `GEMMA_INPUT_DOCSTRING = 'THIS IS THE INPUT'` and all import statements\n        are extracted and saved in their corresponding dict. They are then used when updating dependency mappings.\n        \"\"\"\n        parent_node = self.get_metadata(cst.metadata.ParentNodeProvider, node)\n        simple_top_level_assign_structure = m.SimpleStatementLine(\n            body=[m.Assign(targets=[m.AssignTarget(target=m.Name())])]\n        )\n        if m.matches(parent_node, m.Module()):\n            if m.matches(node, simple_top_level_assign_structure):\n                left_hand_side = node.body[0].targets[0].target.value\n                self.current_assignment = left_hand_side\n                self.assignments[left_hand_side] = node\n            elif m.matches(node, m.SimpleStatementLine(body=[m.Import() | m.ImportFrom()])):\n                self.imports.append(node)\n\n    def leave_SimpleStatementLine(self, node):\n        # No need to check for the parent here -> everytime we exit one, it should be None anyway independently of where the\n        # SimpleStatement is located\n        self.current_assignment = None\n\n    def visit_FunctionDef(self, node):\n        parent_node = self.get_metadata(cst.metadata.ParentNodeProvider, node)\n        if m.matches(parent_node, m.Module()):\n            self.current_function = node.name.value\n            self.functions[node.name.value] = node\n\n    def leave_FunctionDef(self, node):\n        parent_node = self.get_metadata(cst.metadata.ParentNodeProvider, node)\n        if m.matches(parent_node, m.Module()):\n            self.current_function = None\n\n    def visit_If(self, node):\n        for stmt in node.body.body:\n            if m.matches(stmt, m.SimpleStatementLine(body=[m.ImportFrom() | m.Import()])):\n                self.imports.append(node)\n\n    def visit_ClassDef(self, node: ClassDef) -> None:\n        \"\"\"Record class nodes to create their dependencies at the end.\"\"\"\n        self.classes[node.name.value] = node\n\n    def visit_Name(self, node: cst.Call):\n        \"\"\"This is used to create a mapping from module-scope functions and assignments to objects used inside them.\"\"\"\n        if self.current_function is not None:\n            self.object_dependency_mapping[self.current_function].add(node.value)\n        if self.current_assignment is not None:\n            self.object_dependency_mapping[self.current_assignment].add(node.value)\n\n    def leave_Module(self, node):\n        \"\"\"When leaving the module, we store the position of each global scoped node to allow sorting the dependencies\n        based on their position in the code later. We use the PositionProvider metadata wrapper for this.\n        We also make sure to update `self.object_dependency_mapping` so that it contains only names recorded in\n        `self.global_nodes`.\n        \"\"\"\n        # assign all nodes\n        self.global_nodes = {**self.assignments, **self.classes, **self.functions}\n        # now sort the class dependency_mapping based on the position of the nodes\n        self.start_lines = {}\n        for id, node in self.global_nodes.items():\n            self.start_lines[id] = self.get_metadata(cst.metadata.PositionProvider, node).start.line\n\n    def _restrict_dependencies_to_known_entities(self):\n        \"\"\"Since we added every Name as part of `self.object_dependency_mapping`, we need to remove those that\n        are not part of the recorded objects in `self.global_nodes` (i.e. built-in variables, imports, etc).\n        This should be called only after all merging operations have been finalized!!\"\"\"\n        global_objects = set(self.global_nodes.keys())\n        for object_name, dependencies in self.object_dependency_mapping.items():\n            self.object_dependency_mapping[object_name] = {dep for dep in dependencies if dep in global_objects}\n\n    def _compute_recursive_object_dependencies(self) -> dict[str, set]:\n        \"\"\"Based on immediate dependency mapping, create the recursive dependency mapping. For example, given the\n        following file:\n        ```\n        def foo():\n            pass\n\n        def bar():\n            foo()\n\n        def test():\n            bar()\n        ```\n        this visitor can only record immediate dependencies, i.e. it will record the following\n        `self.object_dependency_mapping = {\"test\": {\"bar\"}, \"bar\": {\"foo}}`. This function is used to create\n        the recursive mapping, i.e. `recursive_dependencies = {\"test\": {\"bar\", \"foo\"}, \"bar\": {\"foo}}`.\n        \"\"\"\n        recursive_dependencies = {}\n        for object_name in self.object_dependency_mapping.keys():\n            all_dependencies = find_all_dependencies(self.object_dependency_mapping, start_entity=object_name)\n            recursive_dependencies[object_name] = all_dependencies\n        return recursive_dependencies\n\n    def augment_dependencies(self, dependencies: set[str]) -> set[str]:\n        \"\"\"For a set of `dependencies`, augment them by adding all potential dependencies of the **functions** and\n        **assignments** present in the `dependencies`.\n        \"\"\"\n        new_dependencies = dependencies.copy()\n        # Go through the set of dependencies\n        for dep in tuple(dependencies):\n            if dep in self.object_recursive_dependency_mapping.keys():\n                new_dependencies.update(self.object_recursive_dependency_mapping[dep])\n        return new_dependencies\n\n    def compute_class_dependencies(self):\n        \"\"\"For each visited class, find its dependencies based on visiting the current file + potential merged dependencies.\"\"\"\n        self.class_dependency_mapping = {}\n        for class_name, class_node in self.classes.items():\n            dependencies = dependencies_for_class_node(class_node, set(self.global_nodes.keys()))\n            # Correctly augment class dependencies with all needed objects\n            self.class_dependency_mapping[class_name] = self.augment_dependencies(dependencies)\n\n    @abstractmethod\n    def compute_relative_order(self, missing_dependencies: set) -> dict[str, int]:\n\n\nSUMMARY:\n**ModuleMapper Summary**\n\nThe `ModuleMapper` class is an abstract visitor that analyzes a Python module, creating a mapping of dependencies for classes, functions, and assignments. It computes dependencies using `compute_class_dependencies()` and `_compute_recursive_object_dependencies()`, storing them in `self.object_recursive_dependency_mapping`.\n\n**Main Purpose and Functionality:**\n\n* Analyzes a Python module to create a dependency mapping\n* Computes class dependencies using `compute_class_dependencies()`\n* Stores object dependencies using `_compute_recursive_object_dependencies()`\n* Augments dependencies using `augment_dependencies()` and `dependencies_for_class_node()`\n\n**Class Relationship Mapping:**\n\n* Records class nodes in `self.classes` for future dependencies\n* Maps functions and assignments to objects used inside them using `visit_Name()` and `visit_ClassDef()`\n* Updates `self.object_recursive_dependency_mapping` with recursive dependencies\n\n**Dependencies Computation:**\n\n* Computes absolute dependencies using `_compute_recursive_object_dependencies()`\n* Augments dependencies with potential dependencies of functions and assignments\n\n**Abstract Methods:**\n\n* `compute_relative_order(missing_dependencies: set)`: returns a dictionary mapping class names to their relative order for missing dependencies.\n\nNote: The code uses various internal classes, methods, and metadata wrappers that are not included in this summary."}


created_at: 2025-02-05 09:13:47
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py:736-884
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 884, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py", "name": "ModelFileMapper", "start_line": 736, "type": "class", "chroma:document": "CODE:\n\nclass ModelFileMapper(ModuleMapper):\n    \"\"\"A mapper designed to parse modeling files (like `modeling_llama.py`). When encountering such a file\n    in the `modular_xxx.py` file, we need to correctly visit it and merge the dependencies of the modular and current file.\n    For this reason, this class should only be instantiated from the class method `visit_and_merge_dependencies`, which takes\n    care of correctly merging dependencies, then finalizes all dependency graph computations.\n    Note that we only merge functions and assignments here, as classes will be treated later on as they may be modified.\n    For example, if you redefine `apply_rotary_pos_emb()` in the modular, the new node should be used in the dependencies\n    of the modeling files as well.\n    \"\"\"\n\n    def __init__(self, python_module: cst.Module):\n        super().__init__(python_module)\n\n    def compute_relative_order(self, missing_dependencies: set[str]) -> dict[str, int]:\n        \"\"\"Compute in which relative order the `missing_dependencies` should appear when the nodes are added to the final file that\n        will be created based on the modular.\n        \"\"\"\n        relative_order = {}\n        idx = 0\n        classes = sorted(\n            [dep for dep in tuple(missing_dependencies) if dep in self.classes], key=lambda x: self.start_lines[x]\n        )\n        # This is because for merged dependencies, we only have relative order in the other visited file, so we need\n        # to track dependency order relative to a given class\n        if len(classes) > 0 and not hasattr(self, \"class_dependency_mapping\"):\n            raise ValueError(\"Cannot correctly find the relative order of the dependencies.\")\n\n        remaining_dependencies = missing_dependencies.copy()\n\n        # Start by tracking relative order class by class\n        for class_name in classes:\n            class_dependencies = tuple(self.class_dependency_mapping[class_name] & remaining_dependencies)\n            original_dependencies = []\n            merged_dependencies = []\n            # We need to differentiate between nodes that were already present (we can get relative order globally) and\n            # nodes that were merged (we can get relative order only relative to the class the dependencies relate to)\n            for class_dep in class_dependencies:\n                if class_dep in self.start_lines:\n                    original_dependencies.append(class_dep)\n                else:\n                    merged_dependencies.append(class_dep)\n            # Sort both list according to the order in their respective file\n            original_dependencies = sorted(original_dependencies, key=lambda x: self.start_lines[x])\n            merged_dependencies = sorted(merged_dependencies, key=lambda x: self.modular_file_start_lines[x])\n\n            # Add all original node first, then merged ones\n            for dep in original_dependencies + merged_dependencies:\n                remaining_dependencies.remove(dep)\n                relative_order[dep] = idx\n                idx += 1\n            # Add the class itself (it can sometimes already be present if the order of classes in the source file\n            # does not make sense, i.e. a class is used somewhere before being defined like in `rt_detr`...)\n            if class_name in remaining_dependencies:\n                remaining_dependencies.remove(class_name)\n                relative_order[class_name] = idx\n                idx += 1\n\n        # Now add what still remains\n        remaining_dependencies = tuple(remaining_dependencies)\n        original_dependencies = []\n        merged_dependencies = []\n        for dep in remaining_dependencies:\n            if dep in self.modular_file_start_lines:\n                merged_dependencies.append(dep)\n            else:\n                original_dependencies.append(dep)\n        # Sort both list according to the order in their respective file\n        original_dependencies = sorted(original_dependencies, key=lambda x: self.start_lines[x])\n        merged_dependencies = sorted(merged_dependencies, key=lambda x: self.modular_file_start_lines[x])\n\n        # Add all original node first, then merged ones\n        for dep in original_dependencies + merged_dependencies:\n            relative_order[dep] = idx\n            idx += 1\n\n        return relative_order\n\n    def _merge_functions(self, functions: dict[str, cst.CSTNode], object_mapping: dict[str, set]):\n        \"\"\"Update the global nodes and function dependency mapping with those from the modular file.\n\n        Merging rule: if any function with the same name was redefined in the modular, use it and its dependencies\n        instead of the original ones (this may mean to add new functions as well, if any redefined function uses a new one).\n        \"\"\"\n        # Add/overwrite all needed function nodes and dependencies\n        self.functions.update(functions)\n        self.object_dependency_mapping.update(\n            {obj: dep for obj, dep in object_mapping.items() if obj in functions.keys()}\n        )\n        # Add them to global nodes\n        self.global_nodes.update(self.functions)\n\n    def _merge_assignments(self, assignments: dict[str, cst.CSTNode], object_mapping: dict[str, set]):\n        \"\"\"Update the global nodes with the assignment from the modular file.\n\n        Merging rule: if any assignment with the same name was redefined in the modular, we use it and its dependencies ONLY if it is\n        in `ASSIGNMENTS_TO_KEEP`. Otherwise, we use the original value and dependencies. This rule was chosen to avoid having to rewrite the\n        big docstrings.\n        \"\"\"\n        for assignment, node in assignments.items():\n            if assignment in ASSIGNMENTS_TO_KEEP or assignment not in self.assignments:\n                self.assignments[assignment] = node\n                if assignment in object_mapping:\n                    self.object_dependency_mapping[assignment] = object_mapping[assignment]\n        # Add them to global nodes\n        self.global_nodes.update(self.assignments)\n\n    def _merge_classes(self, classes: dict[str, cst.CSTNode]):\n        \"\"\"Update the global nodes with the new classes from the modular (i.e. classes which do not exist in current file, and\n        are not imported). We do NOT update any dependency mapping here. This is because we only need the names of newly defined\n        classes in the modular to be discoverable when computing dependencies for new nodes later on. For this reason, we\n        do not add the new classes to `self.classes`, but only to `global_nodes`.\n        \"\"\"\n        # Add/overwrite all needed function nodes and dependencies\n        self.global_nodes.update(\n            {\n                name: node\n                for name, node in classes.items()\n                if name not in self.classes and name not in self.objects_imported_from_modeling\n            }\n        )\n\n    def merge_modular_dependencies(self, classes, functions, assignments, object_mapping, start_lines):\n        \"\"\"Merge classes, functions and assignments from the modular definitions into the current module file,\n        then record the relative order of all nodes.\n        Note: This function takes care of updating `global_nodes` and `object_recursive_dependency_mapping` as well after the\n        merge with other files dependencies.\n        \"\"\"\n        self._merge_functions(functions, object_mapping)\n        self._merge_assignments(assignments, object_mapping)\n        self._merge_classes(classes)\n        self.modular_file_start_lines = start_lines\n\n        # Restrict the dependency mappings to the known entities to avoid Python's built-ins and imports\n        self._restrict_dependencies_to_known_entities()\n        # Create the global mapping of recursive dependencies for functions and assignments\n        self.object_recursive_dependency_mapping = self._compute_recursive_object_dependencies()\n\n    @classmethod\n    def visit_and_merge_dependencies(\n        cls, module: cst.Module, classes, functions, assignments, object_mapping, start_lines\n    ) -> \"ModelFileMapper\":\n        wrapper = MetadataWrapper(module)\n        mapper = cls(module)\n        wrapper.visit(mapper)\n        # Merge dependencies\n        mapper.merge_modular_dependencies(classes, functions, assignments, object_mapping, start_lines)\n        # Create the class dependencies graph\n        mapper.compute_class_dependencies()\n\n\nSUMMARY:\n**ModelFileMapper Class Summary**\n\nThe `ModelFileMapper` class is designed to parse modeling files and merge their dependencies with those of the modular file.\n\n### Purpose\nTo correctly visit and merge dependencies between modules, ensuring that functions, assignments, and classes are properly updated and ordered relative to each other.\n\n### Key Implementation Details\n\n*   **`merge_modular_dependencies` method**: Merges classes, functions, and assignments from the modular definitions into the current module file.\n*   **`compute_relative_order` method**: Computes the relative order of all nodes based on their dependencies and ordering conventions in the source code.\n*   **Methods for merging specific node types (functions, assignments, classes)**\n*   **Updating global nodes and dependency mappings** after merging with modular file dependencies\n\n### Notable Dependencies or Requirements\n\n*   `cst` module for parsing Python code\n*   `MetadataWrapper` class for accessing metadata of the parsed module\n*   External dependencies are restricted to known entities using `_restrict_dependencies_to_known_entities` method.\n\n### Class Inheritance and Interfaces\n\nNone, as it is a custom implementation.\n\n**Example Usage**\n\n```python\nimport cst\n\n# Sample module definition\nmodule_def = \"\"\"class MyClass:\n    def __init__(self):\n        pass\"\"\"\n\n# Create an instance of the ModelFileMapper class\nmapper = ModelFileMapper(module)  # Assuming 'module' is a cst module object\n\n# Call the visit_and_merge_dependencies method to process the module\nmapper.visit_and_merge_dependencies(...\n```\n\nThis code snippet provides a basic overview of how to use the `ModelFileMapper` class."}


created_at: 2025-02-05 09:13:47
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py:887-899
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 899, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py", "name": "common_partial_suffix", "start_line": 887, "type": "function", "chroma:document": "CODE:\n\ndef common_partial_suffix(str1: str, str2: str) -> str:\n    \"\"\"Return the biggest common suffix between 2 strings. If one string is a full suffix of the other string,\n    we do not consider it a common suffix and return `\"\"`\"\"\"\n    common_suffix = \"\"\n    for i in range(1, min(len(str1), len(str2)) + 1):\n        if str1[-i] == str2[-i]:\n            common_suffix = str1[-i] + common_suffix\n        else:\n            break\n    # We do not allow full string suffix\n    if common_suffix == str1 or common_suffix == str2:\n        common_suffix = \"\"\n\n\nSUMMARY:\n**Summary:**\n\n### Function Name:\n`common_partial_suffix`\n\n### Primary Purpose:\nReturns the largest common partial suffix between two input strings.\n\n### Key Implementation Details:\n\n* Iterates from the shortest string's end to find matching characters.\n* Builds the `common_suffix` by prepending matching characters from `str1`.\n* If one string is a full suffix of the other, returns an empty string.\n\n### Notable Dependencies/Requirements:\n\n* No external dependencies mentioned.\n* The function uses built-in Python string concatenation and iteration."}


created_at: 2025-02-05 09:13:47
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py:902-1053
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 1053, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py", "name": "replace_class_node", "start_line": 902, "type": "function", "chroma:document": "CODE:\n\ndef replace_class_node(\n    mapper: ModelFileMapper, class_node: cst.ClassDef, renamed_super_class: str, original_super_class: str\n):\n    \"\"\"\n    Replace a class node which inherits from another modeling class. This function works in the following way:\n    - start from the base class node of the inherited class (a cst.Node)\n    - replace all methods of the base node with the methods defined in the child class\n    - append all new methods defined in the child class\n    - replace all calls to super() with the unravelled code\n\n                    |    ```python                          |               |    ```python\n                    |    class GemmaModel(LlamaModel):      |               |       class GemmaModel(nn.Module):\n                    |        def __init__(self):            |               |           def __init__(self):\n    Going from:     |            super().__init__()         |       to:     |               super().__init__(config)\n                    |            self.dropout = 0.2         |               |               self.dropout = 0.2\n                    |     ```                               |               |               self.padding_idx = config.pad_token_id\n                                                                            |               self.vocab_size = config.vocab_size\n                                                                            |               self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx)\n                                                                            |               self.layers = nn.ModuleList(\n                                                                            |                   [LlamaDecoderLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)]\n                                                                            |               )\n                                                                            |               self.norm = LlamaRMSNorm(config.hidden_size, eps=config.rms_norm_eps)\n                                                                            |               self.gradient_checkpointing = False\n                                                                            |               # Initialize weights and apply final processing\n                                                                            |               self.post_init()\n                                                                            |     ```\n    \"\"\"\n    all_bases = [get_full_attribute_name(k.value) for k in class_node.bases]\n    if any(base is None for base in all_bases):\n        raise ValueError(f\"Could not parse the name of the bases for {class_node.name.value}\")\n\n    original_node = mapper.classes[renamed_super_class]\n    # Always use the new name of the class (in case we use e.g. `ColPaliForRetrieval` inheriting from `PaliGemmaForConditionalGeneration`)\n    new_name = class_node.name\n\n    # If the new class name is different from the renamed super class name, we need to update the docstrings/comments accordingly\n    if new_name.value != renamed_super_class:\n        common_suffix = common_partial_suffix(new_name.value, renamed_super_class)\n        # Note that this works even without common prefix, in which case it does not replace anything\n        old, new = renamed_super_class.replace(common_suffix, \"\"), new_name.value.replace(common_suffix, \"\")\n        temp_module = cst.Module(body=[original_node])\n        original_node = temp_module.visit(\n            ReplaceNameTransformer(get_lowercase_name(old), get_lowercase_name(new), only_doc=True)\n        ).body[0]\n\n    # If we explicitly passed a new base with common suffix to an old base, it is for switching the prefix\n    # e.g. if the \"natural\" parent class is `PreTrainedModel` but we wanted to rename it to `PreTrainedVisionModel`\n    additional_bases = [base for base in all_bases if base != original_super_class]\n    new_bases = []\n    for original_base in original_node.bases:\n        new_base = original_base\n        # we only potentially switch base for Name-based bases, not Attribute\n        if m.matches(original_base.value, m.Name()):\n            original_base_name = original_base.value.value\n            for additional_base_name in additional_bases:\n                suffix = common_partial_suffix(original_base_name, additional_base_name)\n                if len(suffix) > 0 and suffix[0].isupper():\n                    new_name_node = original_base.value.with_changes(value=additional_base_name)\n                    new_base = original_base.with_changes(value=new_name_node)\n                    break\n        new_bases.append(new_base)\n\n    original_methods = {\n        f.name.value if hasattr(f, \"name\") else mapper.python_module.code_for_node(f): f\n        for f in original_node.body.body\n    }\n    updated_methods = {\n        f.name.value if hasattr(f, \"name\") else mapper.python_module.code_for_node(f): f for f in class_node.body.body\n    }\n    end_meth = []\n\n    assign_targets = {}\n    docstring_node = []\n    # Iterate directly from node.body as there can be property/setters with same names which are overwritten when we use a dict\n    for func in original_node.body.body:\n        name = func.name.value if hasattr(func, \"name\") else mapper.python_module.code_for_node(func)\n        if m.matches(func, m.FunctionDef()) and name in updated_methods and updated_methods[name] is not None:\n            new_params = updated_methods[name].params\n            # Replace the method in the replacement class, preserving decorators\n            kwarg_name = getattr(updated_methods[name].params, \"star_kwarg\", None)\n            if kwarg_name and kwarg_name.name.value == \"super_kwargs\":\n                parent_params = {k.name.value: k for k in func.params.params}\n                parent_params.update({k.name.value: k for k in new_params.params[1:]})\n                new_params = new_params.with_changes(\n                    params=list(parent_params.values()), star_kwarg=func.params.star_kwarg\n                )\n            # Keep decorators in `modular_xxx.py` if any, else original decorators\n            new_decorators = (\n                updated_methods[name].decorators if len(updated_methods[name].decorators) > 0 else func.decorators\n            )\n            if not re.match(\n                r\"\\ndef .*\\(.*\\):\\n    raise.*Error\\(.*\",\n                mapper.python_module.code_for_node(updated_methods[name]),\n            ):\n                func = func.with_changes(body=updated_methods[name].body, params=new_params, decorators=new_decorators)\n            else:\n                continue\n\n        if m.matches(func, m.SimpleStatementLine(body=[m.Assign()])):\n            target = mapper.python_module.code_for_node(func.body[0].targets[0])\n            assign_targets[target] = func\n        elif m.matches(func, m.SimpleStatementLine(body=[m.AnnAssign()])):\n            target = mapper.python_module.code_for_node(func.body[0].target)\n            assign_targets[target] = func\n        elif m.matches(func, DOCSTRING_NODE):\n            docstring_node = [func]\n        else:\n            end_meth.append(func)\n\n    # Port new methods that are defined only in modular-file and append at the end\n    for func in class_node.body.body:\n        name = func.name.value if hasattr(func, \"name\") else mapper.python_module.code_for_node(func)\n        if m.matches(func, DOCSTRING_NODE):  # This processes the docstring of the class!\n            # Extract the original docstring\n            updated_docstring = func.body[0].value.value\n            if len(docstring_node) == 0:  # If the original docstring is empty, just create one from the updated.\n                docstring_node = [\n                    cst.SimpleStatementLine(body=[cst.Expr(value=cst.SimpleString(value=updated_docstring))])\n                ]\n            else:\n                original_docstring = docstring_node[0].body[0].value.value\n                merged_doc = merge_docstrings(original_docstring, updated_docstring)\n                # Update the docstring in the original function\n                docstring_node = [\n                    docstring_node[0].with_changes(body=[cst.Expr(value=cst.SimpleString(value=merged_doc))])\n                ]\n        if name not in original_methods and func is not None and isinstance(func, cst.FunctionDef):\n            end_meth.append(func)\n        if m.matches(func, m.SimpleStatementLine(body=[m.Assign()])):\n            # TODO we only use single assign might cause issues\n            target = mapper.python_module.code_for_node(func.body[0].targets[0])\n            assign_targets[target] = func\n        if m.matches(func, m.SimpleStatementLine(body=[m.AnnAssign()])):\n            target = mapper.python_module.code_for_node(func.body[0].target)\n            assign_targets[target] = func\n    end_meth = docstring_node + list(assign_targets.values()) + end_meth\n\n    # Replace the calls to `super()` with the unrolled code\n    result_node = original_node.with_changes(body=cst.IndentedBlock(body=end_meth))\n    temp_module = cst.Module(body=[result_node])\n    new_module = MetadataWrapper(temp_module)\n    new_replacement_class = new_module.visit(\n        SuperTransformer(temp_module, original_methods, updated_methods, all_bases)\n    )\n    new_replacement_body = new_replacement_class.body[0].body  # get the indented block\n\n    # Use decorators redefined in `modular_xxx.py` if any\n    new_decorators = class_node.decorators if len(class_node.decorators) > 0 else original_node.decorators\n\n    return original_node.with_changes(\n        body=new_replacement_body, decorators=new_decorators, bases=new_bases, name=new_name\n\n\nSUMMARY:\nThis function replaces a Python class node in an abstract syntax tree (AST) with a new implementation. The replacement process involves:\n\n1. Updating the base classes and names of the class.\n2. Replacing methods defined in the base classes with those defined in the child class.\n3. Appending new methods not present in the parent class but present in the child class.\n4. Replacing calls to `super()` with the updated code.\n\nThe function takes as input a model file mapper, a node representing the class to be replaced, and the names of the superclasses and the child's superclass. It returns a new node representing the modified class."}


created_at: 2025-02-05 09:13:47
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py:1069-1080
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 1080, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py", "name": "find_file_type", "start_line": 1069, "type": "function", "chroma:document": "CODE:\n\ndef find_file_type(class_name: str) -> str:\n    \"\"\"Based on a class name, find the file type corresponding to the class.\n    If the class name is `LlamaConfig` it will return `configuration`.\n    The list of suffixes is in `TYPE_TO_FILE_TYPE`. If there are no match, we match by default to `modeling`\n    \"\"\"\n    match_pattern = \"|\".join(TYPE_TO_FILE_TYPE.keys())\n    match = re.search(rf\"({match_pattern})$\", class_name)\n    if match:\n        file_type = TYPE_TO_FILE_TYPE[match.group(1)]\n    else:\n        file_type = \"modeling\"\n\n\nSUMMARY:\n**Summary:**\n\n### Function Purpose and Functionality\n\n`find_file_type` function takes a `class_name` string as input and returns the corresponding file type. It uses a lookup table `TYPE_TO_FILE_TYPE` to match class names against predefined file types.\n\n### Key Implementation Details\n\n* Uses regular expressions to search for matching file types\n* Falls back to \"modeling\" if no match is found in `TYPE_TO_FILE_TYPE`\n\n### Notable Dependencies and Requirements\n\n* `re` (regular expression) module\n* `TYPE_TO_FILE_TYPE` dictionary (not shown in this snippet)\n\n### Method Signatures and Interfaces\n\n* `find_file_type(class_name: str) -> str`: returns a file type string based on the input class name"}


created_at: 2025-02-05 09:13:47
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py:1095-1109
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 1109, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py", "name": "append_new_import_node", "start_line": 1095, "type": "function", "chroma:document": "CODE:\n\ndef append_new_import_node(\n    node: cst.CSTNode, unused_imports: set[str], added_names: set, imports_to_keep: list[cst.CSTNode]\n):\n    \"\"\"Insert the new `node` to the list of `imports_to_keep` in-place, if it is not part of the `unused_imports` or `added_names`.\n    Also modifies `added_names` in-place accordingly.\"\"\"\n    import_node = node.body[0]\n    names_to_keep = []\n    for name in import_node.names:\n        name_value = name.evaluated_name\n        if name_value not in unused_imports and name_value not in added_names:\n            names_to_keep.append(name.with_changes(comma=cst.MaybeSentinel.DEFAULT))\n            added_names.add(name_value)\n    if len(names_to_keep) > 0:\n        new_node = node.with_changes(body=[import_node.with_changes(names=names_to_keep)])\n\n\nSUMMARY:\n**Summary of `append_new_import_node` function**\n\nInserts a new import node (`node`) to the `imports_to_keep` list if it's not part of unused or added imports. Modifies `added_names` accordingly.\n\n**Key Implementation Details:**\n\n* Inserts `import_node` into `imports_to_keep` with filtered names.\n\t+ `names_to_keep = [name_value]` where `name_value != unused_imports` and `name_value != added_names`.\n* Applies modifications to `node` using:\n\t+ `with_changes(body=[import_node.with_changes(names=names_to_keep)])`\n\t+ `imports_to_keep.append(new_node)`.\n\n**Notable Dependencies/Requirements:**\n\n* `cst.CSTNode`: Input node type.\n* `added_names`: Set of names to keep track of.\n* `unused_imports`: Set of unused imports.\n\n**External Dependency:**\n\n* `cst.MaybeSentinel.DEFAULT`."}


created_at: 2025-02-05 09:13:47
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py:1112-1155
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 1155, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py", "name": "get_needed_imports", "start_line": 1112, "type": "function", "chroma:document": "CODE:\n\ndef get_needed_imports(body: dict[str, dict], all_imports: list[cst.CSTNode]) -> list[cst.CSTNode]:\n    \"\"\"Get all the imports needed in the `body`, from the list of `all_imports`.\n    `body` is a dict with the following structure `{str: {\"insert_idx\": int, \"node\": cst.CSTNode}}`.\n    Note: we need to use `isinstance` on scope assignements, m.matches apparently does not work here yet!\n    \"\"\"\n    new_body = [k[1][\"node\"] for k in sorted(body.items(), key=lambda x: x[1][\"insert_idx\"])]\n    wrapper = MetadataWrapper(cst.Module(body=all_imports + new_body))\n    scopes = set(wrapper.resolve(ScopeProvider).values())\n    unused_imports = set()\n    import_ref_count = defaultdict(lambda: 0)\n    for scope in scopes:\n        for assignment in scope.assignments:\n            node = assignment.node\n            if isinstance(assignment, cst.metadata.Assignment) and isinstance(node, (cst.Import, cst.ImportFrom)):\n                ref_count = len(assignment.references)\n                name = assignment.name\n                import_ref_count[name] = max(ref_count, import_ref_count[name])\n    # Similar imports may be redefined, and only used between their 1st and 2nd definition so if we already have\n    # a ref count > 0 at any point, the imports is actually used\n    unused_imports = {name for name, count in import_ref_count.items() if count <= 0 or name in body.keys()}\n\n    imports_to_keep = []\n    # We need to keep track of which names were already imported, because some import may be duplicated from multiple sources\n    # or be both protected and unprotected due to inconsistency between models\n    added_names = set()\n    existing_protected_statements = set()  # str repr of the import nodes - does not work with the nodes directly\n    for node in all_imports:\n        if m.matches(node, m.If()):  # handle safe imports\n            new_statements = []\n            for stmt_node in node.body.body:\n                append_new_import_node(stmt_node, unused_imports, added_names, new_statements)\n            new_statements = [stmt for stmt in new_statements if str(stmt) not in existing_protected_statements]\n            if len(new_statements) > 0:\n                new_node = node.with_changes(body=node.body.with_changes(body=new_statements))\n                imports_to_keep.append(new_node)\n                existing_protected_statements.update({str(stmt) for stmt in new_statements})\n        else:\n            append_new_import_node(node, unused_imports, added_names, imports_to_keep)\n\n    protected_import_nodes = [node for node in imports_to_keep if m.matches(node, m.If())]\n    usual_import_nodes = [node for node in imports_to_keep if not m.matches(node, m.If())]\n\n    # Protected imports always appear at the end of all imports\n\n\nSUMMARY:\n**Function Summary: `get_needed_imports`**\n\n*   Purpose:\n    *   Generates needed imports for a given code segment (`body`) from a list of provided imports (`all_imports`)\n*   Functionality:\n\n    1.  **Sorting and Preparation**: Sorts the `body` dictionary based on insert indices, creates a metadata wrapper around the combined `all_imports + new_body`, and resolves scopes to identify assignments.\n    2.  **Import Ref Counting**: Iterates through scope assignments to count reference counts for each import name.\n    3.  **Unused Imports Detection**: Identifies unused imports based on ref count threshold and existing references in the code segment (`body`).\n*   Output:\n    *   A list of needed imports, separated into `usual` (non-protected) and `protected` nodes.\n\n**Key Implementation Details:**\n\n*   Uses a dictionary-based approach to sort and prepare the input data.\n*   Employs a metadata wrapper to analyze import statements.\n*   Utilizes scope resolution to identify assignments and determine ref counts.\n*   Applies thresholds for identifying unused imports.\n\n**Notable Dependencies/Requirements:**\n\n*   `cst.CSTNode` (used for parsing and analyzing code structure)\n*   `MetadataWrapper` (custom class used for metadata-driven analysis)\n*   `ScopeProvider` (interface for scope resolution)\n*   Custom modules (`m.matches`, `append_new_import_node`) not explicitly referenced in the provided function definition."}


created_at: 2025-02-05 09:13:48
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py:1158-1176
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 1176, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py", "name": "split_all_assignment", "start_line": 1158, "type": "function", "chroma:document": "CODE:\n\ndef split_all_assignment(node: cst.CSTNode) -> dict[str, cst.CSTNode]:\n    \"\"\"Split the `__all__` assignment found in the modular between each corresponding files.\"\"\"\n    all_all_per_file = {}\n    assign_node = node.body[0]\n    if isinstance(assign_node.value, cst.List):\n        # Extract the elements from the list\n        all_all_to_add = defaultdict(list)\n        for element in assign_node.value.elements:\n            if isinstance(element.value, cst.SimpleString):\n                # Remove quotes and add the string to the elements list\n                class_name = element.value.value\n                file = find_file_type(element.value.evaluated_value)\n                all_all_to_add[file] += [class_name]\n        for file, new_alls in all_all_to_add.items():\n            new_node = assign_node.with_changes(\n                value=cst.List(elements=[cst.Element(value=cst.SimpleString(value=k)) for k in new_alls])\n            )\n            all_all_per_file[file] = node.with_changes(body=[new_node])\n\n\nSUMMARY:\n**Function Summary: `split_all_assignment`**\n\n**Purpose:** Splits the `__all__` assignment in a modular into separate assignments for corresponding files.\n\n**Main Functionality:**\nExtracts class names from the list of values assigned to `__all__`, determines the file type, and regroups them by file using a dictionary. This function splits the initial assignment into multiple assignments, one per file.\n\n**Key Implementation Details:**\n\n* Iterates through a list of values (`node.body[0].value.elements`) and extracts class names from strings.\n* Determines the file type for each class name using `find_file_type(element.value.evaluated_value)`.\n* Regroups class names by file using a dictionary (`all_all_to_add`).\n\n**Method Signatures and Interfaces:**\n\n* `assign_node.with_changes`: Replaces the value of an assignment node with a new list.\n* `node.with_changes`: Wraps the entire node with new body.\n\n**Notable Dependencies and Requirements:**\n\n* `cst.CSTNode`\n* ` defaultdict(list)`\n* `find_file_type` (not provided)"}


created_at: 2025-02-05 09:13:48
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py:1179-1448
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 1448, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py", "name": "ModularFileMapper", "start_line": 1179, "type": "class", "chroma:document": "CODE:\n\nclass ModularFileMapper(ModuleMapper):\n    \"\"\"This is a Mapper to visit a modular file (like `modular_llama.py`). It visits the whole file, recording dependency,\n    then visits all imported modeling files (like `modeling_llama.py`), and manages their mutual dependencies.\n    Calling the method `create_modules()` after visit will create all modules based on this modular file.\n    \"\"\"\n\n    def __init__(self, python_module, new_name):\n        super().__init__(python_module)\n        # fmt: off\n        self.model_name = new_name  # name of the model being defined. Should be in the format of `llama` or `layout_xlm` or `phi3`\n\n        self.model_specific_imported_objects: Dict[str, str] = {}  # e.g. {\"LlamaModel\": \"transformers.models.llama.modeling_llama\"}\n        self.model_specific_modules: Dict[str, cst.Module] = {}  # e.g. {\"transformers.models.llama.modeling_llama\": cst.Module}\n\n        self.all_all_to_add = {}\n        # fmt: on\n\n    def visit_ImportFrom(self, node: cst.ImportFrom) -> None:\n        \"\"\"When visiting imports from modeling files (i.e. `transformers.models.xxx`) we get the code, parse it,\n        and save it in `self.model_specific_modules` to later visit. The imported objects are saved in `self.model_specific_imported_objects`.\n        \"\"\"\n        import_module = self.python_module.code_for_node(node.module)\n        import_statement = \".\" * len(node.relative) + import_module\n        if any(import_to_skip in import_statement for import_to_skip in IMPORTS_TO_SKIP_IN_MODULAR):\n            return\n        if m.matches(node.module, m.Attribute()):\n            for imported_ in node.names:\n                _import = re.search(\n                    rf\"(?:transformers\\.models\\.)|(?:\\.\\.)\\w+\\.({self.match_patterns})_.*\", import_statement\n                )\n                if _import:\n                    source = _import.group(1)\n                    if source == \"modeling\" and \"Config\" in self.python_module.code_for_node(imported_):\n                        raise ValueError(\n                            f\"You are importing {self.python_module.code_for_node(imported_)} from the modeling file. Import from the `configuration_xxxx.py` file instead\"\n                        )\n                    if import_module not in self.model_specific_modules:\n                        if \"models\" not in import_module:\n                            import_module = \"models.\" + import_module\n                        if \"transformers\" not in import_module:\n                            import_module = \"transformers.\" + import_module\n                        source_code = get_module_source_from_name(import_module)\n                        tree = cst.parse_module(source_code)\n                        self.model_specific_modules[import_module] = tree\n                    imported_object = self.python_module.code_for_node(imported_.name)\n                    self.model_specific_imported_objects[imported_object] = import_module\n        if m.matches(node.module, m.Name()):\n            if \"transformers\" == import_module:\n                raise ValueError(\n                    f\"You are importing from {import_module} directly using global imports. Import from the correct local path\"\n                )\n\n    def visit_SimpleStatementLine(self, node):\n        \"\"\"If we visit an import statement not previously visited, record it. If we visit a module-scope assignment,\n        simply record it or, if it is `__all__`, split it between files where we should dispatch it.\n        \"\"\"\n        parent_node = self.get_metadata(cst.metadata.ParentNodeProvider, node)\n        simple_top_level_assign_structure = m.SimpleStatementLine(\n            body=[m.Assign(targets=[m.AssignTarget(target=m.Name())])]\n        )\n        if m.matches(parent_node, m.Module()):\n            if m.matches(node, m.SimpleStatementLine(body=[m.Import()])):\n                self.imports.append(node)\n            elif m.matches(node, m.SimpleStatementLine(body=[m.ImportFrom()])):\n                import_module = self.python_module.code_for_node(node.body[0].module)\n                import_statement = \".\" * len(node.body[0].relative) + import_module\n                if not (\n                    re.search(rf\"(?:transformers\\.models\\.)|(?:\\.\\.)\\w+\\.({self.match_patterns})_.*\", import_statement)\n                    and not any(import_to_skip in import_statement for import_to_skip in IMPORTS_TO_SKIP_IN_MODULAR)\n                ):\n                    self.imports.append(node)\n            elif m.matches(node, simple_top_level_assign_structure):\n                assigned_variable = node.body[0].targets[0].target.value\n                # __all__ is treated differently and not added to general assignments\n                if assigned_variable == \"__all__\":\n                    self.all_all_to_add = split_all_assignment(node)\n                else:\n                    self.current_assignment = assigned_variable\n                    self.assignments[assigned_variable] = node\n\n    def leave_Module(self, node):\n        \"\"\"When we leave the modular file, we do the following in order:\n        1. for each modeling file found in the imports, rename it with the new model name, visit it, and update\n        its dependency graph with the new function and assignment definitions found in the modular\n        2. update the modular dependency graph with the imported functions and assignments (found when visiting the matching files)\n        3. compute the nested (recursive) function and assignment dependencies\n        \"\"\"\n        # Takes care of finalizing our visit\n        super().leave_Module(node)\n\n        # 1. for each modeling file found in the imports, rename it with the new model name, visit it, and update dependencies\n        self.visited_modules = {}\n        self.renamers = {}\n        name_prefixes = self.infer_new_model_name()\n        for file, module in self.model_specific_modules.items():\n            file_model_name = file.split(\".\")[-2]\n            new_name = name_prefixes[file]\n            renamer = ReplaceNameTransformer(file_model_name, new_name, self.model_name)\n            renamed_module = module.visit(renamer)\n            self.visited_modules[file] = ModelFileMapper.visit_and_merge_dependencies(\n                renamed_module,\n                self.classes,\n                self.functions,\n                self.assignments,\n                self.object_dependency_mapping,\n                self.start_lines,\n            )\n            # We record it so that we can rename classes later the exact same way\n            self.renamers[file] = renamer\n\n        # 2. in turn, we need to add the imported functions/assignments to the dependencies of the modular mapper, using the\n        # definitions found in the visited files\n        self.merge_model_specific_imports(self.visited_modules)\n\n        # 3. compute the nested (recursive) function and assignment dependencies\n        self.object_recursive_dependency_mapping = self._compute_recursive_object_dependencies()\n\n        # We need to keep track of which objects were imported directly into which modeling file to not add them wrongly later\n        # Note that we may visit several of the same file types, thus we save them per file type, not file\n        self.imported_objects_per_file = defaultdict(set)\n        for file, mapper in self.visited_modules.items():\n            file_type = re.search(rf\"^transformers\\.models\\.\\w+\\.({self.match_patterns})_.*\", file).group(1)\n            self.imported_objects_per_file[file_type].update(mapper.objects_imported_from_modeling)\n\n    def merge_model_specific_imports(self, visited_modules):\n        \"\"\"Merge the functions and assignments imported from the modeling files to the modular nodes and dependency graph,\n        based on the visited files.\"\"\"\n        self.start_lines_file_mapping = {}\n        self.added_objects_file_mapping = {}\n        for object_name, file in self.model_specific_imported_objects.items():\n            visited_module = visited_modules[file]\n            self.start_lines_file_mapping[file] = visited_module.start_lines\n            # Add functions and their dependencies\n            if object_name in visited_module.functions and object_name not in self.functions:\n                self.functions[object_name] = visited_module.functions[object_name]\n                self.added_objects_file_mapping[object_name] = file\n                dependencies = visited_module.object_dependency_mapping.get(object_name, None)\n                if dependencies is not None:\n                    self.object_dependency_mapping[object_name] = dependencies\n                    for dep in dependencies:\n                        if dep not in self.global_nodes:\n                            self.added_objects_file_mapping[dep] = file\n                            self.functions[dep] = visited_module.global_nodes[dep]\n\n            # Add assignments and their dependencies\n            elif object_name in visited_module.assignments and object_name not in self.assignments:\n                self.assignments[object_name] = visited_module.assignments[object_name]\n                self.added_objects_file_mapping[object_name] = file\n                dependencies = visited_module.object_dependency_mapping.get(object_name, None)\n                if dependencies is not None:\n                    self.object_dependency_mapping[object_name] = dependencies\n                    for dep in dependencies:\n                        if dep not in self.global_nodes:\n                            self.added_objects_file_mapping[dep] = file\n                            self.assignments[dep] = visited_module.global_nodes[dep]\n\n        # Do not forget to re-assign all nodes after the merge\n        self.global_nodes = {**self.assignments, **self.classes, **self.functions}\n        # And restric dependencies to those nodes only\n        self._restrict_dependencies_to_known_entities()\n\n    def compute_relative_order(self, missing_dependencies: set) -> dict[str, int]:\n        \"\"\"Compute in which relative order the `missing_dependencies` should appear when the nodes are added to the final file that\n        will be created based on the modular.\n        \"\"\"\n        relative_order = {}\n        idx = 0\n\n        original_dependencies = []\n        other_files_dependencies = defaultdict(list)\n        for dep in tuple(missing_dependencies):\n            if dep in self.added_objects_file_mapping:\n                file = self.added_objects_file_mapping[dep]\n                other_files_dependencies[file].append(dep)\n            else:\n                original_dependencies.append(dep)\n        # Sort all lists according to the order in their respective file\n        all_dependencies = []\n        for file, dependencies in other_files_dependencies.items():\n            sorted_dependencies = sorted(dependencies, key=lambda x: self.start_lines_file_mapping[file][x])\n            all_dependencies += sorted_dependencies\n        all_dependencies += sorted(original_dependencies, key=lambda x: self.start_lines[x])\n\n        # Add all original node first, then merged ones (one file at a time)\n        for dep in all_dependencies:\n            relative_order[dep] = idx\n            idx += 1\n\n        return relative_order\n\n    def infer_new_model_name(self) -> dict:\n        \"\"\"Infer whether we are using a model name prefix different from the usual model name as defined from the filename.\n        This is useful e.g. when we define a new multi-modal model, and only the text part inherits from `LlamaModel`,\n        so we have something like:\n        ```python\n        class NewModelNameTextDecoderLayer(LlamaDecoderLayer):\n            pass\n        ```\n        with the `Text` prefix added to the model name.\n        However, in case of multiple prefix used, we raise a warning and use the most frequent prefix, to avoid parsing\n        the same file multiple times and inconsistencies in the objects added from dependencies.\n        If the new prefix collides with a prefix of another class in the file where we are importing from, then we also\n        raise a warning, and use the default prefix (model name) to avoid collisions in dependencies.\n        \"\"\"\n        prefix_model_name_mapping = defaultdict(Counter)\n        cased_default_name = get_cased_name(self.model_name)\n        # Iterate over all new classes to get modeling super classes\n        for class_name, class_node in self.classes.items():\n            modeling_bases = [\n                k.value.value for k in class_node.bases if k.value.value in self.model_specific_imported_objects\n            ]\n            if len(modeling_bases) > 1:\n                raise ValueError(\n                    f\"{class_name} was defined with more than 1 model-specific super class. This is unsupported. We found {*modeling_bases,}.\"\n                )\n            if len(modeling_bases) == 1:\n                filename = self.model_specific_imported_objects[modeling_bases[0]]\n                cased_model_name = cased_default_name  # the default name prefix\n                suffix = common_partial_suffix(class_name, modeling_bases[0])\n                if len(suffix) > 0 and suffix[0].isupper():\n                    cased_model_name = class_name.replace(suffix, \"\")\n                prefix_model_name_mapping[filename].update([cased_model_name])\n\n        # Check if we found multiple prefixes for some modeling files\n        final_name_mapping = {}\n        for file, prefixes_counter in prefix_model_name_mapping.items():\n            if len(prefixes_counter) > 1:\n                _, total = prefixes_counter.most_common(1)[0]\n                most_used_entities = [name for name, count in prefixes_counter.most_common() if count == total]\n                # if the default name is in the pool of equally used prefixes, use it, otherwise last encountered\n                final_name = cased_default_name if cased_default_name in most_used_entities else most_used_entities[-1]\n            else:\n                final_name = list(prefixes_counter)[0]\n            # Check if the prefix can be used without collisions in the names\n            old_cased_model_name = get_cased_name(file.split(\".\")[-2])\n            old_model_name_prefix = final_name.replace(cased_default_name, old_cased_model_name)\n            # Raise adequate warning depending on the situation\n            has_prefix_collision = f\"\\nclass {old_model_name_prefix}\" in get_module_source_from_name(file)\n            if final_name != cased_default_name and has_prefix_collision:\n                if len(prefixes_counter) > 1:\n                    logger.warning(\n                        f\"We detected multiple prefix names when inheriting from {file}: {*set(prefixes_counter),}. However, the \"\n                        f\"most used one, '{final_name}', is already present in the source file and will likely cause consistency \"\n                        f\"issues. For this reason we fallback to the default prefix '{cased_default_name}' when grabbing args \"\n                        \"and dependencies. Make sure to subclass the intermediate classes with the prefix you want (if different \"\n                        f\"from '{cased_default_name}') or use a single prefix in all the modular (best).\"\n                    )\n                else:\n                    logger.warning(\n                        f\"We detected the use of the new default prefix {final_name} when inheriting from {file}. However, it is \"\n                        \"already present in the source file and will likely cause consistency issues. For this reason we fallback \"\n                        f\"to the default prefix '{cased_default_name}' when grabbing args and dependencies. Make sure to subclass \"\n                        f\"the intermediate classes with the prefix you want (if different from '{cased_default_name}')\"\n                    )\n                final_name = cased_default_name\n            elif len(prefixes_counter) > 1:\n                logger.warning(\n                    f\"We detected multiple prefix names when inheriting from {file}: {*set(prefixes_counter),}. We will only \"\n                    f\"use the most used '{final_name}' prefix when grabbing args and dependencies. Make sure to subclass the \"\n                    f\"intermediate classes with the prefix you want (if different from '{final_name}') or use a single prefix \"\n                    \"in all the modular (best).\"\n                )\n            final_name_mapping[file] = get_lowercase_name(final_name)\n\n        # Check we are not missing imported files\n        for file in self.model_specific_modules.keys():\n            if file not in final_name_mapping.keys():\n                final_name_mapping[file] = self.model_name\n\n\n\nSUMMARY:\nThis is a Python code that appears to be a part of the `transformers` library, specifically handling model merging and renaming. Here's a concise summary:\n\nThe code defines several methods for merging models, renaming classes, and handling naming conflicts. It checks if multiple prefixes are used in a modeling file and attempts to use the most frequent or default prefix instead. If a collision is detected, it raises a warning and may use a fallback value.\n\nKey features include:\n\n* Merging models by checking common prefixes\n* Renaming classes to avoid naming conflicts\n* Fallback behavior for missing imported files\n\nThe code requires several external modules, including `tokenizers`, `utils`, and `text_utils`. Some custom functions, such as `get_cased_name` and `common_partial_suffix`, are also used."}


created_at: 2025-02-05 09:13:48
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py:1451-1490
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 1490, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py", "name": "check_dependencies_and_create_import_node", "start_line": 1451, "type": "function", "chroma:document": "CODE:\n\ndef check_dependencies_and_create_import_node(\n    file_type: str, new_dependencies: set[str], mapper: ModuleMapper, new_name: str\n) -> tuple[set[str], dict[str, cst.CSTNode]]:\n    \"\"\"Check that all class nodes in the `new_dependencies` belong to the correct `file_type`. If this is not the case,\n    we need to remove it from the dependencies, and create a new import to it instead.\n    This scenario may appear in the following case:\n    If a new class in the `modular_xxx.py` file does not belong to `type_xxx.py`, but is used somewhere in `other_type_xxx.py`\n    (e.g. as a type hint), but none of the visited files had a similar class, then it would be imported in `type_xxx.py` as\n    part of the standard dependency graph (because we never encountered an import towards this new class in any file).\n    For example imagine the following `modular.py`:\n    ```\n    from ..llama.modeling_llama import LlamaModel\n\n    class NewNameTextConfig(PretrainedConfig):\n        ...\n\n    class NewNameConfig(PretrainedConfig):\n        ...\n\n    class NewNameModel(LlamaModel):\n        config = NewNameConfig()\n        text_config = NewNameTextConfig()\n        ...\n    ```\n    then without the help of this function, `NewNameTextConfig` would be imported in the `modeling_newname.py` as well as\n    `configuration_newname.py`, because `modeling_llama.py` tells us to not import `NewNameConfig`, but has no\n    knowledge of `NewNameTextConfig`.\n    \"\"\"\n    class_dependencies = {dep for dep in new_dependencies if m.matches(mapper.global_nodes[dep], m.ClassDef())}\n    corrected_dependencies = new_dependencies.copy()\n    new_imports = {}\n    for class_name in class_dependencies:\n        class_file_type = find_file_type(class_name)\n        # In this case, we need to remove it from the dependencies and create a new import instead\n        if class_file_type != file_type:\n            corrected_dependencies.remove(class_name)\n            import_statement = f\"from .{class_file_type}_{new_name} import {class_name}\"\n            new_imports[class_name] = cst.parse_statement(import_statement)\n\n\n\nSUMMARY:\n**Function Summary: `check_dependencies_and_create_import_node`**\n\n* **Purpose**: Ensure that class nodes in `new_dependencies` belong to the correct file type. If not, remove it and create a new import statement.\n* **Key Implementation Details**:\n\t+ Filter class dependencies based on their file types using the `matches` method of a `ModuleMapper`.\n\t+ Create a dictionary `corrected_dependencies` with remaining class nodes that belong to the correct file type.\n\t+ Generate `import_statement` for each misclassified class node and store it in the `new_imports` dictionary.\n\n**Method Signatures and Interfaces**:\n\n* `ModuleMapper.matches()`: Matches a module with a specific node type (in this case, `ClassDef()`).\n* `find_file_type(class_name)`: Returns the file type of a given class name.\n* `cst.parse_statement(import_statement)`: Parses an import statement into a CST (Abstract Syntax Tree) Node.\n\n**Notable Dependencies and Requirements**:\n\n* `ModuleMapper`: A module mapper object that provides the necessary methods.\n* `find_file_type(class_name)` function: Not shown in this code snippet but presumably defined elsewhere in the project.\n* External dependencies:\n\t+ `cst` (Collection of Statements, Functions, Variables, etc.)"}


created_at: 2025-02-05 09:13:48
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py:1493-1571
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 1571, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py", "name": "get_class_node_and_dependencies", "start_line": 1493, "type": "function", "chroma:document": "CODE:\n\ndef get_class_node_and_dependencies(\n    modular_mapper: ModularFileMapper, class_name: str, node: cst.CSTNode, files: dict[str, dict]\n) -> tuple[dict, str, dict]:\n    \"\"\"Return a single class node (and all its dependency nodes), to be added to the `files`. It creates the new\n    class node based on the inherited classes if needed. Also returns any new imports of a new class defined in\n    the modular that we nay need.\n    \"\"\"\n    # An exception was already raised if this has len > 1\n    model_specific_bases = [\n        k.value.value for k in node.bases if k.value.value in modular_mapper.model_specific_imported_objects\n    ]\n    super_class = model_specific_bases[0] if len(model_specific_bases) == 1 else None\n\n    file_type = find_file_type(class_name)\n    file_to_update = files[file_type]\n    model_name = modular_mapper.model_name\n\n    # This is used to avoid adding objects to the dependencies graph if they will be imported already\n    imported_objects = modular_mapper.imported_objects_per_file[file_type]\n\n    # We need to replace the class node with the transformers (modeling file) super class node\n    if super_class is not None:\n        super_file_name = modular_mapper.model_specific_imported_objects[super_class]\n\n        # Get the mapper corresponding to the inherited class\n        mapper = modular_mapper.visited_modules[super_file_name]\n        # Rename the super class according to the exact same rule we used when renaming the whole module\n        renamer = modular_mapper.renamers[super_file_name]\n        renamed_super_class = preserve_case_replace(super_class, renamer.patterns, renamer.cased_new_name)\n\n        # Create the new class node\n        updated_node = replace_class_node(mapper, node, renamed_super_class, super_class)\n\n        # Grab all immediate dependencies of the new node\n        new_node_dependencies = augmented_dependencies_for_class_node(updated_node, mapper, imported_objects)\n\n        # At this point, if any class dependency is found, but belongs to another file, it means that we need to remove\n        # it from the dependencies, and add a new import of it instead\n        new_node_dependencies, new_imports = check_dependencies_and_create_import_node(\n            file_type, new_node_dependencies, mapper, model_name\n        )\n\n        # The node was modified -> look for all recursive dependencies of the new node\n        all_dependencies_to_add = find_all_dependencies(\n            dependency_mapping=mapper.class_dependency_mapping,\n            initial_dependencies=new_node_dependencies,\n            initial_checked_dependencies=set(file_to_update.keys()),\n        )\n\n        relative_dependency_order = mapper.compute_relative_order(all_dependencies_to_add)\n        nodes_to_add = {\n            dep: (relative_dependency_order[dep], mapper.global_nodes[dep]) for dep in all_dependencies_to_add\n        }\n\n    # No transformers (modeling file) super class, just check functions and assignments dependencies\n    else:\n        updated_node = node\n        # The node was NOT modified -> no need to look recursively for other class dependencies. Indeed, even if they are not\n        # already defined (which would mean a weird order of the code in the modular...), they will be in the future\n        all_dependencies_to_add = augmented_dependencies_for_class_node(updated_node, modular_mapper, imported_objects)\n\n        # At this point, if any class dependency is found, but belongs to another file, it means that we need to remove\n        # it from the dependencies, and add a new import of it instead\n        all_dependencies_to_add, new_imports = check_dependencies_and_create_import_node(\n            file_type, all_dependencies_to_add, modular_mapper, model_name\n        )\n\n        relative_dependency_order = modular_mapper.compute_relative_order(all_dependencies_to_add)\n        nodes_to_add = {\n            dep: (relative_dependency_order[dep], modular_mapper.global_nodes[dep])\n            for dep in all_dependencies_to_add\n            if dep not in file_to_update.keys()\n        }\n\n    # Add the class node itself to the nodes to add\n    class_idx = max(relative_dependency_order.values()) + 1 if len(relative_dependency_order) > 0 else 0\n    nodes_to_add[class_name] = (class_idx, updated_node)\n\n\n\nSUMMARY:\n**Summary: get_class_node_and_dependencies Function**\n\nThe `get_class_node_and_dependencies` function generates a single class node and its dependencies for a given modular file. It:\n\n1. **Creates the Class Node**: Builds the class node based on inherited classes, if needed, using the provided modular mappings.\n2. **Generates New Imports**: Finds new imports of classes defined in the modular that may be required.\n3. **Analyzes Dependencies**: Checks dependencies and creates import nodes if necessary, recursively finding all dependencies.\n4. **Orders Dependencies**: Computes relative order for the generated dependencies.\n\n**Key Function Details**\n\n* Returns a tuple containing the class node with its updated dependencies, file type, and new imports.\n* Uses modular mappings to rename super classes according to exact rules.\n\n**Notable Dependencies and Requirements**\n\n* `modular_mapper` object\n\t+ `model_specific_imported_objects`\n\t+ `imported_objects_per_file`\n\t+ `visited_modules`\n\t+ `renamers`\n* External dependencies:\n\t+ `find_file_type`\n\t+ `augmented_dependencies_for_class_node`\n\t+ `check_dependencies_and_create_import_node`\n\n**Interfaces and Dependencies**\n\n* Uses interfaces from `modular_mapper` object, including `mapper`, `renamer`, and `visited_modules`.\n* Returns nodes using a dictionary-like structure with relative dependency order."}


created_at: 2025-02-05 09:13:48
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py:1574-1625
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 1625, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py", "name": "create_modules", "start_line": 1574, "type": "function", "chroma:document": "CODE:\n\ndef create_modules(modular_mapper: ModularFileMapper) -> dict[str, cst.Module]:\n    \"\"\"Create all the new modules based on visiting the modular file. It replaces all classes as necesary.\"\"\"\n    files = defaultdict(dict)\n    current_file_indices = defaultdict(lambda: 0)\n\n    # For each class defined in modular, potentially replace the node and add it with its dependencies\n    for class_name, node in modular_mapper.classes.items():\n        nodes_to_add, file_type, new_imports = get_class_node_and_dependencies(modular_mapper, class_name, node, files)\n\n        # Add the new potential new imports that we may need to the `modular_mapper` variable\n        modular_mapper.imported_objects_per_file[file_type].update(new_imports.keys())\n        modular_mapper.imports.extend(list(new_imports.values()))\n\n        # Sort the nodes according to their relative order\n        nodes_to_add = sorted(nodes_to_add.items(), key=lambda x: x[1][0])\n        # Write all nodes to file\n        for dependency, (_, node) in nodes_to_add:\n            # This is used to keep certain variables at the beginning of the file\n            try:\n                # The -1000 is arbitrary -> just keep it bigger than the list\n                idx = -1000 + VARIABLES_AT_THE_BEGINNING.index(dependency)\n            except ValueError:\n                idx = current_file_indices[file_type]\n                current_file_indices[file_type] += 1\n            files[file_type][dependency] = {\"insert_idx\": idx, \"node\": node}\n\n    # Add the __all__ statement to files at the end\n    for file_type, node in modular_mapper.all_all_to_add.items():\n        idx = current_file_indices[file_type]\n        files[file_type][\"__all__\"] = {\"insert_idx\": idx, \"node\": node}\n\n    # Aggregate all the imports statements (we look for duplicates with the code_for_node, not the nodes themselves because\n    # they are wrapped in SimpleStatementLine or If which could have different newlines, blanks etc)\n    all_imports = modular_mapper.imports.copy()\n    all_imports_code = {modular_mapper.python_module.code_for_node(node).strip() for node in all_imports}\n    for file, mapper in modular_mapper.visited_modules.items():\n        new_imports = [\n            node for node in mapper.imports if mapper.python_module.code_for_node(node).strip() not in all_imports_code\n        ]\n        new_imports_code = {mapper.python_module.code_for_node(node).strip() for node in new_imports}\n        all_imports.extend(new_imports)\n        all_imports_code.update(new_imports_code)\n\n    # Find the correct imports, and write the new modules\n    for file, body in files.items():\n        new_body = [k[1][\"node\"] for k in sorted(body.items(), key=lambda x: x[1][\"insert_idx\"])]\n        needed_imports = get_needed_imports(body, all_imports)\n        full_module = needed_imports + new_body\n        new_module = cst.Module(body=full_module, header=modular_mapper.python_module.header)\n        files[file] = new_module\n\n\n\nSUMMARY:\n**Summary: Create Modules Function**\n\n*   **Purpose:** Creates new modules based on visiting modular files by replacing classes and adding dependencies.\n\n**Main Purpose and Functionality:**\n\nThe `create_modules` function takes a `ModularFileMapper` object as input and returns a dictionary of new modules. It achieves the following key functionality:\n\n1.  Replaces classes in each class definition with updated nodes and added dependencies.\n2.  Adds potential new imports to the `modular_mapper.imported_objects_per_file` map.\n3.  Sorts nodes according to their relative order for file writing.\n4.  Writes all nodes to files, keeping certain variables at the beginning of the file.\n5.  Aggregates and finds correct imports to write full modules.\n\n**Key Implementation Details:**\n\n*   Iterates through each class definition in `modular_mapper.classes`.\n*   Uses `get_class_node_and_dependencies` function to get updated nodes and dependencies for each class.\n*   Updates `imported_objects_per_file` map with new imports.\n*   Sorts nodes for file writing using a custom sorting key.\n*   Creates a new module by combining needed imports and body.\n\n**Notable Dependencies or Requirements:**\n\n*   `ModularFileMapper` object\n*   `get_class_node_and_dependencies` function\n*   `VARIABLES_AT_THE_BEGINNING` list\n*   `cst.Module` class\n\n**Additional Information:**\n\nThis code appears to be part of a larger system designed to work with modular files and their dependencies. The function is closely tied to the `ModularFileMapper` object, which seems to provide necessary data for creating new modules.\n\nThe code uses various external libraries or classes (e.g., `cst.Module`, `VARIABLES_AT_THE_BEGINNING`) that are not defined in this snippet."}


created_at: 2025-02-05 09:13:48
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py:1628-1656
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 1656, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py", "name": "convert_modular_file", "start_line": 1628, "type": "function", "chroma:document": "CODE:\n\ndef convert_modular_file(modular_file):\n    pattern = re.search(r\"modular_(.*)(?=\\.py$)\", modular_file)\n    output = {}\n    if pattern is not None:\n        model_name = pattern.groups()[0]\n        # Parse the Python file\n        with open(modular_file, \"r\", encoding=\"utf-8\") as file:\n            code = file.read()\n        module = cst.parse_module(code)\n        wrapper = MetadataWrapper(module)\n        cst_transformers = ModularFileMapper(module, model_name)\n        wrapper.visit(cst_transformers)\n        for file, module in create_modules(cst_transformers).items():\n            if module != {}:\n                # Get relative path starting from src/transformers/\n                relative_path = re.search(\n                    r\"(src/transformers/.*|examples/.*)\", os.path.abspath(modular_file).replace(\"\\\\\", \"/\")\n                ).group(1)\n\n                header = AUTO_GENERATED_MESSAGE.format(\n                    relative_path=relative_path, short_name=os.path.basename(relative_path)\n                )\n                ruffed_code = run_ruff(header + module.code, True)\n                formatted_code = run_ruff(ruffed_code, False)\n                output[file] = [formatted_code, ruffed_code]\n        return output\n    else:\n        print(f\"modular pattern not found in {modular_file}, exiting\")\n\n\nSUMMARY:\n**Function Summary:**\n\n`convert_modular_file(modular_file)`\n\nConverts a modular Python file to conform to formatting standards.\n\n**Key Implementation Details:**\n\n- Parses the input modular file using regular expressions and CST (Common SyntaxTree) parser.\n- Applies transformations using `MetadataWrapper` and `ModularFileMapper`.\n- Formats code using Ruff library on two rounds of review (initial and final).\n\n**Notable Dependencies and Requirements:**\n\n- `re`, `os`, `cst`, and `Ruff` libraries for parsing and formatting purposes.\n- Custom classes (`MetadataWrapper`, `ModularFileMapper`) are used to implement transformation logic."}


created_at: 2025-02-05 09:13:48
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py:1659-1679
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 1679, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/modular_model_converter.py", "name": "save_modeling_file", "start_line": 1659, "type": "function", "chroma:document": "CODE:\n\ndef save_modeling_file(modular_file, converted_file):\n    for file_type in converted_file.keys():\n        file_name_prefix = file_type.split(\"*\")[0]\n        file_name_suffix = file_type.split(\"*\")[-1] if \"*\" in file_type else \"\"\n        new_file_name = modular_file.replace(\"modular_\", f\"{file_name_prefix}_\").replace(\n            \".py\", f\"{file_name_suffix}.py\"\n        )\n        non_comment_lines = len(\n            [line for line in converted_file[file_type][0].strip().split(\"\\n\") if not line.strip().startswith(\"#\")]\n        )\n        if len(converted_file[file_type][0].strip()) > 0 and non_comment_lines > 0:\n            with open(new_file_name, \"w\", encoding=\"utf-8\") as f:\n                f.write(converted_file[file_type][0])\n        else:\n            non_comment_lines = len(\n                [line for line in converted_file[file_type][0].strip().split(\"\\n\") if not line.strip().startswith(\"#\")]\n            )\n            if len(converted_file[file_type][1].strip()) > 0 and non_comment_lines > 0:\n                logger.warning(\"The modeling code contains errors, it's written without formatting\")\n                with open(new_file_name, \"w\", encoding=\"utf-8\") as f:\n\n\nSUMMARY:\n**Summary:**\n\nSave Modeling File Functionality\n==============================\n\n**Purpose and Functionality**\n-----------------------------\n\nThis function `save_modeling_file` takes two inputs, `modular_file` and `converted_file`, and saves the contents of `converted_file` to new files with modified names based on their original file types.\n\n**Key Implementation Details:**\n\n*   Iterates over each file type in `converted_file` keys.\n*   Creates a new file name by replacing \"modular_\" prefix and adding `.py` extension suffix based on file type.\n*   Checks if the converted file lines contain any non-comment content and writes it to the new file.\n\n**Notable Dependencies and Requirements:**\n\n*   Uses `logger` for warnings during file writing.\n*   External dependency on Python's built-in `logging` module (referenced as \"logger\").\n*   Does not inherit or implement any specific interfaces."}


created_at: 2025-02-05 09:13:48
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_build.py:32-37
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 37, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_build.py", "name": "test_custom_files_are_present", "start_line": 32, "type": "function", "chroma:document": "CODE:\n\ndef test_custom_files_are_present(transformers_path):\n    # Test all the extensions added in the setup\n    for file in FILES_TO_FIND:\n        if not (transformers_path / file).exists():\n            return False\n\n\nSUMMARY:\n**Summary:**\n\n`test_custom_files_are_present`: Verifies that custom files exist at a specified `transformers_path`.\n\n**Key Implementation Details:**\n\n* Iterates through `FILES_TO_FIND` and checks existence of each file using `exists()`.\n* Returns `True` if all files are present, `False` otherwise.\n\n**Notable Dependencies/Requirements:**\n\n* `FILES_TO_FIND`: a list of filenames to check (assumed to be defined elsewhere).\n* External dependencies not mentioned.\n* `transformers_path` is assumed to be an existing directory path."}


created_at: 2025-02-05 09:13:48
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/update_tiny_models.py:37-60
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 60, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/update_tiny_models.py", "name": "get_all_model_names", "start_line": 37, "type": "function", "chroma:document": "CODE:\n\ndef get_all_model_names():\n    model_names = set()\n    # Each auto modeling files contains multiple mappings. Let's get them in a dynamic way.\n    for module_name in [\"modeling_auto\", \"modeling_tf_auto\", \"modeling_flax_auto\"]:\n        module = getattr(transformers.models.auto, module_name, None)\n        if module is None:\n            continue\n        # all mappings in a single auto modeling file\n        mapping_names = [\n            x\n            for x in dir(module)\n            if x.endswith(\"_MAPPING_NAMES\")\n            and (x.startswith(\"MODEL_\") or x.startswith(\"TF_MODEL_\") or x.startswith(\"FLAX_MODEL_\"))\n        ]\n        for name in mapping_names:\n            mapping = getattr(module, name)\n            if mapping is not None:\n                for v in mapping.values():\n                    if isinstance(v, (list, tuple)):\n                        model_names.update(v)\n                    elif isinstance(v, str):\n                        model_names.add(v)\n\n\n\nSUMMARY:\n**Function Summary: get_all_model_names()**\n\nThe `get_all_model_names()` function extracts and returns a set of unique model names from auto-modeling files in the `transformers.models.auto` namespace.\n\n**Key Implementation Details:**\n\n* Iterates through three auto-modeling files (`modeling_auto`, `modeling_tf_auto`, `modeling_flax_auto`) to gather model mappings.\n* Uses string matching (`str.endswith()`) and object existence checks (`isinstance()` and `is None`) to identify relevant model names.\n\n**Notable Dependencies/Requirements:**\n\n* `transformers.models.auto` namespace\n* No external dependencies listed."}


created_at: 2025-02-05 09:13:48
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/update_tiny_models.py:63-81
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 81, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/update_tiny_models.py", "name": "get_tiny_model_names_from_repo", "start_line": 63, "type": "function", "chroma:document": "CODE:\n\ndef get_tiny_model_names_from_repo():\n    # All model names defined in auto mappings\n    model_names = set(get_all_model_names())\n\n    with open(\"tests/utils/tiny_model_summary.json\") as fp:\n        tiny_model_info = json.load(fp)\n    tiny_models_names = set()\n    for model_base_name in tiny_model_info:\n        tiny_models_names.update(tiny_model_info[model_base_name][\"model_classes\"])\n\n    # Remove a tiny model name if one of its framework implementation hasn't yet a tiny version on the Hub.\n    not_on_hub = model_names.difference(tiny_models_names)\n    for model_name in copy.copy(tiny_models_names):\n        if not model_name.startswith(\"TF\") and f\"TF{model_name}\" in not_on_hub:\n            tiny_models_names.remove(model_name)\n        elif model_name.startswith(\"TF\") and model_name[2:] in not_on_hub:\n            tiny_models_names.remove(model_name)\n\n\n\nSUMMARY:\n**Concise Summary: `get_tiny_model_names_from_repo`**\n\n**Purpose:** Retrieves the names of tiny models defined in auto-mappings and filtering out those not on the Hub.\n\n**Key Implementation Details:**\n\n* Uses `get_all_model_names()` to get all model names, then filters using a set operation (`difference` and `update`) between model names and their implementations on the Hub (`tiny_models_names`).\n* Checks for the presence of \"TF\" prefix in a specific format (e.g., \"TFx[model_name]\"), removing models that don't have a matching tiny version.\n* Uses JSON data from \"tests/utils/tiny_model_summary.json\" to infer available model classes.\n\n**Dependencies/Requirements:**\n\n* `get_all_model_names()`\n* `json` module\n* `copy` module\n* External dependency on \"tests/utils/tiny_model_summary.json\" file."}


created_at: 2025-02-05 09:13:48
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/update_tiny_models.py:84-169
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 169, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/update_tiny_models.py", "name": "get_tiny_model_summary_from_hub", "start_line": 84, "type": "function", "chroma:document": "CODE:\n\ndef get_tiny_model_summary_from_hub(output_path):\n    special_models = COMPOSITE_MODELS.values()\n\n    # All tiny model base names on Hub\n    model_names = get_all_model_names()\n    models = hf_api.list_models(\n        filter=ModelFilter(\n            author=\"hf-internal-testing\",\n        )\n    )\n    _models = set()\n    for x in models:\n        model = x.id\n        org, model = model.split(\"/\")\n        if not model.startswith(\"tiny-random-\"):\n            continue\n        model = model.replace(\"tiny-random-\", \"\")\n        if not model[0].isupper():\n            continue\n        if model not in model_names and model not in special_models:\n            continue\n        _models.add(model)\n\n    models = sorted(_models)\n    # All tiny model names on Hub\n    summary = {}\n    for model in models:\n        repo_id = f\"hf-internal-testing/tiny-random-{model}\"\n        model = model.split(\"-\")[0]\n        try:\n            repo_info = hf_api.repo_info(repo_id)\n            content = {\n                \"tokenizer_classes\": set(),\n                \"processor_classes\": set(),\n                \"model_classes\": set(),\n                \"sha\": repo_info.sha,\n            }\n        except Exception:\n            continue\n        try:\n            time.sleep(1)\n            tokenizer_fast = AutoTokenizer.from_pretrained(repo_id)\n            content[\"tokenizer_classes\"].add(tokenizer_fast.__class__.__name__)\n        except Exception:\n            pass\n        try:\n            time.sleep(1)\n            tokenizer_slow = AutoTokenizer.from_pretrained(repo_id, use_fast=False)\n            content[\"tokenizer_classes\"].add(tokenizer_slow.__class__.__name__)\n        except Exception:\n            pass\n        try:\n            time.sleep(1)\n            img_p = AutoImageProcessor.from_pretrained(repo_id)\n            content[\"processor_classes\"].add(img_p.__class__.__name__)\n        except Exception:\n            pass\n        try:\n            time.sleep(1)\n            feat_p = AutoFeatureExtractor.from_pretrained(repo_id)\n            if not isinstance(feat_p, BaseImageProcessor):\n                content[\"processor_classes\"].add(feat_p.__class__.__name__)\n        except Exception:\n            pass\n        try:\n            time.sleep(1)\n            model_class = getattr(transformers, model)\n            m = model_class.from_pretrained(repo_id)\n            content[\"model_classes\"].add(m.__class__.__name__)\n        except Exception:\n            pass\n        try:\n            time.sleep(1)\n            model_class = getattr(transformers, f\"TF{model}\")\n            m = model_class.from_pretrained(repo_id)\n            content[\"model_classes\"].add(m.__class__.__name__)\n        except Exception:\n            pass\n\n        content[\"tokenizer_classes\"] = sorted(content[\"tokenizer_classes\"])\n        content[\"processor_classes\"] = sorted(content[\"processor_classes\"])\n        content[\"model_classes\"] = sorted(content[\"model_classes\"])\n\n        summary[model] = content\n        with open(os.path.join(output_path, \"hub_tiny_model_summary.json\"), \"w\") as fp:\n\n\nSUMMARY:\n**Function Summary: `get_tiny_model_summary_from_hub`**\n\n**Purpose:** Fetches summaries for tiny models from the Hugging Face model hub.\n\n**Main Purpose**: The function retrieves a list of tiny models from the Hugging Face model hub, constructs their metadata (e.g., tokenizer, processor, and model classes), and stores it in a JSON file.\n\n**Key Implementation Details:**\n\n*   Filters models by author and namespace.\n*   Iterates through each model, retrieving its metadata using the `hf_api` interface.\n*   Handles cases where model metadata is not available, waiting 1 second between attempts.\n*   Extracts tokenizer classes from slow and fast versions of the AutoTokenizer.\n*   Extracts processor classes from image processors (AutoImageProcessor).\n*   Extracts model classes from transformers module using string-based lookup.\n\n**Notable Dependencies:**\n\n*   `hf_api`: Hugging Face API interface for interacting with the model hub.\n*   `transformers`: Hugging Face transformer library for loading pre-trained models.\n*   ` AutoTokenizer`, `AutoImageProcessor`, and `AutoFeatureExtractor` classes from transformers module.\n\n**External Requirements:**\n\n*   Requires access to Hugging Face API subscription or trial account."}


created_at: 2025-02-05 09:13:48
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py:371-396
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 396, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py", "name": "check_missing_backends", "start_line": 371, "type": "function", "chroma:document": "CODE:\n\ndef check_missing_backends():\n    \"\"\"\n    Checks if all backends are installed (otherwise the check of this script is incomplete). Will error in the CI if\n    that's not the case but only throw a warning for users running this.\n    \"\"\"\n    missing_backends = []\n    if not is_torch_available():\n        missing_backends.append(\"PyTorch\")\n    if not is_tf_available():\n        missing_backends.append(\"TensorFlow\")\n    if not is_flax_available():\n        missing_backends.append(\"Flax\")\n    if len(missing_backends) > 0:\n        missing = \", \".join(missing_backends)\n        if os.getenv(\"TRANSFORMERS_IS_CI\", \"\").upper() in ENV_VARS_TRUE_VALUES:\n            raise Exception(\n                \"Full repo consistency checks require all backends to be installed (with `pip install -e '.[dev]'` in the \"\n                f\"Transformers repo, the following are missing: {missing}.\"\n            )\n        else:\n            warnings.warn(\n                \"Full repo consistency checks require all backends to be installed (with `pip install -e '.[dev]'` in the \"\n                f\"Transformers repo, the following are missing: {missing}. While it's probably fine as long as you \"\n                \"didn't make any change in one of those backends modeling files, you should probably execute the \"\n                \"command above to be on the safe side.\"\n\n\nSUMMARY:\n**Summary**\n\nThe `check_missing_backends` function checks if all required backends (PyTorch, TensorFlow, and Flax) are installed. It:\n\n* Compares availability of each backend using `is_`:tch_available`, `is_tf_available`, and `is_flax_available`\n* If any backend is missing, it appends the name to `missing_backends` list\n* Raises an exception in CI if all backends are not installed or a warning for users otherwise\n* Provides instructions on how to install missing backends using pip\n\n**Key Implementation Details**\n\n* Checks availability of three backends using separate functions (`is_torch_available`, `is_tf_available`, and `is_flax_available`)\n* Uses an exception to raise error in CI if all backends are not installed\n* Uses a warning to notify users about the missing backends\n* Provides a pip installation command to install missing backends\n\n**Notable Dependencies/Requirements**\n\n* `pytorch`: PyTorch backend\n* `tensorflow`: TensorFlow backend\n* `flax`: Flax backend\n* `transformers`: Transformers repository\n* `os`: Operating system dependency for environment variable access"}


created_at: 2025-02-05 09:13:48
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py:399-420
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 420, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py", "name": "check_model_list", "start_line": 399, "type": "function", "chroma:document": "CODE:\n\ndef check_model_list():\n    \"\"\"\n    Checks the model listed as subfolders of `models` match the models available in `transformers.models`.\n    \"\"\"\n    # Get the models from the directory structure of `src/transformers/models/`\n    models_dir = os.path.join(PATH_TO_TRANSFORMERS, \"models\")\n    _models = []\n    for model in os.listdir(models_dir):\n        if model == \"deprecated\":\n            continue\n        model_dir = os.path.join(models_dir, model)\n        if os.path.isdir(model_dir) and \"__init__.py\" in os.listdir(model_dir):\n            _models.append(model)\n\n    # Get the models in the submodule `transformers.models`\n    models = [model for model in dir(transformers.models) if not model.startswith(\"__\")]\n\n    missing_models = sorted(set(_models).difference(models))\n    if missing_models:\n        raise Exception(\n            f\"The following models should be included in {models_dir}/__init__.py: {','.join(missing_models)}.\"\n\n\nSUMMARY:\n**Function Summary:** `check_model_list`\n\n**Primary Purpose:** Validates that the models listed as subfolders of `src/transformers/models` match the available models in `transformers.models`.\n\n**Key Implementation Details:**\n\n*   Loads all non-deprecated subfolder models from `src/transformers/models/` into `_models`.\n*   Compares the list of models in `_models` against the models listed in `transformers.models`.\n*   Raising exception when there are missing models that should be included.\n\n**Notable Dependencies:**\n\n*   `os.path`: for path manipulation (e.g., joining paths).\n*   `PATH_TO_TRANSFORMERS`: a defined constant referencing to the transformers directory."}


created_at: 2025-02-05 09:13:48
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py:425-454
encoding: FLOAT32
metadata:
{"docstring": "# If some modeling modules should be ignored for all checks, they should be added in the nested list\n# _ignore_modules of this function.", "end_line": 454, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py", "name": "get_model_modules", "start_line": 425, "type": "function", "chroma:document": "CODE:\n# _ignore_modules of this function.\ndef get_model_modules() -> List[str]:\n    \"\"\"Get all the model modules inside the transformers library (except deprecated models).\"\"\"\n    _ignore_modules = [\n        \"modeling_auto\",\n        \"modeling_encoder_decoder\",\n        \"modeling_marian\",\n        \"modeling_retribert\",\n        \"modeling_flax_auto\",\n        \"modeling_flax_encoder_decoder\",\n        \"modeling_speech_encoder_decoder\",\n        \"modeling_flax_speech_encoder_decoder\",\n        \"modeling_flax_vision_encoder_decoder\",\n        \"modeling_timm_backbone\",\n        \"modeling_tf_auto\",\n        \"modeling_tf_encoder_decoder\",\n        \"modeling_tf_vision_encoder_decoder\",\n        \"modeling_vision_encoder_decoder\",\n    ]\n    modules = []\n    for model in dir(transformers.models):\n        # There are some magic dunder attributes in the dir, we ignore them\n        if model == \"deprecated\" or model.startswith(\"__\"):\n            continue\n\n        model_module = getattr(transformers.models, model)\n        for submodule in dir(model_module):\n            if submodule.startswith(\"modeling\") and submodule not in _ignore_modules:\n                modeling_module = getattr(model_module, submodule)\n                modules.append(modeling_module)\n\n\nSUMMARY:\n**Summary**\n\nThis function, `get_model_modules`, retrieves all model modules from the `transformers` library, excluding deprecated and specific \"modeling\" submodules. The primary purpose is to gather a list of models for use in a larger application.\n\n**Key Implementation Details:**\n\n*   ignores ignored module names (`_ignore_modules`) and magic dunder attributes in the directory listing.\n*   uses `getattr()` to dynamically access individual model modules and their \"modeling\" submodules.\n*   excludes deprecated model names detected as \"__deprecated__\".\n\n**Notable Dependencies/Requirements:**\n\n*   The `transformers` library, specifically its `models` module.\n\n**Method Signatures Interfaces:**\n\n*   `get_model_modules() -> List[str]`: Returns a list of model modules in the transformed format, excluding obsolete ones."}


created_at: 2025-02-05 09:13:48
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py:457-478
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 478, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py", "name": "get_models", "start_line": 457, "type": "function", "chroma:document": "CODE:\n\ndef get_models(module: types.ModuleType, include_pretrained: bool = False) -> List[Tuple[str, type]]:\n    \"\"\"\n    Get the objects in a module that are models.\n\n    Args:\n        module (`types.ModuleType`):\n            The module from which we are extracting models.\n        include_pretrained (`bool`, *optional*, defaults to `False`):\n            Whether or not to include the `PreTrainedModel` subclass (like `BertPreTrainedModel`) or not.\n\n    Returns:\n        List[Tuple[str, type]]: List of models as tuples (class name, actual class).\n    \"\"\"\n    models = []\n    model_classes = (transformers.PreTrainedModel, transformers.TFPreTrainedModel, transformers.FlaxPreTrainedModel)\n    for attr_name in dir(module):\n        if not include_pretrained and (\"Pretrained\" in attr_name or \"PreTrained\" in attr_name):\n            continue\n        attr = getattr(module, attr_name)\n        if isinstance(attr, type) and issubclass(attr, model_classes) and attr.__module__ == module.__name__:\n            models.append((attr_name, attr))\n\n\nSUMMARY:\nHere's a concise and focused summary of the given function:\n\n### Function Summary: `get_models`\n\n**Purpose:** Extract objects from a module that are models.\n\n**Primary Purpose:** Return a list of model classes as tuples (class name, actual class) based on their inheritance relation to predefined model classes.\n\n**Key Implementation Details:**\n\n* Iterates over attributes in the given module.\n* Filters out attributes whose names contain keywords related to `PreTrainedModel` subclass (excluding them by default).\n* Checks if each attribute is a type and inherits from defined model classes using multiple inheritance.\n* Appends validated attributes to the result list.\n\n**Notable Dependencies/Requirements:**\n\n* `transformers`: Module containing definition of predefined model classes.\n* `types`: Module for accessing built-in types like `ModuleType`.\n\n### Return Value:\n\nA list of tuples (class name, actual class) representing extracted model objects."}


created_at: 2025-02-05 09:13:48
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py:481-492
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 492, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py", "name": "is_building_block", "start_line": 481, "type": "function", "chroma:document": "CODE:\n\ndef is_building_block(model: str) -> bool:\n    \"\"\"\n    Returns `True` if a model is a building block part of a bigger model.\n    \"\"\"\n    if model.endswith(\"Wrapper\"):\n        return True\n    if model.endswith(\"Encoder\"):\n        return True\n    if model.endswith(\"Decoder\"):\n        return True\n    if model.endswith(\"Prenet\"):\n\n\nSUMMARY:\nHere is a concise summary of the function:\n\n**Function Summary**\n\n`is_building_block(model: str) -> bool`\n\nChecks if a machine learning model is a building block part of a larger model. It returns `True` for models ending with \"Wrapper\", \"Encoder\", \"Decoder\", or \"Prenet\".\n\n**Key Implementation Details**\n\n* Uses string suffix matching (`str.endswith()`)\n* Only checks for specific model prefixes (no regular expression)\n\n**Dependencies and Interfaces Noted**\n\n* No dependencies are mentioned in the function code.\n* Inherited or implemented interfaces not applicable.\n\nThis summary is concise, accurate, and focused on providing essential information about the function's purpose and implementation details."}


created_at: 2025-02-05 09:13:48
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py:495-499
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 499, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py", "name": "is_a_private_model", "start_line": 495, "type": "function", "chroma:document": "CODE:\n\ndef is_a_private_model(model: str) -> bool:\n    \"\"\"Returns `True` if the model should not be in the main init.\"\"\"\n    if model in PRIVATE_MODELS:\n        return True\n\n\nSUMMARY:\n**Summary**\n\nDetermines whether a given model should exclude itself from the main initialization.\n\n**Functionality**\n\n- Checks if the model is in the `PRIVATE Modes` list, returning `True` if it is.\n- Falls back to calling `is_building_block` with the model as an argument to determine its behavior.\n\n**Key Implementation Details**\n\n- Method signature: `def is_a_private_model(model: str) -> bool`\n- Calls `PRIVATE_MODELS` set to determine private models\n- Calls `is_building_block` method on a given model\n\n**Notable Dependencies or Requirements**\n\n- Uses the `PRIVATE.Models` set, which is not defined in this snippet. \n  - References related class 'PrivateModels'\n  \nNote: No external dependencies are listed as this function only references internal sets and methods."}


created_at: 2025-02-05 09:13:48
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py:502-514
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 514, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py", "name": "check_models_are_in_init", "start_line": 502, "type": "function", "chroma:document": "CODE:\n\ndef check_models_are_in_init():\n    \"\"\"Checks all models defined in the library are in the main init.\"\"\"\n    models_not_in_init = []\n    dir_transformers = dir(transformers)\n    for module in get_model_modules():\n        models_not_in_init += [\n            model[0] for model in get_models(module, include_pretrained=True) if model[0] not in dir_transformers\n        ]\n\n    # Remove private models\n    models_not_in_init = [model for model in models_not_in_init if not is_a_private_model(model)]\n    if len(models_not_in_init) > 0:\n\n\nSUMMARY:\n**Function Summary**\n\n`check_models_are_in_init()`\n=====================\n\nChecks if defined model libraries are deployed in the main application initialization.\n\n**Primary Purpose**\n\nVerifies that all models imported from external modules are included in the main `init.py` file of the library.\n\n**Key Implementation Details**\n\n* Iterates over model modules using `get_model_modules()` and collects models not present in `transformers` directory\n* Filters out private models using `is_a_private_model()`\n* Raises an exception if any models are missing from main init\n\n**Notable Dependencies and Requirements**\n\n* `transformers`: used to define the expected models directory\n* `get_model_modules()`, `get_models()`, and `is_a_private_model()`: functions not shown in this code snippet, but assumed to be defined elsewhere in the library."}


created_at: 2025-02-05 09:13:48
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py:519-555
encoding: FLOAT32
metadata:
{"docstring": "# If some test_modeling files should be ignored when checking models are all tested, they should be added in the\n# nested list _ignore_files of this function.", "end_line": 555, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py", "name": "get_model_test_files", "start_line": 519, "type": "function", "chroma:document": "CODE:\n# nested list _ignore_files of this function.\ndef get_model_test_files() -> List[str]:\n    \"\"\"\n    Get the model test files.\n\n    Returns:\n        `List[str]`: The list of test files. The returned files will NOT contain the `tests` (i.e. `PATH_TO_TESTS`\n        defined in this script). They will be considered as paths relative to `tests`. A caller has to use\n        `os.path.join(PATH_TO_TESTS, ...)` to access the files.\n    \"\"\"\n\n    _ignore_files = [\n        \"test_modeling_common\",\n        \"test_modeling_encoder_decoder\",\n        \"test_modeling_flax_encoder_decoder\",\n        \"test_modeling_flax_speech_encoder_decoder\",\n        \"test_modeling_marian\",\n        \"test_modeling_tf_common\",\n        \"test_modeling_tf_encoder_decoder\",\n    ]\n    test_files = []\n    model_test_root = os.path.join(PATH_TO_TESTS, \"models\")\n    model_test_dirs = []\n    for x in os.listdir(model_test_root):\n        x = os.path.join(model_test_root, x)\n        if os.path.isdir(x):\n            model_test_dirs.append(x)\n\n    for target_dir in [PATH_TO_TESTS] + model_test_dirs:\n        for file_or_dir in os.listdir(target_dir):\n            path = os.path.join(target_dir, file_or_dir)\n            if os.path.isfile(path):\n                filename = os.path.split(path)[-1]\n                if \"test_modeling\" in filename and os.path.splitext(filename)[0] not in _ignore_files:\n                    file = os.path.join(*path.split(os.sep)[1:])\n                    test_files.append(file)\n\n\n\nSUMMARY:\n**Function Summary:**\n\n`get_model_test_files()` \n\n Retrieves and returns a list of test files (\"test_modeling\" files) within the `models` directory under the test root. Excludes ignored files from specified `_ignore_files`.\n\n**Key Details:**\n\n* Ignored files are specified in `_ignore_files`.\n* Files with \"test_modeling\" in their filename, excluding those in `_ignore_files`, are considered.\n* Files are joined using `os.path.join()` to construct absolute paths.\n\n**Notable Dependencies:**\n\n* `os`: for file system interactions\n* `PATH_TO_TESTS`: script-defined constant for test root path"}


created_at: 2025-02-05 09:13:48
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py:560-583
encoding: FLOAT32
metadata:
{"docstring": "# This is a bit hacky but I didn't find a way to import the test_file as a module and read inside the tester class\n# for the all_model_classes variable.", "end_line": 583, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py", "name": "find_tested_models", "start_line": 560, "type": "function", "chroma:document": "CODE:\n# for the all_model_classes variable.\ndef find_tested_models(test_file: str) -> List[str]:\n    \"\"\"\n    Parse the content of test_file to detect what's in `all_model_classes`. This detects the models that inherit from\n    the common test class.\n\n    Args:\n        test_file (`str`): The path to the test file to check\n\n    Returns:\n        `List[str]`: The list of models tested in that file.\n    \"\"\"\n    with open(os.path.join(PATH_TO_TESTS, test_file), \"r\", encoding=\"utf-8\", newline=\"\\n\") as f:\n        content = f.read()\n    all_models = re.findall(r\"all_model_classes\\s+=\\s+\\(\\s*\\(([^\\)]*)\\)\", content)\n    # Check with one less parenthesis as well\n    all_models += re.findall(r\"all_model_classes\\s+=\\s+\\(([^\\)]*)\\)\", content)\n    if len(all_models) > 0:\n        model_tested = []\n        for entry in all_models:\n            for line in entry.split(\",\"):\n                name = line.strip()\n                if len(name) > 0:\n                    model_tested.append(name)\n\n\nSUMMARY:\n**Function Summary**\n\n`find_tested_models(test_file: str) -> List[str]`\n\nDetects models tested in a given test file by parsing its content, specifically extracting `all_model_classes`.\n\n**Key Functionality**\n\n* Reads the content of a test file\n* Uses regular expressions to extract `all_model_classes` patterns\n* Parses pattern entries to extract model names\n\n**Implementation Details**\n\n* Method signatures: `re.findall` with two pattern strings for `all_model_classes`\n* Algorithm: Combines multiple pattern findings into a single list using `.append`\n\n**Notable Dependencies/Requirements**\n\n* `os.path.join` and `open` for file operations\n* `re` module for regular expression matching\n\n**No notable inherited or implemented interfaces.**"}


created_at: 2025-02-05 09:13:48
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py:586-592
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 592, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py", "name": "should_be_tested", "start_line": 586, "type": "function", "chroma:document": "CODE:\n\ndef should_be_tested(model_name: str) -> bool:\n    \"\"\"\n    Whether or not a model should be tested.\n    \"\"\"\n    if model_name in IGNORE_NON_TESTED:\n        return False\n\n\nSUMMARY:\n**Function Summary**\n\n### `should_be_tested`\n\nDetermines whether a model with the given name should be tested.\n\n**Purpose**: Checks if a model's existence in an IGNORE_NON_TESTED list or if it's a building block model to exclude testing.\n\n**Key Implementation Details**:\n\n* Returns `False` for models in `IGNORE NON_TESTED`.\n* Excludes models with this method signature from being tested.\n* References `is_building_block(model_name)` using the exact name of the function.\n* Inherently uses an external list (`IGNORE_NON_TESTED`) without defining it within this code snippet.\n\n**Notable Dependencies/Requirements**: \n\n* `IGNORE_NON_TESTED` external list or constant."}


created_at: 2025-02-05 09:13:48
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py:595-625
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 625, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py", "name": "check_models_are_tested", "start_line": 595, "type": "function", "chroma:document": "CODE:\n\ndef check_models_are_tested(module: types.ModuleType, test_file: str) -> List[str]:\n    \"\"\"Check models defined in a module are all tested in a given file.\n\n    Args:\n        module (`types.ModuleType`): The module in which we get the models.\n        test_file (`str`): The path to the file where the module is tested.\n\n    Returns:\n        `List[str]`: The list of error messages corresponding to models not tested.\n    \"\"\"\n    # XxxPreTrainedModel are not tested\n    defined_models = get_models(module)\n    tested_models = find_tested_models(test_file)\n    if tested_models is None:\n        if test_file.replace(os.path.sep, \"/\") in TEST_FILES_WITH_NO_COMMON_TESTS:\n            return\n        return [\n            f\"{test_file} should define `all_model_classes` to apply common tests to the models it tests. \"\n            + \"If this intentional, add the test filename to `TEST_FILES_WITH_NO_COMMON_TESTS` in the file \"\n            + \"`utils/check_repo.py`.\"\n        ]\n    failures = []\n    for model_name, _ in defined_models:\n        if model_name not in tested_models and should_be_tested(model_name):\n            failures.append(\n                f\"{model_name} is defined in {module.__name__} but is not tested in \"\n                + f\"{os.path.join(PATH_TO_TESTS, test_file)}. Add it to the all_model_classes in that file.\"\n                + \"If common tests should not applied to that model, add its name to `IGNORE_NON_TESTED`\"\n                + \"in the file `utils/check_repo.py`.\"\n            )\n\n\nSUMMARY:\n**Summary:**\n\nThis function checks if all models defined in a module (`module`) are tested in a given test file (`test_file`). It returns a list of error messages for models that are not tested.\n\n**Key Details:**\n\n* Defines and tests models from `defined_models` and `tested_models` respectively\n* Returns error messages for models that are not tested but should be (`model_name`)\n* Notifies user to add model to `all_model_classes` in test file or ignore with `IGNORE_NON_TESTED`\n \n**Function Signature:**\n```python\ndef check_models_are_tested(module: types.ModuleType, test_file: str) -> List[str]:\n```\n\n**Notable Dependencies/Requirements:**\n\n* `get_models`: module for getting defined models\n* `find_tested_models`: function for finding tested models in a file\n* `TEST_FILES_WITH_NO_COMMON_TESTS`: dictionary of files with no common tests\n* `utils/check_repo.py`: configuration file for repository checks\n* `types.ModuleType`: type hint for a Python module"}


created_at: 2025-02-05 09:13:48
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py:628-646
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 646, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py", "name": "check_all_models_are_tested", "start_line": 628, "type": "function", "chroma:document": "CODE:\n\ndef check_all_models_are_tested():\n    \"\"\"Check all models are properly tested.\"\"\"\n    modules = get_model_modules()\n    test_files = get_model_test_files()\n    failures = []\n    for module in modules:\n        # Matches a module to its test file.\n        test_file = [file for file in test_files if f\"test_{module.__name__.split('.')[-1]}.py\" in file]\n        if len(test_file) == 0:\n            failures.append(f\"{module.__name__} does not have its corresponding test file {test_file}.\")\n        elif len(test_file) > 1:\n            failures.append(f\"{module.__name__} has several test files: {test_file}.\")\n        else:\n            test_file = test_file[0]\n            new_failures = check_models_are_tested(module, test_file)\n            if new_failures is not None:\n                failures += new_failures\n    if len(failures) > 0:\n\n\nSUMMARY:\n**Summary**\n\n*   Function: `check_all_models_are_tested()`\n*   **Purpose:** Verifies if all models have corresponding test files and reports any mismatches.\n*   Key implementation:\n    *   Retrieves module metadata (`get_model_modules()`).\n    *   Iterates over each module, matching it to its test file based on the module name prefix.\n    *   Calls a separate function (`check_models_are_tested()`) for modules with test files and combines any returned errors.\n*   **Notable dependencies:**\n    *   `get_model_modules()` (module metadata retrieval)\n    *   `get_model_test_files()` (test file retrieval)\n\n**External interfaces/deependencies:**\n\n    - Check Model Functions\n   "}


created_at: 2025-02-05 09:13:49
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py:649-664
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 664, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py", "name": "get_all_auto_configured_models", "start_line": 649, "type": "function", "chroma:document": "CODE:\n\ndef get_all_auto_configured_models() -> List[str]:\n    \"\"\"Return the list of all models in at least one auto class.\"\"\"\n    result = set()  # To avoid duplicates we concatenate all model classes in a set.\n    if is_torch_available():\n        for attr_name in dir(transformers.models.auto.modeling_auto):\n            if attr_name.startswith(\"MODEL_\") and attr_name.endswith(\"MAPPING_NAMES\"):\n                result = result | set(get_values(getattr(transformers.models.auto.modeling_auto, attr_name)))\n    if is_tf_available():\n        for attr_name in dir(transformers.models.auto.modeling_tf_auto):\n            if attr_name.startswith(\"TF_MODEL_\") and attr_name.endswith(\"MAPPING_NAMES\"):\n                result = result | set(get_values(getattr(transformers.models.auto.modeling_tf_auto, attr_name)))\n    if is_flax_available():\n        for attr_name in dir(transformers.models.auto.modeling_flax_auto):\n            if attr_name.startswith(\"FLAX_MODEL_\") and attr_name.endswith(\"MAPPING_NAMES\"):\n                result = result | set(get_values(getattr(transformers.models.auto.modeling_flax_auto, attr_name)))\n\n\nSUMMARY:\n**Function Summary: `get_all_auto_configured_models`**\n\n**Purpose:** Extracts and returns a list of model names from auto-configured models in at least one available framework (PyTorch, TensorFlow, or Flax).\n\n**Implementation Details:**\n\n* Iterates over specific attribute names (`MODEL_`, `TF_MODEL_`, and `FLAX_MODEL_`) and their corresponding mapping names within each auto-configured model.\n* Uses set operations to combine unique model names from different frameworks.\n\n**Notable Dependencies:**\n\n* `transformers.models.auto.modeling_auto`\n* `transformers.models.auto.modeling_tf_auto`\n* `transformers.models.auto.modeling_flax_auto`\n* `is_torch_available()`, `is_tf_available()`, and `is_flax_available()` (functions to check availability of respective frameworks)\n\n**Return Value:** A list of strings representing model names from auto-configured models in at least one available framework."}


created_at: 2025-02-05 09:13:49
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py:667-675
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 675, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py", "name": "ignore_unautoclassed", "start_line": 667, "type": "function", "chroma:document": "CODE:\n\ndef ignore_unautoclassed(model_name: str) -> bool:\n    \"\"\"Rules to determine if a model should be in an auto class.\"\"\"\n    # Special white list\n    if model_name in IGNORE_NON_AUTO_CONFIGURED:\n        return True\n    # Encoder and Decoder should be ignored\n    if \"Encoder\" in model_name or \"Decoder\" in model_name:\n        return True\n\n\nSUMMARY:\n**Summary:**\nThis function determines whether a model with the given `model_name` should be in an auto-class. It checks against a special whitelist and ensures that specific models like \"Encoder\" and \"Decoder\" are ignored.\n\n**Key Details:**\n\n* **Functionality:** `ignore_unautoclassed(model_name) -> bool`\n* **Purpose:** Determine if a model is auto-configured\n* **Design Choice:** Whitelisting approach with specific exceptions for \"Encoder\" and \"Decoder\" models.\n* **Dependencies/Requirements:**\n\t+ `IGNORE_NON_AUTO_CONFIGURED`: A predefined list of accepted non-auto-configured model names.\n\nNote: The original code snippet included the `IGNORE_NON_AUTO_CONFIGURED` variable, which is likely defined elsewhere in the project. This summary assumes its existence for the purpose of analysis."}


created_at: 2025-02-05 09:13:49
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py:678-700
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 700, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py", "name": "check_models_are_auto_configured", "start_line": 678, "type": "function", "chroma:document": "CODE:\n\ndef check_models_are_auto_configured(module: types.ModuleType, all_auto_models: List[str]) -> List[str]:\n    \"\"\"\n    Check models defined in module are each in an auto class.\n\n    Args:\n        module (`types.ModuleType`):\n            The module in which we get the models.\n        all_auto_models (`List[str]`):\n            The list of all models in an auto class (as obtained with `get_all_auto_configured_models()`).\n\n    Returns:\n        `List[str]`: The list of error messages corresponding to models not tested.\n    \"\"\"\n    defined_models = get_models(module)\n    failures = []\n    for model_name, _ in defined_models:\n        if model_name not in all_auto_models and not ignore_unautoclassed(model_name):\n            failures.append(\n                f\"{model_name} is defined in {module.__name__} but is not present in any of the auto mapping. \"\n                \"If that is intended behavior, add its name to `IGNORE_NON_AUTO_CONFIGURED` in the file \"\n                \"`utils/check_repo.py`.\"\n            )\n\n\nSUMMARY:\n**Summary:**\n\n### Function Purpose:\n\n`check_models_are_auto_configured` checks if models defined in a module are part of an auto class.\n\n### Key Implementation Details:\n\n*   Calculates the list of defined models from the input `module`.\n*   Iterates over each model, verifying if it exists in the `all_auto_models` list and ignoring unautoclassed models.\n*   Appends error messages to the `failures` list for non-existent models.\n\n### Notable Dependencies/Requirements:\n\n*   `get_all_auto_configured_models()`: Returns a list of all models in an auto class.\n*   `ignore_unautoclassed(model_name)`: Function indicating whether a model is intended to be ignored.\n*   `IGNORE_NON_AUTO_CONFIGURED` constant: File location for modifying ignore behavior."}


created_at: 2025-02-05 09:13:49
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py:703-715
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 715, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py", "name": "check_all_models_are_auto_configured", "start_line": 703, "type": "function", "chroma:document": "CODE:\n\ndef check_all_models_are_auto_configured():\n    \"\"\"Check all models are each in an auto class.\"\"\"\n    # This is where we need to check we have all backends or the check is incomplete.\n    check_missing_backends()\n    modules = get_model_modules()\n    all_auto_models = get_all_auto_configured_models()\n    failures = []\n    for module in modules:\n        new_failures = check_models_are_auto_configured(module, all_auto_models)\n        if new_failures is not None:\n            failures += new_failures\n    if len(failures) > 0:\n\n\nSUMMARY:\n**Summary:**\n\n`check_all_models_are_auto_configured()` function checks whether all models are configured as auto-configurable. It verifies that all required backends exist and iterates through modules to validate individual model configurations, accumulating failures in a list.\n\n**Key Details:**\n\n* Verifies existence of all required backends\n* Iterates through modules to validate model configurations\n* Checks for `None` return value from `check_models_are_auto_configured()`\n* Accumulates failure messages and raises an exception\n\n**Notable Dependencies/Requirements:**\n\n* `get_model_modules()` function\n* `get_all_auto_configured_models()` function\n* `check_missing_backends()` function"}


created_at: 2025-02-05 09:13:49
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py:718-759
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 759, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py", "name": "check_all_auto_object_names_being_defined", "start_line": 718, "type": "function", "chroma:document": "CODE:\n\ndef check_all_auto_object_names_being_defined():\n    \"\"\"Check all names defined in auto (name) mappings exist in the library.\"\"\"\n    # This is where we need to check we have all backends or the check is incomplete.\n    check_missing_backends()\n\n    failures = []\n    mappings_to_check = {\n        \"TOKENIZER_MAPPING_NAMES\": TOKENIZER_MAPPING_NAMES,\n        \"IMAGE_PROCESSOR_MAPPING_NAMES\": IMAGE_PROCESSOR_MAPPING_NAMES,\n        \"FEATURE_EXTRACTOR_MAPPING_NAMES\": FEATURE_EXTRACTOR_MAPPING_NAMES,\n        \"PROCESSOR_MAPPING_NAMES\": PROCESSOR_MAPPING_NAMES,\n    }\n\n    # Each auto modeling files contains multiple mappings. Let's get them in a dynamic way.\n    for module_name in [\"modeling_auto\", \"modeling_tf_auto\", \"modeling_flax_auto\"]:\n        module = getattr(transformers.models.auto, module_name, None)\n        if module is None:\n            continue\n        # all mappings in a single auto modeling file\n        mapping_names = [x for x in dir(module) if x.endswith(\"_MAPPING_NAMES\")]\n        mappings_to_check.update({name: getattr(module, name) for name in mapping_names})\n\n    for name, mapping in mappings_to_check.items():\n        for _, class_names in mapping.items():\n            if not isinstance(class_names, tuple):\n                class_names = (class_names,)\n                for class_name in class_names:\n                    if class_name is None:\n                        continue\n                    # dummy object is accepted\n                    if not hasattr(transformers, class_name):\n                        # If the class name is in a model name mapping, let's not check if there is a definition in any modeling\n                        # module, if it's a private model defined in this file.\n                        if name.endswith(\"MODEL_MAPPING_NAMES\") and is_a_private_model(class_name):\n                            continue\n                        if name.endswith(\"MODEL_FOR_IMAGE_MAPPING_NAMES\") and is_a_private_model(class_name):\n                            continue\n                        failures.append(\n                            f\"`{class_name}` appears in the mapping `{name}` but it is not defined in the library.\"\n                        )\n    if len(failures) > 0:\n\n\nSUMMARY:\n**Summary:**\n\nThe `check_all_auto_object_names_being_defined()` function checks if all names defined in auto mapping mappings exist in the library. It iterates through various auto modeling files, maps, and class definitions to identify missing objects.\n\n**Key Functionality:**\n\n* Checks for missing classes derived from `TOKENIZER_MAPPING_NAMES`, `IMAGE_PROCESSOR_MAPPING_NAMES`, etc.\n* Iterates through auto modeling files (`modeling_auto`, `modeling_tf_auto`, `modeling_flax_auto`) and extracts mapping names and their corresponding class definitions\n* Reports missing classes in the following format: \"<class_name> appears in <mapping-name> but it is not defined in the library.\"\n\n**Notable Dependencies/Requirements:**\n\n* `transformers.models.auto` module\n* External dependencies:\n\t+ `transformers`\n\t+ `torch`\n\t+ `jittor`\n\t(+ other transformer-based libraries)\n* Inherited Interfaces:\n\t+ `TOKENIZER_MAPPING_NAMES`, `IMAGE_PROCESSOR_MAPPING_NAMES`, etc.\n\n**Implementation Details:**\n\n* Uses dynamic imports and attribute access to extract mapping names and class definitions.\n* Includes private model checks for certain mapping types.\n\n**Primary Purpose:**\nThe primary purpose of this function is to identify missing classes derived from auto mapping mappings, raising an exception if any are found."}


created_at: 2025-02-05 09:13:49
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py:762-792
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 792, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py", "name": "check_all_auto_mapping_names_in_config_mapping_names", "start_line": 762, "type": "function", "chroma:document": "CODE:\n\ndef check_all_auto_mapping_names_in_config_mapping_names():\n    \"\"\"Check all keys defined in auto mappings (mappings of names) appear in `CONFIG_MAPPING_NAMES`.\"\"\"\n    # This is where we need to check we have all backends or the check is incomplete.\n    check_missing_backends()\n\n    failures = []\n    # `TOKENIZER_PROCESSOR_MAPPING_NAMES` and `AutoTokenizer` is special, and don't need to follow the rule.\n    mappings_to_check = {\n        \"IMAGE_PROCESSOR_MAPPING_NAMES\": IMAGE_PROCESSOR_MAPPING_NAMES,\n        \"FEATURE_EXTRACTOR_MAPPING_NAMES\": FEATURE_EXTRACTOR_MAPPING_NAMES,\n        \"PROCESSOR_MAPPING_NAMES\": PROCESSOR_MAPPING_NAMES,\n    }\n\n    # Each auto modeling files contains multiple mappings. Let's get them in a dynamic way.\n    for module_name in [\"modeling_auto\", \"modeling_tf_auto\", \"modeling_flax_auto\"]:\n        module = getattr(transformers.models.auto, module_name, None)\n        if module is None:\n            continue\n        # all mappings in a single auto modeling file\n        mapping_names = [x for x in dir(module) if x.endswith(\"_MAPPING_NAMES\")]\n        mappings_to_check.update({name: getattr(module, name) for name in mapping_names})\n\n    for name, mapping in mappings_to_check.items():\n        for model_type in mapping:\n            if model_type not in CONFIG_MAPPING_NAMES:\n                failures.append(\n                    f\"`{model_type}` appears in the mapping `{name}` but it is not defined in the keys of \"\n                    \"`CONFIG_MAPPING_NAMES`.\"\n                )\n    if len(failures) > 0:\n\n\nSUMMARY:\n**CONCISE SUMMARY**\n\n### Function Purpose:\n\nDefectsCheckAutoMappingNames checks if all keys defined in auto mapping names appear in CONFIG_MAPPING_NAMES.\n\n### Key Implementation Details:\n\n- Iterates through special mapping names and their corresponding values from `TOKENIZER_PROCESSOR_MAPPING_NAMES` and `AutoTokenizer`.\n- Compares model types in auto modeling files with those in CONFIG_MAPPING_NAMES.\n- Raises an exception when inconsistencies are found.\n\n### Notable Dependencies/Requirements:\n\n- `CONFIG_MAPPING_NAMES`\n- `IMAGE_PROCESSOR_MAPPING_NAMES`\n- `FEATURE_EXTRACTOR_MAPPING_NAMES`\n- `PROCESSOR_MAPPING_NAMES`\n- `transformers.models.auto` (dynamic import)"}


created_at: 2025-02-05 09:13:49
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py:795-816
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 816, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py", "name": "check_all_auto_mappings_importable", "start_line": 795, "type": "function", "chroma:document": "CODE:\n\ndef check_all_auto_mappings_importable():\n    \"\"\"Check all auto mappings can be imported.\"\"\"\n    # This is where we need to check we have all backends or the check is incomplete.\n    check_missing_backends()\n\n    failures = []\n    mappings_to_check = {}\n    # Each auto modeling files contains multiple mappings. Let's get them in a dynamic way.\n    for module_name in [\"modeling_auto\", \"modeling_tf_auto\", \"modeling_flax_auto\"]:\n        module = getattr(transformers.models.auto, module_name, None)\n        if module is None:\n            continue\n        # all mappings in a single auto modeling file\n        mapping_names = [x for x in dir(module) if x.endswith(\"_MAPPING_NAMES\")]\n        mappings_to_check.update({name: getattr(module, name) for name in mapping_names})\n\n    for name in mappings_to_check:\n        name = name.replace(\"_MAPPING_NAMES\", \"_MAPPING\")\n        if not hasattr(transformers, name):\n            failures.append(f\"`{name}`\")\n    if len(failures) > 0:\n\n\nSUMMARY:\n**Summary**\n\n`check_all_auto_mappings_importable` function checks whether all auto-mapping functions can be imported.\n\n**Primary Purpose**: Verify if all auto-mapping files are properly installed and importable.\n\n**Key Implementation Details**:\n\n* Iterates through `modeling_auto`, `modeling_tf_auto`, and `modeling_flax_auto` modules to collect their mapping names.\n* Checks for the existence of each mapped function in the `transformers` namespace using `hasattr`.\n\n**Notable Detections**:\n\n* Failure: `MAPPING_NAME` (e.g. `_MAPPING_NAMES`) not found in `transformers`.\n* Raised exception if any failure is detected, listing all failed mappings.\n\n**Dependencies and Requirements**:\n- `transformers.models.auto`\n- External dependencies: Not explicitly listed, but assumed to be part of the `transformers` package.\n- Inherited/Implemeted interfaces: No notable references."}


created_at: 2025-02-05 09:13:49
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py:819-861
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 861, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py", "name": "check_objects_being_equally_in_main_init", "start_line": 819, "type": "function", "chroma:document": "CODE:\n\ndef check_objects_being_equally_in_main_init():\n    \"\"\"\n    Check if a (TensorFlow or Flax) object is in the main __init__ iif its counterpart in PyTorch is.\n    \"\"\"\n    attrs = dir(transformers)\n\n    failures = []\n    for attr in attrs:\n        obj = getattr(transformers, attr)\n        if not hasattr(obj, \"__module__\") or \"models.deprecated\" in obj.__module__:\n            continue\n\n        module_path = obj.__module__\n        module_name = module_path.split(\".\")[-1]\n        module_dir = \".\".join(module_path.split(\".\")[:-1])\n        if (\n            module_name.startswith(\"modeling_\")\n            and not module_name.startswith(\"modeling_tf_\")\n            and not module_name.startswith(\"modeling_flax_\")\n        ):\n            parent_module = sys.modules[module_dir]\n\n            frameworks = []\n            if is_tf_available():\n                frameworks.append(\"TF\")\n            if is_flax_available():\n                frameworks.append(\"Flax\")\n\n            for framework in frameworks:\n                other_module_path = module_path.replace(\"modeling_\", f\"modeling_{framework.lower()}_\")\n                if os.path.isfile(\"src/\" + other_module_path.replace(\".\", \"/\") + \".py\"):\n                    other_module_name = module_name.replace(\"modeling_\", f\"modeling_{framework.lower()}_\")\n                    other_module = getattr(parent_module, other_module_name)\n                    if hasattr(other_module, f\"{framework}{attr}\"):\n                        if not hasattr(transformers, f\"{framework}{attr}\"):\n                            if f\"{framework}{attr}\" not in OBJECT_TO_SKIP_IN_MAIN_INIT_CHECK:\n                                failures.append(f\"{framework}{attr}\")\n                    if hasattr(other_module, f\"{framework}_{attr}\"):\n                        if not hasattr(transformers, f\"{framework}_{attr}\"):\n                            if f\"{framework}_{attr}\" not in OBJECT_TO_SKIP_IN_MAIN_INIT_CHECK:\n                                failures.append(f\"{framework}_{attr}\")\n    if len(failures) > 0:\n\n\nSUMMARY:\n**Summary:**\n\nThis function, `check_objects_being_equally_in_main_init`, checks if the objects present in TensorFlow/Flax counterpart are also present in PyTorch's main module. It iterates through all attributes of the `transformers` object and verifies the existence of corresponding attributes with prefix \"modeling_\". If found, it raises an exception indicating failures.\n\n**Key Details:**\n\n- **Implementation:** This function uses attribute iteration to check for equivalent objects between TensorFlow/Flax and PyTorch.\n- **Important Method Signatures:**\n  - `is_tf_available()`: checks if the TF framework is available.\n  - `is_flax_available()`: checks if the Flax framework is available.\n  \n**Notable Dependencies and Requirements:**\n\n- **Related Classes/Fuctions:** \n  - `transformers` (the object to be checked)\n  - `sys`, `os`\n- **External Dependencies:**\n  - TensorFlow\n  - Flax\n  - PyTorch\n- **Implemented Interfaces:**\n    - No specific interfaces are mentioned, but the function appears to follow a generic structure for comparing attributes between frameworks.\n-"}


created_at: 2025-02-05 09:13:49
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py:867-890
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 890, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py", "name": "check_decorator_order", "start_line": 867, "type": "function", "chroma:document": "CODE:\n\ndef check_decorator_order(filename: str) -> List[int]:\n    \"\"\"\n    Check that in a given test file, the slow decorator is always last.\n\n    Args:\n        filename (`str`): The path to a test file to check.\n\n    Returns:\n        `List[int]`: The list of failures as a list of indices where there are problems.\n    \"\"\"\n    with open(filename, \"r\", encoding=\"utf-8\", newline=\"\\n\") as f:\n        lines = f.readlines()\n    decorator_before = None\n    errors = []\n    for i, line in enumerate(lines):\n        search = _re_decorator.search(line)\n        if search is not None:\n            decorator_name = search.groups()[0]\n            if decorator_before is not None and decorator_name.startswith(\"parameterized\"):\n                errors.append(i)\n            decorator_before = decorator_name\n        elif decorator_before is not None:\n            decorator_before = None\n\n\nSUMMARY:\n**Summary: `check_decorator_order` Function**\n\n* **Purpose**: Verifies that in a given test file, the slow decorator \"parameterized\" always follows other decorators.\n* **Implementation Details**:\n\t+ Extracts lines from the specified test file and searches for occurrences of the \"_re_decorator\" regular expression.\n\t+ If the search finds \"parameterized\", adds the current line index to the error list if a preceding \"parameterized\" decorator was found.\n\t+ Resets the tracking of preceding decorators when a new one is encountered.\n* **Notable Dependencies**: `_re_decorator` regular expression (internal), `List[int]` type hint for return value.\n* **Assumptions/Requirements**: None external dependencies mentioned, no inherited or implemented interfaces listed."}


created_at: 2025-02-05 09:13:49
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py:893-906
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 906, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py", "name": "check_all_decorator_order", "start_line": 893, "type": "function", "chroma:document": "CODE:\n\ndef check_all_decorator_order():\n    \"\"\"Check that in all test files, the slow decorator is always last.\"\"\"\n    errors = []\n    for fname in os.listdir(PATH_TO_TESTS):\n        if fname.endswith(\".py\"):\n            filename = os.path.join(PATH_TO_TESTS, fname)\n            new_errors = check_decorator_order(filename)\n            errors += [f\"- {filename}, line {i}\" for i in new_errors]\n    if len(errors) > 0:\n        msg = \"\\n\".join(errors)\n        raise ValueError(\n            \"The parameterized decorator (and its variants) should always be first, but this is not the case in the\"\n            f\" following files:\\n{msg}\"\n\n\nSUMMARY:\n**Code Summary**\n\n```markdown\n### Functionality\n\nChecks if the slow decorator is last in every test file.\nRaises ValueError with error messages if not.\n\n### Key Implementation Details\n\n* Iterates over test files (ending with .py) and checks decorator order using `check_decorator_order` method.\n* Compares actual order against expected order (slow decorator first).\n* If mismatch, raises `ValueError` with list of error messages and corresponding file locations.\n\n### Notable Dependencies\n\n- [os] library for accessing file system\n- [PATH_TO_TESTS] environment variable or constant for path to test files\n```\n\nNote: This summary preserves the exact names of classes/functions/variables mentioned in the original code, uses precise technical terms, and focuses on key implementation details."}


created_at: 2025-02-05 09:13:49
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py:909-934
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 934, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py", "name": "find_all_documented_objects", "start_line": 909, "type": "function", "chroma:document": "CODE:\n\ndef find_all_documented_objects() -> List[str]:\n    \"\"\"\n    Parse the content of all doc files to detect which classes and functions it documents.\n\n    Returns:\n        `List[str]`: The list of all object names being documented.\n        `Dict[str, List[str]]`: A dictionary mapping the object name (full import path, e.g.\n            `integrations.PeftAdapterMixin`) to its documented methods\n    \"\"\"\n    documented_obj = []\n    documented_methods_map = {}\n    for doc_file in Path(PATH_TO_DOC).glob(\"**/*.md\"):\n        with open(doc_file, \"r\", encoding=\"utf-8\", newline=\"\\n\") as f:\n            content = f.read()\n        raw_doc_objs = re.findall(r\"\\[\\[autodoc\\]\\]\\s+(\\S+)\\s+\", content)\n        documented_obj += [obj.split(\".\")[-1] for obj in raw_doc_objs]\n\n        for obj in raw_doc_objs:\n            obj_public_methods = re.findall(rf\"\\[\\[autodoc\\]\\] {obj}((\\n\\s+-.*)+)\", content)\n            # Some objects have no methods documented\n            if len(obj_public_methods) == 0:\n                continue\n            else:\n                documented_methods_map[obj] = re.findall(r\"(?<=-\\s).*\", obj_public_methods[0][0])\n\n\n\nSUMMARY:\n**Summary**\n\nThis function, `find_all_documented_objects`, parses Markdown files in the specified directory to detect class and method documentation. It returns two data structures: a list of object names being documented and a dictionary mapping object names to their documented methods.\n\n**Key Implementation Details**\n\n* Uses regular expressions to extract object names and methods from Markdown content\n* Iterates over Markdown files in the specified directory, extracting object objects with autodoc tags\n* Uses dictionary comprehension to create a mapping between object names and their documented methods\n\n**Primary Purpose**: Detects class and method documentation in Markdown files.\n\n**Notable Dependencies**\n\n* `re` for regular expression pattern matching\n* `pathlib` for working with file paths"}


created_at: 2025-02-05 09:13:49
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py:1036-1071
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 1071, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py", "name": "ignore_undocumented", "start_line": 1036, "type": "function", "chroma:document": "CODE:\n\ndef ignore_undocumented(name: str) -> bool:\n    \"\"\"Rules to determine if `name` should be undocumented (returns `True` if it should not be documented).\"\"\"\n    # NOT DOCUMENTED ON PURPOSE.\n    # Constants uppercase are not documented.\n    if name.isupper():\n        return True\n    # PreTrainedModels / Encoders / Decoders / Layers / Embeddings / Attention are not documented.\n    if (\n        name.endswith(\"PreTrainedModel\")\n        or name.endswith(\"Decoder\")\n        or name.endswith(\"Encoder\")\n        or name.endswith(\"Layer\")\n        or name.endswith(\"Embeddings\")\n        or name.endswith(\"Attention\")\n    ):\n        return True\n    # Submodules are not documented.\n    if os.path.isdir(os.path.join(PATH_TO_TRANSFORMERS, name)) or os.path.isfile(\n        os.path.join(PATH_TO_TRANSFORMERS, f\"{name}.py\")\n    ):\n        return True\n    # All load functions are not documented.\n    if name.startswith(\"load_tf\") or name.startswith(\"load_pytorch\"):\n        return True\n    # is_xxx_available functions are not documented.\n    if name.startswith(\"is_\") and name.endswith(\"_available\"):\n        return True\n    # Deprecated objects are not documented.\n    if name in DEPRECATED_OBJECTS or name in UNDOCUMENTED_OBJECTS:\n        return True\n    # MMBT model does not really work.\n    if name.startswith(\"MMBT\"):\n        return True\n    if name in SHOULD_HAVE_THEIR_OWN_PAGE:\n        return True\n\n\nSUMMARY:\n**Summary**\n\nThis function, `ignore_undocumented`, determines whether a given string `name` should be included in the documentation based on specific rules. It checks for:\n\n* Uppercase constants and pre-trained models\n* Submodules (e.g., transformer.py)\n* Load functions (e.g., load_tf())\n* is_xxx_available functions\n* Deprecated objects\n* MMBT model\n\n**Key Implementation Details**\n\n* The function uses a combination of string matching and path checking using `os.path`.\n* The `PATH_TO_TRANSFORMERS` constant is assumed to be defined elsewhere in the codebase.\n\n**Notable Dependencies and Requirements**\n\n* `os`: for file system operations\n* `PATH_TO_TRANSFORMERS`: implied path to transformer submodules\n\n**Primary Purpose**\nThe primary purpose of this function is to determine whether a given string should be excluded from documentation based on specific rules."}


created_at: 2025-02-05 09:13:49
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py:1074-1086
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 1086, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py", "name": "check_all_objects_are_documented", "start_line": 1074, "type": "function", "chroma:document": "CODE:\n\ndef check_all_objects_are_documented():\n    \"\"\"Check all models are properly documented.\"\"\"\n    documented_objs, documented_methods_map = find_all_documented_objects()\n    modules = transformers._modules\n    objects = [c for c in dir(transformers) if c not in modules and not c.startswith(\"_\")]\n    undocumented_objs = [c for c in objects if c not in documented_objs and not ignore_undocumented(c)]\n    if len(undocumented_objs) > 0:\n        raise Exception(\n            \"The following objects are in the public init so should be documented:\\n - \"\n            + \"\\n - \".join(undocumented_objs)\n        )\n    check_model_type_doc_match()\n\n\nSUMMARY:\n**Summary**\n\n`check_all_objects_are_documented()` function:\n\n*   **Purpose and functionality**: Ensures all models in the `transformers` namespace are properly documented by checking for public initialization objects and documenting methods that exist in a mapping.\n\n**Key implementation details**:\n\n*   Checks objects in the `transformers` namespace using dir() to find non-module, non-internal private objects.\n*   Verifies if objects are not already documented by calling `find_all_documented_objects()` and ignoring certain method names using `ignore_undocumented(c)`.\n*   Raises an exception with a list of undocumented objects if any exist.\n\n**Notable dependencies or requirements**:\n\n*   Calls other functions: `find_all_documented_objects()`, `check_model_type_doc_match()`, and `check_public_method_exists(documented_methods_map)`. \n*   References classes/functions: `transformers._modules`, `dir(transformers)`.\n*   External dependencies: None mentioned.\n*   Implements interfaces: [Unspecified, as it appears to be a custom class or module within the `transformers` namespace."}


created_at: 2025-02-05 09:13:49
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py:1089-1135
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 1135, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py", "name": "check_public_method_exists", "start_line": 1089, "type": "function", "chroma:document": "CODE:\n\ndef check_public_method_exists(documented_methods_map):\n    \"\"\"Check that all explicitly documented public methods are defined in the corresponding class.\"\"\"\n    failures = []\n    for obj, methods in documented_methods_map.items():\n        # Let's ensure there is no repetition\n        if len(set(methods)) != len(methods):\n            failures.append(f\"Error in the documentation of {obj}: there are repeated documented methods.\")\n\n        # Navigates into the object, given the full import path\n        nested_path = obj.split(\".\")\n        submodule = transformers\n        if len(nested_path) > 1:\n            nested_submodules = nested_path[:-1]\n            for submodule_name in nested_submodules:\n                if submodule_name == \"transformers\":\n                    continue\n\n                try:\n                    submodule = getattr(submodule, submodule_name)\n                except AttributeError:\n                    failures.append(f\"Could not parse {submodule_name}. Are the required dependencies installed?\")\n                continue\n\n        class_name = nested_path[-1]\n\n        try:\n            obj_class = getattr(submodule, class_name)\n        except AttributeError:\n            failures.append(f\"Could not parse {class_name}. Are the required dependencies installed?\")\n            continue\n\n        # Checks that all explicitly documented methods are defined in the class\n        for method in methods:\n            if method == \"all\":  # Special keyword to document all public methods\n                continue\n            try:\n                if not hasattr(obj_class, method):\n                    failures.append(\n                        \"The following public method is explicitly documented but not defined in the corresponding \"\n                        f\"class. class: {obj}, method: {method}. If the method is defined, this error can be due to \"\n                        f\"lacking dependencies.\"\n                    )\n            except ImportError:\n                pass\n\n    if len(failures) > 0:\n\n\nSUMMARY:\n**Function Summary:**\n\n`check_public_method_exists(documented_methods_map)`\n=============================================\n\n**Purpose:** Ensures that all explicitly documented public methods are defined in the corresponding class.\n\n**Main Functionality:**\n- Verifies consistency of documented public methods across the specified map.\n- Resolves method definitions within nested object hierarchies using dynamic imports.\n\n**Key Implementation Details:**\n\n- Iterates through each object and its associated classes, ensuring unique documented method names.\n- Handles class and method name resolutions via `getattr` with conditional error handling for unknown submodules or dependencies.\n\n**External Dependencies:**\n- `transformers`: A required submodule.\n\n**Interfaces and Methods:** \n\n- The function iterates over a dictionary mapping objects to their listed public methods (`documented_methods_map`)."}


created_at: 2025-02-05 09:13:49
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py:1138-1162
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 1162, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py", "name": "check_model_type_doc_match", "start_line": 1138, "type": "function", "chroma:document": "CODE:\n\ndef check_model_type_doc_match():\n    \"\"\"Check all doc pages have a corresponding model type.\"\"\"\n    model_doc_folder = Path(PATH_TO_DOC) / \"model_doc\"\n    model_docs = [m.stem for m in model_doc_folder.glob(\"*.md\")]\n\n    model_types = list(transformers.models.auto.configuration_auto.MODEL_NAMES_MAPPING.keys())\n    model_types = [MODEL_TYPE_TO_DOC_MAPPING[m] if m in MODEL_TYPE_TO_DOC_MAPPING else m for m in model_types]\n\n    errors = []\n    for m in model_docs:\n        if m not in model_types and m != \"auto\":\n            close_matches = get_close_matches(m, model_types)\n            error_message = f\"{m} is not a proper model identifier.\"\n            if len(close_matches) > 0:\n                close_matches = \"/\".join(close_matches)\n                error_message += f\" Did you mean {close_matches}?\"\n            errors.append(error_message)\n\n    if len(errors) > 0:\n        raise ValueError(\n            \"Some model doc pages do not match any existing model type:\\n\"\n            + \"\\n\".join(errors)\n            + \"\\nYou can add any missing model type to the `MODEL_NAMES_MAPPING` constant in \"\n            \"models/auto/configuration_auto.py.\"\n\n\nSUMMARY:\n**Summary**\n\nThis function, `check_model_type_doc_match`, checks if all documentation pages under `PATH_TO_DOC/model_doc` correspond to existing models. It iterates through the doc pages, finds their corresponding model types using the `MODEL_TYPE_TO_DOC_MAPPING` dictionary, and compiles a list of errors. The function raises a `ValueError` with a list of error messages if any errors are found.\n\n**Key Implementation Details**\n\n* Iterates through documentation pages under `PATH_TO_DOC/model_doc`\n* Uses `transformers.models.auto.configuration_auto.MODEL_NAMES_MAPPING` to get existing model types\n* Compares each doc page's name with the model types, providing close matches for non-matching names\n* Raises an error if any doc pages do not match an existing model type\n\n**Note**\n\nThe function relies on external dependencies:\n\n* `transformers.models.auto.configuration_auto.MODEL_NAMES_MAPPING`\n* `get_close_matches` (not shown in this code snippet)\n* `models/auto/configuration_auto`"}


created_at: 2025-02-05 09:13:49
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py:1165-1191
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 1191, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py", "name": "check_deprecated_constant_is_up_to_date", "start_line": 1165, "type": "function", "chroma:document": "CODE:\n\ndef check_deprecated_constant_is_up_to_date():\n    \"\"\"\n    Check if the constant `DEPRECATED_MODELS` in `models/auto/configuration_auto.py` is up to date.\n    \"\"\"\n    deprecated_folder = os.path.join(PATH_TO_TRANSFORMERS, \"models\", \"deprecated\")\n    deprecated_models = [m for m in os.listdir(deprecated_folder) if not m.startswith(\"_\")]\n\n    constant_to_check = transformers.models.auto.configuration_auto.DEPRECATED_MODELS\n    message = []\n    missing_models = sorted(set(deprecated_models) - set(constant_to_check))\n    if len(missing_models) != 0:\n        missing_models = \", \".join(missing_models)\n        message.append(\n            \"The following models are in the deprecated folder, make sure to add them to `DEPRECATED_MODELS` in \"\n            f\"`models/auto/configuration_auto.py`: {missing_models}.\"\n        )\n\n    extra_models = sorted(set(constant_to_check) - set(deprecated_models))\n    if len(extra_models) != 0:\n        extra_models = \", \".join(extra_models)\n        message.append(\n            \"The following models are in the `DEPRECATED_MODELS` constant but not in the deprecated folder. Either \"\n            f\"remove them from the constant or move to the deprecated folder: {extra_models}.\"\n        )\n\n    if len(message) > 0:\n\n\nSUMMARY:\n**Summary**\n\nThe `check_deprecated_constant_is_up_to_date` function checks the consistency of the `DEPRECATED_MODELS` constant in `models/auto/configuration_auto.py`. It ensures that:\n\n* All models in the deprecated folder are included in the constant.\n* None of the models listed in the constant are not present in the deprecated folder.\n\n**Key Implementation Details**\n\n* The function uses a list comprehension to extract model names from the deprecated folder.\n* It compares the models in the `DEPRECATED_MODELS` constant with those in the deprecated folder using set differences.\n* If any inconsistencies are found, it raises an exception with a descriptive message.\n\n**Notable Dependencies and Requirements**\n\n* `transformers`: A dependency for accessing the `models.auto.configuration_auto` module.\n* The function assumes that the constants and folders referenced are located within this module."}


created_at: 2025-02-05 09:13:49
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py:1194-1217
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 1217, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_repo.py", "name": "check_repo_quality", "start_line": 1194, "type": "function", "chroma:document": "CODE:\n\ndef check_repo_quality():\n    \"\"\"Check all models are tested and documented.\"\"\"\n    print(\"Repository-wide checks:\")\n    print(\"    - checking all models are included.\")\n    check_model_list()\n    print(\"    - checking all models are public.\")\n    check_models_are_in_init()\n    print(\"    - checking all models have tests.\")\n    check_all_decorator_order()\n    check_all_models_are_tested()\n    print(\"    - checking all objects have documentation.\")\n    check_all_objects_are_documented()\n    print(\"    - checking all models are in at least one auto class.\")\n    check_all_models_are_auto_configured()\n    print(\"    - checking all names in auto name mappings are defined.\")\n    check_all_auto_object_names_being_defined()\n    print(\"    - checking all keys in auto name mappings are defined in `CONFIG_MAPPING_NAMES`.\")\n    check_all_auto_mapping_names_in_config_mapping_names()\n    print(\"    - checking all auto mappings could be imported.\")\n    check_all_auto_mappings_importable()\n    print(\"    - checking all objects are equally (across frameworks) in the main __init__.\")\n    check_objects_being_equally_in_main_init()\n    print(\"    - checking the DEPRECATED_MODELS constant is up to date.\")\n\n\nSUMMARY:\n**Summary:**\n\nThe `check_repo_quality` function runs a sequence of checks on a repository, evaluating various aspects of model quality and configuration. Specifically, it:\n\n*   Verifies models' inclusion\n*   Inspects public visibility of models\n*   Confirms test existence for models\n*   Checks for documentation presence in objects\n*   Validates auto-class configurations\n*   Examines auto-mapping definitions and importability\n*   Ensures object distribution in `__init__`\n*   Verifies the `DEPRECATED_MODELS` constant's accuracy\n\n**Key Functions/Methods:**\n\n- `check_model_list()`: Verifies models' existence\n- `check_models_are_in_init()`: Inspects public visibility of models\n- `check_all_decorator_order()`: Confirms test existence for models\n- `check_all_models_are_tested()`: Confirms documentation presence in objects\n- `check_all_objects_are_documented()`: Validates auto-mapping definitions and importability\n\n**Notable Dependencies/Requirements:**\n\n- The function relies on the `CONFIG_MAPPING_NAMES` constant, indicating an external dependency.\n- It also expects the existence of a `DEPRECATED_MODELS` constant, implying external configuration or setup.\n\nPlease note that further context is necessary to provide a more accurate analysis."}


created_at: 2025-02-05 09:13:49
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_doctest_list.py:43-75
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 75, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_doctest_list.py", "name": "clean_doctest_list", "start_line": 43, "type": "function", "chroma:document": "CODE:\n\ndef clean_doctest_list(doctest_file: str, overwrite: bool = False):\n    \"\"\"\n    Cleans the doctest in a given file.\n\n    Args:\n        doctest_file (`str`):\n            The path to the doctest file to check or clean.\n        overwrite (`bool`, *optional*, defaults to `False`):\n            Whether or not to fix problems. If `False`, will error when the file is not clean.\n    \"\"\"\n    non_existent_paths = []\n    all_paths = []\n    with open(doctest_file, \"r\", encoding=\"utf-8\") as f:\n        for line in f:\n            line = line.strip().split(\" \")[0]\n            path = os.path.join(REPO_PATH, line)\n            if not (os.path.isfile(path) or os.path.isdir(path)):\n                non_existent_paths.append(line)\n            all_paths.append(line)\n\n    if len(non_existent_paths) > 0:\n        non_existent_paths = \"\\n\".join([f\"- {f}\" for f in non_existent_paths])\n        raise ValueError(f\"`{doctest_file}` contains non-existent paths:\\n{non_existent_paths}\")\n\n    sorted_paths = sorted(all_paths)\n    if all_paths != sorted_paths:\n        if not overwrite:\n            raise ValueError(\n                f\"Files in `{doctest_file}` are not in alphabetical order, run `make fix-copies` to fix \"\n                \"this automatically.\"\n            )\n        with open(doctest_file, \"w\", encoding=\"utf-8\") as f:\n\n\nSUMMARY:\n**Code Summary**\n\n### Function Purpose\n\n`clean_doctest_list` cleans and organizes a doctest file by removing non-existent paths and sorting the remaining paths in alphabetical order.\n\n### Key Implementation Details\n\n* The function reads a doctest file line-by-line, stripping leading whitespace and splitting on spaces.\n* It checks if each path exists and creates a list of non-existent paths.\n* If any non-existent paths are found, they are reported as an error.\n* The function then sorts the remaining paths alphabetically and writes them back to the original file. The sorting can be skipped with the `overwrite` parameter.\n\n### Notable Dependencies\n\n* `os.path`: used for path manipulation (e.g., `join`, `resolve`)\n* `REPO_PATH`: assumed to be a predefined constant containing the repository root path\n\n### Method Signature and Interfaces\n\n| Method/Interface | Description |\n| --- | --- |\n| `clean_doctest_list(doctest_file: str, overwrite: bool = False)` | Cleans and organizes a doctest file. |\n\n### External Dependencies\n\nNone mentioned in this snippet, but `os` module is used implicitly."}


created_at: 2025-02-05 09:13:49
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py:175-177
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 177, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py", "name": "_is_definition_header_ending_line", "start_line": 175, "type": "function", "chroma:document": "CODE:\n\ndef _is_definition_header_ending_line(line: str) -> bool:\n    # Helper function. Returns `True` if `line` is the end parenthesis of a class/function definition\n\n\nSUMMARY:\n**Summary:**\n\nThis function, `_is_definition_header_ending_line`, determines whether a given line of code indicates the end of a class or function definition.\n\n```markdown\n### Main Purpose and Functionality\n\nDetermines if a line ends an opening parenthesis of a class or function definition.\n\n### Key Implementation Details\n\n- Uses regular expression (`re.search`) to search for the pattern `^\\s*\\)(\\s*->.*:|:)\\s*$` at the start of the line.\n- Returns `True` when the pattern is found, indicating an ending definition line.\n- The primary purpose is to identify lines that signal the completion of a class or function definition.\n\n### Notable Dependencies and Requirements\n\nNone listed, but uses Python's built-in `re` module for regular expression matching.\n```\n\n**Additional Information:**\n\nThis function does not depend on any external libraries beyond the `re` module."}


created_at: 2025-02-05 09:13:49
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py:180-183
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 183, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py", "name": "_should_continue", "start_line": 180, "type": "function", "chroma:document": "CODE:\n\ndef _should_continue(line: str, indent: str) -> bool:\n    # Helper function. Returns `True` if `line` is empty, starts with the `indent` or is the end parenthesis of a\n    # class/function definition\n\n\nSUMMARY:\n**Summary:**\n\n- `should_continue(line: str, indent: str) -> bool`: Returns True if the given line is empty, starts with the specified indent, or marks the end of a class or function definition.\n\n**Implementation Details:**\n\n- Checks for three conditions:\n  - Starts with the specified indent (`line.startswith(indent)`)\n  - Is an empty line after stripping whitespace (`len(line.strip()) == 0`)\n  - Marks the end of a definition header using `_is_definition_header_ending_line(line)`"}


created_at: 2025-02-05 09:13:49
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py:186-252
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 252, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py", "name": "_sanity_check_splits", "start_line": 186, "type": "function", "chroma:document": "CODE:\n\ndef _sanity_check_splits(splits_1, splits_2, is_class, filename):\n    \"\"\"Check the two (inner) block structures of the corresponding code block given by `split_code_into_blocks` match.\n\n    For the case of `class`, they must be of one of the following 3 cases:\n\n        - a single block without name:\n\n            class foo:\n                a = 1\n\n        - a consecutive sequence of (1 or more) blocks with name\n\n            class foo:\n\n                def f(x):\n                    return x\n\n        - a block without name, followed by a consecutive sequence of (1 or more) blocks with name\n\n            class foo:\n                a = 1\n\n                def f(x):\n                    return x\n\n                def g(x):\n                    return None\n\n    The 2 code snippets that give `splits_1` and `splits_2` have to be in the same case to pass this check, but the\n    number of blocks with name in the consecutive sequence is not taken into account.\n\n    For the case of `function or method`, we don't require it to be in one of the above 3 cases. However, the structure\n    of`splits_1` and `splits_2` have to match exactly. In particular, the number of blocks with name in a consecutive\n    sequence is taken into account.\n    \"\"\"\n    block_names_1 = []\n    block_names_2 = []\n\n    for block in splits_1[1:]:\n        if block[0].startswith(\"_block_without_name_\"):\n            block_names_1.append(\"block_without_name\")\n        elif not block[0].startswith(\"_empty_block_\") and (\n            not is_class or len(block_names_1) == 0 or block_names_1[-1].startswith(\"block_without_name\")\n        ):\n            block_names_1.append(\"block_with_name\")\n\n    for block in splits_2[1:]:\n        if block[0].startswith(\"_block_without_name_\"):\n            block_names_2.append(\"block_without_name\")\n        elif not block[0].startswith(\"_empty_block_\") and (\n            not is_class or len(block_names_2) == 0 or block_names_2[-1].startswith(\"block_without_name\")\n        ):\n            block_names_2.append(\"block_with_name\")\n\n    if is_class:\n        if block_names_1 not in [\n            [\"block_without_name\"],\n            [\"block_with_name\"],\n            [\"block_without_name\", \"block_with_name\"],\n        ]:\n            raise ValueError(\n                f\"\"\"Class defined in {filename} doesn't have the expected stucture.\n                See the docstring of `_sanity_check_splits` in the file `utils/check_copies.py`\"\"\",\n            )\n\n    if block_names_1 != block_names_2:\n\n\nSUMMARY:\n**Function Summary: `_sanity_check_splits`**\n\n### Purpose and Functionality\n\nVerifies the structure of paired code blocks in `splits_1` and `splits_2`. For classes, ensures one of three predefined block structures is matched. For functions/methods, verifies exact matching of block arrangements.\n\n### Key Implementation Details\n\n* Class block structure checking:\n\t+ Supports single block without name and consecutive sequence of blocks with name\n\t+ Number of blocks with name in consecutive sequence not considered\n* Function/method block structure checking:\n\t+ Exact matching of block arrangement required\n\n### Notable Dependencies and Requirements\n\n* `is_class` flag indicating class definition or not\n* `filename` parameter for logging purposes\n* Internal constants and interface (`_block_without_name_`, `_empty_block_`)"}


created_at: 2025-02-05 09:13:49
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py:255-280
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 280, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py", "name": "find_block_end", "start_line": 255, "type": "function", "chroma:document": "CODE:\n\ndef find_block_end(lines: List[str], start_index: int, indent: int) -> int:\n    \"\"\"\n    Find the end of the class/func block starting at `start_index` in a source code (defined by `lines`).\n\n    Args:\n        lines (`List[str]`):\n            The source code, represented by a list of lines.\n        start_index (`int`):\n            The starting index of the target class/func block.\n        indent (`int`):\n            The indent of the class/func body.\n\n    Returns:\n        `int`: The index of the block's ending line plus by 1 (i.e. exclusive).\n    \"\"\"\n    indent = \" \" * indent\n    # enter the block body\n    line_index = start_index + 1\n\n    while line_index < len(lines) and _should_continue(lines[line_index], indent):\n        line_index += 1\n    # Clean up empty lines at the end (if any).\n    while len(lines[line_index - 1]) <= 1:\n        line_index -= 1\n\n\n\nSUMMARY:\n**Function Summary:**\n`find_block_end`\n---------------\n\nFind the index of the block's ending line in a source code.\n\n```python\ndef find_block_end(lines: List[str], start_index: int, indent: int) -> int:\n    # ...\n```\n\n### Purpose and Functionality:\n\nThis function finds the end of a class/func block in a source code, starting from `start_index`.\nIt iterates through the lines until it encounters an invalid continuation line or reaches the end of the code.\n\n### Key Implementation Details:\n\n- Iterates until a blank or invalid line (`_should_continue`) is encountered.\n- Cleans up empty lines at the end.\n- Returns the index of the block's ending line (inclusive).\n\n### Dependencies and Requirements:\n No external dependencies."}


created_at: 2025-02-05 09:13:49
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py:283-388
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 388, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py", "name": "split_code_into_blocks", "start_line": 283, "type": "function", "chroma:document": "CODE:\n\ndef split_code_into_blocks(\n    lines: List[str], start_index: int, end_index: int, indent: int, backtrace: bool = False\n) -> List[Tuple[str, int, int]]:\n    \"\"\"\n    Split the class/func block starting at `start_index` in a source code (defined by `lines`) into *inner blocks*.\n\n    The block's header is included as the first element. The contiguous regions (without empty lines) that are not\n    inside any inner block are included as blocks. The contiguous regions of empty lines that are not inside any inner\n    block are also included as (dummy) blocks.\n\n    Args:\n        lines (`List[str]`):\n            The source code, represented by a list of lines.\n        start_index (`int`):\n            The starting index of the target class/func block.\n        end_index (`int`):\n            The ending index of the target class/func block.\n        indent (`int`):\n            The indent of the class/func body.\n        backtrace (`bool`, *optional*, defaults to `False`):\n            Whether or not to include the lines before the inner class/func block's header (e.g. comments, decorators,\n            etc.) until an empty line is encountered.\n\n    Returns:\n        `List[Tuple[str, int, int]]`: A list of elements with the form `(block_name, start_index, end_index)`.\n    \"\"\"\n    splits = []\n    # `indent - 4` is the indent level of the target class/func header\n    try:\n        target_block_name = re.search(\n            rf\"^{' ' * (indent - 4)}((class|def)\\s+\\S+)(\\(|\\:)\", lines[start_index]\n        ).groups()[0]\n    except Exception:\n        start_context = min(start_index - 10, 0)\n        end_context = min(end_index + 10, len(lines))\n        raise ValueError(\n            f\"Tried to split a class or function. It did not work. Error comes from line {start_index}: \\n```\\n\"\n            + \"\".join(lines[start_context:end_context])\n            + \"```\\n\"\n        )\n\n    # from now on, the `block` means inner blocks unless explicitly specified\n    indent_str = \" \" * indent\n    block_without_name_idx = 0\n    empty_block_idx = 0\n\n    # Find the lines for the definition header\n    index = start_index\n    if \"(\" in lines[start_index] and \"):\" not in lines[start_index] in lines[start_index]:\n        while index < end_index:\n            if _is_definition_header_ending_line(lines[index]):\n                break\n            index += 1\n\n    # the first line outside the definition header\n    index += 1\n    splits.append((target_block_name, start_index, index))\n\n    block_start_index, prev_block_end_index = index, index\n    while index < end_index:\n        # if found, it will be an inner block\n        block_found = re.search(rf\"^{indent_str}((class|def)\\s+\\S+)(\\(|\\:)\", lines[index])\n        if block_found:\n            name = block_found.groups()[0]\n\n            block_end_index = find_block_end(lines, index, indent + 4)\n\n            # backtrace to include the lines before the found block's definition header (e.g. comments, decorators,\n            # etc.) until an empty line is encountered.\n            block_start_index = index\n            if index > prev_block_end_index and backtrace:\n                idx = index - 1\n                for idx in range(index - 1, prev_block_end_index - 2, -1):\n                    if not (len(lines[idx].strip()) > 0 and lines[idx].startswith(indent_str)):\n                        break\n                idx += 1\n                if idx < index:\n                    block_start_index = idx\n\n            # between the current found block and the previous found block\n            if block_start_index > prev_block_end_index:\n                # give it a dummy name\n                if len(\"\".join(lines[prev_block_end_index:block_start_index]).strip()) == 0:\n                    prev_block_name = f\"_empty_block_{empty_block_idx}\"\n                    empty_block_idx += 1\n                else:\n                    prev_block_name = f\"_block_without_name_{block_without_name_idx}\"\n                    block_without_name_idx += 1\n                # Add it as a block\n                splits.append((prev_block_name, prev_block_end_index, block_start_index))\n\n            # Add the current found block\n            splits.append((name, block_start_index, block_end_index))\n            prev_block_end_index = block_end_index\n            index = block_end_index - 1\n\n        index += 1\n\n    if index > prev_block_end_index:\n        if len(\"\".join(lines[prev_block_end_index:index]).strip()) == 0:\n            prev_block_name = f\"_empty_block_{empty_block_idx}\"\n        else:\n            prev_block_name = f\"_block_without_name_{block_without_name_idx}\"\n        splits.append((prev_block_name, prev_block_end_index, index))\n\n\n\nSUMMARY:\n**Function Summary: `split_code_into_blocks`**\n\n### Purpose and Functionality\nSplits the code block in the specified range into inner blocks, including contiguous regions without empty lines.\n\n### Key Implementation Details\n\n* Takes source code `_lines`, start index `_start_index`, end index `_end_index`, and `indent_level`.\n* Uses a regular expression to find the definition header.\n* Iterates through the code to find inner blocks using a regular expression.\n* Handles backtrace by including lines before the inner block's definition header.\n\n### Method Signatures and Interfaces\n\n* Takes a list of strings (`lines`) as input.\n* Returns a list of tuples containing string, start index, and end index (`List[Tuple[str, int, int]]`).\n\n### Notable Dependencies or Requirements\n\n* Uses the `re` module for regular expressions.\n\n### Output Format\nReturns a list of tuples containing block names, start indices, and end indices.\n\n### Summary\n\nSplits code into inner blocks using a regular expression-based approach. It handles backtrace and returns a list of tuples containing block information."}


created_at: 2025-02-05 09:13:50
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py:391-465
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 465, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py", "name": "find_code_in_transformers", "start_line": 391, "type": "function", "chroma:document": "CODE:\n\ndef find_code_in_transformers(\n    object_name: str, base_path: str = None, return_indices: bool = False\n) -> Union[str, Tuple[List[str], int, int]]:\n    \"\"\"\n    Find and return the source code of an object.\n\n    Args:\n        object_name (`str`):\n            The name of the object we want the source code of.\n        base_path (`str`, *optional*):\n            The path to the base folder where files are checked. If not set, it will be set to `TRANSFORMERS_PATH`.\n        return_indices(`bool`, *optional*, defaults to `False`):\n            If `False`, will only return the code (as a string), otherwise it will also return the whole lines of the\n            file where the object specified by `object_name` is defined, together the start/end indices of the block in\n            the file that defines the object.\n\n    Returns:\n        `Union[str, Tuple[List[str], int, int]]`: If `return_indices=False`, only the source code of the object will be\n        returned. Otherwise, it also returns the whole lines of the file where the object specified by `object_name` is\n        defined, together the start/end indices of the block in the file that defines the object.\n    \"\"\"\n    parts = object_name.split(\".\")\n    i = 0\n\n    # We can't set this as the default value in the argument, otherwise `CopyCheckTester` will fail, as it uses a\n    # patched temp directory.\n    if base_path is None:\n        base_path = TRANSFORMERS_PATH\n\n    # Detail: the `Copied from` statement is originally designed to work with the last part of `TRANSFORMERS_PATH`,\n    # (which is `transformers`). The same should be applied for `MODEL_TEST_PATH`. However, its last part is `models`\n    # (to only check and search in it) which is a bit confusing. So we keep the copied statement staring with\n    # `tests.models.` and change it to `tests` here.\n    if base_path == MODEL_TEST_PATH:\n        base_path = \"tests\"\n\n    # First let's find the module where our object lives.\n    module = parts[i]\n    while i < len(parts) and not os.path.isfile(os.path.join(base_path, f\"{module}.py\")):\n        i += 1\n        if i < len(parts):\n            module = os.path.join(module, parts[i])\n    if i >= len(parts):\n        raise ValueError(\n            f\"`object_name` should begin with the name of a module of transformers but got {object_name}.\"\n        )\n\n    with open(os.path.join(base_path, f\"{module}.py\"), \"r\", encoding=\"utf-8\", newline=\"\\n\") as f:\n        lines = f.readlines()\n\n    # Now let's find the class / func in the code!\n    indent = \"\"\n    line_index = 0\n    for name in parts[i + 1 :]:\n        while (\n            line_index < len(lines) and re.search(rf\"^{indent}(class|def)\\s+{name}(\\(|\\:)\", lines[line_index]) is None\n        ):\n            line_index += 1\n        # find the target specified in the current level in `parts` -> increase `indent` so we can search the next\n        indent += \"    \"\n        # the index of the first line in the (currently found) block *body*\n        line_index += 1\n\n    if line_index >= len(lines):\n        raise ValueError(f\" {object_name} does not match any function or class in {module}.\")\n\n    # `indent` is already one level deeper than the (found) class/func block's definition header\n\n    # We found the beginning of the class / func, now let's find the end (when the indent diminishes).\n    # `start_index` is the index of the class/func block's definition header\n    start_index = line_index - 1\n    end_index = find_block_end(lines, start_index, len(indent))\n\n    code = \"\".join(lines[start_index:end_index])\n\n\nSUMMARY:\n**Summary: `find_code_in_transformers` Function**\n\nThe `find_code_in_transformers` function finds and returns the source code of an object in the transformers library.\n\n### Purpose and Functionality\n\n* Takes an object name, base path (optional), and return indices as input arguments.\n* Returns either the object's source code or a tuple containing the source code, file lines, start index, and end index.\n\n### Key Implementation Details\n\n* Finds the module where the object lives by splitting the object name and checking existing files in the base path.\n* Parses the Python code to find the class or function specified by the object name, returning the source code if `return_indices` is False.\n\n### Notable Dependencies and Requirements\n\n* Uses external dependencies:\n\t+ `os` for file operations\n\t+ `re` for regular expression matching\n\t+ `transformers` library (not explicitly imported)\n* Inherited or implemented interfaces:\n\t+ None, but uses Python's built-in functions such as `open`, `readlines`, and `find_block_end`.\n\n### Method Signatures\n\nThe function takes the following input arguments:\n\n* `object_name`: str\n* `base_path`: str (optional)\n* `return_indices`: bool (optional)\n\nThe function returns a Union type containing either str or a tuple with the following elements:\n\t+ str: source code of the object, if `return_indices` is False.\n\t+ Tuple[List[str], int, int]: file lines, start index, and end index, if `return_indices` is True."}


