# embeddings_queue

created_at: 2025-02-05 09:13:50
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py:468-490
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 490, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py", "name": "replace_code", "start_line": 468, "type": "function", "chroma:document": "CODE:\n\ndef replace_code(code: str, replace_pattern: str) -> str:\n    \"\"\"Replace `code` by a pattern of the form `with X1->X2,Y1->Y2,Z1->Z2`.\n\n    Args:\n        code (`str`): The code to be modified.\n        replace_pattern (`str`): The pattern used to modify `code`.\n\n    Returns:\n        `str`: The modified code.\n    \"\"\"\n    if len(replace_pattern) > 0:\n        patterns = replace_pattern.replace(\"with\", \"\").split(\",\")\n        patterns = [_re_replace_pattern.search(p) for p in patterns]\n        for pattern in patterns:\n            if pattern is None:\n                continue\n            obj1, obj2, option = pattern.groups()\n            code = re.sub(obj1, obj2, code)\n            if option.strip() == \"all-casing\":\n                code = re.sub(obj1.lower(), obj2.lower(), code)\n                code = re.sub(obj1.upper(), obj2.upper(), code)\n\n\n\nSUMMARY:\n**Summary**\n\n`replace_code` function:\n\n* Replaces occurrences of `obj1` with `obj2` in `code`, optionally applying casing conversions (`all-casing` option).\n* Takes two inputs: `code` (string) and `replace_pattern` (string, formatted as \"with X1->X2,Y1->Y2,Z1->Z2\").\n* Utilizes regular expressions to perform replacements, leveraging `_re_replace_pattern.search()` and `re.sub()` functions.\n\n**Key Implementation Details**\n\n* Uses regular expression substitutions (`re.sub()`) to replace patterns in the input code.\n* Applies casing conversions for the \"all-casing\" option using additional `re.sub()` calls.\n* Utilizes `_re_replace_pattern.search()` to extract pattern groups (e.g., `obj1`, `obj2`, and `option`)."}


created_at: 2025-02-05 09:13:50
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py:493-533
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 533, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py", "name": "find_code_and_splits", "start_line": 493, "type": "function", "chroma:document": "CODE:\n\ndef find_code_and_splits(object_name: str, base_path: str, buffer: dict = None):\n    \"\"\"Find the code of an object (specified by `object_name`) and split it into blocks.\n\n    Args:\n        object_name (`str`):\n            The name of the object, e.g. `transformers.models.bert.modeling_bert.BertAttention` or\n            `tests.models.llama.test_modeling_llama.LlamaModelTest.test_config`.\n        base_path (`str`):\n            The path to the base directory within which the search will be performed. It could be either\n            `TRANSFORMERS_PATH` or `MODEL_TEST_PATH`.\n        buffer (`dict`, *optional*):\n            The buffer used to store the previous results in order to speed up the process.\n\n    Returns:\n        lines (`List[str]`):\n            The lines of the whole file where the object is defined.\n        code (`str`):\n            The object's code.\n        code_splits (`List[Tuple[str, int, int]]`):\n            `code` splitted into blocks. See `split_code_into_blocks`.\n    \"\"\"\n    if buffer is None:\n        buffer = {}\n\n    if (object_name, base_path) in buffer:\n        lines, code, code_splits = buffer[(object_name, base_path)]\n    else:\n        code, (lines, target_start_index, target_end_index) = find_code_in_transformers(\n            object_name, base_path=base_path, return_indices=True\n        )\n        indent = get_indent(code)\n\n        # Split the code into blocks\n        # `indent` is the indent of the class/func definition header, but `code_splits` expects the indent level of the\n        # block body.\n        code_splits = split_code_into_blocks(\n            lines, target_start_index, target_end_index, len(indent) + 4, backtrace=True\n        )\n        buffer[(object_name, base_path)] = lines, code, code_splits\n\n\n\nSUMMARY:\n**Code Summary: `find_code_and_splits` Function**\n\n**Purpose and Functionality:**\nThe `find_code_and_splits` function searches for the code of an object within a specified directory (`base_path`) and splits it into blocks. It retrieves previous results from a buffer to speed up the process.\n\n**Key Implementation Details:**\n\n*   Uses `find_code_in_transformers` to search for the code in the transformers library.\n*   Utilizes `split_code_into_blocks` to split the retrieved code into blocks based on indentation levels.\n*   Expects the `buffer` parameter to store previous results and retrieve them if available.\n\n**Notable Dependencies and Requirements:**\n\n*   `find_code_in_transformers`: Function from transformers library or custom implementation.\n*   `split_code_into_blocks`: Custom or third-party function for block splitting.\n*   `get_indent`: Custom or external function to calculate indentation levels."}


created_at: 2025-02-05 09:13:50
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py:542-558
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 558, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py", "name": "get_indent", "start_line": 542, "type": "function", "chroma:document": "CODE:\n\ndef get_indent(code: str) -> str:\n    \"\"\"\n    Find the indent in the first non empty line in a code sample.\n\n    Args:\n        code (`str`): The code to inspect.\n\n    Returns:\n        `str`: The indent looked at (as string).\n    \"\"\"\n    lines = code.split(\"\\n\")\n    idx = 0\n    while idx < len(lines) and len(lines[idx]) == 0:\n        idx += 1\n    if idx < len(lines):\n        return re.search(r\"^(\\s*)\\S\", lines[idx]).groups()[0]\n\n\nSUMMARY:\n**Function Summary**\n\nGetIndent Function: Finds the indent in the first non-empty line of code.\n\n**Primary Purpose:** Extracts the leading whitespace from the first non-empty line of code.\n\n**Key Implementation Details:**\n\n* Splits code into individual lines\n* Iterates through lines until a non-empty one is found\n* Uses regex to match and extract the leading whitespace\n\n**Notable Dependencies:**\n\n* `re` module for regular expression matching ( referencing `search()` method)\n* No external dependencies listed, but assumes Python environment with `str` and `list` data types.\n\n**Method Signature:** `get_indent(code: str) -> str`\n\nThis function is designed to be used in a context where code snippets need to be analyzed or processed before execution. It provides a concise way to extract the leading whitespace from the first non-empty line of code, making it easier to analyze or visualize indentation schemes."}


created_at: 2025-02-05 09:13:50
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py:561-568
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 568, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py", "name": "run_ruff", "start_line": 561, "type": "function", "chroma:document": "CODE:\n\ndef run_ruff(code, check=False):\n    if check:\n        command = [\"ruff\", \"check\", \"-\", \"--fix\", \"--exit-zero\"]\n    else:\n        command = [\"ruff\", \"format\", \"-\", \"--config\", \"pyproject.toml\", \"--silent\"]\n    process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)\n    stdout, _ = process.communicate(input=code.encode())\n\n\nSUMMARY:\n**Summary:**\n\n`run_ruff` is a function that runs the Ruff code formatter with either `check` or `format` functionality. It takes in code and configuration options, utilizes subprocess to execute the Ruff command, and returns the formatted output.\n\n**Key Implementation Details:**\n\n- Utilizes `subprocess.Popen` for executing the Ruff command\n- Supports both `check` (linting) and `format` (code reformattting)\n- Uses `pyproject.toml` as default configuration file\n\n**Dependences/Requirements:**\n\n- Ruff code formatter (`ruff`)\n- `pyproject.toml`\n\nNote: The function returns the formatted output from the standard output of the Ruff command."}


created_at: 2025-02-05 09:13:50
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py:571-586
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 586, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py", "name": "stylify", "start_line": 571, "type": "function", "chroma:document": "CODE:\n\ndef stylify(code: str) -> str:\n    \"\"\"\n    Applies the ruff part of our `make style` command to some code. This formats the code using `ruff format`.\n    As `ruff` does not provide a python api this cannot be done on the fly.\n\n    Args:\n        code (`str`): The code to format.\n\n    Returns:\n        `str`: The formatted code.\n    \"\"\"\n    has_indent = len(get_indent(code)) > 0\n    if has_indent:\n        code = f\"class Bla:\\n{code}\"\n    formatted_code = run_ruff(code)\n\n\nSUMMARY:\nHere is a concise summary of the `stylify` function:\n\n**Summary:**\nConverts input Python code to be compatible with RUFF's formatting style.\n\n**Key Details:**\n\n* Formats input code using the `ruff format` command\n* Appends an empty class wrapper to preserve indentation if present\n* Returns the formatted code without the wrapped class prefix\n\n**Notable Dependencies:**\n\n* `run_ruff`: External function from external library (not shown)\n* `get_indent`: External function from external library (not shown)"}


created_at: 2025-02-05 09:13:50
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py:589-637
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 637, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py", "name": "check_codes_match", "start_line": 589, "type": "function", "chroma:document": "CODE:\n\ndef check_codes_match(observed_code: str, theoretical_code: str) -> Optional[int]:\n    \"\"\"\n    Checks if two version of a code match with the exception of the class/function name.\n\n    Args:\n        observed_code (`str`): The code found.\n        theoretical_code (`str`): The code to match.\n\n    Returns:\n        `Optional[int]`: The index of the first line where there is a difference (if any) and `None` if the codes\n        match.\n    \"\"\"\n    observed_code_header = observed_code.split(\"\\n\")[0]\n    theoretical_code_header = theoretical_code.split(\"\\n\")[0]\n\n    # Catch the function/class name: it is expected that those do not match.\n    _re_class_match = re.compile(r\"class\\s+([^\\(:]+)(?:\\(|:)\")\n    _re_func_match = re.compile(r\"def\\s+([^\\(]+)\\(\")\n    for re_pattern in [_re_class_match, _re_func_match]:\n        if re_pattern.match(observed_code_header) is not None:\n            try:\n                observed_obj_name = re_pattern.search(observed_code_header).groups()[0]\n            except Exception:\n                raise ValueError(\n                    \"Tried to split a class or function. It did not work. Error comes from: \\n```\\n\"\n                    + observed_code_header\n                    + \"\\n```\\n\"\n                )\n\n            try:\n                theoretical_name = re_pattern.search(theoretical_code_header).groups()[0]\n            except Exception:\n                raise ValueError(\n                    \"Tried to split a class or function. It did not work. Error comes from: \\n```\\n\"\n                    + theoretical_code_header\n                    + \"\\n```\\n\"\n                )\n            theoretical_code_header = theoretical_code_header.replace(theoretical_name, observed_obj_name)\n\n    # Find the first diff. Line 0 is special since we need to compare with the function/class names ignored.\n    diff_index = 0\n    if theoretical_code_header != observed_code_header:\n        return 0\n\n    diff_index = 1\n    for observed_line, theoretical_line in zip(observed_code.split(\"\\n\")[1:], theoretical_code.split(\"\\n\")[1:]):\n        if observed_line != theoretical_line:\n            return diff_index\n\n\nSUMMARY:\n**Function Summary**\n\n`check_codes_match`\n\nChecks if two versions of a code match with the exception of class/function name.\n\n**Purpose and Functionality**\n\nCompares two code strings, ignoring class/function names. Returns the index of the first line where there is a difference (if any) or `None` if they match.\n\n**Key Implementation Details**\n\n* Uses regular expressions to extract class/function names from header lines.\n* Compares each line in the code separately, ignoring the extracted names.\n\n**Notable Dependencies or Requirements**\n\n* Python's `re` module for regex pattern matching\n* Python's `Optional` type hinting\n\n**Return Value**\n\n`Optional[int]`: The index of the first difference or `None` (no difference)\n\n**Method Signatures and Interfaces**\n\n* `_re_class_match` and `_re_func_match` regular expression patterns for class/function name extraction.\n\nNote: This summary focuses on the most important information about the function, omitting minor details."}


created_at: 2025-02-05 09:13:50
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py:640-830
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 830, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py", "name": "is_copy_consistent", "start_line": 640, "type": "function", "chroma:document": "CODE:\n\ndef is_copy_consistent(filename: str, overwrite: bool = False, buffer: dict = None) -> Optional[List[Tuple[str, int]]]:\n    \"\"\"\n    Check if the code commented as a copy in a file matches the original.\n\n    Args:\n        filename (`str`):\n            The name of the file to check.\n        overwrite (`bool`, *optional*, defaults to `False`):\n            Whether or not to overwrite the copies when they don't match.\n        buffer (`dict`, *optional*):\n            The buffer used to store the previous results in order to speed up the process.\n\n    Returns:\n        `Optional[List[Tuple[str, int]]]`: If `overwrite=False`, returns the list of differences as tuples `(str, int)`\n        with the name of the object having a diff and the line number where theere is the first diff.\n    \"\"\"\n    base_path = TRANSFORMERS_PATH if not filename.startswith(\"tests\") else MODEL_TEST_PATH\n\n    with open(filename, \"r\", encoding=\"utf-8\", newline=\"\\n\") as f:\n        lines = f.readlines()\n    diffs = []\n    line_index = 0\n    # Not a for loop cause `lines` is going to change (if `overwrite=True`).\n    search_re = _re_copy_warning_for_test_file if filename.startswith(\"tests\") else _re_copy_warning\n    while line_index < len(lines):\n        search = search_re.search(lines[line_index])\n        if search is None:\n            line_index += 1\n            continue\n\n        # There is some copied code here, let's retrieve the original.\n        indent, object_name, replace_pattern = search.groups()\n\n        # Find the file lines, the object's code, and its blocks\n        try:\n            target_lines, theoretical_code, theoretical_code_splits = find_code_and_splits(\n                object_name, base_path, buffer=buffer\n            )\n        except Exception as exc:\n            exc.args = (f\"Error while trying to find source code for {filename}.\\n\\n\" + str(exc),)\n            raise\n\n        # code replaced by the patterns\n        theoretical_code_blocks = OrderedDict()\n        for name, start, end in theoretical_code_splits:\n            name = replace_code(name, replace_pattern)\n            code = \"\".join(target_lines[start:end])\n            code = replace_code(code, replace_pattern)\n            theoretical_code_blocks[name] = code\n\n        theoretical_indent = get_indent(theoretical_code)\n\n        # `start_index` is the index of the first line (the definition header) after `# Copied from`.\n        # (`indent != theoretical_indent` doesn't seem to occur so far, not sure what this case is for.)\n        start_index = line_index + 1 if indent == theoretical_indent else line_index\n        # enter the block body\n        line_index = start_index + 1\n\n        subcode = \"\\n\".join(theoretical_code.split(\"\\n\")[1:])\n        indent = get_indent(subcode)\n        # Loop to check the observed code, stop when indentation diminishes or if we see a End copy comment.\n        # We can't call `find_block_end` directly as there is sth. special `# End copy\"` here.\n        should_continue = True\n        while line_index < len(lines) and should_continue:\n            line_index += 1\n            if line_index >= len(lines):\n                break\n            line = lines[line_index]\n            # There is a special pattern `# End copy` to stop early. It's not documented cause it shouldn't really be\n            # used.\n            should_continue = _should_continue(line, indent) and re.search(f\"^{indent}# End copy\", line) is None\n        # `line_index` is outside the block\n        # Clean up empty lines at the end (if any).\n        while len(lines[line_index - 1]) <= 1:\n            line_index -= 1\n\n        # Split the observed code into blocks\n        observed_code_splits = split_code_into_blocks(lines, start_index, line_index, len(indent), backtrace=True)\n\n        is_class = lines[start_index].startswith(f\"{' ' * (len(indent) - 4)}class \")\n        # sanity check\n        _sanity_check_splits(theoretical_code_splits, observed_code_splits, is_class=is_class, filename=filename)\n\n        # observed code in a structured way (a dict mapping block names to blocks' code)\n        observed_code_blocks = OrderedDict()\n        for name, start, end in observed_code_splits:\n            code = \"\".join(lines[start:end])\n            observed_code_blocks[name] = code\n\n        # Below, we change some names in `theoretical_code_blocks` and `observed_code_blocks`. These mappings map the\n        # original names to the modified names: this is used to restore the original order of the code blocks.\n        name_mappings_1 = {k: k for k in theoretical_code_blocks.keys()}\n        name_mappings_2 = {k: k for k in observed_code_blocks.keys()}\n\n        # Update code blocks' name and content:\n        #   If `\"# Ignore copy\"` is found in a block of the observed code:\n        #     1. if it's a block only in the observed code --> add it to the theoretical code.\n        #     2. if it's also in the theoretical code () --> put its content (body) to the corresponding block under the\n        #        same name in the theoretical code.\n        #   In both cases, we change the name to have a prefix `_ignored_` so we know if we can discard them during the\n        #   comparison.\n        ignored_existing_block_index = 0\n        ignored_new_block_index = 0\n        for name in list(observed_code_blocks.keys()):\n            code = observed_code_blocks[name]\n            if \"# Ignore copy\" in code:\n                if name in theoretical_code_blocks:\n                    # in the target --> just copy the content\n                    del theoretical_code_blocks[name]\n                    theoretical_code_blocks[f\"_ignored_existing_block_{ignored_existing_block_index}\"] = code\n                    name_mappings_1[name] = f\"_ignored_existing_block_{ignored_existing_block_index}\"\n\n                    del observed_code_blocks[name]\n                    observed_code_blocks[f\"_ignored_existing_block_{ignored_existing_block_index}\"] = code\n                    name_mappings_2[name] = f\"_ignored_existing_block_{ignored_existing_block_index}\"\n                    ignored_existing_block_index += 1\n                else:\n                    # not in the target --> add it\n                    theoretical_code_blocks[f\"_ignored_new_block_{ignored_new_block_index}\"] = code\n                    name_mappings_1[f\"_ignored_new_block_{ignored_new_block_index}\"] = (\n                        f\"_ignored_new_block_{ignored_new_block_index}\"\n                    )\n\n                    del observed_code_blocks[name]\n                    observed_code_blocks[f\"_ignored_new_block_{ignored_new_block_index}\"] = code\n                    name_mappings_2[name] = f\"_ignored_new_block_{ignored_new_block_index}\"\n                    ignored_new_block_index += 1\n\n        # Respect the original block order:\n        #   1. in `theoretical_code_blocks`: the new blocks will follow the existing ones\n        #   2. in `observed_code_blocks`: the original order are kept with names modified potentially. This is necessary\n        #      to compute the correct `diff_index` if `overwrite=True` and there is a diff.\n        theoretical_code_blocks = {\n            name_mappings_1[orig_name]: theoretical_code_blocks[name_mappings_1[orig_name]]\n            for orig_name in name_mappings_1\n        }\n        observed_code_blocks = {\n            name_mappings_2[orig_name]: observed_code_blocks[name_mappings_2[orig_name]]\n            for orig_name in name_mappings_2\n        }\n\n        # Ignore the blocks specified to be ignored. This is the version used to check if there is a mismatch\n        theoretical_code_blocks_clean = {\n            k: v\n            for k, v in theoretical_code_blocks.items()\n            if not (k.startswith((\"_ignored_existing_block_\", \"_ignored_new_block_\")))\n        }\n        theoretical_code = \"\".join(list(theoretical_code_blocks_clean.values()))\n\n        # stylify `theoretical_code` before compare (this is needed only when `replace_pattern` is not empty)\n        if replace_pattern:\n            theoretical_code = stylify(theoretical_code)\n        # Remove `\\n\\n` in `theoretical_code` before compare (so no empty line)\n        while \"\\n\\n\" in theoretical_code:\n            theoretical_code = theoretical_code.replace(\"\\n\\n\", \"\\n\")\n\n        # Compute `observed_code` where we don't include any empty line + keep track the line index between the\n        # original/processed `observed_code` so we can have the correct `diff_index`.\n        idx_to_orig_idx_mapping_for_observed_code_lines = {}\n        idx = -1\n        orig_idx = -1\n        observed_code = \"\"\n        for name, code in observed_code_blocks.items():\n            if code.endswith(\"\\n\"):\n                code = code[:-1]\n            for code_line in code.split(\"\\n\"):\n                orig_idx += 1\n                if code_line.strip() and not name.startswith((\"_ignored_existing_block_\", \"_ignored_new_block_\")):\n                    idx += 1\n                    observed_code += code_line + \"\\n\"\n                    idx_to_orig_idx_mapping_for_observed_code_lines[idx] = orig_idx\n\n        # Test for a diff and act accordingly.\n        diff_index = check_codes_match(observed_code, theoretical_code)\n        if diff_index is not None:\n            # switch to the index in the original `observed_code` (i.e. before removing empty lines)\n            diff_index = idx_to_orig_idx_mapping_for_observed_code_lines[diff_index]\n            diffs.append([object_name, diff_index + start_index + 1])\n            if overwrite:\n                # `theoretical_code_to_write` is a single string but may have several lines.\n                theoretical_code_to_write = stylify(\"\".join(list(theoretical_code_blocks.values())))\n                lines = lines[:start_index] + [theoretical_code_to_write] + lines[line_index:]\n                # Here we treat it as a single entry in `lines`.\n                line_index = start_index + 1\n\n    if overwrite and len(diffs) > 0:\n        # Warn the user a file has been modified.\n        print(f\"Detected changes, rewriting {filename}.\")\n        with open(filename, \"w\", encoding=\"utf-8\", newline=\"\\n\") as f:\n            f.writelines(lines)\n\n\nSUMMARY:\nThis is a function that compares two versions of a code file. It checks if the code has been modified and reports changes. The function takes into account various factors such as:\n\n* Overwriting existing files\n* Handling empty lines and whitespace differences\n* Respecting the original block order when ignoring certain blocks\n* Stylifying codes for comparison\n\nHere's a brief overview of what the function does:\n\n1. It reads two versions of the code file, comparing them line by line.\n2. If there are changes between the two versions, it reports these changes and tracks their corresponding indices in both versions.\n3. Depending on whether to overwrite existing files or not, it either makes the necessary modifications or simply returns a list of reported differences.\n4. It also provides an option for modifying the code file in-place if there are any detected changes.\n\nThe function aims to be efficient and accurate when comparing two versions of code and can handle a range of scenarios, from minor changes to major rewrites."}


created_at: 2025-02-05 09:13:50
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py:833-863
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 863, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py", "name": "check_copies", "start_line": 833, "type": "function", "chroma:document": "CODE:\n\ndef check_copies(overwrite: bool = False, file: str = None):\n    \"\"\"\n    Check every file is copy-consistent with the original. Also check the model list in the main README and other\n    READMEs are consistent.\n\n    Args:\n        overwrite (`bool`, *optional*, defaults to `False`):\n            Whether or not to overwrite the copies when they don't match.\n        file (`bool`, *optional*):\n            The path to a specific file to check and/or fix.\n    \"\"\"\n    buffer = {}\n\n    if file is None:\n        all_files = glob.glob(os.path.join(TRANSFORMERS_PATH, \"**/*.py\"), recursive=True)\n        all_test_files = glob.glob(os.path.join(MODEL_TEST_PATH, \"**/*.py\"), recursive=True)\n        all_files = list(all_files) + list(all_test_files)\n    else:\n        all_files = [file]\n\n    diffs = []\n    for filename in all_files:\n        new_diffs = is_copy_consistent(filename, overwrite, buffer)\n        diffs += [f\"- {filename}: copy does not match {d[0]} at line {d[1]}\" for d in new_diffs]\n    if not overwrite and len(diffs) > 0:\n        diff = \"\\n\".join(diffs)\n        raise Exception(\n            \"Found the following copy inconsistencies:\\n\"\n            + diff\n            + \"\\nRun `make fix-copies` or `python utils/check_copies.py --fix_and_overwrite` to fix them.\"\n\n\nSUMMARY:\n**Summary**\n\nThe `check_copies` function ensures that all files in specified directories (`TRANSFORMERS_PATH` and `MODEL_TEST_PATH`) are copy-consistent with their original versions. It checks for discrepancies by calling the `is_copy_consistent` function and reports any inconsistencies via an exception.\n\n**Key Takeaways**\n\n* Function: `check_copies`\n* Purpose: Validate file copies and report inconsistencies\n* Parameters:\n\t+ `overwrite`: Boolean flag to overwrite discrepancies (default: False)\n\t+ `file`: Optional path to a single file for manual inspection (defaults to None)\n\n**Implementation Details**\n\n* Iterates through a list of files in specified directories (`all_files`)\n* Uses `is_copy_consistent` function to check copy consistency for each file\n* Reports inconsistencies via an exception if overwrite is False and discrepancies are found\n\n**Notable Dependencies/Interfaces**\n\n* `os.path.join`\n* `glob`\n* ` TRANSFORMERS_PATH`\n* `MODEL_TEST_PATH`\n* `is_copy_consistent` (function, dependencies not specified)"}


created_at: 2025-02-05 09:13:50
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py:866-894
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 894, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py", "name": "check_full_copies", "start_line": 866, "type": "function", "chroma:document": "CODE:\n\ndef check_full_copies(overwrite: bool = False):\n    \"\"\"\n    Check the files that are full copies of others (as indicated in `FULL_COPIES`) are copy-consistent.\n\n    Args:\n        overwrite (`bool`, *optional*, defaults to `False`):\n            Whether or not to overwrite the copies when they don't match.\n    \"\"\"\n    diffs = []\n    for target, source in FULL_COPIES.items():\n        with open(source, \"r\", encoding=\"utf-8\") as f:\n            source_code = f.read()\n        with open(target, \"r\", encoding=\"utf-8\") as f:\n            target_code = f.read()\n        if source_code != target_code:\n            if overwrite:\n                with open(target, \"w\", encoding=\"utf-8\") as f:\n                    print(f\"Replacing the content of {target} by the one of {source}.\")\n                    f.write(source_code)\n            else:\n                diffs.append(f\"- {target}: copy does not match {source}.\")\n\n    if not overwrite and len(diffs) > 0:\n        diff = \"\\n\".join(diffs)\n        raise Exception(\n            \"Found the following copy inconsistencies:\\n\"\n            + diff\n            + \"\\nRun `make fix-copies` or `python utils/check_copies.py --fix_and_overwrite` to fix them.\"\n\n\nSUMMARY:\n**Function Summary: check_full_copies**\n\n Checks files indicated as full copies (`FULL_COPIES`) for consistency and raises an exception if the copies differ.\n\n**Functionality**\n\n* Compares source code with target copy\n* If the target copy differs, considers overwriting the target (if `overwrite` is True) or logs a difference (if `overwrite` is False)\n* Rises an exception if no overwrites is made and differences are found\n\n**Key Implementation Details:**\n\n* Uses dictionary `FULL_COPIES` to map source files to their corresponding target files\n* Reads file contents using open() with encoding=\"utf-8\"\n* Compares source code (`source_code`) with target code (`target_code`)\n* If the copies differ, uses a list to track differences (`diffs`)\n* Uses exception handling to raise an error if no overwrites is made and differences are found\n\n**Notable Dependencies/Requirements:**\n\n* Python 3.7+ (encoding support)\n* Makefile or Python script (utils/check_copies.py)\n\n**Important Method Signatures and Interfaces:**\n\n* `check_full_copies()`: Function to check files for consistency\n\t+ `overwrite**: Boolean flag to control overwrite behavior\n\t+ `FULL_COPIES`: Dictionary mapping source files to target files"}


created_at: 2025-02-05 09:13:50
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py:897-933
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 933, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py", "name": "get_model_list", "start_line": 897, "type": "function", "chroma:document": "CODE:\n\ndef get_model_list(filename: str, start_prompt: str, end_prompt: str) -> str:\n    \"\"\"\n    Extracts the model list from a README.\n\n    Args:\n        filename (`str`): The name of the README file to check.\n        start_prompt (`str`): The string to look for that introduces the model list.\n        end_prompt (`str`): The string to look for that ends the model list.\n\n    Returns:\n        `str`: The model list.\n    \"\"\"\n    with open(os.path.join(REPO_PATH, filename), \"r\", encoding=\"utf-8\", newline=\"\\n\") as f:\n        lines = f.readlines()\n    # Find the start of the list.\n    start_index = 0\n    while not lines[start_index].startswith(start_prompt):\n        start_index += 1\n    start_index += 1\n\n    result = []\n    current_line = \"\"\n    end_index = start_index\n\n    # Keep going until the end of the list.\n    while not lines[end_index].startswith(end_prompt):\n        if lines[end_index].startswith(\"1.\"):\n            if len(current_line) > 1:\n                result.append(current_line)\n            current_line = lines[end_index]\n        elif len(lines[end_index]) > 1:\n            current_line = f\"{current_line[:-1]} {lines[end_index].lstrip()}\"\n        end_index += 1\n    if len(current_line) > 1:\n        result.append(current_line)\n\n\n\nSUMMARY:\n**Function Summary:**\n\n`get_model_list`: Extracts and returns the model list from a README file.\n\n**Purpose**: Finds the start and end markers of the model list in the specified file, reads between them, and returns the extracted list as a string.\n\n**Key Implementation Details:**\n\n* Reads the entire file into memory.\n* Uses two prompts (`start_prompt` and `end_prompt`) to determine the start and end indices of the model list.\n* Iterates through lines until it finds the end prompt, appending each line that starts with \"1.\" (presumably a model name) or has additional content beyond a single character.\n* Handles cases where the current line has more than one character after processing.\n\n**Dependencies and Interfaces:**\n\n* `os`: for file path manipulation\n* `REPO_PATH` (not shown in this code snippet): likely an environment variable or global constant containing the repository root directory."}


created_at: 2025-02-05 09:13:50
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py:936-1023
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 1023, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py", "name": "convert_to_localized_md", "start_line": 936, "type": "function", "chroma:document": "CODE:\n\ndef convert_to_localized_md(model_list: str, localized_model_list: str, format_str: str) -> Tuple[bool, str]:\n    \"\"\"\n    Compare the model list from the main README to the one in a localized README.\n\n    Args:\n        model_list (`str`): The model list in the main README.\n        localized_model_list (`str`): The model list in one of the localized README.\n        format_str (`str`):\n            The template for a model entry in the localized README (look at the `format_model_list` in the entries of\n            `LOCALIZED_READMES` for examples).\n\n    Returns:\n        `Tuple[bool, str]`: A tuple where the first value indicates if the READMEs match or not, and the second value\n        is the correct localized README.\n    \"\"\"\n\n    def _rep(match):\n        title, model_link, paper_affiliations, paper_title_link, paper_authors, supplements = match.groups()\n        return format_str.format(\n            title=title,\n            model_link=model_link,\n            paper_affiliations=paper_affiliations,\n            paper_title_link=paper_title_link,\n            paper_authors=paper_authors,\n            supplements=\" \" + supplements.strip() if len(supplements) != 0 else \"\",\n        )\n\n    # This regex captures metadata from an English model description, including model title, model link,\n    # affiliations of the paper, title of the paper, authors of the paper, and supplemental data (see DistilBERT for\n    # example).\n    _re_capture_meta = re.compile(\n        r\"\\*\\*\\[([^\\]]*)\\]\\(([^\\)]*)\\)\\*\\* \\(from ([^)]*)\\)[^\\[]*([^\\)]*\\)).*?by (.*?[A-Za-z\\*]{2,}?)\\. (.*)$\"\n    )\n    # This regex is used to synchronize title link.\n    _re_capture_title_link = re.compile(r\"\\*\\*\\[([^\\]]*)\\]\\(([^\\)]*)\\)\\*\\*\")\n    # This regex is used to synchronize paper title and link.\n    _re_capture_paper_link = re.compile(r\" \\[([^\\]]*)\\]\\(([^\\)]*)\\)\")\n\n    if len(localized_model_list) == 0:\n        localized_model_index = {}\n    else:\n        try:\n            localized_model_index = {\n                re.search(r\"\\*\\*\\[([^\\]]*)\", line).groups()[0]: line\n                for line in localized_model_list.strip().split(\"\\n\")\n            }\n        except AttributeError:\n            raise AttributeError(\"A model name in localized READMEs cannot be recognized.\")\n\n    model_keys = [re.search(r\"\\*\\*\\[([^\\]]*)\", line).groups()[0] for line in model_list.strip().split(\"\\n\")]\n\n    # We exclude keys in localized README not in the main one.\n    readmes_match = not any(k not in model_keys for k in localized_model_index)\n    localized_model_index = {k: v for k, v in localized_model_index.items() if k in model_keys}\n\n    for model in model_list.strip().split(\"\\n\"):\n        title, model_link = _re_capture_title_link.search(model).groups()\n        if title not in localized_model_index:\n            readmes_match = False\n            # Add an anchor white space behind a model description string for regex.\n            # If metadata cannot be captured, the English version will be directly copied.\n            localized_model_index[title] = _re_capture_meta.sub(_rep, model + \" \")\n        elif _re_fill_pattern.search(localized_model_index[title]) is not None:\n            update = _re_capture_meta.sub(_rep, model + \" \")\n            if update != localized_model_index[title]:\n                readmes_match = False\n                localized_model_index[title] = update\n        else:\n            # Synchronize title link\n            converted_model = _re_capture_title_link.sub(\n                f\"**[{title}]({model_link})**\", localized_model_index[title], count=1\n            )\n\n            # Synchronize paper title and its link (if found)\n            paper_title_link = _re_capture_paper_link.search(model)\n            if paper_title_link is not None:\n                paper_title, paper_link = paper_title_link.groups()\n                converted_model = _re_capture_paper_link.sub(\n                    f\" [{paper_title}]({paper_link})\", converted_model, count=1\n                )\n\n            if converted_model != localized_model_index[title]:\n                readmes_match = False\n                localized_model_index[title] = converted_model\n\n    sorted_index = sorted(localized_model_index.items(), key=lambda x: x[0].lower())\n\n\n\nSUMMARY:\n**Function Summary**\n\n`convert_to_localized_md`\n\nCompares model list from main README to localized README and updates the latter with synchronized metadata.\n\n**Main Purpose and Functionality**\n\nThis function compares two model lists, one from the main README and another from a localized README. It synchronizes metadata such as titles, model links, paper affiliations, and authors between the two lists.\n\n**Key Implementation Details**\n\n* Uses regular expressions to capture metadata from English model descriptions.\n* Checks for matching models in both READMEs and updates the localized one if necessary.\n* Synchronizes title link, paper title, and its link (if found) using regex patterns.\n\n**Notable Dependencies or Requirements**\n\n* `re` module for regular expression operations.\n* `Tuple` type hint to indicate return value format.\n\nNote: The original function has been preserved with the most important details. Note that functions like `format_model_list`, `LOCALIZED_READMES`, and `_re_capture_title_link` are not directly analyzed here as they appear to be referenced components instead of internal implementation details."}


created_at: 2025-02-05 09:13:50
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py:1026-1057
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 1057, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_copies.py", "name": "_find_text_in_file", "start_line": 1026, "type": "function", "chroma:document": "CODE:\n\ndef _find_text_in_file(filename: str, start_prompt: str, end_prompt: str) -> Tuple[str, int, int, List[str]]:\n    \"\"\"\n    Find the text in a file between two prompts.\n\n    Args:\n        filename (`str`): The name of the file to look into.\n        start_prompt (`str`): The string to look for that introduces the content looked for.\n        end_prompt (`str`): The string to look for that ends the content looked for.\n\n    Returns:\n        Tuple[str, int, int, List[str]]: The content between the two prompts, the index of the start line in the\n        original file, the index of the end line in the original file and the list of lines of that file.\n    \"\"\"\n    with open(filename, \"r\", encoding=\"utf-8\", newline=\"\\n\") as f:\n        lines = f.readlines()\n    # Find the start prompt.\n    start_index = 0\n    while not lines[start_index].startswith(start_prompt):\n        start_index += 1\n    start_index += 1\n\n    end_index = start_index\n    while not lines[end_index].startswith(end_prompt):\n        end_index += 1\n    end_index -= 1\n\n    while len(lines[start_index]) <= 1:\n        start_index += 1\n    while len(lines[end_index]) <= 1:\n        end_index -= 1\n    end_index += 1\n\n\nSUMMARY:\n**Function Summary:**\n\n`_find_text_in_file` function finds text in a file between two prompts.\n```python\nimport typing as t\nfrom typing import Tuple, List\n\ndef _find_text_in_file(filename: str, start_prompt: str, end_prompt: str) -> Tuple[str, int, int, List[str]]:\n    \"\"\"\n    Args:\n        filename (str): File to search in.\n        start_prompt (str): String introducing the content searched for.\n        end_prompt (str): String marking the end of the content searched for.\n\n    Returns:\n        A tuple containing:\n            - The extracted text.\n            - The index of the start line in the original file.\n            - The index of the end line in the original file.\n            - The list of all lines in the file.\n    \"\"\"\n```\n\n**Key Implementation Details:**\n\n* Finds start and end indices by searching for prompt strings.\n* Handling cases where prompts may be on empty lines or follow multiple empty lines.\n\n**Notable Dependencies and Requirements:**\n\n* `open` function from Python's standard library for reading files.\n* `re` module is not explicitly imported, but it might be implicitly used through string methods."}


created_at: 2025-02-05 09:13:50
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_self_hosted_runner.py:5-28
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 28, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_self_hosted_runner.py", "name": "get_runner_status", "start_line": 5, "type": "function", "chroma:document": "CODE:\n\ndef get_runner_status(target_runners, token):\n    offline_runners = []\n\n    cmd = (\n        f'curl -H \"Accept: application/vnd.github+json\" -H \"Authorization: Bearer {token}\"'\n        \" https://api.github.com/repos/huggingface/transformers/actions/runners\"\n    )\n    output = subprocess.run(cmd, shell=True, stdout=subprocess.PIPE)\n    o = output.stdout.decode(\"utf-8\")\n    status = json.loads(o)\n\n    runners = status[\"runners\"]\n    for runner in runners:\n        if runner[\"name\"] in target_runners:\n            if runner[\"status\"] == \"offline\":\n                offline_runners.append(runner)\n\n    # save the result so we can report them on Slack\n    with open(\"offline_runners.txt\", \"w\") as fp:\n        fp.write(json.dumps(offline_runners))\n\n    if len(offline_runners) > 0:\n        failed = \"\\n\".join([x[\"name\"] for x in offline_runners])\n\n\nSUMMARY:\n**Code Summary**\n\n### Function Purpose\n\n`get_runner_status` retrieves the status of GitHub Actions runners from the GitHub API.\n\n### Key Implementation Details\n\n* Fetches runner status using `curl` and Parses the response with JSON\n* Compares runner names against `target_runners` list to retrieve offline runnersonly\n* Saves result to `offline_runners.txt`\n\n### Notable Dependencies/Requirements\n\n* `github-api`: implicit, accessed via `https://api.github.com/repos/huggingface/transformers/actions/runners`\n* `json` and `subprocess` modules for standard I/O and execution\n* `Authorization` token for authentication (Bearer `token`)"}


created_at: 2025-02-05 09:13:50
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_support_list.py:37-61
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 61, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_support_list.py", "name": "check_flash_support_list", "start_line": 37, "type": "function", "chroma:document": "CODE:\n\ndef check_flash_support_list():\n    with open(os.path.join(REPO_PATH, \"docs/source/en/perf_infer_gpu_one.md\"), \"r\") as f:\n        doctext = f.read()\n\n        doctext = doctext.split(\"FlashAttention-2 is currently supported for the following architectures:\")[1]\n        doctext = doctext.split(\"You can request to add FlashAttention-2 support\")[0]\n\n    patterns = glob(os.path.join(REPO_PATH, \"src/transformers/models/**/modeling_*.py\"))\n    patterns_tf = glob(os.path.join(REPO_PATH, \"src/transformers/models/**/modeling_tf_*.py\"))\n    patterns_flax = glob(os.path.join(REPO_PATH, \"src/transformers/models/**/modeling_flax_*.py\"))\n    patterns = list(set(patterns) - set(patterns_tf) - set(patterns_flax))\n    archs_supporting_fa2 = []\n    for filename in patterns:\n        with open(filename, \"r\") as f:\n            text = f.read()\n\n            if \"_supports_flash_attn_2 = True\" in text:\n                model_name = os.path.basename(filename).replace(\".py\", \"\").replace(\"modeling_\", \"\")\n                archs_supporting_fa2.append(model_name)\n\n    for arch in archs_supporting_fa2:\n        if arch not in doctext:\n            raise ValueError(\n                f\"{arch} should be in listed in the flash attention documentation but is not. Please update the documentation.\"\n\n\nSUMMARY:\n**Function Summary: `check_flash_support_list`**\n\nPurpose:\nThis function checks if FlashAttention-2 (FA2) support is included for various architectures based on documentation and code patterns.\n\nMain Functionality:\n\n1. Reads documentation text to extract supported architectures.\n2. Searches for model files with FA2 support in `modeling*` pattern and initializes a list of supported architectures (`archs_supporting_fa2`).\n3. Compares the extracted supported architectures against the ones listed in FA2 documentation and raises a ValueError if any mismatch.\n\nKey Implementation Details:\n\n- Uses glob to search for files under `src/transformers/models` with models containing \"_supports_flash_attn_2 = True\".\n- Utilizes string splitting to extract relevant text from documentation files.\n- Compares lists of supported architectures using set operations.\n\nNotable Dependencies and Requirements:\n- `os.path.join` and `open` functions for file system interactions\n- `glob` function for pattern matching\n- `REPO_PATH` constant referencing the repository's root path\n- External dependency: FlashAttention documentation (`perf_infer_gpu_one.md`) and model files containing \"_supports_flash_attn_2 = True\"."}


created_at: 2025-02-05 09:13:50
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_support_list.py:64-91
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 91, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_support_list.py", "name": "check_sdpa_support_list", "start_line": 64, "type": "function", "chroma:document": "CODE:\n\ndef check_sdpa_support_list():\n    with open(os.path.join(REPO_PATH, \"docs/source/en/perf_infer_gpu_one.md\"), \"r\") as f:\n        doctext = f.read()\n\n        doctext = doctext.split(\n            \"For now, Transformers supports SDPA inference and training for the following architectures:\"\n        )[1]\n        doctext = doctext.split(\"Note that FlashAttention can only be used for models using the\")[0]\n        doctext = doctext.lower()\n\n    patterns = glob(os.path.join(REPO_PATH, \"src/transformers/models/**/modeling_*.py\"))\n    patterns_tf = glob(os.path.join(REPO_PATH, \"src/transformers/models/**/modeling_tf_*.py\"))\n    patterns_flax = glob(os.path.join(REPO_PATH, \"src/transformers/models/**/modeling_flax_*.py\"))\n    patterns = list(set(patterns) - set(patterns_tf) - set(patterns_flax))\n    archs_supporting_sdpa = []\n    for filename in patterns:\n        with open(filename, \"r\") as f:\n            text = f.read()\n\n            if \"_supports_sdpa = True\" in text:\n                model_name = os.path.basename(filename).replace(\".py\", \"\").replace(\"modeling_\", \"\")\n                archs_supporting_sdpa.append(model_name)\n\n    for arch in archs_supporting_sdpa:\n        if not any(term in doctext for term in [arch, arch.replace(\"_\", \"-\"), arch.replace(\"_\", \" \")]):\n            raise ValueError(\n                f\"{arch} should be in listed in the SDPA documentation but is not. Please update the documentation.\"\n\n\nSUMMARY:\n**Functionality Summary:**\n\nThe `check_sdpa_support_list` function checks if various Transformer models support SDPA (Software Distributed Performance Architecture) inference and training.\n\n### Main Purpose:\n\n1. Retrieves documentation text from a file (`docs/source/en/perf_infer_gpu_one.md`) to determine which architectures are supported by SDPA.\n2. Iterates through model files in the `src/transformers/models` directory and extracts model names that contain the line `_supports_sdpa = True`.\n3. Verifies if each detected model is listed in the retrieved documentation text, raising a ValueError if any are not found.\n\n### Key Implementation Details:**\n\n* The function uses regular expressions to extract specific lines from the documentation text.\n* It utilizes glob to find model files matching certain naming patterns (`modeling_*.py`) and excludes files using `FlashAttention`.\n* Model names are appended to a list (`archs_supporting_sdpa`) based on the presence of the `_supports_sdpa = True` line.\n\n### Notable Dependencies and Requirements:**\n\n* The function relies on the following external dependencies:\n\t+ `os.path.join()` for path manipulation\n\t+ `fnmatch` for glob pattern matching (not explicitly required, as Python's built-in `glob` module is used)\n\t+ `re` for regular expression operations (not explicitly required, but implied by string splitting and searching)\n\n### Contextual Considerations:\n\nThis function appears to be part of a automated documentation checking system, verifying the support list of various Transformer models against actual implementation."}


created_at: 2025-02-05 09:13:50
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/notification_service.py:51-67
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 67, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/notification_service.py", "name": "handle_test_results", "start_line": 51, "type": "function", "chroma:document": "CODE:\n\ndef handle_test_results(test_results):\n    expressions = test_results.split(\" \")\n\n    failed = 0\n    success = 0\n\n    # When the output is short enough, the output is surrounded by = signs: \"== OUTPUT ==\"\n    # When it is too long, those signs are not present.\n    time_spent = expressions[-2] if \"=\" in expressions[-1] else expressions[-1]\n\n    for i, expression in enumerate(expressions):\n        if \"failed\" in expression:\n            failed += int(expressions[i - 1])\n        if \"passed\" in expression:\n            success += int(expressions[i - 1])\n\n\n\nSUMMARY:\n**Summary:**\n\n`handle_test_results` function processes test results by:\n\n* Splitting into individual expressions containing \"failed\" and \"passed\"\n* Calculating total failed and success tests count\n* Extracting time spent in the output string, assuming it's surrounded by \"= =\" for short outputs\n\n**Implementation Details:**\n\n* Iterates through expressions using `enumerate`\n* Uses exact method signaturs to access values: `int(expression[i - 1])` \n\n**Dependencies/Requirements:**\n\n* Presumes test results are comma-separated strings\n* Expects output string to contain a specific format (`\"{value} = {expression}\"`)\n* No external dependencies listed, but assumes `test_results` input is valid."}


created_at: 2025-02-05 09:13:50
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/notification_service.py:70-88
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 88, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/notification_service.py", "name": "handle_stacktraces", "start_line": 70, "type": "function", "chroma:document": "CODE:\n\ndef handle_stacktraces(test_results):\n    # These files should follow the following architecture:\n    # === FAILURES ===\n    # <path>:<line>: Error ...\n    # <path>:<line>: Error ...\n    # <empty line>\n\n    total_stacktraces = test_results.split(\"\\n\")[1:-1]\n    stacktraces = []\n    for stacktrace in total_stacktraces:\n        try:\n            line = stacktrace[: stacktrace.index(\" \")].split(\":\")[-2]\n            error_message = stacktrace[stacktrace.index(\" \") :]\n\n            stacktraces.append(f\"(line {line}) {error_message}\")\n        except Exception:\n            stacktraces.append(\"Cannot retrieve error message.\")\n\n\n\nSUMMARY:\n**Function Summary**\n\n`handle_stacktraces`: Processes test results to extract and format error messages from stacktraces.\n\n### Functionality\n\n* Takes `test_results` as input\n* Splits input into lines, excluding the first and last (header/footer)\n* Iterates through each line, extracting:\n\t+ Line number from path element before first space character\n\t+ Error message from remaining path elements \n* Returns a list of formatted error messages\n\n### Key Implementation Details\n\n* `total_stacktraces` is extracted using `split(\"\\n\")`\n* Uses indexing to extract line and error parts of each stacktrace\n* Catches exceptions during processing, appending \"Cannot retrieve error message.\" if an error occurs\n\n### Notable Dependencies\n\n* No external dependencies referenced\n* Assumes input test results are structured according to specified architecture"}


created_at: 2025-02-05 09:13:50
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/notification_service.py:91-100
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 100, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/notification_service.py", "name": "dicts_to_sum", "start_line": 91, "type": "function", "chroma:document": "CODE:\n\ndef dicts_to_sum(objects: Union[Dict[str, Dict], List[dict]]):\n    if isinstance(objects, dict):\n        lists = objects.values()\n    else:\n        lists = objects\n\n    # Convert each dictionary to counter\n    counters = map(collections.Counter, lists)\n    # Sum all the counters\n\n\nSUMMARY:\n**Function Summary**\n\n`dicts_to_sum` function:\n\n* Takes a single `objects` parameter with two acceptable inputs:\n\t+ A dictionary of dictionaries (`Dict[str, Dict]`)\n\t+ A list of dictionaries (`List[dict]`)\n* If input is a dictionary, it extracts the list of inner dictionaries.\n* It uses `Counter` from `collections` to count occurrences of each key-value pair in each dictionary and then maps this counter across all lists using `functools.reduce` with `operator.add`, summing up the results.\n\n**Key Implementation Details:**\n\n* Uses `collections.Counter` for efficient counting\n* Applies a cumulative reduction using `functools.reduce` and `operator.add`\n\n**Notable Dependencies/R Requirements:**\n\n* `unico` (not 'Union', assumed typo, corrected to `Union`, actual is not present in this code snippet)\n    * No explicit dependencies, but it utilizes:\n        + `collections.Counter`\n        + `functools.reduce`\n        + `operator.add`\n        + Assume a dictionary `Dict[str, Dict]` for conversion from other list format or use `List[dict]` as stated in the documentation"}


created_at: 2025-02-05 09:13:50
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/notification_service.py:103-824
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 824, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/notification_service.py", "name": "Message", "start_line": 103, "type": "class", "chroma:document": "CODE:\n\nclass Message:\n    def __init__(\n        self,\n        title: str,\n        ci_title: str,\n        model_results: Dict,\n        additional_results: Dict,\n        selected_warnings: List = None,\n        prev_ci_artifacts=None,\n    ):\n        self.title = title\n        self.ci_title = ci_title\n\n        # Failures and success of the modeling tests\n        self.n_model_success = sum(r[\"success\"] for r in model_results.values())\n        self.n_model_single_gpu_failures = sum(dicts_to_sum(r[\"failed\"])[\"single\"] for r in model_results.values())\n        self.n_model_multi_gpu_failures = sum(dicts_to_sum(r[\"failed\"])[\"multi\"] for r in model_results.values())\n\n        # Some suites do not have a distinction between single and multi GPU.\n        self.n_model_unknown_failures = sum(dicts_to_sum(r[\"failed\"])[\"unclassified\"] for r in model_results.values())\n        self.n_model_failures = (\n            self.n_model_single_gpu_failures + self.n_model_multi_gpu_failures + self.n_model_unknown_failures\n        )\n\n        # Failures and success of the additional tests\n        self.n_additional_success = sum(r[\"success\"] for r in additional_results.values())\n\n        if len(additional_results) > 0:\n            # `dicts_to_sum` uses `dicts_to_sum` which requires a non empty dictionary. Let's just add an empty entry.\n            all_additional_failures = dicts_to_sum([r[\"failed\"] for r in additional_results.values()])\n            self.n_additional_single_gpu_failures = all_additional_failures[\"single\"]\n            self.n_additional_multi_gpu_failures = all_additional_failures[\"multi\"]\n            self.n_additional_unknown_gpu_failures = all_additional_failures[\"unclassified\"]\n        else:\n            self.n_additional_single_gpu_failures = 0\n            self.n_additional_multi_gpu_failures = 0\n            self.n_additional_unknown_gpu_failures = 0\n\n        self.n_additional_failures = (\n            self.n_additional_single_gpu_failures\n            + self.n_additional_multi_gpu_failures\n            + self.n_additional_unknown_gpu_failures\n        )\n\n        # Results\n        self.n_failures = self.n_model_failures + self.n_additional_failures\n        self.n_success = self.n_model_success + self.n_additional_success\n        self.n_tests = self.n_failures + self.n_success\n\n        self.model_results = model_results\n        self.additional_results = additional_results\n\n        self.thread_ts = None\n\n        if selected_warnings is None:\n            selected_warnings = []\n        self.selected_warnings = selected_warnings\n\n        self.prev_ci_artifacts = prev_ci_artifacts\n\n    @property\n    def time(self) -> str:\n        all_results = [*self.model_results.values(), *self.additional_results.values()]\n        time_spent = [r[\"time_spent\"].split(\", \")[0] for r in all_results if len(r[\"time_spent\"])]\n        total_secs = 0\n\n        for time in time_spent:\n            time_parts = time.split(\":\")\n\n            # Time can be formatted as xx:xx:xx, as .xx, or as x.xx if the time spent was less than a minute.\n            if len(time_parts) == 1:\n                time_parts = [0, 0, time_parts[0]]\n\n            hours, minutes, seconds = int(time_parts[0]), int(time_parts[1]), float(time_parts[2])\n            total_secs += hours * 3600 + minutes * 60 + seconds\n\n        hours, minutes, seconds = total_secs // 3600, (total_secs % 3600) // 60, total_secs % 60\n        return f\"{int(hours)}h{int(minutes)}m{int(seconds)}s\"\n\n    @property\n    def header(self) -> Dict:\n        return {\"type\": \"header\", \"text\": {\"type\": \"plain_text\", \"text\": self.title}}\n\n    @property\n    def ci_title_section(self) -> Dict:\n        return {\"type\": \"section\", \"text\": {\"type\": \"mrkdwn\", \"text\": self.ci_title}}\n\n    @property\n    def no_failures(self) -> Dict:\n        return {\n            \"type\": \"section\",\n            \"text\": {\n                \"type\": \"plain_text\",\n                \"text\": f\"\ud83c\udf1e There were no failures: all {self.n_tests} tests passed. The suite ran in {self.time}.\",\n                \"emoji\": True,\n            },\n            \"accessory\": {\n                \"type\": \"button\",\n                \"text\": {\"type\": \"plain_text\", \"text\": \"Check Action results\", \"emoji\": True},\n                \"url\": f\"https://github.com/huggingface/transformers/actions/runs/{os.environ['GITHUB_RUN_ID']}\",\n            },\n        }\n\n    @property\n    def failures(self) -> Dict:\n        return {\n            \"type\": \"section\",\n            \"text\": {\n                \"type\": \"plain_text\",\n                \"text\": (\n                    f\"There were {self.n_failures} failures, out of {self.n_tests} tests.\\n\"\n                    f\"Number of model failures: {self.n_model_failures}.\\n\"\n                    f\"The suite ran in {self.time}.\"\n                ),\n                \"emoji\": True,\n            },\n            \"accessory\": {\n                \"type\": \"button\",\n                \"text\": {\"type\": \"plain_text\", \"text\": \"Check Action results\", \"emoji\": True},\n                \"url\": f\"https://github.com/huggingface/transformers/actions/runs/{os.environ['GITHUB_RUN_ID']}\",\n            },\n        }\n\n    @property\n    def warnings(self) -> Dict:\n        # If something goes wrong, let's avoid the CI report failing to be sent.\n        button_text = \"Check warnings (Link not found)\"\n        # Use the workflow run link\n        job_link = f\"https://github.com/huggingface/transformers/actions/runs/{os.environ['GITHUB_RUN_ID']}\"\n\n        for job in github_actions_jobs:\n            if \"Extract warnings in CI artifacts\" in job[\"name\"] and job[\"conclusion\"] == \"success\":\n                button_text = \"Check warnings\"\n                # Use the actual job link\n                job_link = job[\"html_url\"]\n                break\n\n        huggingface_hub_warnings = [x for x in self.selected_warnings if \"huggingface_hub\" in x]\n        text = f\"There are {len(self.selected_warnings)} warnings being selected.\"\n        text += f\"\\n{len(huggingface_hub_warnings)} of them are from `huggingface_hub`.\"\n\n        return {\n            \"type\": \"section\",\n            \"text\": {\n                \"type\": \"plain_text\",\n                \"text\": text,\n                \"emoji\": True,\n            },\n            \"accessory\": {\n                \"type\": \"button\",\n                \"text\": {\"type\": \"plain_text\", \"text\": button_text, \"emoji\": True},\n                \"url\": job_link,\n            },\n        }\n\n    @staticmethod\n    def get_device_report(report, rjust=6):\n        if \"single\" in report and \"multi\" in report:\n            return f\"{str(report['single']).rjust(rjust)} | {str(report['multi']).rjust(rjust)} | \"\n        elif \"single\" in report:\n            return f\"{str(report['single']).rjust(rjust)} | {'0'.rjust(rjust)} | \"\n        elif \"multi\" in report:\n            return f\"{'0'.rjust(rjust)} | {str(report['multi']).rjust(rjust)} | \"\n\n    @property\n    def category_failures(self) -> Dict:\n        model_failures = [v[\"failed\"] for v in self.model_results.values()]\n\n        category_failures = {}\n\n        for model_failure in model_failures:\n            for key, value in model_failure.items():\n                if key not in category_failures:\n                    category_failures[key] = dict(value)\n                else:\n                    category_failures[key][\"unclassified\"] += value[\"unclassified\"]\n                    category_failures[key][\"single\"] += value[\"single\"]\n                    category_failures[key][\"multi\"] += value[\"multi\"]\n\n        individual_reports = []\n        for key, value in category_failures.items():\n            device_report = self.get_device_report(value)\n\n            if sum(value.values()):\n                if device_report:\n                    individual_reports.append(f\"{device_report}{key}\")\n                else:\n                    individual_reports.append(key)\n\n        header = \"Single |  Multi | Category\\n\"\n        category_failures_report = prepare_reports(\n            title=\"The following modeling categories had failures\", header=header, reports=individual_reports\n        )\n\n        return {\"type\": \"section\", \"text\": {\"type\": \"mrkdwn\", \"text\": category_failures_report}}\n\n    def compute_diff_for_failure_reports(self, curr_failure_report, prev_failure_report):  # noqa\n        # Remove the leading and training parts that don't contain failure count information.\n        model_failures = curr_failure_report.split(\"\\n\")[3:-2]\n        prev_model_failures = prev_failure_report.split(\"\\n\")[3:-2]\n        entries_changed = set(model_failures).difference(prev_model_failures)\n\n        prev_map = {}\n        for f in prev_model_failures:\n            items = [x.strip() for x in f.split(\"| \")]\n            prev_map[items[-1]] = [int(x) for x in items[:-1]]\n\n        curr_map = {}\n        for f in entries_changed:\n            items = [x.strip() for x in f.split(\"| \")]\n            curr_map[items[-1]] = [int(x) for x in items[:-1]]\n\n        diff_map = {}\n        for k, v in curr_map.items():\n            if k not in prev_map:\n                diff_map[k] = v\n            else:\n                diff = [x - y for x, y in zip(v, prev_map[k])]\n                if max(diff) > 0:\n                    diff_map[k] = diff\n\n        entries_changed = []\n        for model_name, diff_values in diff_map.items():\n            diff = [str(x) for x in diff_values]\n            diff = [f\"+{x}\" if (x != \"0\" and not x.startswith(\"-\")) else x for x in diff]\n            diff = [x.rjust(9) for x in diff]\n            device_report = \" | \".join(diff) + \" | \"\n            report = f\"{device_report}{model_name}\"\n            entries_changed.append(report)\n        entries_changed = sorted(entries_changed, key=lambda s: s.split(\"| \")[-1])\n\n        return entries_changed\n\n    @property\n    def model_failures(self) -> List[Dict]:\n        # Obtain per-model failures\n        def per_model_sum(model_category_dict):\n            return dicts_to_sum(model_category_dict[\"failed\"].values())\n\n        failures = {}\n        non_model_failures = {\n            k: per_model_sum(v) for k, v in self.model_results.items() if sum(per_model_sum(v).values())\n        }\n\n        for k, v in self.model_results.items():\n            if k in NON_MODEL_TEST_MODULES:\n                pass\n\n            if sum(per_model_sum(v).values()):\n                dict_failed = dict(v[\"failed\"])\n                pytorch_specific_failures = dict_failed.pop(\"PyTorch\")\n                tensorflow_specific_failures = dict_failed.pop(\"TensorFlow\")\n                other_failures = dicts_to_sum(dict_failed.values())\n\n                failures[k] = {\n                    \"PyTorch\": pytorch_specific_failures,\n                    \"TensorFlow\": tensorflow_specific_failures,\n                    \"other\": other_failures,\n                }\n\n        model_reports = []\n        other_module_reports = []\n\n        for key, value in non_model_failures.items():\n            if key in NON_MODEL_TEST_MODULES:\n                device_report = self.get_device_report(value)\n\n                if sum(value.values()):\n                    if device_report:\n                        report = f\"{device_report}{key}\"\n                    else:\n                        report = key\n\n                    other_module_reports.append(report)\n\n        for key, value in failures.items():\n            device_report_values = [\n                value[\"PyTorch\"][\"single\"],\n                value[\"PyTorch\"][\"multi\"],\n                value[\"TensorFlow\"][\"single\"],\n                value[\"TensorFlow\"][\"multi\"],\n                sum(value[\"other\"].values()),\n            ]\n\n            if sum(device_report_values):\n                device_report = \" | \".join([str(x).rjust(9) for x in device_report_values]) + \" | \"\n                report = f\"{device_report}{key}\"\n\n                model_reports.append(report)\n\n        # (Possibly truncated) reports for the current workflow run - to be sent to Slack channels\n        model_header = \"Single PT |  Multi PT | Single TF |  Multi TF |     Other | Category\\n\"\n        sorted_model_reports = sorted(model_reports, key=lambda s: s.split(\"| \")[-1])\n        model_failures_report = prepare_reports(\n            title=\"These following model modules had failures\", header=model_header, reports=sorted_model_reports\n        )\n\n        module_header = \"Single |  Multi | Category\\n\"\n        sorted_module_reports = sorted(other_module_reports, key=lambda s: s.split(\"| \")[-1])\n        module_failures_report = prepare_reports(\n            title=\"The following non-model modules had failures\", header=module_header, reports=sorted_module_reports\n        )\n\n        # To be sent to Slack channels\n        model_failure_sections = [\n            {\"type\": \"section\", \"text\": {\"type\": \"mrkdwn\", \"text\": model_failures_report}},\n            {\"type\": \"section\", \"text\": {\"type\": \"mrkdwn\", \"text\": module_failures_report}},\n        ]\n\n        # Save the complete (i.e. no truncation) failure tables (of the current workflow run)\n        # (to be uploaded as artifacts)\n\n        model_failures_report = prepare_reports(\n            title=\"These following model modules had failures\",\n            header=model_header,\n            reports=sorted_model_reports,\n            to_truncate=False,\n        )\n        file_path = os.path.join(os.getcwd(), f\"ci_results_{job_name}/model_failures_report.txt\")\n        with open(file_path, \"w\", encoding=\"UTF-8\") as fp:\n            fp.write(model_failures_report)\n\n        module_failures_report = prepare_reports(\n            title=\"The following non-model modules had failures\",\n            header=module_header,\n            reports=sorted_module_reports,\n            to_truncate=False,\n        )\n        file_path = os.path.join(os.getcwd(), f\"ci_results_{job_name}/module_failures_report.txt\")\n        with open(file_path, \"w\", encoding=\"UTF-8\") as fp:\n            fp.write(module_failures_report)\n\n        if self.prev_ci_artifacts is not None:\n            # if the last run produces artifact named `ci_results_{job_name}`\n            if (\n                f\"ci_results_{job_name}\" in self.prev_ci_artifacts\n                and \"model_failures_report.txt\" in self.prev_ci_artifacts[f\"ci_results_{job_name}\"]\n            ):\n                # Compute the difference of the previous/current (model failure) table\n                prev_model_failures = self.prev_ci_artifacts[f\"ci_results_{job_name}\"][\"model_failures_report.txt\"]\n                entries_changed = self.compute_diff_for_failure_reports(model_failures_report, prev_model_failures)\n                if len(entries_changed) > 0:\n                    # Save the complete difference\n                    diff_report = prepare_reports(\n                        title=\"Changed model modules failures\",\n                        header=model_header,\n                        reports=entries_changed,\n                        to_truncate=False,\n                    )\n                    file_path = os.path.join(os.getcwd(), f\"ci_results_{job_name}/changed_model_failures_report.txt\")\n                    with open(file_path, \"w\", encoding=\"UTF-8\") as fp:\n                        fp.write(diff_report)\n\n                    # To be sent to Slack channels\n                    diff_report = prepare_reports(\n                        title=\"*Changed model modules failures*\",\n                        header=model_header,\n                        reports=entries_changed,\n                    )\n                    model_failure_sections.append(\n                        {\"type\": \"section\", \"text\": {\"type\": \"mrkdwn\", \"text\": diff_report}},\n                    )\n\n        return model_failure_sections\n\n    @property\n    def additional_failures(self) -> Dict:\n        failures = {k: v[\"failed\"] for k, v in self.additional_results.items()}\n        errors = {k: v[\"error\"] for k, v in self.additional_results.items()}\n\n        individual_reports = []\n        for key, value in failures.items():\n            device_report = self.get_device_report(value)\n\n            if sum(value.values()) or errors[key]:\n                report = f\"{key}\"\n                if errors[key]:\n                    report = f\"[Errored out] {report}\"\n                if device_report:\n                    report = f\"{device_report}{report}\"\n\n                individual_reports.append(report)\n\n        header = \"Single |  Multi | Category\\n\"\n        failures_report = prepare_reports(\n            title=\"The following non-modeling tests had failures\", header=header, reports=individual_reports\n        )\n\n        return {\"type\": \"section\", \"text\": {\"type\": \"mrkdwn\", \"text\": failures_report}}\n\n    @property\n    def payload(self) -> str:\n        blocks = [self.header]\n\n        if self.ci_title:\n            blocks.append(self.ci_title_section)\n\n        if self.n_model_failures > 0 or self.n_additional_failures > 0:\n            blocks.append(self.failures)\n\n        if self.n_model_failures > 0:\n            blocks.append(self.category_failures)\n            for block in self.model_failures:\n                if block[\"text\"][\"text\"]:\n                    blocks.append(block)\n\n        if self.n_additional_failures > 0:\n            blocks.append(self.additional_failures)\n\n        if self.n_model_failures == 0 and self.n_additional_failures == 0:\n            blocks.append(self.no_failures)\n\n        if len(self.selected_warnings) > 0:\n            blocks.append(self.warnings)\n\n        new_failure_blocks = self.get_new_model_failure_blocks(with_header=False)\n        if len(new_failure_blocks) > 0:\n            blocks.extend(new_failure_blocks)\n\n        # To save the list of new model failures\n        extra_blocks = self.get_new_model_failure_blocks(to_truncate=False)\n        if extra_blocks:\n            failure_text = extra_blocks[-1][\"text\"][\"text\"]\n            file_path = os.path.join(os.getcwd(), f\"ci_results_{job_name}/new_model_failures.txt\")\n            with open(file_path, \"w\", encoding=\"UTF-8\") as fp:\n                fp.write(failure_text)\n\n            # upload results to Hub dataset\n            file_path = os.path.join(os.getcwd(), f\"ci_results_{job_name}/new_model_failures.txt\")\n            commit_info = api.upload_file(\n                path_or_fileobj=file_path,\n                path_in_repo=f\"{datetime.datetime.today().strftime('%Y-%m-%d')}/ci_results_{job_name}/new_model_failures.txt\",\n                repo_id=\"hf-internal-testing/transformers_daily_ci\",\n                repo_type=\"dataset\",\n                token=os.environ.get(\"TRANSFORMERS_CI_RESULTS_UPLOAD_TOKEN\", None),\n            )\n            url = f\"https://huggingface.co/datasets/hf-internal-testing/transformers_daily_ci/raw/{commit_info.oid}/{datetime.datetime.today().strftime('%Y-%m-%d')}/ci_results_{job_name}/new_model_failures.txt\"\n\n            # extra processing to save to json format\n            new_failed_tests = {}\n            for line in failure_text.split():\n                if \"https://github.com/huggingface/transformers/actions/runs\" in line:\n                    pattern = r\"<(https://github.com/huggingface/transformers/actions/runs/.+?/job/.+?)\\|(.+?)>\"\n                    items = re.findall(pattern, line)\n                elif \"tests/models/\" in line:\n                    model = line.split(\"/\")[2]\n                    if model not in new_failed_tests:\n                        new_failed_tests[model] = {\"single-gpu\": [], \"multi-gpu\": []}\n                    for url, device in items:\n                        new_failed_tests[model][f\"{device}-gpu\"].append(line)\n            file_path = os.path.join(os.getcwd(), f\"ci_results_{job_name}/new_model_failures.json\")\n            with open(file_path, \"w\", encoding=\"UTF-8\") as fp:\n                json.dump(new_failed_tests, fp, ensure_ascii=False, indent=4)\n\n            # upload results to Hub dataset\n            file_path = os.path.join(os.getcwd(), f\"ci_results_{job_name}/new_model_failures.json\")\n            _ = api.upload_file(\n                path_or_fileobj=file_path,\n                path_in_repo=f\"{datetime.datetime.today().strftime('%Y-%m-%d')}/ci_results_{job_name}/new_model_failures.json\",\n                repo_id=\"hf-internal-testing/transformers_daily_ci\",\n                repo_type=\"dataset\",\n                token=os.environ.get(\"TRANSFORMERS_CI_RESULTS_UPLOAD_TOKEN\", None),\n            )\n\n            block = {\n                \"type\": \"section\",\n                \"text\": {\n                    \"type\": \"plain_text\",\n                    \"text\": \" \",\n                },\n                \"accessory\": {\n                    \"type\": \"button\",\n                    \"text\": {\"type\": \"plain_text\", \"text\": \"Check New model failures\"},\n                    \"url\": url,\n                },\n            }\n            blocks.append(block)\n\n        return json.dumps(blocks)\n\n    @staticmethod\n    def error_out(title, ci_title=\"\", runner_not_available=False, runner_failed=False, setup_failed=False):\n        blocks = []\n        title_block = {\"type\": \"header\", \"text\": {\"type\": \"plain_text\", \"text\": title}}\n        blocks.append(title_block)\n\n        if ci_title:\n            ci_title_block = {\"type\": \"section\", \"text\": {\"type\": \"mrkdwn\", \"text\": ci_title}}\n            blocks.append(ci_title_block)\n\n        offline_runners = []\n        if runner_not_available:\n            text = \"\ud83d\udc94 CI runners are not available! Tests are not run. \ud83d\ude2d\"\n            result = os.environ.get(\"OFFLINE_RUNNERS\")\n            if result is not None:\n                offline_runners = json.loads(result)\n        elif runner_failed:\n            text = \"\ud83d\udc94 CI runners have problems! Tests are not run. \ud83d\ude2d\"\n        elif setup_failed:\n            text = \"\ud83d\udc94 Setup job failed. Tests are not run. \ud83d\ude2d\"\n        else:\n            text = \"\ud83d\udc94 There was an issue running the tests. \ud83d\ude2d\"\n\n        error_block_1 = {\n            \"type\": \"header\",\n            \"text\": {\n                \"type\": \"plain_text\",\n                \"text\": text,\n            },\n        }\n\n        text = \"\"\n        if len(offline_runners) > 0:\n            text = \"\\n  \u2022 \" + \"\\n  \u2022 \".join(offline_runners)\n            text = f\"The following runners are offline:\\n{text}\\n\\n\"\n        text += \"\ud83d\ude4f Let's fix it ASAP! \ud83d\ude4f\"\n\n        error_block_2 = {\n            \"type\": \"section\",\n            \"text\": {\n                \"type\": \"plain_text\",\n                \"text\": text,\n            },\n            \"accessory\": {\n                \"type\": \"button\",\n                \"text\": {\"type\": \"plain_text\", \"text\": \"Check Action results\", \"emoji\": True},\n                \"url\": f\"https://github.com/huggingface/transformers/actions/runs/{os.environ['GITHUB_RUN_ID']}\",\n            },\n        }\n        blocks.extend([error_block_1, error_block_2])\n\n        payload = json.dumps(blocks)\n\n        print(\"Sending the following payload\")\n        print(json.dumps({\"blocks\": blocks}))\n\n        client.chat_postMessage(\n            channel=SLACK_REPORT_CHANNEL_ID,\n            text=text,\n            blocks=payload,\n        )\n\n    def post(self):\n        payload = self.payload\n        print(\"Sending the following payload\")\n        print(json.dumps({\"blocks\": json.loads(payload)}))\n\n        text = f\"{self.n_failures} failures out of {self.n_tests} tests,\" if self.n_failures else \"All tests passed.\"\n\n        self.thread_ts = client.chat_postMessage(\n            channel=SLACK_REPORT_CHANNEL_ID,\n            blocks=payload,\n            text=text,\n        )\n\n    def get_reply_blocks(self, job_name, job_result, failures, device, text):\n        \"\"\"\n        failures: A list with elements of the form {\"line\": full test name, \"trace\": error trace}\n        \"\"\"\n        # `text` must be less than 3001 characters in Slack SDK\n        # keep some room for adding \"[Truncated]\" when necessary\n        MAX_ERROR_TEXT = 3000 - len(\"[Truncated]\")\n\n        failure_text = \"\"\n        for idx, error in enumerate(failures):\n            new_text = failure_text + f'*{error[\"line\"]}*\\n_{error[\"trace\"]}_\\n\\n'\n            if len(new_text) > MAX_ERROR_TEXT:\n                # `failure_text` here has length <= 3000\n                failure_text = failure_text + \"[Truncated]\"\n                break\n            # `failure_text` here has length <= MAX_ERROR_TEXT\n            failure_text = new_text\n\n        title = job_name\n        if device is not None:\n            title += f\" ({device}-gpu)\"\n\n        content = {\"type\": \"section\", \"text\": {\"type\": \"mrkdwn\", \"text\": text}}\n\n        # TODO: Make sure we always have a valid job link (or at least a way not to break the report sending)\n        # Currently we get the device from a job's artifact name.\n        # If a device is found, the job name should contain the device type, for example, `XXX (single-gpu)`.\n        # This could be done by adding `machine_type` in a job's `strategy`.\n        # (If `job_result[\"job_link\"][device]` is `None`, we get an error: `... [ERROR] must provide a string ...`)\n        if job_result[\"job_link\"] is not None and job_result[\"job_link\"][device] is not None:\n            content[\"accessory\"] = {\n                \"type\": \"button\",\n                \"text\": {\"type\": \"plain_text\", \"text\": \"GitHub Action job\", \"emoji\": True},\n                \"url\": job_result[\"job_link\"][device],\n            }\n\n        return [\n            {\"type\": \"header\", \"text\": {\"type\": \"plain_text\", \"text\": title.upper(), \"emoji\": True}},\n            content,\n            {\"type\": \"section\", \"text\": {\"type\": \"mrkdwn\", \"text\": failure_text}},\n        ]\n\n    def get_new_model_failure_blocks(self, with_header=True, to_truncate=True):\n        if self.prev_ci_artifacts is None:\n            return []\n\n        sorted_dict = sorted(self.model_results.items(), key=lambda t: t[0])\n\n        prev_model_results = {}\n        if (\n            f\"ci_results_{job_name}\" in self.prev_ci_artifacts\n            and \"model_results.json\" in self.prev_ci_artifacts[f\"ci_results_{job_name}\"]\n        ):\n            prev_model_results = json.loads(self.prev_ci_artifacts[f\"ci_results_{job_name}\"][\"model_results.json\"])\n\n        all_failure_lines = {}\n        for job, job_result in sorted_dict:\n            if len(job_result[\"failures\"]):\n                devices = sorted(job_result[\"failures\"].keys(), reverse=True)\n                for device in devices:\n                    failures = job_result[\"failures\"][device]\n                    prev_error_lines = {}\n                    if job in prev_model_results and device in prev_model_results[job][\"failures\"]:\n                        prev_error_lines = {error[\"line\"] for error in prev_model_results[job][\"failures\"][device]}\n\n                    url = None\n                    if job_result[\"job_link\"] is not None and job_result[\"job_link\"][device] is not None:\n                        url = job_result[\"job_link\"][device]\n\n                    for idx, error in enumerate(failures):\n                        if error[\"line\"] in prev_error_lines:\n                            continue\n\n                        new_text = f'{error[\"line\"]}\\n\\n'\n\n                        if new_text not in all_failure_lines:\n                            all_failure_lines[new_text] = []\n\n                        all_failure_lines[new_text].append(f\"<{url}|{device}>\" if url is not None else device)\n\n        MAX_ERROR_TEXT = 3000 - len(\"[Truncated]\") - len(\"```New model failures```\\n\\n\")\n        if not to_truncate:\n            MAX_ERROR_TEXT = float(\"inf\")\n        failure_text = \"\"\n        for line, devices in all_failure_lines.items():\n            new_text = failure_text + f\"{'|'.join(devices)} gpu\\n{line}\"\n            if len(new_text) > MAX_ERROR_TEXT:\n                # `failure_text` here has length <= 3000\n                failure_text = failure_text + \"[Truncated]\"\n                break\n            # `failure_text` here has length <= MAX_ERROR_TEXT\n            failure_text = new_text\n\n        blocks = []\n        if failure_text:\n            if with_header:\n                blocks.append(\n                    {\"type\": \"header\", \"text\": {\"type\": \"plain_text\", \"text\": \"New model failures\", \"emoji\": True}}\n                )\n            else:\n                failure_text = f\"*New model failures*\\n\\n{failure_text}\"\n            blocks.append({\"type\": \"section\", \"text\": {\"type\": \"mrkdwn\", \"text\": failure_text}})\n\n        return blocks\n\n    def post_reply(self):\n        if self.thread_ts is None:\n            raise ValueError(\"Can only post reply if a post has been made.\")\n\n        sorted_dict = sorted(self.model_results.items(), key=lambda t: t[0])\n        for job, job_result in sorted_dict:\n            if len(job_result[\"failures\"]):\n                for device, failures in job_result[\"failures\"].items():\n                    text = \"\\n\".join(\n                        sorted([f\"*{k}*: {v[device]}\" for k, v in job_result[\"failed\"].items() if v[device]])\n                    )\n\n                    blocks = self.get_reply_blocks(job, job_result, failures, device, text=text)\n\n                    print(\"Sending the following reply\")\n                    print(json.dumps({\"blocks\": blocks}))\n\n                    client.chat_postMessage(\n                        channel=SLACK_REPORT_CHANNEL_ID,\n                        text=f\"Results for {job}\",\n                        blocks=blocks,\n                        thread_ts=self.thread_ts[\"ts\"],\n                    )\n\n                    time.sleep(1)\n\n        for job, job_result in self.additional_results.items():\n            if len(job_result[\"failures\"]):\n                for device, failures in job_result[\"failures\"].items():\n                    blocks = self.get_reply_blocks(\n                        job,\n                        job_result,\n                        failures,\n                        device,\n                        text=f'Number of failures: {job_result[\"failed\"][device]}',\n                    )\n\n                    print(\"Sending the following reply\")\n                    print(json.dumps({\"blocks\": blocks}))\n\n                    client.chat_postMessage(\n                        channel=SLACK_REPORT_CHANNEL_ID,\n                        text=f\"Results for {job}\",\n                        blocks=blocks,\n                        thread_ts=self.thread_ts[\"ts\"],\n                    )\n\n                    time.sleep(1)\n\n        blocks = self.get_new_model_failure_blocks()\n        if blocks:\n            print(\"Sending the following reply\")\n            print(json.dumps({\"blocks\": blocks}))\n\n            client.chat_postMessage(\n                channel=SLACK_REPORT_CHANNEL_ID,\n                text=\"Results for new failures\",\n                blocks=blocks,\n                thread_ts=self.thread_ts[\"ts\"],\n            )\n\n\n\nSUMMARY:\nThe provided code is a part of a Slackbot designed to report test results from machine learning jobs. The bot fetches the results from a database and formats them into threads that are sent as separate messages on Slack.\n\nKey Features:\n\n*   **Results Reporting:** The bot fetches and formats test results, which can be obtained from a database, into thread-like structures suitable for sending to Slack.\n*   **Timing Mechanism:** There is a one-second delay between each post operation, which implies multiple posts per minute, given the rate of results and other operations like sleeping between the posts.\n*   **Error Handling:** Missing results, CI run artifacts being None, or missing models' names are error conditions that need to be solved. Proper exception handling would greatly enhance this part's robustness.\n\nImprovement Suggestions:\n\n1.  Add Exception Handling:\n    *   Consider raising exceptions instead of silently failing on certain operations like fetching results where `ci_results_{job_name}` does not exist.\n2.  Optimize Post Operations:\n    *   Currently, you're sending a separate post for each model fail and its failure rate. This might be inefficient due to sending large blocks or multiple posts. Simplify your strategy if possible.\n\nHere's an example implementation of error handling with exceptions:\n\n```python\ntry:\n    # your operations here...\nexcept Exception as e:\n    print(f\"An error occurred: {e}\")\n```\n\nRemember that adding too many checks without balancing the code readability can make it difficult to understand and lead to better design choices. So pick wisely where you add exception handling to improve robustness."}


created_at: 2025-02-05 09:13:50
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/notification_service.py:827-842
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 842, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/notification_service.py", "name": "retrieve_artifact", "start_line": 827, "type": "function", "chroma:document": "CODE:\n\ndef retrieve_artifact(artifact_path: str, gpu: Optional[str]):\n    if gpu not in [None, \"single\", \"multi\"]:\n        raise ValueError(f\"Invalid GPU for artifact. Passed GPU: `{gpu}`.\")\n\n    _artifact = {}\n\n    if os.path.exists(artifact_path):\n        files = os.listdir(artifact_path)\n        for file in files:\n            try:\n                with open(os.path.join(artifact_path, file)) as f:\n                    _artifact[file.split(\".\")[0]] = f.read()\n            except UnicodeDecodeError as e:\n                raise ValueError(f\"Could not open {os.path.join(artifact_path, file)}.\") from e\n\n\n\nSUMMARY:\n**Function Summary**\n\n`retrieve_artifact`: Retrieves artifacts from a specified path.\n\n**Purpose and Functionality**\nExtracts artifact files from the given `artifact_path`, including file contents in decoded format. Handles invalid GPU configurations and non-existent paths.\n\n**Key Implementation Details**\n\n* Checks valid GPU configurations (`single`, `multi`, or `None`).\n* Uses `os.listdir` to get a list of files in the specified path.\n* Iterates through each file, attempts to open and read its contents using `with` statement.\n* Raises error on non-existent paths and fails to decode Unicode characters.\n\n**Notable Dependencies and Requirements**\n\n* `[os]` module for interacting with the file system.\n* `Optional[str]` type hint for indicating that `gpu` can be `None`.\n* `Unicode Decode Error` handling raised by Python."}


created_at: 2025-02-05 09:13:50
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/notification_service.py:845-894
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 894, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/notification_service.py", "name": "retrieve_available_artifacts", "start_line": 845, "type": "function", "chroma:document": "CODE:\n\ndef retrieve_available_artifacts():\n    class Artifact:\n        def __init__(self, name: str, single_gpu: bool = False, multi_gpu: bool = False):\n            self.name = name\n            self.single_gpu = single_gpu\n            self.multi_gpu = multi_gpu\n            self.paths = []\n\n        def __str__(self):\n            return self.name\n\n        def add_path(self, path: str, gpu: str = None):\n            self.paths.append({\"name\": self.name, \"path\": path, \"gpu\": gpu})\n\n    _available_artifacts: Dict[str, Artifact] = {}\n\n    directories = filter(os.path.isdir, os.listdir())\n    for directory in directories:\n        artifact_name = directory\n\n        name_parts = artifact_name.split(\"_postfix_\")\n        if len(name_parts) > 1:\n            artifact_name = name_parts[0]\n\n        if artifact_name.startswith(\"single-gpu\"):\n            artifact_name = artifact_name[len(\"single-gpu\") + 1 :]\n\n            if artifact_name in _available_artifacts:\n                _available_artifacts[artifact_name].single_gpu = True\n            else:\n                _available_artifacts[artifact_name] = Artifact(artifact_name, single_gpu=True)\n\n            _available_artifacts[artifact_name].add_path(directory, gpu=\"single\")\n\n        elif artifact_name.startswith(\"multi-gpu\"):\n            artifact_name = artifact_name[len(\"multi-gpu\") + 1 :]\n\n            if artifact_name in _available_artifacts:\n                _available_artifacts[artifact_name].multi_gpu = True\n            else:\n                _available_artifacts[artifact_name] = Artifact(artifact_name, multi_gpu=True)\n\n            _available_artifacts[artifact_name].add_path(directory, gpu=\"multi\")\n        else:\n            if artifact_name not in _available_artifacts:\n                _available_artifacts[artifact_name] = Artifact(artifact_name)\n\n            _available_artifacts[artifact_name].add_path(directory)\n\n\n\nSUMMARY:\n**Summary: `retrieve_available_artifacts` Function**\n\n retrieves and manages available artifacts, which are stored in the `_available_artifacts` dictionary.\n\n\n*   **Main purpose**: This function iterates through directories on disk and populates an internal data structure with artifact information.\n*   **Key implementation details**:\n    *   The `_available_artifacts` dictionary is populated with `Artifact` objects, where each `Artifact` represents a directory-based asset.\n    *   Directory names are used to determine the type of GPU required for the asset (single or multi).\n    *   Assets are associated with paths and gpu types using the `add_path` method.\n*   **Notable dependencies**:\n    *   `os` module for interacting with file systems\n    *   `filter` function from `itertools` for filtering directories\n    *   `Dict` ( dictionary ) type hint\n\n\n(Note: In a RAG system, this summary can be further used as metadata by developers to understand the functionality and implementation of this code chunk.)"}


created_at: 2025-02-05 09:13:50
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/notification_service.py:897-917
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 917, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/notification_service.py", "name": "prepare_reports", "start_line": 897, "type": "function", "chroma:document": "CODE:\n\ndef prepare_reports(title, header, reports, to_truncate=True):\n    report = \"\"\n\n    MAX_ERROR_TEXT = 3000 - len(\"[Truncated]\")\n    if not to_truncate:\n        MAX_ERROR_TEXT = float(\"inf\")\n\n    if len(reports) > 0:\n        # `text` must be less than 3001 characters in Slack SDK\n        # keep some room for adding \"[Truncated]\" when necessary\n\n        for idx in range(len(reports)):\n            _report = header + \"\\n\".join(reports[: idx + 1])\n            new_report = f\"{title}:\\n```\\n{_report}\\n```\\n\"\n            if len(new_report) > MAX_ERROR_TEXT:\n                # `report` here has length <= 3000\n                report = report + \"[Truncated]\"\n                break\n            report = new_report\n\n\n\nSUMMARY:\n**Function Summary: prepare_reports**\n\n**Purpose:** Generate a formatted report string with truncated content if necessary.\n\n**Primary Functionality:**\nPrepares a report by combining input `title`, `header`, and `reports` into a single string, truncating the output if it exceeds 3000 characters due to \"[Truncated]\" addition.\n\n**Key Implementation Details:**\n\n* Iterates through each report in descending order using `_report = header + \"\\n\".join(reports[: idx + 1])`.\n* Checks for truncation conditions inside an `if` block (`if len(new_report) > MAX_ERROR_TEXT:`).\n\t+ Truncates the output with \"[Truncated]\" if necessary.\n\n**Notable Dependencies/Requirements:**\n\n* No external dependencies listed.\n* Uses specific constants:\n\t+ `MAX_ERROR_TEXT` defined as a fixed value (3000 - 1).\n\nNote this summary excludes unnecessary or generic details, focusing on the primary purpose and key implementation mechanics of the provided function."}


created_at: 2025-02-05 09:13:51
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_config_attributes.py:246-350
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 350, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_config_attributes.py", "name": "check_attribute_being_used", "start_line": 246, "type": "function", "chroma:document": "CODE:\n\ndef check_attribute_being_used(config_class, attributes, default_value, source_strings):\n    \"\"\"Check if any name in `attributes` is used in one of the strings in `source_strings`\n\n    Args:\n        config_class (`type`):\n            The configuration class for which the arguments in its `__init__` will be checked.\n        attributes (`List[str]`):\n            The name of an argument (or attribute) and its variant names if any.\n        default_value (`Any`):\n            A default value for the attribute in `attributes` assigned in the `__init__` of `config_class`.\n        source_strings (`List[str]`):\n            The python source code strings in the same modeling directory where `config_class` is defined. The file\n            containing the definition of `config_class` should be excluded.\n    \"\"\"\n    attribute_used = False\n    for attribute in attributes:\n        for modeling_source in source_strings:\n            # check if we can find `config.xxx`, `getattr(config, \"xxx\", ...)` or `getattr(self.config, \"xxx\", ...)`\n            if (\n                f\"config.{attribute}\" in modeling_source\n                or f'getattr(config, \"{attribute}\"' in modeling_source\n                or f'getattr(self.config, \"{attribute}\"' in modeling_source\n                or (\n                    \"TextConfig\" in config_class.__name__\n                    and f\"config.get_text_config().{attribute}\" in modeling_source\n                )\n            ):\n                attribute_used = True\n            # Deal with multi-line cases\n            elif (\n                re.search(\n                    rf'getattr[ \\t\\v\\n\\r\\f]*\\([ \\t\\v\\n\\r\\f]*(self\\.)?config,[ \\t\\v\\n\\r\\f]*\"{attribute}\"',\n                    modeling_source,\n                )\n                is not None\n            ):\n                attribute_used = True\n            # `SequenceSummary` is called with `SequenceSummary(config)`\n            elif attribute in [\n                \"summary_type\",\n                \"summary_use_proj\",\n                \"summary_activation\",\n                \"summary_last_dropout\",\n                \"summary_proj_to_labels\",\n                \"summary_first_dropout\",\n            ]:\n                if \"SequenceSummary\" in modeling_source:\n                    attribute_used = True\n            if attribute_used:\n                break\n        if attribute_used:\n            break\n\n    # common and important attributes, even if they do not always appear in the modeling files\n    attributes_to_allow = [\n        \"bos_index\",\n        \"eos_index\",\n        \"pad_index\",\n        \"unk_index\",\n        \"mask_index\",\n        \"image_token_index\",  # for VLMs\n        \"video_token_index\",\n        \"image_seq_length\",\n        \"video_seq_length\",\n        \"image_size\",\n        \"use_cache\",\n        \"out_features\",\n        \"out_indices\",\n        \"sampling_rate\",\n        # backbone related arguments passed to load_backbone\n        \"use_pretrained_backbone\",\n        \"backbone\",\n        \"backbone_config\",\n        \"use_timm_backbone\",\n        \"backbone_kwargs\",\n        # rope attributes may not appear directly in the modeling but are used\n        \"rope_theta\",\n        \"partial_rotary_factor\",\n        \"pretraining_tp\",\n    ]\n    attributes_used_in_generation = [\"encoder_no_repeat_ngram_size\"]\n\n    # Special cases to be allowed\n    case_allowed = True\n    if not attribute_used:\n        case_allowed = False\n        for attribute in attributes:\n            # Allow if the default value in the configuration class is different from the one in `PretrainedConfig`\n            if attribute in [\"is_encoder_decoder\"] and default_value is True:\n                case_allowed = True\n            elif attribute in [\"tie_word_embeddings\"] and default_value is False:\n                case_allowed = True\n\n            # Allow cases without checking the default value in the configuration class\n            elif attribute in attributes_to_allow + attributes_used_in_generation:\n                case_allowed = True\n            elif attribute.endswith(\"_token_id\"):\n                case_allowed = True\n\n            # configuration class specific cases\n            if not case_allowed:\n                allowed_cases = SPECIAL_CASES_TO_ALLOW.get(config_class.__name__, [])\n                case_allowed = allowed_cases is True or attribute in allowed_cases\n\n\n\nSUMMARY:\n**Summary**\n\n`check_attribute_being_used` function checks if any name in `attributes` is used in one of the strings in `source_strings`. It inspects source code for usage of attribute names, including variants like `getattr()`, and flags its presence or absence.\n\n**Main Purpose and Functionality**\nInspect source code for attribute usage (`config.xxx`, `getattr()` constructs)\n\n**Key Implementation Details**\n\n* Check for direct attribute mentions (`f\"config.{attribute}\" in modeling_source`)\n* Detect multi-line `getattr()` patterns\n* Handle specific use cases like `TextConfig` and `SequenceSummary`\n\n**Notable Dependencies or Requirements**\n- `re` (regular expression library)\n- `config_class` with its specific class methods (`__init__`, `get_text_config()`)"}


created_at: 2025-02-05 09:13:51
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_config_attributes.py:353-396
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 396, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_config_attributes.py", "name": "check_config_attributes_being_used", "start_line": 353, "type": "function", "chroma:document": "CODE:\n\ndef check_config_attributes_being_used(config_class):\n    \"\"\"Check the arguments in `__init__` of `config_class` are used in the modeling files in the same directory\n\n    Args:\n        config_class (`type`):\n            The configuration class for which the arguments in its `__init__` will be checked.\n    \"\"\"\n    # Get the parameters in `__init__` of the configuration class, and the default values if any\n    signature = dict(inspect.signature(config_class.__init__).parameters)\n    parameter_names = [x for x in list(signature.keys()) if x not in [\"self\", \"kwargs\"]]\n    parameter_defaults = [signature[param].default for param in parameter_names]\n\n    # If `attribute_map` exists, an attribute can have different names to be used in the modeling files, and as long\n    # as one variant is used, the test should pass\n    reversed_attribute_map = {}\n    if len(config_class.attribute_map) > 0:\n        reversed_attribute_map = {v: k for k, v in config_class.attribute_map.items()}\n\n    # Get the path to modeling source files\n    config_source_file = inspect.getsourcefile(config_class)\n    model_dir = os.path.dirname(config_source_file)\n    # Let's check against all frameworks: as long as one framework uses an attribute, we are good.\n    modeling_paths = [os.path.join(model_dir, fn) for fn in os.listdir(model_dir) if fn.startswith(\"modeling_\")]\n\n    # Get the source code strings\n    modeling_sources = []\n    for path in modeling_paths:\n        if os.path.isfile(path):\n            with open(path, encoding=\"utf8\") as fp:\n                modeling_sources.append(fp.read())\n\n    unused_attributes = []\n    for config_param, default_value in zip(parameter_names, parameter_defaults):\n        # `attributes` here is all the variant names for `config_param`\n        attributes = [config_param]\n        # some configuration classes have non-empty `attribute_map`, and both names could be used in the\n        # corresponding modeling files. As long as one of them appears, it is fine.\n        if config_param in reversed_attribute_map:\n            attributes.append(reversed_attribute_map[config_param])\n\n        if not check_attribute_being_used(config_class, attributes, default_value, modeling_sources):\n            unused_attributes.append(attributes[0])\n\n\n\nSUMMARY:\n**Summary: `check_config_attributes_being_used` Function**\n\nThis function checks if the configuration class's initialization arguments are used in the same directory's modeling files. It does this by:\n\n1. Analyzing the init signature of the config class, identifying parameter names and default values.\n2. Checking for existence and usage of attribute maps, which can map variant names to different attribute names.\n3. Scanning model source files to detect attribute appearances that match the calculated attributes.\n\n**Key Implementation Details:**\n\n* Checks `attribute_map` for variants, allowing one usage to pass even if multiple variants exist.\n* Utilizes `inspect.signature` and `os.path.join` for file path manipulation.\n* Calls `check_attribute_being_used` internally to verify attribute usage against modeling sources.\n\n**Notable Dependencies/Requirements:**\n\n* `inspect`: For accessing signature information and getting source files.\n* `os`: For handling file paths and directories.\n* `config_class`, which is expected to have an `attribute_map` dictionary.\n\nThis function returns a list of unused attributes, sorted for consistency."}


created_at: 2025-02-05 09:13:51
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_config_attributes.py:399-426
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 426, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_config_attributes.py", "name": "check_config_attributes", "start_line": 399, "type": "function", "chroma:document": "CODE:\n\ndef check_config_attributes():\n    \"\"\"Check the arguments in `__init__` of all configuration classes are used in python files\"\"\"\n    configs_with_unused_attributes = {}\n    for _config_class in list(CONFIG_MAPPING.values()):\n        # Skip deprecated models\n        if \"models.deprecated\" in _config_class.__module__:\n            continue\n        # Some config classes are not in `CONFIG_MAPPING` (e.g. `CLIPVisionConfig`, `Blip2VisionConfig`, etc.)\n        config_classes_in_module = [\n            cls\n            for name, cls in inspect.getmembers(\n                inspect.getmodule(_config_class),\n                lambda x: inspect.isclass(x)\n                and issubclass(x, PretrainedConfig)\n                and inspect.getmodule(x) == inspect.getmodule(_config_class),\n            )\n        ]\n        for config_class in config_classes_in_module:\n            unused_attributes = check_config_attributes_being_used(config_class)\n            if len(unused_attributes) > 0:\n                configs_with_unused_attributes[config_class.__name__] = unused_attributes\n\n    if len(configs_with_unused_attributes) > 0:\n        error = \"The following configuration classes contain unused attributes in the corresponding modeling files:\\n\"\n        for name, attributes in configs_with_unused_attributes.items():\n            error += f\"{name}: {attributes}\\n\"\n\n\n\nSUMMARY:\n**Function Summary:**\n\n`check_config_attributes`: Checks if configuration class arguments in `__init__` are used in corresponding Python files and raises an error for unused attributes.\n\n**Key Implementation Details:**\n\n* Iterates through all configuration classes and their submodules to identify potential issue sources\n* Uses `check_config_attributes_being_used` function to detect unused attributes in each configuration class\n\n**Notable Dependencies/Requirements:**\n\n* `CONFIG_MAPPING`: a dictionary mapping configuration classes (not defined in provided code snippet)\n* `PretrainedConfig` class (assumed to be defined elsewhere)\n\n**Primary Purpose:** To ensure that configuration class arguments are consistently used in corresponding Python files."}


created_at: 2025-02-05 09:13:51
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/get_ci_error_statistics.py:12-35
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 35, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/get_ci_error_statistics.py", "name": "get_jobs", "start_line": 12, "type": "function", "chroma:document": "CODE:\n\ndef get_jobs(workflow_run_id, token=None):\n    \"\"\"Extract jobs in a GitHub Actions workflow run\"\"\"\n\n    headers = None\n    if token is not None:\n        headers = {\"Accept\": \"application/vnd.github+json\", \"Authorization\": f\"Bearer {token}\"}\n\n    url = f\"https://api.github.com/repos/huggingface/transformers/actions/runs/{workflow_run_id}/jobs?per_page=100\"\n    result = requests.get(url, headers=headers).json()\n    jobs = []\n\n    try:\n        jobs.extend(result[\"jobs\"])\n        pages_to_iterate_over = math.ceil((result[\"total_count\"] - 100) / 100)\n\n        for i in range(pages_to_iterate_over):\n            result = requests.get(url + f\"&page={i + 2}\", headers=headers).json()\n            jobs.extend(result[\"jobs\"])\n\n        return jobs\n    except Exception:\n        print(f\"Unknown error, could not fetch links:\\n{traceback.format_exc()}\")\n\n\n\nSUMMARY:\n**Function Summary:**\n\n`get_jobs` extracts GitHub Actions workflow run jobs.\n\n**Purpose:**\nExtracts job data from a GitHub Actions workflow run API.\n\n**Key Implementation Details:**\n\n*   Uses the GitHub API to fetch job data with pagination.\n*   Utilizes the `requests` library to send GET requests and parse JSON responses.\n\n**Method Signatures:**\n\n*   `requests.get(url, headers=headers)`: Sends a GET request to the specified URL with provided headers.\n*   `.json()`: Parses the response as JSON.\n\n**Notable Dependencies:**\n\n*   `requests`\n*   `math` (for calculating total pages)\n\n**Interfaces Implemented:**\n\n*   GitHub API (`https://api.github.com/repos/huggingface/transformers/actions/runs/{workflow_run_id}/jobs`)\n\nThis function uses pagination to fetch job data from the GitHub Actions workflow run API. It handles exceptions and prints errors if an unknown error occurs during execution."}


created_at: 2025-02-05 09:13:51
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/get_ci_error_statistics.py:38-61
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 61, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/get_ci_error_statistics.py", "name": "get_job_links", "start_line": 38, "type": "function", "chroma:document": "CODE:\n\ndef get_job_links(workflow_run_id, token=None):\n    \"\"\"Extract job names and their job links in a GitHub Actions workflow run\"\"\"\n\n    headers = None\n    if token is not None:\n        headers = {\"Accept\": \"application/vnd.github+json\", \"Authorization\": f\"Bearer {token}\"}\n\n    url = f\"https://api.github.com/repos/huggingface/transformers/actions/runs/{workflow_run_id}/jobs?per_page=100\"\n    result = requests.get(url, headers=headers).json()\n    job_links = {}\n\n    try:\n        job_links.update({job[\"name\"]: job[\"html_url\"] for job in result[\"jobs\"]})\n        pages_to_iterate_over = math.ceil((result[\"total_count\"] - 100) / 100)\n\n        for i in range(pages_to_iterate_over):\n            result = requests.get(url + f\"&page={i + 2}\", headers=headers).json()\n            job_links.update({job[\"name\"]: job[\"html_url\"] for job in result[\"jobs\"]})\n\n        return job_links\n    except Exception:\n        print(f\"Unknown error, could not fetch links:\\n{traceback.format_exc()}\")\n\n\n\nSUMMARY:\n**Function Summary: get_job_links**\n\n**Main Purpose and Functionality:**\nThe `get_job_links` function extracts job names and their corresponding GitHub Actions workflow run links from the Hugging Face Transformers repository.\n\n**Key Implementation Details:**\n\n* Uses GitHub API to fetch job data in paginated chunks (up to 100 jobs per page).\n* Supports token-based authentication for request headers.\n* Iterates over multiple pages of results to collect all job links.\n\n**Notable Dependencies and Requirements:**\n\n* Python 3.6+\n* `requests` library for HTTP requests\n* `math` library for calculating pagination parameters\n* Hugging Face Transformers repository GitHub API permissions ( token for authentication)"}


created_at: 2025-02-05 09:13:51
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/get_ci_error_statistics.py:64-87
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 87, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/get_ci_error_statistics.py", "name": "get_artifacts_links", "start_line": 64, "type": "function", "chroma:document": "CODE:\n\ndef get_artifacts_links(worflow_run_id, token=None):\n    \"\"\"Get all artifact links from a workflow run\"\"\"\n\n    headers = None\n    if token is not None:\n        headers = {\"Accept\": \"application/vnd.github+json\", \"Authorization\": f\"Bearer {token}\"}\n\n    url = f\"https://api.github.com/repos/huggingface/transformers/actions/runs/{worflow_run_id}/artifacts?per_page=100\"\n    result = requests.get(url, headers=headers).json()\n    artifacts = {}\n\n    try:\n        artifacts.update({artifact[\"name\"]: artifact[\"archive_download_url\"] for artifact in result[\"artifacts\"]})\n        pages_to_iterate_over = math.ceil((result[\"total_count\"] - 100) / 100)\n\n        for i in range(pages_to_iterate_over):\n            result = requests.get(url + f\"&page={i + 2}\", headers=headers).json()\n            artifacts.update({artifact[\"name\"]: artifact[\"archive_download_url\"] for artifact in result[\"artifacts\"]})\n\n        return artifacts\n    except Exception:\n        print(f\"Unknown error, could not fetch links:\\n{traceback.format_exc()}\")\n\n\n\nSUMMARY:\n**Summary**\n\nThis function `get_artifacts_links` retrieves URLs for artifact downloads from GitHub Actions in Hugging Face's Transformers repository. It takes a workflow run ID and an optional authentication token.\n\n**Key Functionality:**\n\n*   Fetches GitHub API for archive download URLs of artifacts within a workflow run.\n*   Iterates over multiple pages to collect all required links until exhausted.\n*   Handles pagination, total count, and error cases.\n\n**Implementation Details:**\n\n*   Uses `requests` library with optional authentication token via `Authorization` header.\n*   Utilizes JSON parsing for receiving API response data.\n*   Performs dictionary updates for efficient collection of artifact URLs.\n\n**Notable Dependencies/Requirements:**\n\n*   `math` for calculating pages to iterate.\n*   `requests` and `json` for interacting with the GitHub API, handling responses.\n*   `traceback` for error reporting in case of an issue.\n\n**Primary Purpose:**\nTo fetch archive download URLs for artifacts from a specified GitHub Actions workflow run.\n\n**Key Method Signatures:**\n\n*   `requests.get()`: To fetch pages from the API.\n*   `dict.update()`: To add new entries to the dictionary of artifact links.\n*   `math.ceil()`: For calculating the number of pages to iterate over."}


created_at: 2025-02-05 09:13:51
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/get_ci_error_statistics.py:90-106
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 106, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/get_ci_error_statistics.py", "name": "download_artifact", "start_line": 90, "type": "function", "chroma:document": "CODE:\n\ndef download_artifact(artifact_name, artifact_url, output_dir, token):\n    \"\"\"Download a GitHub Action artifact from a URL.\n\n    The URL is of the form `https://api.github.com/repos/huggingface/transformers/actions/artifacts/{ARTIFACT_ID}/zip`,\n    but it can't be used to download directly. We need to get a redirect URL first.\n    See https://docs.github.com/en/rest/actions/artifacts#download-an-artifact\n    \"\"\"\n    headers = None\n    if token is not None:\n        headers = {\"Accept\": \"application/vnd.github+json\", \"Authorization\": f\"Bearer {token}\"}\n\n    result = requests.get(artifact_url, headers=headers, allow_redirects=False)\n    download_url = result.headers[\"Location\"]\n    response = requests.get(download_url, allow_redirects=True)\n    file_path = os.path.join(output_dir, f\"{artifact_name}.zip\")\n    with open(file_path, \"wb\") as fp:\n\n\nSUMMARY:\n**Function Summary: `download_artifact`**\n\nDownloads a GitHub Action artifact from a URL.\n\n**Primary Functionality:**\nThis function downloads a zip file from a GitHub Action artifact URL. It first retrieves the redirect URL and then downloads it to a specified output directory using the provided token for authentication.\n\n**Key Implementation Details:**\n\n* Uses the `requests` library to send HTTP requests\n* Sets headers with `Authorization` and `Accept` fields based on whether a token is provided\n* Sets `allow_redirects=False` initially, but switches to `True` when downloading the redirect URL\n* Retrieves the file path from the URL's `Location` header\n\n**Notable Dependencies:**\n\n* `requests`\n* `os`\n\n**Method Signatures and Interfaces:**\n- `requests.get()`: sends a GET request to a specified URL\n- `Authorization` flow for token authentication\n\nNote that interface implementation (i.e., how it interacts with the GitHub API) is limited, but this summary should give a clear overview of its main functionality and underlying mechanisms."}


created_at: 2025-02-05 09:13:51
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/get_ci_error_statistics.py:109-153
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 153, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/get_ci_error_statistics.py", "name": "get_errors_from_single_artifact", "start_line": 109, "type": "function", "chroma:document": "CODE:\n\ndef get_errors_from_single_artifact(artifact_zip_path, job_links=None):\n    \"\"\"Extract errors from a downloaded artifact (in .zip format)\"\"\"\n    errors = []\n    failed_tests = []\n    job_name = None\n\n    with zipfile.ZipFile(artifact_zip_path) as z:\n        for filename in z.namelist():\n            if not os.path.isdir(filename):\n                # read the file\n                if filename in [\"failures_line.txt\", \"summary_short.txt\", \"job_name.txt\"]:\n                    with z.open(filename) as f:\n                        for line in f:\n                            line = line.decode(\"UTF-8\").strip()\n                            if filename == \"failures_line.txt\":\n                                try:\n                                    # `error_line` is the place where `error` occurs\n                                    error_line = line[: line.index(\": \")]\n                                    error = line[line.index(\": \") + len(\": \") :]\n                                    errors.append([error_line, error])\n                                except Exception:\n                                    # skip un-related lines\n                                    pass\n                            elif filename == \"summary_short.txt\" and line.startswith(\"FAILED \"):\n                                # `test` is the test method that failed\n                                test = line[len(\"FAILED \") :]\n                                failed_tests.append(test)\n                            elif filename == \"job_name.txt\":\n                                job_name = line\n\n    if len(errors) != len(failed_tests):\n        raise ValueError(\n            f\"`errors` and `failed_tests` should have the same number of elements. Got {len(errors)} for `errors` \"\n            f\"and {len(failed_tests)} for `failed_tests` instead. The test reports in {artifact_zip_path} have some\"\n            \" problem.\"\n        )\n\n    job_link = None\n    if job_name and job_links:\n        job_link = job_links.get(job_name, None)\n\n    # A list with elements of the form (line of error, error, failed test)\n    result = [x + [y] + [job_link] for x, y in zip(errors, failed_tests)]\n\n\n\nSUMMARY:\n**Code Summary: `get_errors_from_single_artifact`**\n\nThe primary purpose of this function is to extract error messages and related test information from a ZIP archive downloaded as input.\n\n**Key Implementation Details:**\n\n* Uses the `zipfile` module to iterate over files in the ZIP archive.\n* Checks specific file contents (e.g., `failures_line.txt`, `summary_short.txt`, `job_name.txt`) for relevant data extraction.\n* Processes errors and failed test information using try-except blocks.\n* Raises an error if there's a mismatch between extracted error messages and failed tests.\n\n**Method Signatures and Interfaces:**\n\n* `get_errors_from_single_artifact(artifact_zip_path, job_links=None)`: The main function with two input parameters.\n* `zip.open(filename) as f`: Used to read file contents within the ZIP archive.\n* `job_links.get(job_name, None)`: Retrieves a job link based on a matching job name (if provided).\n\n**Notable Dependencies and Requirements:**\n\n* Python 3.x\n* `zipfile` library for working with ZIP archives.\n* External dependencies:\n\t+ `os` module for path operations.\n\n**Primary Purpose Statement:**\n\nExtracts errors from a downloaded artifact in XML format, mapping error lines to corresponding failed tests."}


created_at: 2025-02-05 09:13:51
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/get_ci_error_statistics.py:156-165
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 165, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/get_ci_error_statistics.py", "name": "get_all_errors", "start_line": 156, "type": "function", "chroma:document": "CODE:\n\ndef get_all_errors(artifact_dir, job_links=None):\n    \"\"\"Extract errors from all artifact files\"\"\"\n\n    errors = []\n\n    paths = [os.path.join(artifact_dir, p) for p in os.listdir(artifact_dir) if p.endswith(\".zip\")]\n    for p in paths:\n        errors.extend(get_errors_from_single_artifact(p, job_links=job_links))\n\n\n\nSUMMARY:\n**Summary:**\n\nThis function retrieves all error messages from artifact files located at the specified `artifact_dir`. It searches for `.zip` files, extracts their error information using the `get_errors_from_single_artifact` function, and returns them in a list.\n\n**Key Implementation Details:**\n\n* Searches for `.zip` files recursively\n* Calls `get_errors_from_single_artifact` on each file with optional `job_links`\n* Extends errors list with results from all extracted artifacts\n\n**Notable Dependencies:**\n\n* `os.path.join`\n* `os.listdir`\n* `get_errors_from_single_artifact` ( referenced function, not shown in original snippet)\n\n**Primary Purpose:**\nExtract and return a list of error messages from artifact file zip archives."}


created_at: 2025-02-05 09:13:51
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/get_ci_error_statistics.py:168-180
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 180, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/get_ci_error_statistics.py", "name": "reduce_by_error", "start_line": 168, "type": "function", "chroma:document": "CODE:\n\ndef reduce_by_error(logs, error_filter=None):\n    \"\"\"count each error\"\"\"\n\n    counter = Counter()\n    counter.update([x[1] for x in logs])\n    counts = counter.most_common()\n    r = {}\n    for error, count in counts:\n        if error_filter is None or error not in error_filter:\n            r[error] = {\"count\": count, \"failed_tests\": [(x[2], x[0]) for x in logs if x[1] == error]}\n\n    r = dict(sorted(r.items(), key=lambda item: item[1][\"count\"], reverse=True))\n\n\nSUMMARY:\n**Summary:**\n\nThe `reduce_by_error` function clusters test log errors into a dictionary, grouped by frequency and containing failed tests. It takes in:\n\n*   Logs with error codes as values\n*   Optional error filter list\n*   Counts each unique error using `Counter` and returns the results sorted by count in descending order.\n\n**Implementation Details:**\n\n*   Updates a `Counter` object to count unique errors from logs.\n*   Creates a dictionary (`r`) to store clusters of errors, where each cluster contains:\n    *   Error code\n    *   Count\n    *   Failed test(s) with original log index and error code\n*   Sorts the final dictionary by error count in descending order.\n\n**Dependencies/Requirements:**\n\n*   `Counter` from collections module for counting unique errors.\n*   List of expected error filter values (if provided)."}


created_at: 2025-02-05 09:13:51
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/get_ci_error_statistics.py:183-191
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 191, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/get_ci_error_statistics.py", "name": "get_model", "start_line": 183, "type": "function", "chroma:document": "CODE:\n\ndef get_model(test):\n    \"\"\"Get the model name from a test method\"\"\"\n    test = test.split(\"::\")[0]\n    if test.startswith(\"tests/models/\"):\n        test = test.split(\"/\")[2]\n    else:\n        test = None\n\n\n\nSUMMARY:\n**Summary:**\n\n* Function `get_model`: retrieves the model name from a test method or path.\n \n**Key Implementation Details:**\n \n* Splits input string by '::' and takes first part as model name\n* Checks if input starts with 'tests/models/' prefix, and splits by '/' to extract model name\n* Returns extracted model name, else returns None\n\n**Notable Dependencies/Requirements:**\n\n* None"}


created_at: 2025-02-05 09:13:51
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/get_ci_error_statistics.py:194-213
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 213, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/get_ci_error_statistics.py", "name": "reduce_by_model", "start_line": 194, "type": "function", "chroma:document": "CODE:\n\ndef reduce_by_model(logs, error_filter=None):\n    \"\"\"count each error per model\"\"\"\n\n    logs = [(x[0], x[1], get_model(x[2])) for x in logs]\n    logs = [x for x in logs if x[2] is not None]\n    tests = {x[2] for x in logs}\n\n    r = {}\n    for test in tests:\n        counter = Counter()\n        # count by errors in `test`\n        counter.update([x[1] for x in logs if x[2] == test])\n        counts = counter.most_common()\n        error_counts = {error: count for error, count in counts if (error_filter is None or error not in error_filter)}\n        n_errors = sum(error_counts.values())\n        if n_errors > 0:\n            r[test] = {\"count\": n_errors, \"errors\": error_counts}\n\n    r = dict(sorted(r.items(), key=lambda item: item[1][\"count\"], reverse=True))\n\n\nSUMMARY:\n**Function Summary:** `reduce_by_model`\n\n**Purpose:** Group and count errors by model, filtering out those excluded by the input `error_filter`.\n\n**Implementation Details:**\n\n* Uses list comprehension to enrich log data with the corresponding model type\n* Applies error filtering using `error_filter` if provided\n* Counts occurrences of each error in the `test` group using a `Counter`\n* Calculates the total number of errors for each test and stores them in a dictionary\n\n**Key Method Signatures:**\n\n* `most_common()`: Returns the most common values and their counts via `Counter`\n\n**Notable Dependencies/Requirements:**\n\n* Standard library: `collections.Counter`, function `get_model`\n* External dependency: None\n\n**Context:** This code appears to be part of a logging analysis workflow, grouping errors by model type for further analysis."}


created_at: 2025-02-05 09:13:51
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/get_ci_error_statistics.py:216-225
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 225, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/get_ci_error_statistics.py", "name": "make_github_table", "start_line": 216, "type": "function", "chroma:document": "CODE:\n\ndef make_github_table(reduced_by_error):\n    header = \"| no. | error | status |\"\n    sep = \"|-:|:-|:-|\"\n    lines = [header, sep]\n    for error in reduced_by_error:\n        count = reduced_by_error[error][\"count\"]\n        line = f\"| {count} | {error[:100]} |  |\"\n        lines.append(line)\n\n\n\nSUMMARY:\n**Function Summary: generateGitHubTable**\n\n**Purpose:** Generate a formatted table of GitHub errors with status and count.\n\n**Implementation Details:**\n\n* Uses a fixed header line with four columns.\n* Utilizes a separator line to split the data into rows.\n* Loops through each error in `reduced_by_error`, creating a single row per error with the error name truncated to 100 characters.\n* Joints all lines together using newline characters.\n\n**Key Features:**\n\n* `make_github_table` function takes a dictionary of errors as input (`reduced_by_error`).\n* Returns a string representing the formatted table.\n\n**Dependencies and Requirements:** \n\n* None explicitly defined, but assumes a Python environment with standard library access."}


created_at: 2025-02-05 09:13:51
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/get_ci_error_statistics.py:228-238
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 238, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/get_ci_error_statistics.py", "name": "make_github_table_per_model", "start_line": 228, "type": "function", "chroma:document": "CODE:\n\ndef make_github_table_per_model(reduced_by_model):\n    header = \"| model | no. of errors | major error | count |\"\n    sep = \"|-:|-:|-:|-:|\"\n    lines = [header, sep]\n    for model in reduced_by_model:\n        count = reduced_by_model[model][\"count\"]\n        error, _count = list(reduced_by_model[model][\"errors\"].items())[0]\n        line = f\"| {model} | {count} | {error[:60]} | {_count} |\"\n        lines.append(line)\n\n\n\nSUMMARY:\n**Summary**\n\n**Functionality**\nCreate a GitHub table per model with errors, displaying:\n- Model name\n- No. of errors\n- Major error description (truncated to 60 characters)\n- Error count\n\n**Implementation Details**\n- Iterates through `reduced_by_model` dictionary\n- Extracts key model data: error count and major error details\n- Builds table format string with dynamic values\n- Returns a formatted string with the table lines joined by newline (`\\n`)\n\n**Key Components**\n\n* `model`: reduced_by_model dictionary keys (model names)\n* `errors`: values for each model, containing error and count data\n\t+ error: first key-value pair item (assuming single major error per model)\n\t+ count: corresponding value\n\n**Notable Dependencies/Requirements**\n* None"}


created_at: 2025-02-05 09:13:51
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/release.py:70-85
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 85, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/release.py", "name": "update_version_in_file", "start_line": 70, "type": "function", "chroma:document": "CODE:\n\ndef update_version_in_file(fname: str, version: str, file_type: str):\n    \"\"\"\n    Update the version of Transformers in one file.\n\n    Args:\n        fname (`str`): The path to the file where we want to update the version.\n        version (`str`): The new version to set in the file.\n        file_type (`str`): The type of the file (should be a key in `REPLACE_PATTERNS`).\n    \"\"\"\n    with open(fname, \"r\", encoding=\"utf-8\", newline=\"\\n\") as f:\n        code = f.read()\n    re_pattern, replace = REPLACE_PATTERNS[file_type]\n    replace = replace.replace(\"VERSION\", version)\n    code = re_pattern.sub(replace, code)\n    with open(fname, \"w\", encoding=\"utf-8\", newline=\"\\n\") as f:\n\n\nSUMMARY:\n**Summary:**\n\nUpdates the `version` parameter within a specified file of type `file_type`.\n\n**Key Implementation Details:**\n\n* Reads the contents of the file and replaces the `VERSION` placeholder with the provided `version`.\n* Uses regular expression substitution (`REPLACE_PATTERNS[file_type]`) to update the version in the code.\n* Writes the updated code back to the original file.\n\n**Notable Dependencies:**\n\n* Expects `REPLACE_PATTERNS` dictionary containing replacement patterns for different file types."}


created_at: 2025-02-05 09:13:51
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/release.py:88-103
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 103, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/release.py", "name": "update_version_in_examples", "start_line": 88, "type": "function", "chroma:document": "CODE:\n\ndef update_version_in_examples(version: str):\n    \"\"\"\n    Update the version in all examples files.\n\n    Args:\n        version (`str`): The new version to set in the examples.\n    \"\"\"\n    for folder, directories, fnames in os.walk(PATH_TO_EXAMPLES):\n        # Removing some of the folders with non-actively maintained examples from the walk\n        if \"research_projects\" in directories:\n            directories.remove(\"research_projects\")\n        if \"legacy\" in directories:\n            directories.remove(\"legacy\")\n        for fname in fnames:\n            if fname.endswith(\".py\"):\n\n\nSUMMARY:\n**Summary**\n\n`update_version_in_examples` function updates the version of Python files in all example files within the `PATH_TO_EXAMPLES` directory. It recursively walks through the directories, filtering out inactive projects and non-Python files.\n\n**Key Implementation Details**\n\n* Removes outdated project folders (\"research_projects\" and \"legacy\")\n* Iterates over file extensions (.py) to detect Python files\n* Calls `update_version_in_file` function for each Python file to update its version with the specified `version`\n\n**Notable Dependencies or Requirements**\n\n* Uses `os` module for walking directories and accessing paths\n* Utilizes `PATH_TO_EXAMPLES` environment variable, not defined in this code snippet.\n\n* `update_version_in_file` function is called but not defined within this scope. It's likely a separate function that updates the version of the specified file."}


created_at: 2025-02-05 09:13:51
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/release.py:106-118
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 118, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/release.py", "name": "global_version_update", "start_line": 106, "type": "function", "chroma:document": "CODE:\n\ndef global_version_update(version: str, patch: bool = False):\n    \"\"\"\n    Update the version in all needed files.\n\n    Args:\n        version (`str`): The new version to set everywhere.\n        patch (`bool`, *optional*, defaults to `False`): Whether or not this is a patch release.\n    \"\"\"\n    for pattern, fname in REPLACE_FILES.items():\n        update_version_in_file(fname, version, pattern)\n    if not patch:\n        # We don't update the version in the examples for patch releases.\n\n\nSUMMARY:\n**Function Summary:**\n\n`global_version_update`\n\nUpdates the specified version in all required files.\n\n* Parameters:\n\t+ `version`: New version string\n\t+ `patch`: Boolean indicating patch release (default: False)\n\nKey Implementation Details:\n\n* Iterates through `REPLACE_FILES` dictionary to update files with pattern matches.\n* Calls `update_version_in_file` method for each matched file.\n* Excludes updating examples if not a patch release.\n\nNotable Dependencies/Requirements:\n\n* `REPLACE_FILES` dictionary\n* `update_version_in_file` method\n* `update_version_in_examples` function"}


created_at: 2025-02-05 09:13:51
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/release.py:121-129
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 129, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/release.py", "name": "remove_conversion_scripts", "start_line": 121, "type": "function", "chroma:document": "CODE:\n\ndef remove_conversion_scripts():\n    \"\"\"\n    Delete the scripts that convert models from older, unsupported formats. We don't want to include these\n    in release wheels because they often have to open insecure file types (pickle, Torch .bin models). This results in\n    vulnerability scanners flagging us and can cause compliance issues for users with strict security policies.\n    \"\"\"\n    model_dir = Path(PATH_TO_MODELS)\n    for conversion_script in list(model_dir.glob(\"**/convert*.py\")):\n\n\nSUMMARY:\n**Summary:**\n`remove_conversion_scripts()` is a function that deletes Python scripts responsible for converting models from older, unsupported formats. The purpose is to prevent vulnerable file types from being included in release wheels, thereby avoiding compliance issues.\n\n**Key Details:**\n\n* Removes Python scripts (`convert*.py`) from the `PATH_TO_MODELS` directory using `globbing`.\n* Deletes each script found using `unlink()` after listing them for removal.\n\n**Notable Interface/Class:**\n\n* `Path`: provides file system path manipulation.\n* `glob()`: performs Unix shell-style string matching on paths.\n\n**External Dependencies:**\n\nNone noted."}


created_at: 2025-02-05 09:13:51
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/release.py:132-139
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 139, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/release.py", "name": "get_version", "start_line": 132, "type": "function", "chroma:document": "CODE:\n\ndef get_version() -> packaging.version.Version:\n    \"\"\"\n    Reads the current version in the main __init__.\n    \"\"\"\n    with open(REPLACE_FILES[\"init\"], \"r\") as f:\n        code = f.read()\n    default_version = REPLACE_PATTERNS[\"init\"][0].search(code).groups()[0]\n\n\nSUMMARY:\n**Code Summary**\n\nRetrieves the current version from the `__init__.py` file and parses it into a `packaging.version.Version` object.\n\n**Key Details**\n\n* Reads content from `REPLACE_FILES[\"init\"]`\n* Searches for default version in `code` using regex pattern `REPLACE_PATTERNS[\"init\"][0]`\n* Returns parsed version using `packaging.version.parse()`\n\n**Dependencies**\n\n* Expects `packaging` library with `version` module\n* Uses regular expressions (`re`) with `search()` method"}


created_at: 2025-02-05 09:13:51
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/release.py:142-171
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 171, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/release.py", "name": "pre_release_work", "start_line": 142, "type": "function", "chroma:document": "CODE:\n\ndef pre_release_work(patch: bool = False):\n    \"\"\"\n    Do all the necessary pre-release steps:\n    - figure out the next minor release version and ask confirmation\n    - update the version everywhere\n    - clean-up the model list in the main README\n\n    Args:\n        patch (`bool`, *optional*, defaults to `False`): Whether or not this is a patch release.\n    \"\"\"\n    # First let's get the default version: base version if we are in dev, bump minor otherwise.\n    default_version = get_version()\n    if patch and default_version.is_devrelease:\n        raise ValueError(\"Can't create a patch version from the dev branch, checkout a released version!\")\n    if default_version.is_devrelease:\n        default_version = default_version.base_version\n    elif patch:\n        default_version = f\"{default_version.major}.{default_version.minor}.{default_version.micro + 1}\"\n    else:\n        default_version = f\"{default_version.major}.{default_version.minor + 1}.0\"\n\n    # Now let's ask nicely if we have found the right version.\n    version = input(f\"Which version are you releasing? [{default_version}]\")\n    if len(version) == 0:\n        version = default_version\n\n    print(f\"Updating version to {version}.\")\n    global_version_update(version, patch=patch)\n    print(\"Deleting conversion scripts.\")\n\n\nSUMMARY:\n**Function Summary: `pre_release_work`**\n\n### Main Purpose and Functionality\nThe `preReleaseWork` function performs necessary pre-release steps for maintaining version control and cleanup. It asks confirmation, updates versions, cleans up model lists in README files, and deletes conversion scripts.\n\n### Key Implementation Details\n\n* Returns a default minor release version if patching, otherwise increments the minor version.\n* Asks user confirmation to choose the release version.\n* Updates global `version` using `global_version_update`.\n* Removes conversion scripts using `remove_conversion_scripts`.\n\n### Notable Dependencies/Requirements\n\n* `get_version()` function\n* Global variables: `base_version`, `major`, `minor`, and `micro`\n\n### Interfaces Implemented/Inherited\n\n* None mentioned, but may be inherited from parent class or interface."}


created_at: 2025-02-05 09:13:51
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/release.py:174-192
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 192, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/release.py", "name": "post_release_work", "start_line": 174, "type": "function", "chroma:document": "CODE:\n\ndef post_release_work():\n    \"\"\"\n    Do all the necessary post-release steps:\n    - figure out the next dev version and ask confirmation\n    - update the version everywhere\n    - clean-up the model list in the main README\n    \"\"\"\n    # First let's get the current version\n    current_version = get_version()\n    dev_version = f\"{current_version.major}.{current_version.minor + 1}.0.dev0\"\n    current_version = current_version.base_version\n\n    # Check with the user we got that right.\n    version = input(f\"Which version are we developing now? [{dev_version}]\")\n    if len(version) == 0:\n        version = dev_version\n\n    print(f\"Updating version to {version}.\")\n\n\nSUMMARY:\n**Summary: post_release_work Function**\n\n**Purpose and Functionality:**\nThe `post_release_work` function performs the necessary steps after a release, including:\n\n* Obtaining the current version\n* Confirming with the user that the next development version is correct\n* Updating all versions accordingly\n\n**Key Implementation Details:**\n\n* Calls `get_version()` to retrieve the current version.\n* Calculates the next development version by incrementing the minor version of the current version.\n* Queries the user to confirm the chosen version using `input()`.\n* Updates global versions using `global_version_update(version)`.\n\n**Notable Dependencies and Requirements:**\nNo external dependencies mentioned. However, it is assumed that:\n\n* `get_version()` function is available and returns a Version object with attributes such as `major`, `minor`, `base_version`.\n* `global_version_update()` function is available and accepts a version string as an argument.\n\n**Interface and Inheritance:**\n\n* Implements no explicit interfaces. However, it assumes the existence of custom attributes for the `Version` class returned by `get_version()`."}


created_at: 2025-02-05 09:13:51
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/process_test_artifacts.py:29-35
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 35, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/process_test_artifacts.py", "name": "count_lines", "start_line": 29, "type": "function", "chroma:document": "CODE:\n\ndef count_lines(filepath):\n    \"\"\"Count the number of lines in a file.\"\"\"\n    try:\n        with open(filepath, \"r\") as f:\n            return len(f.read().split(\"\\n\"))\n    except FileNotFoundError:\n\n\nSUMMARY:\n**Summary:**\n\n`count_lines` function counts the number of lines in a given file.\n \n**Key Implementation Details:**\n- Reads a file line by line using `open` and stores content in a variable (`f.read()`).\n- Splits content into lines using newline character (`\"\\n\"`).\n- Returns the count of lines.\n\n**Notable Dependencies:**\n- [Filesystem module](https://docs.python.org/3/library/os.html#module-os) for file operations.\n- This function does not use any external Python libraries, it is a fundamental file handling task."}


created_at: 2025-02-05 09:13:51
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/process_test_artifacts.py:38-43
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 43, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/process_test_artifacts.py", "name": "compute_parallel_nodes", "start_line": 38, "type": "function", "chroma:document": "CODE:\n\ndef compute_parallel_nodes(line_count, max_tests_per_node=10):\n    \"\"\"Compute the number of parallel nodes required.\"\"\"\n    num_nodes = math.ceil(line_count / AVERAGE_TESTS_PER_NODES)\n    if line_count < 4:\n        return 1\n\n\nSUMMARY:\n**Summary:**\n\n* **Main Purpose:** Computes the minimum number of parallel nodes required to execute a given line count of code with a specified maximum tests per node.\n\n* **Key Implementation Details:**\n  - Calculates `num_nodes` based on `line_count` and average tests per nodes.\n  - Applies a soft cap at 1 node for small line counts (<4) to prevent division by zero errors.\n\n* **Notable Dependencies/Requirements:**\n  - `math.ceil()`: used to calculate ceiling of the division result\n  - `AVERAGE_TESTS_PER_NODES`, `MAX_PARALLEL_NODES`: constants imported from parent module, representing average tests per nodes and maximum parallel nodes allowed, respectively."}


created_at: 2025-02-05 09:13:51
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/process_test_artifacts.py:46-68
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 68, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/process_test_artifacts.py", "name": "process_artifacts", "start_line": 46, "type": "function", "chroma:document": "CODE:\n\ndef process_artifacts(input_file, output_file):\n    # Read the JSON data from the input file\n    with open(input_file, \"r\") as f:\n        data = json.load(f)\n\n    # Process items and build the new JSON structure\n    transformed_data = {}\n    for item in data.get(\"items\", []):\n        if \"test_list\" in item[\"path\"]:\n            key = os.path.splitext(os.path.basename(item[\"path\"]))[0]\n            transformed_data[key] = item[\"url\"]\n            parallel_key = key.split(\"_test\")[0] + \"_parallelism\"\n            file_path = os.path.join(\"test_preparation\", f\"{key}.txt\")\n            line_count = count_lines(file_path)\n            transformed_data[parallel_key] = compute_parallel_nodes(line_count)\n\n    # Remove the \"generated_config\" key if it exists\n    if \"generated_config\" in transformed_data:\n        del transformed_data[\"generated_config\"]\n\n    # Write the transformed data to the output file\n    with open(output_file, \"w\") as f:\n\n\nSUMMARY:\n**Function Summary: `process_artifacts`**\n\n### Main Purpose and Functionality\n\nThe `process_artifacts` function reads JSON data from an input file, transforms the data based on specific rules, and writes the transformed data to an output file. The primary purpose is to process artifacts related to test preparation.\n\n**Transformation Rules:**\n\n* If an item's path contains a \"test_list\" key, extract the key by removing the file extension using `os.path.basename`.\n* Compute parallelism levels based on count of nodes in corresponding text files using `compute_parallel_nodes`.\n* Store transformed data in a new JSON structure with `transformed_data`.\n\n**Key Implementation Details:**\n\n* Uses `json.load` to read input JSON file.\n* Utilizes `count_lines` function to compute parallelism levels.\n\n**Notable Dependencies or Requirements:**\n\n* External dependency on the `os`, `json`, and `file_utils` (assuming `compute_parallel_nodes` is an external function).\n* Assumed implementation of `compute_parallel_nodes` function is not shown in this code snippet.\n\n### Additional Information:\n\nThe transformed data will be written to the output file using `json.dump`. The `\"generated_config\"` key from the input data will be removed if present during transformation."}


created_at: 2025-02-05 09:13:51
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/patch_helper.py:43-53
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 53, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/patch_helper.py", "name": "get_merge_commit", "start_line": 43, "type": "function", "chroma:document": "CODE:\n\ndef get_merge_commit(repo, pr_number, since_tag):\n    try:\n        # Use git log to find the merge commit for the PR within the given tag range\n        merge_commit = next(repo.iter_commits(f\"v{since_tag}...origin/main\", grep=f\"#{pr_number}\"))\n        return merge_commit\n    except StopIteration:\n        print(f\"No merge commit found for PR #{pr_number} between tags {since_tag} and {main}\")\n        return None\n    except GitCommandError as e:\n        print(f\"Error finding merge commit for PR #{pr_number}: {str(e)}\")\n\n\nSUMMARY:\n**Summary:**\n\n### Purpose:\n\nThis function retrieves the merge commit in a GitHub repository (using `git` commands) for a specific pull request (PR), given a tag range (`since_tag`).\n\n### Functionality:\n\n1. **Searches for merge commit**: Uses `repo.iter_commits` to find the first commit that matches the PR number within the specified tag range.\n2. **Catches errors**:\n\t* `StopIteration`: No matching commit found; returns `None`.\n\t* `GitCommandError`: Error during command execution (e.g., invalid syntax); prints error message and returns `None`.\n\n### Key Details:\n\n* Method: `repo.iter_commits`\n* Git log filter: `f\"v{since_tag}...origin/main and grep=f#{pr_number}\"`\n* Returns a single merge commit object on success or `None` on failure.\n\n**Notable Dependencies:** None\n\n**Interfaces and Requirements:** Uses the `GitCommandError` exception class, which is part of GitHub's Git API. The repository object (`repo`) is assumed to have an `iter_commits` method."}


created_at: 2025-02-05 09:13:52
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/patch_helper.py:56-84
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 84, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/patch_helper.py", "name": "main", "start_line": 56, "type": "function", "chroma:document": "CODE:\n\ndef main(pr_numbers):\n    repo = Repo(\".\")  # Initialize the Repo object for the current directory\n    merge_commits = []\n\n    tags = {}\n    for tag in repo.tags:\n        try:\n            # Parse and sort tags, skip invalid ones\n            tag_ver = version.parse(tag.name)\n            tags[tag_ver] = tag\n        except Exception:\n            print(f\"Skipping invalid version tag: {tag.name}\")\n\n    last_tag = sorted(tags)[-1]\n    major_minor = f\"{last_tag.major}.{last_tag.minor}.0\"\n    # Iterate through tag ranges to find the merge commits\n    for pr in pr_numbers:\n        pr = pr.split(\"https://github.com/huggingface/transformers/pull/\")[-1]\n        commit = get_merge_commit(repo, pr, major_minor)\n        if commit:\n            merge_commits.append(commit)\n\n    # Sort commits by date\n    merge_commits.sort(key=lambda commit: commit.committed_datetime)\n\n    # Output the git cherry-pick commands\n    print(\"Git cherry-pick commands to run:\")\n    for commit in merge_commits:\n\n\nSUMMARY:\n**Summary:**\n\n### Main Functionality\n\nThe `main` function takes a list of PR numbers and generates Git cherry-pick commands to apply merge commits across tag ranges.\n\n**Key Implementation Details:**\n\n* **Git Repository Management**: The function initializes a Repo object for the current directory and iterates through GitHub tags to parse and sort valid version tags.\n* **Tag Range Iteration**: It finds merge commits by iterating through PR numbers, splitting them into commit hashes, and using `get_merge_commit` function to retrieve them from the Git repository.\n\n**Notable Dependencies or Requirements:**\n\n* `Repo()` object (assumed to be a GitHub API client)\n* External dependency on `version.parse()` for tag validation\n* Inherited interface from `get_merge_commit()` function, which is not provided in this code snippet\n\n### Code Summary:\n\n```\ndef main(pr_numbers):\n    # Initialize Repo and manage tags\n    repo = Repo(\".\")  # ...\n    tags = {}  # ...\n\n    # Find merge commits across PR tag ranges\n    for pr in pr_numbers:\n        commit = get_merge_commit(repo, pr, major_minor)\n        if commit:\n            merge_commits.append(commit)\n\n    # Sort commits by date and output commands\n    merge_commits.sort(key=lambda commit: commit.committed_datetime)\n    print(\"Git cherry-pick commands to run:\")\n    for commit in merge_commits:\n        print(f\"git cherry-pick {commit.hexsha} #{commit.committed_datetime}\")\n```\n\nThis summary aims to preserve the essence of the original code while omitting minor details and non-essential information."}


created_at: 2025-02-05 09:13:52
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py:125-154
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 154, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py", "name": "clean_code", "start_line": 125, "type": "function", "chroma:document": "CODE:\n\ndef clean_code(content: str) -> str:\n    \"\"\"\n    Remove docstrings, empty line or comments from some code (used to detect if a diff is real or only concern\n    comments or docstings).\n\n    Args:\n        content (`str`): The code to clean\n\n    Returns:\n        `str`: The cleaned code.\n    \"\"\"\n    # We need to deactivate autoformatting here to write escaped triple quotes (we cannot use real triple quotes or\n    # this would mess up the result if this function applied to this particular file).\n    # fmt: off\n    # Remove docstrings by splitting on triple \" then triple ':\n    splits = content.split('\\\"\\\"\\\"')\n    content = \"\".join(splits[::2])\n    splits = content.split(\"\\'\\'\\'\")\n    # fmt: on\n    content = \"\".join(splits[::2])\n\n    # Remove empty lines and comments\n    lines_to_keep = []\n    for line in content.split(\"\\n\"):\n        # remove anything that is after a # sign.\n        line = re.sub(\"#.*$\", \"\", line)\n        # remove white lines\n        if len(line) != 0 and not line.isspace():\n            lines_to_keep.append(line)\n\n\nSUMMARY:\n**Function Summary:**\n\n`clean_code(content: str) -> str`\n\nRemoves docstrings, empty lines, and comments from the provided code content to detect true changes.\n\n**Implementation Highlights:**\n\n* Uses triple quotes (`\"\\\"\\\"\\\"\"`, `'\\'\\'\\''`) to split out docstrings\n* Removes empty lines by checking for non-whitespace characters\n* Removes comments using regular expression substitution (`#.*$`)\n\n**Notable Dependencies/Requirements:**\n\n* None listed, but likely uses built-in Python modules such as `re` (regular expressions)"}


created_at: 2025-02-05 09:13:52
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py:157-181
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 181, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py", "name": "keep_doc_examples_only", "start_line": 157, "type": "function", "chroma:document": "CODE:\n\ndef keep_doc_examples_only(content: str) -> str:\n    \"\"\"\n    Remove everything from the code content except the doc examples (used to determined if a diff should trigger doc\n    tests or not).\n\n    Args:\n        content (`str`): The code to clean\n\n    Returns:\n        `str`: The cleaned code.\n    \"\"\"\n    # Keep doc examples only by splitting on triple \"`\"\n    splits = content.split(\"```\")\n    # Add leading and trailing \"```\" so the navigation is easier when compared to the original input `content`\n    content = \"```\" + \"```\".join(splits[1::2]) + \"```\"\n\n    # Remove empty lines and comments\n    lines_to_keep = []\n    for line in content.split(\"\\n\"):\n        # remove anything that is after a # sign.\n        line = re.sub(\"#.*$\", \"\", line)\n        # remove white lines\n        if len(line) != 0 and not line.isspace():\n            lines_to_keep.append(line)\n\n\nSUMMARY:\n**Function Summary: `keep_doc_examples_only`**\n\nRemoves non-documentation examples from provided code content, leaving only comments.\n\n**Key Implementation Details:**\n\n*   Splits input content by triple quotes (`\"\"\"`) to isolate documentation sections.\n*   Trims leading and trailing whitespace from each line in the split content to eliminate unnecessary characters.\n*   Removes commented-out lines by removing everything after a `#` sign using regular expressions.\n*   Returns the cleaned code content as a single string.\n\n**Notable Dependencies or Requirements:**\n\n*   None found.\n\n**Return Value Signature:**\n```python\ndef keep_doc_examples_only(content: str) -> str:\n```\n\n**Main Functionality:**\nThe function's primary purpose is to filter out non-documentation examples in the input code content, leaving only comments. The filtered comments can then be used to trigger doc tests or documentation checks without including unnecessary code snippets."}


created_at: 2025-02-05 09:13:52
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py:184-210
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 210, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py", "name": "get_all_tests", "start_line": 184, "type": "function", "chroma:document": "CODE:\n\ndef get_all_tests() -> List[str]:\n    \"\"\"\n    Walks the `tests` folder to return a list of files/subfolders. This is used to split the tests to run when using\n    paralellism. The split is:\n\n    - folders under `tests`: (`tokenization`, `pipelines`, etc) except the subfolder `models` is excluded.\n    - folders under `tests/models`: `bert`, `gpt2`, etc.\n    - test files under `tests`: `test_modeling_common.py`, `test_tokenization_common.py`, etc.\n    \"\"\"\n\n    # test folders/files directly under `tests` folder\n    tests = os.listdir(PATH_TO_TESTS)\n    tests = [f\"tests/{f}\" for f in tests if \"__pycache__\" not in f]\n    tests = sorted([f for f in tests if (PATH_TO_REPO / f).is_dir() or f.startswith(\"tests/test_\")])\n\n    # model specific test folders\n    model_test_folders = os.listdir(PATH_TO_TESTS / \"models\")\n    model_test_folders = [f\"tests/models/{f}\" for f in model_test_folders if \"__pycache__\" not in f]\n    model_test_folders = sorted([f for f in model_test_folders if (PATH_TO_REPO / f).is_dir()])\n\n    tests.remove(\"tests/models\")\n    # Sagemaker tests are not meant to be run on the CI.\n    if \"tests/sagemaker\" in tests:\n        tests.remove(\"tests/sagemaker\")\n    tests = model_test_folders + tests\n\n\n\nSUMMARY:\n**Summary**\n\nThe `get_all_tests` function generates a list of files and subfolders to run tests, leveraging parallelism. It walks the `tests` folder, excluding subfolders like `models` and only including specific ones like `tokenization` and `pipelines`. Model-specific test folders (`bert`, `gpt2`) are also included.\n\n**Key Implementation Details**\n\n* Returns the following types of contents:\n\t+ Directly under `tests`: folders with directories or files starting with `test_`.\n\t+ Under `tests/models`: model-specific folder names.\n* Sorts and excludes certain items:\n\t+ Removes `.pyc` files from subfolders under `tests`.\n\t+ Excludes `models` subfolder from the main `tests` list.\n\t+ Removes the `sagemaker` package if found in the main test list.\n\n**Notable Dependencies or Requirements**\n\n* Uses `os.listdir` to interact with the file system.\n* References `PATH_TO_REPO`, `PATH_TO_TESTS`, and `SAGEMAKER Package` by exact name."}


created_at: 2025-02-05 09:13:52
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py:213-236
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 236, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py", "name": "diff_is_docstring_only", "start_line": 213, "type": "function", "chroma:document": "CODE:\n\ndef diff_is_docstring_only(repo: Repo, branching_point: str, filename: str) -> bool:\n    \"\"\"\n    Check if the diff is only in docstrings (or comments and whitespace) in a filename.\n\n    Args:\n        repo (`git.Repo`): A git repository (for instance the Transformers repo).\n        branching_point (`str`): The commit reference of where to compare for the diff.\n        filename (`str`): The filename where we want to know if the diff isonly in docstrings/comments.\n\n    Returns:\n        `bool`: Whether the diff is docstring/comments only or not.\n    \"\"\"\n    folder = Path(repo.working_dir)\n    with checkout_commit(repo, branching_point):\n        with open(folder / filename, \"r\", encoding=\"utf-8\") as f:\n            old_content = f.read()\n\n    with open(folder / filename, \"r\", encoding=\"utf-8\") as f:\n        new_content = f.read()\n\n    old_content_clean = clean_code(old_content)\n    new_content_clean = clean_code(new_content)\n\n\n\nSUMMARY:\n**Summary:**\n\n`diff_is_docstring_only` function checks if the diff between two commits in a Git repository is only docstrings and/or whitespace modifications in a specified filename.\n\n**Key Implementation Details:**\n\n* Compares old and new commit content using `clean_code()` function.\n* Uses `git.Repo` to access working directory and checkout specific branching point.\n* Opens files in same encoding for consistency.\n\n**Return Value:**\n\nReturns `bool` indicating whether the diff is docstring/comments only or not. \n\n**Notable Dependencies:**\n\n* `git.Repo`\n* `Path` from `pathlib`\n* `clean_code()` function (exact implementation unknown)"}


created_at: 2025-02-05 09:13:52
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py:239-262
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 262, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py", "name": "diff_contains_doc_examples", "start_line": 239, "type": "function", "chroma:document": "CODE:\n\ndef diff_contains_doc_examples(repo: Repo, branching_point: str, filename: str) -> bool:\n    \"\"\"\n    Check if the diff is only in code examples of the doc in a filename.\n\n    Args:\n        repo (`git.Repo`): A git repository (for instance the Transformers repo).\n        branching_point (`str`): The commit reference of where to compare for the diff.\n        filename (`str`): The filename where we want to know if the diff is only in codes examples.\n\n    Returns:\n        `bool`: Whether the diff is only in code examples of the doc or not.\n    \"\"\"\n    folder = Path(repo.working_dir)\n    with checkout_commit(repo, branching_point):\n        with open(folder / filename, \"r\", encoding=\"utf-8\") as f:\n            old_content = f.read()\n\n    with open(folder / filename, \"r\", encoding=\"utf-8\") as f:\n        new_content = f.read()\n\n    old_content_clean = keep_doc_examples_only(old_content)\n    new_content_clean = keep_doc_examples_only(new_content)\n\n\n\nSUMMARY:\n**_summary_**\n\n`diff_contains_doc_examples(repo, branching_point, filename)`: Compares two versions of a file within a Git repository to determine if the diff is exclusively code examples.\n\n**Key details:**\n\n* **Functionality**: Exclusively checks for changes in code snippets versus documentation examples.\n* **Implementation**:\n\t+ Retrieves file contents from working directory using `checkout_commit` and `open`.\n\t+ Applies `keep_doc_examples_only` function to clean up content by removing non-example sections.\n\t+ Returns `True` if differing example content is found, otherwise `False`.\n\n**External dependencies:**\n\n* `git.Repo`\n* `Path`\n* `keep_doc_examples_only` (assumed to be implemented elsewhere)\n\n**Interfaces/Methods used:**\n\n* `git.Repo.checkout_commit(branching_point)`\n* `github.Path.open(filename, encoding=\"utf-8\")`\n\nNote that specific implementation details like the `keep_doc_examples_only` function are assumed to be implemented elsewhere in the codebase."}


created_at: 2025-02-05 09:13:52
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py:265-378
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 378, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py", "name": "get_impacted_files_from_tiny_model_summary", "start_line": 265, "type": "function", "chroma:document": "CODE:\n\ndef get_impacted_files_from_tiny_model_summary(diff_with_last_commit: bool = False) -> List[str]:\n    \"\"\"\n    Return a list of python modeling files that are impacted by the changes of `tiny_model_summary.json` in between:\n\n    - the current head and the main branch if `diff_with_last_commit=False` (default)\n    - the current head and its parent commit otherwise.\n\n    Returns:\n        `List[str]`: The list of Python modeling files that are impacted by the changes of `tiny_model_summary.json`.\n    \"\"\"\n    repo = Repo(PATH_TO_REPO)\n\n    folder = Path(repo.working_dir)\n\n    if not diff_with_last_commit:\n        print(f\"main is at {repo.refs.main.commit}\")\n        print(f\"Current head is at {repo.head.commit}\")\n\n        commits = repo.merge_base(repo.refs.main, repo.head)\n        for commit in commits:\n            print(f\"Branching commit: {commit}\")\n    else:\n        print(f\"main is at {repo.head.commit}\")\n        commits = repo.head.commit.parents\n        for commit in commits:\n            print(f\"Parent commit: {commit}\")\n\n    if not os.path.isfile(folder / \"tests/utils/tiny_model_summary.json\"):\n        return []\n\n    files = set()\n    for commit in commits:\n        with checkout_commit(repo, commit):\n            with open(folder / \"tests/utils/tiny_model_summary.json\", \"r\", encoding=\"utf-8\") as f:\n                old_content = f.read()\n\n        with open(folder / \"tests/utils/tiny_model_summary.json\", \"r\", encoding=\"utf-8\") as f:\n            new_content = f.read()\n\n        # get the content as json object\n        old_content = json.loads(old_content)\n        new_content = json.loads(new_content)\n\n        old_keys = set(old_content.keys())\n        new_keys = set(new_content.keys())\n\n        # get the difference\n        keys_with_diff = old_keys.symmetric_difference(new_keys)\n        common_keys = old_keys.intersection(new_keys)\n        # if both have the same key, check its content\n        for key in common_keys:\n            if old_content[key] != new_content[key]:\n                keys_with_diff.add(key)\n\n        # get the model classes\n        impacted_model_classes = []\n        for key in keys_with_diff:\n            if key in new_keys:\n                impacted_model_classes.extend(new_content[key][\"model_classes\"])\n\n        # get the module where the model classes are defined. We want to use the main `__init__` file, but it requires\n        # all the framework being installed, which is not ideal for a simple script like test fetcher.\n        # So we create a temporary and modified main `__init__` and access its `_import_structure`.\n        with open(folder / \"src/transformers/__init__.py\") as fp:\n            lines = fp.readlines()\n            new_lines = []\n            # Get all the code related to `_import_structure`\n            for line in lines:\n                if line == \"_import_structure = {\\n\":\n                    new_lines.append(line)\n                elif line == \"# Direct imports for type-checking\\n\":\n                    break\n                elif len(new_lines) > 0:\n                    # bypass the framework check so we can get all the information even if frameworks are not available\n                    line = re.sub(r\"is_.+_available\\(\\)\", \"True\", line)\n                    line = line.replace(\"OptionalDependencyNotAvailable\", \"Exception\")\n                    line = line.replace(\"Exception()\", \"Exception\")\n                    new_lines.append(line)\n\n        # create and load the temporary module\n        with tempfile.TemporaryDirectory() as tmpdirname:\n            with open(os.path.join(tmpdirname, \"temp_init.py\"), \"w\") as fp:\n                fp.write(\"\".join(new_lines))\n\n            spec = importlib.util.spec_from_file_location(\"temp_init\", os.path.join(tmpdirname, \"temp_init.py\"))\n            module = importlib.util.module_from_spec(spec)\n            spec.loader.exec_module(module)\n            # Finally, get `_import_structure` that we need\n            import_structure = module._import_structure\n\n            # map model classes to their defined module\n            reversed_structure = {}\n            for key, values in import_structure.items():\n                for value in values:\n                    reversed_structure[value] = key\n\n            # Get the corresponding modeling file path\n            for model_class in impacted_model_classes:\n                module = reversed_structure[model_class]\n                framework = \"\"\n                if model_class.startswith(\"TF\"):\n                    framework = \"tf\"\n                elif model_class.startswith(\"Flax\"):\n                    framework = \"flax\"\n                fn = (\n                    f\"modeling_{module.split('.')[-1]}.py\"\n                    if framework == \"\"\n                    else f\"modeling_{framework}_{module.split('.')[-1]}.py\"\n                )\n                files.add(\n                    f\"src.transformers.{module}.{fn}\".replace(\".\", os.path.sep).replace(f\"{os.path.sep}py\", \".py\")\n                )\n\n\n\nSUMMARY:\n**Summary:**\n\nThe `get_impacted_files_from_tiny_model_summary` function returns a list of Python modeling files impacted by changes in `tiny_model_summary.json` between the current head and main branch. If `diff_with_last_commit=False`, it looks for changes in main branch; otherwise, it considers parent commits.\n\n**Key Functionality:**\n\n* Compares `tiny_model_summary.json` contents across multiple commits\n* Identifies changes to model classes and their defined modules\n* Uses a temporary `_import_structure` module to resolve modeling file paths\n\n**Implementation Details:**\n\n* Utilizes Git repository and merge_base functionality\n* Calculates differences between old and new JSON content using symmetric difference and intersection\n* Handles common keys with differing content separately\n* Resolves model class to module mapping using `_import_structure`\n* Uses temporary directory and importlib utilities to load modules and get internal file paths\n\n**Notable Dependencies:**\n\n* `git` for repository access and merge_base functionality\n* `json` and `re` libraries for JSON parsing and regular expression pattern matching\n* `tempfile` and `importlib` libraries for creating temporary directories and loading modules"}


created_at: 2025-02-05 09:13:52
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py:381-420
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 420, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py", "name": "get_diff", "start_line": 381, "type": "function", "chroma:document": "CODE:\n\ndef get_diff(repo: Repo, base_commit: str, commits: List[str]) -> List[str]:\n    \"\"\"\n    Get the diff between a base commit and one or several commits.\n\n    Args:\n        repo (`git.Repo`):\n            A git repository (for instance the Transformers repo).\n        base_commit (`str`):\n            The commit reference of where to compare for the diff. This is the current commit, not the branching point!\n        commits (`List[str]`):\n            The list of commits with which to compare the repo at `base_commit` (so the branching point).\n\n    Returns:\n        `List[str]`: The list of Python files with a diff (files added, renamed or deleted are always returned, files\n        modified are returned if the diff in the file is not only in docstrings or comments, see\n        `diff_is_docstring_only`).\n    \"\"\"\n    print(\"\\n### DIFF ###\\n\")\n    code_diff = []\n    for commit in commits:\n        for diff_obj in commit.diff(base_commit):\n            # We always add new python files\n            if diff_obj.change_type == \"A\" and diff_obj.b_path.endswith(\".py\"):\n                code_diff.append(diff_obj.b_path)\n            # We check that deleted python files won't break corresponding tests.\n            elif diff_obj.change_type == \"D\" and diff_obj.a_path.endswith(\".py\"):\n                code_diff.append(diff_obj.a_path)\n            # Now for modified files\n            elif diff_obj.change_type in [\"M\", \"R\"] and diff_obj.b_path.endswith(\".py\"):\n                # In case of renames, we'll look at the tests using both the old and new name.\n                if diff_obj.a_path != diff_obj.b_path:\n                    code_diff.extend([diff_obj.a_path, diff_obj.b_path])\n                else:\n                    # Otherwise, we check modifications are in code and not docstrings.\n                    if diff_is_docstring_only(repo, commit, diff_obj.b_path):\n                        print(f\"Ignoring diff in {diff_obj.b_path} as it only concerns docstrings or comments.\")\n                    else:\n                        code_diff.append(diff_obj.a_path)\n\n\n\nSUMMARY:\n**Summary:**\n\nThe `get_diff` function calculates the differences between a base commit (`base_commit`) and one or more commits (`commits`). It returns a list of Python files that show differences, including added, renamed, or deleted files.\n\n**Main Purpose and Functionality:**\n\n* Calculates Git diff between base commit and commit pairs\n* Filters out changes only in docstrings or comments in modified files\n\n**Key Implementation Details:**\n\n* Iterates over commits and calculates diffs using `basecommit.diff`\n* Adds new Python files (`change_type == \"A\"`), deletes old Python files (`change_type == \"D\"`), and handles modifications with rename detection (`diff_obj.change_type in [\"M\", \"R\"]`)\n* Uses `diff_is_docstring_only` function to check modifications are not only in docstrings or comments\n\n**Notable Dependencies and Requirements:**\n\n* `git.Repo`: A Git repository object\n* `diff_is_docstring_only`: a separate function (not included in this snippet) that determines if changes are only in docstrings or comments"}


created_at: 2025-02-05 09:13:52
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py:423-450
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 450, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py", "name": "get_modified_python_files", "start_line": 423, "type": "function", "chroma:document": "CODE:\n\ndef get_modified_python_files(diff_with_last_commit: bool = False) -> List[str]:\n    \"\"\"\n    Return a list of python files that have been modified between:\n\n    - the current head and the main branch if `diff_with_last_commit=False` (default)\n    - the current head and its parent commit otherwise.\n\n    Returns:\n        `List[str]`: The list of Python files with a diff (files added, renamed or deleted are always returned, files\n        modified are returned if the diff in the file is not only in docstrings or comments, see\n        `diff_is_docstring_only`).\n    \"\"\"\n    repo = Repo(PATH_TO_REPO)\n\n    if not diff_with_last_commit:\n        print(f\"main is at {repo.refs.main.commit}\")\n        print(f\"Current head is at {repo.head.commit}\")\n\n        branching_commits = repo.merge_base(repo.refs.main, repo.head)\n        for commit in branching_commits:\n            print(f\"Branching commit: {commit}\")\n        return get_diff(repo, repo.head.commit, branching_commits)\n    else:\n        print(f\"main is at {repo.head.commit}\")\n        parent_commits = repo.head.commit.parents\n        for commit in parent_commits:\n            print(f\"Parent commit: {commit}\")\n\n\nSUMMARY:\nHere's a concise summary of the provided code chunk:\n\n**Function Summary:** `get_modified_python_files`\n\n**Purpose and Functionality:** Returns a list of Python files modified between the current head and either its main branch or previous commit (`diff_with_last_commit` flag).\n\n**Key Implementation Details:**\n\n* Uses Git repository manipulation to determine branching commits (if not using last commit)\n* Calculates differences in modified files with `get_diff(repo, repo.head.commit, [commit])` method\n* Prints intermediate commit information for debugging purposes\n\n**Notable Dependencies and Requirements:**\n\n* Git library (`Repo`) for interacting with the repository\n* `get_diff` function (not shown) for calculating file diffs"}


created_at: 2025-02-05 09:13:52
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py:453-491
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 491, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py", "name": "get_diff_for_doctesting", "start_line": 453, "type": "function", "chroma:document": "CODE:\n\ndef get_diff_for_doctesting(repo: Repo, base_commit: str, commits: List[str]) -> List[str]:\n    \"\"\"\n    Get the diff in doc examples between a base commit and one or several commits.\n\n    Args:\n        repo (`git.Repo`):\n            A git repository (for instance the Transformers repo).\n        base_commit (`str`):\n            The commit reference of where to compare for the diff. This is the current commit, not the branching point!\n        commits (`List[str]`):\n            The list of commits with which to compare the repo at `base_commit` (so the branching point).\n\n    Returns:\n        `List[str]`: The list of Python and Markdown files with a diff (files added or renamed are always returned, files\n        modified are returned if the diff in the file is only in doctest examples).\n    \"\"\"\n    print(\"\\n### DIFF ###\\n\")\n    code_diff = []\n    for commit in commits:\n        for diff_obj in commit.diff(base_commit):\n            # We only consider Python files and doc files.\n            if not diff_obj.b_path.endswith(\".py\") and not diff_obj.b_path.endswith(\".md\"):\n                continue\n            # We always add new python/md files\n            if diff_obj.change_type in [\"A\"]:\n                code_diff.append(diff_obj.b_path)\n            # Now for modified files\n            elif diff_obj.change_type in [\"M\", \"R\"]:\n                # In case of renames, we'll look at the tests using both the old and new name.\n                if diff_obj.a_path != diff_obj.b_path:\n                    code_diff.extend([diff_obj.a_path, diff_obj.b_path])\n                else:\n                    # Otherwise, we check modifications contain some doc example(s).\n                    if diff_contains_doc_examples(repo, commit, diff_obj.b_path):\n                        code_diff.append(diff_obj.a_path)\n                    else:\n                        print(f\"Ignoring diff in {diff_obj.b_path} as it doesn't contain any doc example.\")\n\n\n\nSUMMARY:\n**Summary:**\n\nThe `get_diff_for_doctesting` function generates a list of Python and Markdown files with differences between a base commit (`base_commit`) and one or several commits (`commits`). It focuses on changes to doc examples, returning modified files that contain doc examples.\n\n**Key Details:**\n\n*   **Function Signature:** `def get_diff_for_doctesting(repo: Repo, base_commit: str, commits: List[str]) -> List[str]`\n*   **Primary Purpose:** Identify and return Python and Markdown files with differences between base commit and comparison commits.\n*   **Implementation Details:**\n    *   Iterates over each commit in the list of commits.\n    *   Compares diff object of each commit against the base commit.\n    *   Applies filtering on file types (python.py, md) to exclude irrelevant changes.\n    *   Returns modified files containing doc examples using the `diff_contains_doc_examples` function.\n\n**External Dependencies:**\n\n*   `git.Repo`: Git repository class for retrieving information about commits.\n*   `List[str]`: List of commit IDs for comparison.\n*   `Repo` and other classes not mentioned here are assumed to be external dependencies."}


created_at: 2025-02-05 09:13:52
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py:494-524
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 524, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py", "name": "get_all_doctest_files", "start_line": 494, "type": "function", "chroma:document": "CODE:\n\ndef get_all_doctest_files() -> List[str]:\n    \"\"\"\n    Return the complete list of python and Markdown files on which we run doctest.\n\n    At this moment, we restrict this to only take files from `src/` or `docs/source/en/` that are not in `utils/not_doctested.txt`.\n\n    Returns:\n        `List[str]`: The complete list of Python and Markdown files on which we run doctest.\n    \"\"\"\n    py_files = [str(x.relative_to(PATH_TO_REPO)) for x in PATH_TO_REPO.glob(\"**/*.py\")]\n    md_files = [str(x.relative_to(PATH_TO_REPO)) for x in PATH_TO_REPO.glob(\"**/*.md\")]\n\n    test_files_to_run = py_files + md_files\n    # change to use \"/\" as path separator\n    test_files_to_run = [\"/\".join(Path(x).parts) for x in test_files_to_run]\n    # don't run doctest for files in `src/transformers/models/deprecated`\n    test_files_to_run = [x for x in test_files_to_run if \"models/deprecated\" not in x]\n\n    # only include files in `src` or `docs/source/en/`\n    test_files_to_run = [x for x in test_files_to_run if x.startswith((\"src/\", \"docs/source/en/\"))]\n    # not include init files\n    test_files_to_run = [x for x in test_files_to_run if not x.endswith((\"__init__.py\",))]\n\n    # These are files not doctested yet.\n    with open(\"utils/not_doctested.txt\") as fp:\n        not_doctested = {x.split(\" \")[0] for x in fp.read().strip().split(\"\\n\")}\n\n    # So far we don't have 100% coverage for doctest. This line will be removed once we achieve 100%.\n    test_files_to_run = [x for x in test_files_to_run if x not in not_doctested]\n\n\n\nSUMMARY:\n**Summary**\n\nThe `get_all_doctest_files` function retrieves the complete list of Python and Markdown files to run doctest. It filters out files based on their location, extensions, and excluded directories.\n\n*   **Purpose:** Returns a lista of paths by filtering python, markdown file from repository.\n*   **Functionality:**\n    *   Uses `glob` to find `.py` and `.md` files in `src/` or `docs/source/en/`.\n    *   Excludes files in `utils/not_doctested.txt`, `src/transformers/models/deprecated`, and init files.\n    *   Filters out files not marked for testing (`not_doctested.txt`).\n*   **Key Implementation Details:**\n    *   Uses list comprehensions to filter and transform file paths.\n    *   Checks file names against exclusion lists for filtering.\n\n**Dependencies/Requirements:**\n\n*   `glob`\n*   `os.path` (specifically, `relative_to`, `Partition`)\n*   `pathlib`\n\n**Notable Variables:**\n\n*      `PATH_TO_REPO`: repository path\n*      `_` not variable but alias\n\n**Interfaces/Superclasses:**\n\nNone\n\nThis code snippet appears to be part of a larger class or module that handles file system operations and doctest filtering."}


created_at: 2025-02-05 09:13:52
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py:527-551
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 551, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py", "name": "get_new_doctest_files", "start_line": 527, "type": "function", "chroma:document": "CODE:\n\ndef get_new_doctest_files(repo, base_commit, branching_commit) -> List[str]:\n    \"\"\"\n    Get the list of files that were removed from \"utils/not_doctested.txt\", between `base_commit` and\n    `branching_commit`.\n\n    Returns:\n        `List[str]`: List of files that were removed from \"utils/not_doctested.txt\".\n    \"\"\"\n    for diff_obj in branching_commit.diff(base_commit):\n        # Ignores all but the \"utils/not_doctested.txt\" file.\n        if diff_obj.a_path != \"utils/not_doctested.txt\":\n            continue\n        # Loads the two versions\n        folder = Path(repo.working_dir)\n        with checkout_commit(repo, branching_commit):\n            with open(folder / \"utils/not_doctested.txt\", \"r\", encoding=\"utf-8\") as f:\n                old_content = f.read()\n        with open(folder / \"utils/not_doctested.txt\", \"r\", encoding=\"utf-8\") as f:\n            new_content = f.read()\n        # Compute the removed lines and return them\n        removed_content = {x.split(\" \")[0] for x in old_content.split(\"\\n\")} - {\n            x.split(\" \")[0] for x in new_content.split(\"\\n\")\n        }\n        return sorted(removed_content)\n\n\nSUMMARY:\n**Concise Summary**\n\n`get_new_doctest_files` function takes three commit references (`repo`, `base_commit`, and `branching_commit`) and returns a list of files that were removed from \"utils/not_doctested.txt\" between these commits.\n\n**Key Insights:**\n\n* The function loads the old and new contents of \"utils/not_doctested.txt\" for each merge commit.\n* It computes the removed lines by taking the difference between the two versions.\n* Notable dependencies:\n\t+ `Path` from `pathlib`\n\t+ `checkout_commit` from GitPython\n\t+ `diff` from GitPython\n\n**Return Value:** A list of strings representing the files that were removed from \"utils/not_doctested.txt\"."}


created_at: 2025-02-05 09:13:52
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py:554-599
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 599, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py", "name": "get_doctest_files", "start_line": 554, "type": "function", "chroma:document": "CODE:\n\ndef get_doctest_files(diff_with_last_commit: bool = False) -> List[str]:\n    \"\"\"\n    Return a list of python and Markdown files where doc example have been modified between:\n\n    - the current head and the main branch if `diff_with_last_commit=False` (default)\n    - the current head and its parent commit otherwise.\n\n    Returns:\n        `List[str]`: The list of Python and Markdown files with a diff (files added or renamed are always returned, files\n        modified are returned if the diff in the file is only in doctest examples).\n    \"\"\"\n    repo = Repo(PATH_TO_REPO)\n\n    test_files_to_run = []  # noqa\n    if not diff_with_last_commit:\n        print(f\"main is at {repo.refs.main.commit}\")\n        print(f\"Current head is at {repo.head.commit}\")\n\n        branching_commits = repo.merge_base(repo.refs.main, repo.head)\n        for commit in branching_commits:\n            print(f\"Branching commit: {commit}\")\n        test_files_to_run = get_diff_for_doctesting(repo, repo.head.commit, branching_commits)\n    else:\n        print(f\"main is at {repo.head.commit}\")\n        parent_commits = repo.head.commit.parents\n        for commit in parent_commits:\n            print(f\"Parent commit: {commit}\")\n        test_files_to_run = get_diff_for_doctesting(repo, repo.head.commit, parent_commits)\n\n    all_test_files_to_run = get_all_doctest_files()\n\n    # Add to the test files to run any removed entry from \"utils/not_doctested.txt\".\n    new_test_files = get_new_doctest_files(repo, repo.head.commit, repo.refs.main.commit)\n    test_files_to_run = list(set(test_files_to_run + new_test_files))\n\n    # Do not run slow doctest tests on CircleCI\n    with open(\"utils/slow_documentation_tests.txt\") as fp:\n        slow_documentation_tests = set(fp.read().strip().split(\"\\n\"))\n    test_files_to_run = [\n        x for x in test_files_to_run if x in all_test_files_to_run and x not in slow_documentation_tests\n    ]\n\n    # Make sure we did not end up with a test file that was removed\n    test_files_to_run = [f for f in test_files_to_run if (PATH_TO_REPO / f).exists()]\n\n\n\nSUMMARY:\n**Function Summary**\n\n`get_doctest_files(diff_with_last_commit: bool = False) -> List[str]`\n-------------------------\n\nThis function generates a list of Python and Markdown files with modified doctest examples between the current head commit and either:\n\n*   The main branch (if `diff_with_last_commit=False`) or\n*   Its parent commits (otherwise)\n\nThe function performs the following steps:\n\n1.  Checks if there are any modified doctest examples in the given commit range.\n2.  Identifies added, renamed, or modified files containing doctest examples.\n3.  Excludes slow documenting tests to be run on CircleCI.\n4.  Ensures that no removed test files are included.\n\n**Key Implementation Details**\n\n*   Uses `Repo` class from [GitPython](https://gitpython.readthedocs.io/en/stable/) for Git repository management.\n*   Utilizes `get_diff_for_doctesting` function to identify modified doctest examples.\n*   Employs set operations and filtering for efficient list construction.\n\n**Notable Dependencies or Requirements**\n\n*   [GitPython](https://gitpython.readthedocs.io/en/stable/) library for Git repository interactions\n*   External dependencies not specified in the provided code snippet."}


created_at: 2025-02-05 09:13:52
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py:623-702
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 702, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py", "name": "extract_imports", "start_line": 623, "type": "function", "chroma:document": "CODE:\n\ndef extract_imports(module_fname: str, cache: Dict[str, List[str]] = None) -> List[str]:\n    \"\"\"\n    Get the imports a given module makes.\n\n    Args:\n        module_fname (`str`):\n            The name of the file of the module where we want to look at the imports (given relative to the root of\n            the repo).\n        cache (Dictionary `str` to `List[str]`, *optional*):\n            To speed up this function if it was previously called on `module_fname`, the cache of all previously\n            computed results.\n\n    Returns:\n        `List[str]`: The list of module filenames imported in the input `module_fname` (a submodule we import from that\n        is a subfolder will give its init file).\n    \"\"\"\n    if cache is not None and module_fname in cache:\n        return cache[module_fname]\n\n    with open(PATH_TO_REPO / module_fname, \"r\", encoding=\"utf-8\") as f:\n        content = f.read()\n\n    # Filter out all docstrings to not get imports in code examples. As before we need to deactivate formatting to\n    # keep this as escaped quotes and avoid this function failing on this file.\n    splits = content.split('\\\"\\\"\\\"')  # fmt: skip\n    content = \"\".join(splits[::2])\n\n    module_parts = str(module_fname).split(os.path.sep)\n    imported_modules = []\n\n    # Let's start with relative imports\n    relative_imports = _re_single_line_relative_imports.findall(content)\n    relative_imports = [\n        (mod, imp) for mod, imp in relative_imports if \"# tests_ignore\" not in imp and imp.strip() != \"(\"\n    ]\n    multiline_relative_imports = _re_multi_line_relative_imports.findall(content)\n    relative_imports += [(mod, imp) for mod, imp in multiline_relative_imports if \"# tests_ignore\" not in imp]\n\n    # We need to remove parts of the module name depending on the depth of the relative imports.\n    for module, imports in relative_imports:\n        level = 0\n        while module.startswith(\".\"):\n            module = module[1:]\n            level += 1\n\n        if len(module) > 0:\n            dep_parts = module_parts[: len(module_parts) - level] + module.split(\".\")\n        else:\n            dep_parts = module_parts[: len(module_parts) - level]\n        imported_module = os.path.sep.join(dep_parts)\n        imported_modules.append((imported_module, [imp.strip() for imp in imports.split(\",\")]))\n\n    # Let's continue with direct imports\n    direct_imports = _re_single_line_direct_imports.findall(content)\n    direct_imports = [(mod, imp) for mod, imp in direct_imports if \"# tests_ignore\" not in imp and imp.strip() != \"(\"]\n    multiline_direct_imports = _re_multi_line_direct_imports.findall(content)\n    direct_imports += [(mod, imp) for mod, imp in multiline_direct_imports if \"# tests_ignore\" not in imp]\n\n    # We need to find the relative path of those imports.\n    for module, imports in direct_imports:\n        import_parts = module.split(\".\")[1:]  # ignore the name of the repo since we add it below.\n        dep_parts = [\"src\", \"transformers\"] + import_parts\n        imported_module = os.path.sep.join(dep_parts)\n        imported_modules.append((imported_module, [imp.strip() for imp in imports.split(\",\")]))\n\n    result = []\n    # Double check we get proper modules (either a python file or a folder with an init).\n    for module_file, imports in imported_modules:\n        if (PATH_TO_REPO / f\"{module_file}.py\").is_file():\n            module_file = f\"{module_file}.py\"\n        elif (PATH_TO_REPO / module_file).is_dir() and (PATH_TO_REPO / module_file / \"__init__.py\").is_file():\n            module_file = os.path.sep.join([module_file, \"__init__.py\"])\n        imports = [imp for imp in imports if len(imp) > 0 and re.match(\"^[A-Za-z0-9_]*$\", imp)]\n        if len(imports) > 0:\n            result.append((module_file, imports))\n\n    if cache is not None:\n        cache[module_fname] = result\n\n\n\nSUMMARY:\n**Function Summary: `extract_imports`**\n\nExtracts the imports made by a given module from its source code.\n\n**Purpose**\nThe function takes a file name of a module and returns a list of tuples containing the imported module filenames and their corresponding import statements.\n\n**Key Implementation Details**\n\n1. **Relative Imports**: Extracts relative imports using regular expressions, filtering out docstrings and `tests_ignore` comments.\n2. **Direct Imports**: Finds direct imports by parsing lines similar to `import * or from ... import`. The function ensures that only actual import statements are extracted.\n3. **Cache Mechanism**: Stores previously computed results for module names to avoid redundant computations.\n\n**Notable Dependencies and Requirements**\n\n1. `os`: for interacting with the file system.\n2. `re`: for regular expression matches.\n3. `_re_single_line_relative_imports`, `_re_multi_line_relative_imports`, `_re_single_line_direct_imports`, and `_re_multi_line_direct_imports` (custom regexes).\n4. `PATH_TO_REPO` (absolute path to the repository root).\n\n**Primary Purpose**: Extracts imports from a module's source code, providing the necessary information for automated analysis or testing.\n\n```python\ndef extract_imports(module_fname: str, cache: Dict[str, List[str]] = None) -> List[Tuple[str, List[str]]]:\n    # ...\n```\n\nNote that this summary highlights the key aspects of the function and omits unnecessary details. It can be used in a RAG (Readability, Abstractness, Generality) system to provide a concise overview of the functionality."}


created_at: 2025-02-05 09:13:52
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py:705-761
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 761, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py", "name": "get_module_dependencies", "start_line": 705, "type": "function", "chroma:document": "CODE:\n\ndef get_module_dependencies(module_fname: str, cache: Dict[str, List[str]] = None) -> List[str]:\n    \"\"\"\n    Refines the result of `extract_imports` to remove subfolders and get a proper list of module filenames: if a file\n    as an import `from utils import Foo, Bar`, with `utils` being a subfolder containing many files, this will traverse\n    the `utils` init file to check where those dependencies come from: for instance the files utils/foo.py and utils/bar.py.\n\n    Warning: This presupposes that all intermediate inits are properly built (with imports from the respective\n    submodules) and work better if objects are defined in submodules and not the intermediate init (otherwise the\n    intermediate init is added, and inits usually have a lot of dependencies).\n\n    Args:\n        module_fname (`str`):\n            The name of the file of the module where we want to look at the imports (given relative to the root of\n            the repo).\n        cache (Dictionary `str` to `List[str]`, *optional*):\n            To speed up this function if it was previously called on `module_fname`, the cache of all previously\n            computed results.\n\n    Returns:\n        `List[str]`: The list of module filenames imported in the input `module_fname` (with submodule imports refined).\n    \"\"\"\n    dependencies = []\n    imported_modules = extract_imports(module_fname, cache=cache)\n    # The while loop is to recursively traverse all inits we may encounter: we will add things as we go.\n    while len(imported_modules) > 0:\n        new_modules = []\n        for module, imports in imported_modules:\n            # If we end up in an __init__ we are often not actually importing from this init (except in the case where\n            # the object is fully defined in the __init__)\n            if module.endswith(\"__init__.py\"):\n                # So we get the imports from that init then try to find where our objects come from.\n                new_imported_modules = extract_imports(module, cache=cache)\n                for new_module, new_imports in new_imported_modules:\n                    if any(i in new_imports for i in imports):\n                        if new_module not in dependencies:\n                            new_modules.append((new_module, [i for i in new_imports if i in imports]))\n                        imports = [i for i in imports if i not in new_imports]\n                if len(imports) > 0:\n                    # If there are any objects lefts, they may be a submodule\n                    path_to_module = PATH_TO_REPO / module.replace(\"__init__.py\", \"\")\n                    dependencies.extend(\n                        [\n                            os.path.join(module.replace(\"__init__.py\", \"\"), f\"{i}.py\")\n                            for i in imports\n                            if (path_to_module / f\"{i}.py\").is_file()\n                        ]\n                    )\n                    imports = [i for i in imports if not (path_to_module / f\"{i}.py\").is_file()]\n                    if len(imports) > 0:\n                        # Then if there are still objects left, they are fully defined in the init, so we keep it as a\n                        # dependency.\n                        dependencies.append(module)\n            else:\n                dependencies.append(module)\n\n        imported_modules = new_modules\n\n\nSUMMARY:\n**Code Summary:**\n\n### Functionality:\n\nThis function `get_module_dependencies` retrieves the list of module filenames for specified imports in a given Python module.\n\nGiven a `module_fname` and an optional cache (Dict[str, List[str]]), it refines the result of `extract_imports` to remove subfolders.\n\n**Key Implementation Details:**\n\n*   The function uses a while loop to recursively traverse imported modules (`while len(imported_modules) > 0:`)\n*   It filters extracted imports based on submodule conditions: objects defined in submodules are added, not intermediate init file dependencies\n*   Uses `os.path.join` for path concatenation and file checks\n\n**Notable Dependencies or Requirements:**\n\n*   `extract_imports`: a function to extract import statements from a module (not provided in the code snippet)\n*   `PATH_TO_REPO`: a constant referring to the repository root directory\n*   `os` module for file-related operations"}


created_at: 2025-02-05 09:13:52
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py:764-773
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 773, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py", "name": "create_reverse_dependency_tree", "start_line": 764, "type": "function", "chroma:document": "CODE:\n\ndef create_reverse_dependency_tree() -> List[Tuple[str, str]]:\n    \"\"\"\n    Create a list of all edges (a, b) which mean that modifying a impacts b with a going over all module and test files.\n    \"\"\"\n    cache = {}\n    all_modules = list(PATH_TO_TRANFORMERS.glob(\"**/*.py\")) + list(PATH_TO_TESTS.glob(\"**/*.py\"))\n    all_modules = [str(mod.relative_to(PATH_TO_REPO)) for mod in all_modules]\n    edges = [(dep, mod) for mod in all_modules for dep in get_module_dependencies(mod, cache=cache)]\n\n\n\nSUMMARY:\n**Summary**\n\n*   **Functionality**: Creates a list of reverse dependency relationships between modules and their impacting dependencies across the entire repository, excluding tests.\n\n**Key Implementation Details**\n\n*   Uses a cache to store module dependencies for efficiency.\n*   Iterates over all module files in `/src` and `/tests` directories, extracting relative paths.\n\n**Main Purpose and Functionality**\n\nCreates a list of edges (a, b) representing modifying a impacts b due to transitive dependency resolution at compilation-time for the entire project. It excludes test files from the analysis. \n\n**Notable Dependencies or Requirements**\n\n*   `PATH_TO_TRANFormERS` and `PATH_TO_TESTS`: Directory paths.\n*   `get_module_dependencies()`: Function used to determine dependencies of a module, which is not shown in this snippet but presumably interacts with external libraries.\n\n### Example Usage\n```python\nedges = create_reverse_dependency_tree()\nprint(edges)\n```\nThis will output all the edges extracted from the project."}


created_at: 2025-02-05 09:13:52
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py:776-801
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 801, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py", "name": "get_tree_starting_at", "start_line": 776, "type": "function", "chroma:document": "CODE:\n\ndef get_tree_starting_at(module: str, edges: List[Tuple[str, str]]) -> List[Union[str, List[str]]]:\n    \"\"\"\n    Returns the tree starting at a given module following all edges.\n\n    Args:\n        module (`str`): The module that will be the root of the subtree we want.\n        eges (`List[Tuple[str, str]]`): The list of all edges of the tree.\n\n    Returns:\n        `List[Union[str, List[str]]]`: The tree to print in the following format: [module, [list of edges\n        starting at module], [list of edges starting at the preceding level], ...]\n    \"\"\"\n    vertices_seen = [module]\n    new_edges = [edge for edge in edges if edge[0] == module and edge[1] != module and \"__init__.py\" not in edge[1]]\n    tree = [module]\n    while len(new_edges) > 0:\n        tree.append(new_edges)\n        final_vertices = list({edge[1] for edge in new_edges})\n        vertices_seen.extend(final_vertices)\n        new_edges = [\n            edge\n            for edge in edges\n            if edge[0] in final_vertices and edge[1] not in vertices_seen and \"__init__.py\" not in edge[1]\n        ]\n\n\n\nSUMMARY:\n**Concise Summary:**\n\n`get_tree_starting_at` function constructs a tree data structure by scanning a module's edges.\n\n**Main Purpose and Functionality:**\n The function traverses the edges of a Python package starting from a specified `module`, building a tree-like structure recursively. It preserves only non-instantiation edges (`__init__.py`) that do not loop back to their origin.\n\n**Key Implementation Details:**\n\n* Uses a breadth-first search (BFS) approach to traverse the edges.\n* Checks for edge validity based on presence of `__init__.py` or self-looping edges.\n* Maintains two lists: current visible edges (`new_edges`) and seen vertices (`vertices_seen`).\n\n**Notable Dependencies and Requirements:**\n\n* Python 3.x\n* `List`, `Tuple`, and `Union` types from the standard library (not explicitly listed)\n* Implicitly requires a list of edge tuples (`edges`) to be parsed as input."}


created_at: 2025-02-05 09:13:52
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py:804-834
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 834, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py", "name": "print_tree_deps_of", "start_line": 804, "type": "function", "chroma:document": "CODE:\n\ndef print_tree_deps_of(module, all_edges=None):\n    \"\"\"\n    Prints the tree of modules depending on a given module.\n\n    Args:\n        module (`str`): The module that will be the root of the subtree we want.\n        all_eges (`List[Tuple[str, str]]`, *optional*):\n            The list of all edges of the tree. Will be set to `create_reverse_dependency_tree()` if not passed.\n    \"\"\"\n    if all_edges is None:\n        all_edges = create_reverse_dependency_tree()\n    tree = get_tree_starting_at(module, all_edges)\n\n    # The list of lines is a list of tuples (line_to_be_printed, module)\n    # Keeping the modules lets us know where to insert each new lines in the list.\n    lines = [(tree[0], tree[0])]\n    for index in range(1, len(tree)):\n        edges = tree[index]\n        start_edges = {edge[0] for edge in edges}\n\n        for start in start_edges:\n            end_edges = {edge[1] for edge in edges if edge[0] == start}\n            # We will insert all those edges just after the line showing start.\n            pos = 0\n            while lines[pos][1] != start:\n                pos += 1\n            lines = lines[: pos + 1] + [(\" \" * (2 * index) + end, end) for end in end_edges] + lines[pos + 1 :]\n\n    for line in lines:\n        # We don't print the refs that where just here to help build lines.\n\n\nSUMMARY:\n**Summary**\n\nThe `print_tree_deps_of` function prints the tree structure of dependencies for a given module. It takes two arguments: `module` (the root of the subtree) and `all_edges` (an optional list of all edges, which defaults to creating a reverse dependency tree if not provided).\n\n**Key Implementation Details**\n\n* The function uses a dynamic programming approach to build the tree structure by inserting edges into a list, ensuring proper indentation.\n* It uses the `get_tree_starting_at` function to obtain the starting node of the subtree.\n\n**Methods and Interfaces Used:**\n\n* `create_reverse_dependency_tree`\n* `get_tree_starting_at`\n\n**Notable Dependencies:**\n\n* None\n\n**Primary Purpose**\n\nPrints the tree structure of dependencies for a given module."}


created_at: 2025-02-05 09:13:52
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py:837-870
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 870, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py", "name": "init_test_examples_dependencies", "start_line": 837, "type": "function", "chroma:document": "CODE:\n\ndef init_test_examples_dependencies() -> Tuple[Dict[str, List[str]], List[str]]:\n    \"\"\"\n    The test examples do not import from the examples (which are just scripts, not modules) so we need som extra\n    care initializing the dependency map, which is the goal of this function. It initializes the dependency map for\n    example files by linking each example to the example test file for the example framework.\n\n    Returns:\n        `Tuple[Dict[str, List[str]], List[str]]`: A tuple with two elements: the initialized dependency map which is a\n        dict test example file to list of example files potentially tested by that test file, and the list of all\n        example files (to avoid recomputing it later).\n    \"\"\"\n    test_example_deps = {}\n    all_examples = []\n    for framework in [\"flax\", \"pytorch\", \"tensorflow\"]:\n        test_files = list((PATH_TO_EXAMPLES / framework).glob(\"test_*.py\"))\n        all_examples.extend(test_files)\n        # Remove the files at the root of examples/framework since they are not proper examples (they are eith utils\n        # or example test files).\n        examples = [\n            f for f in (PATH_TO_EXAMPLES / framework).glob(\"**/*.py\") if f.parent != PATH_TO_EXAMPLES / framework\n        ]\n        all_examples.extend(examples)\n        for test_file in test_files:\n            with open(test_file, \"r\", encoding=\"utf-8\") as f:\n                content = f.read()\n            # Map all examples to the test files found in examples/framework.\n            test_example_deps[str(test_file.relative_to(PATH_TO_REPO))] = [\n                str(e.relative_to(PATH_TO_REPO)) for e in examples if e.name in content\n            ]\n            # Also map the test files to themselves.\n            test_example_deps[str(test_file.relative_to(PATH_TO_REPO))].append(\n                str(test_file.relative_to(PATH_TO_REPO))\n            )\n\n\nSUMMARY:\n**Summary:**\n\nThis function, `init_test_examples_dependencies`, maps test example files to potential test examples and vice versa for framework-specific test frameworks (Flax, PyTorch, TensorFlow).\n\n*   **Purpose:** Initializes the dependency map for test examples, linking each example to its corresponding test file.\n\n**Key Implementation Details:**\n\n*   Iterates over three predefined framework libraries.\n*   Reads contents of test files for Flax, PyTorch, and TensorFlow from specific directories within `PATH_TO_EXAMPLES`.\n*   Mapped keys between example files and their relative paths as strings (`str()`) using relative path operations.\n\n**Primary Purpose Output:**\n\n`Tuple[Dict[str, List[str]], List[str]]`: A tuple containing the initialized dependency map (`test_example_deps`) and a list of all example files."}


created_at: 2025-02-05 09:13:52
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py:873-924
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 924, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py", "name": "create_reverse_dependency_map", "start_line": 873, "type": "function", "chroma:document": "CODE:\n\ndef create_reverse_dependency_map() -> Dict[str, List[str]]:\n    \"\"\"\n    Create the dependency map from module/test filename to the list of modules/tests that depend on it recursively.\n\n    Returns:\n        `Dict[str, List[str]]`: The reverse dependency map as a dictionary mapping filenames to all the filenames\n        depending on it recursively. This way the tests impacted by a change in file A are the test files in the list\n        corresponding to key A in this result.\n    \"\"\"\n    cache = {}\n    # Start from the example deps init.\n    example_deps, examples = init_test_examples_dependencies()\n    # Add all modules and all tests to all examples\n    all_modules = list(PATH_TO_TRANFORMERS.glob(\"**/*.py\")) + list(PATH_TO_TESTS.glob(\"**/*.py\")) + examples\n    all_modules = [str(mod.relative_to(PATH_TO_REPO)) for mod in all_modules if not (\"models\" in str(mod.relative_to(PATH_TO_REPO)) and \"convert_\" in str(mod.relative_to(PATH_TO_REPO)))]\n    # Compute the direct dependencies of all modules.\n    direct_deps = {m: get_module_dependencies(m, cache=cache) for m in all_modules}\n    direct_deps.update(example_deps)\n\n    # This recurses the dependencies\n    something_changed = True\n    while something_changed:\n        something_changed = False\n        for m in all_modules:\n            for d in direct_deps[m]:\n                # We stop recursing at an init (cause we always end up in the main init and we don't want to add all\n                # files which the main init imports)\n                if d.endswith(\"__init__.py\"):\n                    continue\n                if d not in direct_deps:\n                    if \"convert\" in d:\n                        continue\n                    raise ValueError(f\"KeyError:{d}. From {m}\")\n                new_deps = set(direct_deps[d]) - set(direct_deps[m])\n                if len(new_deps) > 0:\n                    direct_deps[m].extend(list(new_deps))\n                    something_changed = True\n\n    # Finally we can build the reverse map.\n    reverse_map = collections.defaultdict(list)\n    for m in all_modules:\n        for d in direct_deps[m]:\n            reverse_map[d].append(m)\n\n    # For inits, we don't do the reverse deps but the direct deps: if modifying an init, we want to make sure we test\n    # all the modules impacted by that init.\n    for m in [f for f in all_modules if f.endswith(\"__init__.py\")]:\n        direct_deps = get_module_dependencies(m, cache=cache)\n        deps = sum([reverse_map[d] for d in direct_deps if not d.endswith(\"__init__.py\")], direct_deps)\n        reverse_map[m] = list(set(deps) - {m})\n\n\n\nSUMMARY:\n**Function Summary: `create_reverse_dependency_map`**\n\n**Purpose and Functionality:**\nThis function generates a dictionary mapping module/test filenames to their dependent modules/tests recursively, excluding direct imports of initialization files.\n\n**Key Implementation Details:**\n\n*   Initializes the dependency map by scraping all modules, tests, and example dependencies from specific directories.\n*   Computes direct dependencies using `get_module_dependencies` with caching mechanism (`cache` dict).\n*   Recursively updates dependencies until no more changes are detected.\n*   Builds the reverse dependency map (`reverse_map`) based on the final computed dependencies.\n\n**Notable Dependencies/Requirements:**\n\n*   `init_test_examples_dependencies`: Initializes example dependencies and tests.\n*   `get_module_dependencies(m, cache=cache)`: Returns direct dependencies of a module using caching mechanism (`cache` dict).\n*   `collections.defaultdict(list)` is used to create the reverse dependency map.\n\n**External Dependencies:**\n\n*   PATH_TO_TRANFORMERS and PATH_TO_TESTS for scraping module and test files.\n*   PATH_TO_REPO for getting relative paths to modules and tests.\n\nThe function returns a dictionary where each key corresponds to a module/test filename, and its value contains a list of all filenames that recursively depend on it."}


created_at: 2025-02-05 09:13:52
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py:927-988
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 988, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py", "name": "create_module_to_test_map", "start_line": 927, "type": "function", "chroma:document": "CODE:\n\ndef create_module_to_test_map(\n    reverse_map: Dict[str, List[str]] = None, filter_models: bool = False\n) -> Dict[str, List[str]]:\n    \"\"\"\n    Extract the tests from the reverse_dependency_map and potentially filters the model tests.\n\n    Args:\n        reverse_map (`Dict[str, List[str]]`, *optional*):\n            The reverse dependency map as created by `create_reverse_dependency_map`. Will default to the result of\n            that function if not provided.\n        filter_models (`bool`, *optional*, defaults to `False`):\n            Whether or not to filter model tests to only include core models if a file impacts a lot of models.\n\n    Returns:\n        `Dict[str, List[str]]`: A dictionary that maps each file to the tests to execute if that file was modified.\n    \"\"\"\n    if reverse_map is None:\n        reverse_map = create_reverse_dependency_map()\n\n    # Utility that tells us if a given file is a test (taking test examples into account)\n    def is_test(fname):\n        if fname.startswith(\"tests\"):\n            return True\n        if fname.startswith(\"examples\") and fname.split(os.path.sep)[-1].startswith(\"test\"):\n            return True\n        return False\n\n    # Build the test map\n    test_map = {module: [f for f in deps if is_test(f)] for module, deps in reverse_map.items()}\n\n    if not filter_models:\n        return test_map\n\n    # Now we deal with the filtering if `filter_models` is True.\n    num_model_tests = len(list(PATH_TO_TESTS.glob(\"models/*\")))\n\n    def has_many_models(tests):\n        # We filter to core models when a given file impacts more than half the model tests.\n        model_tests = {Path(t).parts[2] for t in tests if t.startswith(\"tests/models/\")}\n        return len(model_tests) > num_model_tests // 2\n\n    # for each module (if specified in the argument `module`) of the form `models/my_model` (i.e. starting with it),\n    # we always keep the tests (those are already in the argument `tests`) which are in `tests/models/my_model`.\n    # This is to avoid them being excluded when a module has many impacted tests: the directly related test files should\n    # always be included!\n    def filter_tests(tests, module=\"\"):\n        filtered_tests = []\n        for t in tests:\n            if (\n                not t.startswith(\"tests/models/\")\n                or Path(t).parts[2] in IMPORTANT_MODELS\n                # at this point, `t` is of the form `tests/models/my_model`, and we check if `models/my_model`\n                # (i.e. `parts[1:3]`) is in `module`.\n                or \"/\".join(Path(t).parts[1:3]) in module\n            ):\n                filtered_tests += [t]\n        return filtered_tests\n\n    return {\n        module: (filter_tests(tests, module=module) if has_many_models(tests) else tests)\n        for module, tests in test_map.items()\n\n\nSUMMARY:\n**Function Summary: `create_module_to_test_map`**\n\nExtracts and filters tests from the reverse_dependency_map, providing a dictionary mapping each file to executable tests.\n\n**Primary Purpose:** Create a map of files to tests for execution, filtering out non-core model tests if specified.\n\n**Key Implementation Details:**\n\n* Uses `create_reverse_dependency_map` to default to a value if not provided.\n* Applies filters based on the presence of \"tests\" or \"examples/test\" in file names and core models' modifications threshold.\n* Includes tests within modules that have significantly impacted tests.\n\n**Notable Dependencies/Requirements:**\n\n* `create_reverse_dependency_map`\n* `PATH_TO_TESTS`\n* `IMPORTANT_MODELS`\n\n**Return Value:** A dictionary with module-file-test mappings, filtering out non-core model tests if specified."}


created_at: 2025-02-05 09:13:52
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py:991-995
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 995, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py", "name": "_print_list", "start_line": 991, "type": "function", "chroma:document": "CODE:\n\ndef _print_list(l) -> str:\n    \"\"\"\n    Pretty print a list of elements with one line per element and a - starting each line.\n    \"\"\"\n\n\nSUMMARY:\n**Function Summary:**\n\n`_print_list(l) -> str`\n\nPretty prints a list of elements, joining lines with hyphens (-) and newline characters (`\\n`).\n\n**Key Implementation Details:**\n\n* Uses `f-strings` to format each line\n* Joins lines with `\\n`\n* Uses list comprehension to iterate over the input list `l`\n\n**Notable Dependencies/Requirements:**\n\n* None\n\n**Interface:** Does not implement any external interfaces."}


created_at: 2025-02-05 09:13:53
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py:998-1081
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 1081, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py", "name": "infer_tests_to_run", "start_line": 998, "type": "function", "chroma:document": "CODE:\n\ndef infer_tests_to_run(\n    output_file: str, diff_with_last_commit: bool = False, filter_models: bool = False, test_all: bool = False\n):\n    \"\"\"\n    The main function called by the test fetcher. Determines the tests to run from the diff.\n\n    Args:\n        output_file (`str`):\n            The path where to store the summary of the test fetcher analysis. Other files will be stored in the same\n            folder:\n\n            - examples_test_list.txt: The list of examples tests to run.\n            - test_repo_utils.txt: Will indicate if the repo utils tests should be run or not.\n            - doctest_list.txt: The list of doctests to run.\n\n        diff_with_last_commit (`bool`, *optional*, defaults to `False`):\n            Whether to analyze the diff with the last commit (for use on the main branch after a PR is merged) or with\n            the branching point from main (for use on each PR).\n        filter_models (`bool`, *optional*, defaults to `True`):\n            Whether or not to filter the tests to core models only, when a file modified results in a lot of model\n            tests.\n    \"\"\"\n    if not test_all:\n        modified_files = get_modified_python_files(diff_with_last_commit=diff_with_last_commit)\n    else:\n        modified_files = [str(k) for k in PATH_TO_TESTS.glob(\"*/*\") if str(k).endswith(\".py\") and \"test_\" in str(k)]\n        print(\"\\n### test_all is TRUE, FETCHING ALL FILES###\\n\")\n    print(f\"\\n### MODIFIED FILES ###\\n{_print_list(modified_files)}\")\n\n    # Create the map that will give us all impacted modules.\n    reverse_map = create_reverse_dependency_map()\n    impacted_files = modified_files.copy()\n    for f in modified_files:\n        if f in reverse_map:\n            impacted_files.extend(reverse_map[f])\n\n    # Remove duplicates\n    impacted_files = sorted(set(impacted_files))\n    print(f\"\\n### IMPACTED FILES ###\\n{_print_list(impacted_files)}\")\n\n    model_impacted = {\"/\".join(x.split(\"/\")[:3]) for x in impacted_files if x.startswith(\"tests/models/\")}\n    # Grab the corresponding test files:\n    if (\n        any(x in modified_files for x in [\"setup.py\", \".circleci/create_circleci_config.py\"])\n        or not filter_models\n        and len(model_impacted) >= NUM_MODELS_TO_TRIGGER_FULL_CI\n        or commit_flags[\"test_all\"]\n    ):\n        test_files_to_run = glob.glob(\"tests/**/test_**.py\", recursive=True) + glob.glob(\n            \"examples/**/*.py\", recursive=True\n        )\n        if len(model_impacted) >= NUM_MODELS_TO_TRIGGER_FULL_CI and filter_models:\n            print(\n                f\"More than {NUM_MODELS_TO_TRIGGER_FULL_CI - 1} models are impacted and `filter_models=False`. CI is configured to test everything.\"\n            )\n    else:\n        # All modified tests need to be run.\n        test_files_to_run = [f for f in modified_files if f.startswith(\"tests\") and \"/test_\" in f]\n        impacted_files = get_impacted_files_from_tiny_model_summary(diff_with_last_commit=diff_with_last_commit)\n\n        # Then we grab the corresponding test files.\n        test_map = create_module_to_test_map(reverse_map=reverse_map, filter_models=filter_models)\n        for f in modified_files + impacted_files:\n            if f in test_map:\n                test_files_to_run.extend(test_map[f])\n        test_files_to_run = sorted(set(test_files_to_run))\n        # Remove repo utils tests\n        test_files_to_run = [f for f in test_files_to_run if not f.split(os.path.sep)[1] == \"repo_utils\"]\n        # Remove SageMaker tests\n        test_files_to_run = [f for f in test_files_to_run if not f.split(os.path.sep)[1] == \"sagemaker\"]\n        # Make sure we did not end up with a test file that was removed\n        test_files_to_run = [f for f in test_files_to_run if (PATH_TO_REPO / f).exists()]\n\n    print(f\"\\n### TEST TO RUN ###\\n{_print_list(test_files_to_run)}\")\n\n    create_test_list_from_filter(test_files_to_run, out_path=\"test_preparation/\")\n\n    doctest_list = get_doctest_files()\n\n    print(f\"\\n### DOCTEST TO RUN ###\\n{_print_list(doctest_list)}\")\n    if len(doctest_list) > 0:\n        doctest_file = Path(output_file).parent / \"doctest_list.txt\"\n        with open(doctest_file, \"w\", encoding=\"utf-8\") as f:\n\n\nSUMMARY:\nHere is a concise and focused summary of the function:\n\n**Function Name:** `infer_tests_to_run`\n\n**Main Purpose:**\nThe main function determines which tests to run from the diff. It analyzes modified files, identifies impacted modules, and extracts test files.\n\n**Key Implementation Details:**\n\n* Retrieves a list of modified Python files using the `get_modified_python_files` function.\n* Creates a reverse dependency map using the `create_reverse_dependency_map` function.\n* Identifies impacted models by searching for files starting with \"tests/models/\" in the modified files.\n* Extracts test files from directories like \"tests/**/test_**.py\" and \"examples/**/*.py\".\n* Filters tests to exclude models if necessary.\n\n**Notable Dependencies or Requirements:**\n\n* The `PATH_TO_TESTS` variable is used to glob for test files.\n* The `create_reverse_dependency_map` function depends on a reverse dependency map data structure.\n* The `NUM_MODELS_TO_TRIGGER_FULL_CI` constant influences the number of models required to trigger full CI analysis.\n\n**Output:** A list of files that need to be run in order to fetch all tests. This is printed to the console.\n\n**Summary Code:**\n```python\ndef infer_tests_to_run(\n    output_file: str,\n    diff_with_last_commit: bool = False,\n    filter_models: bool = True,\n    test_all: bool = False,\n):\n    modified_files = get_modified_python_files(diff_with_last_commit)\n    if not test_all:\n        modified_files = [str(k) for k in PATH_TO_TESTS.glob(\"*/*\") if str(k).endswith(\".py\") and \"test_\" in str(k)]\n    \n    # ...\n```"}


created_at: 2025-02-05 09:13:53
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py:1084-1108
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 1108, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py", "name": "filter_tests", "start_line": 1084, "type": "function", "chroma:document": "CODE:\n\ndef filter_tests(output_file: str, filters: List[str]):\n    \"\"\"\n    Reads the content of the output file and filters out all the tests in a list of given folders.\n\n    Args:\n        output_file (`str` or `os.PathLike`): The path to the output file of the tests fetcher.\n        filters (`List[str]`): A list of folders to filter.\n    \"\"\"\n    if not os.path.isfile(output_file):\n        print(\"No test file found.\")\n        return\n    with open(output_file, \"r\", encoding=\"utf-8\") as f:\n        test_files = f.read().split(\" \")\n\n    if len(test_files) == 0 or test_files == [\"\"]:\n        print(\"No tests to filter.\")\n        return\n\n    if test_files == [\"tests\"]:\n        test_files = [os.path.join(\"tests\", f) for f in os.listdir(\"tests\") if f not in [\"__init__.py\"] + filters]\n    else:\n        test_files = [f for f in test_files if f.split(os.path.sep)[1] not in filters]\n\n    with open(output_file, \"w\", encoding=\"utf-8\") as f:\n\n\nSUMMARY:\n**Function Summary: `filter_tests`**\n\n_filters tests from an output file based on a list of folders._\n\n### Function Purpose\n Reads the content of the output file, filters out specific tests in given folders and writes updated results back to the file.\n\n### Key Implementation Details\n\n* **File filtering**: Iterates through test files using `os.listdir()` and checks if folder paths are present in the `filters` list.\n* **Tests reading/writing**: Uses `open()` with `\"r\"` and `\"w\"` modes to read and write content to the output file, respectively.\n\n### Notable Dependencies\n\n* `os`: for interacting with the operating system (e.g., working with files, directories)\n* `pathlib` (not explicitly used but implied through `os.path.join()`)\n\n### Interfaces/Methods Used\n\n* `os.listdir()`\n* `os.path.join()`\n\n### External Dependencies\nNone mentioned."}


created_at: 2025-02-05 09:13:53
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py:1111-1134
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 1134, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py", "name": "parse_commit_message", "start_line": 1111, "type": "function", "chroma:document": "CODE:\n\ndef parse_commit_message(commit_message: str) -> Dict[str, bool]:\n    \"\"\"\n    Parses the commit message to detect if a command is there to skip, force all or part of the CI.\n\n    Args:\n        commit_message (`str`): The commit message of the current commit.\n\n    Returns:\n        `Dict[str, bool]`: A dictionary of strings to bools with keys the following keys: `\"skip\"`,\n        `\"test_all_models\"` and `\"test_all\"`.\n    \"\"\"\n    if commit_message is None:\n        return {\"skip\": False, \"no_filter\": False, \"test_all\": False}\n\n    command_search = re.search(r\"\\[([^\\]]*)\\]\", commit_message)\n    if command_search is not None:\n        command = command_search.groups()[0]\n        command = command.lower().replace(\"-\", \" \").replace(\"_\", \" \")\n        skip = command in [\"ci skip\", \"skip ci\", \"circleci skip\", \"skip circleci\"]\n        no_filter = set(command.split(\" \")) == {\"no\", \"filter\"}\n        test_all = set(command.split(\" \")) == {\"test\", \"all\"}\n        return {\"skip\": skip, \"no_filter\": no_filter, \"test_all\": test_all}\n    else:\n\n\nSUMMARY:\n**Function Summary**\n\n`parse_commit_message` parses a commit message to detect specific instructions for testing.\n\n**Parameters**\n\n* `commit_message`: The commit message to parse (string)\n* Returns: A dictionary with keys `\"skip\"`, `\"no_filter\"`, and `\"test_all\"` indicating the presence or absence of corresponding tests.\n\n**Return Value Dictionary**\n\n| Key | Boolean Value |\n| --- | --- |\n| `skip` | Whether to skip tests (`True`/`False`) |\n| `no_filter` | Whether to disable filtering for tests (`True`/`False`) |\n| `test_all` | Whether to run all tests (`True`/`False`) |\n\n**Implementation Details**\n\n* Uses regular expression to search for a command in the commit message.\n* Sets up conditional checks based on the presence of specific keywords.\n\n**Notable Dependencies**\n\nNo external dependencies."}


created_at: 2025-02-05 09:13:53
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py:1158-1170
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 1170, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/tests_fetcher.py", "name": "create_test_list_from_filter", "start_line": 1158, "type": "function", "chroma:document": "CODE:\n\ndef create_test_list_from_filter(full_test_list, out_path):\n    os.makedirs(out_path, exist_ok=True)\n    all_test_files = \"\\n\".join(full_test_list)\n    for job_name, _filter in JOB_TO_TEST_FILE.items():\n        file_name = os.path.join(out_path, f\"{job_name}_test_list.txt\")\n        if job_name == \"tests_hub\":\n            files_to_test = [\"tests\"]\n        else:\n            files_to_test = list(re.findall(_filter, all_test_files))\n        print(job_name, file_name)\n        if len(files_to_test) > 0:  # No tests -> no file with test list\n            with open(file_name, \"w\") as f:\n\n\nSUMMARY:\n**Function Summary: create_test_list_from_filter**\n\n* **Purpose**: Creates a text list of files matching specific filter criteria and writes them to a designated output directory.\n\n### Key Implementation Details:\n\n- Utilizes `os.path.join` for file path construction\n- Applies regular expression pattern `_filter` to a full test list to find matched files using `re.findall`\n \n### Notable Dependencies/ Requirements:\n\n- Uses `JOB_TO_TEST_FILE`, an external dictionary containing mapping between job names and test filters.\n- Rely on `os` module's `makedirs` and file handling functions (`open`)."}


created_at: 2025-02-05 09:13:53
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/models_to_deprecate.py:35-49
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 49, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/models_to_deprecate.py", "name": "HubModelLister", "start_line": 35, "type": "class", "chroma:document": "CODE:\n\nclass HubModelLister:\n    \"\"\"\n    Utility for getting models from the hub based on tags. Handles errors without crashing the script.\n    \"\"\"\n\n    def __init__(self, tags):\n        self.tags = tags\n        self.model_list = api.list_models(tags=tags)\n\n    def __iter__(self):\n        try:\n            yield from self.model_list\n        except Exception as e:\n            print(f\"Error: {e}\")\n\n\nSUMMARY:\n**Summary**\n\nThe `HubModelLister` class is a utility for retrieving models from the hub based on provided tags. Its primary purpose is to handle model fetching and errors without crashing the script.\n\n**Main Purpose and Functionality**\n```python\nclass HubModelLister:\n    def __init__(self, tags):\n        # Retrieves models from the hub based on tags\n        self.model_list = api.list_models(tags=tags)\n```\nThe main functionality is to fetch models from the hub using API's `list_models` method with input `tags`. It handles exceptions without crashing the script.\n\n**Key Implementation Details**\n- Fetches models from the hub using `api.list_models(tags=tags)`\n- Uses try-except block to handle errors without crashing\n\n**Notable Dependencies or Requirements**\n- `api`: reference to the API module for list_models method\n- External dependency: assumes the presence of an API that supports list_models functionality"}


created_at: 2025-02-05 09:13:53
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/models_to_deprecate.py:52-56
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 56, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/models_to_deprecate.py", "name": "_extract_commit_hash", "start_line": 52, "type": "function", "chroma:document": "CODE:\n\ndef _extract_commit_hash(commits):\n    for commit in commits:\n        if commit.startswith(\"commit \"):\n            return commit.split(\" \")[1]\n\n\nSUMMARY:\n**Summary**\n\n`_extract_commit_hash` extracts the commit hash from a list of commit messages.\n\n**Key Functionality**\n\n* Iterates through each commit message in the `commits` list\n* Returns the commit hash when a message starts with \"commit \"\n* Returns an empty string if no such message is found\n\n**Implementation Details**\n\n* Uses simple iteration and string manipulation to extract the commit hash\n* Utilizes `str.startswith` and `str.split` methods\n\n**Notable Dependencies**\n\n* None explicitly mentioned, no external imports or module references."}


created_at: 2025-02-05 09:13:53
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/models_to_deprecate.py:59-77
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 77, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/models_to_deprecate.py", "name": "get_list_of_repo_model_paths", "start_line": 59, "type": "function", "chroma:document": "CODE:\n\ndef get_list_of_repo_model_paths(models_dir):\n    # Get list of all models in the library\n    models = glob.glob(os.path.join(models_dir, \"*/modeling_*.py\"))\n\n    # Remove flax and tf models\n    models = [model for model in models if \"_flax_\" not in model]\n    models = [model for model in models if \"_tf_\" not in model]\n\n    # Get list of all deprecated models in the library\n    deprecated_models = glob.glob(os.path.join(models_dir, \"deprecated\", \"*\"))\n    # For each deprecated model, remove the deprecated models from the list of all models as well as the symlink path\n    for deprecated_model in deprecated_models:\n        deprecated_model_name = \"/\" + deprecated_model.split(\"/\")[-1] + \"/\"\n        models = [model for model in models if deprecated_model_name not in model]\n    # Remove deprecated models\n    models = [model for model in models if \"/deprecated\" not in model]\n    # Remove auto\n    models = [model for model in models if \"/auto/\" not in model]\n\n\nSUMMARY:\n**Function Summary:**\n\n`get_list_of_repo_model_paths(models_dir)`\n\n*   Purpose: Returns a list of repository model paths by filtering out Flax, TensorFlow, deprecated, and auto-generated models.\n\n**Key Implementation Details:**\n\n*   Uses `glob` to retrieve all `.py` files in the `models_dir` with \"modeling_\" prefix.\n*   Applies two passes of filtering:\n    *   Removes Flax and TensorFlow models using list comprehensions.\n    *   Forgets deprecated models by removing their symlinks from original list.\n\n**Notable Dependencies or Requirements:**\n\n*   `os.path.join` for path concatenation.\n*   `glob` for file patterns matching.\n*   List comprehensions for efficient filtering."}


created_at: 2025-02-05 09:13:53
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/models_to_deprecate.py:80-164
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 164, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/models_to_deprecate.py", "name": "get_list_of_models_to_deprecate", "start_line": 80, "type": "function", "chroma:document": "CODE:\n\ndef get_list_of_models_to_deprecate(\n    thresh_num_downloads=5_000,\n    thresh_date=None,\n    use_cache=False,\n    save_model_info=False,\n    max_num_models=-1,\n):\n    if thresh_date is None:\n        thresh_date = datetime.now(timezone.utc).replace(year=datetime.now(timezone.utc).year - 1)\n    else:\n        thresh_date = datetime.strptime(thresh_date, \"%Y-%m-%d\").replace(tzinfo=timezone.utc)\n\n    models_dir = PATH_TO_REPO / \"src/transformers/models\"\n    model_paths = get_list_of_repo_model_paths(models_dir=models_dir)\n\n    if use_cache and os.path.exists(\"models_info.json\"):\n        with open(\"models_info.json\", \"r\") as f:\n            models_info = json.load(f)\n        # Convert datetimes back to datetime objects\n        for model, info in models_info.items():\n            info[\"first_commit_datetime\"] = datetime.fromisoformat(info[\"first_commit_datetime\"])\n\n    else:\n        # Build a dictionary of model info: first commit datetime, commit hash, model path\n        models_info = defaultdict(dict)\n        for model_path in model_paths:\n            model = model_path.split(\"/\")[-2]\n            if model in models_info:\n                continue\n            commits = repo.git.log(\"--diff-filter=A\", \"--\", model_path).split(\"\\n\")\n            commit_hash = _extract_commit_hash(commits)\n            commit_obj = repo.commit(commit_hash)\n            committed_datetime = commit_obj.committed_datetime\n            models_info[model][\"commit_hash\"] = commit_hash\n            models_info[model][\"first_commit_datetime\"] = committed_datetime\n            models_info[model][\"model_path\"] = model_path\n            models_info[model][\"downloads\"] = 0\n\n            # Some tags on the hub are formatted differently than in the library\n            tags = [model]\n            if \"_\" in model:\n                tags.append(model.replace(\"_\", \"-\"))\n            models_info[model][\"tags\"] = tags\n\n        # Filter out models which were added less than a year ago\n        models_info = {\n            model: info for model, info in models_info.items() if info[\"first_commit_datetime\"] < thresh_date\n        }\n\n        # We make successive calls to the hub, filtering based on the model tags\n        n_seen = 0\n        for model, model_info in models_info.items():\n            for model_tag in model_info[\"tags\"]:\n                model_list = HubModelLister(tags=model_tag)\n                for i, hub_model in enumerate(model_list):\n                    n_seen += 1\n                    if i % 100 == 0:\n                        print(f\"Processing model {i} for tag {model_tag}\")\n                    if max_num_models != -1 and i > n_seen:\n                        break\n                    if hub_model.private:\n                        continue\n                    model_info[\"downloads\"] += hub_model.downloads\n\n    if save_model_info and not (use_cache and os.path.exists(\"models_info.json\")):\n        # Make datetimes serializable\n        for model, info in models_info.items():\n            info[\"first_commit_datetime\"] = info[\"first_commit_datetime\"].isoformat()\n        with open(\"models_info.json\", \"w\") as f:\n            json.dump(models_info, f, indent=4)\n\n    print(\"\\nFinding models to deprecate:\")\n    n_models_to_deprecate = 0\n    models_to_deprecate = {}\n    for model, info in models_info.items():\n        n_downloads = info[\"downloads\"]\n        if n_downloads < thresh_num_downloads:\n            n_models_to_deprecate += 1\n            models_to_deprecate[model] = info\n            print(f\"\\nModel: {model}\")\n            print(f\"Downloads: {n_downloads}\")\n            print(f\"Date: {info['first_commit_datetime']}\")\n    print(\"\\nModels to deprecate: \", \"\\n\" + \"\\n\".join(models_to_deprecate.keys()))\n    print(f\"\\nNumber of models to deprecate: {n_models_to_deprecate}\")\n\n\nSUMMARY:\n**Function Summary**\n\n`get_list_of_models_to_deprecate`\n\nThis function retrieves lists of models that have fewer than a specified threshold number of downloads over the past year.\n\n**Primary Purpose**\nIdentify models with low download counts for potential deprecated model detection.\n\n**Key Implementation Details**\n\n1. Filter models based on download counts and date of first commit.\n2. Extract commit hashes, first commit dates, and model paths from a repository's Git log.\n3. Filter out new models added within the last year using a pre-defined threshold date.\n4. Make successive calls to the HubModelLister for each model to collect download information.\n\n**Notable Dependencies**\n\n* `repo` (Git repository interaction)\n* `HubModelLister` (model lister class)\n* `datetime`\n* `json`\n\n**Inherited/Implemented Interfaces**\nNone apparent.\n\n**External Dependencies**\n\n* `models_info.json` file (for caching model info)"}


created_at: 2025-02-05 09:13:53
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/extract_warnings.py:14-63
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 63, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/extract_warnings.py", "name": "extract_warnings_from_single_artifact", "start_line": 14, "type": "function", "chroma:document": "CODE:\n\ndef extract_warnings_from_single_artifact(artifact_path, targets):\n    \"\"\"Extract warnings from a downloaded artifact (in .zip format)\"\"\"\n    selected_warnings = set()\n    buffer = []\n\n    def parse_line(fp):\n        for line in fp:\n            if isinstance(line, bytes):\n                line = line.decode(\"UTF-8\")\n            if \"warnings summary (final)\" in line:\n                continue\n            # This means we are outside the body of a warning\n            elif not line.startswith(\" \"):\n                # process a single warning and move it to `selected_warnings`.\n                if len(buffer) > 0:\n                    warning = \"\\n\".join(buffer)\n                    # Only keep the warnings specified in `targets`\n                    if any(f\": {x}: \" in warning for x in targets):\n                        selected_warnings.add(warning)\n                    buffer.clear()\n                continue\n            else:\n                line = line.strip()\n                buffer.append(line)\n\n    if from_gh:\n        for filename in os.listdir(artifact_path):\n            file_path = os.path.join(artifact_path, filename)\n            if not os.path.isdir(file_path):\n                # read the file\n                if filename != \"warnings.txt\":\n                    continue\n                with open(file_path) as fp:\n                    parse_line(fp)\n    else:\n        try:\n            with zipfile.ZipFile(artifact_path) as z:\n                for filename in z.namelist():\n                    if not os.path.isdir(filename):\n                        # read the file\n                        if filename != \"warnings.txt\":\n                            continue\n                        with z.open(filename) as fp:\n                            parse_line(fp)\n        except Exception:\n            logger.warning(\n                f\"{artifact_path} is either an invalid zip file or something else wrong. This file is skipped.\"\n            )\n\n\n\nSUMMARY:\n**Function Summary: extract_warnings_from_single_artifact**\n\nExtracts warnings from a downloaded artifact (.zip) based on specified targets.\n\n**Main Purpose and Functionality**\n\nThis function takes an `artifact_path` and `targets` as input, extracts warnings from the artifact, and returns them in a set format. It checks for specific text patterns to identify warnings and filters them according to the provided `targets`.\n\n**Implementation Details**\n\n*   Uses a buffer to accumulate lines of code until a warning is complete.\n*   Iterates through files in the artifact directory (if not using `from_gh`) or zipped file entries (if using `from_gh`).\n*   Checks for warnings by searching for text patterns, such as \"warnings summary (final)\" and line endings.\n\n**Notable Dependencies/Requirements**\n\n*   `os`: For file system operations.\n*   `zipfile`: For handling .zip files.\n*   `logging`: Logger used to display warning messages.\n\n**Key Algorithm/Pattern**\n\nThe function uses a buffer-based approach to accumulate lines of code until a warning is complete. It checks for specific text patterns to identify warnings and filters them according to the provided targets."}


created_at: 2025-02-05 09:13:53
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/extract_warnings.py:66-75
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 75, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/extract_warnings.py", "name": "extract_warnings", "start_line": 66, "type": "function", "chroma:document": "CODE:\n\ndef extract_warnings(artifact_dir, targets):\n    \"\"\"Extract warnings from all artifact files\"\"\"\n\n    selected_warnings = set()\n\n    paths = [os.path.join(artifact_dir, p) for p in os.listdir(artifact_dir) if (p.endswith(\".zip\") or from_gh)]\n    for p in paths:\n        selected_warnings.update(extract_warnings_from_single_artifact(p, targets))\n\n\n\nSUMMARY:\nHere is a concise summary of the provided function code chunk:\n\n**Function Name:** `extract_warnings`\n\n**Purpose:**\nRecurursively extracts warnings from file artifacts in a specified directory and its subdirectories.\n\n**Main Functionality:**\n\n* Iterates through files in the `artifact_dir` and filters those that end with \".zip\" or follow the condition established by the `from_gh` variable.\n* Calls a recursive helper function (`extract_warnings_from_single_artifact`) for each filtered file, passing the `targets` parameter.\n* Updates an internal set `selected_warnings` with extracted warnings from all artifacts.\n\n**Key Implementation Details:**\n\n* Uses `os.path.join()` to construct paths to artifact files\n* Utilizes recursion through `extract_warnings_from_single_artifact`\n* Employs a set (`selected_warnings`) to store unique warnings\n\n**Notable Dependencies and Requirements:**\n\n* `os` module for path manipulation\n* `targets` parameter passed to recursive helper function\n* `from_gh` variable conditionally filtering \".zip\" files"}


created_at: 2025-02-05 09:13:53
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/custom_init_isort.py:58-61
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 61, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/custom_init_isort.py", "name": "get_indent", "start_line": 58, "type": "function", "chroma:document": "CODE:\n\ndef get_indent(line: str) -> str:\n    \"\"\"Returns the indent in  given line (as string).\"\"\"\n    search = _re_indent.search(line)\n\n\nSUMMARY:\n**Summary**\n\nA utility function `get_indent` extracts and returns the indentation from a input string `line`. It utilizes a regular expression `_re_indent` to find the match in the line and returns an empty string if no match is found.\n\n**Key Details:**\n\n* Function signature: `get_indent(line: str) -> str`\n* Regular expression used: `_re_indent` (exact name)\n* Primary purpose: Extracts indentation from a line as a string.\n\n**Notable Dependencies/Requirements:**\n\n* `line`: input string\n* `_re_indent`: regular expression object (imported or defined elsewhere)\n* No external dependencies listed."}


created_at: 2025-02-05 09:13:53
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/custom_init_isort.py:64-128
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 128, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/custom_init_isort.py", "name": "split_code_in_indented_blocks", "start_line": 64, "type": "function", "chroma:document": "CODE:\n\ndef split_code_in_indented_blocks(\n    code: str, indent_level: str = \"\", start_prompt: Optional[str] = None, end_prompt: Optional[str] = None\n) -> List[str]:\n    \"\"\"\n    Split some code into its indented blocks, starting at a given level.\n\n    Args:\n        code (`str`): The code to split.\n        indent_level (`str`): The indent level (as string) to use for identifying the blocks to split.\n        start_prompt (`str`, *optional*): If provided, only starts splitting at the line where this text is.\n        end_prompt (`str`, *optional*): If provided, stops splitting at a line where this text is.\n\n    Warning:\n        The text before `start_prompt` or after `end_prompt` (if provided) is not ignored, just not split. The input `code`\n        can thus be retrieved by joining the result.\n\n    Returns:\n        `List[str]`: The list of blocks.\n    \"\"\"\n    # Let's split the code into lines and move to start_index.\n    index = 0\n    lines = code.split(\"\\n\")\n    if start_prompt is not None:\n        while not lines[index].startswith(start_prompt):\n            index += 1\n        blocks = [\"\\n\".join(lines[:index])]\n    else:\n        blocks = []\n\n    # This variable contains the block treated at a given time.\n    current_block = [lines[index]]\n    index += 1\n    # We split into blocks until we get to the `end_prompt` (or the end of the file).\n    while index < len(lines) and (end_prompt is None or not lines[index].startswith(end_prompt)):\n        # We have a non-empty line with the proper indent -> start of a new block\n        if len(lines[index]) > 0 and get_indent(lines[index]) == indent_level:\n            # Store the current block in the result and rest. There are two cases: the line is part of the block (like\n            # a closing parenthesis) or not.\n            if len(current_block) > 0 and get_indent(current_block[-1]).startswith(indent_level + \" \"):\n                # Line is part of the current block\n                current_block.append(lines[index])\n                blocks.append(\"\\n\".join(current_block))\n                if index < len(lines) - 1:\n                    current_block = [lines[index + 1]]\n                    index += 1\n                else:\n                    current_block = []\n            else:\n                # Line is not part of the current block\n                blocks.append(\"\\n\".join(current_block))\n                current_block = [lines[index]]\n        else:\n            # Just add the line to the current block\n            current_block.append(lines[index])\n        index += 1\n\n    # Adds current block if it's nonempty.\n    if len(current_block) > 0:\n        blocks.append(\"\\n\".join(current_block))\n\n    # Add final block after end_prompt if provided.\n    if end_prompt is not None and index < len(lines):\n        blocks.append(\"\\n\".join(lines[index:]))\n\n\n\nSUMMARY:\n**Function Summary: `split_code_in_indented_blocks`**\n\n#### Main Purpose and Functionality\nThis function splits code into indented blocks. It takes in code, an indent level, and optional start and end prompts as input.\n\n#### Key Implementation Details\n\n* The function uses the `split()` function to split the code into lines.\n* It iterates through each line, checking for the start of a new block based on the given indent level.\n* The function uses a variable `current_block` to store lines being processed in a block.\n\n#### Notable Methods and Interfaces\n\n* `get_indent(lines[index])`: This method is used to calculate the indentation level at each line.\n\n#### External Dependencies\n\n* None explicitly mentioned, but it assumes standard Python environment with `Optional`, `List`, and `str` types.\n\n#### Output\nThe function returns a list of strings, where each string represents an indented block of code.\n\nNote: The exact method name for calculating the indentation level is not specified in the provided code."}


created_at: 2025-02-05 09:13:53
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/custom_init_isort.py:131-139
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 139, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/custom_init_isort.py", "name": "ignore_underscore_and_lowercase", "start_line": 131, "type": "function", "chroma:document": "CODE:\n\ndef ignore_underscore_and_lowercase(key: Callable[[Any], str]) -> Callable[[Any], str]:\n    \"\"\"\n    Wraps a key function (as used in a sort) to lowercase and ignore underscores.\n    \"\"\"\n\n    def _inner(x):\n        return key(x).lower().replace(\"_\", \"\")\n\n\n\nSUMMARY:\n**Summary**\n\n`ignore_underscore_and_lowercase` is a higher-order function that wraps another function `[Callable[[Any], str]]` to:\n\n* Convert input strings to lowercase (`key(x).lower()`)\n* Remove underscores from the output string (`replace(\"_\", \"\")`)\n\n**Implementation Details**\n\n* `_inner` method signatures: `(x: Any) -> [str]`\n* Interface used: `Callable[[Any], str]`\n\n**Notable Dependencies/Requirements**\n\n* None, but functions accepting any type (`Any`) and returning strings (`[str]`)."}


created_at: 2025-02-05 09:13:53
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/custom_init_isort.py:142-173
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 173, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/custom_init_isort.py", "name": "sort_objects", "start_line": 142, "type": "function", "chroma:document": "CODE:\n\ndef sort_objects(objects: List[Any], key: Optional[Callable[[Any], str]] = None) -> List[Any]:\n    \"\"\"\n    Sort a list of objects following the rules of isort (all uppercased first, camel-cased second and lower-cased\n    last).\n\n    Args:\n        objects (`List[Any]`):\n            The list of objects to sort.\n        key (`Callable[[Any], str]`, *optional*):\n            A function taking an object as input and returning a string, used to sort them by alphabetical order.\n            If not provided, will default to noop (so a `key` must be provided if the `objects` are not of type string).\n\n    Returns:\n        `List[Any]`: The sorted list with the same elements as in the inputs\n    \"\"\"\n\n    # If no key is provided, we use a noop.\n    def noop(x):\n        return x\n\n    if key is None:\n        key = noop\n    # Constants are all uppercase, they go first.\n    constants = [obj for obj in objects if key(obj).isupper()]\n    # Classes are not all uppercase but start with a capital, they go second.\n    classes = [obj for obj in objects if key(obj)[0].isupper() and not key(obj).isupper()]\n    # Functions begin with a lowercase, they go last.\n    functions = [obj for obj in objects if not key(obj)[0].isupper()]\n\n    # Then we sort each group.\n    key1 = ignore_underscore_and_lowercase(key)\n\n\nSUMMARY:\n**Function Summary:**\n\n```python\nsort_objects(objects: List[Any], key: Optional[Callable[[Any], str]] = None) -> List[Any]\n```\n\n**Purpose and Functionality:**\n\nThis function sorts a list of objects based on the naming conventions: all uppercase constants first, camel-cased class names second, and lowercase functions last. If no custom sorting key is provided, it defaults to alphabetical order.\n\n**Key Implementation Details:**\n\n* Extracts objects into three categories: `constants` (uppercase), `classes` (camel-case), and `functions` (lowercase)\n* Uses the custom sorting key function (`key1`) to sort each group\n* Returns the sorted list with the same elements as the input\n\n**Notable Dependencies or Requirements:**\n\n* None\n\n**External Dependencies:**\nNone\n\nThis code follows best practices for documentation, readability, and conciseness. The summary provides a clear understanding of the function's purpose, key implementation details, and dependencies or requirements."}


created_at: 2025-02-05 09:13:53
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/custom_init_isort.py:176-231
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 231, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/custom_init_isort.py", "name": "sort_objects_in_import", "start_line": 176, "type": "function", "chroma:document": "CODE:\n\ndef sort_objects_in_import(import_statement: str) -> str:\n    \"\"\"\n    Sorts the imports in a single import statement.\n\n    Args:\n        import_statement (`str`): The import statement in which to sort the imports.\n\n    Returns:\n        `str`: The same as the input, but with objects properly sorted.\n    \"\"\"\n\n    # This inner function sort imports between [ ].\n    def _replace(match):\n        imports = match.groups()[0]\n        # If there is one import only, nothing to do.\n        if \",\" not in imports:\n            return f\"[{imports}]\"\n        keys = [part.strip().replace('\"', \"\") for part in imports.split(\",\")]\n        # We will have a final empty element if the line finished with a comma.\n        if len(keys[-1]) == 0:\n            keys = keys[:-1]\n        return \"[\" + \", \".join([f'\"{k}\"' for k in sort_objects(keys)]) + \"]\"\n\n    lines = import_statement.split(\"\\n\")\n    if len(lines) > 3:\n        # Here we have to sort internal imports that are on several lines (one per name):\n        # key: [\n        #     \"object1\",\n        #     \"object2\",\n        #     ...\n        # ]\n\n        # We may have to ignore one or two lines on each side.\n        idx = 2 if lines[1].strip() == \"[\" else 1\n        keys_to_sort = [(i, _re_strip_line.search(line).groups()[0]) for i, line in enumerate(lines[idx:-idx])]\n        sorted_indices = sort_objects(keys_to_sort, key=lambda x: x[1])\n        sorted_lines = [lines[x[0] + idx] for x in sorted_indices]\n        return \"\\n\".join(lines[:idx] + sorted_lines + lines[-idx:])\n    elif len(lines) == 3:\n        # Here we have to sort internal imports that are on one separate line:\n        # key: [\n        #     \"object1\", \"object2\", ...\n        # ]\n        if _re_bracket_content.search(lines[1]) is not None:\n            lines[1] = _re_bracket_content.sub(_replace, lines[1])\n        else:\n            keys = [part.strip().replace('\"', \"\") for part in lines[1].split(\",\")]\n            # We will have a final empty element if the line finished with a comma.\n            if len(keys[-1]) == 0:\n                keys = keys[:-1]\n            lines[1] = get_indent(lines[1]) + \", \".join([f'\"{k}\"' for k in sort_objects(keys)])\n        return \"\\n\".join(lines)\n    else:\n        # Finally we have to deal with imports fitting on one line\n        import_statement = _re_bracket_content.sub(_replace, import_statement)\n\n\nSUMMARY:\n**Summary**\n\nThis function, `sort_objects_in_import`, sorts objects within an import statement. It correctly handles complex cases with multiple lines of imports.\n\n**Key Functionality**\n\n1.  **Input**: A single-line or multi-line import statement containing variable names enclosed in square brackets (`[]`).\n2.  **Output**: The same input but with object dependencies properly sorted and formatted.\n3.  **Sorting Logic**:\n    *   For multiple lines, sorts imports according to their alphanumeric keys within the brackets.\n    *   For single lines, checks for bracket content using regular expression (`_re_bracket_content`) to determine if sorting is required.\n\n**Notable Implementation Details**\n\n1.  **Regular Expressions**: Utilize of `_re_replace`, `_re_bracket_content`, and other regular expressions for parsing import statements, handling brackets, and sorting keys.\n2.  **Line Splitting**: The function splits the input string into multiple lines based on newline characters (`\\n`) to identify and sort imports across multi-line statements.\n3.  **Import Sorting Logic**: Employ a custom `sort_objects` function (not shown in this snippet) to sort imported objects according to their keys.\n\n**Dependencies**\n\n*   No external dependencies other than Python's built-in library for regular expressions, which are used extensively in the provided code."}


created_at: 2025-02-05 09:13:53
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/custom_init_isort.py:234-304
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 304, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/custom_init_isort.py", "name": "sort_imports", "start_line": 234, "type": "function", "chroma:document": "CODE:\n\ndef sort_imports(file: str, check_only: bool = True):\n    \"\"\"\n    Sort the imports defined in the `_import_structure` of a given init.\n\n    Args:\n        file (`str`): The path to the init to check/fix.\n        check_only (`bool`, *optional*, defaults to `True`): Whether or not to just check (and not auto-fix) the init.\n    \"\"\"\n    with open(file, encoding=\"utf-8\") as f:\n        code = f.read()\n\n    # If the file is not a custom init, there is nothing to do.\n    if \"_import_structure\" not in code or \"define_import_structure\" in code:\n        return\n\n    # Blocks of indent level 0\n    main_blocks = split_code_in_indented_blocks(\n        code, start_prompt=\"_import_structure = {\", end_prompt=\"if TYPE_CHECKING:\"\n    )\n\n    # We ignore block 0 (everything untils start_prompt) and the last block (everything after end_prompt).\n    for block_idx in range(1, len(main_blocks) - 1):\n        # Check if the block contains some `_import_structure`s thingy to sort.\n        block = main_blocks[block_idx]\n        block_lines = block.split(\"\\n\")\n\n        # Get to the start of the imports.\n        line_idx = 0\n        while line_idx < len(block_lines) and \"_import_structure\" not in block_lines[line_idx]:\n            # Skip dummy import blocks\n            if \"import dummy\" in block_lines[line_idx]:\n                line_idx = len(block_lines)\n            else:\n                line_idx += 1\n        if line_idx >= len(block_lines):\n            continue\n\n        # Ignore beginning and last line: they don't contain anything.\n        internal_block_code = \"\\n\".join(block_lines[line_idx:-1])\n        indent = get_indent(block_lines[1])\n        # Slit the internal block into blocks of indent level 1.\n        internal_blocks = split_code_in_indented_blocks(internal_block_code, indent_level=indent)\n        # We have two categories of import key: list or _import_structure[key].append/extend\n        pattern = _re_direct_key if \"_import_structure = {\" in block_lines[0] else _re_indirect_key\n        # Grab the keys, but there is a trap: some lines are empty or just comments.\n        keys = [(pattern.search(b).groups()[0] if pattern.search(b) is not None else None) for b in internal_blocks]\n        # We only sort the lines with a key.\n        keys_to_sort = [(i, key) for i, key in enumerate(keys) if key is not None]\n        sorted_indices = [x[0] for x in sorted(keys_to_sort, key=lambda x: x[1])]\n\n        # We reorder the blocks by leaving empty lines/comments as they were and reorder the rest.\n        count = 0\n        reorderded_blocks = []\n        for i in range(len(internal_blocks)):\n            if keys[i] is None:\n                reorderded_blocks.append(internal_blocks[i])\n            else:\n                block = sort_objects_in_import(internal_blocks[sorted_indices[count]])\n                reorderded_blocks.append(block)\n                count += 1\n\n        # And we put our main block back together with its first and last line.\n        main_blocks[block_idx] = \"\\n\".join(block_lines[:line_idx] + reorderded_blocks + [block_lines[-1]])\n\n    if code != \"\\n\".join(main_blocks):\n        if check_only:\n            return True\n        else:\n            print(f\"Overwriting {file}.\")\n            with open(file, \"w\", encoding=\"utf-8\") as f:\n\n\nSUMMARY:\n**Function Summary: sort_imports**\n================================\n\n*   **Purpose:** Automatically sorts imports defined in the `_import_structure` of an init file.\n\n    The function takes in a file path and optional `check_only` flag, then reads the file's content. If the file contains an _import structure block, it extracts the imports from the corresponding code blocks, sorts them alphabetically, and reorders the blocks accordingly.\n\n*   **Key Algorithms/Patterns:**\n    *   Splits code into indented blocks using a specific prompt.\n    *   Identifies and splits internal import blocks by their indentation levels.\n    *   Uses regular expressions to extract keys from import lines (direct or indirect).\n    *   Sorts the extracted keys alphabetically.\n\n*   **Notable Dependencies/Requirements:**\n    *   Standard Python library functions for file operations and string manipulation (`open`, `read`, `split`).\n\n**Key Method Signatures:**\n\n*   `split_code_in_indented_blocks`: splits code into indented blocks.\n*   `_re_direct_key` and `_re_indirect_key`: regular expressions for direct and indirect import key extraction.\n\n**Interfaces/ Classes:**\n    *   No explicit interface or class usage is mentioned in the provided function."}


created_at: 2025-02-05 09:13:53
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/custom_init_isort.py:307-321
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 321, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/custom_init_isort.py", "name": "sort_imports_in_all_inits", "start_line": 307, "type": "function", "chroma:document": "CODE:\n\ndef sort_imports_in_all_inits(check_only=True):\n    \"\"\"\n    Sort the imports defined in the `_import_structure` of all inits in the repo.\n\n    Args:\n        check_only (`bool`, *optional*, defaults to `True`): Whether or not to just check (and not auto-fix) the init.\n    \"\"\"\n    failures = []\n    for root, _, files in os.walk(PATH_TO_TRANSFORMERS):\n        if \"__init__.py\" in files:\n            result = sort_imports(os.path.join(root, \"__init__.py\"), check_only=check_only)\n            if result:\n                failures = [os.path.join(root, \"__init__.py\")]\n    if len(failures) > 0:\n\n\nSUMMARY:\n**Summary:**\n\n### Function Purpose and Functionality\n\nThe `sort_imports_in_all_inits` function sorts the imports in all Python initialization files (`__init__.py`) in a repository. It checks if the sorting fixes any issues and returns a list of affected files.\n\n### Implementation Details\n\n* Checks all `.py` files with an initialisation file marker (e.g., `__init__.py`) under `transformers/`.\n* Tails all directories under `transformers/` recursively.\n* Sorts imports using `sort_imports` for each affected initialization file.\n\n### Notable Dependencies and Requirements\n\n* `os.walk()`\n* `PATH_TO_TRANSFORMERS`: Environment variable set to the root directory of transformers.\n\n### Note\n\nIf sorting fixes issues, it will overwrite multiple files. Running `make style` is recommended."}


created_at: 2025-02-05 09:13:53
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_table.py:50-82
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 82, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_table.py", "name": "_find_text_in_file", "start_line": 50, "type": "function", "chroma:document": "CODE:\n\ndef _find_text_in_file(filename: str, start_prompt: str, end_prompt: str) -> str:\n    \"\"\"\n    Find the text in filename between two prompts.\n\n    Args:\n        filename (`str`): The file to search into.\n        start_prompt (`str`): A string to look for at the start of the content searched.\n        end_prompt (`str`): A string that will mark the end of the content to look for.\n\n    Returns:\n        `str`: The content between the prompts.\n    \"\"\"\n    with open(filename, \"r\", encoding=\"utf-8\", newline=\"\\n\") as f:\n        lines = f.readlines()\n\n    # Find the start prompt.\n    start_index = 0\n    while not lines[start_index].startswith(start_prompt):\n        start_index += 1\n    start_index += 1\n\n    # Now go until the end prompt.\n    end_index = start_index\n    while not lines[end_index].startswith(end_prompt):\n        end_index += 1\n    end_index -= 1\n\n    while len(lines[start_index]) <= 1:\n        start_index += 1\n    while len(lines[end_index]) <= 1:\n        end_index -= 1\n    end_index += 1\n\n\nSUMMARY:\n**Function Summary: `_find_text_in_file`**\n\nThis function searches for text within a specified `filename` based on two prompts. It finds the content between these two markers and returns it.\n\n**Primary Purpose:** Find text in file, filtered by start and end prompts.\n\n**Implementation Details:**\n\n- Uses a loop to find the index of the first line starting with the `start_prompt`.\n- Continues searching after that using another loop to identify the last line ending with the `end_prompt`.\n- Ensures non-empty lines are included at the edges by incrementing/decrementing indices.\n- Returns the content between the prompts, along with start and end indices.\n\n**Notable Dependencies:**\n\n- `open()` for file operations\n- External dependencies not specified in provided snippet. However, the code assumes familiarity with external libraries and file system access.\n\n**Interfaces/Methods Used:**\n\n- File I/O (`open()`)\n- String manipulation (`readlines(), join()`)"}


created_at: 2025-02-05 09:13:53
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_table.py:96-115
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 115, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_table.py", "name": "camel_case_split", "start_line": 96, "type": "function", "chroma:document": "CODE:\n\ndef camel_case_split(identifier: str) -> List[str]:\n    \"\"\"\n    Split a camel-cased name into words.\n\n    Args:\n        identifier (`str`): The camel-cased name to parse.\n\n    Returns:\n        `List[str]`: The list of words in the identifier (as seprated by capital letters).\n\n    Example:\n\n    ```py\n    >>> camel_case_split(\"CamelCasedClass\")\n    [\"Camel\", \"Cased\", \"Class\"]\n    ```\n    \"\"\"\n    # Regex thanks to https://stackoverflow.com/questions/29916065/how-to-do-camelcase-split-in-python\n    matches = re.finditer(\".+?(?:(?<=[a-z])(?=[A-Z])|(?<=[A-Z])(?=[A-Z][a-z])|$)\", identifier)\n\n\nSUMMARY:\n**Function Summary: `camel_case_split`**\n\nThe `camel_case_split` function takes a camel-cased string as input and returns a list of words separated by capital letters.\n\n**Key Implementation Details:**\n\n* Uses regular expressions (regex) to split the input string using the pattern specified in the comments.\n* The regex pattern `.+?(?=(?=[a-z])(?=[A-Z])|(?<=[A-Z])(?=[A-Z][a-z]))|$` breaks down into:\n\t+ `.+?`: Matches one or more characters (except newline) non-greedy (`?`)\n\t+ `(?=[a-z]|(?=[A-Z])` and similar patterns: Looks for capital letters, ignoring whitespace\n\n**Notable Dependencies and Requirements:**\n\n* Python's built-in `re` module (regular expression library)\n* None, as it only relies on the standard library.\n\nThis code snippet achieves its purpose by utilizing regex to split the input string into words based on capital letter transitions."}


created_at: 2025-02-05 09:13:53
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_table.py:118-132
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 132, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_table.py", "name": "_center_text", "start_line": 118, "type": "function", "chroma:document": "CODE:\n\ndef _center_text(text: str, width: int) -> str:\n    \"\"\"\n    Utility that will add spaces on the left and right of a text to make it centered for a given width.\n\n    Args:\n        text (`str`): The text to center.\n        width (`int`): The desired length of the result.\n\n    Returns:\n        `str`: A text of length `width` with the original `text` in the middle.\n    \"\"\"\n    text_length = 2 if text == \"\u2705\" or text == \"\u274c\" else len(text)\n    left_indent = (width - text_length) // 2\n    right_indent = width - text_length - left_indent\n\n\nSUMMARY:\n**Summary**\n\nThe `_center_text` function centers a input `text` within a specified `width`, padding with spaces on the left and right. It optimizes padding for emojis (`\u2705` or `\u274c`) to avoid uneven indentation.\n\n**Key Details:**\n\n* Utilizes string multiplication (`\" \" *`) for efficient padding.\n* Calculates padding using integer division (`//`) to align text center.\n* Ignores emoji inputs by reducing text length by 2 characters.\n* Returns a centered string with exactly specified `width`.\n\n**Notable Dependencies/Requirements:**\n\n* Supports strings (including emojis) as input `text`.\n* Assumes explicit `int` type for `width`."}


created_at: 2025-02-05 09:13:53
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_table.py:185-268
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 268, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_table.py", "name": "get_model_table_from_auto_modules", "start_line": 185, "type": "function", "chroma:document": "CODE:\n\ndef get_model_table_from_auto_modules() -> str:\n    \"\"\"\n    Generates an up-to-date model table from the content of the auto modules.\n    \"\"\"\n    # Dictionary model names to config.\n    config_maping_names = transformers_module.models.auto.configuration_auto.CONFIG_MAPPING_NAMES\n    model_name_to_config = {\n        name: config_maping_names[code]\n        for code, name in transformers_module.MODEL_NAMES_MAPPING.items()\n        if code in config_maping_names\n    }\n    model_name_to_prefix = {name: config.replace(\"Config\", \"\") for name, config in model_name_to_config.items()}\n\n    # Dictionaries flagging if each model prefix has a backend in PT/TF/Flax.\n    pt_models = collections.defaultdict(bool)\n    tf_models = collections.defaultdict(bool)\n    flax_models = collections.defaultdict(bool)\n\n    # Let's lookup through all transformers object (once).\n    for attr_name in dir(transformers_module):\n        lookup_dict = None\n        if _re_tf_models.match(attr_name) is not None:\n            lookup_dict = tf_models\n            attr_name = _re_tf_models.match(attr_name).groups()[0]\n        elif _re_flax_models.match(attr_name) is not None:\n            lookup_dict = flax_models\n            attr_name = _re_flax_models.match(attr_name).groups()[0]\n        elif _re_pt_models.match(attr_name) is not None:\n            lookup_dict = pt_models\n            attr_name = _re_pt_models.match(attr_name).groups()[0]\n\n        if lookup_dict is not None:\n            while len(attr_name) > 0:\n                if attr_name in model_name_to_prefix.values():\n                    lookup_dict[attr_name] = True\n                    break\n                # Try again after removing the last word in the name\n                attr_name = \"\".join(camel_case_split(attr_name)[:-1])\n\n    # Let's build that table!\n    model_names = list(model_name_to_config.keys()) + list(MODEL_NAMES_WITH_SAME_CONFIG.keys())\n\n    # model name to doc link mapping\n    model_names_mapping = transformers_module.models.auto.configuration_auto.MODEL_NAMES_MAPPING\n    model_name_to_link_mapping = {value: f\"[{value}](model_doc/{key})\" for key, value in model_names_mapping.items()}\n    # update mapping with special model names\n    model_name_to_link_mapping = {\n        k: SPECIAL_MODEL_NAME_LINK_MAPPING[k] if k in SPECIAL_MODEL_NAME_LINK_MAPPING else v\n        for k, v in model_name_to_link_mapping.items()\n    }\n\n    # MaskFormerSwin and TimmBackbone are backbones and so not meant to be loaded and used on their own. Instead, they define architectures which can be loaded using the AutoBackbone API.\n    names_to_exclude = [\"MaskFormerSwin\", \"TimmBackbone\", \"Speech2Text2\"]\n    model_names = [name for name in model_names if name not in names_to_exclude]\n    model_names.sort(key=str.lower)\n\n    columns = [\"Model\", \"PyTorch support\", \"TensorFlow support\", \"Flax Support\"]\n    # We'll need widths to properly display everything in the center (+2 is to leave one extra space on each side).\n\n    widths = [len(c) + 2 for c in columns]\n    widths[0] = max([len(doc_link) for doc_link in model_name_to_link_mapping.values()]) + 2\n\n    # Build the table per se\n    table = \"|\" + \"|\".join([_center_text(c, w) for c, w in zip(columns, widths)]) + \"|\\n\"\n    # Use \":-----:\" format to center-aligned table cell texts\n    table += \"|\" + \"|\".join([\":\" + \"-\" * (w - 2) + \":\" for w in widths]) + \"|\\n\"\n\n    check = {True: \"\u2705\", False: \"\u274c\"}\n\n    for name in model_names:\n        if name in MODEL_NAMES_TO_IGNORE:\n            continue\n        if name in MODEL_NAMES_WITH_SAME_CONFIG.keys():\n            prefix = model_name_to_prefix[MODEL_NAMES_WITH_SAME_CONFIG[name]]\n        else:\n            prefix = model_name_to_prefix[name]\n        line = [\n            model_name_to_link_mapping[name],\n            check[pt_models[prefix]],\n            check[tf_models[prefix]],\n            check[flax_models[prefix]],\n        ]\n        table += \"|\" + \"|\".join([_center_text(l, w) for l, w in zip(line, widths)]) + \"|\\n\"\n\n\nSUMMARY:\n**Summary:**\n\n`get_model_table_from_auto_modules` function generates an up-to-date model table from the content of auto modules.\n\n**Main Purpose and Functionality:**\n\n* Generates a table with model names, PyTorch, TensorFlow, and Flax support information.\n* Retrieves configuration mapping, models, and prefixes from `transformers_module`.\n* Flags models for each backend (PT/TF/Flax) using regular expressions.\n* Builds the table by iterating through generated model names, checking backend support, and formatting text.\n\n**Notable Implementation Details:**\n\n* Uses dictionaries to store configuration mapping, model names, and prefixes.\n* Utilizes regular expressions to flag models for each backend.\n* Uses `_center_text` function to format table cell texts.\n\n**Notable Dependencies or Requirements:**\n\n* `transformers_module` module (contains constants and functions).\n* `_re_tf_models`, `_re_flax_models`, `_re_pt_models` regular expression patterns (defined in `transformers_module`)."}


created_at: 2025-02-05 09:13:53
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_table.py:271-293
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 293, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_table.py", "name": "check_model_table", "start_line": 271, "type": "function", "chroma:document": "CODE:\n\ndef check_model_table(overwrite=False):\n    \"\"\"\n    Check the model table in the index.md is consistent with the state of the lib and potentially fix it.\n\n    Args:\n        overwrite (`bool`, *optional*, defaults to `False`):\n            Whether or not to overwrite the table when it's not up to date.\n    \"\"\"\n    current_table, start_index, end_index, lines = _find_text_in_file(\n        filename=os.path.join(PATH_TO_DOCS, \"index.md\"),\n        start_prompt=\"<!--This table is updated automatically from the auto modules\",\n        end_prompt=\"<!-- End table-->\",\n    )\n    new_table = get_model_table_from_auto_modules()\n\n    if current_table != new_table:\n        if overwrite:\n            with open(os.path.join(PATH_TO_DOCS, \"index.md\"), \"w\", encoding=\"utf-8\", newline=\"\\n\") as f:\n                f.writelines(lines[:start_index] + [new_table] + lines[end_index:])\n        else:\n            raise ValueError(\n                \"The model table in the `index.md` has not been updated. Run `make fix-copies` to fix this.\"\n\n\nSUMMARY:\n**Summary**\n\nThis is a Python function named `check_model_table` that checks if the content of the `model table` in the `index.md` file is consistent with its creation via auto-modules. It returns two tables, one previous (`current_table`) and one generated from auto-modules (`new_table`). The primary purpose of this function is to detect inconsistencies and update the `index.md` file accordingly.\n\n**Key Implementation Details**\n\n- This function uses `_find_text_in_file` to extract text ranges containing the model table from `index.md`. \n- It generates a new table using `get_model_table_from_auto_modules`.\n- If the tables are different, it either overwrites the outdated table or raises an error with a suggestion to update via `make fix-copies`.\n\n**Notable Dependencies/Requirements**\n\n- `os` (file system operations)\n- `PATH_TO_DOCS`, assumed to be a path constant defined elsewhere in the code.\n- `_find_text_in_file` and `get_model_table_from_auto_modules` functions, assumed to be implemented elsewhere."}


created_at: 2025-02-05 09:13:53
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/add_pipeline_model_mapping_to_test.py:49-59
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 59, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/add_pipeline_model_mapping_to_test.py", "name": "get_framework", "start_line": 49, "type": "function", "chroma:document": "CODE:\n\ndef get_framework(test_class):\n    \"\"\"Infer the framework from the test class `test_class`.\"\"\"\n\n    if \"ModelTesterMixin\" in [x.__name__ for x in test_class.__bases__]:\n        return \"pt\"\n    elif \"TFModelTesterMixin\" in [x.__name__ for x in test_class.__bases__]:\n        return \"tf\"\n    elif \"FlaxModelTesterMixin\" in [x.__name__ for x in test_class.__bases__]:\n        return \"flax\"\n    else:\n\n\nSUMMARY:\n**Summary:**\n\n`get_framework` function infers the framework associated with a given test class. It searches inheritance hierarchy of `test_class` and returns `\"pt\"`, `\"tf\"`, or `\"flax\"` based on the presence of `ModelTesterMixin` or its subclasses.\n\n**Key Details:**\n\n* Function signature: `def get_framework(test_class)`\n* Primary purpose: Infer framework type from test class\n* Algorithm:\n\t+ Search inheritance hierarchy of `test_class`\n\t+ Check for `ModelTesterMixin`, `TFModelTesterMixin`, or `FlaxModelTesterMixin` subclasses\n\t+ Return corresponding framework ID (\"pt\", \"tf\", or \"flax\")\n\n**Dependencies/Requirements:**\n\n* None"}


created_at: 2025-02-05 09:13:54
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/add_pipeline_model_mapping_to_test.py:62-81
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 81, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/add_pipeline_model_mapping_to_test.py", "name": "get_mapping_for_task", "start_line": 62, "type": "function", "chroma:document": "CODE:\n\ndef get_mapping_for_task(task, framework):\n    \"\"\"Get mappings defined in `XXXPipelineTests` for the task `task`.\"\"\"\n    # Use the cached results\n    if PIPELINE_TEST_MAPPING[task].get(framework, None) is not None:\n        return PIPELINE_TEST_MAPPING[task][framework]\n\n    pipeline_test_class = pipeline_test_mapping[task][\"test\"]\n    mapping = None\n\n    if framework == \"pt\":\n        mapping = getattr(pipeline_test_class, \"model_mapping\", None)\n    elif framework == \"tf\":\n        mapping = getattr(pipeline_test_class, \"tf_model_mapping\", None)\n\n    if mapping is not None:\n        mapping = dict(mapping.items())\n\n    # cache the results\n    PIPELINE_TEST_MAPPING[task][framework] = mapping\n\n\nSUMMARY:\n**Function Summary: `get_mapping_for_task`**\n\n**Purpose:** Returns the task-specific model mapping for a given task and framework, retrieving or caching the result.\n\n**Key Implementation Details:**\n\n* Retrieves cached result if available through `PIPELINE_TEST_MAPPING[task].get(framework, None)`.\n* Uses frameworkspecific attributes (`model_mapping`, `tf_model_mapping`) from `pipeline_test_class` to construct the mapping.\n* Caches the result in `PIPELINE_TEST_MAPPING[task][framework]`.\n\n**Notable Dependencies and Requirements:**\n\n* `pipeline_test_mapping`: a dictionary containing task-specific pipeline test data.\n* `PIPELINE TestingMapping`: an external cache storage for mapped results.\n\n**Method Signatures and Interfaces:**\n\n* `get(framework) -> Mapping dict`"}


created_at: 2025-02-05 09:13:54
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/add_pipeline_model_mapping_to_test.py:84-102
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 102, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/add_pipeline_model_mapping_to_test.py", "name": "get_model_for_pipeline_test", "start_line": 84, "type": "function", "chroma:document": "CODE:\n\ndef get_model_for_pipeline_test(test_class, task):\n    \"\"\"Get the model architecture(s) related to the test class `test_class` for a pipeline `task`.\"\"\"\n    framework = get_framework(test_class)\n    if framework is None:\n        return None\n    mapping = get_mapping_for_task(task, framework)\n    if mapping is None:\n        return None\n\n    config_classes = list({model_class.config_class for model_class in test_class.all_model_classes})\n    if len(config_classes) != 1:\n        raise ValueError(\"There should be exactly one configuration class from `test_class.all_model_classes`.\")\n\n    # This could be a list/tuple of model classes, but it's rare.\n    model_class = mapping.get(config_classes[0], None)\n    if isinstance(model_class, (tuple, list)):\n        model_class = sorted(model_class, key=lambda x: x.__name__)\n\n\n\nSUMMARY:\n**Summary:**\n\nThe `get_model_for_pipeline_test` function retrieves the model architecture(s) associated with a given test class `test_class` and pipeline task `task`. It achieves this by:\n\n*   Retrieving the framework related to `test_class` using `get_framework`.\n*   Resolving the mapping for `task` based on the retrieved framework using `get_mapping_for_task`.\n*   Identifying the sole configuration class from `test_class.all_model_classes` and retrieving the corresponding model architecture(s) using this class's configuration.\n\n**Key Details:**\n\n*   The function raises a `ValueError` if more than one configuration class is present in `test_class.all_model_classes`.\n*   The model architecture may be returned as a single class, list, or tuple of classes, sorted by their names for consistency.\n\n**Notable Dependencies/Requirements:**\n\n*   `get_framework`\n*   `get_mapping_for_task`\n*   `test_class` (assuming it has methods like `all_model_classes`).\n*   Import statements assume existing models are defined with a `config_class`."}


created_at: 2025-02-05 09:13:54
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/add_pipeline_model_mapping_to_test.py:105-110
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 110, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/add_pipeline_model_mapping_to_test.py", "name": "get_pipeline_model_mapping", "start_line": 105, "type": "function", "chroma:document": "CODE:\n\ndef get_pipeline_model_mapping(test_class):\n    \"\"\"Get `pipeline_model_mapping` for `test_class`.\"\"\"\n    mapping = [(task, get_model_for_pipeline_test(test_class, task)) for task in pipeline_test_mapping]\n    mapping = sorted([(task, model) for task, model in mapping if model is not None], key=lambda x: x[0])\n\n\n\nSUMMARY:\n**Function Summary**\n\n`get_pipeline_model_mapping` returns a dictionary mapping test classes to pipelines, where each pipeline is executed and mapped to its corresponding model. \n\n**Key Details**\n\n* Iterates through `pipeline_test_mapping` using `test_class` as input.\n* Applies `get_model_for_pipeline_test` function for each task.\n* Filters out None values and sorts the results by task name.\n\n**Return Value**\n\nA dictionary with task names as keys and pipeline models as values. \n\n**Dependences/Requirements**\n\n* `pipeline_test_mapping`: A list of tuples containing test class names and corresponding pipeline tasks.\n* `get_model_for_pipeline_test(test_class, task)`: A function that returns a model for a given pipeline task."}


created_at: 2025-02-05 09:13:54
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/add_pipeline_model_mapping_to_test.py:113-139
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 139, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/add_pipeline_model_mapping_to_test.py", "name": "get_pipeline_model_mapping_string", "start_line": 113, "type": "function", "chroma:document": "CODE:\n\ndef get_pipeline_model_mapping_string(test_class):\n    \"\"\"Get `pipeline_model_mapping` for `test_class` as a string (to be added to the test file).\n\n    This will be a 1-line string. After this is added to a test file, `make style` will format it beautifully.\n    \"\"\"\n    framework = get_framework(test_class)\n    if framework == \"pt\":\n        framework = \"torch\"\n    default_value = \"{}\"\n\n    mapping = get_pipeline_model_mapping(test_class)\n    if len(mapping) == 0:\n        return \"\"\n\n    texts = []\n    for task, model_classes in mapping.items():\n        if isinstance(model_classes, (tuple, list)):\n            # A list/tuple of model classes\n            value = \"(\" + \", \".join([x.__name__ for x in model_classes]) + \")\"\n        else:\n            # A single model class\n            value = model_classes.__name__\n        texts.append(f'\"{task}\": {value}')\n    text = \"{\" + \", \".join(texts) + \"}\"\n    text = f\"pipeline_model_mapping = {text} if is_{framework}_available() else {default_value}\"\n\n\n\nSUMMARY:\n**Function Summary: `get_pipeline_model_mapping_string`**\n\n**Purpose and Functionality:** This function generates a 1-line string representing the pipeline model mapping for a given test class, which can be added to a test file.\n\n**Key Implementation Details:**\n\n* Uses the `get_framework` function to determine the framework type (\"pt\" or \"torch\") based on the input test class.\n* Calls `get_pipeline_model_mapping` to retrieve the pipeline model mapping.\n* Constructs a list of text strings (`\"task\": value`) representing each model class/class tuple pair in the mapping.\n* Joins these strings into a single JSON-like string and formats it with conditional logic using the `is_{framework}_available()` function.\n\n**Notable Dependencies and Requirements:**\n\n* The `get_framework` function\n* The `get_pipeline_model_mapping` function\n* Conditional availability check using `is_{framework}_available()`\n* Python dictionary-like formatting (`{key: value}`) for JSON-style strings."}


created_at: 2025-02-05 09:13:54
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/add_pipeline_model_mapping_to_test.py:142-147
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 147, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/add_pipeline_model_mapping_to_test.py", "name": "is_valid_test_class", "start_line": 142, "type": "function", "chroma:document": "CODE:\n\ndef is_valid_test_class(test_class):\n    \"\"\"Restrict to `XXXModelTesterMixin` and should be a subclass of `unittest.TestCase`.\"\"\"\n    base_class_names = {\"ModelTesterMixin\", \"TFModelTesterMixin\", \"FlaxModelTesterMixin\"}\n    if not issubclass(test_class, unittest.TestCase):\n        return False\n\n\nSUMMARY:\n**Summary**\n\nValidates if a given test class inherits from `unittest.TestCase` and is derived from one of the specified model tester mixins:\n```python\ndef is_valid_test_class(test_class):\n    \"\"\"Restricts to `XXXModelTesterMixin` and should be a subclass of `unittest.TestCase`.\n    \n    Parameters:\n        test_class (class): The class to validate.\n    \n    Returns:\n        bool: True if valid, False otherwise.\n    \"\"\"\n    # Constants\n    base ClassNames = {\"ModelTesterMixin\", \"TFModelTesterMixin\", \"FlaxModelTesterMixin\"}\n    \n    # Validation check\n    def isValidBaseClass(base_class):\n        return len(base_classNames.intersection([x.__name__ for x in base_class.__bases__])) > 0\n    \n    # Check if the test class is a subclass of TestCase and has one of the model tester mixins as an base class\n    return not issubclass(test_class, unittest.TestCase) or isValidBaseClass(test_class)\n```\n**Key Details**\n\n* `unittest.TestCase` class validation\n* `ModelTesterMixin`, `TFModelTesterMixin`, and `FlaxModelTesterMixin` derived classes are expected\n\nNote: The `is_valid_test_class` function returns False if the test class does not inherit from `unittest_TestCase` or if it does not have a mixin as an base class, whereas the original code would return False immediately without checking for any mixin."}


created_at: 2025-02-05 09:13:54
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/add_pipeline_model_mapping_to_test.py:150-164
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 164, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/add_pipeline_model_mapping_to_test.py", "name": "find_test_class", "start_line": 150, "type": "function", "chroma:document": "CODE:\n\ndef find_test_class(test_file):\n    \"\"\"Find a test class in `test_file` to which we will add `pipeline_model_mapping`.\"\"\"\n    test_classes = [x for x in get_test_classes(test_file) if is_valid_test_class(x)]\n\n    target_test_class = None\n    for test_class in test_classes:\n        # If a test class has defined `pipeline_model_mapping`, let's take it\n        if getattr(test_class, \"pipeline_model_mapping\", None) is not None:\n            target_test_class = test_class\n            break\n    # Take the test class with the shortest name (just a heuristic)\n    if target_test_class is None and len(test_classes) > 0:\n        target_test_class = sorted(test_classes, key=lambda x: (len(x.__name__), x.__name__))[0]\n\n\n\nSUMMARY:\n**Function Summary:**\n\n`find_test_class(test_file)` finds and selects the test class from the `test_file` to which `pipeline_model_mapping` will be added.\n\n**Key Implementation Details:**\n\n* Iterates over valid test classes in `test_file`.\n* Checks for presence of `pipeline_model_mapping` attribute; selects first matching class.\n* Fallbacks to shortest class name heuristic if no match found.\n\n**Notable Dependencies and Requirements:**\n\n* `get_test_classes(test_file)`: returns list of test classes in `test_file`.\n* `is_valid_test_class(x)`: checks validity of a test class.\n* External dependency: `test_file` (required)."}


created_at: 2025-02-05 09:13:54
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/add_pipeline_model_mapping_to_test.py:167-177
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 177, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/add_pipeline_model_mapping_to_test.py", "name": "find_block_ending", "start_line": 167, "type": "function", "chroma:document": "CODE:\n\ndef find_block_ending(lines, start_idx, indent_level):\n    end_idx = start_idx\n    for idx, line in enumerate(lines[start_idx:]):\n        indent = len(line) - len(line.lstrip())\n        if idx == 0 or indent > indent_level or (indent == indent_level and line.strip() == \")\"):\n            end_idx = start_idx + idx\n        elif idx > 0 and indent <= indent_level:\n            # Outside the definition block of `pipeline_model_mapping`\n            break\n\n\n\nSUMMARY:\n**Summary:**\n\n`find_block_ending`: Finds the index of the last line in a code block, ending with a matching bracket.\n\n**Purpose:** The function locates the closing bracket of a block defined by indentation levels.\n\n**Key Implementation Details:**\n\n* Iterates through lines starting from `start_idx`.\n* Checks for matching bracket using indentation level.\n* Returns the index of the block's end (`end_idx`).\n\n**Note:** The function ensures the block does not exceed a line containing the string \")\" while also considering lines outside the block defined by \"pipeline_model_mapping\"."}


created_at: 2025-02-05 09:13:54
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/add_pipeline_model_mapping_to_test.py:180-289
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 289, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/add_pipeline_model_mapping_to_test.py", "name": "add_pipeline_model_mapping", "start_line": 180, "type": "function", "chroma:document": "CODE:\n\ndef add_pipeline_model_mapping(test_class, overwrite=False):\n    \"\"\"Add `pipeline_model_mapping` to `test_class`.\"\"\"\n    if getattr(test_class, \"pipeline_model_mapping\", None) is not None:\n        if not overwrite:\n            return \"\", -1\n\n    line_to_add = get_pipeline_model_mapping_string(test_class)\n    if len(line_to_add) == 0:\n        return \"\", -1\n    line_to_add = line_to_add + \"\\n\"\n\n    # The code defined the class `test_class`\n    class_lines, class_start_line_no = inspect.getsourcelines(test_class)\n    # `inspect` gives the code for an object, including decorator(s) if any.\n    # We (only) need the exact line of the class definition.\n    for idx, line in enumerate(class_lines):\n        if line.lstrip().startswith(\"class \"):\n            class_lines = class_lines[idx:]\n            class_start_line_no += idx\n            break\n    class_end_line_no = class_start_line_no + len(class_lines) - 1\n\n    # The index in `class_lines` that starts the definition of `all_model_classes`, `all_generative_model_classes` or\n    # `pipeline_model_mapping`. This assumes they are defined in such order, and we take the start index of the last\n    # block that appears in a `test_class`.\n    start_idx = None\n    # The indent level of the line at `class_lines[start_idx]` (if defined)\n    indent_level = 0\n    # To record if `pipeline_model_mapping` is found in `test_class`.\n    def_line = None\n    for idx, line in enumerate(class_lines):\n        if line.strip().startswith(\"all_model_classes = \"):\n            indent_level = len(line) - len(line.lstrip())\n            start_idx = idx\n        elif line.strip().startswith(\"all_generative_model_classes = \"):\n            indent_level = len(line) - len(line.lstrip())\n            start_idx = idx\n        elif line.strip().startswith(\"pipeline_model_mapping = \"):\n            indent_level = len(line) - len(line.lstrip())\n            start_idx = idx\n            def_line = line\n            break\n\n    if start_idx is None:\n        return \"\", -1\n    # Find the ending index (inclusive) of the above found block.\n    end_idx = find_block_ending(class_lines, start_idx, indent_level)\n\n    # Extract `is_xxx_available()` from existing blocks: some models require specific libraries like `timm` and use\n    # `is_timm_available()` instead of `is_torch_available()`.\n    # Keep leading and trailing whitespaces\n    r = re.compile(r\"\\s(is_\\S+?_available\\(\\))\\s\")\n    for line in class_lines[start_idx : end_idx + 1]:\n        backend_condition = r.search(line)\n        if backend_condition is not None:\n            # replace the leading and trailing whitespaces to the space character \" \".\n            target = \" \" + backend_condition[0][1:-1] + \" \"\n            line_to_add = r.sub(target, line_to_add)\n            break\n\n    if def_line is None:\n        # `pipeline_model_mapping` is not defined. The target index is set to the ending index (inclusive) of\n        # `all_model_classes` or `all_generative_model_classes`.\n        target_idx = end_idx\n    else:\n        # `pipeline_model_mapping` is defined. The target index is set to be one **BEFORE** its start index.\n        target_idx = start_idx - 1\n        # mark the lines of the currently existing `pipeline_model_mapping` to be removed.\n        for idx in range(start_idx, end_idx + 1):\n            # These lines are going to be removed before writing to the test file.\n            class_lines[idx] = None  # noqa\n\n    # Make sure the test class is a subclass of `PipelineTesterMixin`.\n    parent_classes = [x.__name__ for x in test_class.__bases__]\n    if \"PipelineTesterMixin\" not in parent_classes:\n        # Put `PipelineTesterMixin` just before `unittest.TestCase`\n        _parent_classes = [x for x in parent_classes if x != \"TestCase\"] + [\"PipelineTesterMixin\"]\n        if \"TestCase\" in parent_classes:\n            # Here we **assume** the original string is always with `unittest.TestCase`.\n            _parent_classes.append(\"unittest.TestCase\")\n        parent_classes = \", \".join(_parent_classes)\n        for idx, line in enumerate(class_lines):\n            # Find the ending of the declaration of `test_class`\n            if line.strip().endswith(\"):\"):\n                # mark the lines of the declaration of `test_class` to be removed\n                for _idx in range(idx + 1):\n                    class_lines[_idx] = None  # noqa\n                break\n        # Add the new, one-line, class declaration for `test_class`\n        class_lines[0] = f\"class {test_class.__name__}({parent_classes}):\\n\"\n\n    # Add indentation\n    line_to_add = \" \" * indent_level + line_to_add\n    # Insert `pipeline_model_mapping` to `class_lines`.\n    # (The line at `target_idx` should be kept by definition!)\n    class_lines = class_lines[: target_idx + 1] + [line_to_add] + class_lines[target_idx + 1 :]\n    # Remove the lines that are marked to be removed\n    class_lines = [x for x in class_lines if x is not None]\n\n    # Move from test class to module (in order to write to the test file)\n    module_lines = inspect.getsourcelines(inspect.getmodule(test_class))[0]\n    # Be careful with the 1-off between line numbers and array indices\n    module_lines = module_lines[: class_start_line_no - 1] + class_lines + module_lines[class_end_line_no:]\n    code = \"\".join(module_lines)\n\n    moddule_file = inspect.getsourcefile(test_class)\n    with open(moddule_file, \"w\", encoding=\"UTF-8\", newline=\"\\n\") as fp:\n        fp.write(code)\n\n\n\nSUMMARY:\nHere is a concise summary of the provided function code chunk:\n\n**Function Purpose:**\nThe `add_pipeline_model_mapping` function adds or ovewrites the `pipeline_model_mapping` attribute to a given test class (`test_class`).\n\n**Key Implementation Details:**\n\n1.  The function checks if `test_class` already has a defined `pipeline_model_mapping` and only overwrites it if set to `False`.\n2.  It searches for the line that defines the previous block (e.g., `all_model_classes`, `all_generative_model_classes`, `pipeline_model_mapping`) to determine the ending index of the block.\n3.  If a matching line is found, the function removes any leading whitespaces and replaces it with a specific regex pattern (`is_\\S+_(available)\\s`) before extracting the new line of code for `pipeline_model_mapping`.\n4.  It ensures that the modified test class name is appended after its parent classes using the `\"unittest.TestCase\"`\n5.  After adding or overwriting `pipeline_model_mapping`, it removes the lines marked to be removed.\n\n**Dependencies and Requirements:**\n\n1.  Uses the `inspect` module for code inspection.\n2.  References related classes/functions by exact name, such as getting class lines using its sourcelines method.\n3.  Requires external dependencies like regular expression (`re`) library.\n\n**Return Value:** The added or overwritten line of new `pipeline_model_mapping`."}


created_at: 2025-02-05 09:13:54
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/add_pipeline_model_mapping_to_test.py:292-296
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 296, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/add_pipeline_model_mapping_to_test.py", "name": "add_pipeline_model_mapping_to_test_file", "start_line": 292, "type": "function", "chroma:document": "CODE:\n\ndef add_pipeline_model_mapping_to_test_file(test_file, overwrite=False):\n    \"\"\"Add `pipeline_model_mapping` to `test_file`.\"\"\"\n    test_class = find_test_class(test_file)\n    if test_class:\n\n\nSUMMARY:\n**Summary:**\n\n`add_pipeline_model_mapping_to_test_file`: Adds the `pipeline_model_mapping` configuration to a `test_file`, overwriting if specified.\n\n**Key Details:**\n\n- Calls `find_test_class` with `test_file` as input.\n- If a test class is found, it calls `add_pipeline_model_mapping` with the class and an optional overwrite flag.\n- The primary purpose of using `overwrite=False` in `add_pipeline_model_mapping` if `test_class` exists.\n\n Relevant methods: \n - `find_test_class(test_file)`\n - `add_pipeline_model_mapping(test_class, overwrite)`\n\nNote:\n- External dependency: `find_test_class` and `add_pipeline_model_mapping` functions are assumed to be defined elsewhere."}


created_at: 2025-02-05 09:13:54
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py:149-188
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 188, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py", "name": "get_processor_types_from_config_class", "start_line": 149, "type": "function", "chroma:document": "CODE:\n\ndef get_processor_types_from_config_class(config_class, allowed_mappings=None):\n    \"\"\"Return a tuple of processors for `config_class`.\n\n    We use `tuple` here to include (potentially) both slow & fast tokenizers.\n    \"\"\"\n\n    # To make a uniform return type\n    def _to_tuple(x):\n        if not isinstance(x, collections.abc.Sequence):\n            x = (x,)\n        else:\n            x = tuple(x)\n        return x\n\n    if allowed_mappings is None:\n        allowed_mappings = [\"processor\", \"tokenizer\", \"image_processor\", \"feature_extractor\"]\n\n    processor_types = ()\n\n    # Check first if a model has `ProcessorMixin`. Otherwise, check if it has tokenizers, and/or an image processor or\n    # a feature extractor\n    if config_class in PROCESSOR_MAPPING and \"processor\" in allowed_mappings:\n        processor_types = _to_tuple(PROCESSOR_MAPPING[config_class])\n    else:\n        if config_class in TOKENIZER_MAPPING and \"tokenizer\" in allowed_mappings:\n            processor_types = TOKENIZER_MAPPING[config_class]\n\n        if config_class in IMAGE_PROCESSOR_MAPPING and \"image_processor\" in allowed_mappings:\n            processor_types += _to_tuple(IMAGE_PROCESSOR_MAPPING[config_class])\n        elif config_class in FEATURE_EXTRACTOR_MAPPING and \"feature_extractor\" in allowed_mappings:\n            processor_types += _to_tuple(FEATURE_EXTRACTOR_MAPPING[config_class])\n\n    # Remark: some configurations have no processor at all. For example, generic composite models like\n    # `EncoderDecoderModel` is used for any (compatible) text models. Also, `DecisionTransformer` doesn't\n    # require any processor.\n\n    # We might get `None` for some tokenizers - remove them here.\n    processor_types = tuple(p for p in processor_types if p is not None)\n\n\n\nSUMMARY:\n**Code Summary**\n\n```\nget_processor_types_from_config_class(config_class, allowed_mappings=None)\n-------------------------\n\nPurpose:\nExtracts and returns a tuple of supported processors or tokenizer types based on the provided config class.\n\nFunctionality:\n\n* Takes a config class and an optional list of allowed mappings as input.\n* Checks if the config class has a direct mapping in PROCESSOR_MAPPING, then uses that.\n* If not, checks for indirect mappings through TOKENIZER_MAPPING, IMAGE_PROCESSOR_MAPPING, or FEATURE_EXTRACTOR_MAPPING.\n* Removes None values from the resulting processor types tuple.\n\nKey Implementation Details:\n\n* Uses `_to_tuple` function to ensure uniform return type (sequence or scalar).\n* Checks for specific allowed mappings and their corresponding mappings in PROCESSOR_MAPPING and other alias dictionaries.\n* Uses tuple concatenation to combine multiple processor types into a single tuple.\n\nDependencies/Requirements:\n\n* PROCESSOR_MAPPING, TOKENIZER_MAPPING, IMAGE_PROCESSOR_MAPPING, FEATURE_EXTRACTOR_MAPPING\n* collections.abc.Sequence (for isinstance check)\n```\n\nNote: The code summary is written in a concise and focused manner, highlighting the main purpose, functionality, key implementation details, and dependencies."}


created_at: 2025-02-05 09:13:54
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py:191-218
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 218, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py", "name": "get_architectures_from_config_class", "start_line": 191, "type": "function", "chroma:document": "CODE:\n\ndef get_architectures_from_config_class(config_class, arch_mappings, models_to_skip=None):\n    \"\"\"Return a tuple of all possible architectures attributed to a configuration class `config_class`.\n\n    For example, BertConfig -> [BertModel, BertForMaskedLM, ..., BertForQuestionAnswering].\n    \"\"\"\n    # A model architecture could appear in several mappings. For example, `BartForConditionalGeneration` is in\n    #   - MODEL_FOR_PRETRAINING_MAPPING_NAMES\n    #   - MODEL_WITH_LM_HEAD_MAPPING_NAMES\n    #   - MODEL_FOR_MASKED_LM_MAPPING_NAMES\n    #   - MODEL_FOR_SEQ_TO_SEQ_CAUSAL_LM_MAPPING_NAMES\n    # We avoid the duplication.\n    architectures = set()\n\n    if models_to_skip is None:\n        models_to_skip = []\n    models_to_skip = UNCONVERTIBLE_MODEL_ARCHITECTURES.union(models_to_skip)\n\n    for mapping in arch_mappings:\n        if config_class in mapping:\n            models = mapping[config_class]\n            models = tuple(models) if isinstance(models, collections.abc.Sequence) else (models,)\n            for model in models:\n                if model.__name__ not in models_to_skip:\n                    architectures.add(model)\n\n    architectures = tuple(architectures)\n\n\n\nSUMMARY:\n**Code Summary:**\n\n### Purpose and Functionality\n\n`get_architectures_from_config_class` returns a tuple of possible architectures attributed to a configuration class `config_class`, filtering out excluded models using provided mappings.\n\n### Key Implementation Details\n\n*   Uses a set to avoid duplications across multiple mapping sources.\n*   Filters out models based on a boolean `models_to_skip`.\n\n### Notable Dependencies or Requirements\n\n*   `arch_mappings`: dictionary of configuration classes mapped to architectures\n*   `models_to_skip`: optional list of excluded model names (defaults to empty set)\n*   `UNCONVERTIBLE_MODEL_ARCHITECTURES`: pre-defined set of inconvertible architecture models"}


created_at: 2025-02-05 09:13:54
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py:221-242
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 242, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py", "name": "get_config_class_from_processor_class", "start_line": 221, "type": "function", "chroma:document": "CODE:\n\ndef get_config_class_from_processor_class(processor_class):\n    \"\"\"Get the config class from a processor class.\n\n    Some config/model classes use tokenizers/feature_extractors from other models. For example, `GPT-J` uses\n    `GPT2Tokenizer`. If no checkpoint is found for a config class, or a checkpoint is found without necessary file(s) to\n    create the processor for `processor_class`, we get the config class that corresponds to `processor_class` and use it\n    to find a checkpoint in order to create the processor.\n    \"\"\"\n\n    processor_prefix = processor_class.__name__\n    for postfix in [\"TokenizerFast\", \"Tokenizer\", \"ImageProcessor\", \"FeatureExtractor\", \"Processor\"]:\n        processor_prefix = processor_prefix.replace(postfix, \"\")\n\n    # `Wav2Vec2CTCTokenizer` -> `Wav2Vec2Config`\n    if processor_prefix == \"Wav2Vec2CTC\":\n        processor_prefix = \"Wav2Vec2\"\n\n    # Find the new configuration class\n    new_config_name = f\"{processor_prefix}Config\"\n    new_config_class = getattr(transformers_module, new_config_name)\n\n\n\nSUMMARY:\n**Summary:**\n\nThis function, `get_config_class_from_processor_class`, retrieves the corresponding config class from a given processor class. It checks for prefixes in the processor class name to determine the necessary config class, and returns it if a matching checkpoint is found.\n\n**Key Implementation Details:**\n\n* Replaced the original prefix with a shortened version (e.g., \"Tokenizer\" -> \"\")\n* Handles specific cases for Wav2Vec2CTC and other models\n* Uses the `getattr` function to dynamically retrieve the config class from the `transformers_module`\n\n**Notable Dependencies and Requirements:**\n\n* `transformers_module`: assumed to be a module with config classes registered under their name (e.g., `Wav2Vec2Config`, `GPT2Tokenizer`)\n* Tokenizer/feature extractor dependencies specific to each model are not explicitly handled; may require additional configuration or file presence checks."}


created_at: 2025-02-05 09:13:54
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py:245-372
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 372, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py", "name": "build_processor", "start_line": 245, "type": "function", "chroma:document": "CODE:\n\ndef build_processor(config_class, processor_class, allow_no_checkpoint=False):\n    \"\"\"Create a processor for `processor_class`.\n\n    If a processor is not able to be built with the original arguments, this method tries to change the arguments and\n    call itself recursively, by inferring a new `config_class` or a new `processor_class` from another one, in order to\n    find a checkpoint containing the necessary files to build a processor.\n\n    The processor is not saved here. Instead, it will be saved in `convert_processors` after further changes in\n    `convert_processors`. For each model architecture`, a copy will be created and saved along the built model.\n    \"\"\"\n    # Currently, this solely uses the docstring in the source file of `config_class` to find a checkpoint.\n    checkpoint = get_checkpoint_from_config_class(config_class)\n\n    if checkpoint is None:\n        # try to get the checkpoint from the config class for `processor_class`.\n        # This helps cases like `XCLIPConfig` and `VideoMAEFeatureExtractor` to find a checkpoint from `VideoMAEConfig`.\n        config_class_from_processor_class = get_config_class_from_processor_class(processor_class)\n        checkpoint = get_checkpoint_from_config_class(config_class_from_processor_class)\n\n    processor = None\n    try:\n        processor = processor_class.from_pretrained(checkpoint)\n    except Exception as e:\n        logger.error(f\"{e.__class__.__name__}: {e}\")\n\n    # Try to get a new processor class from checkpoint. This is helpful for a checkpoint without necessary file to load\n    # processor while `processor_class` is an Auto class. For example, `sew` has `Wav2Vec2Processor` in\n    # `PROCESSOR_MAPPING_NAMES`, its `tokenizer_class` is `AutoTokenizer`, and the checkpoint\n    # `https://huggingface.co/asapp/sew-tiny-100k` has no tokenizer file, but we can get\n    # `tokenizer_class: Wav2Vec2CTCTokenizer` from the config file. (The new processor class won't be able to load from\n    # `checkpoint`, but it helps this recursive method to find a way to build a processor).\n    if (\n        processor is None\n        and checkpoint is not None\n        and issubclass(processor_class, (PreTrainedTokenizerBase, AutoTokenizer))\n    ):\n        try:\n            config = AutoConfig.from_pretrained(checkpoint)\n        except Exception as e:\n            logger.error(f\"{e.__class__.__name__}: {e}\")\n            config = None\n        if config is not None:\n            if not isinstance(config, config_class):\n                raise ValueError(\n                    f\"`config` (which is of type {config.__class__.__name__}) should be an instance of `config_class`\"\n                    f\" ({config_class.__name__})!\"\n                )\n            tokenizer_class = config.tokenizer_class\n            new_processor_class = None\n            if tokenizer_class is not None:\n                new_processor_class = getattr(transformers_module, tokenizer_class)\n                if new_processor_class != processor_class:\n                    processor = build_processor(config_class, new_processor_class)\n            # If `tokenizer_class` is not specified in `config`, let's use `config` to get the process class via auto\n            # mappings, but only allow the tokenizer mapping being used. This is to make `Wav2Vec2Conformer` build\n            if processor is None:\n                new_processor_classes = get_processor_types_from_config_class(\n                    config.__class__, allowed_mappings=[\"tokenizer\"]\n                )\n                # Used to avoid infinite recursion between a pair of fast/slow tokenizer types\n                names = [\n                    x.__name__.replace(\"Fast\", \"\") for x in [processor_class, new_processor_class] if x is not None\n                ]\n                new_processor_classes = [\n                    x for x in new_processor_classes if x is not None and x.__name__.replace(\"Fast\", \"\") not in names\n                ]\n                if len(new_processor_classes) > 0:\n                    new_processor_class = new_processor_classes[0]\n                    # Let's use fast tokenizer if there is any\n                    for x in new_processor_classes:\n                        if x.__name__.endswith(\"Fast\"):\n                            new_processor_class = x\n                            break\n                    processor = build_processor(config_class, new_processor_class)\n\n    if processor is None:\n        # Try to build each component (tokenizer & feature extractor) of a `ProcessorMixin`.\n        if issubclass(processor_class, ProcessorMixin):\n            attrs = {}\n            for attr_name in processor_class.attributes:\n                attrs[attr_name] = []\n                # This could be a tuple (for tokenizers). For example, `CLIPProcessor` has\n                #   - feature_extractor_class = \"CLIPFeatureExtractor\"\n                #   - tokenizer_class = (\"CLIPTokenizer\", \"CLIPTokenizerFast\")\n                attr_class_names = getattr(processor_class, f\"{attr_name}_class\")\n                if not isinstance(attr_class_names, tuple):\n                    attr_class_names = (attr_class_names,)\n\n                for name in attr_class_names:\n                    attr_class = getattr(transformers_module, name)\n                    attr = build_processor(config_class, attr_class)\n                    if attr is not None:\n                        attrs[attr_name].append(attr)\n\n            # try to build a `ProcessorMixin`, so we can return a single value\n            if all(len(v) > 0 for v in attrs.values()):\n                try:\n                    processor = processor_class(**{k: v[0] for k, v in attrs.items()})\n                except Exception as e:\n                    logger.error(f\"{e.__class__.__name__}: {e}\")\n        else:\n            # `checkpoint` might lack some file(s) to load a processor. For example, `facebook/hubert-base-ls960`\n            # has no tokenizer file to load `Wav2Vec2CTCTokenizer`. In this case, we try to build a processor\n            # with the configuration class (for example, `Wav2Vec2Config`) corresponding to `processor_class`.\n            config_class_from_processor_class = get_config_class_from_processor_class(processor_class)\n            if config_class_from_processor_class != config_class:\n                processor = build_processor(config_class_from_processor_class, processor_class)\n\n    # Try to create an image processor or a feature extractor without any checkpoint\n    if (\n        processor is None\n        and allow_no_checkpoint\n        and (issubclass(processor_class, BaseImageProcessor) or issubclass(processor_class, FeatureExtractionMixin))\n    ):\n        try:\n            processor = processor_class()\n        except Exception as e:\n            logger.error(f\"{e.__class__.__name__}: {e}\")\n\n    # validation\n    if processor is not None:\n        if not (isinstance(processor, processor_class) or processor_class.__name__.startswith(\"Auto\")):\n            raise ValueError(\n                f\"`processor` (which is of type {processor.__class__.__name__}) should be an instance of\"\n                f\" {processor_class.__name__} or an Auto class!\"\n            )\n\n\n\nSUMMARY:\n**Function Summary: `build_processor`**\n\n**Purpose:** Create a processor for the specified `processor_class` using a configuration class from `config_class`.\n\n**Main Functionality:**\n\n1. Use `get_checkpoint_from_config_class` to find a checkpoint in the source file of `config_class`.\n2. If no checkpoint is found, attempt to get one from `processore_class`.\n3. Try to create an instance of `processor_class` from the checkpoint.\n4. If creation fails, use recursive approach to try alternative configurations.\n\n**Key Implementation Details:**\n\n*   Uses `get_config_class_from_processor_class` to map `processor_class` to a configuration class.\n*   Tries to get a new processor class using AutoMapper or by extracting tokenizer classes from the config file.\n*   Creates an image processor or feature extractor without a checkpoint if `allow_no_checkpoint` is set.\n\n**Notable Dependencies and Requirements:**\n\n*   Uses `transformers_module` for loading processor classes.\n*   References related classes/functions like `get_config_class_from_processor_class`, `pretrained`, `logger`.\n*   Assumes the existence of certain configuration files (e.g., `processorer_mapping_names`)."}


created_at: 2025-02-05 09:13:54
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py:375-466
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 466, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py", "name": "get_tiny_config", "start_line": 375, "type": "function", "chroma:document": "CODE:\n\ndef get_tiny_config(config_class, model_class=None, **model_tester_kwargs):\n    \"\"\"Retrieve a tiny configuration from `config_class` using each model's `ModelTester`.\n\n    Args:\n        config_class: Subclass of `PreTrainedConfig`.\n\n    Returns:\n        An instance of `config_class` with tiny hyperparameters\n    \"\"\"\n    model_type = config_class.model_type\n\n    # For model type like `data2vec-vision` and `donut-swin`, we can't get the config/model file name directly via\n    # `model_type` as it would be sth. like `configuration_data2vec_vision.py`.\n    # A simple way is to use `inspect.getsourcefile(config_class)`.\n    config_source_file = inspect.getsourcefile(config_class)\n    # The modeling file name without prefix (`modeling_`) and postfix (`.py`)\n    modeling_name = config_source_file.split(os.path.sep)[-1].replace(\"configuration_\", \"\").replace(\".py\", \"\")\n\n    try:\n        print(\"Importing\", model_type_to_module_name(model_type))\n        module_name = model_type_to_module_name(model_type)\n        if not modeling_name.startswith(module_name):\n            raise ValueError(f\"{modeling_name} doesn't start with {module_name}!\")\n        test_file = os.path.join(\"tests\", \"models\", module_name, f\"test_modeling_{modeling_name}.py\")\n        models_to_model_testers = get_model_to_tester_mapping(test_file)\n        # Find the model tester class\n        model_tester_class = None\n        tester_classes = []\n        if model_class is not None:\n            tester_classes = get_tester_classes_for_model(test_file, model_class)\n        else:\n            for _tester_classes in models_to_model_testers.values():\n                tester_classes.extend(_tester_classes)\n        if len(tester_classes) > 0:\n            # sort with the length of the class names first, then the alphabetical order\n            # This is to avoid `T5EncoderOnlyModelTest` is used instead of `T5ModelTest`, which has\n            # `is_encoder_decoder=False` and causes some pipeline tests failing (also failures in `Optimum` CI).\n            # TODO: More fine grained control of the desired tester class.\n            model_tester_class = sorted(tester_classes, key=lambda x: (len(x.__name__), x.__name__))[0]\n    except ModuleNotFoundError:\n        error = f\"Tiny config not created for {model_type} - cannot find the testing module from the model name.\"\n        raise ValueError(error)\n\n    if model_tester_class is None:\n        error = f\"Tiny config not created for {model_type} - no model tester is found in the testing module.\"\n        raise ValueError(error)\n\n    # CLIP-like models have `text_model_tester` and `vision_model_tester`, and we need to pass `vocab_size` to\n    # `text_model_tester` via `text_kwargs`. The same trick is also necessary for `Flava`.\n\n    if \"vocab_size\" in model_tester_kwargs:\n        if \"text_kwargs\" in inspect.signature(model_tester_class.__init__).parameters.keys():\n            vocab_size = model_tester_kwargs.pop(\"vocab_size\")\n            model_tester_kwargs[\"text_kwargs\"] = {\"vocab_size\": vocab_size}\n\n    # `parent` is an instance of `unittest.TestCase`, but we don't need it here.\n    model_tester = model_tester_class(parent=None, **model_tester_kwargs)\n\n    if hasattr(model_tester, \"get_pipeline_config\"):\n        config = model_tester.get_pipeline_config()\n    elif hasattr(model_tester, \"prepare_config_and_inputs\"):\n        # `PoolFormer` has no `get_config` defined. Furthermore, it's better to use `prepare_config_and_inputs` even if\n        # `get_config` is defined, since there might be some extra changes in `prepare_config_and_inputs`.\n        config = model_tester.prepare_config_and_inputs()[0]\n    elif hasattr(model_tester, \"get_config\"):\n        config = model_tester.get_config()\n    else:\n        error = (\n            f\"Tiny config not created for {model_type} - the model tester {model_tester_class.__name__} lacks\"\n            \" necessary method to create config.\"\n        )\n        raise ValueError(error)\n\n    # make sure this is long enough (some model tester has `20` for this attr.) to pass `text-generation`\n    # pipeline tests.\n    max_positions = []\n    for key in [\"max_position_embeddings\", \"max_source_positions\", \"max_target_positions\"]:\n        if getattr(config, key, 0) > 0:\n            max_positions.append(getattr(config, key))\n        if getattr(config, \"text_config\", None) is not None:\n            if getattr(config.text_config, key, None) is not None:\n                max_positions.append(getattr(config.text_config, key))\n    if len(max_positions) > 0:\n        max_position = max(200, min(max_positions))\n        for key in [\"max_position_embeddings\", \"max_source_positions\", \"max_target_positions\"]:\n            if getattr(config, key, 0) > 0:\n                setattr(config, key, max_position)\n            if getattr(config, \"text_config\", None) is not None:\n                if getattr(config.text_config, key, None) is not None:\n                    setattr(config.text_config, key, max_position)\n\n\n\nSUMMARY:\n**Summary:**\n\n`get_tiny_config` function retrieves a tiny configuration from `config_class` using each model's `ModelTester`. \n\n**Main Purpose and Functionality:**\n\n* The function takes a `config_class` (subclass of `PreTrainedConfig`) as input.\n* It retrieves the corresponding `model_tester_class` from the testing module based on the `model_type`.\n* It creates an instance of the `model_tester_class` with updated arguments to set `vocab_size`.\n* It creates a configuration using the model tester's `get_pipeline_config`, `prepare_config_and_inputs`, or `get_config` method.\n* If necessary, it sets max positions in the configuration.\n\n**Key Implementation Details:**\n\n* Uses dynamic imports and path manipulation to retrieve the testing module based on `model_type`.\n* Sets vocab size for models with a language tokenizer.\n* Uses various methods from `model_tester_class` (e.g., `get_pipeline_config`, `prepare_config_and_inputs`) to create the configuration.\n\n**Notable Dependencies or Requirements:**\n\n* `config_class` (subclass of `PreTrainedConfig`)\n* `model_type` (determined by `config_class.model_type`)\n* Various model testers and corresponding modules\n* Testing framework (e.g., `unittest`)"}


created_at: 2025-02-05 09:13:54
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py:469-478
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 478, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py", "name": "convert_tokenizer", "start_line": 469, "type": "function", "chroma:document": "CODE:\n\ndef convert_tokenizer(tokenizer_fast: PreTrainedTokenizerFast):\n    new_tokenizer = tokenizer_fast.train_new_from_iterator(\n        data[\"training_ds\"][\"text\"], TARGET_VOCAB_SIZE, show_progress=False\n    )\n\n    # Make sure it at least runs\n    if not isinstance(new_tokenizer, LayoutLMv3TokenizerFast):\n        new_tokenizer(data[\"testing_ds\"][\"text\"])\n\n\n\nSUMMARY:\n**Concise Summary**\n\nThis function creates or updates a `LayoutLMv3TokenizerFast` instance using the provided `PreTrainedTokenizerFast`. It:\n\n* Trains a new tokenizer from the training data with a specified vocabulary size (`TARGET_VOCAB_SIZE`)\n* Checks if the created tokenizer is of type `LayoutLMv3TokenizerFast`, and if not, uses the testing data to create an alternative tokenizer\n* Returns the newly created or updated `LayoutLMv3TokenizerFast` instance.\n\n**Details:**\n\n* Main purpose: tokenization for a specific text dataset using `LayoutLMv3TokenizerFast`.\n* Key implementation details:\n\t+ Tokenizer training and update.\n\t+ Type checking for `LayoutLMv3TokenizerFast`.\n* Notable dependencies:\n\t+ `PreTrainedTokenizerFast` from Hugging Face library.\n\t+ `data[\"training_ds\"][\"text\"]` and `data[\"testing_ds\"][\"text\"]` datasets."}


created_at: 2025-02-05 09:13:54
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py:481-527
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 527, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py", "name": "convert_feature_extractor", "start_line": 481, "type": "function", "chroma:document": "CODE:\n\ndef convert_feature_extractor(feature_extractor, tiny_config):\n    to_convert = False\n    kwargs = {}\n    if hasattr(tiny_config, \"image_size\"):\n        kwargs[\"size\"] = tiny_config.image_size\n        kwargs[\"crop_size\"] = tiny_config.image_size\n        to_convert = True\n    elif (\n        hasattr(tiny_config, \"vision_config\")\n        and tiny_config.vision_config is not None\n        and hasattr(tiny_config.vision_config, \"image_size\")\n    ):\n        kwargs[\"size\"] = tiny_config.vision_config.image_size\n        kwargs[\"crop_size\"] = tiny_config.vision_config.image_size\n        to_convert = True\n\n    # Speech2TextModel specific.\n    if hasattr(tiny_config, \"input_feat_per_channel\"):\n        kwargs[\"feature_size\"] = tiny_config.input_feat_per_channel\n        kwargs[\"num_mel_bins\"] = tiny_config.input_feat_per_channel\n        to_convert = True\n\n    if to_convert:\n        feature_extractor = feature_extractor.__class__(**kwargs)\n\n    # Sanity check: on tiny image feature extractors, a large image size results in slow CI -- up to the point where it\n    # can result in timeout issues.\n    if (\n        isinstance(feature_extractor, BaseImageProcessor)\n        and hasattr(feature_extractor, \"size\")\n        and isinstance(feature_extractor.size, dict)\n    ):\n        largest_image_size = max(feature_extractor.size.values())\n        if largest_image_size > 64:\n            # hardcoded exceptions\n            models_with_large_image_size = (\"deformable_detr\", \"flava\", \"grounding_dino\", \"mgp_str\", \"swiftformer\")\n            if any(model_name in tiny_config.model_type for model_name in models_with_large_image_size):\n                pass\n            else:\n                raise ValueError(\n                    f\"Image size of {tiny_config.model_type} is too large ({feature_extractor.size}). \"\n                    \"Please reduce it to 64 or less on each dimension. The following steps are usually the \"\n                    \"easiest solution: 1) confirm that you're setting `image_size` in your ModelTester class; \"\n                    \"2) ensure that it gets passed to the tester config init, `get_config()`.\"\n                )\n\n\n\nSUMMARY:\n**Function Summary: Convert Feature Extractor**\n\nConverts a `feature_extractor` instance to a new configuration based on `tiny_config`.\n\n**Main Purpose and Functionality:**\nThe function `convert_feature_extractor` takes two parameters, `feature_extractor` and `tiny_config`, and converts the former to fit the latter's specifications. It checks for various attributes in `tiny_config` (e.g., `image_size`, `vision_config.image_size`) and updates the `kwargs` dictionary accordingly.\n\n**Key Implementation Details:**\n\n* Updates `kwargs[\"size\"]` and `kwargs[\"crop_size\"]` based on `image_size` or `vision_config.image_size`\n* Sets `feature_size` and `num_mel_bins` for `Speech2TextModel`\n* Checks for large image sizes in BaseImageProcessor instances (up to 64 pixels)\n* Raises a ValueError if an incompatible model type is used with too large an image size\n\n**Notable Dependencies/Requirements:**\n\n* `BaseImageProcessor`, `tiny_config`, and related classes/functions are imported but not listed explicitly\n* External dependencies on ModelTester, get_config(), ImageSize, VisionConfig are implied but not specified"}


created_at: 2025-02-05 09:13:54
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py:530-757
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 757, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py", "name": "convert_processors", "start_line": 530, "type": "function", "chroma:document": "CODE:\n\ndef convert_processors(processors, tiny_config, output_folder, result):\n    \"\"\"Change a processor to work with smaller inputs.\n\n    For tokenizers, we try to reduce their vocabulary size.\n\n    For feature extractor, we use smaller image size or change\n    other attributes using the values from `tiny_config`. See `convert_feature_extractor`.\n\n    This method should not fail: we catch the errors and put them in `result[\"warnings\"]` with descriptive messages.\n    \"\"\"\n\n    def _sanity_check(fast_tokenizer, slow_tokenizer, keep_fast_tokenizer=False):\n        \"\"\"Set tokenizer(s) to `None` if the fast/slow tokenizers have different values for `vocab_size` or `length`.\n\n        If `keep_fast_tokenizer=True`, the fast tokenizer will be kept.\n        \"\"\"\n        # sanity check 1: fast and slow tokenizers should be compatible (vocab_size)\n        if fast_tokenizer is not None and slow_tokenizer is not None:\n            if fast_tokenizer.vocab_size != slow_tokenizer.vocab_size:\n                warning_message = (\n                    \"The fast/slow tokenizers \"\n                    f\"({fast_tokenizer.__class__.__name__}/{slow_tokenizer.__class__.__name__}) have different \"\n                    \"vocabulary size: \"\n                    f\"fast_tokenizer.vocab_size = {fast_tokenizer.vocab_size} and \"\n                    f\"slow_tokenizer.vocab_size = {slow_tokenizer.vocab_size}.\"\n                )\n                result[\"warnings\"].append(warning_message)\n                if not keep_fast_tokenizer:\n                    fast_tokenizer = None\n                slow_tokenizer = None\n\n        # sanity check 2: fast and slow tokenizers should be compatible (length)\n        if fast_tokenizer is not None and slow_tokenizer is not None:\n            if len(fast_tokenizer) != len(slow_tokenizer):\n                warning_message = (\n                    f\"The fast/slow tokenizers () have different length: \"\n                    f\"len(fast_tokenizer) = {len(fast_tokenizer)} and \"\n                    f\"len(slow_tokenizer) = {len(slow_tokenizer)}.\"\n                )\n                result[\"warnings\"].append(warning_message)\n                if not keep_fast_tokenizer:\n                    fast_tokenizer = None\n                slow_tokenizer = None\n\n        return fast_tokenizer, slow_tokenizer\n\n    tokenizers = []\n    feature_extractors = []\n    for processor in processors:\n        if isinstance(processor, PreTrainedTokenizerBase):\n            if processor.__class__.__name__ not in {x.__class__.__name__ for x in tokenizers}:\n                tokenizers.append(processor)\n        elif isinstance(processor, BaseImageProcessor):\n            if processor.__class__.__name__ not in {x.__class__.__name__ for x in feature_extractors}:\n                feature_extractors.append(processor)\n        elif isinstance(processor, FeatureExtractionMixin):\n            if processor.__class__.__name__ not in {x.__class__.__name__ for x in feature_extractors}:\n                feature_extractors.append(processor)\n        elif isinstance(processor, ProcessorMixin):\n            if hasattr(processor, \"tokenizer\"):\n                if processor.tokenizer.__class__.__name__ not in {x.__class__.__name__ for x in tokenizers}:\n                    tokenizers.append(processor.tokenizer)\n            # Currently, we only have these 2 possibilities\n            if hasattr(processor, \"image_processor\"):\n                if processor.image_processor.__class__.__name__ not in {\n                    x.__class__.__name__ for x in feature_extractors\n                }:\n                    feature_extractors.append(processor.image_processor)\n            elif hasattr(processor, \"feature_extractor\"):\n                if processor.feature_extractor.__class__.__name__ not in {\n                    x.__class__.__name__ for x in feature_extractors\n                }:\n                    feature_extractors.append(processor.feature_extractor)\n\n    # check the built processors have the unique type\n    num_types = len({x.__class__.__name__ for x in feature_extractors})\n    if num_types >= 2:\n        raise ValueError(f\"`feature_extractors` should contain at most 1 type, but it contains {num_types} types!\")\n    num_types = len({x.__class__.__name__.replace(\"Fast\", \"\") for x in tokenizers})\n    if num_types >= 2:\n        raise ValueError(f\"`tokenizers` should contain at most 1 tokenizer type, but it contains {num_types} types!\")\n\n    fast_tokenizer = None\n    slow_tokenizer = None\n\n    for tokenizer in tokenizers:\n        if isinstance(tokenizer, PreTrainedTokenizerFast):\n            fast_tokenizer = tokenizer\n        else:\n            slow_tokenizer = tokenizer\n\n    # If the (original) fast/slow tokenizers don't correspond, keep only the fast tokenizer.\n    # This doesn't necessarily imply the fast/slow tokenizers in a single Hub repo. has issues.\n    # It's more of an issue in `build_processor` which tries to get a checkpoint with as much effort as possible.\n    # For `YosoModel` (which uses `AlbertTokenizer(Fast)`), its real (Hub) checkpoint doesn't contain valid files to\n    # load the slower tokenizer (`AlbertTokenizer`), and it ends up finding the (canonical) checkpoint of `AlbertModel`,\n    # which has different vocabulary.\n    # TODO: Try to improve `build_processor`'s definition and/or usage to avoid the above situation in the first place.\n    fast_tokenizer, slow_tokenizer = _sanity_check(fast_tokenizer, slow_tokenizer, keep_fast_tokenizer=True)\n    original_fast_tokenizer, original_slow_tokenizer = fast_tokenizer, slow_tokenizer\n\n    if fast_tokenizer:\n        try:\n            # Wav2Vec2ForCTC , ByT5Tokenizer etc. all are already small enough and have no fast version that can\n            # be retrained\n            if fast_tokenizer.vocab_size > TARGET_VOCAB_SIZE:\n                fast_tokenizer = convert_tokenizer(fast_tokenizer)\n        except Exception:\n            result[\"warnings\"].append(\n                (\n                    f\"Failed to convert the fast tokenizer for {fast_tokenizer.__class__.__name__}.\",\n                    traceback.format_exc(),\n                )\n            )\n\n    # If `fast_tokenizer` exists, `slow_tokenizer` should correspond to it.\n    if fast_tokenizer:\n        # Make sure the fast tokenizer can be saved\n        try:\n            # We don't save it to `output_folder` at this moment - only at the end of this function.\n            with tempfile.TemporaryDirectory() as tmpdir:\n                fast_tokenizer.save_pretrained(tmpdir)\n                try:\n                    slow_tokenizer = AutoTokenizer.from_pretrained(tmpdir, use_fast=False)\n                except Exception:\n                    result[\"warnings\"].append(\n                        (\n                            f\"Failed to load the slow tokenizer saved from {fast_tokenizer.__class__.__name__}.\",\n                            traceback.format_exc(),\n                        )\n                    )\n                    # Let's just keep the fast version\n                    slow_tokenizer = None\n        except Exception:\n            result[\"warnings\"].append(\n                (\n                    f\"Failed to save the fast tokenizer for {fast_tokenizer.__class__.__name__}.\",\n                    traceback.format_exc(),\n                )\n            )\n            fast_tokenizer = None\n\n    # If the (possibly converted) fast/slow tokenizers don't correspond, set them to `None`, and use the original\n    # tokenizers.\n    fast_tokenizer, slow_tokenizer = _sanity_check(fast_tokenizer, slow_tokenizer, keep_fast_tokenizer=False)\n\n    # If there is any conversion failed, we keep the original tokenizers.\n    if (original_fast_tokenizer is not None and fast_tokenizer is None) or (\n        original_slow_tokenizer is not None and slow_tokenizer is None\n    ):\n        warning_messagae = (\n            \"There are some issues when converting the fast/slow tokenizers. The original tokenizers from the Hub \"\n            \" will be used instead.\"\n        )\n        result[\"warnings\"].append(warning_messagae)\n        # Let's use the original version at the end (`original_fast_tokenizer` and `original_slow_tokenizer`)\n        fast_tokenizer = original_fast_tokenizer\n        slow_tokenizer = original_slow_tokenizer\n\n    # Make sure the fast tokenizer can be saved\n    if fast_tokenizer:\n        # We don't save it to `output_folder` at this moment - only at the end of this function.\n        with tempfile.TemporaryDirectory() as tmpdir:\n            try:\n                fast_tokenizer.save_pretrained(tmpdir)\n            except Exception:\n                result[\"warnings\"].append(\n                    (\n                        f\"Failed to save the fast tokenizer for {fast_tokenizer.__class__.__name__}.\",\n                        traceback.format_exc(),\n                    )\n                )\n                fast_tokenizer = None\n    # Make sure the slow tokenizer can be saved\n    if slow_tokenizer:\n        # We don't save it to `output_folder` at this moment - only at the end of this function.\n        with tempfile.TemporaryDirectory() as tmpdir:\n            try:\n                slow_tokenizer.save_pretrained(tmpdir)\n            except Exception:\n                result[\"warnings\"].append(\n                    (\n                        f\"Failed to save the slow tokenizer for {slow_tokenizer.__class__.__name__}.\",\n                        traceback.format_exc(),\n                    )\n                )\n                slow_tokenizer = None\n\n    # update feature extractors using the tiny config\n    try:\n        feature_extractors = [convert_feature_extractor(p, tiny_config) for p in feature_extractors]\n    except Exception:\n        result[\"warnings\"].append(\n            (\n                \"Failed to convert feature extractors.\",\n                traceback.format_exc(),\n            )\n        )\n        feature_extractors = []\n\n    if hasattr(tiny_config, \"max_position_embeddings\") and tiny_config.max_position_embeddings > 0:\n        if fast_tokenizer is not None:\n            if fast_tokenizer.__class__.__name__ in [\n                \"RobertaTokenizerFast\",\n                \"XLMRobertaTokenizerFast\",\n                \"LongformerTokenizerFast\",\n                \"MPNetTokenizerFast\",\n            ]:\n                fast_tokenizer.model_max_length = tiny_config.max_position_embeddings - 2\n            else:\n                fast_tokenizer.model_max_length = tiny_config.max_position_embeddings\n        if slow_tokenizer is not None:\n            if slow_tokenizer.__class__.__name__ in [\n                \"RobertaTokenizer\",\n                \"XLMRobertaTokenizer\",\n                \"LongformerTokenizer\",\n                \"MPNetTokenizer\",\n            ]:\n                slow_tokenizer.model_max_length = tiny_config.max_position_embeddings - 2\n            else:\n                slow_tokenizer.model_max_length = tiny_config.max_position_embeddings\n\n    processors = [fast_tokenizer, slow_tokenizer] + feature_extractors\n    processors = [p for p in processors if p is not None]\n    for p in processors:\n        p.save_pretrained(output_folder)\n\n\n\nSUMMARY:\nHere's a concise summary:\n\nThe code creates and saves a model with modified tokenizers (fast and slow) when using the Hugging Face library. It performs the following tasks:\n\n1. Checks available tokenizers on the Hugging Face Hub.\n2. Converts fast and slow tokenizers to smaller versions based on a provided `tiny_config`.\n3. Verifies that the converted tokenizers correspond to each other.\n4. Saves modified feature extractors (if applicable) using the tiny config.\n5. Calls back the modified processors to ensure they can be saved in their updated form.\n\nThe code maintains the functionality of original tokenizers and saves only modified versions, if conversions are successful."}


created_at: 2025-02-05 09:13:54
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py:760-769
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 769, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py", "name": "get_checkpoint_dir", "start_line": 760, "type": "function", "chroma:document": "CODE:\n\ndef get_checkpoint_dir(output_dir, model_arch):\n    \"\"\"Get framework-agnostic architecture name. Used to save all PT/TF/Flax models into the same directory.\"\"\"\n\n    arch_name = model_arch.__name__\n    if arch_name.startswith(\"TF\"):\n        arch_name = arch_name[2:]\n    elif arch_name.startswith(\"Flax\"):\n        arch_name = arch_name[4:]\n\n\n\nSUMMARY:\n**Summary**\n\nThe `get_checkpoint_dir` function creates a framework-agnostic architecture name by removing the framework prefix (\"TF\" or \"Flax\") from the model architecture. It then joins this modified name with the provided output directory using `os.path.join`. \n\n**Key Details:**\n\n* Function purpose: Returns a framework-agnostic checkpoint directory.\n* Implementation:\n\t+ Removes framework prefix from model architecture (`arch_name`).\n\t+ Uses `os.path.join` to combine filename and directory.\n\n**Dependency:**\n\n* Imports an externally-used module: `os`.\n* Assumes the existence of a string object containing model architecture information.\n* Relies on string manipulation for framework-agnostic namingConvention."}


created_at: 2025-02-05 09:13:54
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py:772-795
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 795, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py", "name": "build_model", "start_line": 772, "type": "function", "chroma:document": "CODE:\n\ndef build_model(model_arch, tiny_config, output_dir):\n    \"\"\"Create and save a model for `model_arch`.\n\n    Also copy the set of processors to each model (under the same model type) output folder.\n    \"\"\"\n\n    checkpoint_dir = get_checkpoint_dir(output_dir, model_arch)\n\n    processor_output_dir = os.path.join(output_dir, \"processors\")\n    # copy the (same set of) processors (for a model type) to the model arch. specific folder\n    if os.path.isdir(processor_output_dir):\n        shutil.copytree(processor_output_dir, checkpoint_dir, dirs_exist_ok=True)\n\n    tiny_config = copy.deepcopy(tiny_config)\n\n    if any(model_arch.__name__.endswith(x) for x in [\"ForCausalLM\", \"LMHeadModel\"]):\n        tiny_config.is_encoder_decoder = False\n        tiny_config.is_decoder = True\n\n    model = model_arch(config=tiny_config)\n    model.save_pretrained(checkpoint_dir)\n    model.from_pretrained(checkpoint_dir)\n\n\n\nSUMMARY:\nHere is a concise and focused code summary:\n\n**Functionality:**\n\n* Creates and saves a model for the specified `model_arch` using a custom configuration (`tiny_config`) and output directory (`output_dir`)\n* Copies processor files to the same folder as the checkpoint directory if it exists\n* Saves and loads the pre-trained model with the specified `checkpoint_dir`\n\n**Key Implementation Details:**\n\n* Uses `shutil.copytree` to copy processors, allowing for reuse across models of the same type\n* Applies settings from `tiny_config` based on the model architecture name (`model_arch.__name__`)\n\n**Notable Dependencies and Requirements:**\n\n* Hugged library (not explicitly listed, but implied by usage)\n* os library (for file system operations)\n* shutil library (used for copying files)"}


created_at: 2025-02-05 09:13:54
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py:798-808
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 808, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py", "name": "fill_result_with_error", "start_line": 798, "type": "function", "chroma:document": "CODE:\n\ndef fill_result_with_error(result, error, trace, models_to_create):\n    \"\"\"Fill `result` with errors for all target model arch if we can't build processor\"\"\"\n    error = (error, trace)\n    result[\"error\"] = error\n    for framework in FRAMEWORKS:\n        if framework in models_to_create:\n            result[framework] = {}\n            for model_arch in models_to_create[framework]:\n                result[framework][model_arch.__name__] = {\"model\": None, \"checkpoint\": None, \"error\": error}\n\n\n\nSUMMARY:\n**Summary:**\n\n`fill_result_with_error` function:\n```python\ndef fill_result_with_error(result, error, trace, models_to_create):\n    \"\"\"\n    Fill `result` with errors for various framework models and processor.\n\n    Parameters:\n    result (dict): Target dictionary to be filled with errors.\n    error (tuple): Error message and traceback.\n    trace (traceback object): Debugging information.\n    models_to_create (dict): Models to create for each framework.\n    \"\"\"\n```\n\n**Functionality:**\n\n1. Fills `result` dictionary with errors for all target model architectures (model_arch in `models_to_create`) if the processor cannot be built.\n2. Initializes processor configuration with only class names for each component.\n\n**Key Implementation Details:**\n\n* Iterates through `FRAMEWORKS` and updates corresponding framework models in `result`.\n* Uses nested dictionaries to store model configurations for each architecture.\n\n**Notable Dependencies or Requirements:**\n\n* `FRAMEWORKS` (list of strings)\n* `models_to_create` (dictionary with framework models)\n* `traceback` object (`trace`)\n* Processor components with their class names\n\nThis summary aims to provide a concise overview of the function's purpose, key implementation details, and any notable dependencies or requirements."}


created_at: 2025-02-05 09:13:54
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py:811-858
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 858, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py", "name": "upload_model", "start_line": 811, "type": "function", "chroma:document": "CODE:\n\ndef upload_model(model_dir, organization, token):\n    \"\"\"Upload the tiny models\"\"\"\n\n    arch_name = model_dir.split(os.path.sep)[-1]\n    repo_name = f\"tiny-random-{arch_name}\"\n    repo_id = f\"{organization}/{repo_name}\"\n\n    repo_exist = False\n    error = None\n    try:\n        create_repo(repo_id=repo_id, exist_ok=False, repo_type=\"model\", token=token)\n    except Exception as e:\n        error = e\n        if \"You already created\" in str(e):\n            error = None\n            logger.warning(\"Remote repository exists and will be cloned.\")\n            repo_exist = True\n            try:\n                create_repo(repo_id=repo_id, exist_ok=True, repo_type=\"model\", token=token)\n            except Exception as e:\n                error = e\n    if error is not None:\n        raise error\n\n    with tempfile.TemporaryDirectory() as tmpdir:\n        repo = Repository(local_dir=tmpdir, clone_from=repo_id, token=token)\n        repo.git_pull()\n        shutil.copytree(model_dir, tmpdir, dirs_exist_ok=True)\n\n        if repo_exist:\n            # Open a PR on the existing Hub repo.\n            hub_pr_url = upload_folder(\n                folder_path=model_dir,\n                repo_id=repo_id,\n                repo_type=\"model\",\n                commit_message=f\"Update tiny models for {arch_name}\",\n                commit_description=f\"Upload tiny models for {arch_name}\",\n                create_pr=True,\n                token=token,\n            )\n            logger.warning(f\"PR open in {hub_pr_url}.\")\n            # TODO: We need this information?\n        else:\n            # Push to Hub repo directly\n            repo.git_add(auto_lfs_track=True)\n            repo.git_commit(f\"Upload tiny models for {arch_name}\")\n            repo.git_push(blocking=True)  # this prints a progress bar with the upload\n\n\nSUMMARY:\n**Summary:**\n\nThis function, `upload_model`, uploads a small model library (`model_dir`) to a remote repository on Hub. The primary purpose is to:\n\n1. Create or clone an existing repository in Hub (`create_repo` method) if it doesn't exist or already exists and needs to be updated.\n2. Import the models from the local directory and update the repository with the latest changes.\n\n**Key Implementation Details:**\n\n* Creates a temporary directory and clones the remote repository using `Repository.git_pull()`.\n* Copies the model directory into the temporary directory.\n* If the repository exists, opens a Pull Request (PR) on Hub (`upload_folder` method).\n* If not, directly pushes the updated models to the Hub repository.\n\n**Notable Dependencies or Requirements:**\n\n* `Hub` API (Remote Repository creation/cloning and PR creation)\n* `Repository` class for interacting with the remote repository\n* `TemporaryDirectory` from `tempfile` module for creating a temporary directory\n* `shutil` for copying files and directories\n* `os.path.sep` for splitting the model directory path\n\n**Method Signatures and Interfaces:**\n\n* `upload_model(model_dir, organization, token)`\n* `create_repo(repo_id, exist_ok=False, repo_type=\"model\", token=token)`\n* `Repository(local_dir=tmpdir, clone_from=repo_id, token=token)`\n* `upload_folder(folder_path=model_dir, repo_id=repo_id, repo_type=\"model\", ...)`\n* `logger.warning` and similar logging functions for output"}


created_at: 2025-02-05 09:13:54
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py:861-999
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 999, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py", "name": "build_composite_models", "start_line": 861, "type": "function", "chroma:document": "CODE:\n\ndef build_composite_models(config_class, output_dir):\n    import tempfile\n\n    from transformers import (\n        BertConfig,\n        BertLMHeadModel,\n        BertModel,\n        BertTokenizer,\n        BertTokenizerFast,\n        EncoderDecoderModel,\n        GPT2Config,\n        GPT2LMHeadModel,\n        GPT2Tokenizer,\n        GPT2TokenizerFast,\n        SpeechEncoderDecoderModel,\n        TFEncoderDecoderModel,\n        TFVisionEncoderDecoderModel,\n        TFVisionTextDualEncoderModel,\n        VisionEncoderDecoderModel,\n        VisionTextDualEncoderModel,\n        ViTConfig,\n        ViTFeatureExtractor,\n        ViTModel,\n        Wav2Vec2Config,\n        Wav2Vec2Model,\n        Wav2Vec2Processor,\n    )\n\n    # These will be removed at the end if they are empty\n    result = {\"error\": None, \"warnings\": []}\n\n    if config_class.model_type == \"encoder-decoder\":\n        encoder_config_class = BertConfig\n        decoder_config_class = BertConfig\n        encoder_processor = (BertTokenizerFast, BertTokenizer)\n        decoder_processor = (BertTokenizerFast, BertTokenizer)\n        encoder_class = BertModel\n        decoder_class = BertLMHeadModel\n        model_class = EncoderDecoderModel\n        tf_model_class = TFEncoderDecoderModel\n    elif config_class.model_type == \"vision-encoder-decoder\":\n        encoder_config_class = ViTConfig\n        decoder_config_class = GPT2Config\n        encoder_processor = (ViTFeatureExtractor,)\n        decoder_processor = (GPT2TokenizerFast, GPT2Tokenizer)\n        encoder_class = ViTModel\n        decoder_class = GPT2LMHeadModel\n        model_class = VisionEncoderDecoderModel\n        tf_model_class = TFVisionEncoderDecoderModel\n    elif config_class.model_type == \"speech-encoder-decoder\":\n        encoder_config_class = Wav2Vec2Config\n        decoder_config_class = BertConfig\n        encoder_processor = (Wav2Vec2Processor,)\n        decoder_processor = (BertTokenizerFast, BertTokenizer)\n        encoder_class = Wav2Vec2Model\n        decoder_class = BertLMHeadModel\n        model_class = SpeechEncoderDecoderModel\n        tf_model_class = None\n    elif config_class.model_type == \"vision-text-dual-encoder\":\n        # Not encoder-decoder, but encoder-encoder. We just keep the same name as above to make code easier\n        encoder_config_class = ViTConfig\n        decoder_config_class = BertConfig\n        encoder_processor = (ViTFeatureExtractor,)\n        decoder_processor = (BertTokenizerFast, BertTokenizer)\n        encoder_class = ViTModel\n        decoder_class = BertModel\n        model_class = VisionTextDualEncoderModel\n        tf_model_class = TFVisionTextDualEncoderModel\n\n    with tempfile.TemporaryDirectory() as tmpdir:\n        try:\n            # build encoder\n            models_to_create = {\"processor\": encoder_processor, \"pytorch\": (encoder_class,), \"tensorflow\": []}\n            encoder_output_dir = os.path.join(tmpdir, \"encoder\")\n            build(encoder_config_class, models_to_create, encoder_output_dir)\n\n            # build decoder\n            models_to_create = {\"processor\": decoder_processor, \"pytorch\": (decoder_class,), \"tensorflow\": []}\n            decoder_output_dir = os.path.join(tmpdir, \"decoder\")\n            build(decoder_config_class, models_to_create, decoder_output_dir)\n\n            # build encoder-decoder\n            encoder_path = os.path.join(encoder_output_dir, encoder_class.__name__)\n            decoder_path = os.path.join(decoder_output_dir, decoder_class.__name__)\n\n            if config_class.model_type != \"vision-text-dual-encoder\":\n                # Specify these explicitly for encoder-decoder like models, but not for `vision-text-dual-encoder` as it\n                # has no decoder.\n                decoder_config = decoder_config_class.from_pretrained(decoder_path)\n                decoder_config.is_decoder = True\n                decoder_config.add_cross_attention = True\n                model = model_class.from_encoder_decoder_pretrained(\n                    encoder_path,\n                    decoder_path,\n                    decoder_config=decoder_config,\n                )\n            elif config_class.model_type == \"vision-text-dual-encoder\":\n                model = model_class.from_vision_text_pretrained(encoder_path, decoder_path)\n\n            model_path = os.path.join(\n                output_dir,\n                f\"{model_class.__name__}-{encoder_config_class.model_type}-{decoder_config_class.model_type}\",\n            )\n            model.save_pretrained(model_path)\n\n            if tf_model_class is not None:\n                model = tf_model_class.from_pretrained(model_path)\n                model.save_pretrained(model_path)\n\n            # copy the processors\n            encoder_processor_path = os.path.join(encoder_output_dir, \"processors\")\n            decoder_processor_path = os.path.join(decoder_output_dir, \"processors\")\n            if os.path.isdir(encoder_processor_path):\n                shutil.copytree(encoder_processor_path, model_path, dirs_exist_ok=True)\n            if os.path.isdir(decoder_processor_path):\n                shutil.copytree(decoder_processor_path, model_path, dirs_exist_ok=True)\n\n            # fill `result`\n            result[\"processor\"] = {x.__name__: x.__name__ for x in encoder_processor + decoder_processor}\n\n            result[\"pytorch\"] = {model_class.__name__: {\"model\": model_class.__name__, \"checkpoint\": model_path}}\n\n            result[\"tensorflow\"] = {}\n            if tf_model_class is not None:\n                result[\"tensorflow\"] = {\n                    tf_model_class.__name__: {\"model\": tf_model_class.__name__, \"checkpoint\": model_path}\n                }\n        except Exception:\n            result[\"error\"] = (\n                f\"Failed to build models for {config_class.__name__}.\",\n                traceback.format_exc(),\n            )\n\n    if not result[\"error\"]:\n        del result[\"error\"]\n    if not result[\"warnings\"]:\n        del result[\"warnings\"]\n\n\n\nSUMMARY:\n**Summary**\n\nThe `build_composite_models` function builds composite models from various pre-trained transformer architectures. Its primary purpose is to create a full-fledged encoder-decoder or vision-text-dual-encoder model from its input configuration, pre-trained models, and output directory.\n\n**Key Implementation Details:**\n\n*   The function relies on the Hugging Face Transformers library.\n*   It handles three types of composite models:\n    *   Encoder-decoder (`config_class.model_type == \"encoder-decoder\"`): Uses PyTorch.\n    *   Vision-encoder-decoder (`config_class.model_type == \"vision-encoder-decoder\"`): Uses PyTorch.\n    *   Speech-encoder-decoder (`config_class.model_type == \"speech-encoder-decoder\"`): Supports both PyTorch and TensorFlow.\n    *   Vision-text-dual-encoder (`config_class.model_type == \"vision-text-dual-encoder\"`): Only uses PyTorch.\n*   Pre-trained models are loaded, configured, and combined to form the final composite model using the `model_from_pretrained`, `model_from_encoder_decoder`, or `model_from_vision_text` methods.\n\n**Notable Dependencies:**\n\n*   The `transformers` library from Hugging Face.\n*   Temporary directories for storing pre-built models.\n*   External functions and classes (e.g., `build`, `os.path.join`, `shutil.copytree`) from Python's standard library or other dependencies not explicitly mentioned here.\n\n**Model Output:**\n\nThe function generates a directory with a composite model, along with various output files, including:\n        *   Model checkpoint\n        *   PyTorch models and checkpoints (when supported)\n        *   TensorFlow models and checkpoints (if specified)\n        *   PyTorch processors\n\n**Interfaces:**\n\n*   `build`: An external function for building pre-trained models.\n*   `TransformerModel` interface from the Hugging Face Transformers library.\n\n```python\ndef build_composite_models(config_class, output_dir):\n    # ... [function implementation]\n    \n    return result\n```\n\nThis concise summary should provide essential information for developers to understand this code snippet and replicate its functionality in their own projects."}


created_at: 2025-02-05 09:13:55
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py:1002-1021
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 1021, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py", "name": "get_token_id_from_tokenizer", "start_line": 1002, "type": "function", "chroma:document": "CODE:\n\ndef get_token_id_from_tokenizer(token_id_name, tokenizer, original_token_id):\n    \"\"\"Use `tokenizer` to get the values of `bos_token_id`, `eos_token_ids`, etc.\n\n    The argument `token_id_name` should be a string ending with `_token_id`, and `original_token_id` should be an\n    integer that will be return if `tokenizer` has no token corresponding to `token_id_name`.\n    \"\"\"\n\n    token_id = original_token_id\n\n    if not token_id_name.endswith(\"_token_id\"):\n        raise ValueError(f\"`token_id_name` is {token_id_name}, which doesn't end with `_token_id`!\")\n\n    token = getattr(tokenizer, token_id_name.replace(\"_token_id\", \"_token\"), None)\n    if token is not None:\n        if isinstance(tokenizer, PreTrainedTokenizerFast):\n            token_id = tokenizer._convert_token_to_id_with_added_voc(token)\n        else:\n            token_id = tokenizer._convert_token_to_id(token)\n\n\n\nSUMMARY:\n**Summary**\n\n`get_token_id_from_tokenizer` function retrieves the ID of a specific token from a `tokenizer` object. It:\n\n* Takes in `token_id_name` (string) and `original_token_id` (integer)\n* Initializes `token_id` to `original_token_id`\n* Raises ValueError if `token_id_name` does not end with `_token_id`\n* Uses `tokenizer` to retrieve the token ID, optionally converting it with added vocabulary\n\n**Key Implementation Details**\n\n* Replaces `_token` in `token_id_name` with `_token` from `tokenizer` for attribute access\n* Checks if `tokenizer` is a `PreTrainedTokenizerFast` and applies conversion using its `_convert_token_to_id_with_added_voc` method\n\n**Notable Dependencies or Requirements**\n\n* External dependencies:\n\t+ `transformers` library (specifically, `PreTrainedTokenizerFast`)\n* Related classes/functions:\n\t+ ` tokenizer._convert_token_to_id`\n\t+ `tokenizer._convert_token_to_id_with_added_voc`"}


created_at: 2025-02-05 09:13:55
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py:1024-1086
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 1086, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py", "name": "get_config_overrides", "start_line": 1024, "type": "function", "chroma:document": "CODE:\n\ndef get_config_overrides(config_class, processors):\n    # `Bark` configuration is too special. Let's just not handle this for now.\n    if config_class.__name__ == \"BarkConfig\":\n        return {}\n\n    config_overrides = {}\n\n    # Check if there is any tokenizer (prefer fast version if any)\n    tokenizer = None\n    for processor in processors:\n        if isinstance(processor, PreTrainedTokenizerFast):\n            tokenizer = processor\n            break\n        elif isinstance(processor, PreTrainedTokenizer):\n            tokenizer = processor\n\n    if tokenizer is None:\n        return config_overrides\n\n    # Get some properties of the (already converted) tokenizer (smaller vocab size, special token ids, etc.)\n    # We use `len(tokenizer)` instead of `tokenizer.vocab_size` to avoid potential issues for tokenizers with non-empty\n    # `added_tokens_encoder`. One example is the `DebertaV2Tokenizer` where the mask token is the extra token.\n    vocab_size = len(tokenizer)\n\n    # The original checkpoint has length `35998`, but it doesn't have ids `30400` and `30514` but instead `35998` and\n    # `35999`.\n    if config_class.__name__ == \"GPTSanJapaneseConfig\":\n        vocab_size += 2\n\n    config_overrides[\"vocab_size\"] = vocab_size\n\n    # Used to create a new model tester with `tokenizer.vocab_size` in order to get the (updated) special token ids.\n    model_tester_kwargs = {\"vocab_size\": vocab_size}\n    # `FSMTModelTester` accepts `src_vocab_size` and `tgt_vocab_size` but not `vocab_size`.\n    if config_class.__name__ == \"FSMTConfig\":\n        del model_tester_kwargs[\"vocab_size\"]\n        model_tester_kwargs[\"src_vocab_size\"] = tokenizer.src_vocab_size\n        model_tester_kwargs[\"tgt_vocab_size\"] = tokenizer.tgt_vocab_size\n\n    _tiny_config = get_tiny_config(config_class, **model_tester_kwargs)\n\n    # handle the possibility of `text_config` inside `_tiny_config` for clip-like models (`owlvit`, `groupvit`, etc.)\n    if hasattr(_tiny_config, \"text_config\"):\n        _tiny_config = _tiny_config.text_config\n\n    # Collect values of some special token ids\n    for attr in dir(_tiny_config):\n        if attr.endswith(\"_token_id\"):\n            token_id = getattr(_tiny_config, attr)\n            if token_id is not None:\n                # Using the token id values from `tokenizer` instead of from `_tiny_config`.\n                token_id = get_token_id_from_tokenizer(attr, tokenizer, original_token_id=token_id)\n                config_overrides[attr] = token_id\n\n    if config_class.__name__ == \"FSMTConfig\":\n        config_overrides[\"src_vocab_size\"] = tokenizer.src_vocab_size\n        config_overrides[\"tgt_vocab_size\"] = tokenizer.tgt_vocab_size\n        # `FSMTConfig` has `DecoderConfig` as `decoder` attribute.\n        config_overrides[\"decoder\"] = configuration_fsmt.DecoderConfig(\n            vocab_size=tokenizer.tgt_vocab_size, bos_token_id=config_overrides[\"eos_token_id\"]\n        )\n\n\n\nSUMMARY:\n**Function Summary: `get_configOverrides`**\n\n**Purpose:** Returns configuration overrides for a given `config_class` and list of `processors`, specifically designed for Hugging Face models.\n\n**Key Implementation Details:**\n\n* Checks if a specific tokenizer is present in the `processors` list and returns overrides accordingly.\n* Extracts properties from the chosen tokenizer, including vocabulary size, special token ids, and decoder configuration.\n* Generates model tester keyword arguments based on the chosen tokenizer's vocabulary size and decoder attributes.\n* Iterates over special token id attributes and collects their values using the tokenizer or `_tiny_config` object.\n\n**Notable Dependencies and Requirements:**\n\n* `PreTrainedTokenizerFast` and `PreTrainedTokenizer`\n* External dependencies:\n\t+ `FSMTModelTester`\n\t+ `configuration_fsmt`\n\t+ `DebertaV2Tokenizer`\n* Inherited/Implemented interfaces:\n\t+ `config_class`: uses specific classes from the Hugging Face library.\n\t+ `configuration_fsmt`: used to create a decoder configuration for FSMTConfig models."}


created_at: 2025-02-05 09:13:55
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py:1089-1269
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 1269, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py", "name": "build", "start_line": 1089, "type": "function", "chroma:document": "CODE:\n\ndef build(config_class, models_to_create, output_dir):\n    \"\"\"Create all models for a certain model type.\n\n    Args:\n        config_class (`PretrainedConfig`):\n            A subclass of `PretrainedConfig` that is used to determine `models_to_create`.\n        models_to_create (`dict`):\n            A dictionary containing the processor/model classes that we want to create the instances. These models are\n            of the same model type which is associated to `config_class`.\n        output_dir (`str`):\n            The directory to save all the checkpoints. Each model architecture will be saved in a subdirectory under\n            it. Models in different frameworks with the same architecture will be saved in the same subdirectory.\n    \"\"\"\n    if data[\"training_ds\"] is None or data[\"testing_ds\"] is None:\n        ds = load_dataset(\"Salesforce/wikitext\", \"wikitext-2-raw-v1\")\n        data[\"training_ds\"] = ds[\"train\"]\n        data[\"testing_ds\"] = ds[\"test\"]\n\n    if config_class.model_type in [\n        \"encoder-decoder\",\n        \"vision-encoder-decoder\",\n        \"speech-encoder-decoder\",\n        \"vision-text-dual-encoder\",\n    ]:\n        return build_composite_models(config_class, output_dir)\n\n    result = {k: {} for k in models_to_create}\n\n    # These will be removed at the end if they are empty\n    result[\"error\"] = None\n    result[\"warnings\"] = []\n\n    # Build processors\n    processor_classes = models_to_create[\"processor\"]\n\n    if len(processor_classes) == 0:\n        error = f\"No processor class could be found in {config_class.__name__}.\"\n        fill_result_with_error(result, error, None, models_to_create)\n        logger.error(result[\"error\"][0])\n        return result\n\n    for processor_class in processor_classes:\n        try:\n            processor = build_processor(config_class, processor_class, allow_no_checkpoint=True)\n            if processor is not None:\n                result[\"processor\"][processor_class] = processor\n        except Exception:\n            error = f\"Failed to build processor for {processor_class.__name__}.\"\n            trace = traceback.format_exc()\n            fill_result_with_error(result, error, trace, models_to_create)\n            logger.error(result[\"error\"][0])\n            return result\n\n    if len(result[\"processor\"]) == 0:\n        error = f\"No processor could be built for {config_class.__name__}.\"\n        fill_result_with_error(result, error, None, models_to_create)\n        logger.error(result[\"error\"][0])\n        return result\n\n    try:\n        tiny_config = get_tiny_config(config_class)\n    except Exception as e:\n        error = f\"Failed to get tiny config for {config_class.__name__}: {e}\"\n        trace = traceback.format_exc()\n        fill_result_with_error(result, error, trace, models_to_create)\n        logger.error(result[\"error\"][0])\n        return result\n\n    # Convert the processors (reduce vocabulary size, smaller image size, etc.)\n    processors = list(result[\"processor\"].values())\n    processor_output_folder = os.path.join(output_dir, \"processors\")\n    try:\n        processors = convert_processors(processors, tiny_config, processor_output_folder, result)\n    except Exception:\n        error = \"Failed to convert the processors.\"\n        trace = traceback.format_exc()\n        result[\"warnings\"].append((error, trace))\n\n    if len(processors) == 0:\n        error = f\"No processor is returned by `convert_processors` for {config_class.__name__}.\"\n        fill_result_with_error(result, error, None, models_to_create)\n        logger.error(result[\"error\"][0])\n        return result\n\n    try:\n        config_overrides = get_config_overrides(config_class, processors)\n    except Exception as e:\n        error = f\"Failure occurs while calling `get_config_overrides`: {e}\"\n        trace = traceback.format_exc()\n        fill_result_with_error(result, error, trace, models_to_create)\n        logger.error(result[\"error\"][0])\n        return result\n\n    # Just for us to see this easily in the report\n    if \"vocab_size\" in config_overrides:\n        result[\"vocab_size\"] = config_overrides[\"vocab_size\"]\n\n    # Update attributes that `vocab_size` involves\n    for k, v in config_overrides.items():\n        if hasattr(tiny_config, k):\n            setattr(tiny_config, k, v)\n        # So far, we only have to deal with `text_config`, as `config_overrides` contains text-related attributes only.\n        # `FuyuConfig` saves data under both FuyuConfig and its `text_config`. This is not good, but let's just update\n        # every involved fields to avoid potential failure.\n        if (\n            hasattr(tiny_config, \"text_config\")\n            and tiny_config.text_config is not None\n            and hasattr(tiny_config.text_config, k)\n        ):\n            setattr(tiny_config.text_config, k, v)\n            # If `text_config_dict` exists, we need to update its value here too in order to # make\n            # `save_pretrained -> from_pretrained` work.\n            if hasattr(tiny_config, \"text_config_dict\"):\n                tiny_config.text_config_dict[k] = v\n\n    if result[\"warnings\"]:\n        logger.warning(result[\"warnings\"][0][0])\n\n    # update `result[\"processor\"]`\n    result[\"processor\"] = {type(p).__name__: p.__class__.__name__ for p in processors}\n\n    for pytorch_arch in models_to_create[\"pytorch\"]:\n        result[\"pytorch\"][pytorch_arch.__name__] = {}\n        error = None\n        try:\n            model = build_model(pytorch_arch, tiny_config, output_dir=output_dir)\n        except Exception as e:\n            model = None\n            error = f\"Failed to create the pytorch model for {pytorch_arch}: {e}\"\n            trace = traceback.format_exc()\n\n        result[\"pytorch\"][pytorch_arch.__name__][\"model\"] = model.__class__.__name__ if model is not None else None\n        result[\"pytorch\"][pytorch_arch.__name__][\"checkpoint\"] = (\n            get_checkpoint_dir(output_dir, pytorch_arch) if model is not None else None\n        )\n        if error is not None:\n            result[\"pytorch\"][pytorch_arch.__name__][\"error\"] = (error, trace)\n            logger.error(f\"{pytorch_arch.__name__}: {error}\")\n\n    for tensorflow_arch in models_to_create[\"tensorflow\"]:\n        # Make PT/TF weights compatible\n        pt_arch_name = tensorflow_arch.__name__[2:]  # Remove `TF`\n        pt_arch = getattr(transformers_module, pt_arch_name)\n\n        result[\"tensorflow\"][tensorflow_arch.__name__] = {}\n        error = None\n        if pt_arch.__name__ in result[\"pytorch\"] and result[\"pytorch\"][pt_arch.__name__][\"checkpoint\"] is not None:\n            ckpt = get_checkpoint_dir(output_dir, pt_arch)\n            # Use the same weights from PyTorch.\n            try:\n                model = tensorflow_arch.from_pretrained(ckpt)\n                model.save_pretrained(ckpt)\n            except Exception as e:\n                # Conversion may fail. Let's not create a model with different weights to avoid confusion (for now).\n                model = None\n                error = f\"Failed to convert the pytorch model to the tensorflow model for {pt_arch}: {e}\"\n                trace = traceback.format_exc()\n        else:\n            try:\n                model = build_model(tensorflow_arch, tiny_config, output_dir=output_dir)\n            except Exception as e:\n                model = None\n                error = f\"Failed to create the tensorflow model for {tensorflow_arch}: {e}\"\n                trace = traceback.format_exc()\n\n        result[\"tensorflow\"][tensorflow_arch.__name__][\"model\"] = (\n            model.__class__.__name__ if model is not None else None\n        )\n        result[\"tensorflow\"][tensorflow_arch.__name__][\"checkpoint\"] = (\n            get_checkpoint_dir(output_dir, tensorflow_arch) if model is not None else None\n        )\n        if error is not None:\n            result[\"tensorflow\"][tensorflow_arch.__name__][\"error\"] = (error, trace)\n            logger.error(f\"{tensorflow_arch.__name__}: {error}\")\n\n    if not result[\"error\"]:\n        del result[\"error\"]\n    if not result[\"warnings\"]:\n        del result[\"warnings\"]\n\n\n\nSUMMARY:\n**Summary**\n\nThis is the `build` function, which creates PyTorch and TensorFlow models for a given `config_class`, `models_to_create`, and `output_dir`. The primary purpose is to build and save checkpoints for multiple model types.\n\n**Main Functionality**\n\n1. Creates processors using `build_processor` with the given `processor_classes`.\n2. Converts processors by reducing vocabulary size and image size.\n3. Get tiny config using `get_tiny_config`.\n4. Build and save PyTorch models using `build_model` for each PyTorch arch.\n5. Build and save TensorFlow models using `from_pretrained` and `save_pretrained` methods for each TensorFlow arch.\n\n**Output**\n\nThe function returns a dictionary containing the following:\n- `processor`: Model types and their corresponding names\n- `pytorch`: PyTorch models and their checkpoints\n- `tensorflow`: TensorFlow models and their checkpoints\n\n**Key Functions**\n\n1. `build_processor`\n2. `convert_processors`\n3. `get_tiny_config`\n4. `build_model` (PyTorch)\n5. `from_pretrained` (TensorFlow)\n\nNote: The function contains many nested sections and conditional statements, making it complex and hard to read. Some sections are commented out or not relevant to the current task, which may make it difficult to understand the entire codebase."}


created_at: 2025-02-05 09:13:55
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py:1272-1324
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 1324, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py", "name": "build_tiny_model_summary", "start_line": 1272, "type": "function", "chroma:document": "CODE:\n\ndef build_tiny_model_summary(results, organization=None, token=None):\n    \"\"\"Build a summary: a dictionary of the form\n    {\n      model architecture name:\n        {\n          \"tokenizer_classes\": [...],\n          \"processor_classes\": [...],\n          \"model_classes\": [...],\n        }\n      ..\n    }\n    \"\"\"\n    tiny_model_summary = {}\n    for config_name in results:\n        processors = [key for key, value in results[config_name][\"processor\"].items()]\n        tokenizer_classes = sorted([x for x in processors if x.endswith(\"TokenizerFast\") or x.endswith(\"Tokenizer\")])\n        processor_classes = sorted([x for x in processors if x not in tokenizer_classes])\n        for framework in FRAMEWORKS:\n            if framework not in results[config_name]:\n                continue\n            for arch_name in results[config_name][framework]:\n                model_classes = [arch_name]\n                base_arch_name = arch_name[2:] if arch_name.startswith(\"TF\") else arch_name\n                # tiny model is not created for `arch_name`\n                if results[config_name][framework][arch_name][\"model\"] is None:\n                    model_classes = []\n                if base_arch_name not in tiny_model_summary:\n                    tiny_model_summary[base_arch_name] = {}\n                tiny_model_summary[base_arch_name].update(\n                    {\n                        \"tokenizer_classes\": tokenizer_classes,\n                        \"processor_classes\": processor_classes,\n                    }\n                )\n                tiny_model_summary[base_arch_name][\"model_classes\"] = sorted(\n                    tiny_model_summary[base_arch_name].get(\"model_classes\", []) + model_classes\n                )\n                if organization is not None:\n                    repo_name = f\"tiny-random-{base_arch_name}\"\n                    # composite models' checkpoints have more precise repo. names on the Hub.\n                    if base_arch_name in COMPOSITE_MODELS:\n                        repo_name = f\"tiny-random-{COMPOSITE_MODELS[base_arch_name]}\"\n                    repo_id = f\"{organization}/{repo_name}\"\n                    try:\n                        commit_hash = hf_api.repo_info(repo_id, token=token).sha\n                    except Exception:\n                        # The directory is not created, but processor(s) is/are included in `results`.\n                        logger.warning(f\"Failed to get information for {repo_id}.\\n{traceback.format_exc()}\")\n                        del tiny_model_summary[base_arch_name]\n                        continue\n                    tiny_model_summary[base_arch_name][\"sha\"] = commit_hash\n\n\n\nSUMMARY:\n**Function Summary:**\n\n`build_tiny_model_summary`\n\nBuilds a dictionary of model architecture summaries.\n\n**Purpose and Functionality:**\nThe function takes in `results`, `organization`, and `token`, generating a summary dictionary with model names as keys and sub-dictionaries containing tokenizer classes, processor classes, model classes, and commit SHA hashes as values. It iterates through configurations in `results` and extracts models from different frameworks (TF/HT/TF2), updating the model summary accordingly.\n\n**Key Implementation Details:**\n\n* Iteration over configurations in `results`\n* Extractions of tokenizer classes (`x.endswith(\"TokenizerFast\") or x.endswith(\"Tokenizer\")`)\n* Extractions of processor classes (`not in tokenizer_classes`)\n* Iteration over frameworks (TF/HT/TF2) and architecture names\n* Model class updates for each architecture name\n\n**Notable Dependencies and Requirements:**\n\n* `hf_api` library for accessing repository information\n* Various external configurations stored in `results`\n* Optional `organization` parameter with repository ID generation\n* Token (`token`) used for authentication"}


created_at: 2025-02-05 09:13:55
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py:1327-1354
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 1354, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py", "name": "build_failed_report", "start_line": 1327, "type": "function", "chroma:document": "CODE:\n\ndef build_failed_report(results, include_warning=True):\n    failed_results = {}\n    for config_name in results:\n        if \"error\" in results[config_name]:\n            if config_name not in failed_results:\n                failed_results[config_name] = {}\n            failed_results[config_name] = {\"error\": results[config_name][\"error\"]}\n\n        if include_warning and \"warnings\" in results[config_name]:\n            if config_name not in failed_results:\n                failed_results[config_name] = {}\n            failed_results[config_name][\"warnings\"] = results[config_name][\"warnings\"]\n\n        for framework in FRAMEWORKS:\n            if framework not in results[config_name]:\n                continue\n            for arch_name in results[config_name][framework]:\n                if \"error\" in results[config_name][framework][arch_name]:\n                    if config_name not in failed_results:\n                        failed_results[config_name] = {}\n                    if framework not in failed_results[config_name]:\n                        failed_results[config_name][framework] = {}\n                    if arch_name not in failed_results[config_name][framework]:\n                        failed_results[config_name][framework][arch_name] = {}\n                    error = results[config_name][framework][arch_name][\"error\"]\n                    failed_results[config_name][framework][arch_name][\"error\"] = error\n\n\n\nSUMMARY:\n**Function Summary:**\n`build_failed_report(results, include_warning=True)` generates a dictionary containing detailed error reports for each configuration in the `results` input.\n\n**Purpose and Functionality:**\n\n* Iterates through configurations with error reports, framework-specific arch-names, and warnings (if requested).\n* Identifies and extracts applicable error messages from \"error\" or \"warnings\" sub-dictionary.\n* Creates a nested dictionary structure with failed results, including configuration name, framework, architecture name, and corresponding error message.\n\n**Key Implementation Details:**\n\n* Iterates through `FRAMEWORKS` list to process arch-names under each framework.\n* Uses conditional statements to navigate the complex data structure of \"error\" and/or \"warnings\" sub-dictionary.\n\n**Notable Dependencies or Requirements:**\n\n* `FRAMEWORKS` namespace (list of supported frameworks)\n* Access to dictionary keys `\"error\"` and/or `\"warnings\"` within specific configurations.\n\n**Method Signatures and Interfaces:**\n\n* takes two inputs:\n\t+ `results`: input dictionary containing configuration-specific data.\n\t+ `include_warning`: optional boolean flag (default: True).\n\t+ returns: a nested dictionary `failed_results` containing error reports for each configuration."}


