# embeddings_queue

created_at: 2025-02-05 09:13:55
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py:1357-1372
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 1372, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py", "name": "build_simple_report", "start_line": 1357, "type": "function", "chroma:document": "CODE:\n\ndef build_simple_report(results):\n    text = \"\"\n    failed_text = \"\"\n    for config_name in results:\n        for framework in FRAMEWORKS:\n            if framework not in results[config_name]:\n                continue\n            for arch_name in results[config_name][framework]:\n                if \"error\" in results[config_name][framework][arch_name]:\n                    result = results[config_name][framework][arch_name][\"error\"]\n                    failed_text += f\"{arch_name}: {result[0]}\\n\"\n                else:\n                    result = (\"OK\",)\n                text += f\"{arch_name}: {result[0]}\\n\"\n\n\n\nSUMMARY:\n**Summary**\n\nThe `build_simple_report` function generates two report strings: `text` and `failed_text`. It iterates through a nested dictionary of test results to generate a simple text report. The primary purpose is to extract and format pass/fail information for each framework, architecture, and configuration.\n\n**Key Implementation Details**\n\n* Iterates through three loops to gather data:\n\t+ Outer loop: Configuration names (`config_name`)\n\t+ Middle loop: Frameworks (`framework` from `FRAMEWORKS`)\n\t+ Inner loop: Architecture names (`arch_name`) within a given framework\n* Checks for \"error\" in the result, and formats either an OK or failed message\n* Formats and concatenates results into two separate strings using `text` and `failed_text`\n\n**Notable Dependencies/Requirements**\n\n* `FRAMEWORKS`: An iterable of list-like objects representing frameworks to report on\n* Related classes/functions not shown in this snippet"}


created_at: 2025-02-05 09:13:55
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py:1375-1399
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 1399, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py", "name": "update_tiny_model_summary_file", "start_line": 1375, "type": "function", "chroma:document": "CODE:\n\ndef update_tiny_model_summary_file(report_path):\n    with open(os.path.join(report_path, \"tiny_model_summary.json\")) as fp:\n        new_data = json.load(fp)\n    with open(\"tests/utils/tiny_model_summary.json\") as fp:\n        data = json.load(fp)\n    for key, value in new_data.items():\n        if key not in data:\n            data[key] = value\n        else:\n            for attr in [\"tokenizer_classes\", \"processor_classes\", \"model_classes\"]:\n                # we might get duplication here. We will remove them below when creating `updated_data`.\n                data[key][attr].extend(value[attr])\n            new_sha = value.get(\"sha\", None)\n            if new_sha is not None:\n                data[key][\"sha\"] = new_sha\n\n    updated_data = {}\n    for key in sorted(data.keys()):\n        updated_data[key] = {}\n        for attr, value in data[key].items():\n            # deduplication and sort\n            updated_data[key][attr] = sorted(set(value)) if attr != \"sha\" else value\n\n    with open(os.path.join(report_path, \"updated_tiny_model_summary.json\"), \"w\") as fp:\n\n\nSUMMARY:\n**Concise Summary:**\n\nThe `update_tiny_model_summary_file` function updates the contents of two JSON files, comparing data from two sources and replacing duplicates or updating specific attributes. The main purpose is to merge and validate tiny model summaries.\n\n* **Primary Purpose:** Merge data from two sources into a unified summary.\n* **Key Implementation Details:**\n\t+ Iterates over duplicate key-value pairs, updating the lower priority value with the higher priority one.\n\t+ Appends values from lists while preserving duplicates, except for the SHA hash attribute.\n\t+ Sorts and deduplicates list values, excluding the SHA hash field.\n* **Notable Dependencies/Requirements:**\n\t+ `json` module for parsing file contents\n\t+ `os` module for path manipulation\n\t+ Access to files \"tiny_model_summary.json\" and \"updated_tiny_model_summary.json\".\n\n**RAG System Notes:**"}


created_at: 2025-02-05 09:13:55
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py:1402-1517
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 1517, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/create_dummy_models.py", "name": "create_tiny_models", "start_line": 1402, "type": "function", "chroma:document": "CODE:\n\ndef create_tiny_models(\n    output_path,\n    all,\n    model_types,\n    models_to_skip,\n    no_check,\n    upload,\n    organization,\n    token,\n    num_workers=1,\n):\n    clone_path = os.path.abspath(os.path.dirname(os.path.dirname(__file__)))\n    if os.getcwd() != clone_path:\n        raise ValueError(f\"This script should be run from the root of the clone of `transformers` {clone_path}\")\n\n    report_path = os.path.join(output_path, \"reports\")\n    os.makedirs(report_path, exist_ok=True)\n\n    _pytorch_arch_mappings = [\n        x\n        for x in dir(transformers_module)\n        if x.startswith(\"MODEL_\") and x.endswith(\"_MAPPING\") and x != \"MODEL_NAMES_MAPPING\"\n    ]\n    _tensorflow_arch_mappings = [\n        x for x in dir(transformers_module) if x.startswith(\"TF_MODEL_\") and x.endswith(\"_MAPPING\")\n    ]\n\n    pytorch_arch_mappings = [getattr(transformers_module, x) for x in _pytorch_arch_mappings]\n    tensorflow_arch_mappings = [getattr(transformers_module, x) for x in _tensorflow_arch_mappings]\n\n    config_classes = CONFIG_MAPPING.values()\n    if not all:\n        config_classes = [CONFIG_MAPPING[model_type] for model_type in model_types]\n\n    # A map from config classes to tuples of processors (tokenizer, feature extractor, processor) classes\n    processor_type_map = {c: get_processor_types_from_config_class(c) for c in config_classes}\n\n    to_create = {}\n    for c in config_classes:\n        processors = processor_type_map[c]\n        models = get_architectures_from_config_class(c, pytorch_arch_mappings, models_to_skip)\n        tf_models = get_architectures_from_config_class(c, tensorflow_arch_mappings, models_to_skip)\n        if len(models) + len(tf_models) > 0:\n            to_create[c] = {\"processor\": processors, \"pytorch\": models, \"tensorflow\": tf_models}\n\n    results = {}\n    if num_workers <= 1:\n        for c, models_to_create in list(to_create.items()):\n            print(f\"Create models for {c.__name__} ...\")\n            result = build(c, models_to_create, output_dir=os.path.join(output_path, c.model_type))\n            results[c.__name__] = result\n            print(\"=\" * 40)\n    else:\n        all_build_args = []\n        for c, models_to_create in list(to_create.items()):\n            all_build_args.append((c, models_to_create, os.path.join(output_path, c.model_type)))\n        with multiprocessing.Pool() as pool:\n            results = pool.starmap(build, all_build_args)\n            results = {buid_args[0].__name__: result for buid_args, result in zip(all_build_args, results)}\n\n    if upload:\n        if organization is None:\n            raise ValueError(\"The argument `organization` could not be `None`. No model is uploaded\")\n\n        to_upload = []\n        for model_type in os.listdir(output_path):\n            # This is the directory containing the reports\n            if model_type == \"reports\":\n                continue\n            for arch in os.listdir(os.path.join(output_path, model_type)):\n                if arch == \"processors\":\n                    continue\n                to_upload.append(os.path.join(output_path, model_type, arch))\n        to_upload = sorted(to_upload)\n\n        upload_results = {}\n        if len(to_upload) > 0:\n            for model_dir in to_upload:\n                try:\n                    upload_model(model_dir, organization, token)\n                except Exception as e:\n                    error = f\"Failed to upload {model_dir}. {e.__class__.__name__}: {e}\"\n                    logger.error(error)\n                    upload_results[model_dir] = error\n\n        with open(os.path.join(report_path, \"failed_uploads.json\"), \"w\") as fp:\n            json.dump(upload_results, fp, indent=4)\n\n    # Build the tiny model summary file. The `tokenizer_classes` and `processor_classes` could be both empty lists.\n    # When using the items in this file to update the file `tests/utils/tiny_model_summary.json`, the model\n    # architectures with `tokenizer_classes` and `processor_classes` being both empty should **NOT** be added to\n    # `tests/utils/tiny_model_summary.json`.\n    tiny_model_summary = build_tiny_model_summary(results, organization=organization, token=token)\n    with open(os.path.join(report_path, \"tiny_model_summary.json\"), \"w\") as fp:\n        json.dump(tiny_model_summary, fp, indent=4)\n\n    with open(os.path.join(report_path, \"tiny_model_creation_report.json\"), \"w\") as fp:\n        json.dump(results, fp, indent=4)\n\n    # Build the warning/failure report (json format): same format as the complete `results` except this contains only\n    # warnings or errors.\n    failed_results = build_failed_report(results)\n    with open(os.path.join(report_path, \"failed_report.json\"), \"w\") as fp:\n        json.dump(failed_results, fp, indent=4)\n\n    simple_report, failed_report = build_simple_report(results)\n    # The simplified report: a .txt file with each line of format:\n    # {model architecture name}: {OK or error message}\n    with open(os.path.join(report_path, \"simple_report.txt\"), \"w\") as fp:\n        fp.write(simple_report)\n\n    # The simplified failure report: same above except this only contains line with errors\n    with open(os.path.join(report_path, \"simple_failed_report.txt\"), \"w\") as fp:\n        fp.write(failed_report)\n\n\n\nSUMMARY:\n**Function Summary**\n\nThe `create_tiny_models` function creates and builds models for various transformer architectures. Its primary purpose is to generate reports on the creation process.\n\n**Key Functionality:**\n\n1.  Checks if the current directory is the root of the cloned 'transformers' repository.\n2.  Creates a \"reports\" directory in the output path and sets it up with proper permissions.\n3.  Lists all the model types, processes it through `model_types` list or retrieves them from all by using CONFIG_MAPPING.values() if required, and then maps to specific architectures like 'tokenizer_classes' and 'processor_classes'.\n4.  If num_workers is greater than 1, uses multiprocessing pool with star map function to build models in parallel.\n5.  Builds the tiny model summary file based on results.\n\n**Implementation Details:**\n\n-   Uses `os.path` module to work with paths.\n-   Utilizes the dir() function of objects to get specific architectures like pytorch and tensorflow models from transformers_module.\n\n**Notable Dependencies or Requirements:**\n\n1.  The functions builds on top of the os, multiprocessing and json modules as shown below:\n    -   `os`: used for working with file paths.\n    -   `multiprocessing.Pool` : used to build models in parallel.\n    -   `json`: used to dump results in JSON format.\n\n2.  References transformers_module which is not a custom module but likely part of an existing library or framework, containing classes and methods like \"build\" as mentioned above which builds models and also processes to_create dictionary which maps config class names to processors and architectures built based on pytorch arch mappings.\n\n3.  The organization and token are requirements used during upload process.\n4.  It uses a custom function build_tiny_model_summary where model type, tokenizer_classes, processor_classes are passed as arguments."}


created_at: 2025-02-05 09:13:55
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/create_dependency_mapping.py:5-27
encoding: FLOAT32
metadata:
{"docstring": "# Function to perform topological sorting", "end_line": 27, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/create_dependency_mapping.py", "name": "topological_sort", "start_line": 5, "type": "function", "chroma:document": "CODE:\n# Function to perform topological sorting\ndef topological_sort(dependencies: dict):\n    # Nodes are the name of the models to convert (we only add those to the graph)\n    nodes = {node.rsplit(\"modular_\", 1)[1].replace(\".py\", \"\") for node in dependencies.keys()}\n    # This will be a graph from models to convert, to models to convert that should be converted before (as they are a dependency)\n    graph = {}\n    name_mapping = {}\n    for node, deps in dependencies.items():\n        node_name = node.rsplit(\"modular_\", 1)[1].replace(\".py\", \"\")\n        dep_names = {dep.split(\".\")[-2] for dep in deps}\n        dependencies = {dep for dep in dep_names if dep in nodes and dep != node_name}\n        graph[node_name] = dependencies\n        name_mapping[node_name] = node\n\n    sorting_list = []\n    while len(graph) > 0:\n        # Find the nodes with 0 out-degree\n        leaf_nodes = {node for node in graph if len(graph[node]) == 0}\n        # Add them to the list\n        sorting_list += list(leaf_nodes)\n        # Remove the leafs from the graph (and from the deps of other nodes)\n        graph = {node: deps - leaf_nodes for node, deps in graph.items() if node not in leaf_nodes}\n\n\n\nSUMMARY:\n**Concise Summary**\n\nThe `topological_sort` function performs a topological sort on a graph of model dependencies. It takes a dictionary `dependencies` as input, where each key is a node (model) and its value is a list of its dependent models.\n\n**Key Features:**\n\n* The function constructs a directed graph from the dependencies.\n* It uses a depth-first search algorithm to find nodes with no outgoing edges (leaf nodes), which are then added to a sorting list.\n* The process repeats until all nodes have been visited, resulting in a sorted list of model names corresponding to the original input dictionary keys.\n\n**Notes:**\n\n* External dependencies include `graph` and `name_mapping`, which represent temporary data structures used during the sorting process.\n* No external libraries are required beyond Python's built-in `dict` and set operations."}


created_at: 2025-02-05 09:13:55
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/create_dependency_mapping.py:31-41
encoding: FLOAT32
metadata:
{"docstring": "# Function to extract class and import info from a file", "end_line": 41, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/create_dependency_mapping.py", "name": "extract_classes_and_imports", "start_line": 31, "type": "function", "chroma:document": "CODE:\n# Function to extract class and import info from a file\ndef extract_classes_and_imports(file_path):\n    with open(file_path, \"r\") as file:\n        tree = ast.parse(file.read(), filename=file_path)\n    imports = set()\n\n    for node in ast.walk(tree):\n        if isinstance(node, (ast.Import, ast.ImportFrom)):\n            module = node.module if isinstance(node, ast.ImportFrom) else None\n            if module and (\".modeling_\" in module or \"transformers.models\" in module):\n                imports.add(module)\n\n\nSUMMARY:\n**Concise Summary**\n\nThe `extract_classes_and_imports` function extracts classes and imports from a Python file. It:\n\n* Parses the file into an Abstract Syntax Tree (AST)\n*walks through the AST to identify import nodes containing \".modeling_\" or \"transformers.models\" modules\n*Returns a set of unique imported module paths.\n\n**Key Implementation Details**\n\nThe key implementation details are:\n\n* The use of `ast.Import` and `ast.ImportFrom` node types to identify imports.\n* The use of `filename` parameter in `ast.parse()` to preserve the file path.\n* The `walk()` function to traverse the AST tree.\n\n**Notable Dependencies or Requirements**\n\nThe function relies on the following dependencies:\n\n* Python's built-in `ast` module (Abstract Syntax Tree).\n* No external libraries are required."}


created_at: 2025-02-05 09:13:55
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/create_dependency_mapping.py:45-53
encoding: FLOAT32
metadata:
{"docstring": "# Function to map dependencies between classes", "end_line": 53, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/create_dependency_mapping.py", "name": "map_dependencies", "start_line": 45, "type": "function", "chroma:document": "CODE:\n# Function to map dependencies between classes\ndef map_dependencies(py_files):\n    dependencies = defaultdict(set)\n    # First pass: Extract all classes and map to files\n    for file_path in py_files:\n        # dependencies[file_path].add(None)\n        class_to_file = extract_classes_and_imports(file_path)\n        for module in class_to_file:\n            dependencies[file_path].add(module)\n\n\nSUMMARY:\n**Summary**\n\n Maps dependencies between Python file locations. \n\n**Key Implementation Details**\n\n* Merges `extract_classes_and_imports` function call into a single loop, iterating over each file in `py_files`\n* Utilizes a `defaultdict(set)` to efficiently store and manage dependencies for each file\n* Returns a dictionary mapping file paths to sets of imported modules\n\n**Notable Dependencies or Requirements**\n\n* None explicitly mentioned; no external dependencies are referenced. \n\n(Note: The exact implementation of the `extract_classes_and_imports` function is not shown in this snippet, but it presumably extracts class definitions and import statements from a given Python file.)"}


created_at: 2025-02-05 09:13:55
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/create_dependency_mapping.py:56-59
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 59, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/create_dependency_mapping.py", "name": "find_priority_list", "start_line": 56, "type": "function", "chroma:document": "CODE:\n\ndef find_priority_list(py_files):\n    dependencies = map_dependencies(py_files)\n    ordered_classes = topological_sort(dependencies)\n\n\nSUMMARY:\n**Summary:**\n\nThis function, `find_priority_list`, takes a list of Python files (`py_files`) as input. It maps dependencies between classes, performs a topological sort on the dependency graph, and returns an ordered list of classes.\n\n**Key Details:**\n\n* Maps class dependencies using `map_dependencies(py_files)`\n* Performs topological sort on dependency graph using `topological_sort(dependencies)`\n* Returns ordered list of classes\n\n**Notable Dependencies/Requirements:**\n\n* None explicitly stated, but `map_dependencies` and `topological_sort` likely rely on external libraries (e.g., Pyrex, graphlib) that are not included in this function summary."}


created_at: 2025-02-05 09:13:55
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_docstrings.py:600-607
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 607, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_docstrings.py", "name": "find_indent", "start_line": 600, "type": "function", "chroma:document": "CODE:\n\ndef find_indent(line: str) -> int:\n    \"\"\"\n    Returns the number of spaces that start a line indent.\n    \"\"\"\n    search = re.search(r\"^(\\s*)(?:\\S|$)\", line)\n    if search is None:\n        return 0\n\n\nSUMMARY:\n**Summary:**\n\n`find_indent(line: str) -> int`: Returns the number of spaces at the start of a line, used for indentation calculation.\n\n**Key Details:**\n\n* Uses `re.search` to search for a pattern of one or more non-breaking spaces (`\\s+`) at the beginning of the line (`^`).\n* If no match is found, returns 0.\n* Returns the length (i.e., number of spaces) in the matched group.\n\n**Example Usage:**"}


created_at: 2025-02-05 09:13:55
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_docstrings.py:610-637
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 637, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_docstrings.py", "name": "stringify_default", "start_line": 610, "type": "function", "chroma:document": "CODE:\n\ndef stringify_default(default: Any) -> str:\n    \"\"\"\n    Returns the string representation of a default value, as used in docstring: numbers are left as is, all other\n    objects are in backtiks.\n\n    Args:\n        default (`Any`): The default value to process\n\n    Returns:\n        `str`: The string representation of that default.\n    \"\"\"\n    if isinstance(default, bool):\n        # We need to test for bool first as a bool passes isinstance(xxx, (int, float))\n        return f\"`{default}`\"\n    elif isinstance(default, enum.Enum):\n        # We need to test for enum first as an enum with int values will pass isinstance(xxx, (int, float))\n        return f\"`{str(default)}`\"\n    elif isinstance(default, int):\n        return str(default)\n    elif isinstance(default, float):\n        result = str(default)\n        return str(round(default, 2)) if len(result) > 6 else result\n    elif isinstance(default, str):\n        return str(default) if default.isnumeric() else f'`\"{default}\"`'\n    elif isinstance(default, type):\n        return f\"`{default.__name__}`\"\n    else:\n\n\nSUMMARY:\n**Summary:**\n\nThis function `stringify_default` converts its input `default` value into a string representation. It handles various data types:\n\n*   Boolean (`True/False`) and integers as is\n*   Enums using their representation as a string\n*   Floats with precise 2-decimal places formatting (unless more than 6 characters long)\n*   Strings with numeric values or enclosed in backticks\n\n**Key implementation details:**\n\n| Implementation | Description |\n| --- | --- |\n| `bool` and `int` conversions | Directly return as strings |\n| `enum.Enum` conversion | Use `str()` to convert to string, preserving enum value |\n| Float formatting | Use `round()` with 2 decimal places for strings longer than 6 characters |\n| String handling | Separate numeric strings from non-numeric strings and apply formatting accordingly |\n\n**Noteable dependencies:**\n\n*   `enum`: Used to handle Enums\n*   `str()`, `round()`: Implemented in Python's built-in functions\n\nThis summary provides a concise overview of the function's primary functionality, key implementation details, and notable dependencies."}


created_at: 2025-02-05 09:13:55
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_docstrings.py:640-666
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 666, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_docstrings.py", "name": "eval_math_expression", "start_line": 640, "type": "function", "chroma:document": "CODE:\n\ndef eval_math_expression(expression: str) -> Optional[Union[float, int]]:\n    # Mainly taken from the excellent https://stackoverflow.com/a/9558001\n    \"\"\"\n    Evaluate (safely) a mathematial expression and returns its value.\n\n    Args:\n        expression (`str`): The expression to evaluate.\n\n    Returns:\n        `Optional[Union[float, int]]`: Returns `None` if the evaluation fails in any way and the value computed\n        otherwise.\n\n    Example:\n\n    ```py\n    >>> eval_expr('2^6')\n    4\n    >>> eval_expr('2**6')\n    64\n    >>> eval_expr('1 + 2*3**(4^5) / (6 + -7)')\n    -5.0\n    ```\n    \"\"\"\n    try:\n        return eval_node(ast.parse(expression, mode=\"eval\").body)\n    except TypeError:\n\n\nSUMMARY:\n**Summary**\n\nThis function, `eval_math_expression`, evaluates a mathematical expression provided as a string and returns its value. It safely handles various mathematical operations, including exponentiation, multiplication, division, and addition/subtraction.\n\n**Key Implementation Details**\n\n* Utilizes the `eval` function from Python's standard library to evaluate the expression.\n* Parses the expression using `ast.parse` with the `mode=\"eval\"` argument.\n* Calls the `eval_node` function (not shown in this code snippet) on the parsed AST.\n\n**Notable Dependencies and Requirements**\n\n* Python's standard library, specifically the `ast` module for parsing.\n* The `eval_node` function is not part of the standard library and is assumed to be implemented separately."}


created_at: 2025-02-05 09:13:55
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_docstrings.py:669-677
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 677, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_docstrings.py", "name": "eval_node", "start_line": 669, "type": "function", "chroma:document": "CODE:\n\ndef eval_node(node):\n    if isinstance(node, ast.Num):  # <number>\n        return node.n\n    elif isinstance(node, ast.BinOp):  # <left> <operator> <right>\n        return MATH_OPERATORS[type(node.op)](eval_node(node.left), eval_node(node.right))\n    elif isinstance(node, ast.UnaryOp):  # <operator> <operand> e.g., -1\n        return MATH_OPERATORS[type(node.op)](eval_node(node.operand))\n    else:\n\n\nSUMMARY:\n**Eval Node Function Summary**\n\n### Purpose and Functionality\n\nThe `eval_node` function is a recursive traverser of abstract syntax tree (AST) nodes, evaluating arithmetic expressions by applying mathematical operators to their operands.\n\n### Key Implementation Details\n\n* Evaluates integer literal nodes (`ast.Num`) as-is.\n* Unifies binary operation nodes (`ast.BinOp`): recursively evaluates left and right operands with operator-dependent functions from `MATH_OPERATORS`.\n* Unifies unary operation nodes (`ast.UnaryOp`): recursively evaluates the operand with an operator-dependent function from `MATH_OPERATORS`.\n* Raises a `TypeError` for unsupported node types.\n\n### Dependencies\n\nNone, as this implementation does not rely on external dependencies. However, it utilizes:\n\n* `ast` (Abstract Syntax Tree) module for parsing expressions.\n* `MATH Operators` dictionary (not shown in provided code), which maps operator types to evaluation functions.\n* No external libraries besides the Python standard library."}


created_at: 2025-02-05 09:13:55
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_docstrings.py:680-742
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 742, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_docstrings.py", "name": "replace_default_in_arg_description", "start_line": 680, "type": "function", "chroma:document": "CODE:\n\ndef replace_default_in_arg_description(description: str, default: Any) -> str:\n    \"\"\"\n    Catches the default value in the description of an argument inside a docstring and replaces it by the value passed.\n\n    Args:\n        description (`str`): The description of an argument in a docstring to process.\n        default (`Any`): The default value that whould be in the docstring of that argument.\n\n    Returns:\n       `str`: The description updated with the new default value.\n    \"\"\"\n    # Lots of docstrings have `optional` or **opational** instead of *optional* so we do this fix here.\n    description = description.replace(\"`optional`\", OPTIONAL_KEYWORD)\n    description = description.replace(\"**optional**\", OPTIONAL_KEYWORD)\n    if default is inspect._empty:\n        # No default, make sure the description doesn't have any either\n        idx = description.find(OPTIONAL_KEYWORD)\n        if idx != -1:\n            description = description[:idx].rstrip()\n            if description.endswith(\",\"):\n                description = description[:-1].rstrip()\n    elif default is None:\n        # Default None are not written, we just set `*optional*`. If there is default that is not None specified in the\n        # description, we do not erase it (as sometimes we set the default to `None` because the default is a mutable\n        # object).\n        idx = description.find(OPTIONAL_KEYWORD)\n        if idx == -1:\n            description = f\"{description}, {OPTIONAL_KEYWORD}\"\n        elif re.search(r\"defaults to `?None`?\", description) is not None:\n            len_optional = len(OPTIONAL_KEYWORD)\n            description = description[: idx + len_optional]\n    else:\n        str_default = None\n        # For numbers we may have a default that is given by a math operation (1/255 is really popular). We don't\n        # want to replace those by their actual values.\n        if isinstance(default, (int, float)) and re.search(\"defaults to `?(.*?)(?:`|$)\", description) is not None:\n            # Grab the default and evaluate it.\n            current_default = re.search(\"defaults to `?(.*?)(?:`|$)\", description).groups()[0]\n            if default == eval_math_expression(current_default):\n                try:\n                    # If it can be directly converted to the type of the default, it's a simple value\n                    str_default = str(type(default)(current_default))\n                except Exception:\n                    # Otherwise there is a math operator so we add a code block.\n                    str_default = f\"`{current_default}`\"\n            elif isinstance(default, enum.Enum) and default.name == current_default.split(\".\")[-1]:\n                # When the default is an Enum (this is often the case for PIL.Image.Resampling), and the docstring\n                # matches the enum name, keep the existing docstring rather than clobbering it with the enum value.\n                str_default = f\"`{current_default}`\"\n\n        if str_default is None:\n            str_default = stringify_default(default)\n        # Make sure default match\n        if OPTIONAL_KEYWORD not in description:\n            description = f\"{description}, {OPTIONAL_KEYWORD}, defaults to {str_default}\"\n        elif _re_parse_description.search(description) is None:\n            idx = description.find(OPTIONAL_KEYWORD)\n            len_optional = len(OPTIONAL_KEYWORD)\n            description = f\"{description[:idx + len_optional]}, defaults to {str_default}\"\n        else:\n            description = _re_parse_description.sub(rf\"*optional*, defaults to {str_default}\", description)\n\n\n\nSUMMARY:\n**Summary:**\n\nFunction `replace_default_in_arg_description` updates the argument description in a docstring to include the provided default value. It performs various fixes and transformations based on the type and content of the default value.\n\n**Key Features:**\n\n* Replaces occurrences of \"optional\" or \"**optional**\" with `OPTIONAL_KEYWORD`.\n* Handles cases where the default value is `None`, `empty string` (e.g., empty line), or a complex math expression.\n* For numbers, evaluates the math expression if it matches the default value and converts it to the correct type.\n\n**Important Variables and Functions:**\n\n* `OPTIONAL_KEYWORD`: \"optional\" or \"**optional**\"\n* `description`: input docstring\n* `default`: default value to be included in the updated description\n\n**Notable Dependencies:**\n\n* `inspect` for checking `default` value\n* `re` for regular expression matching and replacing\n* `eval_math_expression` function not provided in the code snippet, but used for evaluating math expressions\n\n**Return Value:**\n\nThe updated docstring with the default value included."}


created_at: 2025-02-05 09:13:55
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_docstrings.py:745-768
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 768, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_docstrings.py", "name": "get_default_description", "start_line": 745, "type": "function", "chroma:document": "CODE:\n\ndef get_default_description(arg: inspect.Parameter) -> str:\n    \"\"\"\n    Builds a default description for a parameter that was not documented.\n\n    Args:\n        arg (`inspect.Parameter`): The argument in the signature to generate a description for.\n\n    Returns:\n        `str`: The description.\n    \"\"\"\n    if arg.annotation is inspect._empty:\n        arg_type = \"<fill_type>\"\n    elif hasattr(arg.annotation, \"__name__\"):\n        arg_type = arg.annotation.__name__\n    else:\n        arg_type = str(arg.annotation)\n\n    if arg.default is inspect._empty:\n        return f\"`{arg_type}`\"\n    elif arg.default is None:\n        return f\"`{arg_type}`, {OPTIONAL_KEYWORD}\"\n    else:\n        str_default = stringify_default(arg.default)\n\n\nSUMMARY:\n**get_default_description function summary**\n\n```markdown\nProvides default description for an uncovered argument in Python function signatures.\n\n### Key implementation details:\n\n* Uses `inspect.Parameter` and related types (`inspect._empty`) to determine parameter type and defaults.\n* Employs string formatting to construct the final default description.\n\n### Notable dependencies and interfaces:\n\n* `inspect`: Imported module providing type inspection utilities.\n* `optional`: Likely an external library (e.g., `pydantic`) for optional parameter handling, referenced by `OPTIONAL_KEYWORD`.\n```\n\n### Primary purpose and functionality\nBuilds a default description string for an uncovered argument in a Python function signature, considering its type and potential default value."}


created_at: 2025-02-05 09:13:55
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_docstrings.py:771-785
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 785, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_docstrings.py", "name": "find_source_file", "start_line": 771, "type": "function", "chroma:document": "CODE:\n\ndef find_source_file(obj: Any) -> Path:\n    \"\"\"\n    Finds the source file of an object.\n\n    Args:\n        obj (`Any`): The object whose source file we are looking for.\n\n    Returns:\n        `Path`: The source file.\n    \"\"\"\n    module = obj.__module__\n    obj_file = PATH_TO_TRANSFORMERS\n    for part in module.split(\".\")[1:]:\n        obj_file = obj_file / part\n\n\nSUMMARY:\n**Summary:**\n\nFinds the source file of an object given its `__module__` attribute.\n\n*   **Purpose:** Returns the path to the source file associated with the input object's module.\n*   **Implementation Details:**\n    *   Uses the `__module__` attribute of the input object to determine the module name.\n    *   Joins the module name with a prefix (`PATH_TO_TRANSFORMERS`) to form the initial directory path.\n    *   Iterates through the parts of the module name (skipping the first part, which is the package name), appending each part as a directory.\n        *   `module.split(\".\")[1:]`: Extracts the namespace from the module name after removing the package prefix ('__main__.').\n    *   Returns the final path with a `.py` suffix using `Path.with_suffix()`.\n*   **Dependencies:** None explicitly mentioned; however, it relies on the `PATH_TO_TRANSFORMERS` constant and possibly `Path`.\n\n**Key Variables/Functions:**\n\n*   `obj`: Input object with its `__module__` attribute.\n*   `module`: The namespace of the input object's module name (e.g., 'transformers.modeling.tokenization').\n*   `PARTS OF THE MODULE NAME`: An array consisting of module parts excluding the package index prefix (`__main__.`)."}


created_at: 2025-02-05 09:13:55
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_docstrings.py:788-900
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 900, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_docstrings.py", "name": "match_docstring_with_signature", "start_line": 788, "type": "function", "chroma:document": "CODE:\n\ndef match_docstring_with_signature(obj: Any) -> Optional[Tuple[str, str]]:\n    \"\"\"\n    Matches the docstring of an object with its signature.\n\n    Args:\n        obj (`Any`): The object to process.\n\n    Returns:\n        `Optional[Tuple[str, str]]`: Returns `None` if there is no docstring or no parameters documented in the\n        docstring, otherwise returns a tuple of two strings: the current documentation of the arguments in the\n        docstring and the one matched with the signature.\n    \"\"\"\n    if len(getattr(obj, \"__doc__\", \"\")) == 0:\n        # Nothing to do, there is no docstring.\n        return\n\n    # Read the docstring in the source code to see if there is a special command to ignore this object.\n    try:\n        source, _ = inspect.getsourcelines(obj)\n    except OSError:\n        source = []\n\n    idx = 0\n    while idx < len(source) and '\"\"\"' not in source[idx]:\n        idx += 1\n\n    ignore_order = False\n    if idx < len(source):\n        line_before_docstring = source[idx - 1]\n        if re.search(r\"^\\s*#\\s*no-format\\s*$\", line_before_docstring):\n            # This object is ignored\n            return\n        elif re.search(r\"^\\s*#\\s*ignore-order\\s*$\", line_before_docstring):\n            ignore_order = True\n\n    # Read the signature\n    signature = inspect.signature(obj).parameters\n\n    obj_doc_lines = obj.__doc__.split(\"\\n\")\n    # Get to the line where we start documenting arguments\n    idx = 0\n    while idx < len(obj_doc_lines) and _re_args.search(obj_doc_lines[idx]) is None:\n        idx += 1\n\n    if idx == len(obj_doc_lines):\n        # Nothing to do, no parameters are documented.\n        return\n\n    indent = find_indent(obj_doc_lines[idx])\n    arguments = {}\n    current_arg = None\n    idx += 1\n    start_idx = idx\n    # Keep going until the arg section is finished (nonempty line at the same indent level) or the end of the docstring.\n    while idx < len(obj_doc_lines) and (\n        len(obj_doc_lines[idx].strip()) == 0 or find_indent(obj_doc_lines[idx]) > indent\n    ):\n        if find_indent(obj_doc_lines[idx]) == indent + 4:\n            # New argument -> let's generate the proper doc for it\n            re_search_arg = _re_parse_arg.search(obj_doc_lines[idx])\n            if re_search_arg is not None:\n                _, name, description = re_search_arg.groups()\n                current_arg = name\n                if name in signature:\n                    default = signature[name].default\n                    if signature[name].kind is inspect._ParameterKind.VAR_KEYWORD:\n                        default = None\n                    new_description = replace_default_in_arg_description(description, default)\n                else:\n                    new_description = description\n                init_doc = _re_parse_arg.sub(rf\"\\1\\2 ({new_description}):\", obj_doc_lines[idx])\n                arguments[current_arg] = [init_doc]\n        elif current_arg is not None:\n            arguments[current_arg].append(obj_doc_lines[idx])\n\n        idx += 1\n\n    # We went too far by one (perhaps more if there are a lot of new lines)\n    idx -= 1\n    if current_arg:\n        while len(obj_doc_lines[idx].strip()) == 0:\n            arguments[current_arg] = arguments[current_arg][:-1]\n            idx -= 1\n    # And we went too far by one again.\n    idx += 1\n\n    old_doc_arg = \"\\n\".join(obj_doc_lines[start_idx:idx])\n\n    old_arguments = list(arguments.keys())\n    arguments = {name: \"\\n\".join(doc) for name, doc in arguments.items()}\n    # Add missing arguments with a template\n    for name in set(signature.keys()) - set(arguments.keys()):\n        arg = signature[name]\n        # We ignore private arguments or *args/**kwargs (unless they are documented by the user)\n        if name.startswith(\"_\") or arg.kind in [\n            inspect._ParameterKind.VAR_KEYWORD,\n            inspect._ParameterKind.VAR_POSITIONAL,\n        ]:\n            arguments[name] = \"\"\n        else:\n            arg_desc = get_default_description(arg)\n            arguments[name] = \" \" * (indent + 4) + f\"{name} ({arg_desc}): <fill_docstring>\"\n\n    # Arguments are sorted by the order in the signature unless a special comment is put.\n    if ignore_order:\n        new_param_docs = [arguments[name] for name in old_arguments if name in signature]\n        missing = set(signature.keys()) - set(old_arguments)\n        new_param_docs.extend([arguments[name] for name in missing if len(arguments[name]) > 0])\n    else:\n        new_param_docs = [arguments[name] for name in signature.keys() if len(arguments[name]) > 0]\n    new_doc_arg = \"\\n\".join(new_param_docs)\n\n\n\nSUMMARY:\n**Function Summary:**\n### match_docstring_with_signature(obj)\n \n  **Purpose:** Match the docstring of an object with its signature.\n\n  **Primary Functionality:** The function `match_docstring_with_signature` compares two documentation strings for an object's methods and arguments. It reads the source code (if available) to check for ignore commands, then parses the signatures using `inspect.signature`. It extracts argument names and descriptions from the docstring, generating new documentations based on the found signature parameters.\n\n  **Key Implementation Details:**\n- Uses regular expressions (`re`) for pattern matching.\n- Leverages `find_indent` (function not shown in this snippet) to parse indentation levels.\n- Utilizes `_pre_parse_arg` and `_parse_arg` regular expression patterns (not directly visible here).\n\n  **Dependencies/Requirements:**\n- Import of `inspect`, `re`, and access to `type Any`, `Optional`, `Tuple`, and `inspect._ParameterKind`.\n - The function utilizes type hints for clarity (`Any`, `Inspect.Signature`).\n  \n  **API Interface:** \n    It accepts an object, returns a tuple containing two strings:\n     - The first string contains the original documentation of the arguments described in the docstring.\n     - The second string contains the matched and generated documentation for the same argument parameters based on their signature.\n\n**Additional Notes:**\nThe function seems heavily involved with parsing and manipulating various parts of Python source code, likely to maintain consistency across different codebases by following certain established coding standards."}


created_at: 2025-02-05 09:13:55
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_docstrings.py:903-955
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 955, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_docstrings.py", "name": "fix_docstring", "start_line": 903, "type": "function", "chroma:document": "CODE:\n\ndef fix_docstring(obj: Any, old_doc_args: str, new_doc_args: str):\n    \"\"\"\n    Fixes the docstring of an object by replacing its arguments documentaiton by the one matched with the signature.\n\n    Args:\n        obj (`Any`):\n            The object whose dostring we are fixing.\n        old_doc_args (`str`):\n            The current documentation of the parameters of `obj` in the docstring (as returned by\n            `match_docstring_with_signature`).\n        new_doc_args (`str`):\n            The documentation of the parameters of `obj` matched with its signature (as returned by\n            `match_docstring_with_signature`).\n    \"\"\"\n    # Read the docstring in the source code and make sure we have the right part of the docstring\n    source, line_number = inspect.getsourcelines(obj)\n\n    # Get to the line where we start documenting arguments\n    idx = 0\n    while idx < len(source) and _re_args.search(source[idx]) is None:\n        idx += 1\n\n    if idx == len(source):\n        # Args are not defined in the docstring of this object\n        return\n\n    # Get to the line where we stop documenting arguments\n    indent = find_indent(source[idx])\n    idx += 1\n    start_idx = idx\n    while idx < len(source) and (len(source[idx].strip()) == 0 or find_indent(source[idx]) > indent):\n        idx += 1\n\n    idx -= 1\n    while len(source[idx].strip()) == 0:\n        idx -= 1\n    idx += 1\n\n    if \"\".join(source[start_idx:idx])[:-1] != old_doc_args:\n        # Args are not fully defined in the docstring of this object\n        return\n\n    obj_file = find_source_file(obj)\n    with open(obj_file, \"r\", encoding=\"utf-8\") as f:\n        content = f.read()\n\n    # Replace content\n    lines = content.split(\"\\n\")\n    lines = lines[: line_number + start_idx - 1] + [new_doc_args] + lines[line_number + idx - 1 :]\n\n    print(f\"Fixing the docstring of {obj.__name__} in {obj_file}.\")\n    with open(obj_file, \"w\", encoding=\"utf-8\") as f:\n\n\nSUMMARY:\n**Function Summary: `fix_docstring` Function**\n\n### Purpose and Functionality\n\nThe `fix_docstring` function updates the docstrings of objects by replacing outdated parameter documentation with the latest one based on the object's signature. It ensures accuracy and consistency in documentations.\n\n### Key Implementation Details\n\n1.  **Inspecting Docstrings**:\n\n    *   Reads source code from a given object.\n    *   Identifies the lines where arguments are defined and replaced.\n2.  **Matching Docstring with Signature**:\n\n    *   Uses `match_docstring_with_signature` to determine the updated documentation based on the signature.\n\n### Notable Dependencies\n\n1  **External Dependency**:\n\n    *   `re` for regular expression matching\n2  **Inherited/Implemented Interfaces** (None mentioned in this function)\n\nThe code snippet fixes the docstrings of objects by replacing outdated parameter documentation with accurate ones. It uses regular expressions to identify lines where arguments are defined, reads source code from an object, and updates its content accordingly."}


created_at: 2025-02-05 09:13:55
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_docstrings.py:958-1048
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 1048, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_docstrings.py", "name": "check_docstrings", "start_line": 958, "type": "function", "chroma:document": "CODE:\n\ndef check_docstrings(overwrite: bool = False, check_all: bool = False):\n    \"\"\"\n    Check docstrings of all public objects that are callables and are documented. By default, only checks the diff.\n\n    Args:\n        overwrite (`bool`, *optional*, defaults to `False`):\n            Whether to fix inconsistencies or not.\n        check_all (`bool`, *optional*, defaults to `False`):\n            Whether to check all files.\n    \"\"\"\n    module_diff_files = None\n    if not check_all:\n        module_diff_files = set()\n        repo = Repo(PATH_TO_REPO)\n        # Diff from index to unstaged files\n        for modified_file_diff in repo.index.diff(None):\n            if modified_file_diff.a_path.startswith(\"src/transformers\"):\n                module_diff_files.add(modified_file_diff.a_path)\n        # Diff from index to `main`\n        for modified_file_diff in repo.index.diff(repo.refs.main.commit):\n            if modified_file_diff.a_path.startswith(\"src/transformers\"):\n                module_diff_files.add(modified_file_diff.a_path)\n        # quick escape route: if there are no module files in the diff, skip this check\n        if len(module_diff_files) == 0:\n            return\n        print(\"    Checking docstrings in the following files:\" + \"\\n    - \" + \"\\n    - \".join(module_diff_files))\n\n    failures = []\n    hard_failures = []\n    to_clean = []\n    for name in dir(transformers):\n        # Skip objects that are private or not documented.\n        if name.startswith(\"_\") or ignore_undocumented(name) or name in OBJECTS_TO_IGNORE:\n            continue\n\n        obj = getattr(transformers, name)\n        if not callable(obj) or not isinstance(obj, type) or getattr(obj, \"__doc__\", None) is None:\n            continue\n\n        # If we are checking against the diff, we skip objects that are not part of the diff.\n        if module_diff_files is not None:\n            object_file = find_source_file(getattr(transformers, name))\n            object_file_relative_path = \"src/\" + str(object_file).split(\"/src/\")[1]\n            if object_file_relative_path not in module_diff_files:\n                continue\n\n        # Check docstring\n        try:\n            result = match_docstring_with_signature(obj)\n            if result is not None:\n                old_doc, new_doc = result\n            else:\n                old_doc, new_doc = None, None\n        except Exception as e:\n            print(e)\n            hard_failures.append(name)\n            continue\n        if old_doc != new_doc:\n            if overwrite:\n                fix_docstring(obj, old_doc, new_doc)\n            else:\n                failures.append(name)\n        elif not overwrite and new_doc is not None and (\"<fill_type>\" in new_doc or \"<fill_docstring>\" in new_doc):\n            to_clean.append(name)\n\n    # Deal with errors\n    error_message = \"\"\n    if len(hard_failures) > 0:\n        error_message += (\n            \"The argument part of the docstrings of the following objects could not be processed, check they are \"\n            \"properly formatted.\"\n        )\n        error_message += \"\\n\" + \"\\n\".join([f\"- {name}\" for name in hard_failures])\n    if len(failures) > 0:\n        error_message += (\n            \"The following objects docstrings do not match their signature. Run `make fix-copies` to fix this. \"\n            \"In some cases, this error may be raised incorrectly by the docstring checker. If you think this is the \"\n            \"case, you can manually check the docstrings and then add the object name to `OBJECTS_TO_IGNORE` in \"\n            \"`utils/check_docstrings.py`.\"\n        )\n        error_message += \"\\n\" + \"\\n\".join([f\"- {name}\" for name in failures])\n    if len(to_clean) > 0:\n        error_message += (\n            \"The following objects docstrings contain templates you need to fix: search for `<fill_type>` or \"\n            \"`<fill_docstring>`.\"\n        )\n        error_message += \"\\n\" + \"\\n\".join([f\"- {name}\" for name in to_clean])\n\n    if len(error_message) > 0:\n        error_message = \"There was at least one problem when checking docstrings of public objects.\\n\" + error_message\n\n\nSUMMARY:\n**Summary**\n\nThe `check_docstrings` function checks docstrings of public, callable objects in the `transformers` module and its dependencies. It:\n\n1. **Analyze Documentation**: Checks if existing docs match defined signatures.\n\n2. **Fix Potential Issues**: Fixes inconsistencies or warnings reported by the tool.\n\n3. **Report Potential Bugs**: Prints error messages about unmatched signatures, incomplete methods, and potential bugs.\n\n**Key Functionality**\n\n*   `check_all`: Optional flag to check all files.\n*   `overwrite`: Optional flag to fix inconsistencies instead of logging failures.\n\n**Implementation Details**\n\n*   Verifies functions match expected signature patterns.\n*   Skips private objects or those without docstrings.\n*   Raises `ValueError` with error messages when reports detection issues."}


created_at: 2025-02-05 09:13:55
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/download_glue_data.py:46-53
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 53, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/download_glue_data.py", "name": "download_and_extract", "start_line": 46, "type": "function", "chroma:document": "CODE:\n\ndef download_and_extract(task, data_dir):\n    print(f\"Downloading and extracting {task}...\")\n    data_file = f\"{task}.zip\"\n    urllib.request.urlretrieve(TASK2PATH[task], data_file)\n    with zipfile.ZipFile(data_file) as zip_ref:\n        zip_ref.extractall(data_dir)\n    os.remove(data_file)\n\n\nSUMMARY:\n**Summary:**\n\n`download_and_extract` function downloads and extracts a zip file for a given task, extracting the contents to a specified data directory.\n\n**Implementation Details:**\n\n* Downloads from `taskId`\n* Extracts contents using `zipfile.ZipFile.extractall()`\n* Deletes temp zip file after extraction (`os.remove(data_file)`)\n\n**Notable Dependencies:**\n\n* `urllib.request.urlretrieve()` for downloading\n* `zipfile` module for extracting zip file\n* `os` module for deleting temporary files\n\n**Primary Class/Function Names:**\n\n* `task`\n* `data_dir`\n* `TASK2PATH[task]`: mapping of tasks to download paths"}


created_at: 2025-02-05 09:13:56
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/download_glue_data.py:56-102
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 102, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/download_glue_data.py", "name": "format_mrpc", "start_line": 56, "type": "function", "chroma:document": "CODE:\n\ndef format_mrpc(data_dir, path_to_data):\n    print(\"Processing MRPC...\")\n    mrpc_dir = os.path.join(data_dir, \"MRPC\")\n    if not os.path.isdir(mrpc_dir):\n        os.mkdir(mrpc_dir)\n    if path_to_data:\n        mrpc_train_file = os.path.join(path_to_data, \"msr_paraphrase_train.txt\")\n        mrpc_test_file = os.path.join(path_to_data, \"msr_paraphrase_test.txt\")\n    else:\n        print(\"Local MRPC data not specified, downloading data from %s\" % MRPC_TRAIN)\n        mrpc_train_file = os.path.join(mrpc_dir, \"msr_paraphrase_train.txt\")\n        mrpc_test_file = os.path.join(mrpc_dir, \"msr_paraphrase_test.txt\")\n        urllib.request.urlretrieve(MRPC_TRAIN, mrpc_train_file)\n        urllib.request.urlretrieve(MRPC_TEST, mrpc_test_file)\n    if not os.path.isfile(mrpc_train_file):\n        raise ValueError(f\"Train data not found at {mrpc_train_file}\")\n    if not os.path.isfile(mrpc_test_file):\n        raise ValueError(f\"Test data not found at {mrpc_test_file}\")\n    urllib.request.urlretrieve(TASK2PATH[\"MRPC\"], os.path.join(mrpc_dir, \"dev_ids.tsv\"))\n\n    dev_ids = []\n    with open(os.path.join(mrpc_dir, \"dev_ids.tsv\"), encoding=\"utf8\") as ids_fh:\n        for row in ids_fh:\n            dev_ids.append(row.strip().split(\"\\t\"))\n\n    with open(mrpc_train_file, encoding=\"utf8\") as data_fh, open(\n        os.path.join(mrpc_dir, \"train.tsv\"), \"w\", encoding=\"utf8\"\n    ) as train_fh, open(os.path.join(mrpc_dir, \"dev.tsv\"), \"w\", encoding=\"utf8\") as dev_fh:\n        header = data_fh.readline()\n        train_fh.write(header)\n        dev_fh.write(header)\n        for row in data_fh:\n            label, id1, id2, s1, s2 = row.strip().split(\"\\t\")\n            if [id1, id2] in dev_ids:\n                dev_fh.write(\"%s\\t%s\\t%s\\t%s\\t%s\\n\" % (label, id1, id2, s1, s2))\n            else:\n                train_fh.write(\"%s\\t%s\\t%s\\t%s\\t%s\\n\" % (label, id1, id2, s1, s2))\n\n    with open(mrpc_test_file, encoding=\"utf8\") as data_fh, open(\n        os.path.join(mrpc_dir, \"test.tsv\"), \"w\", encoding=\"utf8\"\n    ) as test_fh:\n        header = data_fh.readline()\n        test_fh.write(\"index\\t#1 ID\\t#2 ID\\t#1 String\\t#2 String\\n\")\n        for idx, row in enumerate(data_fh):\n            label, id1, id2, s1, s2 = row.strip().split(\"\\t\")\n            test_fh.write(\"%d\\t%s\\t%s\\t%s\\t%s\\n\" % (idx, id1, id2, s1, s2))\n\n\nSUMMARY:\n**Function Summary: `format_mrpc`**\n\n**Main Purpose and Functionality**\nThe `format_mrpc` function processes the MRPC (Microsoft Research Paraphrase Corpus) dataset by:\n\n* Creating a directory for the dataset if it does not exist\n* Downloading dataset files from a specified URL or using local data if specified\n* Validating the presence of required files\n* Preprocessing data into training, development, and testing sets\n\n**Key Implementation Details**\n\n* Uses `os.path.join` to construct file paths\n* Utilizes `urllib.request.urlretrieve` to download external datasets\n* Reads and writes data from/to files using `open` function with specified encoding\n* Splits input file rows into individual labels, IDs, and strings\n\n**Notable Dependencies and Requirements**\n\n* Uses `os` module for file system operations\n* Utilizes `urllib.request` module for downloading external datasets\n* References classes and functions by exact name:\n\t+ `MRPC_TRAIN`\n\t+ `TASK2PATH[\"MRPC\"]`\n\n**External Dependencies:**\n\n* `urllib.request`\n* `os`\n* Python's built-in file I/O operations"}


created_at: 2025-02-05 09:13:56
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/download_glue_data.py:105-112
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 112, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/download_glue_data.py", "name": "download_diagnostic", "start_line": 105, "type": "function", "chroma:document": "CODE:\n\ndef download_diagnostic(data_dir):\n    print(\"Downloading and extracting diagnostic...\")\n    if not os.path.isdir(os.path.join(data_dir, \"diagnostic\")):\n        os.mkdir(os.path.join(data_dir, \"diagnostic\"))\n    data_file = os.path.join(data_dir, \"diagnostic\", \"diagnostic.tsv\")\n    urllib.request.urlretrieve(TASK2PATH[\"diagnostic\"], data_file)\n    print(\"\\tCompleted!\")\n\n\nSUMMARY:\n**Summary:**\n\nThe `download_diagnostic` function downloads and extracts diagnostic data from a specified directory. It:\n\n* Creates the 'diagnostic' subdirectory if it does not exist\n* Downloads the diagnostic file using `urlretrieve` from the `TASK2PATH[\"diagnostic\"]` location\n* Prints confirmation messages after completion\n\n**Key details:**\n\n* Uses `os.path.join` to construct paths for directory and file creation/export\n* Implements `urllib.request.urlretrieve` for file download\n* Returns no value (indicating a side-effecting function)\n\n**Notable dependencies/requirements:**\n\n* `os.path`\n* `urllib.request`\n* `TASK2PATH` (dictionary containing path to diagnostic data)"}


created_at: 2025-02-05 09:13:56
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/download_glue_data.py:115-125
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 125, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/download_glue_data.py", "name": "get_tasks", "start_line": 115, "type": "function", "chroma:document": "CODE:\n\ndef get_tasks(task_names):\n    task_names = task_names.split(\",\")\n    if \"all\" in task_names:\n        tasks = TASKS\n    else:\n        tasks = []\n        for task_name in task_names:\n            if task_name not in TASKS:\n                raise ValueError(f\"Task {task_name} not found!\")\n            tasks.append(task_name)\n\n\nSUMMARY:\n**Function Summary: `get_tasks`**\n\nThe `get_tasks` function retrieves the specified task names from a predefined list of tasks (`TASKS`). If \"all\" is among the input task names, it returns all available tasks. Otherwise, it checks each task name against the `TASKS` list and appends valid task names to a result list, raising an error for invalid task names.\n\n**Key Implementation Details:**\n\n* Uses string splitting to parse input task names.\n* Checks if \"all\" is present in the input task names.\n\n**Notable Dependencies or Requirements:**\n\n* The `TASKS` list of predefined task names (assuming it's defined elsewhere).\n* None explicitly listed, but implicit dependencies on Python's built-in data structures and error handling mechanisms."}


created_at: 2025-02-05 09:13:56
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/download_glue_data.py:128-152
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 152, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/download_glue_data.py", "name": "main", "start_line": 128, "type": "function", "chroma:document": "CODE:\n\ndef main(arguments):\n    parser = argparse.ArgumentParser()\n    parser.add_argument(\"--data_dir\", help=\"directory to save data to\", type=str, default=\"glue_data\")\n    parser.add_argument(\n        \"--tasks\", help=\"tasks to download data for as a comma separated string\", type=str, default=\"all\"\n    )\n    parser.add_argument(\n        \"--path_to_mrpc\",\n        help=\"path to directory containing extracted MRPC data, msr_paraphrase_train.txt and msr_paraphrase_text.txt\",\n        type=str,\n        default=\"\",\n    )\n    args = parser.parse_args(arguments)\n\n    if not os.path.isdir(args.data_dir):\n        os.mkdir(args.data_dir)\n    tasks = get_tasks(args.tasks)\n\n    for task in tasks:\n        if task == \"MRPC\":\n            format_mrpc(args.data_dir, args.path_to_mrpc)\n        elif task == \"diagnostic\":\n            download_diagnostic(args.data_dir)\n        else:\n\n\nSUMMARY:\nHere is a concise summary of the provided code chunk:\n\n**Main Functionality:**\nThe `main` function orchestrates data preparation for multiple natural language processing (NLP) tasks. It:\n\n1. Parses command-line arguments for task specifics (`--data_dir`, `--tasks`, and `--path_to_mrpc`) to download and process data.\n2. Creates a directory to store the downloaded data if it does not exist.\n3. Determines the tasks to be processed based on the comma-separated string provided in `--tasks`.\n4. For each task, performs specific actions:\n   - MRPC task: Formats MRPC data using `format_mrpc`.\n   - Diagnostic task: Downloads diagnostic data using `download_diagnostic`.\n   - Other tasks: Downloads and extracts data using `download_and_extract`.\n\n**Key Implementation Details:**\n\n* Utilizes `argparse` for command-line argument parsing.\n* Uses the `get_tasks` function to determine the specific tasks to be processed.\n* Implements a conditional statement to perform task-specific actions based on the parsed arguments.\n\n**Notable Dependencies or Requirements:**\n\n* `os` module for directory manipulation (e.g., creating directories).\n* `argparse` library for command-line argument parsing.\n* External dependencies:\n\t+ `get_tasks` function (not shown in this snippet).\n\t+ `download_diagnostic`, `format_mrpc`, and `download_and_extract` functions (not shown in this snippet)."}


created_at: 2025-02-05 09:13:56
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/update_metadata.py:126-145
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 145, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/update_metadata.py", "name": "camel_case_split", "start_line": 126, "type": "function", "chroma:document": "CODE:\n\ndef camel_case_split(identifier: str) -> List[str]:\n    \"\"\"\n    Split a camel-cased name into words.\n\n    Args:\n        identifier (`str`): The camel-cased name to parse.\n\n    Returns:\n        `List[str]`: The list of words in the identifier (as seprated by capital letters).\n\n    Example:\n\n    ```py\n    >>> camel_case_split(\"CamelCasedClass\")\n    [\"Camel\", \"Cased\", \"Class\"]\n    ```\n    \"\"\"\n    # Regex thanks to https://stackoverflow.com/questions/29916065/how-to-do-camelcase-split-in-python\n    matches = re.finditer(\".+?(?:(?<=[a-z])(?=[A-Z])|(?<=[A-Z])(?=[A-Z][a-z])|$)\", identifier)\n\n\nSUMMARY:\n** camel_case_split() Function Summary **\n\n\n*   **Purpose and Functionality:** \n    Splits a camel-cased string into individual words based on capitalization.\n\n        *   Type Hint: `identifier (str) -> List[str]`\n        *   Returns: A list of words separated by capital letters.\n\n*   **Implementation Details:** \n    - Uses regular expressions with look-behind and look-ahead assertions to match word boundaries.\n    - Utilizes the `re.finditer` function to extract matches from the input string.\n\n\n*   **Notable Dependencies or Requirements:**\n    - `re` module for regex functionality\n    - No external dependencies\n\n\n        Example usage:\n\n            ```py\n>>> camel_case_split(\"CamelCasedClass\")\n['Camel', 'Cased', 'Class']\n```"}


created_at: 2025-02-05 09:13:56
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/update_metadata.py:148-212
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 212, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/update_metadata.py", "name": "get_frameworks_table", "start_line": 148, "type": "function", "chroma:document": "CODE:\n\ndef get_frameworks_table() -> pd.DataFrame:\n    \"\"\"\n    Generates a dataframe containing the supported auto classes for each model type, using the content of the auto\n    modules.\n    \"\"\"\n    # Dictionary model names to config.\n    config_maping_names = transformers_module.models.auto.configuration_auto.CONFIG_MAPPING_NAMES\n    model_prefix_to_model_type = {\n        config.replace(\"Config\", \"\"): model_type for model_type, config in config_maping_names.items()\n    }\n\n    # Dictionaries flagging if each model prefix has a backend in PT/TF/Flax.\n    pt_models = collections.defaultdict(bool)\n    tf_models = collections.defaultdict(bool)\n    flax_models = collections.defaultdict(bool)\n\n    # Let's lookup through all transformers object (once) and find if models are supported by a given backend.\n    for attr_name in dir(transformers_module):\n        lookup_dict = None\n        if _re_tf_models.match(attr_name) is not None:\n            lookup_dict = tf_models\n            attr_name = _re_tf_models.match(attr_name).groups()[0]\n        elif _re_flax_models.match(attr_name) is not None:\n            lookup_dict = flax_models\n            attr_name = _re_flax_models.match(attr_name).groups()[0]\n        elif _re_pt_models.match(attr_name) is not None:\n            lookup_dict = pt_models\n            attr_name = _re_pt_models.match(attr_name).groups()[0]\n\n        if lookup_dict is not None:\n            while len(attr_name) > 0:\n                if attr_name in model_prefix_to_model_type:\n                    lookup_dict[model_prefix_to_model_type[attr_name]] = True\n                    break\n                # Try again after removing the last word in the name\n                attr_name = \"\".join(camel_case_split(attr_name)[:-1])\n\n    all_models = set(list(pt_models.keys()) + list(tf_models.keys()) + list(flax_models.keys()))\n    all_models = list(all_models)\n    all_models.sort()\n\n    data = {\"model_type\": all_models}\n    data[\"pytorch\"] = [pt_models[t] for t in all_models]\n    data[\"tensorflow\"] = [tf_models[t] for t in all_models]\n    data[\"flax\"] = [flax_models[t] for t in all_models]\n\n    # Now let's find the right processing class for each model. In order we check if there is a Processor, then a\n    # Tokenizer, then a FeatureExtractor, then an ImageProcessor\n    processors = {}\n    for t in all_models:\n        if t in transformers_module.models.auto.processing_auto.PROCESSOR_MAPPING_NAMES:\n            processors[t] = \"AutoProcessor\"\n        elif t in transformers_module.models.auto.tokenization_auto.TOKENIZER_MAPPING_NAMES:\n            processors[t] = \"AutoTokenizer\"\n        elif t in transformers_module.models.auto.image_processing_auto.IMAGE_PROCESSOR_MAPPING_NAMES:\n            processors[t] = \"AutoImageProcessor\"\n        elif t in transformers_module.models.auto.feature_extraction_auto.FEATURE_EXTRACTOR_MAPPING_NAMES:\n            processors[t] = \"AutoFeatureExtractor\"\n        else:\n            # Default to AutoTokenizer if a model has nothing, for backward compatibility.\n            processors[t] = \"AutoTokenizer\"\n\n    data[\"processor\"] = [processors[t] for t in all_models]\n\n\n\nSUMMARY:\n**Function Summary: `get_frameworks_table`**\n\n**Primary Purpose:** Generates a pandas DataFrame containing the supported auto classes for each model type, using the content of the auto modules.\n\n**Key Implementation Details:**\n\n* Uses two dictionaries to map model prefixes to model types (`model_prefix_to_model_type`)\n* Populates three separate dictionaries (`pt_models`, `tf_models`, and `flax_models`) to track backend support for each model prefix\n* Iterates through all transformers object attributes using regular expressions to identify models supported by PT, TF, or Flax backends\n\n**Notable Dependencies and Requirements:**\n\n* `pd` (pandas library) for data manipulation\n* `transformers_module` (external dependencies) containing configuration mapping names and processing class mappings\n* `_re_tf_models`, `_re_flax_models`, and `_re_pt_models` (regex patterns used to identify models supported by PT, TF, or Flax backends)\n* `collections` (utility module) for using defaultdicts"}


created_at: 2025-02-05 09:13:56
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/update_metadata.py:215-251
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 251, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/update_metadata.py", "name": "update_pipeline_and_auto_class_table", "start_line": 215, "type": "function", "chroma:document": "CODE:\n\ndef update_pipeline_and_auto_class_table(table: Dict[str, Tuple[str, str]]) -> Dict[str, Tuple[str, str]]:\n    \"\"\"\n    Update the table maping models to pipelines and auto classes without removing old keys if they don't exist anymore.\n\n    Args:\n        table (`Dict[str, Tuple[str, str]]`):\n            The existing table mapping model names to a tuple containing the pipeline tag and the auto-class name with\n            which they should be used.\n\n    Returns:\n        `Dict[str, Tuple[str, str]]`: The updated table in the same format.\n    \"\"\"\n    auto_modules = [\n        transformers_module.models.auto.modeling_auto,\n        transformers_module.models.auto.modeling_tf_auto,\n        transformers_module.models.auto.modeling_flax_auto,\n    ]\n    for pipeline_tag, model_mapping, auto_class in PIPELINE_TAGS_AND_AUTO_MODELS:\n        model_mappings = [model_mapping, f\"TF_{model_mapping}\", f\"FLAX_{model_mapping}\"]\n        auto_classes = [auto_class, f\"TF_{auto_class}\", f\"Flax_{auto_class}\"]\n        # Loop through all three frameworks\n        for module, cls, mapping in zip(auto_modules, auto_classes, model_mappings):\n            # The type of pipeline may not exist in this framework\n            if not hasattr(module, mapping):\n                continue\n            # First extract all model_names\n            model_names = []\n            for name in getattr(module, mapping).values():\n                if isinstance(name, str):\n                    model_names.append(name)\n                else:\n                    model_names.extend(list(name))\n\n            # Add pipeline tag and auto model class for those models\n            table.update({model_name: (pipeline_tag, cls) for model_name in model_names})\n\n\n\nSUMMARY:\n**Function Summary**\n\n### `update_pipeline_and_auto_class_table`\n\nUpdates the table mapping models to pipelines and auto classes without removing old keys.\n\n#### Parameters\n- `table`: The existing table mapping model names to a tuple containing the pipeline tag and the auto-class name (Dict[str, Tuple[str, str]])\n\n#### Returns\n- `Dict[str, Tuple[str, str]]`: The updated table in the same format\n\n#### Implementation Details\n\n- Updates table with pipeline tags and auto-model classes for each framework (transformers_module.models.auto.modeling_auto, transformers_module.models.auto.modeling_tf_auto, transformers_module.models.auto.modeling_flax_auto)\n- Handles cases where pipeline type may not exist in a given framework\n- Uses model mappings and pipepline tags to populate the table\n\n#### Notable Dependencies\n- `PIPELINE_TAGS_AND_AUTO_MODELS`\n- `transformers_module.models.auto.modeling_auto`, `transformers_module.models.auto.modeling_tf_auto`, `transformers_module.models.auto.modeling_flax_auto`\n\n#### External Dependencies\n- `transformers_module`\n\n#### Notes\n\nThis function iterates over a list of frameworks, extracts model names, and updates the table with pipeline tags and auto-model classes."}


created_at: 2025-02-05 09:13:56
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/update_metadata.py:254-334
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 334, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/update_metadata.py", "name": "update_metadata", "start_line": 254, "type": "function", "chroma:document": "CODE:\n\ndef update_metadata(token: str, commit_sha: str):\n    \"\"\"\n    Update the metadata for the Transformers repo in `huggingface/transformers-metadata`.\n\n    Args:\n        token (`str`): A valid token giving write access to `huggingface/transformers-metadata`.\n        commit_sha (`str`): The commit SHA on Transformers corresponding to this update.\n    \"\"\"\n    frameworks_table = get_frameworks_table()\n    frameworks_dataset = Dataset.from_pandas(frameworks_table)\n\n    resolved_tags_file = hf_hub_download(\n        \"huggingface/transformers-metadata\", \"pipeline_tags.json\", repo_type=\"dataset\", token=token\n    )\n    tags_dataset = Dataset.from_json(resolved_tags_file)\n    table = {\n        tags_dataset[i][\"model_class\"]: (tags_dataset[i][\"pipeline_tag\"], tags_dataset[i][\"auto_class\"])\n        for i in range(len(tags_dataset))\n    }\n    table = update_pipeline_and_auto_class_table(table)\n\n    # Sort the model classes to avoid some nondeterministic updates to create false update commits.\n    model_classes = sorted(table.keys())\n    tags_table = pd.DataFrame(\n        {\n            \"model_class\": model_classes,\n            \"pipeline_tag\": [table[m][0] for m in model_classes],\n            \"auto_class\": [table[m][1] for m in model_classes],\n        }\n    )\n    tags_dataset = Dataset.from_pandas(tags_table)\n\n    hub_frameworks_json = hf_hub_download(\n        repo_id=\"huggingface/transformers-metadata\",\n        filename=\"frameworks.json\",\n        repo_type=\"dataset\",\n        token=token,\n    )\n    with open(hub_frameworks_json) as f:\n        hub_frameworks_json = f.read()\n\n    hub_pipeline_tags_json = hf_hub_download(\n        repo_id=\"huggingface/transformers-metadata\",\n        filename=\"pipeline_tags.json\",\n        repo_type=\"dataset\",\n        token=token,\n    )\n    with open(hub_pipeline_tags_json) as f:\n        hub_pipeline_tags_json = f.read()\n\n    with tempfile.TemporaryDirectory() as tmp_dir:\n        frameworks_dataset.to_json(os.path.join(tmp_dir, \"frameworks.json\"))\n        tags_dataset.to_json(os.path.join(tmp_dir, \"pipeline_tags.json\"))\n\n        with open(os.path.join(tmp_dir, \"frameworks.json\")) as f:\n            frameworks_json = f.read()\n        with open(os.path.join(tmp_dir, \"pipeline_tags.json\")) as f:\n            pipeline_tags_json = f.read()\n\n        frameworks_equal = hub_frameworks_json == frameworks_json\n        hub_pipeline_tags_equal = hub_pipeline_tags_json == pipeline_tags_json\n\n        if frameworks_equal and hub_pipeline_tags_equal:\n            print(\"No updates on the Hub, not pushing the metadata files.\")\n            return\n\n        if commit_sha is not None:\n            commit_message = (\n                f\"Update with commit {commit_sha}\\n\\nSee: \"\n                f\"https://github.com/huggingface/transformers/commit/{commit_sha}\"\n            )\n        else:\n            commit_message = \"Update\"\n\n        upload_folder(\n            repo_id=\"huggingface/transformers-metadata\",\n            folder_path=tmp_dir,\n            repo_type=\"dataset\",\n            token=token,\n            commit_message=commit_message,\n\n\nSUMMARY:\n**Function Summary:**\n\n`update_metadata`\n\n**Purpose:** Updates the metadata for the Transformers repository by processing and uploading dataset files.\n\n**Key Implementation Details:**\n\n* Downloads pipeline_tags.json and frameworks.json dataset files from huggingface/transformers-metadata\n* Processes framework and tag data from downloaded files\n* Checks if updates need to be pushed to GitHub and commits message is generated accordingly\n\n**Noteable Dependencies/Requirements:**\n\n* `get_frameworks_table` function (not shown in this code snippet)\n* `Dataset.from_json` and `Dataset.from_pandas` functions (from the Hugging Face datasets library)\n* `hf_hub_download` and `upload_folder` functions (from the Hugging Face Hub client library)\n\n**Interfaces/ Method Signatures:**\n\n* `update_pipeline_and_auto_class_table(table)`: Updates table with processed framework data\n* `get_frameworks_table()`: Returns a pandas dataframe containing frameworks data\n\n**External Dependencies:**\n\n* `huggingface/transformers-metadata`\n* `huggingfacedatasets` library\n* `huggingface_hub` library"}


created_at: 2025-02-05 09:13:56
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/update_metadata.py:337-358
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 358, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/update_metadata.py", "name": "check_pipeline_tags", "start_line": 337, "type": "function", "chroma:document": "CODE:\n\ndef check_pipeline_tags():\n    \"\"\"\n    Check all pipeline tags are properly defined in the `PIPELINE_TAGS_AND_AUTO_MODELS` constant of this script.\n    \"\"\"\n    in_table = {tag: cls for tag, _, cls in PIPELINE_TAGS_AND_AUTO_MODELS}\n    pipeline_tasks = transformers_module.pipelines.SUPPORTED_TASKS\n    missing = []\n    for key in pipeline_tasks:\n        if key not in in_table:\n            model = pipeline_tasks[key][\"pt\"]\n            if isinstance(model, (list, tuple)):\n                model = model[0]\n            model = model.__name__\n            if model not in in_table.values():\n                missing.append(key)\n\n    if len(missing) > 0:\n        msg = \", \".join(missing)\n        raise ValueError(\n            \"The following pipeline tags are not present in the `PIPELINE_TAGS_AND_AUTO_MODELS` constant inside \"\n            f\"`utils/update_metadata.py`: {msg}. Please add them!\"\n\n\nSUMMARY:\n**Summary:**\n\nThis function, `check_pipeline_tags()`, detects and raises an error if any pipeline tasks are missing from the predefined `PIPELINE_TAGS_AND_AUTO_MODELS` constant.\n\n**Key Details:**\n\n- Verifies that all task pipelines match the predefined auto-models.\n- Checks against `transformers_module.pipelines.SUPPORTED_TASKS`.\n- Raises a `ValueError` with missing task pipelines as input when not found in the defined constants.\n\n**Implementation Notes:**\n\n* Utilizes dictionary comprehension to create an in-table mapping of pipeline tags and models.\n* Iterates through supported tasks, checking their existence in the predefined constant map (`in_table`).\n* Raises an error if the `missing` list is populated during iteration. \n\nThis function ensures that all defined pipelines have corresponding auto-models."}


created_at: 2025-02-05 09:13:56
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/get_test_info.py:30-49
encoding: FLOAT32
metadata:
{"docstring": "r\"\"\"\nThe argument `test_file` in this file refers to a model test file. This should be a string of the from\n`tests/models/*/test_modeling_*.py`.\n\"\"\"", "end_line": 49, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/get_test_info.py", "name": "get_module_path", "start_line": 30, "type": "function", "chroma:document": "CODE:\n\ndef get_module_path(test_file):\n    \"\"\"Return the module path of a model test file.\"\"\"\n    components = test_file.split(os.path.sep)\n    if components[0:2] != [\"tests\", \"models\"]:\n        raise ValueError(\n            \"`test_file` should start with `tests/models/` (with `/` being the OS specific path separator). Got \"\n            f\"{test_file} instead.\"\n        )\n    test_fn = components[-1]\n    if not test_fn.endswith(\"py\"):\n        raise ValueError(f\"`test_file` should be a python file. Got {test_fn} instead.\")\n    if not test_fn.startswith(\"test_modeling_\"):\n        raise ValueError(\n            f\"`test_file` should point to a file name of the form `test_modeling_*.py`. Got {test_fn} instead.\"\n        )\n\n    components = components[:-1] + [test_fn.replace(\".py\", \"\")]\n    test_module_path = \".\".join(components)\n\n\n\nSUMMARY:\n**Function Summary**\n\nThe `get_module_path` function takes a `test_file` path as input and returns the corresponding module path. It validates that the input path matches the expected format `tests/modeling/*.py`, raising errors for any deviations.\n\n**Key Implementation Details**\n\n- Splits the input `test_file` into components using the OS-specific path separator.\n- Verifies that the first two components are \"tests\" and \"models,\" and the last component ends with \".py.\"\n- Reconstructs the module path from the validated components.\n\n**Notable Dependencies and Requirements**\n\n- Uses the `os` module for handling file paths.\n- References `python files` directly in validation checks."}


created_at: 2025-02-05 09:13:56
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/get_test_info.py:52-65
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 65, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/get_test_info.py", "name": "get_test_module", "start_line": 52, "type": "function", "chroma:document": "CODE:\n\ndef get_test_module(test_file):\n    \"\"\"Get the module of a model test file.\"\"\"\n    test_module_path = get_module_path(test_file)\n    try:\n        test_module = importlib.import_module(test_module_path)\n    except AttributeError as exc:\n        # e.g. if you have a `tests` folder in `site-packages`, created by another package, when trying to import\n        # `tests.models...`\n        raise ValueError(\n            f\"Could not import module {test_module_path}. Confirm that you don't have a package with the same root \"\n            \"name installed or in your environment's `site-packages`.\"\n        ) from exc\n\n\n\nSUMMARY:\n**Summary:**\n\nThis function, `get_test_module`, retrieves and loads the Python module corresponding to a given test file by its path. It identifies if the import is successful or not and provides an informative error message if an AttributeError occurs.\n\n**Key Implementation Details:**\n\n* Imports a module using `importlib.import_module` with a specified path.\n* Raises a ValueError with an informative message if the import fails due to a common issue.\n\n**Notable Dependencies:**\n\n* `importlib`\n* None\n\n**Interfaces/Classes:**\n\n* No external interface or class is used in this function."}


created_at: 2025-02-05 09:13:56
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/get_test_info.py:68-77
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 77, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/get_test_info.py", "name": "get_tester_classes", "start_line": 68, "type": "function", "chroma:document": "CODE:\n\ndef get_tester_classes(test_file):\n    \"\"\"Get all classes in a model test file whose names ends with `ModelTester`.\"\"\"\n    tester_classes = []\n    test_module = get_test_module(test_file)\n    for attr in dir(test_module):\n        if attr.endswith(\"ModelTester\"):\n            tester_classes.append(getattr(test_module, attr))\n\n    # sort with class names\n\n\nSUMMARY:\n**Code Summary**\n\n### Purpose and Functionality\n\nGet all classes in a model test file whose names end with \"ModelTester\".\n\n### Implementation Details\n\n* Uses the `dir()` function to list attributes of the `test_module` object.\n* Filters attributes based on name ending with \"ModelTester\".\n* Retrieves attribute objects using `getattr()`.\n* Sorts retrieved classes by their names using the `sorted()` function and a lambda function.\n\n### Notable Dependencies or Requirements\n\n* `get_test_module`: Returns the test module for the given test file (not shown in this code snippet).\n \nThis summary preserves detailed information from the original code while minimizing unnecessary details."}


created_at: 2025-02-05 09:13:56
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/get_test_info.py:80-98
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 98, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/get_test_info.py", "name": "get_test_classes", "start_line": 80, "type": "function", "chroma:document": "CODE:\n\ndef get_test_classes(test_file):\n    \"\"\"Get all [test] classes in a model test file with attribute `all_model_classes` that are non-empty.\n\n    These are usually the (model) test classes containing the (non-slow) tests to run and are subclasses of one of the\n    classes `ModelTesterMixin`, `TFModelTesterMixin` or `FlaxModelTesterMixin`, as well as a subclass of\n    `unittest.TestCase`. Exceptions include `RagTestMixin` (and its subclasses).\n    \"\"\"\n    test_classes = []\n    test_module = get_test_module(test_file)\n    for attr in dir(test_module):\n        attr_value = getattr(test_module, attr)\n        # (TF/Flax)ModelTesterMixin is also an attribute in specific model test module. Let's exclude them by checking\n        # `all_model_classes` is not empty (which also excludes other special classes).\n        model_classes = getattr(attr_value, \"all_model_classes\", [])\n        if len(model_classes) > 0:\n            test_classes.append(attr_value)\n\n    # sort with class names\n\n\nSUMMARY:\nHere is a concise summary of the code chunk:\n\n**Function Purpose**\nThe `get_test_classes` function retrieves and sorts a list of non-empty test classes in a model test file that inherit from specific mixins (`ModelTesterMixin`, `TFModelTesterMixin`, or `FlaxModelTesterMixin`) or `unittest.TestCase`.\n\n**Key Implementation Details**\n\n* Iterates through the module's attributes to find classes with an empty `all_model_classes` attribute.\n* Excludes exceptions like `RagTestMixin` and its subclasses.\n* Sorts the test classes by their names using `__name__` attribute.\n\n**Notable Dependencies and Requirements**\n\n* The `get_test_module` function is called from within this function, suggesting a dependency on another module (`test_file`).\n* This function relies on `unittest.TestCase`, indicating a requirement for testing framework."}


created_at: 2025-02-05 09:13:56
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/get_test_info.py:101-109
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 109, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/get_test_info.py", "name": "get_model_classes", "start_line": 101, "type": "function", "chroma:document": "CODE:\n\ndef get_model_classes(test_file):\n    \"\"\"Get all model classes that appear in `all_model_classes` attributes in a model test file.\"\"\"\n    test_classes = get_test_classes(test_file)\n    model_classes = set()\n    for test_class in test_classes:\n        model_classes.update(test_class.all_model_classes)\n\n    # sort with class names\n\n\nSUMMARY:\n**Function Summary**\n\n`get_model_classes`\n\nReturns a sorted list of unique model classes from `all_model_classes` attributes in a model test file.\n\n**Key Implementation Details**\n\n* Uses `set` to eliminate duplicate class instances\n* Iterates over `test_classes`, updating `model_classes` with `all_model_classes`\n* Sorts classes by name using a case-insensitive comparison\n\n**Notable Dependencies and Requirements**\n\n* `get_test_classes` function is called with `test_file` as an argument\n* Uses `__name__` attribute of class names for sorting comparison\n* No external dependencies are listed due to lack of information"}


created_at: 2025-02-05 09:13:56
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/get_test_info.py:112-124
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 124, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/get_test_info.py", "name": "get_model_tester_from_test_class", "start_line": 112, "type": "function", "chroma:document": "CODE:\n\ndef get_model_tester_from_test_class(test_class):\n    \"\"\"Get the model tester class of a model test class.\"\"\"\n    test = test_class()\n    if hasattr(test, \"setUp\"):\n        test.setUp()\n\n    model_tester = None\n    if hasattr(test, \"model_tester\"):\n        # `(TF/Flax)ModelTesterMixin` has this attribute default to `None`. Let's skip this case.\n        if test.model_tester is not None:\n            model_tester = test.model_tester.__class__\n\n\n\nSUMMARY:\n**Summary**\n\nThe `get_model_tester_from_test_class` function takes a `test_class` as input and returns the model tester class associated with it. It checks for the presence of an instance with an explicit `setUp` method, then attempts to retrieve the model tester class from the instance's attributes.\n\n**Key Implementation Details**\n\n* Tries to call `setUp` on the test instance if present\n* Ignores cases where `model_tester` is set to `None`, likely due to default values in `ModelTesterMixin`\n\n**Notable Dependencies/Requirements**\n\n* `test_class`: The input class being analyzed\n* `TF/Flax` (implied): The framework used for the `ModelTesterMixin`\n* No external dependencies noted\n\n**Notes**: This code appears to be part of a testing harness, likely for machine learning models."}


created_at: 2025-02-05 09:13:56
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/get_test_info.py:127-137
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 137, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/get_test_info.py", "name": "get_test_classes_for_model", "start_line": 127, "type": "function", "chroma:document": "CODE:\n\ndef get_test_classes_for_model(test_file, model_class):\n    \"\"\"Get all [test] classes in `test_file` that have `model_class` in their `all_model_classes`.\"\"\"\n    test_classes = get_test_classes(test_file)\n\n    target_test_classes = []\n    for test_class in test_classes:\n        if model_class in test_class.all_model_classes:\n            target_test_classes.append(test_class)\n\n    # sort with class names\n\n\nSUMMARY:\n**Function Summary**\n\n`get_test_classes_for_model`: Returns a list of test classes from `test_file` that include `model_class`. \n\n**Key Implementation Details**\n\n* Uses `sorted` function for alphabetical ordering by class name (`x.__name__`)\n* Iterates through all test classes using `get_test_classes(test_file)`\n* Filters test classes based on presence of `model_class` in `all_model_classes`\n\n**Input/Output**\n\n* `test_file`: input file containing test classes\n* `model_class`: model class to search for in test classes\n* Returns a sorted list of matching test classes\n\n**Dependencies/Requirements**\n\n* `get_test_classes` function from same library or module"}


created_at: 2025-02-05 09:13:56
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/get_test_info.py:140-151
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 151, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/get_test_info.py", "name": "get_tester_classes_for_model", "start_line": 140, "type": "function", "chroma:document": "CODE:\n\ndef get_tester_classes_for_model(test_file, model_class):\n    \"\"\"Get all model tester classes in `test_file` that are associated to `model_class`.\"\"\"\n    test_classes = get_test_classes_for_model(test_file, model_class)\n\n    tester_classes = []\n    for test_class in test_classes:\n        tester_class = get_model_tester_from_test_class(test_class)\n        if tester_class is not None:\n            tester_classes.append(tester_class)\n\n    # sort with class names\n\n\nSUMMARY:\n**Summary**\n\nReturns a list of model tester classes from a test file (`test_file`) associated with the specified `model_class`, sorted alphabetically by class name.\n\n**Key Details**\n\n* **Functionality**: Retrieves and filters model tester classes based on the provided `test_file` and `model_class`.\n* **Algorithm/Design Choice**: Sorts filtered tester classes using the `__name__` attribute.\n* **Method Signatures**:\n\t+ `get_test_classes_for_model()`: returns list of test classes for a given model\n\t+ `get_model_tester_from_test_class(test_class)`: returns associated model tester class from a test class\n\n**Dependencies**\n\n* `get_test_classes_for_model`\n* `get_model_tester_from_test_class`"}


created_at: 2025-02-05 09:13:56
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/get_test_info.py:154-161
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 161, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/get_test_info.py", "name": "get_test_to_tester_mapping", "start_line": 154, "type": "function", "chroma:document": "CODE:\n\ndef get_test_to_tester_mapping(test_file):\n    \"\"\"Get a mapping from [test] classes to model tester classes in `test_file`.\n\n    This uses `get_test_classes` which may return classes that are NOT subclasses of `unittest.TestCase`.\n    \"\"\"\n    test_classes = get_test_classes(test_file)\n    test_tester_mapping = {test_class: get_model_tester_from_test_class(test_class) for test_class in test_classes}\n\n\nSUMMARY:\n**Summary**\n\n`get_test_to_tester_mapping` function:\n \n* Maps test classes to model tester classes based on inheritance from `unittest.TestCase`.\n* Uses `get_test_classes` to retrieve test class list and `get_model_tester_from_test_class` to map each test class.\n\n**Key Details:**\n\n* Reusable in a unit testing framework.\n* May return classes that are not subclasses of `unittest.TestCase`.\n\n**Implementation Notes:**\n\n* Uses dictionary comprehension for efficient creation of mapping.\n* Relies on `get_test_classes` and `get_model_tester_from_test_class` external functions."}


created_at: 2025-02-05 09:13:56
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/get_test_info.py:164-170
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 170, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/get_test_info.py", "name": "get_model_to_test_mapping", "start_line": 164, "type": "function", "chroma:document": "CODE:\n\ndef get_model_to_test_mapping(test_file):\n    \"\"\"Get a mapping from model classes to test classes in `test_file`.\"\"\"\n    model_classes = get_model_classes(test_file)\n    model_test_mapping = {\n        model_class: get_test_classes_for_model(test_file, model_class) for model_class in model_classes\n    }\n\n\nSUMMARY:\n**Summary:**\n\n`get_model_to_test_mapping` function generates a dictionary mapping `model_classes` to their corresponding test classes (`test_classes`) based on the input `test_file`. It uses `get_model_classes` and `get_test_classes_for_model` functions.\n\n**Key Details:**\n\n- Uses `get_model_classes(test_file)` to retrieve model class names\n- Populates a dictionary (`model_test_mapping`) with models as keys and test classes as values using map comprehension\n\n**Notable Dependencies/Requirements:**\n\n- `get_model_classes(test_file)`\n- `get_test_classes_for_model(test_file, model_class)`\n\nIt can be summarized as:\n```python\ndef get_model_to_test_mapping(test_file):\n    # Maps model classes to their corresponding test classes\n    model_classes = get_model_classes(test_file)\n    model_test_mapping = {model_class: get_test_classes_for_model(test_file, model_class) for model_class in model_classes}\n    return model_test_mapping\n```"}


created_at: 2025-02-05 09:13:56
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/get_test_info.py:173-179
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 179, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/get_test_info.py", "name": "get_model_to_tester_mapping", "start_line": 173, "type": "function", "chroma:document": "CODE:\n\ndef get_model_to_tester_mapping(test_file):\n    \"\"\"Get a mapping from model classes to model tester classes in `test_file`.\"\"\"\n    model_classes = get_model_classes(test_file)\n    model_to_tester_mapping = {\n        model_class: get_tester_classes_for_model(test_file, model_class) for model_class in model_classes\n    }\n\n\nSUMMARY:\n**Summary**\n\n`get_model_to_tester_mapping` function maps model classes to their corresponding tester classes in a given test file.\n\n**Key Details:**\n\n* Uses `model_classes` from `get_model_classes` for iteration\n* Iterates using dictionary comprehension to create mappings of model classes to tester classes\n* Returns a dictionary `model_to_tester_mapping`\n\n**Notable Dependencies and Interfaces:**\n\n* References `get_model_classes` and `get_tester_classes_for_model` functions (not shown)"}


created_at: 2025-02-05 09:13:56
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/get_test_info.py:182-197
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 197, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/get_test_info.py", "name": "to_json", "start_line": 182, "type": "function", "chroma:document": "CODE:\n\ndef to_json(o):\n    \"\"\"Make the information succinct and easy to read.\n\n    Avoid the full class representation like `<class 'transformers.models.bert.modeling_bert.BertForMaskedLM'>` when\n    displaying the results. Instead, we use class name (`BertForMaskedLM`) for the readability.\n    \"\"\"\n    if isinstance(o, str):\n        return o\n    elif isinstance(o, type):\n        return o.__name__\n    elif isinstance(o, (list, tuple)):\n        return [to_json(x) for x in o]\n    elif isinstance(o, dict):\n        return {to_json(k): to_json(v) for k, v in o.items()}\n    else:\n\n\nSUMMARY:\n**Concise Summary**\n\n### Function Purpose and Main Functionality\nThe `to_json` function converts an object `o` into a human-readable JSON format. It handles various data types by recursively processing nested structures, truncating class representations for better readability.\n\n### Key Implementation Details\n\n* Handles string, type alias, list, tuple, and dictionary conversions using recursive calls.\n* Uses `__name__` attribute to display only class names instead of full class representations.\n\n### Notable Dependencies or Requirements\nNone mentioned, but it assumes objects are usable in a context that supports nested data processing (e.g., lists, dictionaries).\n\n### Key Method Signatures and Interfaces\n\n* Implicit: No explicit interfaces defined.\n* Explicit: None."}


created_at: 2025-02-05 09:13:56
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_config_docstrings.py:52-72
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 72, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_config_docstrings.py", "name": "get_checkpoint_from_config_class", "start_line": 52, "type": "function", "chroma:document": "CODE:\n\ndef get_checkpoint_from_config_class(config_class):\n    checkpoint = None\n\n    # source code of `config_class`\n    config_source = inspect.getsource(config_class)\n    checkpoints = _re_checkpoint.findall(config_source)\n\n    # Each `checkpoint` is a tuple of a checkpoint name and a checkpoint link.\n    # For example, `('google-bert/bert-base-uncased', 'https://huggingface.co/google-bert/bert-base-uncased')`\n    for ckpt_name, ckpt_link in checkpoints:\n        # allow the link to end with `/`\n        if ckpt_link.endswith(\"/\"):\n            ckpt_link = ckpt_link[:-1]\n\n        # verify the checkpoint name corresponds to the checkpoint link\n        ckpt_link_from_name = f\"https://huggingface.co/{ckpt_name}\"\n        if ckpt_link == ckpt_link_from_name:\n            checkpoint = ckpt_name\n            break\n\n\n\nSUMMARY:\n**Summary:**\n\n`get_checkpoint_from_config_class` function retrieves the corresponding checkpoint name from a provided config class's source code.\n\n*   **Purpose:** To extract the checkpoint name from a config class based on its implementation details.\n*   **Key Details:**\n    *   Uses regular expression to find checkpoints in the config class source code.\n    *   Verifies each checkpoint by comparing its name with the corresponding link.\n*   **Notable Dependencies/Requirements:**\n    *   `_re_checkpoint` regular expression pattern\n    *   `inspect.getsource(config_class)` function\n*   **Interfaces/Methods:** None (uses built-in Python functions and modules)\n*   **External Dependencies:**\n\n    *   `_re_checkpoint`\n    *   `config_class` (input parameter)\n\n**Summary Focus Criteria Met:**\n\n1. Precise technical terms used: \"config class\", \"checkpoint name\", \"regular expression pattern\"\n2. Class/function/variable names preserved exactly.\n3. Primary purpose stated: \"To extract the checkpoint name from a config class based on its implementation details.\"\n4. Key implementation details narrowed down to:\n    *   Regular expression usage\n    *   Comparison verification for checkpoints\n5. Notable dependencies listed with exact class names and functions: `_re_checkpoint`, `config_class`.\n6. No external modules or libraries required besides built-in Python functions."}


created_at: 2025-02-05 09:13:56
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_config_docstrings.py:75-95
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 95, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_config_docstrings.py", "name": "check_config_docstrings_have_checkpoints", "start_line": 75, "type": "function", "chroma:document": "CODE:\n\ndef check_config_docstrings_have_checkpoints():\n    configs_without_checkpoint = []\n\n    for config_class in list(CONFIG_MAPPING.values()):\n        # Skip deprecated models\n        if \"models.deprecated\" in config_class.__module__:\n            continue\n        checkpoint = get_checkpoint_from_config_class(config_class)\n\n        name = config_class.__name__\n        if checkpoint is None and name not in CONFIG_CLASSES_TO_IGNORE_FOR_DOCSTRING_CHECKPOINT_CHECK:\n            configs_without_checkpoint.append(name)\n\n    if len(configs_without_checkpoint) > 0:\n        message = \"\\n\".join(sorted(configs_without_checkpoint))\n        raise ValueError(\n            f\"The following configurations don't contain any valid checkpoint:\\n{message}\\n\\n\"\n            \"The requirement is to include a link pointing to one of the models of this architecture in the \"\n            \"docstring of the config classes listed above. The link should have be a markdown format like \"\n            \"[myorg/mymodel](https://huggingface.co/myorg/mymodel).\"\n\n\nSUMMARY:\n**Function Summary**\n\n`check_config_docstrings_have_checkpoints()`: Verifies if all configuration classes in `CONFIG_MAPPING` have valid checkpoints in their docstrings. Raises a ValueError if any configurations lack checkpoint information.\n\n**Key Implementation Details**\n\n* Skips deprecated models by checking for \"models.deprecated\" in the module name.\n* Iterates over config classes in `CONFIG_MAPPING`, appending names of missing checkpoint info to `configs_without_checkpoint` list.\n* If no checkpoints found, raises a ValueError with an error message and sorted list of configuration class names.\n\n**Notable Dependencies and Requirements**\n\n* `CONFIG_MAPPING`: mapping of configuration classes\n* `CONFIG_CLASSES_TO_IGNORE_FOR_DOCSTRING_CHECKPOINT_CHECK`: list of ignored configuration class names for docstring checkpoint check\n* `get_checkpoint_from_config_class(config_class)`: function to retrieve checkpoint from a config class\n* `sorted()` function for sorting configuration class names."}


created_at: 2025-02-05 09:13:56
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/notification_service_doc_tests.py:27-43
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 43, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/notification_service_doc_tests.py", "name": "handle_test_results", "start_line": 27, "type": "function", "chroma:document": "CODE:\n\ndef handle_test_results(test_results):\n    expressions = test_results.split(\" \")\n\n    failed = 0\n    success = 0\n\n    # When the output is short enough, the output is surrounded by = signs: \"== OUTPUT ==\"\n    # When it is too long, those signs are not present.\n    time_spent = expressions[-2] if \"=\" in expressions[-1] else expressions[-1]\n\n    for i, expression in enumerate(expressions):\n        if \"failed\" in expression:\n            failed += int(expressions[i - 1])\n        if \"passed\" in expression:\n            success += int(expressions[i - 1])\n\n\n\nSUMMARY:\n**Summary**\n\nThe `handle_test_results` function parses test results to extract failure and success counts, as well as the timestamp. Key aspects include:\n\n*   **Purpose**: Parse test output to calculate failure and success metrics.\n*   **Implementation Details**:\n    *   Splits test output into expressions using spaces as separators.\n    *   Calculates `time_spent` by detecting presence of `== OUTPUT ==` signs in the last expression.\n    *   Iterates through expressions, incrementing `failed` and `success` counts based on specific checks (e.g., \"failed\" or \"passed\").\n*   **Notable Dependencies/Requirements**:\n    *   None\n\n**Detailed Information**\n\n*   Method Signatures: Not applicable\n*   Interfaces Implemented: None\n*   External Dependencies: None"}


created_at: 2025-02-05 09:13:57
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/notification_service_doc_tests.py:46-58
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 58, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/notification_service_doc_tests.py", "name": "extract_first_line_failure", "start_line": 46, "type": "function", "chroma:document": "CODE:\n\ndef extract_first_line_failure(failures_short_lines):\n    failures = {}\n    file = None\n    in_error = False\n    for line in failures_short_lines.split(\"\\n\"):\n        if re.search(r\"_ \\[doctest\\]\", line):\n            in_error = True\n            file = line.split(\" \")[2]\n        elif in_error and not line.split(\" \")[0].isdigit():\n            failures[file] = line\n            in_error = False\n\n\n\nSUMMARY:\n**Summary:**\n\nThis function, `extract_first_line_failure`, processes text data from \"short lines\" of failures. It categorizes them into file-related failures by identifying the `_ [doctest]` prefix and extracts specific information about the error.\n\n**Key Details:**\n\n- **Algorithm:** Uses regular expressions to scan short lines for a pattern (_ [doctest]).\n- **Method Signatures/Interfaces:** Takes `failures_short_lines` as input and returns a dictionary of failures (`failures`) with file names.\n- **Dependencies:**\n  - Utilizes external `re` (regular expression) module.\n  - No imported classes/functions besides built-in Python.\n\n**Context:**\nThis function appears to be part of a code analysis or testing framework, likely used for data extraction from test output logs."}


created_at: 2025-02-05 09:13:57
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/notification_service_doc_tests.py:61-273
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 273, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/notification_service_doc_tests.py", "name": "Message", "start_line": 61, "type": "class", "chroma:document": "CODE:\n\nclass Message:\n    def __init__(self, title: str, doc_test_results: Dict):\n        self.title = title\n\n        self.n_success = sum(job_result[\"n_success\"] for job_result in doc_test_results.values())\n        self.n_failures = sum(job_result[\"n_failures\"] for job_result in doc_test_results.values())\n        self.n_tests = self.n_success + self.n_failures\n\n        # Failures and success of the modeling tests\n        self.doc_test_results = doc_test_results\n\n    @property\n    def time(self) -> str:\n        all_results = [*self.doc_test_results.values()]\n        time_spent = [r[\"time_spent\"].split(\", \")[0] for r in all_results if len(r[\"time_spent\"])]\n        total_secs = 0\n\n        for time in time_spent:\n            time_parts = time.split(\":\")\n\n            # Time can be formatted as xx:xx:xx, as .xx, or as x.xx if the time spent was less than a minute.\n            if len(time_parts) == 1:\n                time_parts = [0, 0, time_parts[0]]\n\n            hours, minutes, seconds = int(time_parts[0]), int(time_parts[1]), float(time_parts[2])\n            total_secs += hours * 3600 + minutes * 60 + seconds\n\n        hours, minutes, seconds = total_secs // 3600, (total_secs % 3600) // 60, total_secs % 60\n        return f\"{int(hours)}h{int(minutes)}m{int(seconds)}s\"\n\n    @property\n    def header(self) -> Dict:\n        return {\"type\": \"header\", \"text\": {\"type\": \"plain_text\", \"text\": self.title}}\n\n    @property\n    def no_failures(self) -> Dict:\n        return {\n            \"type\": \"section\",\n            \"text\": {\n                \"type\": \"plain_text\",\n                \"text\": f\"\ud83c\udf1e There were no failures: all {self.n_tests} tests passed. The suite ran in {self.time}.\",\n                \"emoji\": True,\n            },\n            \"accessory\": {\n                \"type\": \"button\",\n                \"text\": {\"type\": \"plain_text\", \"text\": \"Check Action results\", \"emoji\": True},\n                \"url\": f\"https://github.com/huggingface/transformers/actions/runs/{os.environ['GITHUB_RUN_ID']}\",\n            },\n        }\n\n    @property\n    def failures(self) -> Dict:\n        return {\n            \"type\": \"section\",\n            \"text\": {\n                \"type\": \"plain_text\",\n                \"text\": (\n                    f\"There were {self.n_failures} failures, out of {self.n_tests} tests.\\nThe suite ran in\"\n                    f\" {self.time}.\"\n                ),\n                \"emoji\": True,\n            },\n            \"accessory\": {\n                \"type\": \"button\",\n                \"text\": {\"type\": \"plain_text\", \"text\": \"Check Action results\", \"emoji\": True},\n                \"url\": f\"https://github.com/huggingface/transformers/actions/runs/{os.environ['GITHUB_RUN_ID']}\",\n            },\n        }\n\n    @property\n    def category_failures(self) -> List[Dict]:\n        failure_blocks = []\n\n        MAX_ERROR_TEXT = 3000 - len(\"The following examples had failures:\\n\\n\\n\\n\") - len(\"[Truncated]\\n\")\n        line_length = 40\n        category_failures = {k: v[\"failed\"] for k, v in doc_test_results.items() if isinstance(v, dict)}\n\n        def single_category_failures(category, failures):\n            text = \"\"\n            if len(failures) == 0:\n                return \"\"\n            text += f\"*{category} failures*:\".ljust(line_length // 2).rjust(line_length // 2) + \"\\n\"\n\n            for idx, failure in enumerate(failures):\n                new_text = text + f\"`{failure}`\\n\"\n                if len(new_text) > MAX_ERROR_TEXT:\n                    text = text + \"[Truncated]\\n\"\n                    break\n                text = new_text\n\n            return text\n\n        for category, failures in category_failures.items():\n            report = single_category_failures(category, failures)\n            if len(report) == 0:\n                continue\n            block = {\n                \"type\": \"section\",\n                \"text\": {\n                    \"type\": \"mrkdwn\",\n                    \"text\": f\"The following examples had failures:\\n\\n\\n{report}\\n\",\n                },\n            }\n            failure_blocks.append(block)\n\n        return failure_blocks\n\n    @property\n    def payload(self) -> str:\n        blocks = [self.header]\n\n        if self.n_failures > 0:\n            blocks.append(self.failures)\n\n        if self.n_failures > 0:\n            blocks.extend(self.category_failures)\n\n        if self.n_failures == 0:\n            blocks.append(self.no_failures)\n\n        return json.dumps(blocks)\n\n    @staticmethod\n    def error_out():\n        payload = [\n            {\n                \"type\": \"section\",\n                \"text\": {\n                    \"type\": \"plain_text\",\n                    \"text\": \"There was an issue running the tests.\",\n                },\n                \"accessory\": {\n                    \"type\": \"button\",\n                    \"text\": {\"type\": \"plain_text\", \"text\": \"Check Action results\", \"emoji\": True},\n                    \"url\": f\"https://github.com/huggingface/transformers/actions/runs/{os.environ['GITHUB_RUN_ID']}\",\n                },\n            }\n        ]\n\n        print(\"Sending the following payload\")\n        print(json.dumps({\"blocks\": json.loads(payload)}))\n\n        client.chat_postMessage(\n            channel=SLACK_REPORT_CHANNEL_ID,\n            text=\"There was an issue running the tests.\",\n            blocks=payload,\n        )\n\n    def post(self):\n        print(\"Sending the following payload\")\n        print(json.dumps({\"blocks\": json.loads(self.payload)}))\n\n        text = f\"{self.n_failures} failures out of {self.n_tests} tests,\" if self.n_failures else \"All tests passed.\"\n\n        self.thread_ts = client.chat_postMessage(\n            channel=SLACK_REPORT_CHANNEL_ID,\n            blocks=self.payload,\n            text=text,\n        )\n\n    def get_reply_blocks(self, job_name, job_link, failures, text):\n        # `text` must be less than 3001 characters in Slack SDK\n        # keep some room for adding \"[Truncated]\" when necessary\n        MAX_ERROR_TEXT = 3000 - len(\"[Truncated]\")\n\n        failure_text = \"\"\n        for key, value in failures.items():\n            new_text = failure_text + f\"*{key}*\\n_{value}_\\n\\n\"\n            if len(new_text) > MAX_ERROR_TEXT:\n                # `failure_text` here has length <= 3000\n                failure_text = failure_text + \"[Truncated]\"\n                break\n            # `failure_text` here has length <= MAX_ERROR_TEXT\n            failure_text = new_text\n\n        title = job_name\n        content = {\"type\": \"section\", \"text\": {\"type\": \"mrkdwn\", \"text\": text}}\n\n        if job_link is not None:\n            content[\"accessory\"] = {\n                \"type\": \"button\",\n                \"text\": {\"type\": \"plain_text\", \"text\": \"GitHub Action job\", \"emoji\": True},\n                \"url\": job_link,\n            }\n\n        return [\n            {\"type\": \"header\", \"text\": {\"type\": \"plain_text\", \"text\": title, \"emoji\": True}},\n            content,\n            {\"type\": \"section\", \"text\": {\"type\": \"mrkdwn\", \"text\": failure_text}},\n        ]\n\n    def post_reply(self):\n        if self.thread_ts is None:\n            raise ValueError(\"Can only post reply if a post has been made.\")\n\n        sorted_dict = sorted(self.doc_test_results.items(), key=lambda t: t[0])\n        for job_name, job_result in sorted_dict:\n            if len(job_result[\"failures\"]) > 0:\n                text = f\"*Num failures* :{len(job_result['failed'])} \\n\"\n                failures = job_result[\"failures\"]\n                blocks = self.get_reply_blocks(job_name, job_result[\"job_link\"], failures, text=text)\n\n                print(\"Sending the following reply\")\n                print(json.dumps({\"blocks\": blocks}))\n\n                client.chat_postMessage(\n                    channel=SLACK_REPORT_CHANNEL_ID,\n                    text=f\"Results for {job_name}\",\n                    blocks=blocks,\n                    thread_ts=self.thread_ts[\"ts\"],\n                )\n\n\n\nSUMMARY:\n**Class Summary:**\n\nThe `Message` class is designed to represent test results and report them in a Slack channel. It has the following primary purposes:\n\n*   Calculate and store metadata about the test results (e.g., number of successes/failures, time spent on tests)\n*   Generate reports with various information types (header, failures by category, payload)\n\n**Notable Properties and Methods:**\n\n*   `n_failures`, `n_tests`: Number of test failures and total tests\n*   `payload`: A Slack-compatible report payload in JSON format\n\nKey methods:\n\n*   `post()`: Posts the test result report to the specified Slack channel.\n*   `error_out()`: Sends a notification with an error message to the same Slack channel.\n*   `get_reply_blocks()`: Generates reply blocks for individual test failures, including links to GitHub Action jobs."}


created_at: 2025-02-05 09:13:57
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/notification_service_doc_tests.py:276-288
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 288, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/notification_service_doc_tests.py", "name": "retrieve_artifact", "start_line": 276, "type": "function", "chroma:document": "CODE:\n\ndef retrieve_artifact(name: str):\n    _artifact = {}\n\n    if os.path.exists(name):\n        files = os.listdir(name)\n        for file in files:\n            try:\n                with open(os.path.join(name, file), encoding=\"utf-8\") as f:\n                    _artifact[file.split(\".\")[0]] = f.read()\n            except UnicodeDecodeError as e:\n                raise ValueError(f\"Could not open {os.path.join(name, file)}.\") from e\n\n\n\nSUMMARY:\n**Summary**\n\n`retrieve_artifact` function retrieves files from a specified directory and returns an artifact dictionary containing file names as keys and their corresponding contents.\n\n**Main Purpose and Functionality**\nThe primary purpose of this function is to read the contents files from a given directory named `name`. The function iterates over each file, attempts to decode its contents, and stores them in a dictionary.\n\n```python\ndef retrieve_artifact(name: str) -> dict:\n    # ... (function implementation remains the same)\n```\n\n**Key Implementation Details**\n\n* Uses `os.path.exists` to check if the specified directory path exists.\n* Lists files using `os.listdir`.\n* Tries to open and read each file with UTF-8 encoding, raising an error for decoding failures.\n\n```python\nfiles = os.listdir(name)\nfor file in files:\n    try:\n        with open(os.path.join(name, file), encoding=\"utf-8\") as f:\n            _artifact[file.split(\".\")[0]] = f.read()\n```\n\n**Notable Dependencies and Requirements**\n\n* `os` module for path manipulation.\n* External dependency: `file extensions (e.g., `.txt`, `.pdf`).\n* No apparent inheritance or interface implementation.\n\n```python\nimport os\n```\n\n**RAG System Representation**\n\n| Key | Value |\n| --- | --- |\n| Purpose | Retrieve files from a specified directory and return an artifact dictionary. |\n| Functionality | Iterates over each file, attempts to decode its contents, and stores them in a dictionary. |\n| Implementation Details | Uses `os.path.exists`, `os.listdir`, and UTF-8 encoding for file reading. |\n| Dependencies | `os` module. |"}


created_at: 2025-02-05 09:13:57
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/notification_service_doc_tests.py:291-313
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 313, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/notification_service_doc_tests.py", "name": "retrieve_available_artifacts", "start_line": 291, "type": "function", "chroma:document": "CODE:\n\ndef retrieve_available_artifacts():\n    class Artifact:\n        def __init__(self, name: str):\n            self.name = name\n            self.paths = []\n\n        def __str__(self):\n            return self.name\n\n        def add_path(self, path: str):\n            self.paths.append({\"name\": self.name, \"path\": path})\n\n    _available_artifacts: Dict[str, Artifact] = {}\n\n    directories = filter(os.path.isdir, os.listdir())\n    for directory in directories:\n        artifact_name = directory\n        if artifact_name not in _available_artifacts:\n            _available_artifacts[artifact_name] = Artifact(artifact_name)\n\n            _available_artifacts[artifact_name].add_path(directory)\n\n\n\nSUMMARY:\n**Concise Summary:**\n\nThis function, `retrieve_available_artifacts`, creates and populates a dictionary of artifacts stored in directories on the system. It utilizes classes to encapsulate artifact details and behavior.\n\n**Key Implementation Details:**\n\n* Creates an `Artifact` class with `name` attribute, which stores directory names as names and their corresponding paths.\n* Utilizes `filter()` and `os.path.isdir()` to retrieve only directories from the system.\n* Popsulates `_available_artifacts` dictionary with the extracted directory information.\n\n**Notable Dependencies:**\n\n* `os`: for interacting with the operating system (e.g., `os.path.isdir()`)\n* `filter()`: a built-in Python function for filtering elements from iterable collections based on condition.\n\n**Primary Purpose and Functionality:**\nThe main purpose of this code is to retrieve available artifacts stored in directories, which can be utilized for subsequent operations or processing. \n\n**Additional Informed Insights:**\n- The code iterates over all files and folders in the system using `os.listdir()`.\n- It populates an internal data structure `_available_artifacts` dictionary, containing only directories as keys.\n- It assumes that the directory names match the artifact names within these instances, allowing for consistent categorization."}


created_at: 2025-02-05 09:13:57
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_doc_toc.py:43-76
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 76, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_doc_toc.py", "name": "clean_model_doc_toc", "start_line": 43, "type": "function", "chroma:document": "CODE:\n\ndef clean_model_doc_toc(model_doc: List[dict]) -> List[dict]:\n    \"\"\"\n    Cleans a section of the table of content of the model documentation (one specific modality) by removing duplicates\n    and sorting models alphabetically.\n\n    Args:\n        model_doc (`List[dict]`):\n            The list of dictionaries extracted from the `_toctree.yml` file for this specific modality.\n\n    Returns:\n        `List[dict]`: List of dictionaries like the input, but cleaned up and sorted.\n    \"\"\"\n    counts = defaultdict(int)\n    for doc in model_doc:\n        counts[doc[\"local\"]] += 1\n    duplicates = [key for key, value in counts.items() if value > 1]\n\n    new_doc = []\n    for duplicate_key in duplicates:\n        titles = list({doc[\"title\"] for doc in model_doc if doc[\"local\"] == duplicate_key})\n        if len(titles) > 1:\n            raise ValueError(\n                f\"{duplicate_key} is present several times in the documentation table of content at \"\n                \"`docs/source/en/_toctree.yml` with different *Title* values. Choose one of those and remove the \"\n                \"others.\"\n            )\n        # Only add this once\n        new_doc.append({\"local\": duplicate_key, \"title\": titles[0]})\n\n    # Add none duplicate-keys\n    new_doc.extend([doc for doc in model_doc if counts[doc[\"local\"]] == 1])\n\n    # Sort\n\n\nSUMMARY:\n**Concise Summary**\n\nThe `clean_model_doc_toc` function cleans the table of content in model documentation by removing duplicates and sorting models alphabetically.\n\n**Main Purpose and Functionality**\n\n* Cleans a list of dictionaries (`model_doc`) extracted from `_toctree.yml` by:\n\t+ Counting occurrences of each local key\n\t+ Identifying duplicate keys with multiple titles\n\t+ Removing all but one version of each duplicate title\n* Returns the cleaned list sorted alphabetically by title.\n\n**Key Implementation Details**\n\n* Uses `defaultdict` to count occurrences of each local key.\n* Utilizes a lambda function for sorting by title (case-insensitive).\n* Raises a ValueError if multiple versions of a title are found for a duplicate key.\n\n**Notable Dependencies or Requirements**\n\n* Uses Python's built-in `List`, `dict`, and `defaultdict` types.\n* Assumes the presence of an `_toctree.yml` file with documentation data."}


created_at: 2025-02-05 09:13:57
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_doc_toc.py:79-125
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 125, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_doc_toc.py", "name": "check_model_doc", "start_line": 79, "type": "function", "chroma:document": "CODE:\n\ndef check_model_doc(overwrite: bool = False):\n    \"\"\"\n    Check that the content of the table of content in `_toctree.yml` is clean (no duplicates and sorted for the model\n    API doc) and potentially auto-cleans it.\n\n    Args:\n        overwrite (`bool`, *optional*, defaults to `False`):\n            Whether to just check if the TOC is clean or to auto-clean it (when `overwrite=True`).\n    \"\"\"\n    with open(PATH_TO_TOC, encoding=\"utf-8\") as f:\n        content = yaml.safe_load(f.read())\n\n    # Get to the API doc\n    api_idx = 0\n    while content[api_idx][\"title\"] != \"API\":\n        api_idx += 1\n    api_doc = content[api_idx][\"sections\"]\n\n    # Then to the model doc\n    model_idx = 0\n    while api_doc[model_idx][\"title\"] != \"Models\":\n        model_idx += 1\n\n    model_doc = api_doc[model_idx][\"sections\"]\n\n    # Extract the modalities and clean them one by one.\n    modalities_docs = [(idx, section) for idx, section in enumerate(model_doc) if \"sections\" in section]\n    diff = False\n    for idx, modality_doc in modalities_docs:\n        old_modality_doc = modality_doc[\"sections\"]\n        new_modality_doc = clean_model_doc_toc(old_modality_doc)\n\n        if old_modality_doc != new_modality_doc:\n            diff = True\n            if overwrite:\n                model_doc[idx][\"sections\"] = new_modality_doc\n\n    if diff:\n        if overwrite:\n            api_doc[model_idx][\"sections\"] = model_doc\n            content[api_idx][\"sections\"] = api_doc\n            with open(PATH_TO_TOC, \"w\", encoding=\"utf-8\") as f:\n                f.write(yaml.dump(content, allow_unicode=True))\n        else:\n            raise ValueError(\n                \"The model doc part of the table of content is not properly sorted, run `make style` to fix this.\"\n\n\nSUMMARY:\n**Summary: check_model_doc function**\n\nThe `check_model_doc` function validates and potentially auto-cleans the table of contents (`_toctree.yml`) for the model API doc. It checks for:\n\n* Duplicates: A duplicate title is found, indicating an invalid TOC structure.\n* Sorting: The TOC contains unsorted sections.\n\n**Key Implementation Details**\n\n1. Reads `PATH_TO_TOC` file using `yaml.safe_load()`.\n2. Iterates to find the \"API\" and \"Models\" section indices in the content tree.\n3. Extracts modalities documents from the \"Models\" section.\n4. Cleans each modality document using `clean_model_doc_toc()`.\n\n**Notable Dependencies/Requirements**\n\n* `yaml`: for parsing the `_toctree.yml` file.\n* `PATH_TO_TOC`: a constant representing the file path to `._toctree.yml`.\n\n**Interfaces and Method Signatures**\n\n1. `yaml.safe_load()`: loads YAML data from a string.\n2. `clean_model_doc_toc()`: cleans a model doc using unknown implementation details.\n\nThis function appears to be part of a larger project for generating and maintaining documentation, possibly in a MarkDown or Sphinx-based environment."}


created_at: 2025-02-05 09:13:57
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_bad_commit.py:25-62
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 62, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_bad_commit.py", "name": "create_script", "start_line": 25, "type": "function", "chroma:document": "CODE:\n\ndef create_script(target_test):\n    \"\"\"Create a python script to be run by `git bisect run` to determine if `target_test` passes or fails.\n    If a test is not found in a commit, the script with exit code `0` (i.e. `Success`).\n\n    Args:\n        target_test (`str`): The test to check.\n\n    Returns:\n        `str`: The script to be run by `git bisect run`.\n    \"\"\"\n\n    script = f\"\"\"\nimport os\nimport subprocess\n\nresult = subprocess.run(\n    [\"python3\", \"-m\", \"pytest\", \"-v\", f\"{target_test}\"],\n    capture_output = True,\n    text=True,\n)\nprint(result.stdout)\n\nif len(result.stderr) > 0:\n    if \"ERROR: file or directory not found: \" in result.stderr:\n        print(\"test not found in this commit\")\n        exit(0)\n    else:\n        print(f\"pytest failed to run: {{result.stderr}}\")\n        exit(-1)\nelif f\"{target_test} FAILED\" in result.stdout:\n    print(\"test failed\")\n    exit(2)\n\nexit(0)\n\"\"\"\n\n    with open(\"target_script.py\", \"w\") as fp:\n\n\nSUMMARY:\n**Summary:**\n\nThe `create_script` function generates Python script to be run by `git bisect run` for verifying the outcome of test `target_test`. The script imports required modules, runs the specified test using `pytest`, and outputs the result.\n\n**Key Implementation Details:**\n\n* Runs `python3 -m pytest -v <test_name>` with specified `target_test`\n* Checks if error or failure message is present in standard output/error\n* Exits with success (0) or failure (-1/2) depending on validation\n\n**Notable Dependencies/Requirements:**\n\n* `os` and `subprocess` modules for executing external commands\n* `pytest` module for running tests"}


created_at: 2025-02-05 09:13:57
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_bad_commit.py:65-125
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 125, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_bad_commit.py", "name": "find_bad_commit", "start_line": 65, "type": "function", "chroma:document": "CODE:\n\ndef find_bad_commit(target_test, start_commit, end_commit):\n    \"\"\"Find (backward) the earliest commit between `start_commit` and `end_commit` at which `target_test` fails.\n\n    Args:\n        target_test (`str`): The test to check.\n        start_commit (`str`): The latest commit.\n        end_commit (`str`): The earliest commit.\n\n    Returns:\n        `str`: The earliest commit at which `target_test` fails.\n    \"\"\"\n\n    if start_commit == end_commit:\n        return start_commit\n\n    create_script(target_test=target_test)\n\n    bash = f\"\"\"\ngit bisect reset\ngit bisect start {start_commit} {end_commit}\ngit bisect run python3 target_script.py\n\"\"\"\n\n    with open(\"run_git_bisect.sh\", \"w\") as fp:\n        fp.write(bash.strip())\n\n    result = subprocess.run(\n        [\"bash\", \"run_git_bisect.sh\"],\n        capture_output=True,\n        text=True,\n    )\n    print(result.stdout)\n\n    if \"error: bisect run failed\" in result.stderr:\n        index = result.stderr.find(\"error: bisect run failed\")\n        bash_error = result.stderr[index:]\n\n        error_msg = f\"Error when running git bisect:\\nbash error: {bash_error}\"\n\n        pattern = \"pytest failed to run: .+\"\n        pytest_errors = re.findall(pattern, result.stdout)\n        if len(pytest_errors) > 0:\n            pytest_error = pytest_errors[0]\n            index = pytest_error.find(\"pytest failed to run: \")\n            index += len(\"pytest failed to run: \")\n            pytest_error = pytest_error[index:]\n            error_msg += f\"pytest error: {pytest_error}\"\n\n        raise ValueError(error_msg)\n\n    pattern = r\"(.+) is the first bad commit\"\n    commits = re.findall(pattern, result.stdout)\n\n    bad_commit = None\n    if len(commits) > 0:\n        bad_commit = commits[0]\n\n    print(f\"Between `start_commit` {start_commit} and `end_commit` {end_commit}\")\n    print(f\"bad_commit: {bad_commit}\\n\")\n\n\n\nSUMMARY:\n**Function Summary: find_bad_commit**\n\n**Purpose:** Find the earliest commit between two given commits at which a specified test fails.\n\n### Functionality:\n\n1.  **Initial Check**: Verifies if `start_commit` and `end_commit` are equal, returning `start_commit` in this case.\n2.  Creates a bash script to reset git bisect mode and start it with the provided `target_test`.\n3.  Runs the `run_git_bisect.sh` script using `subprocess.run`, capturing its output and error messages.\n4.  Checks for errors, specifically \"bisect run failed\". If found, extracts the error message and continues.\n5.  Seeks the first occurrence of \"first bad commit\" in the standard output to identify the earliest failing commit.\n\n### Key Implementation Details:\n\n*   `git bisect` mode is managed using bash scripts and subprocesses.\n*   Standard output and error messages are analyzed for errors or test failures.\n*   Regular expressions (`re`) are used to extract specific information from the output.\n\n**Key Method Signatures:**\n\n*   `create_script(target_test)`\n*   `subprocess.run([\"bash\", \"run_git_bisect.sh\"])`\n\n**Dependencies and Interfaces:**\n\n*   External dependencies:\n    *   `git` command\n    *   `python3` for running script\n    *   `re` (regular expression library) for pattern matching\n\n**Interfaces:**\n\n| Interface | Description                |\n| :-------- | -------------------------- |\n| `target_test` | Target test to check          |\n\n### Notable Variability:\n\n*   Initial commit check logic.\n*   Bash script and subprocess execution.\n\nThis analysis highlights the primary purpose and key implementation details of the provided function, facilitating a deeper understanding of its functionality and potential areas for optimization."}


created_at: 2025-02-05 09:13:57
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_bad_commit.py:128-150
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 150, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_bad_commit.py", "name": "get_commit_info", "start_line": 128, "type": "function", "chroma:document": "CODE:\n\ndef get_commit_info(commit):\n    \"\"\"Get information for a commit via `api.github.com`.\"\"\"\n    pr_number = None\n    author = None\n    merged_author = None\n\n    url = f\"https://api.github.com/repos/huggingface/transformers/commits/{commit}/pulls\"\n    pr_info_for_commit = requests.get(url).json()\n\n    if len(pr_info_for_commit) > 0:\n        pr_number = pr_info_for_commit[0][\"number\"]\n\n        url = f\"https://api.github.com/repos/huggingface/transformers/pulls/{pr_number}\"\n        pr_for_commit = requests.get(url).json()\n        author = pr_for_commit[\"user\"][\"login\"]\n        merged_author = pr_for_commit[\"merged_by\"][\"login\"]\n\n    if author is None:\n        url = f\"https://api.github.com/repos/huggingface/transformers/commits/{commit}\"\n        commit_info = requests.get(url).json()\n        author = commit_info[\"author\"][\"login\"]\n\n\n\nSUMMARY:\n**Summary**\n\nThe `get_commit_info` function retrieves information about a GitHub commit, including the pull request number, author, and merged author. It makes API requests to `api.github.com` to fetch data.\n\n* Purpose: Fetch commit metadata via GitHub API.\n* Implementation Details:\n\t+ Retrieves PR info for a given commit and extracts the PR number.\n\t+ Fetches details about the associated PR, including author and merged author, if available.\n\t+ Fallbacks to fetching original commit information using `commit` URL if no PR author is found.\n\t+ Returns a dictionary with commit-related data.\n\n**Key Method Signsatures/Interfaces**\n\n* `requests.get()`: Used for API requests (`url = f...\"`)\n\n**Notable Dependencies**\n\n* `requests`\n* `json` ( implicit, handling JSON responses from GitHub API)\n* `user`, `login` (accessed through PR and commit responses)\n\n**Interface Inheritance/Implementation**\n\nNone.\n\nThis function leverages the GitHub API to collect metadata. The implementation caters to both cases when a PR is associated with the commit and when it's not."}


created_at: 2025-02-05 09:13:57
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_tf_ops.py:43-82
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 82, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_tf_ops.py", "name": "onnx_compliancy", "start_line": 43, "type": "function", "chroma:document": "CODE:\n\ndef onnx_compliancy(saved_model_path, strict, opset):\n    saved_model = SavedModel()\n    onnx_ops = []\n\n    with open(os.path.join(REPO_PATH, \"utils\", \"tf_ops\", \"onnx.json\")) as f:\n        onnx_opsets = json.load(f)[\"opsets\"]\n\n    for i in range(1, opset + 1):\n        onnx_ops.extend(onnx_opsets[str(i)])\n\n    with open(saved_model_path, \"rb\") as f:\n        saved_model.ParseFromString(f.read())\n\n    model_op_names = set()\n\n    # Iterate over every metagraph in case there is more than one (a saved model can contain multiple graphs)\n    for meta_graph in saved_model.meta_graphs:\n        # Add operations in the graph definition\n        model_op_names.update(node.op for node in meta_graph.graph_def.node)\n\n        # Go through the functions in the graph definition\n        for func in meta_graph.graph_def.library.function:\n            # Add operations in each function\n            model_op_names.update(node.op for node in func.node_def)\n\n    # Convert to list, sorted if you want\n    model_op_names = sorted(model_op_names)\n    incompatible_ops = []\n\n    for op in model_op_names:\n        if op not in onnx_ops and op not in INTERNAL_OPS:\n            incompatible_ops.append(op)\n\n    if strict and len(incompatible_ops) > 0:\n        raise Exception(f\"Found the following incompatible ops for the opset {opset}:\\n\" + incompatible_ops)\n    elif len(incompatible_ops) > 0:\n        print(f\"Found the following incompatible ops for the opset {opset}:\")\n        print(*incompatible_ops, sep=\"\\n\")\n    else:\n\n\nSUMMARY:\n**Function Summary: `onnx_compliancy`**\n\n### Purpose and Functionality\n\nThe `onnx_compliancy` function checks if a saved model (in ONNX format) is compliant with the specified ONNX opset version. It parses the saved model, extracts its operations, and verifies that they are compatible with the given opset.\n\n**Key Implementation Details**\n\n* Extracts ONNX opsets from `onnx.json` file\n* Parses saved model from file using `SavedModel.ParseFromString`\n* Iterates over multiple graphs in saved model (if present)\n* Compares extracted operation names from saved model to list of compatible operations (`onnx_ops`) and internal operations (`INTERNAL_OPS`)\n* Raises exception if strict mode is enabled and incompatible ops are found\n\n**Notable Dependencies and Requirements**\n\n* `SavedModel` class\n* `json` module for parsing `onnx.json` file\n* `os` module for getting repository path (`REPO_PATH`)\n* External dependencies:\n\t+ `onnx_ops` (list of compatible ONNX opsets)\n\t+ `INTERNAL_OPS` (list of internal operations)"}


created_at: 2025-02-05 09:13:57
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/notification_service_quantization.py:39-166
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 166, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/notification_service_quantization.py", "name": "QuantizationMessage", "start_line": 39, "type": "class", "chroma:document": "CODE:\n\nclass QuantizationMessage(Message):\n    def __init__(\n        self,\n        title: str,\n        results: Dict,\n    ):\n        self.title = title\n\n        # Failures and success of the modeling tests\n        self.n_success = sum(r[\"success\"] for r in results.values())\n        self.single_gpu_failures = sum(r[\"failed\"][\"single\"] for r in results.values())\n        self.multi_gpu_failures = sum(r[\"failed\"][\"multi\"] for r in results.values())\n        self.n_failures = self.single_gpu_failures + self.multi_gpu_failures\n\n        self.n_tests = self.n_failures + self.n_success\n        self.results = results\n        self.thread_ts = None\n\n    @property\n    def payload(self) -> str:\n        blocks = [self.header]\n\n        if self.n_failures > 0:\n            blocks.append(self.failures_overwiew)\n            blocks.append(self.failures_detailed)\n\n        if self.n_failures == 0:\n            blocks.append(self.no_failures)\n\n        return json.dumps(blocks)\n\n    @property\n    def time(self) -> str:\n        all_results = self.results.values()\n        time_spent = []\n        for r in all_results:\n            if len(r[\"time_spent\"]):\n                time_spent.extend([x for x in r[\"time_spent\"].split(\", \") if len(x.strip())])\n        total_secs = 0\n\n        for time in time_spent:\n            time_parts = time.split(\":\")\n\n            # Time can be formatted as xx:xx:xx, as .xx, or as x.xx if the time spent was less than a minute.\n            if len(time_parts) == 1:\n                time_parts = [0, 0, time_parts[0]]\n\n            hours, minutes, seconds = int(time_parts[0]), int(time_parts[1]), float(time_parts[2])\n            total_secs += hours * 3600 + minutes * 60 + seconds\n\n        hours, minutes, seconds = total_secs // 3600, (total_secs % 3600) // 60, total_secs % 60\n        return f\"{int(hours)}h{int(minutes)}m{int(seconds)}s\"\n\n    @property\n    def failures_overwiew(self) -> Dict:\n        return {\n            \"type\": \"section\",\n            \"text\": {\n                \"type\": \"plain_text\",\n                \"text\": (\n                    f\"There were {self.n_failures} failures, out of {self.n_tests} tests.\\n\"\n                    f\"The suite ran in {self.time}.\"\n                ),\n                \"emoji\": True,\n            },\n            \"accessory\": {\n                \"type\": \"button\",\n                \"text\": {\"type\": \"plain_text\", \"text\": \"Check Action results\", \"emoji\": True},\n                \"url\": f\"https://github.com/huggingface/transformers/actions/runs/{os.environ['GITHUB_RUN_ID']}\",\n            },\n        }\n\n    @property\n    def failures_detailed(self) -> Dict:\n        failures = {k: v[\"failed\"] for k, v in self.results.items()}\n\n        individual_reports = []\n        for key, value in failures.items():\n            device_report = self.get_device_report(value)\n            if sum(value.values()):\n                report = f\"{device_report}{key}\"\n                individual_reports.append(report)\n\n        header = \"Single |  Multi | Category\\n\"\n        failures_report = prepare_reports(\n            title=\"The following quantization tests had failures\", header=header, reports=individual_reports\n        )\n\n        return {\"type\": \"section\", \"text\": {\"type\": \"mrkdwn\", \"text\": failures_report}}\n\n    def post(self):\n        payload = self.payload\n        print(\"Sending the following payload\")\n        print(json.dumps({\"blocks\": json.loads(payload)}))\n\n        text = f\"{self.n_failures} failures out of {self.n_tests} tests,\" if self.n_failures else \"All tests passed.\"\n\n        self.thread_ts = client.chat_postMessage(\n            channel=SLACK_REPORT_CHANNEL_ID,\n            blocks=payload,\n            text=text,\n        )\n\n    def post_reply(self):\n        if self.thread_ts is None:\n            raise ValueError(\"Can only post reply if a post has been made.\")\n\n        for job, job_result in self.results.items():\n            if len(job_result[\"failures\"]):\n                for device, failures in job_result[\"failures\"].items():\n                    blocks = self.get_reply_blocks(\n                        job,\n                        job_result,\n                        failures,\n                        device,\n                        text=f'Number of failures: {job_result[\"failed\"][device]}',\n                    )\n\n                    print(\"Sending the following reply\")\n                    print(json.dumps({\"blocks\": blocks}))\n\n                    client.chat_postMessage(\n                        channel=\"#transformers-ci-daily-quantization\",\n                        text=f\"Results for {job}\",\n                        blocks=blocks,\n                        thread_ts=self.thread_ts[\"ts\"],\n                    )\n\n\nSUMMARY:\n**Summary: QuantizationMessage Class**\n\nThis class, `QuantizationMessage`, inherits from the `Message` class and represents a message containing quantization testing results.\n\n**Main Purpose and Functionality**\nThe main purpose of this class is to store and format the results of quantization tests in a JSON payload that can be sent through Slack.\n\n**Implementation Details**\n\n*   **Initialization**: The constructor takes two parameters: `title` (a string) and `results` (a dictionary).\n    *   `results`: A dictionary containing test results, including failure information.\n        *   `n_success`\n        *   `single_gpu_failures`\n        *   `multi_gpu_failures`\n        *   `n_failures`\n        *   `n_tests`\n        *   `thread_ts` (initially None)\n*   **Properties**:\n    *   `payload`: Returns the formatted JSON payload as a string, including optional sections for failure overview and detailed reports.\n        *   `blocks`: A list of JSON blocks containing the message content.\n        *   Includes failuer section\n    *   `time`: Calculates and returns the total time spent on all tests in hours, minutes, and seconds.\n*   **Methods**:\n    *   `post()`: Sends the message payload to a Slack channel using the client's chat post message method.\n    *   `post_reply()`: Posts replies for individual jobs with failures."}


created_at: 2025-02-05 09:13:57
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/pr_slow_ci_models.py:40-62
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 62, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/pr_slow_ci_models.py", "name": "get_new_python_files_between_commits", "start_line": 40, "type": "function", "chroma:document": "CODE:\n\ndef get_new_python_files_between_commits(base_commit: str, commits: List[str]) -> List[str]:\n    \"\"\"\n    Get the list of added python files between a base commit and one or several commits.\n\n    Args:\n        repo (`git.Repo`):\n            A git repository (for instance the Transformers repo).\n        base_commit (`str`):\n            The commit reference of where to compare for the diff. This is the current commit, not the branching point!\n        commits (`List[str]`):\n            The list of commits with which to compare the repo at `base_commit` (so the branching point).\n\n    Returns:\n        `List[str]`: The list of python files added between a base commit and one or several commits.\n    \"\"\"\n    code_diff = []\n    for commit in commits:\n        for diff_obj in commit.diff(base_commit):\n            # We always add new python files\n            if diff_obj.change_type == \"A\" and diff_obj.b_path.endswith(\".py\"):\n                code_diff.append(diff_obj.b_path)\n\n\n\nSUMMARY:\n**Function Summary:**\n\n`get_new_python_files_between_commits`\n\nReturns a list of added Python files between a base commit (`base_commit`) and one or several commits (`commits`). Compares each commit's diff against the base commit to identify new Python files.\n\n**Key Implementation Details:**\n\n* Uses `git.Repo.diff()` to compare each commit's diff against the base commit.\n* Iterates through a list of commits (`commits`) using the `diff()` method.\n* Identifies new Python files added in each commit by checking for file creation type (\"A\") and file extension (`.py`).\n* Returns a list of paths to newly created Python files.\n\n**Notable Dependencies/Requirements:**\n\n* `git.Repo`: Git repository access\n* `List[str]`: List of commit references\n\n**Primary Purpose:**\nIdentify new Python files added between a base commit and one or several commits in a Git repository."}


created_at: 2025-02-05 09:13:57
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/pr_slow_ci_models.py:65-87
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 87, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/pr_slow_ci_models.py", "name": "get_new_python_files", "start_line": 65, "type": "function", "chroma:document": "CODE:\n\ndef get_new_python_files() -> List[str]:\n    \"\"\"\n    Return a list of python files that have been added between the current head and the main branch.\n\n    Returns:\n        `List[str]`: The list of python files added.\n    \"\"\"\n    repo = Repo(PATH_TO_REPO)\n\n    try:\n        # For the cases where the main branch exists locally\n        main = repo.refs.main\n    except AttributeError:\n        # On GitHub Actions runners, it doesn't have local main branch\n        main = repo.remotes.origin.refs.main\n\n    print(f\"main is at {main.commit}\")\n    print(f\"Current head is at {repo.head.commit}\")\n\n    branching_commits = repo.merge_base(main, repo.head)\n    for commit in branching_commits:\n        print(f\"Branching commit: {commit}\")\n\n\nSUMMARY:\n**Function Summary**\n\n`get_new_python_files()` returns a list of Python files added between the current Git head and the `main` branch.\n\n**Key Implementation Details**\n\n* Uses `Repo` class to interact with the GitHub repository\n* Checks if `main` branch exists locally, falling back to remote `origin` in GitHub Actions runners\n* Retrieves merge base commits using `repo.merge_base`\n* Returns list of new Python files added using `get_new_python_files_between_commits`\n\n**Notable Dependencies and Requirements**\n\n* `Repo` class from `pathlib` library ( implied but not explicitly stated)\n* External dependency on GitHub repository at `PATH_TO_REPO`"}


created_at: 2025-02-05 09:13:57
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/pr_slow_ci_models.py:90-101
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 101, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/pr_slow_ci_models.py", "name": "get_new_model", "start_line": 90, "type": "function", "chroma:document": "CODE:\n\ndef get_new_model():\n    new_files = get_new_python_files()\n    reg = re.compile(r\"src/transformers/models/(.*)/modeling_.*\\.py\")\n\n    new_model = \"\"\n    for x in new_files:\n        find_new_model = reg.findall(x)\n        if len(find_new_model) > 0:\n            new_model = find_new_model[0]\n            # It's unlikely we have 2 new modeling files in a pull request.\n            break\n\n\nSUMMARY:\n**Function Summary: `get_new_model`**\n\nRetrieves the names of newly created Python model files in a Git repository.\n\n**Primary Functionality:**\nParses file contents to extract and return the name of the new model file, assuming only one new modeling file exists in each pull request.\n\n**Key Implementation Details:**\n\n* Uses regular expression `reg` to extract pattern \"src/transformers/models/(.*)/modeling_.*\\.py\"\n* Iterates over list of new Python files (`new_files`) and uses `re.findall` to find the matching model file name\n* Returns the extracted model file name as a string\n\n**Notable Dependencies or Requirements:**\n\n* `re` (regular expression) module for pattern matching\n* `get_new_python_files()` external function or call, returning list of new Python files"}


created_at: 2025-02-05 09:13:57
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/pr_slow_ci_models.py:104-128
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 128, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/pr_slow_ci_models.py", "name": "parse_message", "start_line": 104, "type": "function", "chroma:document": "CODE:\n\ndef parse_message(message: str) -> str:\n    \"\"\"\n    Parses a GitHub pull request's comment to find the models specified in it to run slow CI.\n\n    Args:\n        message (`str`): The body of a GitHub pull request's comment.\n\n    Returns:\n        `str`: The substring in `message` after `run-slow`, run_slow` or run slow`. If no such prefix is found, the\n        empty string is returned.\n    \"\"\"\n    if message is None:\n        return \"\"\n\n    message = message.strip().lower()\n\n    # run-slow: model_1, model_2\n    if not message.startswith((\"run-slow\", \"run_slow\", \"run slow\")):\n        return \"\"\n    message = message[len(\"run slow\") :]\n    # remove leading `:`\n    while message.strip().startswith(\":\"):\n        message = message.strip()[1:]\n\n\n\nSUMMARY:\n**Summary**\n\n### Functionality\n\nThe `parse_message` function parses a GitHub pull request comment to extract models specified in the message to run slow CI.\n\n### Implementation Details\n\n* **Key Algorithm:** Trims and normalizes user input message.\n* **Method Signature:**\n\t+ `parse_message(message: str) -> str`\n* **Design Choice:** Uses multiple prefixes (\"run-slow\", \"run_slow\", \"run slow\") to account for varying notation in comments.\n\n### Dependencies and Requirements\n\n* None explicitly listed, but assumes `str` type for input message."}


created_at: 2025-02-05 09:13:57
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/pr_slow_ci_models.py:131-133
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 133, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/pr_slow_ci_models.py", "name": "get_models", "start_line": 131, "type": "function", "chroma:document": "CODE:\n\ndef get_models(message: str):\n    models = parse_message(message)\n\n\nSUMMARY:\n**Summary**\n\n### Function Overview\n\n`get_models(message: str)`\n\nExtracts and preprocesses a list of models from a given message.\n\n### Primary Purpose\n\nResets and splits the `models` string to separate individual model names, replacing commas with spaces.\n\n### Key Implementation Details\n\n* Uses `parse_message(message)` (exact class/function not provided)\n* Applies whitespace manipulation to output string\n* Utilizes string splitting (`split()`) on `models`\n\n### Notable Dependencies or Requirements\n\n* `parse_message(message)` (related function not provided)\n \n### External Dependencies\n\nList of external dependencies is not specified in the given code chunk"}


created_at: 2025-02-05 09:13:57
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/pr_slow_ci_models.py:136-138
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 138, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/pr_slow_ci_models.py", "name": "check_model_names", "start_line": 136, "type": "function", "chroma:document": "CODE:\n\ndef check_model_names(model_name: str):\n    allowed = string.ascii_letters + string.digits + \"_\"\n\n\nSUMMARY:\n**Summary**\n\nChecks if a `model_name` is valid. A valid model name consists of letters, digits, and underscores only.\n\n* **Primary Purpose**: Validate model names to ensure they comply with the specified format.\n* **Key Implementation Details**:\n\t+ Uses regular expression for string validation (`string.ascii_letters + string.digits + \"_\"`).\n\t+ Checks for leading/trailing underscore prefixes using `startswith()` and `endswith()`.\n* **Notable Dependencies**:\n\t+ `string` module for character classification.\n* **Interfaces/Inherited Class**: None.\n\nIn RAG system notation: \n### model_name_validation\n- Description: Validates a model name to ensure it contains only letters, digits, or underscores.\n- Primary Purpose: Ensure model names meet the specified format requirement."}


created_at: 2025-02-05 09:13:57
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_modular_conversion.py:19-51
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 51, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_modular_conversion.py", "name": "process_file", "start_line": 19, "type": "function", "chroma:document": "CODE:\n\ndef process_file(modular_file_path, generated_modeling_content, file_type=\"modeling_\", fix_and_overwrite=False):\n    file_name_prefix = file_type.split(\"*\")[0]\n    file_name_suffix = file_type.split(\"*\")[-1] if \"*\" in file_type else \"\"\n    file_path = modular_file_path.replace(\"modular_\", f\"{file_name_prefix}_\").replace(\".py\", f\"{file_name_suffix}.py\")\n    # Read the actual modeling file\n    with open(file_path, \"r\") as modeling_file:\n        content = modeling_file.read()\n    output_buffer = StringIO(generated_modeling_content[file_type][0])\n    output_buffer.seek(0)\n    output_content = output_buffer.read()\n    diff = difflib.unified_diff(\n        output_content.splitlines(),\n        content.splitlines(),\n        fromfile=f\"{file_path}_generated\",\n        tofile=f\"{file_path}\",\n        lineterm=\"\",\n    )\n    diff_list = list(diff)\n    # Check for differences\n    if diff_list:\n        if fix_and_overwrite:\n            with open(file_path, \"w\") as modeling_file:\n                modeling_file.write(generated_modeling_content[file_type][0])\n            console.print(f\"[bold blue]Overwritten {file_path} with the generated content.[/bold blue]\")\n        else:\n            console.print(f\"\\n[bold red]Differences found between the generated code and {file_path}:[/bold red]\\n\")\n            diff_text = \"\\n\".join(diff_list)\n            syntax = Syntax(diff_text, \"diff\", theme=\"ansi_dark\", line_numbers=True)\n            console.print(syntax)\n        return 1\n    else:\n        console.print(f\"[bold green]No differences found for {file_path}.[/bold green]\")\n\n\nSUMMARY:\n**Function Summary**\n\n`process_file(modular_file_path, generated_modeling_content, file_type=\"modeling_\", fix_and_overwrite=False)`\n\n**Purpose and Functionality**\nProcess and compare the content of two modeling files located in `modular_file_path`.\n\n**Main Algorithms/P Patterns/Design Choices**\n\n* Reading and comparing file contents using `difflib.unified_diff` to identify differences.\n* Overwriting or printing differences based on `fix_and_overwrite` flag.\n\n**Key Method Signatures and Interfaces**\n\n- `difflib.unified_diff(fromfile, tofile, lineterm=\"\")`\n- `console.print()`\n\n**Notable Dependencies and Requirements**\n\n- `StringIO`\n- `difflib`\n- `console` (assuming a custom console object or module)\n- `generated_modeling_content[file_type]` referencing an external data structure."}


created_at: 2025-02-05 09:13:57
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_modular_conversion.py:54-60
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 60, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_modular_conversion.py", "name": "compare_files", "start_line": 54, "type": "function", "chroma:document": "CODE:\n\ndef compare_files(modular_file_path, fix_and_overwrite=False):\n    # Generate the expected modeling content\n    generated_modeling_content = convert_modular_file(modular_file_path)\n    diff = 0\n    for file_type in generated_modeling_content.keys():\n        diff += process_file(modular_file_path, generated_modeling_content, file_type, fix_and_overwrite)\n\n\nSUMMARY:\n**Summary:**\n\n`compare_files` function compares the expected modeling content with actual files specified by `modular_file_path`. It iterates through each file type, processing them using `process_file`, and returns the cumulative difference (`diff`). \n\n**Key Implementation Details:**\n\n* Generates expected modeling content using `convert_modular_file(modular_file_path)`\n\t+ Input parameter: `modular_file_path`\n\t+ Output parameter: `generated_modeling_content` dictionary containing different file types\n* Iterates through each file type in `generated_modeling_content` and processes them using `process_file(modular_file_path, generated_modeling_content, file_type, fix_and_overwrite)`\n\t+ Input parameters: `modular_file_path`, `generated_modeling_content`, `file_type`\n\t+ Output parameter: cumulative difference (`diff`)\n* Returns the cumulative difference (`diff`) after processing all files\n\n**Notable Dependencies/Requirements:**\n\n* `convert_modular_file` function\n* `process_file` function with parameters `modular_file_path`, `generated_modeling_content`, `file_type`, and `fix_and_overwrite`\n \nThis summary aims to preserve the original method's flow, focus on key implementation details, and reference related functions by exact name."}


created_at: 2025-02-05 09:13:57
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_inits.py:72-88
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 88, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_inits.py", "name": "find_backend", "start_line": 72, "type": "function", "chroma:document": "CODE:\n\ndef find_backend(line: str) -> Optional[str]:\n    \"\"\"\n    Find one (or multiple) backend in a code line of the init.\n\n    Args:\n        line (`str`): A code line of the main init.\n\n    Returns:\n        Optional[`str`]: If one (or several) backend is found, returns it. In the case of multiple backends (the line\n        contains `if is_xxx_available() and `is_yyy_available()`) returns all backends joined on `_and_` (so\n        `xxx_and_yyy` for instance).\n    \"\"\"\n    if _re_test_backend.search(line) is None:\n        return None\n    backends = [b[0] for b in _re_backend.findall(line)]\n    backends.sort()\n\n\nSUMMARY:\n**Summary**\n\n`find_backend` function identifies and extracts multiple backend names (`xxx_and_yyy`) from a given code line of the main `init` file. It performs a case-insensitive search using regular expression `_re_test_backend.search(line)`.\n\n**Implementation Details**\n\n* Uses two regular expressions: `_re_test_backend` to search for the backend pattern and `_re_backend` to extract the backend names.\n* Extracts backend names from the line using a list comprehension (`[b[0] for b in _re_backend.findall(line)]`) and sorts them alphabetically.\n\n**Return Value**\n\nReturns all extracted backend names joined on `_and_`, or `None` if no backend is found (`if _re_test_backend.search(line) is None`)."}


created_at: 2025-02-05 09:13:57
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_inits.py:91-231
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 231, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_inits.py", "name": "parse_init", "start_line": 91, "type": "function", "chroma:document": "CODE:\n\ndef parse_init(init_file) -> Optional[Tuple[Dict[str, List[str]], Dict[str, List[str]]]]:\n    \"\"\"\n    Read an init_file and parse (per backend) the `_import_structure` objects defined and the `TYPE_CHECKING` objects\n    defined.\n\n    Args:\n        init_file (`str`): Path to the init file to inspect.\n\n    Returns:\n        `Optional[Tuple[Dict[str, List[str]], Dict[str, List[str]]]]`: A tuple of two dictionaries mapping backends to list of\n        imported objects, one for the `_import_structure` part of the init and one for the `TYPE_CHECKING` part of the\n        init. Returns `None` if the init is not a custom init.\n    \"\"\"\n    with open(init_file, \"r\", encoding=\"utf-8\", newline=\"\\n\") as f:\n        lines = f.readlines()\n\n    # Get the to `_import_structure` definition.\n    line_index = 0\n    while line_index < len(lines) and not lines[line_index].startswith(\"_import_structure = {\"):\n        line_index += 1\n\n    # If this is a traditional init, just return.\n    if line_index >= len(lines):\n        return None\n\n    # First grab the objects without a specific backend in _import_structure\n    objects = []\n    while not lines[line_index].startswith(\"if TYPE_CHECKING\") and find_backend(lines[line_index]) is None:\n        line = lines[line_index]\n        # If we have everything on a single line, let's deal with it.\n        if _re_one_line_import_struct.search(line):\n            content = _re_one_line_import_struct.search(line).groups()[0]\n            imports = re.findall(r\"\\[([^\\]]+)\\]\", content)\n            for imp in imports:\n                objects.extend([obj[1:-1] for obj in imp.split(\", \")])\n            line_index += 1\n            continue\n        single_line_import_search = _re_import_struct_key_value.search(line)\n        if single_line_import_search is not None:\n            imports = [obj[1:-1] for obj in single_line_import_search.groups()[0].split(\", \") if len(obj) > 0]\n            objects.extend(imports)\n        elif line.startswith(\" \" * 8 + '\"'):\n            objects.append(line[9:-3])\n        line_index += 1\n\n    # Those are stored with the key \"none\".\n    import_dict_objects = {\"none\": objects}\n\n    # Let's continue with backend-specific objects in _import_structure\n    while not lines[line_index].startswith(\"if TYPE_CHECKING\"):\n        # If the line is an if not is_backend_available, we grab all objects associated.\n        backend = find_backend(lines[line_index])\n        # Check if the backend declaration is inside a try block:\n        if _re_try.search(lines[line_index - 1]) is None:\n            backend = None\n\n        if backend is not None:\n            line_index += 1\n\n            # Scroll until we hit the else block of try-except-else\n            while _re_else.search(lines[line_index]) is None:\n                line_index += 1\n\n            line_index += 1\n\n            objects = []\n            # Until we unindent, add backend objects to the list\n            while len(lines[line_index]) <= 1 or lines[line_index].startswith(\" \" * 4):\n                line = lines[line_index]\n                if _re_import_struct_add_one.search(line) is not None:\n                    objects.append(_re_import_struct_add_one.search(line).groups()[0])\n                elif _re_import_struct_add_many.search(line) is not None:\n                    imports = _re_import_struct_add_many.search(line).groups()[0].split(\", \")\n                    imports = [obj[1:-1] for obj in imports if len(obj) > 0]\n                    objects.extend(imports)\n                elif _re_between_brackets.search(line) is not None:\n                    imports = _re_between_brackets.search(line).groups()[0].split(\", \")\n                    imports = [obj[1:-1] for obj in imports if len(obj) > 0]\n                    objects.extend(imports)\n                elif _re_quote_object.search(line) is not None:\n                    objects.append(_re_quote_object.search(line).groups()[0])\n                elif line.startswith(\" \" * 8 + '\"'):\n                    objects.append(line[9:-3])\n                elif line.startswith(\" \" * 12 + '\"'):\n                    objects.append(line[13:-3])\n                line_index += 1\n\n            import_dict_objects[backend] = objects\n        else:\n            line_index += 1\n\n    # At this stage we are in the TYPE_CHECKING part, first grab the objects without a specific backend\n    objects = []\n    while (\n        line_index < len(lines)\n        and find_backend(lines[line_index]) is None\n        and not lines[line_index].startswith(\"else\")\n    ):\n        line = lines[line_index]\n        single_line_import_search = _re_import.search(line)\n        if single_line_import_search is not None:\n            objects.extend(single_line_import_search.groups()[0].split(\", \"))\n        elif line.startswith(\" \" * 8):\n            objects.append(line[8:-2])\n        line_index += 1\n\n    type_hint_objects = {\"none\": objects}\n\n    # Let's continue with backend-specific objects\n    while line_index < len(lines):\n        # If the line is an if is_backend_available, we grab all objects associated.\n        backend = find_backend(lines[line_index])\n        # Check if the backend declaration is inside a try block:\n        if _re_try.search(lines[line_index - 1]) is None:\n            backend = None\n\n        if backend is not None:\n            line_index += 1\n\n            # Scroll until we hit the else block of try-except-else\n            while _re_else.search(lines[line_index]) is None:\n                line_index += 1\n\n            line_index += 1\n\n            objects = []\n            # Until we unindent, add backend objects to the list\n            while len(lines[line_index]) <= 1 or lines[line_index].startswith(\" \" * 8):\n                line = lines[line_index]\n                single_line_import_search = _re_import.search(line)\n                if single_line_import_search is not None:\n                    objects.extend(single_line_import_search.groups()[0].split(\", \"))\n                elif line.startswith(\" \" * 12):\n                    objects.append(line[12:-2])\n                line_index += 1\n\n            type_hint_objects[backend] = objects\n        else:\n            line_index += 1\n\n\n\nSUMMARY:\n**Function Summary: `parse_init`**\n\nParsing an init file to extract import and TYPE_CHECKING definitions.\n\n**Purpose:** Read an init file, identifying and parsing the `_import_structure` and `TYPE_CHECKING` objects defined in it.\n\n**Primary Purpose:** The function reads a `.init` file and returns two dictionaries mapping backends to lists of imported objects: one for the `_import_structure` section and one for the `TYPE_CHECKING` section.\n\n**Key Implementation Details:**\n\n*   Checks for `_import_structure = { ... }` definitions to parse backend-specific imports.\n*   Iterates through lines to identify and extract:\n    *   Import structures without a specific backend (`_re_one_line_import_struct.search`)\n    *   Backend-specific imports within the try-except-else block (`_re_backend_find`)\n    *   TYPE_CHECKING objects without a specific backend (`_re_import.search`)\n*   Parses import definitions using regular expressions (e.g., `_import_struct_add_many`, `_re_between_brackets`)\n\n**Notable Dependencies and Requirements:**\n\n*   The function uses regular expression classes (`_re_one_line_import_struct`, `_re_backend_find`, etc.) to parse the init file.\n*   It relies on `find_backend()` function, but its implementation is not provided in this code snippet.\n\n**Return Value:** A tuple of two dictionaries:\n    *   `import_dict_objects`: Mapping backends to lists of import structures without a specific backend (`\"none\"`).\n    *   `type_hint_objects`: Mapping backends (if applicable) to lists of TYPE_CHECKING objects without a specific backend (`\"none\"`)."}


created_at: 2025-02-05 09:13:57
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_inits.py:234-278
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 278, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_inits.py", "name": "analyze_results", "start_line": 234, "type": "function", "chroma:document": "CODE:\n\ndef analyze_results(import_dict_objects: Dict[str, List[str]], type_hint_objects: Dict[str, List[str]]) -> List[str]:\n    \"\"\"\n    Analyze the differences between _import_structure objects and TYPE_CHECKING objects found in an init.\n\n    Args:\n        import_dict_objects (`Dict[str, List[str]]`):\n            A dictionary mapping backend names (`\"none\"` for the objects independent of any specific backend) to\n            list of imported objects.\n        type_hint_objects (`Dict[str, List[str]]`):\n            A dictionary mapping backend names (`\"none\"` for the objects independent of any specific backend) to\n            list of imported objects.\n\n    Returns:\n        `List[str]`: The list of errors corresponding to mismatches.\n    \"\"\"\n\n    def find_duplicates(seq):\n        return [k for k, v in collections.Counter(seq).items() if v > 1]\n\n    # If one backend is missing from the other part of the init, error early.\n    if list(import_dict_objects.keys()) != list(type_hint_objects.keys()):\n        return [\"Both sides of the init do not have the same backends!\"]\n\n    errors = []\n    # Find all errors.\n    for key in import_dict_objects.keys():\n        # Duplicate imports in any half.\n        duplicate_imports = find_duplicates(import_dict_objects[key])\n        if duplicate_imports:\n            errors.append(f\"Duplicate _import_structure definitions for: {duplicate_imports}\")\n        duplicate_type_hints = find_duplicates(type_hint_objects[key])\n        if duplicate_type_hints:\n            errors.append(f\"Duplicate TYPE_CHECKING objects for: {duplicate_type_hints}\")\n\n        # Missing imports in either part of the init.\n        if sorted(set(import_dict_objects[key])) != sorted(set(type_hint_objects[key])):\n            name = \"base imports\" if key == \"none\" else f\"{key} backend\"\n            errors.append(f\"Differences for {name}:\")\n            for a in type_hint_objects[key]:\n                if a not in import_dict_objects[key]:\n                    errors.append(f\"  {a} in TYPE_HINT but not in _import_structure.\")\n            for a in import_dict_objects[key]:\n                if a not in type_hint_objects[key]:\n                    errors.append(f\"  {a} in _import_structure but not in TYPE_HINT.\")\n\n\nSUMMARY:\n**Function Summary**\n\n`analyze_results` analyzes the discrepancies between `_import_structure` objects and `TYPE_CHECKING` objects found in an init.\n\n**Main Purpose**\nIdentifies duplicate imports, missing imports, and incorrect backend names across both structures.\n\n**Key Implementation Details**\n\n* Verifies that both input dictionaries have identical keys (backends).\n* Uses a custom function `find_duplicates` to detect duplicate values in lists.\n* Iterates through each key-value pair, comparing the two dictionaries for:\n\n    * Duplicate imports\n    * Missing imports\n\n**Notable Dependencies/Requirements**\n\n* Python built-in libraries: `collections`\n* External dependencies:\n\t+ None (self-contained)\n* Implemented interfaces:\n\t+ No explicit interface implementation; relies on dictionary comparisons."}


created_at: 2025-02-05 09:13:57
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_inits.py:281-297
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 297, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_inits.py", "name": "check_all_inits", "start_line": 281, "type": "function", "chroma:document": "CODE:\n\ndef check_all_inits():\n    \"\"\"\n    Check all inits in the transformers repo and raise an error if at least one does not define the same objects in\n    both halves.\n    \"\"\"\n    failures = []\n    for root, _, files in os.walk(PATH_TO_TRANSFORMERS):\n        if \"__init__.py\" in files:\n            fname = os.path.join(root, \"__init__.py\")\n            objects = parse_init(fname)\n            if objects is not None:\n                errors = analyze_results(*objects)\n                if len(errors) > 0:\n                    errors[0] = f\"Problem in {fname}, both halves do not define the same objects.\\n{errors[0]}\"\n                    failures.append(\"\\n\".join(errors))\n    if len(failures) > 0:\n\n\nSUMMARY:\n**Function Summary:**\n\n`check_all_inits()`\n\nChecks all initalization files (`__init__.py`) in the transformers repository for potential object definition discrepancies between both halves. Raises an error if any file fails to define the same objects.\n\n**Key Implementation Details:**\n\n* Iterates through `__init__.py` files in the transformers repository using `os.walk`.\n* Parses initialization file contents using `parse_init(fname)`.\n* Analyzes results using `analyze_results(*objects)` and accumulates errors.\n* Raises a `ValueError` with error messages if any discrepancies are found.\n\n**Notable Dependencies:**\n\n* `os`, `PATH_TO_TRANSFORMERS`\n* `parse_init`, `analyze_results` (external functions not shown in this code snippet)"}


created_at: 2025-02-05 09:13:58
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_inits.py:300-324
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 324, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_inits.py", "name": "get_transformers_submodules", "start_line": 300, "type": "function", "chroma:document": "CODE:\n\ndef get_transformers_submodules() -> List[str]:\n    \"\"\"\n    Returns the list of Transformers submodules.\n    \"\"\"\n    submodules = []\n    for path, directories, files in os.walk(PATH_TO_TRANSFORMERS):\n        for folder in directories:\n            # Ignore private modules\n            if folder.startswith(\"_\"):\n                directories.remove(folder)\n                continue\n            # Ignore leftovers from branches (empty folders apart from pycache)\n            if len(list((Path(path) / folder).glob(\"*.py\"))) == 0:\n                continue\n            short_path = str((Path(path) / folder).relative_to(PATH_TO_TRANSFORMERS))\n            submodule = short_path.replace(os.path.sep, \".\")\n            submodules.append(submodule)\n        for fname in files:\n            if fname == \"__init__.py\":\n                continue\n            short_path = str((Path(path) / fname).relative_to(PATH_TO_TRANSFORMERS))\n            submodule = short_path.replace(\".py\", \"\").replace(os.path.sep, \".\")\n            if len(submodule.split(\".\")) == 1:\n                submodules.append(submodule)\n\n\nSUMMARY:\n**Summary**\n\n`get_transformers_submodules()` function returns a list of Transformers submodules by traversing the directory tree rooted at `PATH_TO_TRANSFORMERS`. It filters out private modules, empty folders, and initializes that are not submodule names.\n\n**Key Implementation Details**\n\n* Uses `os.walk()` to traverse directory tree\n* Checks for file patterns in folders (`*.py`, `__init__.py`) and files (excluding `__init__.py`)\n* Normalizes folder and file paths to standardize submodule names\n\n**Notable Dependencies or Requirements**\n\n* `PATH_TO_TRANSFORMERS`\n* `os` module for walking directories\n* `Path` class from `pathlib` library for path manipulation"}


created_at: 2025-02-05 09:13:58
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/check_inits.py:339-368
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 368, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/check_inits.py", "name": "check_submodules", "start_line": 339, "type": "function", "chroma:document": "CODE:\n\ndef check_submodules():\n    \"\"\"\n    Check all submodules of Transformers are properly registered in the main init. Error otherwise.\n    \"\"\"\n    # This is to make sure the transformers module imported is the one in the repo.\n    from transformers.utils import direct_transformers_import\n\n    transformers = direct_transformers_import(PATH_TO_TRANSFORMERS)\n\n    import_structure_keys = set(transformers._import_structure.keys())\n    # This contains all the base keys of the _import_structure object defined in the init, but if the user is missing\n    # some optional dependencies, they may not have all of them. Thus we read the init to read all additions and\n    # (potentiall re-) add them.\n    with open(os.path.join(PATH_TO_TRANSFORMERS, \"__init__.py\"), \"r\") as f:\n        init_content = f.read()\n    import_structure_keys.update(set(re.findall(r\"import_structure\\[\\\"([^\\\"]*)\\\"\\]\", init_content)))\n\n    module_not_registered = [\n        module\n        for module in get_transformers_submodules()\n        if module not in IGNORE_SUBMODULES and module not in import_structure_keys\n    ]\n\n    if len(module_not_registered) > 0:\n        list_of_modules = \"\\n\".join(f\"- {module}\" for module in module_not_registered)\n        raise ValueError(\n            \"The following submodules are not properly registed in the main init of Transformers:\\n\"\n            f\"{list_of_modules}\\n\"\n            \"Make sure they appear somewhere in the keys of `_import_structure` with an empty list as value.\"\n\n\nSUMMARY:\n**Summary: `check_submodules()` Function**\n\n* **Purpose:** Validate that all submodules in `Transformers` are properly initialized with their respective import structures.\n* **Key Implementation Details:**\n\t+ Verifies `transformers._import_structure` keys against the expected structure and user-defined imports from `_init.py`.\n\t+ Compares submodule lists against imported `import_structure_keys`.\n\t+ Raises a `ValueError` if any submodules are not properly registered.\n* **Notable Dependencies & Requirements:**\n\t+ `direct_transformers_import`: imports transformers module ensuring it's the one in the repo.\n\t+ `get_transformers_submodules`: recursively finds all submodules of Transformers.\n\t+ `_init.py`: provides user-defined import structure additions for optional dependencies."}


created_at: 2025-02-05 09:13:58
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/get_previous_daily_ci.py:7-28
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 28, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/get_previous_daily_ci.py", "name": "get_daily_ci_runs", "start_line": 7, "type": "function", "chroma:document": "CODE:\n\ndef get_daily_ci_runs(token, num_runs=7):\n    \"\"\"Get the workflow runs of the scheduled (daily) CI.\n\n    This only selects the runs triggered by the `schedule` event on the `main` branch.\n    \"\"\"\n    headers = None\n    if token is not None:\n        headers = {\"Accept\": \"application/vnd.github+json\", \"Authorization\": f\"Bearer {token}\"}\n\n    # The id of a workflow (not of a workflow run).\n    # From a given workflow run (where we have workflow run id), we can get the workflow id by going to\n    # https://api.github.com/repos/huggingface/transformers/actions/runs/{workflow_run_id}\n    # and check the `workflow_id` key.\n    workflow_id = \"90575235\"\n\n    url = f\"https://api.github.com/repos/huggingface/transformers/actions/workflows/{workflow_id}/runs\"\n    # On `main` branch + event being `schedule` + not returning PRs + only `num_runs` results\n    url += f\"?branch=main&event=schedule&exclude_pull_requests=true&per_page={num_runs}\"\n\n    result = requests.get(url, headers=headers).json()\n\n\n\nSUMMARY:\n**Function Summary: `get_daily_ci_runs`**\n\n**Purpose and Functionality:** Retrieves the workflow run history of scheduled CI for the GitHub repository \"huggingface/transformers\" on the `main` branch filtered by schedule events.\n\n**Key Implementation Details:**\n\n* Employs the API endpoint `https://api.github.com/repos/huggingface/transformers/actions/workflows/{workflow_id}/runs` with query parameters:\n\t+ Branch: `main`\n\t+ Event: `schedule`\n\t+ Exclude pull requests: `true`\n\t+ Page size: `num_runs`\n* Utilizes the `requests.get()` method to send a GET request to the API endpoint, authenticated via Bearer token.\n\n**Notable Dependencies and Requirements:**\n\n* `token`: GitHub API authentication token\n* `requests` library for API calls\n* External dependencies:\n\t+ github.com ( GitHub repository)\n\t+ transformrsactions API"}


created_at: 2025-02-05 09:13:58
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/get_previous_daily_ci.py:31-40
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 40, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/get_previous_daily_ci.py", "name": "get_last_daily_ci_runs", "start_line": 31, "type": "function", "chroma:document": "CODE:\n\ndef get_last_daily_ci_runs(token):\n    \"\"\"Get the last completed workflow run id of the scheduled (daily) CI.\"\"\"\n    workflow_runs = get_daily_ci_runs(token)\n    workflow_run_id = None\n    for workflow_run in workflow_runs:\n        if workflow_run[\"status\"] == \"completed\":\n            workflow_run_id = workflow_run[\"id\"]\n            break\n\n\n\nSUMMARY:\n**Summary:**\n\n```markdown\n### Function:\n#### `get_last_daily_ci_runs(token)`\n\n#### Purpose:\n Retrieves the ID of the most recent completed daily Continuous Integration (CI) workflow.\n\n#### Implementation Details:\n\n* Calls `get_daily_ci_runs(token)` to retrieve daily CI runs\n* Iterates through runs, returning the ID of the first complete run (`if \"status\" in workflow_run:`)\n* Returns the most recent completed daily CI run's ID\n\n#### Notable Dependencies:\nNone\n\n#### Interfaces:\n- `get_daily_ci_runs(token)`\n```\n\n**RAG Code Tag:**\n`func: get_last_daily_ci_runs(token)`:"}


created_at: 2025-02-05 09:13:58
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/get_previous_daily_ci.py:43-52
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 52, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/get_previous_daily_ci.py", "name": "get_last_daily_ci_run_commit", "start_line": 43, "type": "function", "chroma:document": "CODE:\n\ndef get_last_daily_ci_run_commit(token):\n    \"\"\"Get the commit sha of the last completed scheduled daily CI workflow run.\"\"\"\n    workflow_runs = get_daily_ci_runs(token)\n    head_sha = None\n    for workflow_run in workflow_runs:\n        if workflow_run[\"status\"] == \"completed\":\n            head_sha = workflow_run[\"head_sha\"]\n            break\n\n\n\nSUMMARY:\n**Summary**\n\nThe `get_last_daily_ci_run_commit` function retrieves the commit SHA of the last completed scheduled daily CI workflow run.\n\n**Key Points:**\n\n* **Purpose:** Returns the SHA of the latest completed CI run.\n* **Functionality:** Iterates through completed daily CI runs to find the head SHA associated with the completion.\n* **Implementation Details:**\n\t+ Calls `get_daily_ci_runs` function to retrieve successful workflow runs.\n\t+ Loops through runs, identifying completed ones and saving their corresponding head SHA.\n\t+ Returns the head SHA of the last completed run.\n\n**Notable Dependencies/Requirements:**\n\n* `get_daily_ci_runs(token)` - Function called to retrieve daily CI runs."}


created_at: 2025-02-05 09:13:58
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/get_previous_daily_ci.py:55-65
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 65, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/get_previous_daily_ci.py", "name": "get_last_daily_ci_artifacts", "start_line": 55, "type": "function", "chroma:document": "CODE:\n\ndef get_last_daily_ci_artifacts(artifact_names, output_dir, token):\n    \"\"\"Get the artifacts of last completed workflow run id of the scheduled (daily) CI.\"\"\"\n    workflow_run_id = get_last_daily_ci_runs(token)\n    if workflow_run_id is not None:\n        artifacts_links = get_artifacts_links(worflow_run_id=workflow_run_id, token=token)\n        for artifact_name in artifact_names:\n            if artifact_name in artifacts_links:\n                artifact_url = artifacts_links[artifact_name]\n                download_artifact(\n                    artifact_name=artifact_name, artifact_url=artifact_url, output_dir=output_dir, token=token\n\n\nSUMMARY:\n**Function Summary:**\n\n`get_last_daily_ci_artifacts`\n\n Retrieves and downloads artifacts from the last completed daily CI workflow run.\n\n**Key Implementation Details:**\n\n* Fetches `workflow_run_id` using `get_last_daily_ci_runs`.\n* Retrieves artifact links for the specified `workflow_run_id` using `get_artifacts_links`.\n* Iterates over `artifact_names`, downloading artifacts matching those names using `download_artifact`.\n\n**Notable Dependencies or Requirements:**\n\n* Utilizes `token` in multiple calls.\n* Calls `get_artifacts_links` with `workflow_run_id` and `token`.\n* Expects implementation of `download_artifact` function separate from this snippet.\n\nThis summary prescribes the main purpose, key algorithms/patterns, and dependencies used within `get_last_daily_ci_artifacts`."}


created_at: 2025-02-05 09:13:58
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/get_previous_daily_ci.py:68-84
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 84, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/get_previous_daily_ci.py", "name": "get_last_daily_ci_reports", "start_line": 68, "type": "function", "chroma:document": "CODE:\n\ndef get_last_daily_ci_reports(artifact_names, output_dir, token):\n    \"\"\"Get the artifacts' content of the last completed workflow run id of the scheduled (daily) CI.\"\"\"\n    get_last_daily_ci_artifacts(artifact_names, output_dir, token)\n\n    results = {}\n    for artifact_name in artifact_names:\n        artifact_zip_path = os.path.join(output_dir, f\"{artifact_name}.zip\")\n        if os.path.isfile(artifact_zip_path):\n            results[artifact_name] = {}\n            with zipfile.ZipFile(artifact_zip_path) as z:\n                for filename in z.namelist():\n                    if not os.path.isdir(filename):\n                        # read the file\n                        with z.open(filename) as f:\n                            results[artifact_name][filename] = f.read().decode(\"UTF-8\")\n\n\n\nSUMMARY:\n**Summary**\n\nThe `get_last_daily_ci_reports` function retrieves the artifacts' content from the last completed daily CI workflow run. It takes three parameters: `artifact_names`, `output_dir`, and `token`.\n\n**Purpose and Functionality**\n\nRetrieves artifacts from the last daily CI workflow run, saving their contents in a dictionary for each artifact.\n\n**Key Implementation Details**\n\n* Iterates over `.zip` files with artifact names matching in the provided directories.\n* Reads and decodes file contents using UTF-8 encoding.\n* Returns a nested dictionary containing artifact names as keys and their corresponding file contents as values.\n\n**Notable Dependencies or Requirements**\n\n* `os.path.isfile`\n* `os.open`\n* `zipped zipfile.ZipFile`\n\nNote: This summary assumes the absence of any external dependencies not mentioned in the provided code. If such dependencies exist, please provide them for a more comprehensive analysis."}


created_at: 2025-02-05 09:13:58
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/get_github_job_time.py:8-25
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 25, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/get_github_job_time.py", "name": "extract_time_from_single_job", "start_line": 8, "type": "function", "chroma:document": "CODE:\n\ndef extract_time_from_single_job(job):\n    \"\"\"Extract time info from a single job in a GitHub Actions workflow run\"\"\"\n\n    job_info = {}\n\n    start = job[\"started_at\"]\n    end = job[\"completed_at\"]\n\n    start_datetime = date_parser.parse(start)\n    end_datetime = date_parser.parse(end)\n\n    duration_in_min = round((end_datetime - start_datetime).total_seconds() / 60.0)\n\n    job_info[\"started_at\"] = start\n    job_info[\"completed_at\"] = end\n    job_info[\"duration\"] = duration_in_min\n\n\n\nSUMMARY:\n**Summary**\n\nThis function, `extract_time_from_single_job`, extracts time-related information from a single job in a GitHub Actions workflow run. It:\n\n* Retrieves the start and end timestamps of the job\n* Calculates the duration in minutes between these two timestamps\n* Stores this duration along with the original timestamps in a dictionary, `job_info`\n* Returns the populated `job_info` dictionary\n\n**Key Implementation Details**\n\n* Retrieves dates using date parsing (`date_parser.parse`)\n* Calculates duration in minutes from the timestamp difference ( converted to seconds)\n\n**Notable Dependencies or Requirements**\n\n* `date_parser`: assumed dependency for parsing timestamps into datetimes\n* Expected input: `job` dictionary with keys \"started_at\" and \"completed_at\"\n* Returns a dictionary (`job_info`) with keys \"started_at\", \"completed_at\", and \"duration\"."}


created_at: 2025-02-05 09:13:58
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/get_github_job_time.py:28-51
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 51, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/get_github_job_time.py", "name": "get_job_time", "start_line": 28, "type": "function", "chroma:document": "CODE:\n\ndef get_job_time(workflow_run_id, token=None):\n    \"\"\"Extract time info for all jobs in a GitHub Actions workflow run\"\"\"\n\n    headers = None\n    if token is not None:\n        headers = {\"Accept\": \"application/vnd.github+json\", \"Authorization\": f\"Bearer {token}\"}\n\n    url = f\"https://api.github.com/repos/huggingface/transformers/actions/runs/{workflow_run_id}/jobs?per_page=100\"\n    result = requests.get(url, headers=headers).json()\n    job_time = {}\n\n    try:\n        job_time.update({job[\"name\"]: extract_time_from_single_job(job) for job in result[\"jobs\"]})\n        pages_to_iterate_over = math.ceil((result[\"total_count\"] - 100) / 100)\n\n        for i in range(pages_to_iterate_over):\n            result = requests.get(url + f\"&page={i + 2}\", headers=headers).json()\n            job_time.update({job[\"name\"]: extract_time_from_single_job(job) for job in result[\"jobs\"]})\n\n        return job_time\n    except Exception:\n        print(f\"Unknown error, could not fetch links:\\n{traceback.format_exc()}\")\n\n\n\nSUMMARY:\n**Function Summary**\n\n`get_job_time`: Extracts time information for all jobs in a GitHub Actions workflow run.\n\n**Primary Purpose:** Retrieves and extracts time info from multiple GitHub Actions workflow jobs.\n\n**Implementation Details:**\n\n- Uses `requests` to fetch GitHub API data\n- Iterates through 2 pages of jobs (due to pagination) to collect time info\n- Utilizes `extract_time_from_single_job` function (not shown in this snippet)\n- Handles exceptions and prints error messages if an issue occurs\n\n**Design Choices:**\n\n* Uses Bearer token authentication for API requests\n* Sets a page size of 100 jobs per request (per GitHub's rate limiting guidelines)\n\n**Notable Dependencies:**\n\n* `requests`\n* `math` (for calculating total pages to iterate over)\n* `extract_time_from_single_job` (not shown in this snippet, assumed to be implemented elsewhere)"}


created_at: 2025-02-05 09:13:58
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/deprecate_models.py:28-41
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 41, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/deprecate_models.py", "name": "get_last_stable_minor_release", "start_line": 28, "type": "function", "chroma:document": "CODE:\n\ndef get_last_stable_minor_release():\n    # Get the last stable release of transformers\n    url = \"https://pypi.org/pypi/transformers/json\"\n    release_data = requests.get(url).json()\n\n    # Find the last stable release of of transformers (version below current version)\n    major_version, minor_version, patch_version, _ = current_version.split(\".\")\n    last_major_minor = f\"{major_version}.{int(minor_version) - 1}\"\n    last_stable_minor_releases = [\n        release for release in release_data[\"releases\"] if release.startswith(last_major_minor)\n    ]\n    last_stable_release = sorted(last_stable_minor_releases, key=version.parse)[-1]\n\n\n\nSUMMARY:\n**Summary:**\n\nThe `get_last_stable_minor_release` function retrieves the last stable minor release of the `transformers` library by querying the PyPI JSON API. It compares the current major version to determine the last stable major and minor version, then filters releases to find those with versions below this threshold.\n\n**Key Implementation Details:**\n\n* Fetches `transformers` Pypi JSON data using `requests`\n* Compares current version to find last stable minor release\n* Uses `version.parse()` to sort and compare release versions\n\n**Notable Dependencies/Requirements:**\n\n* `requests` library for HTTP client\n* `pyperpapi` library (`version.parse()`) for parsing release versions"}


created_at: 2025-02-05 09:13:58
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/deprecate_models.py:44-55
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 55, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/deprecate_models.py", "name": "build_tip_message", "start_line": 44, "type": "function", "chroma:document": "CODE:\n\ndef build_tip_message(last_stable_release):\n    return (\n        \"\"\"\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we don't accept any new PRs changing its code.\n\"\"\"\n        + f\"\"\"If you run into any issues running this model, please reinstall the last version that supported this model: v{last_stable_release}.\nYou can do so by running the following command: `pip install -U transformers=={last_stable_release}`.\n\n</Tip>\"\"\"\n\n\nSUMMARY:\n**Function Summary: build_tip_message**\n\n* **Purpose**: Generates a maintenance mode warning message for a machine learning model, indicating that new PRs changing the code are not accepted.\n\t+ Primary purpose: Warn users about model maintenance mode and provide instructions to reinstall previous version.\n\n**Key Implementation Details:**\n- Uses f-string formatting to insert `last_stable_release` into the message.\n- Includes a Git installation command for reusing the last stable version supported by the model (Transformer vX.X).\n\n**Notable Dependencies or Requirements:** None mentioned explicitly."}


created_at: 2025-02-05 09:13:58
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/deprecate_models.py:58-76
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 76, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/deprecate_models.py", "name": "insert_tip_to_model_doc", "start_line": 58, "type": "function", "chroma:document": "CODE:\n\ndef insert_tip_to_model_doc(model_doc_path, tip_message):\n    tip_message_lines = tip_message.split(\"\\n\")\n\n    with open(model_doc_path, \"r\") as f:\n        model_doc = f.read()\n\n    # Add the tip message to the model doc page directly underneath the title\n    lines = model_doc.split(\"\\n\")\n\n    new_model_lines = []\n    for line in lines:\n        if line.startswith(\"# \"):\n            new_model_lines.append(line)\n            new_model_lines.extend(tip_message_lines)\n        else:\n            new_model_lines.append(line)\n\n    with open(model_doc_path, \"w\") as f:\n\n\nSUMMARY:\n**Summary:**\n\n`insert_tip_to_model_doc` function appends a tip message below the title section of an existing Markdown document file.\n\n**Key Details:**\n\n* Reads an input Markdown document file (`model_doc_path`) and tip message (`tip_message`)\n* Splits the document into lines, separating those with title markers (`# `) from others\n* Inserts the tip message lines below the original title lines, preserving blank line separation\n\n**Implementation Notes:**\n\n* Uses `file read/write` operations to modify an external file resource.\n\n**External Dependencies: None specified. No known inherited or implemented interfaces are assumed to be used in this code snippet.**"}


created_at: 2025-02-05 09:13:58
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/deprecate_models.py:79-89
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 89, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/deprecate_models.py", "name": "get_model_doc_path", "start_line": 79, "type": "function", "chroma:document": "CODE:\n\ndef get_model_doc_path(model: str) -> Tuple[Optional[str], Optional[str]]:\n    # Possible variants of the model name in the model doc path\n    model_names = [model, model.replace(\"_\", \"-\"), model.replace(\"_\", \"\")]\n\n    model_doc_paths = [REPO_PATH / f\"docs/source/en/model_doc/{model_name}.md\" for model_name in model_names]\n\n    for model_doc_path, model_name in zip(model_doc_paths, model_names):\n        if os.path.exists(model_doc_path):\n            return model_doc_path, model_name\n\n\n\nSUMMARY:\n**Summary:**\n\nThis function retrieves the path to an English model documentation file based on the provided `model` name. It checks for three possible variant names and searches for a non-empty documentation directory within the repository.\n\n**Implementation Details:**\n\n* The function returns two optional strings, (`model_doc_path`, `model_name`) if a matching documentation file exists.\n* The search considers three variant names:\n\t1. Original model name\n\t2. Model name with hyphens instead of underscores\n\t3. Model name without hyphens\n* It utilizes the `os` module for path existence checks.\n\n**Notable Dependencies:**\n\n* `REPO_PATH`: assumed to be a constants or an external variable referencing the repository's root directory.\n* `os.path.exists()`\n* `zip()` function"}


created_at: 2025-02-05 09:13:58
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/deprecate_models.py:92-108
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 108, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/deprecate_models.py", "name": "extract_model_info", "start_line": 92, "type": "function", "chroma:document": "CODE:\n\ndef extract_model_info(model):\n    model_info = {}\n    model_doc_path, model_doc_name = get_model_doc_path(model)\n    model_path = REPO_PATH / f\"src/transformers/models/{model}\"\n\n    if model_doc_path is None:\n        print(f\"Model doc path does not exist for {model}\")\n        return None\n    model_info[\"model_doc_path\"] = model_doc_path\n    model_info[\"model_doc_name\"] = model_doc_name\n\n    if not os.path.exists(model_path):\n        print(f\"Model path does not exist for {model}\")\n        return None\n    model_info[\"model_path\"] = model_path\n\n\n\nSUMMARY:\n**Function Summary: extract_model_info**\n\n**Purpose and Functionality:**\nThis function retrieves information about a given machine learning model, including its documentation path and file location.\n\n**Key Implementation Details:**\n\n* Retrieves `model` using the `get_model_doc_path` function to get its documentation path.\n* Returns a dictionary containing the following keys:\n\t+ `model_doc_path`: Path to the model's documentation.\n\t+ `model_doc_name`: Name of the model's documentation.\n\n**Notable Dependencies and Requirements:**\n\n* `REPO_PATH`: A constant representing the repository path.\n* `get_model_doc_path` function.\n* External dependencies: This function relies on the existence of the `get_model_doc_path` function, which itself may depend on other external libraries."}


created_at: 2025-02-05 09:13:58
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/deprecate_models.py:111-123
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 123, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/deprecate_models.py", "name": "update_relative_imports", "start_line": 111, "type": "function", "chroma:document": "CODE:\n\ndef update_relative_imports(filename, model):\n    with open(filename, \"r\") as f:\n        filelines = f.read()\n\n    new_file_lines = []\n    for line in filelines.split(\"\\n\"):\n        if line.startswith(\"from ..\"):\n            new_file_lines.append(line.replace(\"from ..\", \"from ...\"))\n        else:\n            new_file_lines.append(line)\n\n    with open(filename, \"w\") as f:\n\n\nSUMMARY:\n**Summary**\n\n`update_relative_imports()` function updates relative import paths in a Python file. It replaces `..` with `...` to update absolute imports on all lines. The updated file is written back to the original location.\n\n**Key Implementation Details**\n\n* Replaces `from ..` with `from ...` across all lines to convert relative imports to absolute ones.\n* Uses `split(\"\\n\")` to split the file into individual lines for processing.\n\n**Notable Dependencies/Requirements**\n\n* None explicit dependencies. However, it's assumed that this function runs in a Python environment with necessary modules and libraries available."}


created_at: 2025-02-05 09:13:58
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/deprecate_models.py:126-142
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 142, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/deprecate_models.py", "name": "remove_copied_from_statements", "start_line": 126, "type": "function", "chroma:document": "CODE:\n\ndef remove_copied_from_statements(model):\n    model_path = REPO_PATH / f\"src/transformers/models/{model}\"\n    for file in os.listdir(model_path):\n        if file == \"__pycache__\":\n            continue\n        file_path = model_path / file\n        with open(file_path, \"r\") as f:\n            file_lines = f.read()\n\n        new_file_lines = []\n        for line in file_lines.split(\"\\n\"):\n            if \"# Copied from\" in line:\n                continue\n            new_file_lines.append(line)\n\n        with open(file_path, \"w\") as f:\n\n\nSUMMARY:\n**Summary**\n\nThis function, `remove_copied_from_statements`, removes commented lines starting with \"# Copied from\" from the specified model files. The main purpose is to clean upTransformer models.\n\n**Key Implementation Details:**\n\n* Iterates over lines in each file in the model directory, skipping lines containing \"# Copied from\".\n* Uses `os` and `re` modules implicitly for listing directory contents and file reading/writing.\n* Writes modified line-by-line to new file, omitting lines with the specified comment.\n\n**Notable Dependencies/Requirements:**\n\n* `REPO_PATH`: reference path (relative to current working directory) containing model files.\n* `os`: file system operations.\n* `re`: regular expression matching used implicitly for pattern searching."}


created_at: 2025-02-05 09:13:58
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/deprecate_models.py:145-158
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 158, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/deprecate_models.py", "name": "move_model_files_to_deprecated", "start_line": 145, "type": "function", "chroma:document": "CODE:\n\ndef move_model_files_to_deprecated(model):\n    model_path = REPO_PATH / f\"src/transformers/models/{model}\"\n    deprecated_model_path = REPO_PATH / f\"src/transformers/models/deprecated/{model}\"\n\n    if not os.path.exists(deprecated_model_path):\n        os.makedirs(deprecated_model_path)\n\n    for file in os.listdir(model_path):\n        if file == \"__pycache__\":\n            continue\n        repo.git.mv(f\"{model_path}/{file}\", f\"{deprecated_model_path}/{file}\")\n\n        # For deprecated files, we then need to update the relative imports\n\n\nSUMMARY:\n**Summary: Migrate Model Files to Deprecation**\n\n### Purpose and Functionality\n\nMigrates model files from `src/transformers/models/{model}` to `src/transformers/models/deprecated/{model}`.\n\n### Key Implementation Details\n\n* Uses `repo.git.mv` to rename files, updating them at the new location.\n* Calls `update_relative_imports` to update relative imports for deprecated files.\n\n### Notable Dependencies or Requirements\n\n* `REPO_PATH`: Base path of the repository\n* `os.path.exists`, `os.makedirs`, `repo.git.mv`\n* `update_relative_imports`\n\nNote: The exact implementation details may vary based on the `update_relative_imports` function, which is not shown in this snippet."}


created_at: 2025-02-05 09:13:58
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/deprecate_models.py:161-165
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 165, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/deprecate_models.py", "name": "delete_model_tests", "start_line": 161, "type": "function", "chroma:document": "CODE:\n\ndef delete_model_tests(model):\n    tests_path = REPO_PATH / f\"tests/models/{model}\"\n\n    if os.path.exists(tests_path):\n\n\nSUMMARY:\n**Function Summary: `delete_model_tests`**\n\n Deletes model test files from the repository, removing related Git tracking.\n\n**Main Purpose and Functionality:**\nThe primary purpose is to delete model test files at the specified path if they exist. It utilizes the Git command-line interface (`repo.git.rm`) to remove these files and their associated Git tracking.\n\n**Key Implementation Details:**\n\n* Uses `os.path.exists` to check for file existence.\n* Utilizes `REPO_PATH / f\"tests/models/{model}\"` as the test path, referencing the `REPO_PATH` constant.\n* Executes `repo.git.rm(\"-r\", tests_path)` to remove the files and their tracking using Git.\n\n**Notable Dependencies or Requirements:**\n\n* References `repo`, an existing Git repository object.\n* Utilizes external tool, `os`.\n* `REPO_PATH`, a constant referencing the repository path."}


created_at: 2025-02-05 09:13:58
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/deprecate_models.py:168-169
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 169, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/deprecate_models.py", "name": "get_line_indent", "start_line": 168, "type": "function", "chroma:document": "CODE:\n\ndef get_line_indent(s):\n\n\nSUMMARY:\n**Summary:**\n\nCalculates the line indentation by subtracting the length of the stripped leading spaces from the total string length.\n\n**Key Implementations:**\n\n* `lstrip()` method is used to remove trailing whitespace characters\n* Function takes a single argument `s` (string)\n\n**Notable Dependencies/Requirements:**\n\n* None mentioned\n\nThis function is used to calculate and return the indentation line in a given string. The implementation relies on the built-in `lstrip()` method, providing a concise and efficient solution for checking leading whitespace characters."}


created_at: 2025-02-05 09:13:58
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/deprecate_models.py:172-192
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 192, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/deprecate_models.py", "name": "update_main_init_file", "start_line": 172, "type": "function", "chroma:document": "CODE:\n\ndef update_main_init_file(models):\n    \"\"\"\n    Replace all instances of model.model_name with model.deprecated.model_name in the __init__.py file\n\n    Args:\n        models (List[str]): The models to mark as deprecated\n    \"\"\"\n    filename = REPO_PATH / \"src/transformers/__init__.py\"\n    with open(filename, \"r\") as f:\n        init_file = f.read()\n\n    # 1. For each model, find all the instances of model.model_name and replace with model.deprecated.model_name\n    for model in models:\n        init_file = init_file.replace(f'models.{model}\"', f'models.deprecated.{model}\"')\n        init_file = init_file.replace(f\"models.{model} import\", f\"models.deprecated.{model} import\")\n\n    with open(filename, \"w\") as f:\n        f.write(init_file)\n\n    # 2. Resort the imports\n\n\nSUMMARY:\n**Function Summary: `update_main_init_file`**\n\n**Primary Purpose:** Replaces deprecated model names in `__init__.py` file with new, deprecated versions.\n\n**Key Implementation Details:**\n\n* Replaces instances of `model.model_name` with `model.deprecated.model_name` in the `__init__.py` file.\n* Also updates import statements to use the deprecated module name.\n\n**Notable Dependencies and Requirements:**\n\n* `REPO_PATH`: The absolute path to the repository's source directory (`\"src/transformers/__init__.py\"`).\n* Uses an external function `sort_imports_in_all_inits(check_only=False)` (not shown in the provided code snippet)."}


created_at: 2025-02-05 09:13:58
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/deprecate_models.py:195-215
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 215, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/deprecate_models.py", "name": "remove_model_references_from_file", "start_line": 195, "type": "function", "chroma:document": "CODE:\n\ndef remove_model_references_from_file(filename, models, condition):\n    \"\"\"\n    Remove all references to the given models from the given file\n\n    Args:\n        filename (str): The file to remove the references from\n        models (List[str]): The models to remove\n        condition (Callable): A function that takes the line and model and returns True if the line should be removed\n    \"\"\"\n    filename = REPO_PATH / filename\n    with open(filename, \"r\") as f:\n        init_file = f.read()\n\n    new_file_lines = []\n    for i, line in enumerate(init_file.split(\"\\n\")):\n        if any(condition(line, model) for model in models):\n            continue\n        new_file_lines.append(line)\n\n    with open(filename, \"w\") as f:\n\n\nSUMMARY:\n**Summary:**\n\n`remove_model_references_from_file` function:\n\n* Purpose: Removes references to specified models from a given file by applying a filter condition.\n* Key Implementation Details:\n\t+ Reads the entire file into memory using `init_file = f.read()`.\n\t+ Iterates over each line, skipping those that match any model based on the `condition` function.\n\t+ Writes the new filtered file contents back to disk.\n* Notable Dependencies and Requirements:\n\t+ Uses Python's built-in `open()` and `read/write` functions for file input/output.\n\t+ Assumes a `REPO_PATH` constant or variable defining the repository root path.\n\t+ Accepts a `condition` function with signature `Callable[[str, str], bool]`.\n\n**RAG (Reusable Anonymous Code) System:**\n\n```markdown\n# remove_model_references_from_file\n### Purpose\nRemoves references to specified models from a given file.\n\n### Key Implementation Details\n\n* Reads entire file into memory.\n* Applies filter condition to line-model pairs.\n* Writes new filtered file contents back to disk.\n\n### Dependencies and Requirements\n\n* Python built-in `open` and `read/write`.\n* `REPO_PATH` constant or variable.\n* `condition` function signature: `Callable[[str, str], bool]`.\n```"}


created_at: 2025-02-05 09:13:58
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/deprecate_models.py:218-265
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 265, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/deprecate_models.py", "name": "remove_model_config_classes_from_config_check", "start_line": 218, "type": "function", "chroma:document": "CODE:\n\ndef remove_model_config_classes_from_config_check(model_config_classes):\n    \"\"\"\n    Remove the deprecated model config classes from the check_config_attributes.py file\n\n    Args:\n        model_config_classes (List[str]): The model config classes to remove e.g. [\"BertConfig\", \"DistilBertConfig\"]\n    \"\"\"\n    filename = REPO_PATH / \"utils/check_config_attributes.py\"\n    with open(filename, \"r\") as f:\n        check_config_attributes = f.read()\n\n    # Keep track as we have to delete comment above too\n    in_special_cases_to_allow = False\n    in_indent = False\n    new_file_lines = []\n\n    for line in check_config_attributes.split(\"\\n\"):\n        indent = get_line_indent(line)\n        if (line.strip() == \"SPECIAL_CASES_TO_ALLOW = {\") or (line.strip() == \"SPECIAL_CASES_TO_ALLOW.update(\"):\n            in_special_cases_to_allow = True\n\n        elif in_special_cases_to_allow and indent == 0 and line.strip() in (\"}\", \")\"):\n            in_special_cases_to_allow = False\n\n        if in_indent:\n            if line.strip().endswith((\"]\", \"],\")):\n                in_indent = False\n            continue\n\n        if in_special_cases_to_allow and any(\n            model_config_class in line for model_config_class in model_config_classes\n        ):\n            # Remove comments above the model config class to remove\n            while new_file_lines[-1].strip().startswith(\"#\"):\n                new_file_lines.pop()\n\n            if line.strip().endswith(\"[\"):\n                in_indent = True\n\n            continue\n\n        elif any(model_config_class in line for model_config_class in model_config_classes):\n            continue\n\n        new_file_lines.append(line)\n\n    with open(filename, \"w\") as f:\n\n\nSUMMARY:\n**Function Summary:**\n\n`remove_model_config_classes_from_config_check`\n\nRemoves deprecated model configuration classes from the `check_config_attributes.py` file.\n\n**Purpose and Functionality:**\n\n* Reads the contents of the specified Python file (`utils/check_config_attributes.py`)\n* Iterates through each line, tracking indentation and special cases\n* Removes comments above deprecated model config classes and rewrites the content in a new file\n\n**Key Implementation Details:**\n\n* Uses regex-free text processing to identify lines with model config classes\n* Preserves indentation and formatting when rewriting the content\n* Handles edge cases such as nested brackets and escaped newline characters\n\n**Dependencies and Interfaces:**\n\n* None, but may utilize external `REPO_PATH` constant in a larger context.\n* Does not implement or inherit any specific interfaces.\n\n**Method Signatures:**\n\n* `remove_model_config_classes_from_config_check(model_config_classes)`: takes a list of string model config class names as input"}


created_at: 2025-02-05 09:13:58
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/deprecate_models.py:268-299
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 299, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/deprecate_models.py", "name": "add_models_to_deprecated_models_in_config_auto", "start_line": 268, "type": "function", "chroma:document": "CODE:\n\ndef add_models_to_deprecated_models_in_config_auto(models):\n    \"\"\"\n    Add the models to the DEPRECATED_MODELS list in configuration_auto.py and sorts the list\n    to be in alphabetical order.\n    \"\"\"\n    filepath = REPO_PATH / \"src/transformers/models/auto/configuration_auto.py\"\n    with open(filepath, \"r\") as f:\n        config_auto = f.read()\n\n    new_file_lines = []\n    deprecated_models_list = []\n    in_deprecated_models = False\n    for line in config_auto.split(\"\\n\"):\n        if line.strip() == \"DEPRECATED_MODELS = [\":\n            in_deprecated_models = True\n            new_file_lines.append(line)\n        elif in_deprecated_models and line.strip() == \"]\":\n            in_deprecated_models = False\n            # Add the new models to deprecated models list\n            deprecated_models_list.extend([f'    \"{model}\", ' for model in models])\n            # Sort so they're in alphabetical order in the file\n            deprecated_models_list = sorted(deprecated_models_list)\n            new_file_lines.extend(deprecated_models_list)\n            # Make sure we still have the closing bracket\n            new_file_lines.append(line)\n        elif in_deprecated_models:\n            deprecated_models_list.append(line)\n        else:\n            new_file_lines.append(line)\n\n    with open(filepath, \"w\") as f:\n\n\nSUMMARY:\n**Function Summary:**\n\n`add_models_to_deprecated_models_in_config_auto(models)`\n\nAdds the input `models` to the list of deprecated models in `configuration_auto.py`, updating its contents and sorting the list alphabetically.\n\n**Key Implementation Details:**\n\n- Reads `DEPRECATED_MODELS` from `configuration_auto.py`.\n- Appends new models to the list in alphabetical order.\n- Updates the file with the modified content.\n\n**Notable Dependencies/Requirements:**\n\n- `REPO_PATH`: reference to a repository-specific path ( exact class/function name required)\n- External dependency on:\n  - `os` module for file system operations\n  - `pathlib` module for handling file paths\n  - `shlex` module (not explicitly used but possibly implied by the `split()` and `strip()` functions)\n\n**Interface/Import Notes:**\n\n- No explicit import statements are provided, suggesting that necessary modules and classes will be imported within this code."}


created_at: 2025-02-05 09:13:58
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/deprecate_models.py:302-370
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 370, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/deprecate_models.py", "name": "deprecate_models", "start_line": 302, "type": "function", "chroma:document": "CODE:\n\ndef deprecate_models(models):\n    # Get model info\n    skipped_models = []\n    models_info = defaultdict(dict)\n    for model in models:\n        single_model_info = extract_model_info(model)\n        if single_model_info is None:\n            skipped_models.append(model)\n        else:\n            models_info[model] = single_model_info\n\n    model_config_classes = []\n    for model, model_info in models_info.items():\n        if model in CONFIG_MAPPING:\n            model_config_classes.append(CONFIG_MAPPING[model].__name__)\n        elif model_info[\"model_doc_name\"] in CONFIG_MAPPING:\n            model_config_classes.append(CONFIG_MAPPING[model_info[\"model_doc_name\"]].__name__)\n        else:\n            skipped_models.append(model)\n            print(f\"Model config class not found for model: {model}\")\n\n    # Filter out skipped models\n    models = [model for model in models if model not in skipped_models]\n\n    if skipped_models:\n        print(f\"Skipped models: {skipped_models} as the model doc or model path could not be found.\")\n    print(f\"Models to deprecate: {models}\")\n\n    # Remove model config classes from config check\n    print(\"Removing model config classes from config checks\")\n    remove_model_config_classes_from_config_check(model_config_classes)\n\n    tip_message = build_tip_message(get_last_stable_minor_release())\n\n    for model, model_info in models_info.items():\n        print(f\"Processing model: {model}\")\n        # Add the tip message to the model doc page directly underneath the title\n        print(\"Adding tip message to model doc page\")\n        insert_tip_to_model_doc(model_info[\"model_doc_path\"], tip_message)\n\n        # Remove #Copied from statements from model's files\n        print(\"Removing #Copied from statements from model's files\")\n        remove_copied_from_statements(model)\n\n        # Move the model file to deprecated: src/transfomers/models/model -> src/transformers/models/deprecated/model\n        print(\"Moving model files to deprecated for model\")\n        move_model_files_to_deprecated(model)\n\n        # Delete the model tests: tests/models/model\n        print(\"Deleting model tests\")\n        delete_model_tests(model)\n\n    # # We do the following with all models passed at once to avoid having to re-write the file multiple times\n    print(\"Updating __init__.py file to point to the deprecated models\")\n    update_main_init_file(models)\n\n    # Remove model references from other files\n    print(\"Removing model references from other files\")\n    remove_model_references_from_file(\n        \"src/transformers/models/__init__.py\", models, lambda line, model: model == line.strip().strip(\",\")\n    )\n    remove_model_references_from_file(\n        \"utils/slow_documentation_tests.txt\", models, lambda line, model: \"/\" + model + \"/\" in line\n    )\n    remove_model_references_from_file(\"utils/not_doctested.txt\", models, lambda line, model: \"/\" + model + \"/\" in line)\n\n    # Add models to DEPRECATED_MODELS in the configuration_auto.py\n    print(\"Adding models to DEPRECATED_MODELS in configuration_auto.py\")\n\n\nSUMMARY:\n**Function Summary: deprecate_models**\n\n### Purpose and Functionality:\n\nThe `deprecate_models` function is designed to deprecate specific models by updating their documentation, removing references, and refactoring their configurations. It iterates through a list of models, extracting information about each model from an `extract_model_info` function call. The function checks if the model has a configured class in the `CONFIG_MAPPING`, adds a tip message to the model's documentation page, removes redundant statements, moves the model files to the deprecated directory, deletes test files, and updates other files that reference the models.\n\n### Key Implementation Details:\n\n*   Iterates through a list of models using `models` as input.\n*   Calls `extract_model_info(model)` for each model to gather information.\n*   Checks if a configured class exists in `CONFIG_MAPPING` or by name within `CONFIG_MAPPING`.\n*   Updates the last stable minor release version using `get_last_stable_minor_release()`.\n*   Uses `build_tip_message(tip_message)`, `remove_model_config_classes_from_config_check(model_config_classes)`, and other functions to finalize model depreciation.\n*   Iterates over models, processing each one individually using `insert_tip_to_model_doc`, `remove_copied_from_statements`, `move_model_files_to_deprecated`, `delete_model_tests`, and updates files like `update_main_init_file` to point to the deprecated models.\n\n### Dependencies and Requirements:\n\n*   Uses `defaultdict` from Python's standard library for creating default dictionaries.\n*   References `CONFIG_MAPPING` (a dictionary of model configurations) at least once during execution, which must be defined elsewhere in the codebase.\n*   Calls functions with external dependencies like `remove_model_references_from_file`, `update_main_init_file`, and others that reference files or classes not specified within this function.\n\n### Notable Interfaces:\n\nThe function provides a few notable interfaces to other components of the system:\n    \n*   `CONFIG_MAPPING`: A dictionary mapping model names or documentation lines to their respective configuration classes.\n*   External dependencies such as `update_main_init_file`, `remove_model_references_from_file` (referencing specific files or directories), and `build_tip_message`."}


created_at: 2025-02-05 09:13:58
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/sort_auto_mappings.py:49-98
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 98, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/sort_auto_mappings.py", "name": "sort_auto_mapping", "start_line": 49, "type": "function", "chroma:document": "CODE:\n\ndef sort_auto_mapping(fname: str, overwrite: bool = False) -> Optional[bool]:\n    \"\"\"\n    Sort all auto mappings in a file.\n\n    Args:\n        fname (`str`): The name of the file where we want to sort auto-mappings.\n        overwrite (`bool`, *optional*, defaults to `False`): Whether or not to fix and overwrite the file.\n\n    Returns:\n        `Optional[bool]`: Returns `None` if `overwrite=True`. Otherwise returns `True` if the file has an auto-mapping\n        improperly sorted, `False` if the file is okay.\n    \"\"\"\n    with open(fname, \"r\", encoding=\"utf-8\") as f:\n        content = f.read()\n\n    lines = content.split(\"\\n\")\n    new_lines = []\n    line_idx = 0\n    while line_idx < len(lines):\n        if _re_intro_mapping.search(lines[line_idx]) is not None:\n            # Start of a new mapping!\n            indent = len(re.search(r\"^(\\s*)\\S\", lines[line_idx]).groups()[0]) + 8\n            while not lines[line_idx].startswith(\" \" * indent + \"(\"):\n                new_lines.append(lines[line_idx])\n                line_idx += 1\n\n            blocks = []\n            while lines[line_idx].strip() != \"]\":\n                # Blocks either fit in one line or not\n                if lines[line_idx].strip() == \"(\":\n                    start_idx = line_idx\n                    while not lines[line_idx].startswith(\" \" * indent + \")\"):\n                        line_idx += 1\n                    blocks.append(\"\\n\".join(lines[start_idx : line_idx + 1]))\n                else:\n                    blocks.append(lines[line_idx])\n                line_idx += 1\n\n            # Sort blocks by their identifiers\n            blocks = sorted(blocks, key=lambda x: _re_identifier.search(x).groups()[0])\n            new_lines += blocks\n        else:\n            new_lines.append(lines[line_idx])\n            line_idx += 1\n\n    if overwrite:\n        with open(fname, \"w\", encoding=\"utf-8\") as f:\n            f.write(\"\\n\".join(new_lines))\n    else:\n\n\nSUMMARY:\n**Function Summary: `sort_auto_mapping`**\n\nThe `sort_auto_mapping` function sorts the auto-mappings in a given file.\n\n### Purpose and Functionality:\n\nAutomatically rearranges indented lines containing mappings in the specified file to adhere to a specific format. Compares and returns whether the original formatting is preserved upon sorting (when `overwrite=False`) or if changes were made for overwriting (`overwrite=True`).\n\n### Key Implementation Details:\n\n1.  **Regex-based Pattern Matching**:\n    *   `_re_intro_mapping.search()`: Identifies the start of a new mapping in lines.\n2.  **Blocks and Lines Processing**:\n    *   `indentation calculation`: Calculates indentation length before processing each mapping block.\n    *   `block decomposition` (`if lines[line_idx].strip() == \"(\":`): Tackles blocks divided across multiple lines with indentation matching the first line's formatting.\n3.  **Sorting Algorithm**:\n    *   `blocks sorted by ids` (`sorted(blocks, key=lambda x: _re_identifier.search(x).groups()[0])`): Orders and groups mappings according to their unique identifiers.\n\n### Notable Dependencies or Requirements:\n\n1.  `re` module (regular expressions) for pattern matching.\n2.  External file operations via `open` function with UTF-8 encoding.\n\n**API Reference:**\n\n*   `_re_intro_mapping`, `_re_identifier`: Externally defined regular expression patterns used within the sorting logic, not part of this summary but crucial for implementation context.\n\n### Example Usage:\n```python\nresult = sort_auto_mapping('path/to/file.txt')\nprint(result)  # Returns 'None' if overwrite is False and file has been properly sorted, False otherwise.\n```"}


created_at: 2025-02-05 09:13:59
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/sort_auto_mappings.py:101-116
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 116, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/sort_auto_mappings.py", "name": "sort_all_auto_mappings", "start_line": 101, "type": "function", "chroma:document": "CODE:\n\ndef sort_all_auto_mappings(overwrite: bool = False):\n    \"\"\"\n    Sort all auto mappings in the library.\n\n    Args:\n        overwrite (`bool`, *optional*, defaults to `False`): Whether or not to fix and overwrite the file.\n    \"\"\"\n    fnames = [os.path.join(PATH_TO_AUTO_MODULE, f) for f in os.listdir(PATH_TO_AUTO_MODULE) if f.endswith(\".py\")]\n    diffs = [sort_auto_mapping(fname, overwrite=overwrite) for fname in fnames]\n\n    if not overwrite and any(diffs):\n        failures = [f for f, d in zip(fnames, diffs) if d]\n        raise ValueError(\n            f\"The following files have auto mappings that need sorting: {', '.join(failures)}. Run `make style` to fix\"\n            \" this.\"\n\n\nSUMMARY:\n**Summary**\n\nFunction `sort_all_auto_mappings`: Sorts all auto mappings in the library, fixing inconsistencies if required. \n\n### Purpose\nSynchronizes Python file auto mappings within a module.\n\n### Implementation Details\n\n- Fetches relevant files using list comprehension with path manipulation (`os.path.join`).\n- Iterates through each file, calls `sort_auto_mapping` to sort its mappings.\n- Checks for non-overwritten changes after sorting; raises an error if updates are detected."}


created_at: 2025-02-05 09:13:59
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/test_module/custom_tokenization_fast.py:5-7
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 7, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/test_module/custom_tokenization_fast.py", "name": "CustomTokenizerFast", "start_line": 5, "type": "class", "chroma:document": "CODE:\n\nclass CustomTokenizerFast(BertTokenizerFast):\n    slow_tokenizer_class = CustomTokenizer\n\n\nSUMMARY:\n**Summary:**\n\nThis class `CustomTokenizerFast` inherits from `BertTokenizerFast`. It overrides the default tokenizer by specifying its own tokenizer class as `SlowTokenizer`. The primary purpose is to enable the use of a custom, slow tokenization mechanism in place of the FastTokenizer.\n\n**Key Implementation Details:**\n\n* Inherited Class: `BertTokenizerFast`\n* Implemented Class: `CustomTokenizer` (specifically assigned as `slow_tokenizer_class`)\n* No additional functionality or method implementations provided; primarily serves as an override for the default tokenizer configuration."}


created_at: 2025-02-05 09:13:59
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/test_module/custom_tokenization.py:3-4
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 4, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/test_module/custom_tokenization.py", "name": "CustomTokenizer", "start_line": 3, "type": "class", "chroma:document": "CODE:\n\nclass CustomTokenizer(BertTokenizer):\n\n\nSUMMARY:\n**Summary**\n\nClass: `CustomTokenizer`\nInherits from: `BertTokenizer`\n\n**Purpose and Functionality**: Creates a custom version of the BERT tokenizer.\n\n**Key Implementation Details**:\n\n* No specific tokenization algorithms or patterns are defined, inheriting functionality from `BertTokenizer`.\n* No method signatures or interfaces are overridden by default.\n \n\n**Notable Dependencies/Requirements**:\n* Inherits from: `BertTokenizer`\n* External dependencies not specified.\n\nNo key implementation details or interface implementations to note."}


created_at: 2025-02-05 09:13:59
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/test_module/custom_modeling.py:7-18
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 18, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/test_module/custom_modeling.py", "name": "CustomModel", "start_line": 7, "type": "class", "chroma:document": "CODE:\n\nclass CustomModel(PreTrainedModel):\n    config_class = CustomConfig\n\n    def __init__(self, config):\n        super().__init__(config)\n        self.linear = torch.nn.Linear(config.hidden_size, config.hidden_size)\n\n    def forward(self, x):\n        return self.linear(x)\n\n    def _init_weights(self, module):\n\n\nSUMMARY:\n**Summary:**\n\nThe `CustomModel` class is a custom PyTorch model derived from `PreTrainedModel`. It uses the `CustomConfig` configuration.\n\n**Main Purpose and Functionality:**\nIt defines a basic neural network architecture with one linear layer, where the input `x` passes through this layer to produce an output.\n\n**Key Implementation Details:**\n\n*   A linear layer (`self.linear`) with input/ouput size of `config.hidden_size`.\n*   The `forward` method applies the linear transformation to the input `x`.\n\n**Notable Dependencies and Requirements:**\n\n*   Import from PyTorch (`torch.nn.Linear`).\n*   Inheritance from `PreTrainedModel`."}


created_at: 2025-02-05 09:13:59
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/test_module/custom_modeling.py:21-32
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 32, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/test_module/custom_modeling.py", "name": "NoSuperInitModel", "start_line": 21, "type": "class", "chroma:document": "CODE:\n\nclass NoSuperInitModel(PreTrainedModel):\n    config_class = NoSuperInitConfig\n\n    def __init__(self, config):\n        super().__init__(config)\n        self.linear = torch.nn.Linear(config.attribute, config.attribute)\n\n    def forward(self, x):\n        return self.linear(x)\n\n    def _init_weights(self, module):\n\n\nSUMMARY:\n**Summary:**\n\nThe `NoSuperInitModel` class is a PyTorch-based pre-trained model that extends the `PreTrainedModel` class. It defines:\n\n* A linear layer with a configuration attribute `config.attribute`.\n* A forward pass through the linear layer, mapping input `x` to its output.\n* An `_init_weights` method that sets no weights (implies super-init behavior).\n\n**Key Details:**\n\n* **Linear Layer**: `self.linear = torch.nn.Linear(config.attribute, config.attribute)`\n* **Forward Pass**: `return self.linear(x)`\n* **_init_weights Method**: Implicitly inherits from superclass's behavior\n\n**Dependencies/Requirements:**\n\n* PyTorch (`torch`)\n* `PreTrainedModel` class (inherited)\n* `NoSuperInitConfig` configuration class ( referenced)"}


created_at: 2025-02-05 09:13:59
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/test_module/custom_image_processing.py:3-4
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 4, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/test_module/custom_image_processing.py", "name": "CustomImageProcessor", "start_line": 3, "type": "class", "chroma:document": "CODE:\n\nclass CustomImageProcessor(CLIPImageProcessor):\n\n\nSUMMARY:\n**Code Summary:**\n\n*   Class Name: `CustomImageProcessor`\n*   Inheritance: Extends `CLIPImageProcessor` to create a custom image processor.\n\n**Additional Details:**\n\n*   No specific implementation details are provided, as the class solely inherits from `CLIPImageProcessor`."}


created_at: 2025-02-05 09:13:59
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/test_module/custom_pipeline.py:5-8
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 8, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/test_module/custom_pipeline.py", "name": "softmax", "start_line": 5, "type": "function", "chroma:document": "CODE:\n\ndef softmax(outputs):\n    maxes = np.max(outputs, axis=-1, keepdims=True)\n    shifted_exp = np.exp(outputs - maxes)\n\n\nSUMMARY:\n**Concise Summary**\n\nThis function implements the softmax activation function. It takes an input array `outputs` of shape `(num_classes, batch_size)` and returns a normalized value for each class.\n\nThe primary purpose is to prevent output values from growing exponentially large by normalizing them over their maximum value across classes.\n\n**Key Implementation Details:**\n\n- Uses NumPy's vectorized operations to calculate the softmax output.\n- Employs shift-invariant exponentiation (`outputs - maxes`) to mitigate numerical instability issues.\n\n**Notable Dependencies and Requirements:**\n\n* `numpy` (imported as `np` for array operations)\n* `outputs` input should be an array of shape `(num_classes, batch_size)` containing output values."}


created_at: 2025-02-05 09:13:59
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/test_module/custom_pipeline.py:11-32
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 32, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/test_module/custom_pipeline.py", "name": "PairClassificationPipeline", "start_line": 11, "type": "class", "chroma:document": "CODE:\n\nclass PairClassificationPipeline(Pipeline):\n    def _sanitize_parameters(self, **kwargs):\n        preprocess_kwargs = {}\n        if \"second_text\" in kwargs:\n            preprocess_kwargs[\"second_text\"] = kwargs[\"second_text\"]\n        return preprocess_kwargs, {}, {}\n\n    def preprocess(self, text, second_text=None):\n        return self.tokenizer(text, text_pair=second_text, return_tensors=self.framework)\n\n    def _forward(self, model_inputs):\n        return self.model(**model_inputs)\n\n    def postprocess(self, model_outputs):\n        logits = model_outputs.logits[0].numpy()\n        probabilities = softmax(logits)\n\n        best_class = np.argmax(probabilities)\n        label = self.model.config.id2label[best_class]\n        score = probabilities[best_class].item()\n        logits = logits.tolist()\n\n\nSUMMARY:\n**PairClassificationPipeline Class Summary**\n\n* **Purpose:** This class implements a pipeline for pair classification tasks using a pre-trained model. It sanitizes input parameters, preprocesses text inputs with optional second text, and postprocesses model outputs to extract confidence scores and predicted labels.\n\n* **Key Implementation Details:**\n\t+ `preprocess` method takes in text inputs and an optional second text, and uses the `tokenizer` to generate input tensors in a specific framework.\n\t+ `_forward` method passes the preprocessed input to the trained model for inference.\n\t+ `postprocess` method extracts attention scores from the model output, converts them to probabilities using softmax, and determines the predicted label by finding the class with the highest probability.\n\n* **Notable Dependencies:**\n\t+ `tokenizer`: a tokenizer object used in the `preprocess` method\n\t+ `model`: a pre-trained model instance that takes in input tensors in the same framework as the pipeline\n\t+ `self.framework`: the specific framework used for tensor generation (e.g. PyTorch, TensorFlow)\n\t+ `softmax`, `logits`, and related functions: commonly used mathematical functions for probability calculations"}


created_at: 2025-02-05 09:13:59
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/test_module/custom_configuration.py:3-8
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 8, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/test_module/custom_configuration.py", "name": "CustomConfig", "start_line": 3, "type": "class", "chroma:document": "CODE:\n\nclass CustomConfig(PretrainedConfig):\n    model_type = \"custom\"\n\n    def __init__(self, attribute=1, **kwargs):\n        self.attribute = attribute\n\n\nSUMMARY:\n**Summary**\n\n`CustomConfig` is a custom configuration class for the `PretrainedConfig` base class. It defines a model type as \"custom\" and extends initialization from its parent with an additional `attribute` parameter, defaulting to 1.\n\n**Key Implementation Details:**\n\n* Customizes model type to `model_type = \"custom\"`\n* Extends __init__ from `PretrainedConfig` with `attribute` parameter\n\n**Notable Dependencies or Requirements:**\nNone explicitly listed. Inherited interfaces: `PretrainedConfig`."}


created_at: 2025-02-05 09:13:59
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/test_module/custom_configuration.py:11-15
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 15, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/test_module/custom_configuration.py", "name": "NoSuperInitConfig", "start_line": 11, "type": "class", "chroma:document": "CODE:\n\nclass NoSuperInitConfig(PretrainedConfig):\n    model_type = \"custom\"\n\n    def __init__(self, attribute=1, **kwargs):\n\n\nSUMMARY:\n**Summary**\n\nThis `NoSuperInitConfig` class inherits from `PretrainedConfig`, defining a custom configuration for a specific model. The primary purpose is to initialize the configuration with an optional `attribute` parameter.\n\n**Key Details**\n\n* Inherited from: `PretrainedConfig`\n* Model type: \"custom\"\n* Initialization:\n\t+ Required attribute: `attribute=1` (default value)\n\n**Notable Dependencies/Requirements**\n\n* None explicitly listed, but assumes access to `PretrainedConfig` class and related dependencies."}


created_at: 2025-02-05 09:13:59
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/test_module/custom_processing.py:3-5
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 5, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/test_module/custom_processing.py", "name": "CustomProcessor", "start_line": 3, "type": "class", "chroma:document": "CODE:\n\nclass CustomProcessor(ProcessorMixin):\n    feature_extractor_class = \"AutoFeatureExtractor\"\n\n\nSUMMARY:\n**Summary:**\n\nThis `CustomProcessor` class inherits from `ProcessorMixin` and defines two key attributes:\n\n* `feature_extractor_class`: specifies the use of the `AutoFeatureExtractor` component.\n* `tokenizer_class`: specifies the use of the `AutoTokenizer` component.\n\nNo additional implementation details or dependencies are defined for this class."}


created_at: 2025-02-05 09:13:59
topic: persistent://default/default/d7f79fe7-4053-4e56-8c53-540c743a4d8b
id: /home/gaby/projects/2025/transformers-4.48.1/utils/test_module/custom_feature_extraction.py:3-4
encoding: FLOAT32
metadata:
{"docstring": "", "end_line": 4, "file_path": "/home/gaby/projects/2025/transformers-4.48.1/utils/test_module/custom_feature_extraction.py", "name": "CustomFeatureExtractor", "start_line": 3, "type": "class", "chroma:document": "CODE:\n\nclass CustomFeatureExtractor(Wav2Vec2FeatureExtractor):\n\n\nSUMMARY:\n**Summary**\n\nA custom `CustomFeatureExtractor` class that inherits from the `Wav2Vec2FeatureExtractor` class, effectively extending the functionality of the parent class with no additional implementation.\n\n**Key Implementation Details**\n\n* Inherited from `Wav2Vec2FeatureExtractor`, implying an extension to the Wav2Vec2 model's feature extraction capabilities.\n* No additional methods or attributes are defined, suggesting a focus on leveraging the existing `Wav2Vec2FeatureExtractor` functionality with no new additions.\n\n**Notable Dependencies**\n\n* `Wav2Vec2FeatureExtractor`: The parent class from which `CustomFeatureExtractor` inherits.\n\n**Additional Requirements**\n\nNone."}


