# embedding_metadata

key: chroma:document
string_value:
CODE:

def eval_node(node):
    if isinstance(node, ast.Num):  # <number>
        return node.n
    elif isinstance(node, ast.BinOp):  # <left> <operator> <right>
        return MATH_OPERATORS[type(node.op)](eval_node(node.left), eval_node(node.right))
    elif isinstance(node, ast.UnaryOp):  # <operator> <operand> e.g., -1
        return MATH_OPERATORS[type(node.op)](eval_node(node.operand))
    else:


SUMMARY:
**Eval Node Function Summary**

### Purpose and Functionality

The `eval_node` function is a recursive traverser of abstract syntax tree (AST) nodes, evaluating arithmetic expressions by applying mathematical operators to their operands.

### Key Implementation Details

* Evaluates integer literal nodes (`ast.Num`) as-is.
* Unifies binary operation nodes (`ast.BinOp`): recursively evaluates left and right operands with operator-dependent functions from `MATH_OPERATORS`.
* Unifies unary operation nodes (`ast.UnaryOp`): recursively evaluates the operand with an operator-dependent function from `MATH_OPERATORS`.
* Raises a `TypeError` for unsupported node types.

### Dependencies

None, as this implementation does not rely on external dependencies. However, it utilizes:

* `ast` (Abstract Syntax Tree) module for parsing expressions.
* `MATH Operators` dictionary (not shown in provided code), which maps operator types to evaluation functions.
* No external libraries besides the Python standard library.


key: chroma:document
string_value:
CODE:

def replace_default_in_arg_description(description: str, default: Any) -> str:
    """
    Catches the default value in the description of an argument inside a docstring and replaces it by the value passed.

    Args:
        description (`str`): The description of an argument in a docstring to process.
        default (`Any`): The default value that whould be in the docstring of that argument.

    Returns:
       `str`: The description updated with the new default value.
    """
    # Lots of docstrings have `optional` or **opational** instead of *optional* so we do this fix here.
    description = description.replace("`optional`", OPTIONAL_KEYWORD)
    description = description.replace("**optional**", OPTIONAL_KEYWORD)
    if default is inspect._empty:
        # No default, make sure the description doesn't have any either
        idx = description.find(OPTIONAL_KEYWORD)
        if idx != -1:
            description = description[:idx].rstrip()
            if description.endswith(","):
                description = description[:-1].rstrip()
    elif default is None:
        # Default None are not written, we just set `*optional*`. If there is default that is not None specified in the
        # description, we do not erase it (as sometimes we set the default to `None` because the default is a mutable
        # object).
        idx = description.find(OPTIONAL_KEYWORD)
        if idx == -1:
            description = f"{description}, {OPTIONAL_KEYWORD}"
        elif re.search(r"defaults to `?None`?", description) is not None:
            len_optional = len(OPTIONAL_KEYWORD)
            description = description[: idx + len_optional]
    else:
        str_default = None
        # For numbers we may have a default that is given by a math operation (1/255 is really popular). We don't
        # want to replace those by their actual values.
        if isinstance(default, (int, float)) and re.search("defaults to `?(.*?)(?:`|$)", description) is not None:
            # Grab the default and evaluate it.
            current_default = re.search("defaults to `?(.*?)(?:`|$)", description).groups()[0]
            if default == eval_math_expression(current_default):
                try:
                    # If it can be directly converted to the type of the default, it's a simple value
                    str_default = str(type(default)(current_default))
                except Exception:
                    # Otherwise there is a math operator so we add a code block.
                    str_default = f"`{current_default}`"
            elif isinstance(default, enum.Enum) and default.name == current_default.split(".")[-1]:
                # When the default is an Enum (this is often the case for PIL.Image.Resampling), and the docstring
                # matches the enum name, keep the existing docstring rather than clobbering it with the enum value.
                str_default = f"`{current_default}`"

        if str_default is None:
            str_default = stringify_default(default)
        # Make sure default match
        if OPTIONAL_KEYWORD not in description:
            description = f"{description}, {OPTIONAL_KEYWORD}, defaults to {str_default}"
        elif _re_parse_description.search(description) is None:
            idx = description.find(OPTIONAL_KEYWORD)
            len_optional = len(OPTIONAL_KEYWORD)
            description = f"{description[:idx + len_optional]}, defaults to {str_default}"
        else:
            description = _re_parse_description.sub(rf"*optional*, defaults to {str_default}", description)



SUMMARY:
**Summary:**

Function `replace_default_in_arg_description` updates the argument description in a docstring to include the provided default value. It performs various fixes and transformations based on the type and content of the default value.

**Key Features:**

* Replaces occurrences of "optional" or "**optional**" with `OPTIONAL_KEYWORD`.
* Handles cases where the default value is `None`, `empty string` (e.g., empty line), or a complex math expression.
* For numbers, evaluates the math expression if it matches the default value and converts it to the correct type.

**Important Variables and Functions:**

* `OPTIONAL_KEYWORD`: "optional" or "**optional**"
* `description`: input docstring
* `default`: default value to be included in the updated description

**Notable Dependencies:**

* `inspect` for checking `default` value
* `re` for regular expression matching and replacing
* `eval_math_expression` function not provided in the code snippet, but used for evaluating math expressions

**Return Value:**

The updated docstring with the default value included.


key: chroma:document
string_value:
CODE:

def get_default_description(arg: inspect.Parameter) -> str:
    """
    Builds a default description for a parameter that was not documented.

    Args:
        arg (`inspect.Parameter`): The argument in the signature to generate a description for.

    Returns:
        `str`: The description.
    """
    if arg.annotation is inspect._empty:
        arg_type = "<fill_type>"
    elif hasattr(arg.annotation, "__name__"):
        arg_type = arg.annotation.__name__
    else:
        arg_type = str(arg.annotation)

    if arg.default is inspect._empty:
        return f"`{arg_type}`"
    elif arg.default is None:
        return f"`{arg_type}`, {OPTIONAL_KEYWORD}"
    else:
        str_default = stringify_default(arg.default)


SUMMARY:
**get_default_description function summary**

```markdown
Provides default description for an uncovered argument in Python function signatures.

### Key implementation details:

* Uses `inspect.Parameter` and related types (`inspect._empty`) to determine parameter type and defaults.
* Employs string formatting to construct the final default description.

### Notable dependencies and interfaces:

* `inspect`: Imported module providing type inspection utilities.
* `optional`: Likely an external library (e.g., `pydantic`) for optional parameter handling, referenced by `OPTIONAL_KEYWORD`.
```

### Primary purpose and functionality
Builds a default description string for an uncovered argument in a Python function signature, considering its type and potential default value.


key: chroma:document
string_value:
CODE:

def find_source_file(obj: Any) -> Path:
    """
    Finds the source file of an object.

    Args:
        obj (`Any`): The object whose source file we are looking for.

    Returns:
        `Path`: The source file.
    """
    module = obj.__module__
    obj_file = PATH_TO_TRANSFORMERS
    for part in module.split(".")[1:]:
        obj_file = obj_file / part


SUMMARY:
**Summary:**

Finds the source file of an object given its `__module__` attribute.

*   **Purpose:** Returns the path to the source file associated with the input object's module.
*   **Implementation Details:**
    *   Uses the `__module__` attribute of the input object to determine the module name.
    *   Joins the module name with a prefix (`PATH_TO_TRANSFORMERS`) to form the initial directory path.
    *   Iterates through the parts of the module name (skipping the first part, which is the package name), appending each part as a directory.
        *   `module.split(".")[1:]`: Extracts the namespace from the module name after removing the package prefix ('__main__.').
    *   Returns the final path with a `.py` suffix using `Path.with_suffix()`.
*   **Dependencies:** None explicitly mentioned; however, it relies on the `PATH_TO_TRANSFORMERS` constant and possibly `Path`.

**Key Variables/Functions:**

*   `obj`: Input object with its `__module__` attribute.
*   `module`: The namespace of the input object's module name (e.g., 'transformers.modeling.tokenization').
*   `PARTS OF THE MODULE NAME`: An array consisting of module parts excluding the package index prefix (`__main__.`).


key: chroma:document
string_value:
CODE:

def match_docstring_with_signature(obj: Any) -> Optional[Tuple[str, str]]:
    """
    Matches the docstring of an object with its signature.

    Args:
        obj (`Any`): The object to process.

    Returns:
        `Optional[Tuple[str, str]]`: Returns `None` if there is no docstring or no parameters documented in the
        docstring, otherwise returns a tuple of two strings: the current documentation of the arguments in the
        docstring and the one matched with the signature.
    """
    if len(getattr(obj, "__doc__", "")) == 0:
        # Nothing to do, there is no docstring.
        return

    # Read the docstring in the source code to see if there is a special command to ignore this object.
    try:
        source, _ = inspect.getsourcelines(obj)
    except OSError:
        source = []

    idx = 0
    while idx < len(source) and '"""' not in source[idx]:
        idx += 1

    ignore_order = False
    if idx < len(source):
        line_before_docstring = source[idx - 1]
        if re.search(r"^\s*#\s*no-format\s*$", line_before_docstring):
            # This object is ignored
            return
        elif re.search(r"^\s*#\s*ignore-order\s*$", line_before_docstring):
            ignore_order = True

    # Read the signature
    signature = inspect.signature(obj).parameters

    obj_doc_lines = obj.__doc__.split("\n")
    # Get to the line where we start documenting arguments
    idx = 0
    while idx < len(obj_doc_lines) and _re_args.search(obj_doc_lines[idx]) is None:
        idx += 1

    if idx == len(obj_doc_lines):
        # Nothing to do, no parameters are documented.
        return

    indent = find_indent(obj_doc_lines[idx])
    arguments = {}
    current_arg = None
    idx += 1
    start_idx = idx
    # Keep going until the arg section is finished (nonempty line at the same indent level) or the end of the docstring.
    while idx < len(obj_doc_lines) and (
        len(obj_doc_lines[idx].strip()) == 0 or find_indent(obj_doc_lines[idx]) > indent
    ):
        if find_indent(obj_doc_lines[idx]) == indent + 4:
            # New argument -> let's generate the proper doc for it
            re_search_arg = _re_parse_arg.search(obj_doc_lines[idx])
            if re_search_arg is not None:
                _, name, description = re_search_arg.groups()
                current_arg = name
                if name in signature:
                    default = signature[name].default
                    if signature[name].kind is inspect._ParameterKind.VAR_KEYWORD:
                        default = None
                    new_description = replace_default_in_arg_description(description, default)
                else:
                    new_description = description
                init_doc = _re_parse_arg.sub(rf"\1\2 ({new_description}):", obj_doc_lines[idx])
                arguments[current_arg] = [init_doc]
        elif current_arg is not None:
            arguments[current_arg].append(obj_doc_lines[idx])

        idx += 1

    # We went too far by one (perhaps more if there are a lot of new lines)
    idx -= 1
    if current_arg:
        while len(obj_doc_lines[idx].strip()) == 0:
            arguments[current_arg] = arguments[current_arg][:-1]
            idx -= 1
    # And we went too far by one again.
    idx += 1

    old_doc_arg = "\n".join(obj_doc_lines[start_idx:idx])

    old_arguments = list(arguments.keys())
    arguments = {name: "\n".join(doc) for name, doc in arguments.items()}
    # Add missing arguments with a template
    for name in set(signature.keys()) - set(arguments.keys()):
        arg = signature[name]
        # We ignore private arguments or *args/**kwargs (unless they are documented by the user)
        if name.startswith("_") or arg.kind in [
            inspect._ParameterKind.VAR_KEYWORD,
            inspect._ParameterKind.VAR_POSITIONAL,
        ]:
            arguments[name] = ""
        else:
            arg_desc = get_default_description(arg)
            arguments[name] = " " * (indent + 4) + f"{name} ({arg_desc}): <fill_docstring>"

    # Arguments are sorted by the order in the signature unless a special comment is put.
    if ignore_order:
        new_param_docs = [arguments[name] for name in old_arguments if name in signature]
        missing = set(signature.keys()) - set(old_arguments)
        new_param_docs.extend([arguments[name] for name in missing if len(arguments[name]) > 0])
    else:
        new_param_docs = [arguments[name] for name in signature.keys() if len(arguments[name]) > 0]
    new_doc_arg = "\n".join(new_param_docs)



SUMMARY:
**Function Summary:**
### match_docstring_with_signature(obj)
 
  **Purpose:** Match the docstring of an object with its signature.

  **Primary Functionality:** The function `match_docstring_with_signature` compares two documentation strings for an object's methods and arguments. It reads the source code (if available) to check for ignore commands, then parses the signatures using `inspect.signature`. It extracts argument names and descriptions from the docstring, generating new documentations based on the found signature parameters.

  **Key Implementation Details:**
- Uses regular expressions (`re`) for pattern matching.
- Leverages `find_indent` (function not shown in this snippet) to parse indentation levels.
- Utilizes `_pre_parse_arg` and `_parse_arg` regular expression patterns (not directly visible here).

  **Dependencies/Requirements:**
- Import of `inspect`, `re`, and access to `type Any`, `Optional`, `Tuple`, and `inspect._ParameterKind`.
 - The function utilizes type hints for clarity (`Any`, `Inspect.Signature`).
  
  **API Interface:** 
    It accepts an object, returns a tuple containing two strings:
     - The first string contains the original documentation of the arguments described in the docstring.
     - The second string contains the matched and generated documentation for the same argument parameters based on their signature.

**Additional Notes:**
The function seems heavily involved with parsing and manipulating various parts of Python source code, likely to maintain consistency across different codebases by following certain established coding standards.


key: chroma:document
string_value:
CODE:

def fix_docstring(obj: Any, old_doc_args: str, new_doc_args: str):
    """
    Fixes the docstring of an object by replacing its arguments documentaiton by the one matched with the signature.

    Args:
        obj (`Any`):
            The object whose dostring we are fixing.
        old_doc_args (`str`):
            The current documentation of the parameters of `obj` in the docstring (as returned by
            `match_docstring_with_signature`).
        new_doc_args (`str`):
            The documentation of the parameters of `obj` matched with its signature (as returned by
            `match_docstring_with_signature`).
    """
    # Read the docstring in the source code and make sure we have the right part of the docstring
    source, line_number = inspect.getsourcelines(obj)

    # Get to the line where we start documenting arguments
    idx = 0
    while idx < len(source) and _re_args.search(source[idx]) is None:
        idx += 1

    if idx == len(source):
        # Args are not defined in the docstring of this object
        return

    # Get to the line where we stop documenting arguments
    indent = find_indent(source[idx])
    idx += 1
    start_idx = idx
    while idx < len(source) and (len(source[idx].strip()) == 0 or find_indent(source[idx]) > indent):
        idx += 1

    idx -= 1
    while len(source[idx].strip()) == 0:
        idx -= 1
    idx += 1

    if "".join(source[start_idx:idx])[:-1] != old_doc_args:
        # Args are not fully defined in the docstring of this object
        return

    obj_file = find_source_file(obj)
    with open(obj_file, "r", encoding="utf-8") as f:
        content = f.read()

    # Replace content
    lines = content.split("\n")
    lines = lines[: line_number + start_idx - 1] + [new_doc_args] + lines[line_number + idx - 1 :]

    print(f"Fixing the docstring of {obj.__name__} in {obj_file}.")
    with open(obj_file, "w", encoding="utf-8") as f:


SUMMARY:
**Function Summary: `fix_docstring` Function**

### Purpose and Functionality

The `fix_docstring` function updates the docstrings of objects by replacing outdated parameter documentation with the latest one based on the object's signature. It ensures accuracy and consistency in documentations.

### Key Implementation Details

1.  **Inspecting Docstrings**:

    *   Reads source code from a given object.
    *   Identifies the lines where arguments are defined and replaced.
2.  **Matching Docstring with Signature**:

    *   Uses `match_docstring_with_signature` to determine the updated documentation based on the signature.

### Notable Dependencies

1  **External Dependency**:

    *   `re` for regular expression matching
2  **Inherited/Implemented Interfaces** (None mentioned in this function)

The code snippet fixes the docstrings of objects by replacing outdated parameter documentation with accurate ones. It uses regular expressions to identify lines where arguments are defined, reads source code from an object, and updates its content accordingly.


key: chroma:document
string_value:
CODE:

def check_docstrings(overwrite: bool = False, check_all: bool = False):
    """
    Check docstrings of all public objects that are callables and are documented. By default, only checks the diff.

    Args:
        overwrite (`bool`, *optional*, defaults to `False`):
            Whether to fix inconsistencies or not.
        check_all (`bool`, *optional*, defaults to `False`):
            Whether to check all files.
    """
    module_diff_files = None
    if not check_all:
        module_diff_files = set()
        repo = Repo(PATH_TO_REPO)
        # Diff from index to unstaged files
        for modified_file_diff in repo.index.diff(None):
            if modified_file_diff.a_path.startswith("src/transformers"):
                module_diff_files.add(modified_file_diff.a_path)
        # Diff from index to `main`
        for modified_file_diff in repo.index.diff(repo.refs.main.commit):
            if modified_file_diff.a_path.startswith("src/transformers"):
                module_diff_files.add(modified_file_diff.a_path)
        # quick escape route: if there are no module files in the diff, skip this check
        if len(module_diff_files) == 0:
            return
        print("    Checking docstrings in the following files:" + "\n    - " + "\n    - ".join(module_diff_files))

    failures = []
    hard_failures = []
    to_clean = []
    for name in dir(transformers):
        # Skip objects that are private or not documented.
        if name.startswith("_") or ignore_undocumented(name) or name in OBJECTS_TO_IGNORE:
            continue

        obj = getattr(transformers, name)
        if not callable(obj) or not isinstance(obj, type) or getattr(obj, "__doc__", None) is None:
            continue

        # If we are checking against the diff, we skip objects that are not part of the diff.
        if module_diff_files is not None:
            object_file = find_source_file(getattr(transformers, name))
            object_file_relative_path = "src/" + str(object_file).split("/src/")[1]
            if object_file_relative_path not in module_diff_files:
                continue

        # Check docstring
        try:
            result = match_docstring_with_signature(obj)
            if result is not None:
                old_doc, new_doc = result
            else:
                old_doc, new_doc = None, None
        except Exception as e:
            print(e)
            hard_failures.append(name)
            continue
        if old_doc != new_doc:
            if overwrite:
                fix_docstring(obj, old_doc, new_doc)
            else:
                failures.append(name)
        elif not overwrite and new_doc is not None and ("<fill_type>" in new_doc or "<fill_docstring>" in new_doc):
            to_clean.append(name)

    # Deal with errors
    error_message = ""
    if len(hard_failures) > 0:
        error_message += (
            "The argument part of the docstrings of the following objects could not be processed, check they are "
            "properly formatted."
        )
        error_message += "\n" + "\n".join([f"- {name}" for name in hard_failures])
    if len(failures) > 0:
        error_message += (
            "The following objects docstrings do not match their signature. Run `make fix-copies` to fix this. "
            "In some cases, this error may be raised incorrectly by the docstring checker. If you think this is the "
            "case, you can manually check the docstrings and then add the object name to `OBJECTS_TO_IGNORE` in "
            "`utils/check_docstrings.py`."
        )
        error_message += "\n" + "\n".join([f"- {name}" for name in failures])
    if len(to_clean) > 0:
        error_message += (
            "The following objects docstrings contain templates you need to fix: search for `<fill_type>` or "
            "`<fill_docstring>`."
        )
        error_message += "\n" + "\n".join([f"- {name}" for name in to_clean])

    if len(error_message) > 0:
        error_message = "There was at least one problem when checking docstrings of public objects.\n" + error_message


SUMMARY:
**Summary**

The `check_docstrings` function checks docstrings of public, callable objects in the `transformers` module and its dependencies. It:

1. **Analyze Documentation**: Checks if existing docs match defined signatures.

2. **Fix Potential Issues**: Fixes inconsistencies or warnings reported by the tool.

3. **Report Potential Bugs**: Prints error messages about unmatched signatures, incomplete methods, and potential bugs.

**Key Functionality**

*   `check_all`: Optional flag to check all files.
*   `overwrite`: Optional flag to fix inconsistencies instead of logging failures.

**Implementation Details**

*   Verifies functions match expected signature patterns.
*   Skips private objects or those without docstrings.
*   Raises `ValueError` with error messages when reports detection issues.


key: chroma:document
string_value:
CODE:

def download_and_extract(task, data_dir):
    print(f"Downloading and extracting {task}...")
    data_file = f"{task}.zip"
    urllib.request.urlretrieve(TASK2PATH[task], data_file)
    with zipfile.ZipFile(data_file) as zip_ref:
        zip_ref.extractall(data_dir)
    os.remove(data_file)


SUMMARY:
**Summary:**

`download_and_extract` function downloads and extracts a zip file for a given task, extracting the contents to a specified data directory.

**Implementation Details:**

* Downloads from `taskId`
* Extracts contents using `zipfile.ZipFile.extractall()`
* Deletes temp zip file after extraction (`os.remove(data_file)`)

**Notable Dependencies:**

* `urllib.request.urlretrieve()` for downloading
* `zipfile` module for extracting zip file
* `os` module for deleting temporary files

**Primary Class/Function Names:**

* `task`
* `data_dir`
* `TASK2PATH[task]`: mapping of tasks to download paths


key: chroma:document
string_value:
CODE:

def format_mrpc(data_dir, path_to_data):
    print("Processing MRPC...")
    mrpc_dir = os.path.join(data_dir, "MRPC")
    if not os.path.isdir(mrpc_dir):
        os.mkdir(mrpc_dir)
    if path_to_data:
        mrpc_train_file = os.path.join(path_to_data, "msr_paraphrase_train.txt")
        mrpc_test_file = os.path.join(path_to_data, "msr_paraphrase_test.txt")
    else:
        print("Local MRPC data not specified, downloading data from %s" % MRPC_TRAIN)
        mrpc_train_file = os.path.join(mrpc_dir, "msr_paraphrase_train.txt")
        mrpc_test_file = os.path.join(mrpc_dir, "msr_paraphrase_test.txt")
        urllib.request.urlretrieve(MRPC_TRAIN, mrpc_train_file)
        urllib.request.urlretrieve(MRPC_TEST, mrpc_test_file)
    if not os.path.isfile(mrpc_train_file):
        raise ValueError(f"Train data not found at {mrpc_train_file}")
    if not os.path.isfile(mrpc_test_file):
        raise ValueError(f"Test data not found at {mrpc_test_file}")
    urllib.request.urlretrieve(TASK2PATH["MRPC"], os.path.join(mrpc_dir, "dev_ids.tsv"))

    dev_ids = []
    with open(os.path.join(mrpc_dir, "dev_ids.tsv"), encoding="utf8") as ids_fh:
        for row in ids_fh:
            dev_ids.append(row.strip().split("\t"))

    with open(mrpc_train_file, encoding="utf8") as data_fh, open(
        os.path.join(mrpc_dir, "train.tsv"), "w", encoding="utf8"
    ) as train_fh, open(os.path.join(mrpc_dir, "dev.tsv"), "w", encoding="utf8") as dev_fh:
        header = data_fh.readline()
        train_fh.write(header)
        dev_fh.write(header)
        for row in data_fh:
            label, id1, id2, s1, s2 = row.strip().split("\t")
            if [id1, id2] in dev_ids:
                dev_fh.write("%s\t%s\t%s\t%s\t%s\n" % (label, id1, id2, s1, s2))
            else:
                train_fh.write("%s\t%s\t%s\t%s\t%s\n" % (label, id1, id2, s1, s2))

    with open(mrpc_test_file, encoding="utf8") as data_fh, open(
        os.path.join(mrpc_dir, "test.tsv"), "w", encoding="utf8"
    ) as test_fh:
        header = data_fh.readline()
        test_fh.write("index\t#1 ID\t#2 ID\t#1 String\t#2 String\n")
        for idx, row in enumerate(data_fh):
            label, id1, id2, s1, s2 = row.strip().split("\t")
            test_fh.write("%d\t%s\t%s\t%s\t%s\n" % (idx, id1, id2, s1, s2))


SUMMARY:
**Function Summary: `format_mrpc`**

**Main Purpose and Functionality**
The `format_mrpc` function processes the MRPC (Microsoft Research Paraphrase Corpus) dataset by:

* Creating a directory for the dataset if it does not exist
* Downloading dataset files from a specified URL or using local data if specified
* Validating the presence of required files
* Preprocessing data into training, development, and testing sets

**Key Implementation Details**

* Uses `os.path.join` to construct file paths
* Utilizes `urllib.request.urlretrieve` to download external datasets
* Reads and writes data from/to files using `open` function with specified encoding
* Splits input file rows into individual labels, IDs, and strings

**Notable Dependencies and Requirements**

* Uses `os` module for file system operations
* Utilizes `urllib.request` module for downloading external datasets
* References classes and functions by exact name:
	+ `MRPC_TRAIN`
	+ `TASK2PATH["MRPC"]`

**External Dependencies:**

* `urllib.request`
* `os`
* Python's built-in file I/O operations


key: chroma:document
string_value:
CODE:

def download_diagnostic(data_dir):
    print("Downloading and extracting diagnostic...")
    if not os.path.isdir(os.path.join(data_dir, "diagnostic")):
        os.mkdir(os.path.join(data_dir, "diagnostic"))
    data_file = os.path.join(data_dir, "diagnostic", "diagnostic.tsv")
    urllib.request.urlretrieve(TASK2PATH["diagnostic"], data_file)
    print("\tCompleted!")


SUMMARY:
**Summary:**

The `download_diagnostic` function downloads and extracts diagnostic data from a specified directory. It:

* Creates the 'diagnostic' subdirectory if it does not exist
* Downloads the diagnostic file using `urlretrieve` from the `TASK2PATH["diagnostic"]` location
* Prints confirmation messages after completion

**Key details:**

* Uses `os.path.join` to construct paths for directory and file creation/export
* Implements `urllib.request.urlretrieve` for file download
* Returns no value (indicating a side-effecting function)

**Notable dependencies/requirements:**

* `os.path`
* `urllib.request`
* `TASK2PATH` (dictionary containing path to diagnostic data)


key: chroma:document
string_value:
CODE:

def get_tasks(task_names):
    task_names = task_names.split(",")
    if "all" in task_names:
        tasks = TASKS
    else:
        tasks = []
        for task_name in task_names:
            if task_name not in TASKS:
                raise ValueError(f"Task {task_name} not found!")
            tasks.append(task_name)


SUMMARY:
**Function Summary: `get_tasks`**

The `get_tasks` function retrieves the specified task names from a predefined list of tasks (`TASKS`). If "all" is among the input task names, it returns all available tasks. Otherwise, it checks each task name against the `TASKS` list and appends valid task names to a result list, raising an error for invalid task names.

**Key Implementation Details:**

* Uses string splitting to parse input task names.
* Checks if "all" is present in the input task names.

**Notable Dependencies or Requirements:**

* The `TASKS` list of predefined task names (assuming it's defined elsewhere).
* None explicitly listed, but implicit dependencies on Python's built-in data structures and error handling mechanisms.


key: chroma:document
string_value:
CODE:

def main(arguments):
    parser = argparse.ArgumentParser()
    parser.add_argument("--data_dir", help="directory to save data to", type=str, default="glue_data")
    parser.add_argument(
        "--tasks", help="tasks to download data for as a comma separated string", type=str, default="all"
    )
    parser.add_argument(
        "--path_to_mrpc",
        help="path to directory containing extracted MRPC data, msr_paraphrase_train.txt and msr_paraphrase_text.txt",
        type=str,
        default="",
    )
    args = parser.parse_args(arguments)

    if not os.path.isdir(args.data_dir):
        os.mkdir(args.data_dir)
    tasks = get_tasks(args.tasks)

    for task in tasks:
        if task == "MRPC":
            format_mrpc(args.data_dir, args.path_to_mrpc)
        elif task == "diagnostic":
            download_diagnostic(args.data_dir)
        else:


SUMMARY:
Here is a concise summary of the provided code chunk:

**Main Functionality:**
The `main` function orchestrates data preparation for multiple natural language processing (NLP) tasks. It:

1. Parses command-line arguments for task specifics (`--data_dir`, `--tasks`, and `--path_to_mrpc`) to download and process data.
2. Creates a directory to store the downloaded data if it does not exist.
3. Determines the tasks to be processed based on the comma-separated string provided in `--tasks`.
4. For each task, performs specific actions:
   - MRPC task: Formats MRPC data using `format_mrpc`.
   - Diagnostic task: Downloads diagnostic data using `download_diagnostic`.
   - Other tasks: Downloads and extracts data using `download_and_extract`.

**Key Implementation Details:**

* Utilizes `argparse` for command-line argument parsing.
* Uses the `get_tasks` function to determine the specific tasks to be processed.
* Implements a conditional statement to perform task-specific actions based on the parsed arguments.

**Notable Dependencies or Requirements:**

* `os` module for directory manipulation (e.g., creating directories).
* `argparse` library for command-line argument parsing.
* External dependencies:
	+ `get_tasks` function (not shown in this snippet).
	+ `download_diagnostic`, `format_mrpc`, and `download_and_extract` functions (not shown in this snippet).


key: chroma:document
string_value:
CODE:

def camel_case_split(identifier: str) -> List[str]:
    """
    Split a camel-cased name into words.

    Args:
        identifier (`str`): The camel-cased name to parse.

    Returns:
        `List[str]`: The list of words in the identifier (as seprated by capital letters).

    Example:

    ```py
    >>> camel_case_split("CamelCasedClass")
    ["Camel", "Cased", "Class"]
    ```
    """
    # Regex thanks to https://stackoverflow.com/questions/29916065/how-to-do-camelcase-split-in-python
    matches = re.finditer(".+?(?:(?<=[a-z])(?=[A-Z])|(?<=[A-Z])(?=[A-Z][a-z])|$)", identifier)


SUMMARY:
** camel_case_split() Function Summary **


*   **Purpose and Functionality:** 
    Splits a camel-cased string into individual words based on capitalization.

        *   Type Hint: `identifier (str) -> List[str]`
        *   Returns: A list of words separated by capital letters.

*   **Implementation Details:** 
    - Uses regular expressions with look-behind and look-ahead assertions to match word boundaries.
    - Utilizes the `re.finditer` function to extract matches from the input string.


*   **Notable Dependencies or Requirements:**
    - `re` module for regex functionality
    - No external dependencies


        Example usage:

            ```py
>>> camel_case_split("CamelCasedClass")
['Camel', 'Cased', 'Class']
```


key: chroma:document
string_value:
CODE:

def get_frameworks_table() -> pd.DataFrame:
    """
    Generates a dataframe containing the supported auto classes for each model type, using the content of the auto
    modules.
    """
    # Dictionary model names to config.
    config_maping_names = transformers_module.models.auto.configuration_auto.CONFIG_MAPPING_NAMES
    model_prefix_to_model_type = {
        config.replace("Config", ""): model_type for model_type, config in config_maping_names.items()
    }

    # Dictionaries flagging if each model prefix has a backend in PT/TF/Flax.
    pt_models = collections.defaultdict(bool)
    tf_models = collections.defaultdict(bool)
    flax_models = collections.defaultdict(bool)

    # Let's lookup through all transformers object (once) and find if models are supported by a given backend.
    for attr_name in dir(transformers_module):
        lookup_dict = None
        if _re_tf_models.match(attr_name) is not None:
            lookup_dict = tf_models
            attr_name = _re_tf_models.match(attr_name).groups()[0]
        elif _re_flax_models.match(attr_name) is not None:
            lookup_dict = flax_models
            attr_name = _re_flax_models.match(attr_name).groups()[0]
        elif _re_pt_models.match(attr_name) is not None:
            lookup_dict = pt_models
            attr_name = _re_pt_models.match(attr_name).groups()[0]

        if lookup_dict is not None:
            while len(attr_name) > 0:
                if attr_name in model_prefix_to_model_type:
                    lookup_dict[model_prefix_to_model_type[attr_name]] = True
                    break
                # Try again after removing the last word in the name
                attr_name = "".join(camel_case_split(attr_name)[:-1])

    all_models = set(list(pt_models.keys()) + list(tf_models.keys()) + list(flax_models.keys()))
    all_models = list(all_models)
    all_models.sort()

    data = {"model_type": all_models}
    data["pytorch"] = [pt_models[t] for t in all_models]
    data["tensorflow"] = [tf_models[t] for t in all_models]
    data["flax"] = [flax_models[t] for t in all_models]

    # Now let's find the right processing class for each model. In order we check if there is a Processor, then a
    # Tokenizer, then a FeatureExtractor, then an ImageProcessor
    processors = {}
    for t in all_models:
        if t in transformers_module.models.auto.processing_auto.PROCESSOR_MAPPING_NAMES:
            processors[t] = "AutoProcessor"
        elif t in transformers_module.models.auto.tokenization_auto.TOKENIZER_MAPPING_NAMES:
            processors[t] = "AutoTokenizer"
        elif t in transformers_module.models.auto.image_processing_auto.IMAGE_PROCESSOR_MAPPING_NAMES:
            processors[t] = "AutoImageProcessor"
        elif t in transformers_module.models.auto.feature_extraction_auto.FEATURE_EXTRACTOR_MAPPING_NAMES:
            processors[t] = "AutoFeatureExtractor"
        else:
            # Default to AutoTokenizer if a model has nothing, for backward compatibility.
            processors[t] = "AutoTokenizer"

    data["processor"] = [processors[t] for t in all_models]



SUMMARY:
**Function Summary: `get_frameworks_table`**

**Primary Purpose:** Generates a pandas DataFrame containing the supported auto classes for each model type, using the content of the auto modules.

**Key Implementation Details:**

* Uses two dictionaries to map model prefixes to model types (`model_prefix_to_model_type`)
* Populates three separate dictionaries (`pt_models`, `tf_models`, and `flax_models`) to track backend support for each model prefix
* Iterates through all transformers object attributes using regular expressions to identify models supported by PT, TF, or Flax backends

**Notable Dependencies and Requirements:**

* `pd` (pandas library) for data manipulation
* `transformers_module` (external dependencies) containing configuration mapping names and processing class mappings
* `_re_tf_models`, `_re_flax_models`, and `_re_pt_models` (regex patterns used to identify models supported by PT, TF, or Flax backends)
* `collections` (utility module) for using defaultdicts


key: chroma:document
string_value:
CODE:

def update_pipeline_and_auto_class_table(table: Dict[str, Tuple[str, str]]) -> Dict[str, Tuple[str, str]]:
    """
    Update the table maping models to pipelines and auto classes without removing old keys if they don't exist anymore.

    Args:
        table (`Dict[str, Tuple[str, str]]`):
            The existing table mapping model names to a tuple containing the pipeline tag and the auto-class name with
            which they should be used.

    Returns:
        `Dict[str, Tuple[str, str]]`: The updated table in the same format.
    """
    auto_modules = [
        transformers_module.models.auto.modeling_auto,
        transformers_module.models.auto.modeling_tf_auto,
        transformers_module.models.auto.modeling_flax_auto,
    ]
    for pipeline_tag, model_mapping, auto_class in PIPELINE_TAGS_AND_AUTO_MODELS:
        model_mappings = [model_mapping, f"TF_{model_mapping}", f"FLAX_{model_mapping}"]
        auto_classes = [auto_class, f"TF_{auto_class}", f"Flax_{auto_class}"]
        # Loop through all three frameworks
        for module, cls, mapping in zip(auto_modules, auto_classes, model_mappings):
            # The type of pipeline may not exist in this framework
            if not hasattr(module, mapping):
                continue
            # First extract all model_names
            model_names = []
            for name in getattr(module, mapping).values():
                if isinstance(name, str):
                    model_names.append(name)
                else:
                    model_names.extend(list(name))

            # Add pipeline tag and auto model class for those models
            table.update({model_name: (pipeline_tag, cls) for model_name in model_names})



SUMMARY:
**Function Summary**

### `update_pipeline_and_auto_class_table`

Updates the table mapping models to pipelines and auto classes without removing old keys.

#### Parameters
- `table`: The existing table mapping model names to a tuple containing the pipeline tag and the auto-class name (Dict[str, Tuple[str, str]])

#### Returns
- `Dict[str, Tuple[str, str]]`: The updated table in the same format

#### Implementation Details

- Updates table with pipeline tags and auto-model classes for each framework (transformers_module.models.auto.modeling_auto, transformers_module.models.auto.modeling_tf_auto, transformers_module.models.auto.modeling_flax_auto)
- Handles cases where pipeline type may not exist in a given framework
- Uses model mappings and pipepline tags to populate the table

#### Notable Dependencies
- `PIPELINE_TAGS_AND_AUTO_MODELS`
- `transformers_module.models.auto.modeling_auto`, `transformers_module.models.auto.modeling_tf_auto`, `transformers_module.models.auto.modeling_flax_auto`

#### External Dependencies
- `transformers_module`

#### Notes

This function iterates over a list of frameworks, extracts model names, and updates the table with pipeline tags and auto-model classes.


key: chroma:document
string_value:
CODE:

def update_metadata(token: str, commit_sha: str):
    """
    Update the metadata for the Transformers repo in `huggingface/transformers-metadata`.

    Args:
        token (`str`): A valid token giving write access to `huggingface/transformers-metadata`.
        commit_sha (`str`): The commit SHA on Transformers corresponding to this update.
    """
    frameworks_table = get_frameworks_table()
    frameworks_dataset = Dataset.from_pandas(frameworks_table)

    resolved_tags_file = hf_hub_download(
        "huggingface/transformers-metadata", "pipeline_tags.json", repo_type="dataset", token=token
    )
    tags_dataset = Dataset.from_json(resolved_tags_file)
    table = {
        tags_dataset[i]["model_class"]: (tags_dataset[i]["pipeline_tag"], tags_dataset[i]["auto_class"])
        for i in range(len(tags_dataset))
    }
    table = update_pipeline_and_auto_class_table(table)

    # Sort the model classes to avoid some nondeterministic updates to create false update commits.
    model_classes = sorted(table.keys())
    tags_table = pd.DataFrame(
        {
            "model_class": model_classes,
            "pipeline_tag": [table[m][0] for m in model_classes],
            "auto_class": [table[m][1] for m in model_classes],
        }
    )
    tags_dataset = Dataset.from_pandas(tags_table)

    hub_frameworks_json = hf_hub_download(
        repo_id="huggingface/transformers-metadata",
        filename="frameworks.json",
        repo_type="dataset",
        token=token,
    )
    with open(hub_frameworks_json) as f:
        hub_frameworks_json = f.read()

    hub_pipeline_tags_json = hf_hub_download(
        repo_id="huggingface/transformers-metadata",
        filename="pipeline_tags.json",
        repo_type="dataset",
        token=token,
    )
    with open(hub_pipeline_tags_json) as f:
        hub_pipeline_tags_json = f.read()

    with tempfile.TemporaryDirectory() as tmp_dir:
        frameworks_dataset.to_json(os.path.join(tmp_dir, "frameworks.json"))
        tags_dataset.to_json(os.path.join(tmp_dir, "pipeline_tags.json"))

        with open(os.path.join(tmp_dir, "frameworks.json")) as f:
            frameworks_json = f.read()
        with open(os.path.join(tmp_dir, "pipeline_tags.json")) as f:
            pipeline_tags_json = f.read()

        frameworks_equal = hub_frameworks_json == frameworks_json
        hub_pipeline_tags_equal = hub_pipeline_tags_json == pipeline_tags_json

        if frameworks_equal and hub_pipeline_tags_equal:
            print("No updates on the Hub, not pushing the metadata files.")
            return

        if commit_sha is not None:
            commit_message = (
                f"Update with commit {commit_sha}\n\nSee: "
                f"https://github.com/huggingface/transformers/commit/{commit_sha}"
            )
        else:
            commit_message = "Update"

        upload_folder(
            repo_id="huggingface/transformers-metadata",
            folder_path=tmp_dir,
            repo_type="dataset",
            token=token,
            commit_message=commit_message,


SUMMARY:
**Function Summary:**

`update_metadata`

**Purpose:** Updates the metadata for the Transformers repository by processing and uploading dataset files.

**Key Implementation Details:**

* Downloads pipeline_tags.json and frameworks.json dataset files from huggingface/transformers-metadata
* Processes framework and tag data from downloaded files
* Checks if updates need to be pushed to GitHub and commits message is generated accordingly

**Noteable Dependencies/Requirements:**

* `get_frameworks_table` function (not shown in this code snippet)
* `Dataset.from_json` and `Dataset.from_pandas` functions (from the Hugging Face datasets library)
* `hf_hub_download` and `upload_folder` functions (from the Hugging Face Hub client library)

**Interfaces/ Method Signatures:**

* `update_pipeline_and_auto_class_table(table)`: Updates table with processed framework data
* `get_frameworks_table()`: Returns a pandas dataframe containing frameworks data

**External Dependencies:**

* `huggingface/transformers-metadata`
* `huggingfacedatasets` library
* `huggingface_hub` library


key: chroma:document
string_value:
CODE:

def check_pipeline_tags():
    """
    Check all pipeline tags are properly defined in the `PIPELINE_TAGS_AND_AUTO_MODELS` constant of this script.
    """
    in_table = {tag: cls for tag, _, cls in PIPELINE_TAGS_AND_AUTO_MODELS}
    pipeline_tasks = transformers_module.pipelines.SUPPORTED_TASKS
    missing = []
    for key in pipeline_tasks:
        if key not in in_table:
            model = pipeline_tasks[key]["pt"]
            if isinstance(model, (list, tuple)):
                model = model[0]
            model = model.__name__
            if model not in in_table.values():
                missing.append(key)

    if len(missing) > 0:
        msg = ", ".join(missing)
        raise ValueError(
            "The following pipeline tags are not present in the `PIPELINE_TAGS_AND_AUTO_MODELS` constant inside "
            f"`utils/update_metadata.py`: {msg}. Please add them!"


SUMMARY:
**Summary:**

This function, `check_pipeline_tags()`, detects and raises an error if any pipeline tasks are missing from the predefined `PIPELINE_TAGS_AND_AUTO_MODELS` constant.

**Key Details:**

- Verifies that all task pipelines match the predefined auto-models.
- Checks against `transformers_module.pipelines.SUPPORTED_TASKS`.
- Raises a `ValueError` with missing task pipelines as input when not found in the defined constants.

**Implementation Notes:**

* Utilizes dictionary comprehension to create an in-table mapping of pipeline tags and models.
* Iterates through supported tasks, checking their existence in the predefined constant map (`in_table`).
* Raises an error if the `missing` list is populated during iteration. 

This function ensures that all defined pipelines have corresponding auto-models.


key: docstring
string_value:
r"""
The argument `test_file` in this file refers to a model test file. This should be a string of the from
`tests/models/*/test_modeling_*.py`.
"""


key: chroma:document
string_value:
CODE:

def get_module_path(test_file):
    """Return the module path of a model test file."""
    components = test_file.split(os.path.sep)
    if components[0:2] != ["tests", "models"]:
        raise ValueError(
            "`test_file` should start with `tests/models/` (with `/` being the OS specific path separator). Got "
            f"{test_file} instead."
        )
    test_fn = components[-1]
    if not test_fn.endswith("py"):
        raise ValueError(f"`test_file` should be a python file. Got {test_fn} instead.")
    if not test_fn.startswith("test_modeling_"):
        raise ValueError(
            f"`test_file` should point to a file name of the form `test_modeling_*.py`. Got {test_fn} instead."
        )

    components = components[:-1] + [test_fn.replace(".py", "")]
    test_module_path = ".".join(components)



SUMMARY:
**Function Summary**

The `get_module_path` function takes a `test_file` path as input and returns the corresponding module path. It validates that the input path matches the expected format `tests/modeling/*.py`, raising errors for any deviations.

**Key Implementation Details**

- Splits the input `test_file` into components using the OS-specific path separator.
- Verifies that the first two components are "tests" and "models," and the last component ends with ".py."
- Reconstructs the module path from the validated components.

**Notable Dependencies and Requirements**

- Uses the `os` module for handling file paths.
- References `python files` directly in validation checks.


key: chroma:document
string_value:
CODE:

def get_test_module(test_file):
    """Get the module of a model test file."""
    test_module_path = get_module_path(test_file)
    try:
        test_module = importlib.import_module(test_module_path)
    except AttributeError as exc:
        # e.g. if you have a `tests` folder in `site-packages`, created by another package, when trying to import
        # `tests.models...`
        raise ValueError(
            f"Could not import module {test_module_path}. Confirm that you don't have a package with the same root "
            "name installed or in your environment's `site-packages`."
        ) from exc



SUMMARY:
**Summary:**

This function, `get_test_module`, retrieves and loads the Python module corresponding to a given test file by its path. It identifies if the import is successful or not and provides an informative error message if an AttributeError occurs.

**Key Implementation Details:**

* Imports a module using `importlib.import_module` with a specified path.
* Raises a ValueError with an informative message if the import fails due to a common issue.

**Notable Dependencies:**

* `importlib`
* None

**Interfaces/Classes:**

* No external interface or class is used in this function.


key: chroma:document
string_value:
CODE:

def get_tester_classes(test_file):
    """Get all classes in a model test file whose names ends with `ModelTester`."""
    tester_classes = []
    test_module = get_test_module(test_file)
    for attr in dir(test_module):
        if attr.endswith("ModelTester"):
            tester_classes.append(getattr(test_module, attr))

    # sort with class names


SUMMARY:
**Code Summary**

### Purpose and Functionality

Get all classes in a model test file whose names end with "ModelTester".

### Implementation Details

* Uses the `dir()` function to list attributes of the `test_module` object.
* Filters attributes based on name ending with "ModelTester".
* Retrieves attribute objects using `getattr()`.
* Sorts retrieved classes by their names using the `sorted()` function and a lambda function.

### Notable Dependencies or Requirements

* `get_test_module`: Returns the test module for the given test file (not shown in this code snippet).
 
This summary preserves detailed information from the original code while minimizing unnecessary details.


key: chroma:document
string_value:
CODE:

def get_test_classes(test_file):
    """Get all [test] classes in a model test file with attribute `all_model_classes` that are non-empty.

    These are usually the (model) test classes containing the (non-slow) tests to run and are subclasses of one of the
    classes `ModelTesterMixin`, `TFModelTesterMixin` or `FlaxModelTesterMixin`, as well as a subclass of
    `unittest.TestCase`. Exceptions include `RagTestMixin` (and its subclasses).
    """
    test_classes = []
    test_module = get_test_module(test_file)
    for attr in dir(test_module):
        attr_value = getattr(test_module, attr)
        # (TF/Flax)ModelTesterMixin is also an attribute in specific model test module. Let's exclude them by checking
        # `all_model_classes` is not empty (which also excludes other special classes).
        model_classes = getattr(attr_value, "all_model_classes", [])
        if len(model_classes) > 0:
            test_classes.append(attr_value)

    # sort with class names


SUMMARY:
Here is a concise summary of the code chunk:

**Function Purpose**
The `get_test_classes` function retrieves and sorts a list of non-empty test classes in a model test file that inherit from specific mixins (`ModelTesterMixin`, `TFModelTesterMixin`, or `FlaxModelTesterMixin`) or `unittest.TestCase`.

**Key Implementation Details**

* Iterates through the module's attributes to find classes with an empty `all_model_classes` attribute.
* Excludes exceptions like `RagTestMixin` and its subclasses.
* Sorts the test classes by their names using `__name__` attribute.

**Notable Dependencies and Requirements**

* The `get_test_module` function is called from within this function, suggesting a dependency on another module (`test_file`).
* This function relies on `unittest.TestCase`, indicating a requirement for testing framework.


key: chroma:document
string_value:
CODE:

def get_model_classes(test_file):
    """Get all model classes that appear in `all_model_classes` attributes in a model test file."""
    test_classes = get_test_classes(test_file)
    model_classes = set()
    for test_class in test_classes:
        model_classes.update(test_class.all_model_classes)

    # sort with class names


SUMMARY:
**Function Summary**

`get_model_classes`

Returns a sorted list of unique model classes from `all_model_classes` attributes in a model test file.

**Key Implementation Details**

* Uses `set` to eliminate duplicate class instances
* Iterates over `test_classes`, updating `model_classes` with `all_model_classes`
* Sorts classes by name using a case-insensitive comparison

**Notable Dependencies and Requirements**

* `get_test_classes` function is called with `test_file` as an argument
* Uses `__name__` attribute of class names for sorting comparison
* No external dependencies are listed due to lack of information


key: chroma:document
string_value:
CODE:

def get_model_tester_from_test_class(test_class):
    """Get the model tester class of a model test class."""
    test = test_class()
    if hasattr(test, "setUp"):
        test.setUp()

    model_tester = None
    if hasattr(test, "model_tester"):
        # `(TF/Flax)ModelTesterMixin` has this attribute default to `None`. Let's skip this case.
        if test.model_tester is not None:
            model_tester = test.model_tester.__class__



SUMMARY:
**Summary**

The `get_model_tester_from_test_class` function takes a `test_class` as input and returns the model tester class associated with it. It checks for the presence of an instance with an explicit `setUp` method, then attempts to retrieve the model tester class from the instance's attributes.

**Key Implementation Details**

* Tries to call `setUp` on the test instance if present
* Ignores cases where `model_tester` is set to `None`, likely due to default values in `ModelTesterMixin`

**Notable Dependencies/Requirements**

* `test_class`: The input class being analyzed
* `TF/Flax` (implied): The framework used for the `ModelTesterMixin`
* No external dependencies noted

**Notes**: This code appears to be part of a testing harness, likely for machine learning models.


key: chroma:document
string_value:
CODE:

def get_test_classes_for_model(test_file, model_class):
    """Get all [test] classes in `test_file` that have `model_class` in their `all_model_classes`."""
    test_classes = get_test_classes(test_file)

    target_test_classes = []
    for test_class in test_classes:
        if model_class in test_class.all_model_classes:
            target_test_classes.append(test_class)

    # sort with class names


SUMMARY:
**Function Summary**

`get_test_classes_for_model`: Returns a list of test classes from `test_file` that include `model_class`. 

**Key Implementation Details**

* Uses `sorted` function for alphabetical ordering by class name (`x.__name__`)
* Iterates through all test classes using `get_test_classes(test_file)`
* Filters test classes based on presence of `model_class` in `all_model_classes`

**Input/Output**

* `test_file`: input file containing test classes
* `model_class`: model class to search for in test classes
* Returns a sorted list of matching test classes

**Dependencies/Requirements**

* `get_test_classes` function from same library or module


key: chroma:document
string_value:
CODE:

def get_tester_classes_for_model(test_file, model_class):
    """Get all model tester classes in `test_file` that are associated to `model_class`."""
    test_classes = get_test_classes_for_model(test_file, model_class)

    tester_classes = []
    for test_class in test_classes:
        tester_class = get_model_tester_from_test_class(test_class)
        if tester_class is not None:
            tester_classes.append(tester_class)

    # sort with class names


SUMMARY:
**Summary**

Returns a list of model tester classes from a test file (`test_file`) associated with the specified `model_class`, sorted alphabetically by class name.

**Key Details**

* **Functionality**: Retrieves and filters model tester classes based on the provided `test_file` and `model_class`.
* **Algorithm/Design Choice**: Sorts filtered tester classes using the `__name__` attribute.
* **Method Signatures**:
	+ `get_test_classes_for_model()`: returns list of test classes for a given model
	+ `get_model_tester_from_test_class(test_class)`: returns associated model tester class from a test class

**Dependencies**

* `get_test_classes_for_model`
* `get_model_tester_from_test_class`


key: chroma:document
string_value:
CODE:

def get_test_to_tester_mapping(test_file):
    """Get a mapping from [test] classes to model tester classes in `test_file`.

    This uses `get_test_classes` which may return classes that are NOT subclasses of `unittest.TestCase`.
    """
    test_classes = get_test_classes(test_file)
    test_tester_mapping = {test_class: get_model_tester_from_test_class(test_class) for test_class in test_classes}


SUMMARY:
**Summary**

`get_test_to_tester_mapping` function:
 
* Maps test classes to model tester classes based on inheritance from `unittest.TestCase`.
* Uses `get_test_classes` to retrieve test class list and `get_model_tester_from_test_class` to map each test class.

**Key Details:**

* Reusable in a unit testing framework.
* May return classes that are not subclasses of `unittest.TestCase`.

**Implementation Notes:**

* Uses dictionary comprehension for efficient creation of mapping.
* Relies on `get_test_classes` and `get_model_tester_from_test_class` external functions.


key: chroma:document
string_value:
CODE:

def get_model_to_test_mapping(test_file):
    """Get a mapping from model classes to test classes in `test_file`."""
    model_classes = get_model_classes(test_file)
    model_test_mapping = {
        model_class: get_test_classes_for_model(test_file, model_class) for model_class in model_classes
    }


SUMMARY:
**Summary:**

`get_model_to_test_mapping` function generates a dictionary mapping `model_classes` to their corresponding test classes (`test_classes`) based on the input `test_file`. It uses `get_model_classes` and `get_test_classes_for_model` functions.

**Key Details:**

- Uses `get_model_classes(test_file)` to retrieve model class names
- Populates a dictionary (`model_test_mapping`) with models as keys and test classes as values using map comprehension

**Notable Dependencies/Requirements:**

- `get_model_classes(test_file)`
- `get_test_classes_for_model(test_file, model_class)`

It can be summarized as:
```python
def get_model_to_test_mapping(test_file):
    # Maps model classes to their corresponding test classes
    model_classes = get_model_classes(test_file)
    model_test_mapping = {model_class: get_test_classes_for_model(test_file, model_class) for model_class in model_classes}
    return model_test_mapping
```


key: chroma:document
string_value:
CODE:

def get_model_to_tester_mapping(test_file):
    """Get a mapping from model classes to model tester classes in `test_file`."""
    model_classes = get_model_classes(test_file)
    model_to_tester_mapping = {
        model_class: get_tester_classes_for_model(test_file, model_class) for model_class in model_classes
    }


SUMMARY:
**Summary**

`get_model_to_tester_mapping` function maps model classes to their corresponding tester classes in a given test file.

**Key Details:**

* Uses `model_classes` from `get_model_classes` for iteration
* Iterates using dictionary comprehension to create mappings of model classes to tester classes
* Returns a dictionary `model_to_tester_mapping`

**Notable Dependencies and Interfaces:**

* References `get_model_classes` and `get_tester_classes_for_model` functions (not shown)


key: chroma:document
string_value:
CODE:

def to_json(o):
    """Make the information succinct and easy to read.

    Avoid the full class representation like `<class 'transformers.models.bert.modeling_bert.BertForMaskedLM'>` when
    displaying the results. Instead, we use class name (`BertForMaskedLM`) for the readability.
    """
    if isinstance(o, str):
        return o
    elif isinstance(o, type):
        return o.__name__
    elif isinstance(o, (list, tuple)):
        return [to_json(x) for x in o]
    elif isinstance(o, dict):
        return {to_json(k): to_json(v) for k, v in o.items()}
    else:


SUMMARY:
**Concise Summary**

### Function Purpose and Main Functionality
The `to_json` function converts an object `o` into a human-readable JSON format. It handles various data types by recursively processing nested structures, truncating class representations for better readability.

### Key Implementation Details

* Handles string, type alias, list, tuple, and dictionary conversions using recursive calls.
* Uses `__name__` attribute to display only class names instead of full class representations.

### Notable Dependencies or Requirements
None mentioned, but it assumes objects are usable in a context that supports nested data processing (e.g., lists, dictionaries).

### Key Method Signatures and Interfaces

* Implicit: No explicit interfaces defined.
* Explicit: None.


key: chroma:document
string_value:
CODE:

def get_checkpoint_from_config_class(config_class):
    checkpoint = None

    # source code of `config_class`
    config_source = inspect.getsource(config_class)
    checkpoints = _re_checkpoint.findall(config_source)

    # Each `checkpoint` is a tuple of a checkpoint name and a checkpoint link.
    # For example, `('google-bert/bert-base-uncased', 'https://huggingface.co/google-bert/bert-base-uncased')`
    for ckpt_name, ckpt_link in checkpoints:
        # allow the link to end with `/`
        if ckpt_link.endswith("/"):
            ckpt_link = ckpt_link[:-1]

        # verify the checkpoint name corresponds to the checkpoint link
        ckpt_link_from_name = f"https://huggingface.co/{ckpt_name}"
        if ckpt_link == ckpt_link_from_name:
            checkpoint = ckpt_name
            break



SUMMARY:
**Summary:**

`get_checkpoint_from_config_class` function retrieves the corresponding checkpoint name from a provided config class's source code.

*   **Purpose:** To extract the checkpoint name from a config class based on its implementation details.
*   **Key Details:**
    *   Uses regular expression to find checkpoints in the config class source code.
    *   Verifies each checkpoint by comparing its name with the corresponding link.
*   **Notable Dependencies/Requirements:**
    *   `_re_checkpoint` regular expression pattern
    *   `inspect.getsource(config_class)` function
*   **Interfaces/Methods:** None (uses built-in Python functions and modules)
*   **External Dependencies:**

    *   `_re_checkpoint`
    *   `config_class` (input parameter)

**Summary Focus Criteria Met:**

1. Precise technical terms used: "config class", "checkpoint name", "regular expression pattern"
2. Class/function/variable names preserved exactly.
3. Primary purpose stated: "To extract the checkpoint name from a config class based on its implementation details."
4. Key implementation details narrowed down to:
    *   Regular expression usage
    *   Comparison verification for checkpoints
5. Notable dependencies listed with exact class names and functions: `_re_checkpoint`, `config_class`.
6. No external modules or libraries required besides built-in Python functions.


key: chroma:document
string_value:
CODE:

def check_config_docstrings_have_checkpoints():
    configs_without_checkpoint = []

    for config_class in list(CONFIG_MAPPING.values()):
        # Skip deprecated models
        if "models.deprecated" in config_class.__module__:
            continue
        checkpoint = get_checkpoint_from_config_class(config_class)

        name = config_class.__name__
        if checkpoint is None and name not in CONFIG_CLASSES_TO_IGNORE_FOR_DOCSTRING_CHECKPOINT_CHECK:
            configs_without_checkpoint.append(name)

    if len(configs_without_checkpoint) > 0:
        message = "\n".join(sorted(configs_without_checkpoint))
        raise ValueError(
            f"The following configurations don't contain any valid checkpoint:\n{message}\n\n"
            "The requirement is to include a link pointing to one of the models of this architecture in the "
            "docstring of the config classes listed above. The link should have be a markdown format like "
            "[myorg/mymodel](https://huggingface.co/myorg/mymodel)."


SUMMARY:
**Function Summary**

`check_config_docstrings_have_checkpoints()`: Verifies if all configuration classes in `CONFIG_MAPPING` have valid checkpoints in their docstrings. Raises a ValueError if any configurations lack checkpoint information.

**Key Implementation Details**

* Skips deprecated models by checking for "models.deprecated" in the module name.
* Iterates over config classes in `CONFIG_MAPPING`, appending names of missing checkpoint info to `configs_without_checkpoint` list.
* If no checkpoints found, raises a ValueError with an error message and sorted list of configuration class names.

**Notable Dependencies and Requirements**

* `CONFIG_MAPPING`: mapping of configuration classes
* `CONFIG_CLASSES_TO_IGNORE_FOR_DOCSTRING_CHECKPOINT_CHECK`: list of ignored configuration class names for docstring checkpoint check
* `get_checkpoint_from_config_class(config_class)`: function to retrieve checkpoint from a config class
* `sorted()` function for sorting configuration class names.


key: chroma:document
string_value:
CODE:

def handle_test_results(test_results):
    expressions = test_results.split(" ")

    failed = 0
    success = 0

    # When the output is short enough, the output is surrounded by = signs: "== OUTPUT =="
    # When it is too long, those signs are not present.
    time_spent = expressions[-2] if "=" in expressions[-1] else expressions[-1]

    for i, expression in enumerate(expressions):
        if "failed" in expression:
            failed += int(expressions[i - 1])
        if "passed" in expression:
            success += int(expressions[i - 1])



SUMMARY:
**Summary**

The `handle_test_results` function parses test results to extract failure and success counts, as well as the timestamp. Key aspects include:

*   **Purpose**: Parse test output to calculate failure and success metrics.
*   **Implementation Details**:
    *   Splits test output into expressions using spaces as separators.
    *   Calculates `time_spent` by detecting presence of `== OUTPUT ==` signs in the last expression.
    *   Iterates through expressions, incrementing `failed` and `success` counts based on specific checks (e.g., "failed" or "passed").
*   **Notable Dependencies/Requirements**:
    *   None

**Detailed Information**

*   Method Signatures: Not applicable
*   Interfaces Implemented: None
*   External Dependencies: None


key: chroma:document
string_value:
CODE:

def extract_first_line_failure(failures_short_lines):
    failures = {}
    file = None
    in_error = False
    for line in failures_short_lines.split("\n"):
        if re.search(r"_ \[doctest\]", line):
            in_error = True
            file = line.split(" ")[2]
        elif in_error and not line.split(" ")[0].isdigit():
            failures[file] = line
            in_error = False



SUMMARY:
**Summary:**

This function, `extract_first_line_failure`, processes text data from "short lines" of failures. It categorizes them into file-related failures by identifying the `_ [doctest]` prefix and extracts specific information about the error.

**Key Details:**

- **Algorithm:** Uses regular expressions to scan short lines for a pattern (_ [doctest]).
- **Method Signatures/Interfaces:** Takes `failures_short_lines` as input and returns a dictionary of failures (`failures`) with file names.
- **Dependencies:**
  - Utilizes external `re` (regular expression) module.
  - No imported classes/functions besides built-in Python.

**Context:**
This function appears to be part of a code analysis or testing framework, likely used for data extraction from test output logs.


key: chroma:document
string_value:
CODE:

class Message:
    def __init__(self, title: str, doc_test_results: Dict):
        self.title = title

        self.n_success = sum(job_result["n_success"] for job_result in doc_test_results.values())
        self.n_failures = sum(job_result["n_failures"] for job_result in doc_test_results.values())
        self.n_tests = self.n_success + self.n_failures

        # Failures and success of the modeling tests
        self.doc_test_results = doc_test_results

    @property
    def time(self) -> str:
        all_results = [*self.doc_test_results.values()]
        time_spent = [r["time_spent"].split(", ")[0] for r in all_results if len(r["time_spent"])]
        total_secs = 0

        for time in time_spent:
            time_parts = time.split(":")

            # Time can be formatted as xx:xx:xx, as .xx, or as x.xx if the time spent was less than a minute.
            if len(time_parts) == 1:
                time_parts = [0, 0, time_parts[0]]

            hours, minutes, seconds = int(time_parts[0]), int(time_parts[1]), float(time_parts[2])
            total_secs += hours * 3600 + minutes * 60 + seconds

        hours, minutes, seconds = total_secs // 3600, (total_secs % 3600) // 60, total_secs % 60
        return f"{int(hours)}h{int(minutes)}m{int(seconds)}s"

    @property
    def header(self) -> Dict:
        return {"type": "header", "text": {"type": "plain_text", "text": self.title}}

    @property
    def no_failures(self) -> Dict:
        return {
            "type": "section",
            "text": {
                "type": "plain_text",
                "text": f"🌞 There were no failures: all {self.n_tests} tests passed. The suite ran in {self.time}.",
                "emoji": True,
            },
            "accessory": {
                "type": "button",
                "text": {"type": "plain_text", "text": "Check Action results", "emoji": True},
                "url": f"https://github.com/huggingface/transformers/actions/runs/{os.environ['GITHUB_RUN_ID']}",
            },
        }

    @property
    def failures(self) -> Dict:
        return {
            "type": "section",
            "text": {
                "type": "plain_text",
                "text": (
                    f"There were {self.n_failures} failures, out of {self.n_tests} tests.\nThe suite ran in"
                    f" {self.time}."
                ),
                "emoji": True,
            },
            "accessory": {
                "type": "button",
                "text": {"type": "plain_text", "text": "Check Action results", "emoji": True},
                "url": f"https://github.com/huggingface/transformers/actions/runs/{os.environ['GITHUB_RUN_ID']}",
            },
        }

    @property
    def category_failures(self) -> List[Dict]:
        failure_blocks = []

        MAX_ERROR_TEXT = 3000 - len("The following examples had failures:\n\n\n\n") - len("[Truncated]\n")
        line_length = 40
        category_failures = {k: v["failed"] for k, v in doc_test_results.items() if isinstance(v, dict)}

        def single_category_failures(category, failures):
            text = ""
            if len(failures) == 0:
                return ""
            text += f"*{category} failures*:".ljust(line_length // 2).rjust(line_length // 2) + "\n"

            for idx, failure in enumerate(failures):
                new_text = text + f"`{failure}`\n"
                if len(new_text) > MAX_ERROR_TEXT:
                    text = text + "[Truncated]\n"
                    break
                text = new_text

            return text

        for category, failures in category_failures.items():
            report = single_category_failures(category, failures)
            if len(report) == 0:
                continue
            block = {
                "type": "section",
                "text": {
                    "type": "mrkdwn",
                    "text": f"The following examples had failures:\n\n\n{report}\n",
                },
            }
            failure_blocks.append(block)

        return failure_blocks

    @property
    def payload(self) -> str:
        blocks = [self.header]

        if self.n_failures > 0:
            blocks.append(self.failures)

        if self.n_failures > 0:
            blocks.extend(self.category_failures)

        if self.n_failures == 0:
            blocks.append(self.no_failures)

        return json.dumps(blocks)

    @staticmethod
    def error_out():
        payload = [
            {
                "type": "section",
                "text": {
                    "type": "plain_text",
                    "text": "There was an issue running the tests.",
                },
                "accessory": {
                    "type": "button",
                    "text": {"type": "plain_text", "text": "Check Action results", "emoji": True},
                    "url": f"https://github.com/huggingface/transformers/actions/runs/{os.environ['GITHUB_RUN_ID']}",
                },
            }
        ]

        print("Sending the following payload")
        print(json.dumps({"blocks": json.loads(payload)}))

        client.chat_postMessage(
            channel=SLACK_REPORT_CHANNEL_ID,
            text="There was an issue running the tests.",
            blocks=payload,
        )

    def post(self):
        print("Sending the following payload")
        print(json.dumps({"blocks": json.loads(self.payload)}))

        text = f"{self.n_failures} failures out of {self.n_tests} tests," if self.n_failures else "All tests passed."

        self.thread_ts = client.chat_postMessage(
            channel=SLACK_REPORT_CHANNEL_ID,
            blocks=self.payload,
            text=text,
        )

    def get_reply_blocks(self, job_name, job_link, failures, text):
        # `text` must be less than 3001 characters in Slack SDK
        # keep some room for adding "[Truncated]" when necessary
        MAX_ERROR_TEXT = 3000 - len("[Truncated]")

        failure_text = ""
        for key, value in failures.items():
            new_text = failure_text + f"*{key}*\n_{value}_\n\n"
            if len(new_text) > MAX_ERROR_TEXT:
                # `failure_text` here has length <= 3000
                failure_text = failure_text + "[Truncated]"
                break
            # `failure_text` here has length <= MAX_ERROR_TEXT
            failure_text = new_text

        title = job_name
        content = {"type": "section", "text": {"type": "mrkdwn", "text": text}}

        if job_link is not None:
            content["accessory"] = {
                "type": "button",
                "text": {"type": "plain_text", "text": "GitHub Action job", "emoji": True},
                "url": job_link,
            }

        return [
            {"type": "header", "text": {"type": "plain_text", "text": title, "emoji": True}},
            content,
            {"type": "section", "text": {"type": "mrkdwn", "text": failure_text}},
        ]

    def post_reply(self):
        if self.thread_ts is None:
            raise ValueError("Can only post reply if a post has been made.")

        sorted_dict = sorted(self.doc_test_results.items(), key=lambda t: t[0])
        for job_name, job_result in sorted_dict:
            if len(job_result["failures"]) > 0:
                text = f"*Num failures* :{len(job_result['failed'])} \n"
                failures = job_result["failures"]
                blocks = self.get_reply_blocks(job_name, job_result["job_link"], failures, text=text)

                print("Sending the following reply")
                print(json.dumps({"blocks": blocks}))

                client.chat_postMessage(
                    channel=SLACK_REPORT_CHANNEL_ID,
                    text=f"Results for {job_name}",
                    blocks=blocks,
                    thread_ts=self.thread_ts["ts"],
                )



SUMMARY:
**Class Summary:**

The `Message` class is designed to represent test results and report them in a Slack channel. It has the following primary purposes:

*   Calculate and store metadata about the test results (e.g., number of successes/failures, time spent on tests)
*   Generate reports with various information types (header, failures by category, payload)

**Notable Properties and Methods:**

*   `n_failures`, `n_tests`: Number of test failures and total tests
*   `payload`: A Slack-compatible report payload in JSON format

Key methods:

*   `post()`: Posts the test result report to the specified Slack channel.
*   `error_out()`: Sends a notification with an error message to the same Slack channel.
*   `get_reply_blocks()`: Generates reply blocks for individual test failures, including links to GitHub Action jobs.


key: chroma:document
string_value:
CODE:

def retrieve_artifact(name: str):
    _artifact = {}

    if os.path.exists(name):
        files = os.listdir(name)
        for file in files:
            try:
                with open(os.path.join(name, file), encoding="utf-8") as f:
                    _artifact[file.split(".")[0]] = f.read()
            except UnicodeDecodeError as e:
                raise ValueError(f"Could not open {os.path.join(name, file)}.") from e



SUMMARY:
**Summary**

`retrieve_artifact` function retrieves files from a specified directory and returns an artifact dictionary containing file names as keys and their corresponding contents.

**Main Purpose and Functionality**
The primary purpose of this function is to read the contents files from a given directory named `name`. The function iterates over each file, attempts to decode its contents, and stores them in a dictionary.

```python
def retrieve_artifact(name: str) -> dict:
    # ... (function implementation remains the same)
```

**Key Implementation Details**

* Uses `os.path.exists` to check if the specified directory path exists.
* Lists files using `os.listdir`.
* Tries to open and read each file with UTF-8 encoding, raising an error for decoding failures.

```python
files = os.listdir(name)
for file in files:
    try:
        with open(os.path.join(name, file), encoding="utf-8") as f:
            _artifact[file.split(".")[0]] = f.read()
```

**Notable Dependencies and Requirements**

* `os` module for path manipulation.
* External dependency: `file extensions (e.g., `.txt`, `.pdf`).
* No apparent inheritance or interface implementation.

```python
import os
```

**RAG System Representation**

| Key | Value |
| --- | --- |
| Purpose | Retrieve files from a specified directory and return an artifact dictionary. |
| Functionality | Iterates over each file, attempts to decode its contents, and stores them in a dictionary. |
| Implementation Details | Uses `os.path.exists`, `os.listdir`, and UTF-8 encoding for file reading. |
| Dependencies | `os` module. |


key: chroma:document
string_value:
CODE:

def retrieve_available_artifacts():
    class Artifact:
        def __init__(self, name: str):
            self.name = name
            self.paths = []

        def __str__(self):
            return self.name

        def add_path(self, path: str):
            self.paths.append({"name": self.name, "path": path})

    _available_artifacts: Dict[str, Artifact] = {}

    directories = filter(os.path.isdir, os.listdir())
    for directory in directories:
        artifact_name = directory
        if artifact_name not in _available_artifacts:
            _available_artifacts[artifact_name] = Artifact(artifact_name)

            _available_artifacts[artifact_name].add_path(directory)



SUMMARY:
**Concise Summary:**

This function, `retrieve_available_artifacts`, creates and populates a dictionary of artifacts stored in directories on the system. It utilizes classes to encapsulate artifact details and behavior.

**Key Implementation Details:**

* Creates an `Artifact` class with `name` attribute, which stores directory names as names and their corresponding paths.
* Utilizes `filter()` and `os.path.isdir()` to retrieve only directories from the system.
* Popsulates `_available_artifacts` dictionary with the extracted directory information.

**Notable Dependencies:**

* `os`: for interacting with the operating system (e.g., `os.path.isdir()`)
* `filter()`: a built-in Python function for filtering elements from iterable collections based on condition.

**Primary Purpose and Functionality:**
The main purpose of this code is to retrieve available artifacts stored in directories, which can be utilized for subsequent operations or processing. 

**Additional Informed Insights:**
- The code iterates over all files and folders in the system using `os.listdir()`.
- It populates an internal data structure `_available_artifacts` dictionary, containing only directories as keys.
- It assumes that the directory names match the artifact names within these instances, allowing for consistent categorization.


key: chroma:document
string_value:
CODE:

def clean_model_doc_toc(model_doc: List[dict]) -> List[dict]:
    """
    Cleans a section of the table of content of the model documentation (one specific modality) by removing duplicates
    and sorting models alphabetically.

    Args:
        model_doc (`List[dict]`):
            The list of dictionaries extracted from the `_toctree.yml` file for this specific modality.

    Returns:
        `List[dict]`: List of dictionaries like the input, but cleaned up and sorted.
    """
    counts = defaultdict(int)
    for doc in model_doc:
        counts[doc["local"]] += 1
    duplicates = [key for key, value in counts.items() if value > 1]

    new_doc = []
    for duplicate_key in duplicates:
        titles = list({doc["title"] for doc in model_doc if doc["local"] == duplicate_key})
        if len(titles) > 1:
            raise ValueError(
                f"{duplicate_key} is present several times in the documentation table of content at "
                "`docs/source/en/_toctree.yml` with different *Title* values. Choose one of those and remove the "
                "others."
            )
        # Only add this once
        new_doc.append({"local": duplicate_key, "title": titles[0]})

    # Add none duplicate-keys
    new_doc.extend([doc for doc in model_doc if counts[doc["local"]] == 1])

    # Sort


SUMMARY:
**Concise Summary**

The `clean_model_doc_toc` function cleans the table of content in model documentation by removing duplicates and sorting models alphabetically.

**Main Purpose and Functionality**

* Cleans a list of dictionaries (`model_doc`) extracted from `_toctree.yml` by:
	+ Counting occurrences of each local key
	+ Identifying duplicate keys with multiple titles
	+ Removing all but one version of each duplicate title
* Returns the cleaned list sorted alphabetically by title.

**Key Implementation Details**

* Uses `defaultdict` to count occurrences of each local key.
* Utilizes a lambda function for sorting by title (case-insensitive).
* Raises a ValueError if multiple versions of a title are found for a duplicate key.

**Notable Dependencies or Requirements**

* Uses Python's built-in `List`, `dict`, and `defaultdict` types.
* Assumes the presence of an `_toctree.yml` file with documentation data.


key: chroma:document
string_value:
CODE:

def check_model_doc(overwrite: bool = False):
    """
    Check that the content of the table of content in `_toctree.yml` is clean (no duplicates and sorted for the model
    API doc) and potentially auto-cleans it.

    Args:
        overwrite (`bool`, *optional*, defaults to `False`):
            Whether to just check if the TOC is clean or to auto-clean it (when `overwrite=True`).
    """
    with open(PATH_TO_TOC, encoding="utf-8") as f:
        content = yaml.safe_load(f.read())

    # Get to the API doc
    api_idx = 0
    while content[api_idx]["title"] != "API":
        api_idx += 1
    api_doc = content[api_idx]["sections"]

    # Then to the model doc
    model_idx = 0
    while api_doc[model_idx]["title"] != "Models":
        model_idx += 1

    model_doc = api_doc[model_idx]["sections"]

    # Extract the modalities and clean them one by one.
    modalities_docs = [(idx, section) for idx, section in enumerate(model_doc) if "sections" in section]
    diff = False
    for idx, modality_doc in modalities_docs:
        old_modality_doc = modality_doc["sections"]
        new_modality_doc = clean_model_doc_toc(old_modality_doc)

        if old_modality_doc != new_modality_doc:
            diff = True
            if overwrite:
                model_doc[idx]["sections"] = new_modality_doc

    if diff:
        if overwrite:
            api_doc[model_idx]["sections"] = model_doc
            content[api_idx]["sections"] = api_doc
            with open(PATH_TO_TOC, "w", encoding="utf-8") as f:
                f.write(yaml.dump(content, allow_unicode=True))
        else:
            raise ValueError(
                "The model doc part of the table of content is not properly sorted, run `make style` to fix this."


SUMMARY:
**Summary: check_model_doc function**

The `check_model_doc` function validates and potentially auto-cleans the table of contents (`_toctree.yml`) for the model API doc. It checks for:

* Duplicates: A duplicate title is found, indicating an invalid TOC structure.
* Sorting: The TOC contains unsorted sections.

**Key Implementation Details**

1. Reads `PATH_TO_TOC` file using `yaml.safe_load()`.
2. Iterates to find the "API" and "Models" section indices in the content tree.
3. Extracts modalities documents from the "Models" section.
4. Cleans each modality document using `clean_model_doc_toc()`.

**Notable Dependencies/Requirements**

* `yaml`: for parsing the `_toctree.yml` file.
* `PATH_TO_TOC`: a constant representing the file path to `._toctree.yml`.

**Interfaces and Method Signatures**

1. `yaml.safe_load()`: loads YAML data from a string.
2. `clean_model_doc_toc()`: cleans a model doc using unknown implementation details.

This function appears to be part of a larger project for generating and maintaining documentation, possibly in a MarkDown or Sphinx-based environment.


key: chroma:document
string_value:
CODE:

def create_script(target_test):
    """Create a python script to be run by `git bisect run` to determine if `target_test` passes or fails.
    If a test is not found in a commit, the script with exit code `0` (i.e. `Success`).

    Args:
        target_test (`str`): The test to check.

    Returns:
        `str`: The script to be run by `git bisect run`.
    """

    script = f"""
import os
import subprocess

result = subprocess.run(
    ["python3", "-m", "pytest", "-v", f"{target_test}"],
    capture_output = True,
    text=True,
)
print(result.stdout)

if len(result.stderr) > 0:
    if "ERROR: file or directory not found: " in result.stderr:
        print("test not found in this commit")
        exit(0)
    else:
        print(f"pytest failed to run: {{result.stderr}}")
        exit(-1)
elif f"{target_test} FAILED" in result.stdout:
    print("test failed")
    exit(2)

exit(0)
"""

    with open("target_script.py", "w") as fp:


SUMMARY:
**Summary:**

The `create_script` function generates Python script to be run by `git bisect run` for verifying the outcome of test `target_test`. The script imports required modules, runs the specified test using `pytest`, and outputs the result.

**Key Implementation Details:**

* Runs `python3 -m pytest -v <test_name>` with specified `target_test`
* Checks if error or failure message is present in standard output/error
* Exits with success (0) or failure (-1/2) depending on validation

**Notable Dependencies/Requirements:**

* `os` and `subprocess` modules for executing external commands
* `pytest` module for running tests


key: chroma:document
string_value:
CODE:

def find_bad_commit(target_test, start_commit, end_commit):
    """Find (backward) the earliest commit between `start_commit` and `end_commit` at which `target_test` fails.

    Args:
        target_test (`str`): The test to check.
        start_commit (`str`): The latest commit.
        end_commit (`str`): The earliest commit.

    Returns:
        `str`: The earliest commit at which `target_test` fails.
    """

    if start_commit == end_commit:
        return start_commit

    create_script(target_test=target_test)

    bash = f"""
git bisect reset
git bisect start {start_commit} {end_commit}
git bisect run python3 target_script.py
"""

    with open("run_git_bisect.sh", "w") as fp:
        fp.write(bash.strip())

    result = subprocess.run(
        ["bash", "run_git_bisect.sh"],
        capture_output=True,
        text=True,
    )
    print(result.stdout)

    if "error: bisect run failed" in result.stderr:
        index = result.stderr.find("error: bisect run failed")
        bash_error = result.stderr[index:]

        error_msg = f"Error when running git bisect:\nbash error: {bash_error}"

        pattern = "pytest failed to run: .+"
        pytest_errors = re.findall(pattern, result.stdout)
        if len(pytest_errors) > 0:
            pytest_error = pytest_errors[0]
            index = pytest_error.find("pytest failed to run: ")
            index += len("pytest failed to run: ")
            pytest_error = pytest_error[index:]
            error_msg += f"pytest error: {pytest_error}"

        raise ValueError(error_msg)

    pattern = r"(.+) is the first bad commit"
    commits = re.findall(pattern, result.stdout)

    bad_commit = None
    if len(commits) > 0:
        bad_commit = commits[0]

    print(f"Between `start_commit` {start_commit} and `end_commit` {end_commit}")
    print(f"bad_commit: {bad_commit}\n")



SUMMARY:
**Function Summary: find_bad_commit**

**Purpose:** Find the earliest commit between two given commits at which a specified test fails.

### Functionality:

1.  **Initial Check**: Verifies if `start_commit` and `end_commit` are equal, returning `start_commit` in this case.
2.  Creates a bash script to reset git bisect mode and start it with the provided `target_test`.
3.  Runs the `run_git_bisect.sh` script using `subprocess.run`, capturing its output and error messages.
4.  Checks for errors, specifically "bisect run failed". If found, extracts the error message and continues.
5.  Seeks the first occurrence of "first bad commit" in the standard output to identify the earliest failing commit.

### Key Implementation Details:

*   `git bisect` mode is managed using bash scripts and subprocesses.
*   Standard output and error messages are analyzed for errors or test failures.
*   Regular expressions (`re`) are used to extract specific information from the output.

**Key Method Signatures:**

*   `create_script(target_test)`
*   `subprocess.run(["bash", "run_git_bisect.sh"])`

**Dependencies and Interfaces:**

*   External dependencies:
    *   `git` command
    *   `python3` for running script
    *   `re` (regular expression library) for pattern matching

**Interfaces:**

| Interface | Description                |
| :-------- | -------------------------- |
| `target_test` | Target test to check          |

### Notable Variability:

*   Initial commit check logic.
*   Bash script and subprocess execution.

This analysis highlights the primary purpose and key implementation details of the provided function, facilitating a deeper understanding of its functionality and potential areas for optimization.


key: chroma:document
string_value:
CODE:

def get_commit_info(commit):
    """Get information for a commit via `api.github.com`."""
    pr_number = None
    author = None
    merged_author = None

    url = f"https://api.github.com/repos/huggingface/transformers/commits/{commit}/pulls"
    pr_info_for_commit = requests.get(url).json()

    if len(pr_info_for_commit) > 0:
        pr_number = pr_info_for_commit[0]["number"]

        url = f"https://api.github.com/repos/huggingface/transformers/pulls/{pr_number}"
        pr_for_commit = requests.get(url).json()
        author = pr_for_commit["user"]["login"]
        merged_author = pr_for_commit["merged_by"]["login"]

    if author is None:
        url = f"https://api.github.com/repos/huggingface/transformers/commits/{commit}"
        commit_info = requests.get(url).json()
        author = commit_info["author"]["login"]



SUMMARY:
**Summary**

The `get_commit_info` function retrieves information about a GitHub commit, including the pull request number, author, and merged author. It makes API requests to `api.github.com` to fetch data.

* Purpose: Fetch commit metadata via GitHub API.
* Implementation Details:
	+ Retrieves PR info for a given commit and extracts the PR number.
	+ Fetches details about the associated PR, including author and merged author, if available.
	+ Fallbacks to fetching original commit information using `commit` URL if no PR author is found.
	+ Returns a dictionary with commit-related data.

**Key Method Signsatures/Interfaces**

* `requests.get()`: Used for API requests (`url = f..."`)

**Notable Dependencies**

* `requests`
* `json` ( implicit, handling JSON responses from GitHub API)
* `user`, `login` (accessed through PR and commit responses)

**Interface Inheritance/Implementation**

None.

This function leverages the GitHub API to collect metadata. The implementation caters to both cases when a PR is associated with the commit and when it's not.


key: chroma:document
string_value:
CODE:

def onnx_compliancy(saved_model_path, strict, opset):
    saved_model = SavedModel()
    onnx_ops = []

    with open(os.path.join(REPO_PATH, "utils", "tf_ops", "onnx.json")) as f:
        onnx_opsets = json.load(f)["opsets"]

    for i in range(1, opset + 1):
        onnx_ops.extend(onnx_opsets[str(i)])

    with open(saved_model_path, "rb") as f:
        saved_model.ParseFromString(f.read())

    model_op_names = set()

    # Iterate over every metagraph in case there is more than one (a saved model can contain multiple graphs)
    for meta_graph in saved_model.meta_graphs:
        # Add operations in the graph definition
        model_op_names.update(node.op for node in meta_graph.graph_def.node)

        # Go through the functions in the graph definition
        for func in meta_graph.graph_def.library.function:
            # Add operations in each function
            model_op_names.update(node.op for node in func.node_def)

    # Convert to list, sorted if you want
    model_op_names = sorted(model_op_names)
    incompatible_ops = []

    for op in model_op_names:
        if op not in onnx_ops and op not in INTERNAL_OPS:
            incompatible_ops.append(op)

    if strict and len(incompatible_ops) > 0:
        raise Exception(f"Found the following incompatible ops for the opset {opset}:\n" + incompatible_ops)
    elif len(incompatible_ops) > 0:
        print(f"Found the following incompatible ops for the opset {opset}:")
        print(*incompatible_ops, sep="\n")
    else:


SUMMARY:
**Function Summary: `onnx_compliancy`**

### Purpose and Functionality

The `onnx_compliancy` function checks if a saved model (in ONNX format) is compliant with the specified ONNX opset version. It parses the saved model, extracts its operations, and verifies that they are compatible with the given opset.

**Key Implementation Details**

* Extracts ONNX opsets from `onnx.json` file
* Parses saved model from file using `SavedModel.ParseFromString`
* Iterates over multiple graphs in saved model (if present)
* Compares extracted operation names from saved model to list of compatible operations (`onnx_ops`) and internal operations (`INTERNAL_OPS`)
* Raises exception if strict mode is enabled and incompatible ops are found

**Notable Dependencies and Requirements**

* `SavedModel` class
* `json` module for parsing `onnx.json` file
* `os` module for getting repository path (`REPO_PATH`)
* External dependencies:
	+ `onnx_ops` (list of compatible ONNX opsets)
	+ `INTERNAL_OPS` (list of internal operations)


key: chroma:document
string_value:
CODE:

class QuantizationMessage(Message):
    def __init__(
        self,
        title: str,
        results: Dict,
    ):
        self.title = title

        # Failures and success of the modeling tests
        self.n_success = sum(r["success"] for r in results.values())
        self.single_gpu_failures = sum(r["failed"]["single"] for r in results.values())
        self.multi_gpu_failures = sum(r["failed"]["multi"] for r in results.values())
        self.n_failures = self.single_gpu_failures + self.multi_gpu_failures

        self.n_tests = self.n_failures + self.n_success
        self.results = results
        self.thread_ts = None

    @property
    def payload(self) -> str:
        blocks = [self.header]

        if self.n_failures > 0:
            blocks.append(self.failures_overwiew)
            blocks.append(self.failures_detailed)

        if self.n_failures == 0:
            blocks.append(self.no_failures)

        return json.dumps(blocks)

    @property
    def time(self) -> str:
        all_results = self.results.values()
        time_spent = []
        for r in all_results:
            if len(r["time_spent"]):
                time_spent.extend([x for x in r["time_spent"].split(", ") if len(x.strip())])
        total_secs = 0

        for time in time_spent:
            time_parts = time.split(":")

            # Time can be formatted as xx:xx:xx, as .xx, or as x.xx if the time spent was less than a minute.
            if len(time_parts) == 1:
                time_parts = [0, 0, time_parts[0]]

            hours, minutes, seconds = int(time_parts[0]), int(time_parts[1]), float(time_parts[2])
            total_secs += hours * 3600 + minutes * 60 + seconds

        hours, minutes, seconds = total_secs // 3600, (total_secs % 3600) // 60, total_secs % 60
        return f"{int(hours)}h{int(minutes)}m{int(seconds)}s"

    @property
    def failures_overwiew(self) -> Dict:
        return {
            "type": "section",
            "text": {
                "type": "plain_text",
                "text": (
                    f"There were {self.n_failures} failures, out of {self.n_tests} tests.\n"
                    f"The suite ran in {self.time}."
                ),
                "emoji": True,
            },
            "accessory": {
                "type": "button",
                "text": {"type": "plain_text", "text": "Check Action results", "emoji": True},
                "url": f"https://github.com/huggingface/transformers/actions/runs/{os.environ['GITHUB_RUN_ID']}",
            },
        }

    @property
    def failures_detailed(self) -> Dict:
        failures = {k: v["failed"] for k, v in self.results.items()}

        individual_reports = []
        for key, value in failures.items():
            device_report = self.get_device_report(value)
            if sum(value.values()):
                report = f"{device_report}{key}"
                individual_reports.append(report)

        header = "Single |  Multi | Category\n"
        failures_report = prepare_reports(
            title="The following quantization tests had failures", header=header, reports=individual_reports
        )

        return {"type": "section", "text": {"type": "mrkdwn", "text": failures_report}}

    def post(self):
        payload = self.payload
        print("Sending the following payload")
        print(json.dumps({"blocks": json.loads(payload)}))

        text = f"{self.n_failures} failures out of {self.n_tests} tests," if self.n_failures else "All tests passed."

        self.thread_ts = client.chat_postMessage(
            channel=SLACK_REPORT_CHANNEL_ID,
            blocks=payload,
            text=text,
        )

    def post_reply(self):
        if self.thread_ts is None:
            raise ValueError("Can only post reply if a post has been made.")

        for job, job_result in self.results.items():
            if len(job_result["failures"]):
                for device, failures in job_result["failures"].items():
                    blocks = self.get_reply_blocks(
                        job,
                        job_result,
                        failures,
                        device,
                        text=f'Number of failures: {job_result["failed"][device]}',
                    )

                    print("Sending the following reply")
                    print(json.dumps({"blocks": blocks}))

                    client.chat_postMessage(
                        channel="#transformers-ci-daily-quantization",
                        text=f"Results for {job}",
                        blocks=blocks,
                        thread_ts=self.thread_ts["ts"],
                    )


SUMMARY:
**Summary: QuantizationMessage Class**

This class, `QuantizationMessage`, inherits from the `Message` class and represents a message containing quantization testing results.

**Main Purpose and Functionality**
The main purpose of this class is to store and format the results of quantization tests in a JSON payload that can be sent through Slack.

**Implementation Details**

*   **Initialization**: The constructor takes two parameters: `title` (a string) and `results` (a dictionary).
    *   `results`: A dictionary containing test results, including failure information.
        *   `n_success`
        *   `single_gpu_failures`
        *   `multi_gpu_failures`
        *   `n_failures`
        *   `n_tests`
        *   `thread_ts` (initially None)
*   **Properties**:
    *   `payload`: Returns the formatted JSON payload as a string, including optional sections for failure overview and detailed reports.
        *   `blocks`: A list of JSON blocks containing the message content.
        *   Includes failuer section
    *   `time`: Calculates and returns the total time spent on all tests in hours, minutes, and seconds.
*   **Methods**:
    *   `post()`: Sends the message payload to a Slack channel using the client's chat post message method.
    *   `post_reply()`: Posts replies for individual jobs with failures.


key: chroma:document
string_value:
CODE:

def get_new_python_files_between_commits(base_commit: str, commits: List[str]) -> List[str]:
    """
    Get the list of added python files between a base commit and one or several commits.

    Args:
        repo (`git.Repo`):
            A git repository (for instance the Transformers repo).
        base_commit (`str`):
            The commit reference of where to compare for the diff. This is the current commit, not the branching point!
        commits (`List[str]`):
            The list of commits with which to compare the repo at `base_commit` (so the branching point).

    Returns:
        `List[str]`: The list of python files added between a base commit and one or several commits.
    """
    code_diff = []
    for commit in commits:
        for diff_obj in commit.diff(base_commit):
            # We always add new python files
            if diff_obj.change_type == "A" and diff_obj.b_path.endswith(".py"):
                code_diff.append(diff_obj.b_path)



SUMMARY:
**Function Summary:**

`get_new_python_files_between_commits`

Returns a list of added Python files between a base commit (`base_commit`) and one or several commits (`commits`). Compares each commit's diff against the base commit to identify new Python files.

**Key Implementation Details:**

* Uses `git.Repo.diff()` to compare each commit's diff against the base commit.
* Iterates through a list of commits (`commits`) using the `diff()` method.
* Identifies new Python files added in each commit by checking for file creation type ("A") and file extension (`.py`).
* Returns a list of paths to newly created Python files.

**Notable Dependencies/Requirements:**

* `git.Repo`: Git repository access
* `List[str]`: List of commit references

**Primary Purpose:**
Identify new Python files added between a base commit and one or several commits in a Git repository.


key: chroma:document
string_value:
CODE:

def get_new_python_files() -> List[str]:
    """
    Return a list of python files that have been added between the current head and the main branch.

    Returns:
        `List[str]`: The list of python files added.
    """
    repo = Repo(PATH_TO_REPO)

    try:
        # For the cases where the main branch exists locally
        main = repo.refs.main
    except AttributeError:
        # On GitHub Actions runners, it doesn't have local main branch
        main = repo.remotes.origin.refs.main

    print(f"main is at {main.commit}")
    print(f"Current head is at {repo.head.commit}")

    branching_commits = repo.merge_base(main, repo.head)
    for commit in branching_commits:
        print(f"Branching commit: {commit}")


SUMMARY:
**Function Summary**

`get_new_python_files()` returns a list of Python files added between the current Git head and the `main` branch.

**Key Implementation Details**

* Uses `Repo` class to interact with the GitHub repository
* Checks if `main` branch exists locally, falling back to remote `origin` in GitHub Actions runners
* Retrieves merge base commits using `repo.merge_base`
* Returns list of new Python files added using `get_new_python_files_between_commits`

**Notable Dependencies and Requirements**

* `Repo` class from `pathlib` library ( implied but not explicitly stated)
* External dependency on GitHub repository at `PATH_TO_REPO`


key: chroma:document
string_value:
CODE:

def get_new_model():
    new_files = get_new_python_files()
    reg = re.compile(r"src/transformers/models/(.*)/modeling_.*\.py")

    new_model = ""
    for x in new_files:
        find_new_model = reg.findall(x)
        if len(find_new_model) > 0:
            new_model = find_new_model[0]
            # It's unlikely we have 2 new modeling files in a pull request.
            break


SUMMARY:
**Function Summary: `get_new_model`**

Retrieves the names of newly created Python model files in a Git repository.

**Primary Functionality:**
Parses file contents to extract and return the name of the new model file, assuming only one new modeling file exists in each pull request.

**Key Implementation Details:**

* Uses regular expression `reg` to extract pattern "src/transformers/models/(.*)/modeling_.*\.py"
* Iterates over list of new Python files (`new_files`) and uses `re.findall` to find the matching model file name
* Returns the extracted model file name as a string

**Notable Dependencies or Requirements:**

* `re` (regular expression) module for pattern matching
* `get_new_python_files()` external function or call, returning list of new Python files


key: chroma:document
string_value:
CODE:

def parse_message(message: str) -> str:
    """
    Parses a GitHub pull request's comment to find the models specified in it to run slow CI.

    Args:
        message (`str`): The body of a GitHub pull request's comment.

    Returns:
        `str`: The substring in `message` after `run-slow`, run_slow` or run slow`. If no such prefix is found, the
        empty string is returned.
    """
    if message is None:
        return ""

    message = message.strip().lower()

    # run-slow: model_1, model_2
    if not message.startswith(("run-slow", "run_slow", "run slow")):
        return ""
    message = message[len("run slow") :]
    # remove leading `:`
    while message.strip().startswith(":"):
        message = message.strip()[1:]



SUMMARY:
**Summary**

### Functionality

The `parse_message` function parses a GitHub pull request comment to extract models specified in the message to run slow CI.

### Implementation Details

* **Key Algorithm:** Trims and normalizes user input message.
* **Method Signature:**
	+ `parse_message(message: str) -> str`
* **Design Choice:** Uses multiple prefixes ("run-slow", "run_slow", "run slow") to account for varying notation in comments.

### Dependencies and Requirements

* None explicitly listed, but assumes `str` type for input message.


key: chroma:document
string_value:
CODE:

def get_models(message: str):
    models = parse_message(message)


SUMMARY:
**Summary**

### Function Overview

`get_models(message: str)`

Extracts and preprocesses a list of models from a given message.

### Primary Purpose

Resets and splits the `models` string to separate individual model names, replacing commas with spaces.

### Key Implementation Details

* Uses `parse_message(message)` (exact class/function not provided)
* Applies whitespace manipulation to output string
* Utilizes string splitting (`split()`) on `models`

### Notable Dependencies or Requirements

* `parse_message(message)` (related function not provided)
 
### External Dependencies

List of external dependencies is not specified in the given code chunk


key: chroma:document
string_value:
CODE:

def check_model_names(model_name: str):
    allowed = string.ascii_letters + string.digits + "_"


SUMMARY:
**Summary**

Checks if a `model_name` is valid. A valid model name consists of letters, digits, and underscores only.

* **Primary Purpose**: Validate model names to ensure they comply with the specified format.
* **Key Implementation Details**:
	+ Uses regular expression for string validation (`string.ascii_letters + string.digits + "_"`).
	+ Checks for leading/trailing underscore prefixes using `startswith()` and `endswith()`.
* **Notable Dependencies**:
	+ `string` module for character classification.
* **Interfaces/Inherited Class**: None.

In RAG system notation: 
### model_name_validation
- Description: Validates a model name to ensure it contains only letters, digits, or underscores.
- Primary Purpose: Ensure model names meet the specified format requirement.


key: chroma:document
string_value:
CODE:

def process_file(modular_file_path, generated_modeling_content, file_type="modeling_", fix_and_overwrite=False):
    file_name_prefix = file_type.split("*")[0]
    file_name_suffix = file_type.split("*")[-1] if "*" in file_type else ""
    file_path = modular_file_path.replace("modular_", f"{file_name_prefix}_").replace(".py", f"{file_name_suffix}.py")
    # Read the actual modeling file
    with open(file_path, "r") as modeling_file:
        content = modeling_file.read()
    output_buffer = StringIO(generated_modeling_content[file_type][0])
    output_buffer.seek(0)
    output_content = output_buffer.read()
    diff = difflib.unified_diff(
        output_content.splitlines(),
        content.splitlines(),
        fromfile=f"{file_path}_generated",
        tofile=f"{file_path}",
        lineterm="",
    )
    diff_list = list(diff)
    # Check for differences
    if diff_list:
        if fix_and_overwrite:
            with open(file_path, "w") as modeling_file:
                modeling_file.write(generated_modeling_content[file_type][0])
            console.print(f"[bold blue]Overwritten {file_path} with the generated content.[/bold blue]")
        else:
            console.print(f"\n[bold red]Differences found between the generated code and {file_path}:[/bold red]\n")
            diff_text = "\n".join(diff_list)
            syntax = Syntax(diff_text, "diff", theme="ansi_dark", line_numbers=True)
            console.print(syntax)
        return 1
    else:
        console.print(f"[bold green]No differences found for {file_path}.[/bold green]")


SUMMARY:
**Function Summary**

`process_file(modular_file_path, generated_modeling_content, file_type="modeling_", fix_and_overwrite=False)`

**Purpose and Functionality**
Process and compare the content of two modeling files located in `modular_file_path`.

**Main Algorithms/P Patterns/Design Choices**

* Reading and comparing file contents using `difflib.unified_diff` to identify differences.
* Overwriting or printing differences based on `fix_and_overwrite` flag.

**Key Method Signatures and Interfaces**

- `difflib.unified_diff(fromfile, tofile, lineterm="")`
- `console.print()`

**Notable Dependencies and Requirements**

- `StringIO`
- `difflib`
- `console` (assuming a custom console object or module)
- `generated_modeling_content[file_type]` referencing an external data structure.


key: chroma:document
string_value:
CODE:

def compare_files(modular_file_path, fix_and_overwrite=False):
    # Generate the expected modeling content
    generated_modeling_content = convert_modular_file(modular_file_path)
    diff = 0
    for file_type in generated_modeling_content.keys():
        diff += process_file(modular_file_path, generated_modeling_content, file_type, fix_and_overwrite)


SUMMARY:
**Summary:**

`compare_files` function compares the expected modeling content with actual files specified by `modular_file_path`. It iterates through each file type, processing them using `process_file`, and returns the cumulative difference (`diff`). 

**Key Implementation Details:**

* Generates expected modeling content using `convert_modular_file(modular_file_path)`
	+ Input parameter: `modular_file_path`
	+ Output parameter: `generated_modeling_content` dictionary containing different file types
* Iterates through each file type in `generated_modeling_content` and processes them using `process_file(modular_file_path, generated_modeling_content, file_type, fix_and_overwrite)`
	+ Input parameters: `modular_file_path`, `generated_modeling_content`, `file_type`
	+ Output parameter: cumulative difference (`diff`)
* Returns the cumulative difference (`diff`) after processing all files

**Notable Dependencies/Requirements:**

* `convert_modular_file` function
* `process_file` function with parameters `modular_file_path`, `generated_modeling_content`, `file_type`, and `fix_and_overwrite`
 
This summary aims to preserve the original method's flow, focus on key implementation details, and reference related functions by exact name.


key: chroma:document
string_value:
CODE:

def find_backend(line: str) -> Optional[str]:
    """
    Find one (or multiple) backend in a code line of the init.

    Args:
        line (`str`): A code line of the main init.

    Returns:
        Optional[`str`]: If one (or several) backend is found, returns it. In the case of multiple backends (the line
        contains `if is_xxx_available() and `is_yyy_available()`) returns all backends joined on `_and_` (so
        `xxx_and_yyy` for instance).
    """
    if _re_test_backend.search(line) is None:
        return None
    backends = [b[0] for b in _re_backend.findall(line)]
    backends.sort()


SUMMARY:
**Summary**

`find_backend` function identifies and extracts multiple backend names (`xxx_and_yyy`) from a given code line of the main `init` file. It performs a case-insensitive search using regular expression `_re_test_backend.search(line)`.

**Implementation Details**

* Uses two regular expressions: `_re_test_backend` to search for the backend pattern and `_re_backend` to extract the backend names.
* Extracts backend names from the line using a list comprehension (`[b[0] for b in _re_backend.findall(line)]`) and sorts them alphabetically.

**Return Value**

Returns all extracted backend names joined on `_and_`, or `None` if no backend is found (`if _re_test_backend.search(line) is None`).


key: chroma:document
string_value:
CODE:

def parse_init(init_file) -> Optional[Tuple[Dict[str, List[str]], Dict[str, List[str]]]]:
    """
    Read an init_file and parse (per backend) the `_import_structure` objects defined and the `TYPE_CHECKING` objects
    defined.

    Args:
        init_file (`str`): Path to the init file to inspect.

    Returns:
        `Optional[Tuple[Dict[str, List[str]], Dict[str, List[str]]]]`: A tuple of two dictionaries mapping backends to list of
        imported objects, one for the `_import_structure` part of the init and one for the `TYPE_CHECKING` part of the
        init. Returns `None` if the init is not a custom init.
    """
    with open(init_file, "r", encoding="utf-8", newline="\n") as f:
        lines = f.readlines()

    # Get the to `_import_structure` definition.
    line_index = 0
    while line_index < len(lines) and not lines[line_index].startswith("_import_structure = {"):
        line_index += 1

    # If this is a traditional init, just return.
    if line_index >= len(lines):
        return None

    # First grab the objects without a specific backend in _import_structure
    objects = []
    while not lines[line_index].startswith("if TYPE_CHECKING") and find_backend(lines[line_index]) is None:
        line = lines[line_index]
        # If we have everything on a single line, let's deal with it.
        if _re_one_line_import_struct.search(line):
            content = _re_one_line_import_struct.search(line).groups()[0]
            imports = re.findall(r"\[([^\]]+)\]", content)
            for imp in imports:
                objects.extend([obj[1:-1] for obj in imp.split(", ")])
            line_index += 1
            continue
        single_line_import_search = _re_import_struct_key_value.search(line)
        if single_line_import_search is not None:
            imports = [obj[1:-1] for obj in single_line_import_search.groups()[0].split(", ") if len(obj) > 0]
            objects.extend(imports)
        elif line.startswith(" " * 8 + '"'):
            objects.append(line[9:-3])
        line_index += 1

    # Those are stored with the key "none".
    import_dict_objects = {"none": objects}

    # Let's continue with backend-specific objects in _import_structure
    while not lines[line_index].startswith("if TYPE_CHECKING"):
        # If the line is an if not is_backend_available, we grab all objects associated.
        backend = find_backend(lines[line_index])
        # Check if the backend declaration is inside a try block:
        if _re_try.search(lines[line_index - 1]) is None:
            backend = None

        if backend is not None:
            line_index += 1

            # Scroll until we hit the else block of try-except-else
            while _re_else.search(lines[line_index]) is None:
                line_index += 1

            line_index += 1

            objects = []
            # Until we unindent, add backend objects to the list
            while len(lines[line_index]) <= 1 or lines[line_index].startswith(" " * 4):
                line = lines[line_index]
                if _re_import_struct_add_one.search(line) is not None:
                    objects.append(_re_import_struct_add_one.search(line).groups()[0])
                elif _re_import_struct_add_many.search(line) is not None:
                    imports = _re_import_struct_add_many.search(line).groups()[0].split(", ")
                    imports = [obj[1:-1] for obj in imports if len(obj) > 0]
                    objects.extend(imports)
                elif _re_between_brackets.search(line) is not None:
                    imports = _re_between_brackets.search(line).groups()[0].split(", ")
                    imports = [obj[1:-1] for obj in imports if len(obj) > 0]
                    objects.extend(imports)
                elif _re_quote_object.search(line) is not None:
                    objects.append(_re_quote_object.search(line).groups()[0])
                elif line.startswith(" " * 8 + '"'):
                    objects.append(line[9:-3])
                elif line.startswith(" " * 12 + '"'):
                    objects.append(line[13:-3])
                line_index += 1

            import_dict_objects[backend] = objects
        else:
            line_index += 1

    # At this stage we are in the TYPE_CHECKING part, first grab the objects without a specific backend
    objects = []
    while (
        line_index < len(lines)
        and find_backend(lines[line_index]) is None
        and not lines[line_index].startswith("else")
    ):
        line = lines[line_index]
        single_line_import_search = _re_import.search(line)
        if single_line_import_search is not None:
            objects.extend(single_line_import_search.groups()[0].split(", "))
        elif line.startswith(" " * 8):
            objects.append(line[8:-2])
        line_index += 1

    type_hint_objects = {"none": objects}

    # Let's continue with backend-specific objects
    while line_index < len(lines):
        # If the line is an if is_backend_available, we grab all objects associated.
        backend = find_backend(lines[line_index])
        # Check if the backend declaration is inside a try block:
        if _re_try.search(lines[line_index - 1]) is None:
            backend = None

        if backend is not None:
            line_index += 1

            # Scroll until we hit the else block of try-except-else
            while _re_else.search(lines[line_index]) is None:
                line_index += 1

            line_index += 1

            objects = []
            # Until we unindent, add backend objects to the list
            while len(lines[line_index]) <= 1 or lines[line_index].startswith(" " * 8):
                line = lines[line_index]
                single_line_import_search = _re_import.search(line)
                if single_line_import_search is not None:
                    objects.extend(single_line_import_search.groups()[0].split(", "))
                elif line.startswith(" " * 12):
                    objects.append(line[12:-2])
                line_index += 1

            type_hint_objects[backend] = objects
        else:
            line_index += 1



SUMMARY:
**Function Summary: `parse_init`**

Parsing an init file to extract import and TYPE_CHECKING definitions.

**Purpose:** Read an init file, identifying and parsing the `_import_structure` and `TYPE_CHECKING` objects defined in it.

**Primary Purpose:** The function reads a `.init` file and returns two dictionaries mapping backends to lists of imported objects: one for the `_import_structure` section and one for the `TYPE_CHECKING` section.

**Key Implementation Details:**

*   Checks for `_import_structure = { ... }` definitions to parse backend-specific imports.
*   Iterates through lines to identify and extract:
    *   Import structures without a specific backend (`_re_one_line_import_struct.search`)
    *   Backend-specific imports within the try-except-else block (`_re_backend_find`)
    *   TYPE_CHECKING objects without a specific backend (`_re_import.search`)
*   Parses import definitions using regular expressions (e.g., `_import_struct_add_many`, `_re_between_brackets`)

**Notable Dependencies and Requirements:**

*   The function uses regular expression classes (`_re_one_line_import_struct`, `_re_backend_find`, etc.) to parse the init file.
*   It relies on `find_backend()` function, but its implementation is not provided in this code snippet.

**Return Value:** A tuple of two dictionaries:
    *   `import_dict_objects`: Mapping backends to lists of import structures without a specific backend (`"none"`).
    *   `type_hint_objects`: Mapping backends (if applicable) to lists of TYPE_CHECKING objects without a specific backend (`"none"`).


key: chroma:document
string_value:
CODE:

def analyze_results(import_dict_objects: Dict[str, List[str]], type_hint_objects: Dict[str, List[str]]) -> List[str]:
    """
    Analyze the differences between _import_structure objects and TYPE_CHECKING objects found in an init.

    Args:
        import_dict_objects (`Dict[str, List[str]]`):
            A dictionary mapping backend names (`"none"` for the objects independent of any specific backend) to
            list of imported objects.
        type_hint_objects (`Dict[str, List[str]]`):
            A dictionary mapping backend names (`"none"` for the objects independent of any specific backend) to
            list of imported objects.

    Returns:
        `List[str]`: The list of errors corresponding to mismatches.
    """

    def find_duplicates(seq):
        return [k for k, v in collections.Counter(seq).items() if v > 1]

    # If one backend is missing from the other part of the init, error early.
    if list(import_dict_objects.keys()) != list(type_hint_objects.keys()):
        return ["Both sides of the init do not have the same backends!"]

    errors = []
    # Find all errors.
    for key in import_dict_objects.keys():
        # Duplicate imports in any half.
        duplicate_imports = find_duplicates(import_dict_objects[key])
        if duplicate_imports:
            errors.append(f"Duplicate _import_structure definitions for: {duplicate_imports}")
        duplicate_type_hints = find_duplicates(type_hint_objects[key])
        if duplicate_type_hints:
            errors.append(f"Duplicate TYPE_CHECKING objects for: {duplicate_type_hints}")

        # Missing imports in either part of the init.
        if sorted(set(import_dict_objects[key])) != sorted(set(type_hint_objects[key])):
            name = "base imports" if key == "none" else f"{key} backend"
            errors.append(f"Differences for {name}:")
            for a in type_hint_objects[key]:
                if a not in import_dict_objects[key]:
                    errors.append(f"  {a} in TYPE_HINT but not in _import_structure.")
            for a in import_dict_objects[key]:
                if a not in type_hint_objects[key]:
                    errors.append(f"  {a} in _import_structure but not in TYPE_HINT.")


SUMMARY:
**Function Summary**

`analyze_results` analyzes the discrepancies between `_import_structure` objects and `TYPE_CHECKING` objects found in an init.

**Main Purpose**
Identifies duplicate imports, missing imports, and incorrect backend names across both structures.

**Key Implementation Details**

* Verifies that both input dictionaries have identical keys (backends).
* Uses a custom function `find_duplicates` to detect duplicate values in lists.
* Iterates through each key-value pair, comparing the two dictionaries for:

    * Duplicate imports
    * Missing imports

**Notable Dependencies/Requirements**

* Python built-in libraries: `collections`
* External dependencies:
	+ None (self-contained)
* Implemented interfaces:
	+ No explicit interface implementation; relies on dictionary comparisons.


key: chroma:document
string_value:
CODE:

def check_all_inits():
    """
    Check all inits in the transformers repo and raise an error if at least one does not define the same objects in
    both halves.
    """
    failures = []
    for root, _, files in os.walk(PATH_TO_TRANSFORMERS):
        if "__init__.py" in files:
            fname = os.path.join(root, "__init__.py")
            objects = parse_init(fname)
            if objects is not None:
                errors = analyze_results(*objects)
                if len(errors) > 0:
                    errors[0] = f"Problem in {fname}, both halves do not define the same objects.\n{errors[0]}"
                    failures.append("\n".join(errors))
    if len(failures) > 0:


SUMMARY:
**Function Summary:**

`check_all_inits()`

Checks all initalization files (`__init__.py`) in the transformers repository for potential object definition discrepancies between both halves. Raises an error if any file fails to define the same objects.

**Key Implementation Details:**

* Iterates through `__init__.py` files in the transformers repository using `os.walk`.
* Parses initialization file contents using `parse_init(fname)`.
* Analyzes results using `analyze_results(*objects)` and accumulates errors.
* Raises a `ValueError` with error messages if any discrepancies are found.

**Notable Dependencies:**

* `os`, `PATH_TO_TRANSFORMERS`
* `parse_init`, `analyze_results` (external functions not shown in this code snippet)


key: chroma:document
string_value:
CODE:

def get_transformers_submodules() -> List[str]:
    """
    Returns the list of Transformers submodules.
    """
    submodules = []
    for path, directories, files in os.walk(PATH_TO_TRANSFORMERS):
        for folder in directories:
            # Ignore private modules
            if folder.startswith("_"):
                directories.remove(folder)
                continue
            # Ignore leftovers from branches (empty folders apart from pycache)
            if len(list((Path(path) / folder).glob("*.py"))) == 0:
                continue
            short_path = str((Path(path) / folder).relative_to(PATH_TO_TRANSFORMERS))
            submodule = short_path.replace(os.path.sep, ".")
            submodules.append(submodule)
        for fname in files:
            if fname == "__init__.py":
                continue
            short_path = str((Path(path) / fname).relative_to(PATH_TO_TRANSFORMERS))
            submodule = short_path.replace(".py", "").replace(os.path.sep, ".")
            if len(submodule.split(".")) == 1:
                submodules.append(submodule)


SUMMARY:
**Summary**

`get_transformers_submodules()` function returns a list of Transformers submodules by traversing the directory tree rooted at `PATH_TO_TRANSFORMERS`. It filters out private modules, empty folders, and initializes that are not submodule names.

**Key Implementation Details**

* Uses `os.walk()` to traverse directory tree
* Checks for file patterns in folders (`*.py`, `__init__.py`) and files (excluding `__init__.py`)
* Normalizes folder and file paths to standardize submodule names

**Notable Dependencies or Requirements**

* `PATH_TO_TRANSFORMERS`
* `os` module for walking directories
* `Path` class from `pathlib` library for path manipulation


key: chroma:document
string_value:
CODE:

def check_submodules():
    """
    Check all submodules of Transformers are properly registered in the main init. Error otherwise.
    """
    # This is to make sure the transformers module imported is the one in the repo.
    from transformers.utils import direct_transformers_import

    transformers = direct_transformers_import(PATH_TO_TRANSFORMERS)

    import_structure_keys = set(transformers._import_structure.keys())
    # This contains all the base keys of the _import_structure object defined in the init, but if the user is missing
    # some optional dependencies, they may not have all of them. Thus we read the init to read all additions and
    # (potentiall re-) add them.
    with open(os.path.join(PATH_TO_TRANSFORMERS, "__init__.py"), "r") as f:
        init_content = f.read()
    import_structure_keys.update(set(re.findall(r"import_structure\[\"([^\"]*)\"\]", init_content)))

    module_not_registered = [
        module
        for module in get_transformers_submodules()
        if module not in IGNORE_SUBMODULES and module not in import_structure_keys
    ]

    if len(module_not_registered) > 0:
        list_of_modules = "\n".join(f"- {module}" for module in module_not_registered)
        raise ValueError(
            "The following submodules are not properly registed in the main init of Transformers:\n"
            f"{list_of_modules}\n"
            "Make sure they appear somewhere in the keys of `_import_structure` with an empty list as value."


SUMMARY:
**Summary: `check_submodules()` Function**

* **Purpose:** Validate that all submodules in `Transformers` are properly initialized with their respective import structures.
* **Key Implementation Details:**
	+ Verifies `transformers._import_structure` keys against the expected structure and user-defined imports from `_init.py`.
	+ Compares submodule lists against imported `import_structure_keys`.
	+ Raises a `ValueError` if any submodules are not properly registered.
* **Notable Dependencies & Requirements:**
	+ `direct_transformers_import`: imports transformers module ensuring it's the one in the repo.
	+ `get_transformers_submodules`: recursively finds all submodules of Transformers.
	+ `_init.py`: provides user-defined import structure additions for optional dependencies.


key: chroma:document
string_value:
CODE:

def get_daily_ci_runs(token, num_runs=7):
    """Get the workflow runs of the scheduled (daily) CI.

    This only selects the runs triggered by the `schedule` event on the `main` branch.
    """
    headers = None
    if token is not None:
        headers = {"Accept": "application/vnd.github+json", "Authorization": f"Bearer {token}"}

    # The id of a workflow (not of a workflow run).
    # From a given workflow run (where we have workflow run id), we can get the workflow id by going to
    # https://api.github.com/repos/huggingface/transformers/actions/runs/{workflow_run_id}
    # and check the `workflow_id` key.
    workflow_id = "90575235"

    url = f"https://api.github.com/repos/huggingface/transformers/actions/workflows/{workflow_id}/runs"
    # On `main` branch + event being `schedule` + not returning PRs + only `num_runs` results
    url += f"?branch=main&event=schedule&exclude_pull_requests=true&per_page={num_runs}"

    result = requests.get(url, headers=headers).json()



SUMMARY:
**Function Summary: `get_daily_ci_runs`**

**Purpose and Functionality:** Retrieves the workflow run history of scheduled CI for the GitHub repository "huggingface/transformers" on the `main` branch filtered by schedule events.

**Key Implementation Details:**

* Employs the API endpoint `https://api.github.com/repos/huggingface/transformers/actions/workflows/{workflow_id}/runs` with query parameters:
	+ Branch: `main`
	+ Event: `schedule`
	+ Exclude pull requests: `true`
	+ Page size: `num_runs`
* Utilizes the `requests.get()` method to send a GET request to the API endpoint, authenticated via Bearer token.

**Notable Dependencies and Requirements:**

* `token`: GitHub API authentication token
* `requests` library for API calls
* External dependencies:
	+ github.com ( GitHub repository)
	+ transformrsactions API


key: chroma:document
string_value:
CODE:

def get_last_daily_ci_runs(token):
    """Get the last completed workflow run id of the scheduled (daily) CI."""
    workflow_runs = get_daily_ci_runs(token)
    workflow_run_id = None
    for workflow_run in workflow_runs:
        if workflow_run["status"] == "completed":
            workflow_run_id = workflow_run["id"]
            break



SUMMARY:
**Summary:**

```markdown
### Function:
#### `get_last_daily_ci_runs(token)`

#### Purpose:
 Retrieves the ID of the most recent completed daily Continuous Integration (CI) workflow.

#### Implementation Details:

* Calls `get_daily_ci_runs(token)` to retrieve daily CI runs
* Iterates through runs, returning the ID of the first complete run (`if "status" in workflow_run:`)
* Returns the most recent completed daily CI run's ID

#### Notable Dependencies:
None

#### Interfaces:
- `get_daily_ci_runs(token)`
```

**RAG Code Tag:**
`func: get_last_daily_ci_runs(token)`:


key: chroma:document
string_value:
CODE:

def get_last_daily_ci_run_commit(token):
    """Get the commit sha of the last completed scheduled daily CI workflow run."""
    workflow_runs = get_daily_ci_runs(token)
    head_sha = None
    for workflow_run in workflow_runs:
        if workflow_run["status"] == "completed":
            head_sha = workflow_run["head_sha"]
            break



SUMMARY:
**Summary**

The `get_last_daily_ci_run_commit` function retrieves the commit SHA of the last completed scheduled daily CI workflow run.

**Key Points:**

* **Purpose:** Returns the SHA of the latest completed CI run.
* **Functionality:** Iterates through completed daily CI runs to find the head SHA associated with the completion.
* **Implementation Details:**
	+ Calls `get_daily_ci_runs` function to retrieve successful workflow runs.
	+ Loops through runs, identifying completed ones and saving their corresponding head SHA.
	+ Returns the head SHA of the last completed run.

**Notable Dependencies/Requirements:**

* `get_daily_ci_runs(token)` - Function called to retrieve daily CI runs.


key: chroma:document
string_value:
CODE:

def get_last_daily_ci_artifacts(artifact_names, output_dir, token):
    """Get the artifacts of last completed workflow run id of the scheduled (daily) CI."""
    workflow_run_id = get_last_daily_ci_runs(token)
    if workflow_run_id is not None:
        artifacts_links = get_artifacts_links(worflow_run_id=workflow_run_id, token=token)
        for artifact_name in artifact_names:
            if artifact_name in artifacts_links:
                artifact_url = artifacts_links[artifact_name]
                download_artifact(
                    artifact_name=artifact_name, artifact_url=artifact_url, output_dir=output_dir, token=token


SUMMARY:
**Function Summary:**

`get_last_daily_ci_artifacts`

 Retrieves and downloads artifacts from the last completed daily CI workflow run.

**Key Implementation Details:**

* Fetches `workflow_run_id` using `get_last_daily_ci_runs`.
* Retrieves artifact links for the specified `workflow_run_id` using `get_artifacts_links`.
* Iterates over `artifact_names`, downloading artifacts matching those names using `download_artifact`.

**Notable Dependencies or Requirements:**

* Utilizes `token` in multiple calls.
* Calls `get_artifacts_links` with `workflow_run_id` and `token`.
* Expects implementation of `download_artifact` function separate from this snippet.

This summary prescribes the main purpose, key algorithms/patterns, and dependencies used within `get_last_daily_ci_artifacts`.


key: chroma:document
string_value:
CODE:

def get_last_daily_ci_reports(artifact_names, output_dir, token):
    """Get the artifacts' content of the last completed workflow run id of the scheduled (daily) CI."""
    get_last_daily_ci_artifacts(artifact_names, output_dir, token)

    results = {}
    for artifact_name in artifact_names:
        artifact_zip_path = os.path.join(output_dir, f"{artifact_name}.zip")
        if os.path.isfile(artifact_zip_path):
            results[artifact_name] = {}
            with zipfile.ZipFile(artifact_zip_path) as z:
                for filename in z.namelist():
                    if not os.path.isdir(filename):
                        # read the file
                        with z.open(filename) as f:
                            results[artifact_name][filename] = f.read().decode("UTF-8")



SUMMARY:
**Summary**

The `get_last_daily_ci_reports` function retrieves the artifacts' content from the last completed daily CI workflow run. It takes three parameters: `artifact_names`, `output_dir`, and `token`.

**Purpose and Functionality**

Retrieves artifacts from the last daily CI workflow run, saving their contents in a dictionary for each artifact.

**Key Implementation Details**

* Iterates over `.zip` files with artifact names matching in the provided directories.
* Reads and decodes file contents using UTF-8 encoding.
* Returns a nested dictionary containing artifact names as keys and their corresponding file contents as values.

**Notable Dependencies or Requirements**

* `os.path.isfile`
* `os.open`
* `zipped zipfile.ZipFile`

Note: This summary assumes the absence of any external dependencies not mentioned in the provided code. If such dependencies exist, please provide them for a more comprehensive analysis.


key: chroma:document
string_value:
CODE:

def extract_time_from_single_job(job):
    """Extract time info from a single job in a GitHub Actions workflow run"""

    job_info = {}

    start = job["started_at"]
    end = job["completed_at"]

    start_datetime = date_parser.parse(start)
    end_datetime = date_parser.parse(end)

    duration_in_min = round((end_datetime - start_datetime).total_seconds() / 60.0)

    job_info["started_at"] = start
    job_info["completed_at"] = end
    job_info["duration"] = duration_in_min



SUMMARY:
**Summary**

This function, `extract_time_from_single_job`, extracts time-related information from a single job in a GitHub Actions workflow run. It:

* Retrieves the start and end timestamps of the job
* Calculates the duration in minutes between these two timestamps
* Stores this duration along with the original timestamps in a dictionary, `job_info`
* Returns the populated `job_info` dictionary

**Key Implementation Details**

* Retrieves dates using date parsing (`date_parser.parse`)
* Calculates duration in minutes from the timestamp difference ( converted to seconds)

**Notable Dependencies or Requirements**

* `date_parser`: assumed dependency for parsing timestamps into datetimes
* Expected input: `job` dictionary with keys "started_at" and "completed_at"
* Returns a dictionary (`job_info`) with keys "started_at", "completed_at", and "duration".


key: chroma:document
string_value:
CODE:

def get_job_time(workflow_run_id, token=None):
    """Extract time info for all jobs in a GitHub Actions workflow run"""

    headers = None
    if token is not None:
        headers = {"Accept": "application/vnd.github+json", "Authorization": f"Bearer {token}"}

    url = f"https://api.github.com/repos/huggingface/transformers/actions/runs/{workflow_run_id}/jobs?per_page=100"
    result = requests.get(url, headers=headers).json()
    job_time = {}

    try:
        job_time.update({job["name"]: extract_time_from_single_job(job) for job in result["jobs"]})
        pages_to_iterate_over = math.ceil((result["total_count"] - 100) / 100)

        for i in range(pages_to_iterate_over):
            result = requests.get(url + f"&page={i + 2}", headers=headers).json()
            job_time.update({job["name"]: extract_time_from_single_job(job) for job in result["jobs"]})

        return job_time
    except Exception:
        print(f"Unknown error, could not fetch links:\n{traceback.format_exc()}")



SUMMARY:
**Function Summary**

`get_job_time`: Extracts time information for all jobs in a GitHub Actions workflow run.

**Primary Purpose:** Retrieves and extracts time info from multiple GitHub Actions workflow jobs.

**Implementation Details:**

- Uses `requests` to fetch GitHub API data
- Iterates through 2 pages of jobs (due to pagination) to collect time info
- Utilizes `extract_time_from_single_job` function (not shown in this snippet)
- Handles exceptions and prints error messages if an issue occurs

**Design Choices:**

* Uses Bearer token authentication for API requests
* Sets a page size of 100 jobs per request (per GitHub's rate limiting guidelines)

**Notable Dependencies:**

* `requests`
* `math` (for calculating total pages to iterate over)
* `extract_time_from_single_job` (not shown in this snippet, assumed to be implemented elsewhere)


key: chroma:document
string_value:
CODE:

def get_last_stable_minor_release():
    # Get the last stable release of transformers
    url = "https://pypi.org/pypi/transformers/json"
    release_data = requests.get(url).json()

    # Find the last stable release of of transformers (version below current version)
    major_version, minor_version, patch_version, _ = current_version.split(".")
    last_major_minor = f"{major_version}.{int(minor_version) - 1}"
    last_stable_minor_releases = [
        release for release in release_data["releases"] if release.startswith(last_major_minor)
    ]
    last_stable_release = sorted(last_stable_minor_releases, key=version.parse)[-1]



SUMMARY:
**Summary:**

The `get_last_stable_minor_release` function retrieves the last stable minor release of the `transformers` library by querying the PyPI JSON API. It compares the current major version to determine the last stable major and minor version, then filters releases to find those with versions below this threshold.

**Key Implementation Details:**

* Fetches `transformers` Pypi JSON data using `requests`
* Compares current version to find last stable minor release
* Uses `version.parse()` to sort and compare release versions

**Notable Dependencies/Requirements:**

* `requests` library for HTTP client
* `pyperpapi` library (`version.parse()`) for parsing release versions


key: chroma:document
string_value:
CODE:

def build_tip_message(last_stable_release):
    return (
        """
<Tip warning={true}>

This model is in maintenance mode only, we don't accept any new PRs changing its code.
"""
        + f"""If you run into any issues running this model, please reinstall the last version that supported this model: v{last_stable_release}.
You can do so by running the following command: `pip install -U transformers=={last_stable_release}`.

</Tip>"""


SUMMARY:
**Function Summary: build_tip_message**

* **Purpose**: Generates a maintenance mode warning message for a machine learning model, indicating that new PRs changing the code are not accepted.
	+ Primary purpose: Warn users about model maintenance mode and provide instructions to reinstall previous version.

**Key Implementation Details:**
- Uses f-string formatting to insert `last_stable_release` into the message.
- Includes a Git installation command for reusing the last stable version supported by the model (Transformer vX.X).

**Notable Dependencies or Requirements:** None mentioned explicitly.


key: chroma:document
string_value:
CODE:

def insert_tip_to_model_doc(model_doc_path, tip_message):
    tip_message_lines = tip_message.split("\n")

    with open(model_doc_path, "r") as f:
        model_doc = f.read()

    # Add the tip message to the model doc page directly underneath the title
    lines = model_doc.split("\n")

    new_model_lines = []
    for line in lines:
        if line.startswith("# "):
            new_model_lines.append(line)
            new_model_lines.extend(tip_message_lines)
        else:
            new_model_lines.append(line)

    with open(model_doc_path, "w") as f:


SUMMARY:
**Summary:**

`insert_tip_to_model_doc` function appends a tip message below the title section of an existing Markdown document file.

**Key Details:**

* Reads an input Markdown document file (`model_doc_path`) and tip message (`tip_message`)
* Splits the document into lines, separating those with title markers (`# `) from others
* Inserts the tip message lines below the original title lines, preserving blank line separation

**Implementation Notes:**

* Uses `file read/write` operations to modify an external file resource.

**External Dependencies: None specified. No known inherited or implemented interfaces are assumed to be used in this code snippet.**


key: chroma:document
string_value:
CODE:

def get_model_doc_path(model: str) -> Tuple[Optional[str], Optional[str]]:
    # Possible variants of the model name in the model doc path
    model_names = [model, model.replace("_", "-"), model.replace("_", "")]

    model_doc_paths = [REPO_PATH / f"docs/source/en/model_doc/{model_name}.md" for model_name in model_names]

    for model_doc_path, model_name in zip(model_doc_paths, model_names):
        if os.path.exists(model_doc_path):
            return model_doc_path, model_name



SUMMARY:
**Summary:**

This function retrieves the path to an English model documentation file based on the provided `model` name. It checks for three possible variant names and searches for a non-empty documentation directory within the repository.

**Implementation Details:**

* The function returns two optional strings, (`model_doc_path`, `model_name`) if a matching documentation file exists.
* The search considers three variant names:
	1. Original model name
	2. Model name with hyphens instead of underscores
	3. Model name without hyphens
* It utilizes the `os` module for path existence checks.

**Notable Dependencies:**

* `REPO_PATH`: assumed to be a constants or an external variable referencing the repository's root directory.
* `os.path.exists()`
* `zip()` function


key: chroma:document
string_value:
CODE:

def extract_model_info(model):
    model_info = {}
    model_doc_path, model_doc_name = get_model_doc_path(model)
    model_path = REPO_PATH / f"src/transformers/models/{model}"

    if model_doc_path is None:
        print(f"Model doc path does not exist for {model}")
        return None
    model_info["model_doc_path"] = model_doc_path
    model_info["model_doc_name"] = model_doc_name

    if not os.path.exists(model_path):
        print(f"Model path does not exist for {model}")
        return None
    model_info["model_path"] = model_path



SUMMARY:
**Function Summary: extract_model_info**

**Purpose and Functionality:**
This function retrieves information about a given machine learning model, including its documentation path and file location.

**Key Implementation Details:**

* Retrieves `model` using the `get_model_doc_path` function to get its documentation path.
* Returns a dictionary containing the following keys:
	+ `model_doc_path`: Path to the model's documentation.
	+ `model_doc_name`: Name of the model's documentation.

**Notable Dependencies and Requirements:**

* `REPO_PATH`: A constant representing the repository path.
* `get_model_doc_path` function.
* External dependencies: This function relies on the existence of the `get_model_doc_path` function, which itself may depend on other external libraries.


key: chroma:document
string_value:
CODE:

def update_relative_imports(filename, model):
    with open(filename, "r") as f:
        filelines = f.read()

    new_file_lines = []
    for line in filelines.split("\n"):
        if line.startswith("from .."):
            new_file_lines.append(line.replace("from ..", "from ..."))
        else:
            new_file_lines.append(line)

    with open(filename, "w") as f:


SUMMARY:
**Summary**

`update_relative_imports()` function updates relative import paths in a Python file. It replaces `..` with `...` to update absolute imports on all lines. The updated file is written back to the original location.

**Key Implementation Details**

* Replaces `from ..` with `from ...` across all lines to convert relative imports to absolute ones.
* Uses `split("\n")` to split the file into individual lines for processing.

**Notable Dependencies/Requirements**

* None explicit dependencies. However, it's assumed that this function runs in a Python environment with necessary modules and libraries available.


key: chroma:document
string_value:
CODE:

def remove_copied_from_statements(model):
    model_path = REPO_PATH / f"src/transformers/models/{model}"
    for file in os.listdir(model_path):
        if file == "__pycache__":
            continue
        file_path = model_path / file
        with open(file_path, "r") as f:
            file_lines = f.read()

        new_file_lines = []
        for line in file_lines.split("\n"):
            if "# Copied from" in line:
                continue
            new_file_lines.append(line)

        with open(file_path, "w") as f:


SUMMARY:
**Summary**

This function, `remove_copied_from_statements`, removes commented lines starting with "# Copied from" from the specified model files. The main purpose is to clean upTransformer models.

**Key Implementation Details:**

* Iterates over lines in each file in the model directory, skipping lines containing "# Copied from".
* Uses `os` and `re` modules implicitly for listing directory contents and file reading/writing.
* Writes modified line-by-line to new file, omitting lines with the specified comment.

**Notable Dependencies/Requirements:**

* `REPO_PATH`: reference path (relative to current working directory) containing model files.
* `os`: file system operations.
* `re`: regular expression matching used implicitly for pattern searching.


key: chroma:document
string_value:
CODE:

def move_model_files_to_deprecated(model):
    model_path = REPO_PATH / f"src/transformers/models/{model}"
    deprecated_model_path = REPO_PATH / f"src/transformers/models/deprecated/{model}"

    if not os.path.exists(deprecated_model_path):
        os.makedirs(deprecated_model_path)

    for file in os.listdir(model_path):
        if file == "__pycache__":
            continue
        repo.git.mv(f"{model_path}/{file}", f"{deprecated_model_path}/{file}")

        # For deprecated files, we then need to update the relative imports


SUMMARY:
**Summary: Migrate Model Files to Deprecation**

### Purpose and Functionality

Migrates model files from `src/transformers/models/{model}` to `src/transformers/models/deprecated/{model}`.

### Key Implementation Details

* Uses `repo.git.mv` to rename files, updating them at the new location.
* Calls `update_relative_imports` to update relative imports for deprecated files.

### Notable Dependencies or Requirements

* `REPO_PATH`: Base path of the repository
* `os.path.exists`, `os.makedirs`, `repo.git.mv`
* `update_relative_imports`

Note: The exact implementation details may vary based on the `update_relative_imports` function, which is not shown in this snippet.


key: chroma:document
string_value:
CODE:

def delete_model_tests(model):
    tests_path = REPO_PATH / f"tests/models/{model}"

    if os.path.exists(tests_path):


SUMMARY:
**Function Summary: `delete_model_tests`**

 Deletes model test files from the repository, removing related Git tracking.

**Main Purpose and Functionality:**
The primary purpose is to delete model test files at the specified path if they exist. It utilizes the Git command-line interface (`repo.git.rm`) to remove these files and their associated Git tracking.

**Key Implementation Details:**

* Uses `os.path.exists` to check for file existence.
* Utilizes `REPO_PATH / f"tests/models/{model}"` as the test path, referencing the `REPO_PATH` constant.
* Executes `repo.git.rm("-r", tests_path)` to remove the files and their tracking using Git.

**Notable Dependencies or Requirements:**

* References `repo`, an existing Git repository object.
* Utilizes external tool, `os`.
* `REPO_PATH`, a constant referencing the repository path.


key: chroma:document
string_value:
CODE:

def get_line_indent(s):


SUMMARY:
**Summary:**

Calculates the line indentation by subtracting the length of the stripped leading spaces from the total string length.

**Key Implementations:**

* `lstrip()` method is used to remove trailing whitespace characters
* Function takes a single argument `s` (string)

**Notable Dependencies/Requirements:**

* None mentioned

This function is used to calculate and return the indentation line in a given string. The implementation relies on the built-in `lstrip()` method, providing a concise and efficient solution for checking leading whitespace characters.


key: chroma:document
string_value:
CODE:

def update_main_init_file(models):
    """
    Replace all instances of model.model_name with model.deprecated.model_name in the __init__.py file

    Args:
        models (List[str]): The models to mark as deprecated
    """
    filename = REPO_PATH / "src/transformers/__init__.py"
    with open(filename, "r") as f:
        init_file = f.read()

    # 1. For each model, find all the instances of model.model_name and replace with model.deprecated.model_name
    for model in models:
        init_file = init_file.replace(f'models.{model}"', f'models.deprecated.{model}"')
        init_file = init_file.replace(f"models.{model} import", f"models.deprecated.{model} import")

    with open(filename, "w") as f:
        f.write(init_file)

    # 2. Resort the imports


SUMMARY:
**Function Summary: `update_main_init_file`**

**Primary Purpose:** Replaces deprecated model names in `__init__.py` file with new, deprecated versions.

**Key Implementation Details:**

* Replaces instances of `model.model_name` with `model.deprecated.model_name` in the `__init__.py` file.
* Also updates import statements to use the deprecated module name.

**Notable Dependencies and Requirements:**

* `REPO_PATH`: The absolute path to the repository's source directory (`"src/transformers/__init__.py"`).
* Uses an external function `sort_imports_in_all_inits(check_only=False)` (not shown in the provided code snippet).


key: chroma:document
string_value:
CODE:

def remove_model_references_from_file(filename, models, condition):
    """
    Remove all references to the given models from the given file

    Args:
        filename (str): The file to remove the references from
        models (List[str]): The models to remove
        condition (Callable): A function that takes the line and model and returns True if the line should be removed
    """
    filename = REPO_PATH / filename
    with open(filename, "r") as f:
        init_file = f.read()

    new_file_lines = []
    for i, line in enumerate(init_file.split("\n")):
        if any(condition(line, model) for model in models):
            continue
        new_file_lines.append(line)

    with open(filename, "w") as f:


SUMMARY:
**Summary:**

`remove_model_references_from_file` function:

* Purpose: Removes references to specified models from a given file by applying a filter condition.
* Key Implementation Details:
	+ Reads the entire file into memory using `init_file = f.read()`.
	+ Iterates over each line, skipping those that match any model based on the `condition` function.
	+ Writes the new filtered file contents back to disk.
* Notable Dependencies and Requirements:
	+ Uses Python's built-in `open()` and `read/write` functions for file input/output.
	+ Assumes a `REPO_PATH` constant or variable defining the repository root path.
	+ Accepts a `condition` function with signature `Callable[[str, str], bool]`.

**RAG (Reusable Anonymous Code) System:**

```markdown
# remove_model_references_from_file
### Purpose
Removes references to specified models from a given file.

### Key Implementation Details

* Reads entire file into memory.
* Applies filter condition to line-model pairs.
* Writes new filtered file contents back to disk.

### Dependencies and Requirements

* Python built-in `open` and `read/write`.
* `REPO_PATH` constant or variable.
* `condition` function signature: `Callable[[str, str], bool]`.
```


key: chroma:document
string_value:
CODE:

def remove_model_config_classes_from_config_check(model_config_classes):
    """
    Remove the deprecated model config classes from the check_config_attributes.py file

    Args:
        model_config_classes (List[str]): The model config classes to remove e.g. ["BertConfig", "DistilBertConfig"]
    """
    filename = REPO_PATH / "utils/check_config_attributes.py"
    with open(filename, "r") as f:
        check_config_attributes = f.read()

    # Keep track as we have to delete comment above too
    in_special_cases_to_allow = False
    in_indent = False
    new_file_lines = []

    for line in check_config_attributes.split("\n"):
        indent = get_line_indent(line)
        if (line.strip() == "SPECIAL_CASES_TO_ALLOW = {") or (line.strip() == "SPECIAL_CASES_TO_ALLOW.update("):
            in_special_cases_to_allow = True

        elif in_special_cases_to_allow and indent == 0 and line.strip() in ("}", ")"):
            in_special_cases_to_allow = False

        if in_indent:
            if line.strip().endswith(("]", "],")):
                in_indent = False
            continue

        if in_special_cases_to_allow and any(
            model_config_class in line for model_config_class in model_config_classes
        ):
            # Remove comments above the model config class to remove
            while new_file_lines[-1].strip().startswith("#"):
                new_file_lines.pop()

            if line.strip().endswith("["):
                in_indent = True

            continue

        elif any(model_config_class in line for model_config_class in model_config_classes):
            continue

        new_file_lines.append(line)

    with open(filename, "w") as f:


SUMMARY:
**Function Summary:**

`remove_model_config_classes_from_config_check`

Removes deprecated model configuration classes from the `check_config_attributes.py` file.

**Purpose and Functionality:**

* Reads the contents of the specified Python file (`utils/check_config_attributes.py`)
* Iterates through each line, tracking indentation and special cases
* Removes comments above deprecated model config classes and rewrites the content in a new file

**Key Implementation Details:**

* Uses regex-free text processing to identify lines with model config classes
* Preserves indentation and formatting when rewriting the content
* Handles edge cases such as nested brackets and escaped newline characters

**Dependencies and Interfaces:**

* None, but may utilize external `REPO_PATH` constant in a larger context.
* Does not implement or inherit any specific interfaces.

**Method Signatures:**

* `remove_model_config_classes_from_config_check(model_config_classes)`: takes a list of string model config class names as input


key: chroma:document
string_value:
CODE:

def add_models_to_deprecated_models_in_config_auto(models):
    """
    Add the models to the DEPRECATED_MODELS list in configuration_auto.py and sorts the list
    to be in alphabetical order.
    """
    filepath = REPO_PATH / "src/transformers/models/auto/configuration_auto.py"
    with open(filepath, "r") as f:
        config_auto = f.read()

    new_file_lines = []
    deprecated_models_list = []
    in_deprecated_models = False
    for line in config_auto.split("\n"):
        if line.strip() == "DEPRECATED_MODELS = [":
            in_deprecated_models = True
            new_file_lines.append(line)
        elif in_deprecated_models and line.strip() == "]":
            in_deprecated_models = False
            # Add the new models to deprecated models list
            deprecated_models_list.extend([f'    "{model}", ' for model in models])
            # Sort so they're in alphabetical order in the file
            deprecated_models_list = sorted(deprecated_models_list)
            new_file_lines.extend(deprecated_models_list)
            # Make sure we still have the closing bracket
            new_file_lines.append(line)
        elif in_deprecated_models:
            deprecated_models_list.append(line)
        else:
            new_file_lines.append(line)

    with open(filepath, "w") as f:


SUMMARY:
**Function Summary:**

`add_models_to_deprecated_models_in_config_auto(models)`

Adds the input `models` to the list of deprecated models in `configuration_auto.py`, updating its contents and sorting the list alphabetically.

**Key Implementation Details:**

- Reads `DEPRECATED_MODELS` from `configuration_auto.py`.
- Appends new models to the list in alphabetical order.
- Updates the file with the modified content.

**Notable Dependencies/Requirements:**

- `REPO_PATH`: reference to a repository-specific path ( exact class/function name required)
- External dependency on:
  - `os` module for file system operations
  - `pathlib` module for handling file paths
  - `shlex` module (not explicitly used but possibly implied by the `split()` and `strip()` functions)

**Interface/Import Notes:**

- No explicit import statements are provided, suggesting that necessary modules and classes will be imported within this code.


key: chroma:document
string_value:
CODE:

def deprecate_models(models):
    # Get model info
    skipped_models = []
    models_info = defaultdict(dict)
    for model in models:
        single_model_info = extract_model_info(model)
        if single_model_info is None:
            skipped_models.append(model)
        else:
            models_info[model] = single_model_info

    model_config_classes = []
    for model, model_info in models_info.items():
        if model in CONFIG_MAPPING:
            model_config_classes.append(CONFIG_MAPPING[model].__name__)
        elif model_info["model_doc_name"] in CONFIG_MAPPING:
            model_config_classes.append(CONFIG_MAPPING[model_info["model_doc_name"]].__name__)
        else:
            skipped_models.append(model)
            print(f"Model config class not found for model: {model}")

    # Filter out skipped models
    models = [model for model in models if model not in skipped_models]

    if skipped_models:
        print(f"Skipped models: {skipped_models} as the model doc or model path could not be found.")
    print(f"Models to deprecate: {models}")

    # Remove model config classes from config check
    print("Removing model config classes from config checks")
    remove_model_config_classes_from_config_check(model_config_classes)

    tip_message = build_tip_message(get_last_stable_minor_release())

    for model, model_info in models_info.items():
        print(f"Processing model: {model}")
        # Add the tip message to the model doc page directly underneath the title
        print("Adding tip message to model doc page")
        insert_tip_to_model_doc(model_info["model_doc_path"], tip_message)

        # Remove #Copied from statements from model's files
        print("Removing #Copied from statements from model's files")
        remove_copied_from_statements(model)

        # Move the model file to deprecated: src/transfomers/models/model -> src/transformers/models/deprecated/model
        print("Moving model files to deprecated for model")
        move_model_files_to_deprecated(model)

        # Delete the model tests: tests/models/model
        print("Deleting model tests")
        delete_model_tests(model)

    # # We do the following with all models passed at once to avoid having to re-write the file multiple times
    print("Updating __init__.py file to point to the deprecated models")
    update_main_init_file(models)

    # Remove model references from other files
    print("Removing model references from other files")
    remove_model_references_from_file(
        "src/transformers/models/__init__.py", models, lambda line, model: model == line.strip().strip(",")
    )
    remove_model_references_from_file(
        "utils/slow_documentation_tests.txt", models, lambda line, model: "/" + model + "/" in line
    )
    remove_model_references_from_file("utils/not_doctested.txt", models, lambda line, model: "/" + model + "/" in line)

    # Add models to DEPRECATED_MODELS in the configuration_auto.py
    print("Adding models to DEPRECATED_MODELS in configuration_auto.py")


SUMMARY:
**Function Summary: deprecate_models**

### Purpose and Functionality:

The `deprecate_models` function is designed to deprecate specific models by updating their documentation, removing references, and refactoring their configurations. It iterates through a list of models, extracting information about each model from an `extract_model_info` function call. The function checks if the model has a configured class in the `CONFIG_MAPPING`, adds a tip message to the model's documentation page, removes redundant statements, moves the model files to the deprecated directory, deletes test files, and updates other files that reference the models.

### Key Implementation Details:

*   Iterates through a list of models using `models` as input.
*   Calls `extract_model_info(model)` for each model to gather information.
*   Checks if a configured class exists in `CONFIG_MAPPING` or by name within `CONFIG_MAPPING`.
*   Updates the last stable minor release version using `get_last_stable_minor_release()`.
*   Uses `build_tip_message(tip_message)`, `remove_model_config_classes_from_config_check(model_config_classes)`, and other functions to finalize model depreciation.
*   Iterates over models, processing each one individually using `insert_tip_to_model_doc`, `remove_copied_from_statements`, `move_model_files_to_deprecated`, `delete_model_tests`, and updates files like `update_main_init_file` to point to the deprecated models.

### Dependencies and Requirements:

*   Uses `defaultdict` from Python's standard library for creating default dictionaries.
*   References `CONFIG_MAPPING` (a dictionary of model configurations) at least once during execution, which must be defined elsewhere in the codebase.
*   Calls functions with external dependencies like `remove_model_references_from_file`, `update_main_init_file`, and others that reference files or classes not specified within this function.

### Notable Interfaces:

The function provides a few notable interfaces to other components of the system:
    
*   `CONFIG_MAPPING`: A dictionary mapping model names or documentation lines to their respective configuration classes.
*   External dependencies such as `update_main_init_file`, `remove_model_references_from_file` (referencing specific files or directories), and `build_tip_message`.


key: chroma:document
string_value:
CODE:

def sort_auto_mapping(fname: str, overwrite: bool = False) -> Optional[bool]:
    """
    Sort all auto mappings in a file.

    Args:
        fname (`str`): The name of the file where we want to sort auto-mappings.
        overwrite (`bool`, *optional*, defaults to `False`): Whether or not to fix and overwrite the file.

    Returns:
        `Optional[bool]`: Returns `None` if `overwrite=True`. Otherwise returns `True` if the file has an auto-mapping
        improperly sorted, `False` if the file is okay.
    """
    with open(fname, "r", encoding="utf-8") as f:
        content = f.read()

    lines = content.split("\n")
    new_lines = []
    line_idx = 0
    while line_idx < len(lines):
        if _re_intro_mapping.search(lines[line_idx]) is not None:
            # Start of a new mapping!
            indent = len(re.search(r"^(\s*)\S", lines[line_idx]).groups()[0]) + 8
            while not lines[line_idx].startswith(" " * indent + "("):
                new_lines.append(lines[line_idx])
                line_idx += 1

            blocks = []
            while lines[line_idx].strip() != "]":
                # Blocks either fit in one line or not
                if lines[line_idx].strip() == "(":
                    start_idx = line_idx
                    while not lines[line_idx].startswith(" " * indent + ")"):
                        line_idx += 1
                    blocks.append("\n".join(lines[start_idx : line_idx + 1]))
                else:
                    blocks.append(lines[line_idx])
                line_idx += 1

            # Sort blocks by their identifiers
            blocks = sorted(blocks, key=lambda x: _re_identifier.search(x).groups()[0])
            new_lines += blocks
        else:
            new_lines.append(lines[line_idx])
            line_idx += 1

    if overwrite:
        with open(fname, "w", encoding="utf-8") as f:
            f.write("\n".join(new_lines))
    else:


SUMMARY:
**Function Summary: `sort_auto_mapping`**

The `sort_auto_mapping` function sorts the auto-mappings in a given file.

### Purpose and Functionality:

Automatically rearranges indented lines containing mappings in the specified file to adhere to a specific format. Compares and returns whether the original formatting is preserved upon sorting (when `overwrite=False`) or if changes were made for overwriting (`overwrite=True`).

### Key Implementation Details:

1.  **Regex-based Pattern Matching**:
    *   `_re_intro_mapping.search()`: Identifies the start of a new mapping in lines.
2.  **Blocks and Lines Processing**:
    *   `indentation calculation`: Calculates indentation length before processing each mapping block.
    *   `block decomposition` (`if lines[line_idx].strip() == "(":`): Tackles blocks divided across multiple lines with indentation matching the first line's formatting.
3.  **Sorting Algorithm**:
    *   `blocks sorted by ids` (`sorted(blocks, key=lambda x: _re_identifier.search(x).groups()[0])`): Orders and groups mappings according to their unique identifiers.

### Notable Dependencies or Requirements:

1.  `re` module (regular expressions) for pattern matching.
2.  External file operations via `open` function with UTF-8 encoding.

**API Reference:**

*   `_re_intro_mapping`, `_re_identifier`: Externally defined regular expression patterns used within the sorting logic, not part of this summary but crucial for implementation context.

### Example Usage:
```python
result = sort_auto_mapping('path/to/file.txt')
print(result)  # Returns 'None' if overwrite is False and file has been properly sorted, False otherwise.
```


key: chroma:document
string_value:
CODE:

def sort_all_auto_mappings(overwrite: bool = False):
    """
    Sort all auto mappings in the library.

    Args:
        overwrite (`bool`, *optional*, defaults to `False`): Whether or not to fix and overwrite the file.
    """
    fnames = [os.path.join(PATH_TO_AUTO_MODULE, f) for f in os.listdir(PATH_TO_AUTO_MODULE) if f.endswith(".py")]
    diffs = [sort_auto_mapping(fname, overwrite=overwrite) for fname in fnames]

    if not overwrite and any(diffs):
        failures = [f for f, d in zip(fnames, diffs) if d]
        raise ValueError(
            f"The following files have auto mappings that need sorting: {', '.join(failures)}. Run `make style` to fix"
            " this."


SUMMARY:
**Summary**

Function `sort_all_auto_mappings`: Sorts all auto mappings in the library, fixing inconsistencies if required. 

### Purpose
Synchronizes Python file auto mappings within a module.

### Implementation Details

- Fetches relevant files using list comprehension with path manipulation (`os.path.join`).
- Iterates through each file, calls `sort_auto_mapping` to sort its mappings.
- Checks for non-overwritten changes after sorting; raises an error if updates are detected.


key: chroma:document
string_value:
CODE:

class CustomTokenizerFast(BertTokenizerFast):
    slow_tokenizer_class = CustomTokenizer


SUMMARY:
**Summary:**

This class `CustomTokenizerFast` inherits from `BertTokenizerFast`. It overrides the default tokenizer by specifying its own tokenizer class as `SlowTokenizer`. The primary purpose is to enable the use of a custom, slow tokenization mechanism in place of the FastTokenizer.

**Key Implementation Details:**

* Inherited Class: `BertTokenizerFast`
* Implemented Class: `CustomTokenizer` (specifically assigned as `slow_tokenizer_class`)
* No additional functionality or method implementations provided; primarily serves as an override for the default tokenizer configuration.


key: chroma:document
string_value:
CODE:

class CustomTokenizer(BertTokenizer):


SUMMARY:
**Summary**

Class: `CustomTokenizer`
Inherits from: `BertTokenizer`

**Purpose and Functionality**: Creates a custom version of the BERT tokenizer.

**Key Implementation Details**:

* No specific tokenization algorithms or patterns are defined, inheriting functionality from `BertTokenizer`.
* No method signatures or interfaces are overridden by default.
 

**Notable Dependencies/Requirements**:
* Inherits from: `BertTokenizer`
* External dependencies not specified.

No key implementation details or interface implementations to note.


key: chroma:document
string_value:
CODE:

class CustomModel(PreTrainedModel):
    config_class = CustomConfig

    def __init__(self, config):
        super().__init__(config)
        self.linear = torch.nn.Linear(config.hidden_size, config.hidden_size)

    def forward(self, x):
        return self.linear(x)

    def _init_weights(self, module):


SUMMARY:
**Summary:**

The `CustomModel` class is a custom PyTorch model derived from `PreTrainedModel`. It uses the `CustomConfig` configuration.

**Main Purpose and Functionality:**
It defines a basic neural network architecture with one linear layer, where the input `x` passes through this layer to produce an output.

**Key Implementation Details:**

*   A linear layer (`self.linear`) with input/ouput size of `config.hidden_size`.
*   The `forward` method applies the linear transformation to the input `x`.

**Notable Dependencies and Requirements:**

*   Import from PyTorch (`torch.nn.Linear`).
*   Inheritance from `PreTrainedModel`.


key: chroma:document
string_value:
CODE:

class NoSuperInitModel(PreTrainedModel):
    config_class = NoSuperInitConfig

    def __init__(self, config):
        super().__init__(config)
        self.linear = torch.nn.Linear(config.attribute, config.attribute)

    def forward(self, x):
        return self.linear(x)

    def _init_weights(self, module):


SUMMARY:
**Summary:**

The `NoSuperInitModel` class is a PyTorch-based pre-trained model that extends the `PreTrainedModel` class. It defines:

* A linear layer with a configuration attribute `config.attribute`.
* A forward pass through the linear layer, mapping input `x` to its output.
* An `_init_weights` method that sets no weights (implies super-init behavior).

**Key Details:**

* **Linear Layer**: `self.linear = torch.nn.Linear(config.attribute, config.attribute)`
* **Forward Pass**: `return self.linear(x)`
* **_init_weights Method**: Implicitly inherits from superclass's behavior

**Dependencies/Requirements:**

* PyTorch (`torch`)
* `PreTrainedModel` class (inherited)
* `NoSuperInitConfig` configuration class ( referenced)


key: chroma:document
string_value:
CODE:

class CustomImageProcessor(CLIPImageProcessor):


SUMMARY:
**Code Summary:**

*   Class Name: `CustomImageProcessor`
*   Inheritance: Extends `CLIPImageProcessor` to create a custom image processor.

**Additional Details:**

*   No specific implementation details are provided, as the class solely inherits from `CLIPImageProcessor`.


key: chroma:document
string_value:
CODE:

def softmax(outputs):
    maxes = np.max(outputs, axis=-1, keepdims=True)
    shifted_exp = np.exp(outputs - maxes)


SUMMARY:
**Concise Summary**

This function implements the softmax activation function. It takes an input array `outputs` of shape `(num_classes, batch_size)` and returns a normalized value for each class.

The primary purpose is to prevent output values from growing exponentially large by normalizing them over their maximum value across classes.

**Key Implementation Details:**

- Uses NumPy's vectorized operations to calculate the softmax output.
- Employs shift-invariant exponentiation (`outputs - maxes`) to mitigate numerical instability issues.

**Notable Dependencies and Requirements:**

* `numpy` (imported as `np` for array operations)
* `outputs` input should be an array of shape `(num_classes, batch_size)` containing output values.


key: chroma:document
string_value:
CODE:

class PairClassificationPipeline(Pipeline):
    def _sanitize_parameters(self, **kwargs):
        preprocess_kwargs = {}
        if "second_text" in kwargs:
            preprocess_kwargs["second_text"] = kwargs["second_text"]
        return preprocess_kwargs, {}, {}

    def preprocess(self, text, second_text=None):
        return self.tokenizer(text, text_pair=second_text, return_tensors=self.framework)

    def _forward(self, model_inputs):
        return self.model(**model_inputs)

    def postprocess(self, model_outputs):
        logits = model_outputs.logits[0].numpy()
        probabilities = softmax(logits)

        best_class = np.argmax(probabilities)
        label = self.model.config.id2label[best_class]
        score = probabilities[best_class].item()
        logits = logits.tolist()


SUMMARY:
**PairClassificationPipeline Class Summary**

* **Purpose:** This class implements a pipeline for pair classification tasks using a pre-trained model. It sanitizes input parameters, preprocesses text inputs with optional second text, and postprocesses model outputs to extract confidence scores and predicted labels.

* **Key Implementation Details:**
	+ `preprocess` method takes in text inputs and an optional second text, and uses the `tokenizer` to generate input tensors in a specific framework.
	+ `_forward` method passes the preprocessed input to the trained model for inference.
	+ `postprocess` method extracts attention scores from the model output, converts them to probabilities using softmax, and determines the predicted label by finding the class with the highest probability.

* **Notable Dependencies:**
	+ `tokenizer`: a tokenizer object used in the `preprocess` method
	+ `model`: a pre-trained model instance that takes in input tensors in the same framework as the pipeline
	+ `self.framework`: the specific framework used for tensor generation (e.g. PyTorch, TensorFlow)
	+ `softmax`, `logits`, and related functions: commonly used mathematical functions for probability calculations


key: chroma:document
string_value:
CODE:

class CustomConfig(PretrainedConfig):
    model_type = "custom"

    def __init__(self, attribute=1, **kwargs):
        self.attribute = attribute


SUMMARY:
**Summary**

`CustomConfig` is a custom configuration class for the `PretrainedConfig` base class. It defines a model type as "custom" and extends initialization from its parent with an additional `attribute` parameter, defaulting to 1.

**Key Implementation Details:**

* Customizes model type to `model_type = "custom"`
* Extends __init__ from `PretrainedConfig` with `attribute` parameter

**Notable Dependencies or Requirements:**
None explicitly listed. Inherited interfaces: `PretrainedConfig`.


key: chroma:document
string_value:
CODE:

class NoSuperInitConfig(PretrainedConfig):
    model_type = "custom"

    def __init__(self, attribute=1, **kwargs):


SUMMARY:
**Summary**

This `NoSuperInitConfig` class inherits from `PretrainedConfig`, defining a custom configuration for a specific model. The primary purpose is to initialize the configuration with an optional `attribute` parameter.

**Key Details**

* Inherited from: `PretrainedConfig`
* Model type: "custom"
* Initialization:
	+ Required attribute: `attribute=1` (default value)

**Notable Dependencies/Requirements**

* None explicitly listed, but assumes access to `PretrainedConfig` class and related dependencies.


key: chroma:document
string_value:
CODE:

class CustomProcessor(ProcessorMixin):
    feature_extractor_class = "AutoFeatureExtractor"


SUMMARY:
**Summary:**

This `CustomProcessor` class inherits from `ProcessorMixin` and defines two key attributes:

* `feature_extractor_class`: specifies the use of the `AutoFeatureExtractor` component.
* `tokenizer_class`: specifies the use of the `AutoTokenizer` component.

No additional implementation details or dependencies are defined for this class.


key: chroma:document
string_value:
CODE:

class CustomFeatureExtractor(Wav2Vec2FeatureExtractor):


SUMMARY:
**Summary**

A custom `CustomFeatureExtractor` class that inherits from the `Wav2Vec2FeatureExtractor` class, effectively extending the functionality of the parent class with no additional implementation.

**Key Implementation Details**

* Inherited from `Wav2Vec2FeatureExtractor`, implying an extension to the Wav2Vec2 model's feature extraction capabilities.
* No additional methods or attributes are defined, suggesting a focus on leveraging the existing `Wav2Vec2FeatureExtractor` functionality with no new additions.

**Notable Dependencies**

* `Wav2Vec2FeatureExtractor`: The parent class from which `CustomFeatureExtractor` inherits.

**Additional Requirements**

None.


